Coders Classroom: July 2021

Saturday, July 24, 2021

Kubernetes for Developers #20: Create Automated Tasks using Jobs and CronJobs

In general, we use K8 Deployment object for creating Pods and running continuously without stopping. However, there are cases where we want to run a task and terminate once it is completed. such as data backups, exporting logs, batch process, sending email etc.

Kubernetes Job object is best suitable to do these kind of tasks. It creates a Pod for given task and terminates successfully once task is completed.

Kubernetes Job creates a Pod to run the task
Pod should be stopped once the task is completed and must not run always as K8 Deployment. Hence, Pod restartPolicy must be set to “Never” or “OnFailure”
Kubernetes Job can create multiple Pods by configuring parallelism
Job will be scheduled to another worker node automatically in case of current worker node failure
Set Job “completions” attribute to create total number of pods at each run. Default value is 1
Set Job “parallelism” attribute to spin up total number of pods in parallel at each run. Default value is 1
Set Job “activeDeadlineSeconds” attribute to specify how long Job should wait for the Pod to finish. Pod will be terminated automatically if it is running beyond specified time.

Create Job with Single Pod

As per below yaml, single Pod will be trigged as we are setting completions and parallelism attribute values to 1

apiVersion: batch/v1
kind: Job
metadata:
  name: job-example
spec:
  completions: 1
  parallelism: 1
  activeDeadlineSeconds: 10
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: busybox
          image: k8s.gcr.io/busybox
          command:
            - echo
            - "Hello K8 Job"

Save above yaml content as "job.yaml" and run following kubectl command

// create job object from yaml file
$ kubectl apply -f job.yaml
job.batch/job-example created

// display all job objects
$ kubectl get jobs
NAME          COMPLETIONS   DURATION   AGE
job-example   1/1           6s         1m

// check the pod which is created by job
$ kubectl get po
NAME                    READY   STATUS              RESTARTS   AGE
job-example-pdchw       0/1     Completed           0          1m

// view logs from the pod
$ kubectl logs job-example-pdchw
Hello K8 Job

// delete the job
$ kubectl delete job/job-example
job.batch "job-example" deleted

Create Job with Multiple Pods Sequentially

As per below yaml, multiple Pods gets triggered sequentially one after another (i.e after completion of each Pod) until it reaches to 3 as we are setting completions: 3

First creates one pod, and when the pod completes, it creates second pod and so on, until it completes all 3 pods.

apiVersion: batch/v1
kind: Job
metadata:
  name: job-example-sequential
spec:
  completions: 3
  parallelism: 1
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: busybox
          image: k8s.gcr.io/busybox
          command:
            - echo
            - "Hello K8 Job"

Save above yaml content as "job-sequential.yaml" and run following kubectl command

// create job object from yaml file
$ kubectl apply -f job-sequential.yaml
job.batch/job-example-sequential created

// display all job objects
$ kubectl get job
NAME                    COMPLETIONS   DURATION   AGE
job-example-sequential   1/3           18s        25s

// 2nd Pod is creating after completion of 1st Pod
$ kubectl get pod
NAME                                  READY   STATUS              RESTARTS   AGE
pod/job-example-sequential-hl5zn       0/1     ContainerCreating   0          1s
pod/job-example-sequential-srf62       0/1     Completed           0          7s

// 3rd Pod is creating after completion of 2nd Pod
$ kubectl get pod
NAME                                  READY   STATUS              RESTARTS   AGE
pod/job-example-sequential-671zn       0/1     ContainerCreating   0          1s
pod/job-example-sequential-hl5zn       0/1     Completed           0          11s
pod/job-example-sequential-srf62       0/1     Completed           0          17s

// All three Pods are completed succefully
$ kubectl get pod
NAME                                  READY   STATUS              RESTARTS   AGE
pod/job-example-sequential-671zn       0/1     Completed           0          5s
pod/job-example-sequential-hl5zn       0/1     Completed           0          13s
pod/job-example-sequential-srf62       0/1     Completed           0          19s

Create Job with Multiple Pods in Parallel

As per below yaml, two Pods gets triggered in parallel as we are setting parallelism : 2

First creates two pods in parallel, and when any one pod completes, it creates third pod and so on, until it completes all 3 pods.

apiVersion: batch/v1
kind: Job
metadata:
  name: job-example-parallel
spec:
  completions: 3
  parallelism: 2
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: busybox
          image: k8s.gcr.io/busybox
          command:
            - echo
            - "Hello K8 Job"

Save above yaml content as "job-parallel.yaml" and run following kubectl command

// create job object from yaml file
$ kubectl apply -f job-parallel.yaml
job.batch/job-example-parallel created

// display all job objects
$ kubectl get job
NAME                    COMPLETIONS   DURATION   AGE
job-example-parallel     2/3           18s        25s

// Two Pods are triggered in parallel at same time
$ kubectl get pod
NAME                             READY   STATUS              RESTARTS   AGE
pod/job-example-parallel-g7jgx   0/1     ContainerCreating   0          1s
pod/job-example-parallel-vjl9w   0/1     ContainerCreating   0          1s

// 3rd Pod is creating after completion of one of the pod
$ kubectl get pod
NAME                            READY   STATUS              RESTARTS   AGE
pod/job-example-parallel-vjl9w   0/1     Completed           0          21s
pod/job-example-parallel-g7jgx   0/1     Completed           0          21s
pod/job-example-parallel-58v8j   0/1     ContainerCreating   0          11s

Create CronJob

CronJob is used for creating periodic and recurring tasks. It runs a job periodically on a given schedule, written in Cron format.

CronJob creates Job object from the jobTemplate property configured in the Cronjob yaml

# ┌───────────── minute (0 - 59)
# │ ┌───────────── hour (0 - 23)
# │ │ ┌───────────── day of the month (1 - 31)
# │ │ │ ┌───────────── month (1 - 12)
# │ │ │ │ ┌───────────── day of the week (0 - 6) (Sunday to Saturday;
# │ │ │ │ │                                   7 is also Sunday on some systems)
# │ │ │ │ │
# │ │ │ │ │
# * * * * *

apiVersion: batch/v1
kind: CronJob
metadata:
  name: cronjob-example
spec:
  schedule: "*/2 * * * *"
  jobTemplate:
    spec:
      completions: 1
      parallelism: 1
      template:
        spec:
          restartPolicy: Never
          containers:
            - name: busybox
              image: k8s.gcr.io/busybox
              command:
                - echo
                - "Hello K8 Job"

As per above yaml, job is scheduled for every 2 minutes.

// create cronjob object from yaml file
$ kubectl apply -f cronjob.yaml
cronjob.batch/cronjob-example created

// display all cronjob objects
$ kubectl get cronjob
NAME              SCHEDULE      SUSPEND   ACTIVE   LAST SCHEDULE   AGE
cronjob-example   */2 * * * *   False     0        11s             3m6s

// display pods created by cronjob
$ kubectl get pod
NAME                                 READY   STATUS              RESTARTS   AGE
pod/cronjob-example-27119250-khqs8   0/1     Completed           0          3m4s
pod/cronjob-example-27119251-4c5kc   0/1     Completed           0          2m4s

Kubernetes for Developers Journey.

Happy Coding :)

Saturday, July 17, 2021

Kubernetes for Developers #19: Manage app credentials using Kubernetes Secrets

In the previous article (Kubernetes for Developers #18: Manage app settings using Kubernetes ConfigMap) we discussed about Kubernetes Configmap which is used for storing non-sensitive information. Whereas Kubernetes Secrets used for storing application sensitive information i.e. application encryption keys, certificates, database credentials and etc.

It stores data as key-value pair.
It stored in memory and never saved to physical disk.
It helps us to separate application specific configurations from the code.
Each Pod uses Secrets values as environment variables or volumes.
It supports passing key value as base64 string or plain text.
Key-value pair must be part of “data” or “stringData” attribute
Key name must contain alphanumeric, dash (-), underscore (_), dot(.) only

base64 encoded key-value pair

We should use “data” attribute if we want to pass all values of keys as base64 encoded strings. This is useful if the key value has binary data. These key values will be converted to plain text automatically when the pod is consuming.

apiVersion: v1
kind: Secret
metadata:
  name: secret-db
data:
  db_name: "cmFtYQo="
  db_password: "cGFzc3dvcmQK"

Plain text key-value pair

We should use “stringData” attribute if we want to pass all values of keys as plain strings. However, internally it converted to based64 encoded string while creating.

apiVersion: v1
kind: Secret
metadata:
  name: secret-db
stringData:
  db_name: "rama"
  db_password: "password"

Types of Secret

Kubernetes provides different built-in secret types for handling common use cases. We use “Opaque” type for storing application related credentials.

Type	Description
Opaque	used for creating user-defined data
kubernetes.io/service-account-token	used for mapping service account token
kubernetes.io/dockercfg	used for storing credentials for accessing private docker registry images
kubernetes.io/basic-auth	used for storing credentials for basic authentication
kubernetes.io/ssh-auth	Used for storing SSH privatekey for SSH tunneling
kubernetes.io/tls	Used for storing TLS certificate details.

Create Secret

There are multiple ways to create K8 Secret object

from YAML file

apiVersion: v1
kind: Secret
type: Opaque
metadata:
  name: secret-db
data:
  db_name: "cmFtYQo="
  db_password: "cGFzc3dvcmQK"

As we are using data attribute, we should pass key values in the base64 encoded string format

Save above yaml content as "secret-db.yaml" and run following kubectl command

// create secret object from yaml file
$ kubectl apply -f secret-db.yaml
secret/secret-db created

// display all secret objects
$ kubectl get secret
NAME                  TYPE                                  DATA   AGE
secret-db             Opaque                                2      18s

// view secret object details using describe command
$ kubectl describe secret secret-db

// view secret object details as yaml file 
$ kubectl get secret secret-db -o yaml

// decore base64 string
$ echo "cmFtYQo=" | base64 --decode
rama

2. from literals

In this approach, we pass key-value pair as part of kubectl command directly.

// create secret object from literals
// syntax
// kubectl create secret <secret-type> <secret-name> --from-literal=<key_name>=<key_value>
// we should use "generic" for "Opaque" secret type
$ kubectl create secret generic secret-db-test --from-literal=db_name=rama --from-literal=db_password=password
secret/secret-db-test created

// display all secret objects
$ kubectl get secret
NAME                  TYPE                                  DATA   AGE
secret-db-test        Opaque                                2      6s

// view secret object details using describe command
$ kubectl describe secret secret-db-test

// view secret object details as yaml file 
$ kubectl get secret secret-db-test -o yaml

Passing Secret values to Pod

1. passing Secret values to container environment variables

In the below example, we are setting Pod container individual environment variables from “secret-db” object

apiVersion: v1
kind: Pod
metadata:
  name: secret-pod
spec:
  containers:
    - name: busybox
      image: k8s.gcr.io/busybox
      command: ["/bin/sh", "-c", "env"]
      env:
        - name: DATABASE_HOST
          value: "myrds.amazon.com"
        - name: DATABASE_USER_NAME
          valueFrom:
            secretKeyRef:
              name: secret-db
              key: db_name
        - name: DATABASE_PASSWORD
          valueFrom:
            secretKeyRef:
              name: secret-db
              key: db_password

save above yaml content as "secret-pod-valuefrom.yaml" and run following kubectl command

// create busybox Pod 
$ kubectl apply -f secret-pod-valuefrom.yaml
pod/secret-pod created

// check environment values from busybox Pod
// secret values will be decoded to plaintext automatically when Pod is consuming
$ kubectl logs pod/secret-pod | grep DATABASE
DATABASE_USER_NAME=rama
DATABASE_PASSWORD=password
DATABASE_HOST=myrds.amazon.com

2. passing complete Secret Object as environment variables to the container

In the below example, we are passing all "secret-db" object key-value pairs as container environment variables.

apiVersion: v1
kind: Pod
metadata:
  name: secret-pod-env
spec:
  containers:
    - name: busybox
      image: k8s.gcr.io/busybox
      command: ["/bin/sh", "-c", "env"]
      envFrom:
        - secretRef:
            name: secret-db

save above yaml content as "configmap-pod-envfrom.yaml" and run following kubectl command

// create busybox Pod 
$ kubectl apply -f secret-pod-envfrom.yaml
pod/secret-pod-env created

// check environment values from busybox Pod
$ kubectl logs pod/secret-pod-env | grep db
db_password=password
db_name=rama

3. Attaching Secret as container volume

In the below example,

we are first creating volumes by setting secretName property as "secret-db"
we are accessing this volume by setting volumeMounts property

apiVersion: v1
kind: Pod
metadata:
  name: secret-pod-volume
spec:
  containers:
    - name: busybox
      image: k8s.gcr.io/busybox
      command: ["/bin/sh", "-c", "ls /etc/secret/"]
      volumeMounts:
        - name: secret-volume
          mountPath: /etc/secret
  volumes:
    - name: secret-volume
      secret:
        secretName: secret-db

save above yaml content as "secret-pod-volume.yaml" and run following kubectl command

// create busybox Pod 
$ kubectl apply -f secret-pod-volume.yaml
pod/secret-pod-volume created

// check logs from busybox Pod
$ kubectl logs pod/secret-pod-volume
db_name
db_password

Kubernetes for Developers Journey.

Happy Coding :)

Friday, July 9, 2021

Kubernetes for Developers #18: Manage app settings using Kubernetes ConfigMap

In general, we use configuration files (i.e. web.config and .rc ) or environment files (.env) for separating application settings from the code. The same application settings can be implemented using Kubernetes ConfigMap Object in the K8 world.

It stores data as key-value pair.
It helps us to separate application environment specific configurations from the code.
Each Pod uses ConfigMap values as environment variables, volumes or command-line arguments.
Key name must contain alphanumeric, dash (-), underscore (_) and dot(.) only
It is not suitable for storing confidential data.

Create ConfigMap

There are multiple ways to create K8 ConfigMap object

from YAML file

apiVersion: v1
kind: ConfigMap
metadata:
  name: configmap-db
data:
  db_host: "mydb.rds.amazonaws.com"
  db_port: "3306"
  db_name: "testuser"

Here, we should specify key-value pair as data property in the K8 ConfigMap object

Save above yaml content as "configmapdb.yaml" and run following kubectl command

// create configmap object from yaml file
$ kubectl apply -f configmapdb.yaml
configmap/configmap-db created

// display all configmap objects
$ kubectl get configmap
NAME             DATA   AGE
configmap-db     3      2m

// view configmap object details using describe command
$ kubectl describe configmap configmap-db

// view configmap object details as yaml file 
$ kubectl get configmap configmap-db -o yaml

2. from literals

In this approach, we pass key-value pair as part of kubectl command directly.

// create configmap object from literals

// syntax
kubectl create configmap <configmap-name> --from-literal=<key_name>=<key_value>

$ kubectl create configmap configmap-ui --from-literal=app_name=mytestapp --from-literal=app_theme=bluetheme
configmap/configmap-ui created

// display all configmap objects
$ kubectl get configmap
NAME             DATA   AGE
configmap-db     3      134m
configmap-ui     2      19s

// view configmap object details using describe command
$ kubectl describe configmap configmap-ui

// view configmap object details as yaml file 
$ kubectl get configmap configmap-ui -o yaml

3. from an env-file

In this approach, we create an env file with list of environment variables and values

db_host=mydb.rds.amazonaws.com
db_port=3306
db_name=testuser

save above content as "configmap-db-env.txt" or "configmap-db-env.properties" and run following kubectl command

// create configmap object from env file
$ kubectl create configmap configmap-db-env --from-env-file=configmap-db-env.txt
configmap-db-env created

// display all configmap objects
$ kubectl get configmap
NAME               DATA   AGE
configmap-db       3      153m
configmap-db-env   3      6s
configmap-ui       2      18m

// view configmap object details using describe command
$ kubectl describe configmap configmap-db-env

// view configmap object details as yaml file 
$ kubectl get configmap configmap-db-env -o yaml

Passing ConfigMap values to Pod

1. passing ConfigMap values to container environment variables

In the below example, we are setting Pod container individual environment variables from “configmap-db” object

apiVersion: v1
kind: Pod
metadata:
  name: configmap-pod
spec:
  containers:
    - name: busybox
      image: k8s.gcr.io/busybox
      command: ["/bin/sh", "-c", "env"]
      env:
        - name: DATABASE_PORT
          value: "3306"
        - name: DATABASE_HOST
          valueFrom:
            configMapKeyRef:
              name: configmap-db
              key: db_host
        - name: DATABASE_NAME
          valueFrom:
            configMapKeyRef:
              name: configmap-db
              key: db_name

save above yaml content as "configmap-pod-valuefrom.yaml" and run following kubectl command

// create busybox Pod 
$ kubectl apply -f configmap-pod-valuefrom.yaml
pod/configmap-pod created

// check environment values from busybox Pod
$ kubectl logs pod/configmap-pod | grep DATABASE
DATABASE_PORT=3306
DATABASE_NAME=testuser
DATABASE_HOST=mydb.rds.amazonaws.com

2. passing complete ConfigMap Object as environment variables to the container

In the below example, we are passing all "configmap-db" object key-value pairs as container environment variables.

apiVersion: v1
kind: Pod
metadata:
  name: configmap-pod-env
spec:
  containers:
    - name: busybox
      image: k8s.gcr.io/busybox
      command: ["/bin/sh", "-c", "env"]
      envFrom:
        - configMapRef:
            name: configmap-db

save above yaml content as "configmap-pod-envfrom.yaml" and run following kubectl command

// create busybox Pod 
$ kubectl apply -f configmap-pod-envfrom.yaml
pod/configmap-pod-env created

// check environment values from busybox Pod
$ kubectl logs pod/configmap-pod-env | grep db
db_port=3306
db_name=testuser
db_host=mydb.rds.amazonaws.com

3. passing ConfigMap defined environment variables as container command arguments

In the below example, we are passing environment variables as container command arguments with $(environment_variable) syntax

apiVersion: v1
kind: Pod
metadata:
  name: configmap-pod-command
spec:
  containers:
    - name: busybox
      image: k8s.gcr.io/busybox
      command:
        [
          "/bin/echo",
          "dbhostname:  $(DATABASE_HOST) and dbport:  $(DATABASE_PORT)",
        ]
      env:
        - name: DATABASE_HOST
          valueFrom:
            configMapKeyRef:
              name: configmap-db
              key: db_host
        - name: DATABASE_PORT
          valueFrom:
            configMapKeyRef:
              name: configmap-db
              key: db_port

save above yaml content as "configmap-pod-command.yaml" and run following kubectl command

// create busybox Pod 
$ kubectl apply -f configmap-pod-command.yaml
pod/configmap-pod-command created

// check logs from busybox Pod
$ kubectl logs pod/configmap-pod-command
dbhostname:  mydb.rds.amazonaws.com and dbport:  3306

4. Attaching ConfigMap as container volume

In the below example,

we are first creating volumes by setting configMap name property as "configmap-db"
we are accessing this volume by setting volumeMounts property

apiVersion: v1
kind: Pod
metadata:
  name: configmap-pod-volume
spec:
  containers:
    - name: busybox
      image: k8s.gcr.io/busybox
      command: ["/bin/sh", "-c", "ls /etc/configmap/"]
      volumeMounts:
        - name: configmap-volume
          mountPath: /etc/configmap
  volumes:
    - name: configmap-volume
      configMap:
        name: configmap-db

save above yaml content as "configmap-pod-volume.yaml" and run following kubectl command

// create busybox Pod 
$ kubectl apply -f configmap-pod-volume.yaml
pod/configmap-pod-volume created

// check logs from busybox Pod
$ kubectl logs pod/configmap-pod-volume
db_host
db_name
db_port

Kubernetes for Developers Journey.

Happy Coding :)