Sunday, April 18, 2021

Kubernetes for Developers #13: Effective way of using K8 Readiness Probe

In General, Replication controller will keep running specified number of pods if application is crashed abruptly inside the Pod. However, there are situations where application might have crashed or deadlocked without terminating process. This is the place, where Kubernetes Health Probes comes into the picture.

Liveness Probe

Readiness Probe

The readiness probe determines when a container is ready to start accepting traffic. It invokes the probe periodically and If a pod reports that it is not ready, it is removed from the service. If the Pod then becomes ready again, it will be re-added automatically.

Unlike liveness probes, if a container fails the readiness check, it will not be restarted. The Pod will be detached from the service endpoint. Whenever Pod is ready to accept request, it will be attached to the Service endpoint again.

Liveness probes keep pods healthy by restarting unhealthy containers. whereas readiness probes make sure that only pods that are ready to serve requests receive them.

Kubernetes can probe the container in three different ways.

1. HTTP GET probe

It performs an HTTP GET request on the container’s REST API resource. If the probe receives a response code between 2xx to 3xx is considered successful. If Http server returns different response code (i.e. other than 2xx to 3xx) or if it doesn’t respond at all, the probe is considered a failure and pod will be detached from the service endpoint.

apiVersion: v1
kind: Pod
metadata:
  name: readiness-http
spec:
  containers:
    - name: readiness
      image: luksa/kubia-unhealthy
      ports:
        - containerPort: 8080
      readinessProbe:
        httpGet:
          path: /
          port: 8080
        initialDelaySeconds: 3
        periodSeconds: 3
        failureThreshold: 3

// run the pod yaml file
> kubectl apply -f ./liveness-http-test.yaml

// view running pod details
> kubectl get po liveness-http-test

// view Pod complete details
> kubectl describe po liveness-http-test

// view previous terminated container logs
> kubectl logs liveness-http-test --previous

2.  TCP Socket probe

It opens a TCP connection to the specified port of the container. If the connection is established successfully, the probe is successful. Otherwise, the pod will be detached from the service endpoint.
apiVersion: v1
kind: Pod
metadata:
  name: readiness-tcp
spec:
  containers:
    - name: goproxy
      image: k8s.gcr.io/goproxy:0.1
      ports:
        - containerPort: 8080
      readinessProbe:
        tcpSocket:
          port: 8080
        initialDelaySeconds: 5
        periodSeconds: 3

3. Exec probe

It executes a specified command inside the container and checks the command’s exit status code. If the status code is 0, the probe is successful. Otherwise, the pod will be detached from the service endpoint.
apiVersion: v1
kind: Pod
metadata:
  name: readiness-exec
spec:
  containers:
    - name: readiness
      image: k8s.gcr.io/busybox
      args:
        - /bin/sh
        - -c
        - touch /tmp/healthy; sleep 30; rm -rf /tmp/healthy; sleep 600
      readinessProbe:
        exec:
          command:
            - cat
            - /tmp/healthy
        initialDelaySeconds: 5
        periodSeconds: 5

When the container starts, it executes this command:
/bin/sh -c "touch /tmp/healthy; sleep 30; rm -rf /tmp/healthy; sleep 600"

For the first 30 seconds of the container's life, there is a /tmp/healthy file. So, during the first 30 seconds, the command cat /tmp/healthy returns a success code. After 30 seconds, cat /tmp/healthy returns a failure code.

Effective Readiness probe check

you should always define a Readiness probe. If your application takes too long to start listening for incoming connections, all the client requests hitting to this Pod will fail until an application is ready to accept.
  • Configure the Readiness probe to perform request on a specific URL path (i.e. /ready)
  • Make sure HTTP GET /ready does not require validation or authentication
  • Liveness probes do not wait for readiness probes to succeed. Both probes run parallel. If you want to wait before executing a liveness probe you should use initialDelaySeconds.
  • It is highly useful if your application needs to load lot of configuration/data during startup.
  • Readiness and liveness probes can be used in parallel for the same container. Using both can ensure that traffic does not reach a container that is not ready for it, and that containers are restarted when they fail.
  • liveness Prob failureThreshold default value is 3. So, it is not mandatory to specify again in pod yaml file.
  • Do not set the same specification for Liveness and Readiness Probe
  • Try to avoid "exec" probes as there are known problems with them

Kubernetes for Developers Journey.

Happy Coding :)
Posted by at No comments:
Labels: , , ,

Sunday, April 11, 2021

Kubernetes for Developers #12: Effective way of using K8 Liveness Probe

In General, Replication controller will keep running specified number of pods if application is crashed abruptly inside the Pod. However, there are situations where application might have crashed or deadlocked without terminating process. This is the place, where Kubernetes Health Probes comes into the picture.

Liveness Probe

Kubernetes checks if the container is alive through Liveness Probe and it will let kubelet to know when to restart a container. We can specify a liveness probe for each container in the pod specification. Kubernetes will periodically execute the liveness probe and restart the container if the probe fails.

Kubernetes can probe the container in three different ways.

1. HTTP GET probe

It performs an HTTP GET request on the container’s REST API resource. If the probe receives a response code between 2xx to 3xx is considered successful. If Http server returns different response code (i.e. other than 2xx to 3xx) or if it doesn’t respond at all, the probe is considered a failure and the container will be restarted by kubelet.

apiVersion: v1
kind: Pod
metadata:
  name: liveness-http
spec:
  containers:
    - name: liveness
      image: luksa/kubia-unhealthy
      ports:
        - containerPort: 8080
      livenessProbe:
        httpGet:
          path: /
          port: 8080
        initialDelaySeconds: 3
        periodSeconds: 3
        failureThreshold: 3

As per code written in the specified node.js app, Http GET request returns response code as 500 after each fifth request. So, Pod will restart automatically after 3 consecutive failures.

// run the pod yaml file
> kubectl apply -f ./liveness-http-test.yaml

// view running pod details
> kubectl get po liveness-http-test

// view Pod complete details
> kubectl describe po liveness-http-test

// view previous terminated container logs
> kubectl logs liveness-http-test --previous

Always set an initailDelaySeconds based on your application startup time. If you do not set the initial delay, liveness prob check starts immediately before application accept the requests and which leads to probe failing, eventually Pod restarts in infinite loop.

2.  TCP Socket probe

It opens a TCP connection to the specified port of the container. If the connection is established successfully, the probe is successful. Otherwise, the container will be restarted by kubelet.
apiVersion: v1
kind: Pod
metadata:
  name: liveness-tcp
spec:
  containers:
    - name: goproxy
      image: k8s.gcr.io/goproxy:0.1
      ports:
        - containerPort: 8080
      livenessProbe:
        tcpSocket:
          port: 8080
        initialDelaySeconds: 3
        periodSeconds: 3

3. Exec probe

It executes a specified command inside the container and checks the command’s exit status code. If the status code is 0, the probe is successful. Otherwise, the container will be restarted by kubelet.
apiVersion: v1
kind: Pod
metadata:
  name: liveness-exec
spec:
  containers:
    - name: liveness
      image: k8s.gcr.io/busybox
      args:
        - /bin/sh
        - -c
        - touch /tmp/healthy; sleep 30; rm -rf /tmp/healthy; sleep 600
      livenessProbe:
        exec:
          command:
            - cat
            - /tmp/healthy
        initialDelaySeconds: 5
        periodSeconds: 5


When the container starts, it executes this command:
/bin/sh -c "touch /tmp/healthy; sleep 30; rm -rf /tmp/healthy; sleep 600"

For the first 30 seconds of the container's life, there is a /tmp/healthy file. So, during the first 30 seconds, the command cat /tmp/healthy returns a success code. After 30 seconds, cat /tmp/healthy returns a failure code.

Effective liveness probe check

you should always define a liveness probe. Without one, Kubernetes has no way of knowing whether your app is responding to HTTP requests. So, liveness probe helps us to restart the container when the application is unresponsive state.
  • Configure the liveness probe to perform request on a specific URL path (i.e. /health) 
  • Make sure HTTP GET /health does not require validation or authentication
  • Always check only internals of the app and should not depend on external services/dependencies.
  • liveness probe should not return a failure when the server cannot connect to the database. If the underlying cause is in the database itself, restarting the web server container will not fix the problem.
  • HTTP GET /health probe should not use many computational resources and should not take too long to complete. It supposed get response back in one second.
  • liveness Prob failureThreshold default value is 3. So, it is not mandatory to specify again in pod yaml file.
  • Do not set the same specification for Liveness and Readiness Probe
  • Try to avoid "exec" probes as there are known problems with them
  • Exit the process when uncaught exception occurs (i.e. kubelet will restart the container automatically) instead of signaling to liveness probe 
  • Liveness probe should be used as a recovery mechanism only when the process is not responsive.
Kubernetes for Developers Journey.

Happy Coding :)
Posted by at 2 comments:
Labels: , , ,
Subscribe to: Posts (Atom)