Health Checking of Istio Services
Kubernetes liveness and readiness probes describes several ways to configure liveness and readiness probes:
The command approach works with no changes required, but HTTP requests, TCP probes, and gRPC probes require Istio to make changes to the pod configuration.
The health check requests to the liveness-http
service are sent by Kubelet.
This becomes a problem when mutual TLS is enabled, because the Kubelet does not have an Istio issued certificate.
Therefore the health check requests will fail.
TCP probe checks need special handling, because Istio redirects all incoming traffic into the sidecar, and so all TCP ports appear open. The Kubelet simply checks if some process is listening on the specified port, and so the probe will always succeed as long as the sidecar is running.
Istio solves both these problems by rewriting the application PodSpec
readiness/liveness probe,
so that the probe request is sent to the sidecar agent.
For HTTP and gRPC requests, the sidecar agent redirects the request to the application and strips the response body, only returning the response code. For TCP probes, the sidecar agent will then do the port check while avoiding the traffic redirection.
The rewriting of problematic probes is enabled by default in all built-in Istio configuration profiles but can be disabled as described below.
Liveness and readiness probes using the command approach
Istio provides a liveness sample that implements this approach. To demonstrate it working with mutual TLS enabled, first create a namespace for the example:
$ kubectl create ns istio-io-health
To configure strict mutual TLS, run:
$ kubectl apply -f - <<EOF
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
name: "default"
namespace: "istio-io-health"
spec:
mtls:
mode: STRICT
EOF
Next, change directory to the root of the Istio installation and run the following command to deploy the sample service:
$ kubectl -n istio-io-health apply -f <(istioctl kube-inject -f @samples/health-check/liveness-command.yaml@)
To confirm that the liveness probes are working, check the status of the sample pod to verify that it is running.
$ kubectl -n istio-io-health get pod
NAME READY STATUS RESTARTS AGE
liveness-6857c8775f-zdv9r 2/2 Running 0 4m
Liveness and readiness probes using the HTTP, TCP, and gRPC approach
As stated previously, Istio uses probe rewrite to implement HTTP, TCP, and gRPC probes by default. You can disable this feature either for specific pods, or globally.
Disable the probe rewrite for a pod
You can annotate the pod with sidecar.istio.io/rewriteAppHTTPProbers: "false"
to disable the probe rewrite option. Make sure you add the annotation to the
pod resource because it will be ignored
anywhere else (for example, on an enclosing deployment resource).
kubectl apply -f - <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
name: liveness-http
spec:
selector:
matchLabels:
app: liveness-http
version: v1
template:
metadata:
labels:
app: liveness-http
version: v1
annotations:
sidecar.istio.io/rewriteAppHTTPProbers: "false"
spec:
containers:
- name: liveness-http
image: docker.io/istio/health:example
ports:
- containerPort: 8001
livenessProbe:
httpGet:
path: /foo
port: 8001
initialDelaySeconds: 5
periodSeconds: 5
EOF
kubectl apply -f - <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
name: liveness-grpc
spec:
selector:
matchLabels:
app: liveness-grpc
version: v1
template:
metadata:
labels:
app: liveness-grpc
version: v1
annotations:
sidecar.istio.io/rewriteAppHTTPProbers: "false"
spec:
containers:
- name: etcd
image: registry.k8s.io/etcd:3.5.1-0
command: ["--listen-client-urls", "http://0.0.0.0:2379", "--advertise-client-urls", "http://127.0.0.1:2379", "--log-level", "debug"]
ports:
- containerPort: 2379
livenessProbe:
grpc:
port: 2379
initialDelaySeconds: 10
periodSeconds: 5
EOF
This approach allows you to disable the health check probe rewrite gradually on individual deployments, without reinstalling Istio.
Disable the probe rewrite globally
Install Istio using --set values.sidecarInjectorWebhook.rewriteAppHTTPProbe=false
to disable the probe rewrite globally. Alternatively, update the configuration map for the Istio sidecar injector:
$ kubectl get cm istio-sidecar-injector -n istio-system -o yaml | sed -e 's/"rewriteAppHTTPProbe": true/"rewriteAppHTTPProbe": false/' | kubectl apply -f -
Cleanup
Remove the namespace used for the examples:
$ kubectl delete ns istio-io-health