Citadel health checking

This task shows how to enable Kubernetes health checking for Citadel. Note this is an Alpha feature.

Since Istio 0.6, Citadel has a health checking feature that can be optionally enabled. By default, the normal Istio deployment process does not enable this feature. Currently, the health checking feature is able to detect the failures of Citadel CSR signing service, by periodically sending CSRs to the API. More health checking features are coming shortly.

Citadel contains a prober client module that periodically checks Citadel's status (currently only the health status of the gRPC server). If Citadel is healthy, the prober client updates the modification time of the health status file (the file is always empty). Otherwise, it does nothing. Citadel relies on a K8s liveness and readiness probe with command line to check the modification time of the health status file on the pod. If the file is not updated for a period, the probe will be triggered and Kubelet will restart the Citadel container.

Note: because Citadel health checking currently only monitors the health status of CSR service API, this feature is not needed if the production setup is not using the Istio Mesh Expansion (which requires the CSR service API).

Before you begin

Set up Istio by following the instructions in the quick start with global mutual TLS enabled:
```
$ kubectl apply -f install/kubernetes/istio-demo-auth.yaml
```
OR
Using Helm with global.mtls.enabled to true.

Starting with Istio 0.7, you can use authentication policy to configure mutual TLS for all/selected services in a namespace (repeated for all namespaces to get global setting). See authentication policy task

Deploying Citadel with health checking

Deploy Citadel with health checking enabled.

$ kubectl apply -f install/kubernetes/istio-citadel-with-health-check.yaml

Deploy the istio-citadel service so that the CSR service can be found by the health checker.

cat <<EOF | kubectl create -f -
apiVersion: v1
kind: Service
metadata:
  name: istio-citadel
  namespace: istio-system
  labels:
    istio: citadel
spec:
  ports:
    - port: 8060
  selector:
    istio: citadel
EOF

Verifying the health checker is working

Citadel will log the health checking results. Run the following in command line:

$ kubectl logs `kubectl get po -n istio-system | grep istio-citadel | awk '{print $1}'` -n istio-system

You will see the output similar to:

...
2018-02-27T04:29:56.128081Z     info    CSR successfully signed.
...
2018-02-27T04:30:11.081791Z     info    CSR successfully signed.
...
2018-02-27T04:30:25.485315Z     info    CSR successfully signed.
...

The log above indicates the periodic health checking is working. Observe that the health checking interval is about 15 seconds, which is the default health checking interval.

(Optional) Configuring the health checking

Optionally, adjust the health checking configuration to meet your own needs. Open the file install/kubernetes/istio-citadel-with-health-check.yaml, and locate the following lines.

...
  - --liveness-probe-path=/tmp/ca.liveness # path to the liveness health checking status file
  - --liveness-probe-interval=60s # interval for health checking file update
  - --probe-check-interval=15s    # interval for health status check
livenessProbe:
  exec:
    command:
    - /usr/local/bin/istio_ca
    - probe
    - --probe-path=/tmp/ca.liveness # path to the liveness health checking status file
    - --interval=125s               # the maximum time gap allowed between the file mtime and the current sys clock.
  initialDelaySeconds: 60
  periodSeconds: 60
...

The liveness-probe-path and probe-path are the path to the health status file, configured at Citadel and the prober; the liveness-probe-interval is the interval to update the health status file, if Citadel is healthy; the probe-check-interval is the interval for Citadel health checking. The interval is the maximum time elapsed since the last update of the health status file, for the prober to consider Citadel as healthy. initialDelaySeconds and periodSeconds are the initial delay and the probe running period.

Prolonging probe-check-interval will reduce the health checking overhead, but there will be a greater lagging for the prober to get notified on the unhealthy status. To avoid the prober restarting Citadel due to temporary unavailability, the interval on the prober can be configured to be more than N times of the liveness-probe-interval. This will allow the prober to tolerate N-1 continuously failed health checks.

Cleanup

To disable health checking on Citadel:

$ kubectl apply -f install/kubernetes/istio-demo-auth.yaml

To remove Citadel:

$ kubectl delete -f install/kubernetes/istio-citadel-with-health-check.yaml