Extending Self-Signed Certificate Lifetime

5 minute read

Istio self-signed certificates have historically had a 1 year default lifetime. If you are using Istio self-signed certificates, you need to schedule regular root transitions before they expire. An expiration of a root certificate may lead to an unexpected cluster-wide outage.

To evaluate the lifetime remaining for your root certificate, please refer to the first step in the procedure below.

We provide the following procedure for you to do the root transition. Note that the Envoy instances will be hot restarted to reload the new root certificates, which may impact long-lived connections. For details about the impacts and how Envoy hot restart works, please refer to here and here.

Scenarios

If you are not currently using the mutual TLS feature in Istio and will not use it in the future, you are not affected and no action is required. You may choose to upgrade to 1.0.8, 1.1.8 or later versions to avoid this problem in the future.

If you are not currently using the mutual TLS feature in Istio and may use it in the future, you are recommended to follow the procedure listed below to upgrade.

If you are currently using the mutual TLS feature in Istio with self-signed certificates, please follow the procedure and check whether you will be affected.

Root transition procedure

Check when the root certificate expires:
Download this script on a machine that has kubectl access to the cluster.
```
$ wget https://raw.githubusercontent.com/istio/tools/master/bin/root-transition.sh
$ chmod +x root-transition.sh
$ ./root-transition.sh check
...
===YOU HAVE 30 DAYS BEFORE THE ROOT CERT EXPIRES!=====
```
Execute the remainder of the steps prior to root certificate expiration to avoid system outages.

Execute a root certificate transition:

During the transition, the Envoy sidecars may be hot-restarted to reload the new certificates. This may have some impact on your traffic. Please refer to Envoy hot restart and read this blog post for more details.

$ ./root-transition.sh transition
Create new ca cert, with trust domain as cluster.local
Wed Jun  5 19:11:15 PDT 2019 delete old ca secret
secret "istio-ca-secret" deleted
Wed Jun  5 19:11:15 PDT 2019 create new ca secret
secret/istio-ca-secret created
pod "istio-citadel-8574b88bcd-j7v2d" deleted
Wed Jun  5 19:11:18 PDT 2019 restarted Citadel, checking status
NAME                             READY     STATUS    RESTARTS   AGE
istio-citadel-8574b88bcd-l2g74   1/1       Running   0          3s
New root certificate:
Certificate:
    Data:
        ...
        Validity
            Not Before: Jun  6 03:24:43 2019 GMT
            Not After : Jun  3 03:24:43 2029 GMT
        Subject: O = cluster.local
        ...
Your old certificate is stored as old-ca-cert.pem, and your private key is stored as ca-key.pem
Please save them safely and privately.

Verify the new workload certificates are generated:

$ ./root-transition.sh verify
...
Checking the current root CA certificate is propagated to all the Istio-managed workload secrets in the cluster.
Root cert MD5 is 8fa8229ab89122edba73706e49a55e4c
Checking namespace: default
  Secret default.istio.default is updated.
  Secret default.istio.sleep is updated.
Checking namespace: istio-system
  Secret istio-system.istio.default is updated.
  ...
------All Istio keys and certificates are updated in secret!

If this command fails, wait a minute and run the command again. It takes some time for Citadel to propagate the certificates.

Upgrade to Istio 1.0.8, 1.1.8 or later:
To ensure the control plane components and Envoy sidecars all load the new certificates and keys, this step is mandatory.
Upgrade your control plane and istio-proxy sidecars to 1.0.8, 1.1.8 or later. Please follow the Istio upgrade procedure.
Verify the new workload certificates are loaded by Envoy:
You can verify whether an Envoy has received the new certificates. The following command shows an example to check the Envoy’s certificate for pod foo running in namespace bar.
```
$ kubectl exec -it foo -c istio-proxy -n bar -- curl http://localhost:15000/certs | head -c 1000
{
 "certificates": [
  {
   "ca_cert": [
      ...
      "valid_from": "2019-06-06T03:24:43Z",
      "expiration_time": ...
   ],
   "cert_chain": [
    {
      ...
    }
```
Please inspect the valid\_from value of the ca\_cert. If it matches the _Not_ _Before_ value in the new certificate as shown in Step 2, your Envoy has loaded the new root certificate.

Troubleshooting

Can I upgrade to 1.0.8, 1.1.8 or later first, and then do the root transition?

Yes, you can. You can upgrade to 1.0.8, 1.1.8 or later as normal. After that, follow the root transition steps and in Step 4, manually restart Galley, Pilot and sidecar-injector to ensure they load the new root certificates.

Why my workloads do not pick up the new certificates (in Step 5)?

Please make sure you have updated to 1.0.8, 1.1.8 or later for the istio-proxy sidecars in Step 4.

Why my Pilot does not work and logs “handshake error”?

This may because the Pilot is not using an Envoy sidecar, while the controlPlaneSecurity is enabled. In this case, restart both Galley and Pilot to ensure they load the new certificates. As an example, the following commands redeploy a pod for Galley / Pilot by removing a pod.

$ kubectl delete po <galley-pod> -n istio-system
$ kubectl delete po <pilot-pod> -n istio-system

How to check if Pilot has an Envoy sidecar

If the following command shows 1/1, that means your Pilot does not have an Envoy sidecar, otherwise, if it is showing 2/2, your Pilot is using an Envoy sidecar.

$ kubectl get po -l istio=pilot -n istio-system
istio-pilot-569bc6d9c-tfwjr   1/1     Running   0          11m

I can’t deploy new workloads with the sidecar-injector

This may happen if you did not upgrade to 1.0.8, 1.1.8 or later. Try to restart the sidecar injector. The sidecar injector will reload the certificate after the restart:

$ kubectl delete po -l istio=sidecar-injector -n istio-system
pod "istio-sidecar-injector-788bd8fc48-x9gdc" deleted