Mutual TLS Migration
This task shows how to migrate your existing Istio services’ traffic from plain text to mutual TLS without breaking live traffic.
In the scenario where there are many services communicating over the network, it
may be desirable to gradually migrate them to Istio. During the migration, some services have Envoy
sidecars while some do not. For a service with a sidecar, if you enable
mutual TLS on the service, the connections from legacy clients (i.e., clients without
Envoy) will lose communication since they do not have Envoy sidecars and client certificates.
To solve this issue, Istio authentication policy provides a PERMISSIVE
mode to solve
this problem. When PERMISSIVE
mode is enabled, a service can take both HTTP
and mutual TLS traffic.
You can configure Istio services to send mutual
TLS traffic to that service while connections from legacy services will not
lose communication. Moreover, you can use the
Grafana dashboard to check which services are
still sending plaintext traffic to the service in PERMISSIVE
mode and choose to lock
them down once the migration is done.
Before you begin
Understand Istio authentication policy and related mutual TLS authentication concepts.
Read the authentication policy task to learn how to configure authentication policy.
Have a Kubernetes cluster with Istio installed, without global mutual TLS enabled (e.g use the demo configuration profile as described in installation steps, or set the
global.mtls.enabled
installation option to false).Ensure that your cluster is in
PERMISSIVE
mode before migrating to mutual TLS. Run the following command to check:$ kubectl get meshpolicy default -o yaml ... spec: peers: - mtls: mode: PERMISSIVE
If the output is the same as above, you can skip the next step.
Run the following command to enable
PERMISSIVE
mode for the cluster. In general, this operation does not cause any interruption to your workloads, but note the warning message below.$ kubectl apply -f - <<EOF apiVersion: "authentication.istio.io/v1alpha1" kind: "MeshPolicy" metadata: name: "default" spec: peers: - mtls: mode: PERMISSIVE EOF
The rest of this task is divided into two parts.
If you want to enable mutual TLS for your workloads one by one, refer to gradually enable mutual TLS for services, which describes the process using simple examples.
If you want to enable mutual TLS for the entire cluster, refer to globally enable mutual TLS for the cluster.
Option 1: gradually enable mutual TLS for services
In this section, you can try out the migration process by creating sample workloads and modifying the policies to enforce STRICT mutual TLS between the workloads.
Set up the cluster
Create the following namespaces and deploy httpbin and sleep with sidecars on both of them.
foo
bar
Create the following namespace and deploy sleep without a sidecar
legacy
$ kubectl create ns foo $ kubectl apply -f <(istioctl kube-inject -f @samples/httpbin/httpbin.yaml@) -n foo $ kubectl apply -f <(istioctl kube-inject -f @samples/sleep/sleep.yaml@) -n foo $ kubectl create ns bar $ kubectl apply -f <(istioctl kube-inject -f @samples/httpbin/httpbin.yaml@) -n bar $ kubectl apply -f <(istioctl kube-inject -f @samples/sleep/sleep.yaml@) -n bar $ kubectl create ns legacy $ kubectl apply -f @samples/sleep/sleep.yaml@ -n legacy
Verify setup by sending an http request (using curl command) from any sleep pod (among those in namespace
foo
,bar
orlegacy
) tohttpbin.foo
. All requests should success with HTTP code 200.$ for from in "foo" "bar" "legacy"; do kubectl exec $(kubectl get pod -l app=sleep -n ${from} -o jsonpath={.items..metadata.name}) -c sleep -n ${from} -- curl http://httpbin.foo:8000/ip -s -o /dev/null -w "sleep.${from} to httpbin.foo: %{http_code}\n"; done sleep.foo to httpbin.foo: 200 sleep.bar to httpbin.foo: 200 sleep.legacy to httpbin.foo: 200
Also verify that there are no authentication policies or destination rules (except control plane’s) in the system:
$ kubectl get policies.authentication.istio.io --all-namespaces NAMESPACE NAME AGE istio-system grafana-ports-mtls-disabled 3m
$ kubectl get destinationrule --all-namespaces NAMESPACE NAME HOST AGE istio-system istio-multicluster-destinationrule *.global 35s istio-system istio-policy istio-policy.istio-system.svc.cluster.local 35s istio-system istio-telemetry istio-telemetry.istio-system.svc.cluster.local 33s
Configure clients to send mutual TLS traffic
Configure Istio services to send mutual TLS traffic by setting DestinationRule
.
$ cat <<EOF | kubectl apply -n foo -f -
apiVersion: "networking.istio.io/v1alpha3"
kind: "DestinationRule"
metadata:
name: "example-httpbin-istio-client-mtls"
spec:
host: httpbin.foo.svc.cluster.local
trafficPolicy:
tls:
mode: ISTIO_MUTUAL
EOF
sleep.foo
and sleep.bar
should start sending mutual TLS traffic to httpbin.foo
. And sleep.legacy
still sends plaintext
traffic to httpbin.foo
since it does not have sidecar thus DestinationRule
does not apply.
Now we confirm all requests to httpbin.foo
still succeed.
$ for from in "foo" "bar" "legacy"; do kubectl exec $(kubectl get pod -l app=sleep -n ${from} -o jsonpath={.items..metadata.name}) -c sleep -n ${from} -- curl http://httpbin.foo:8000/ip -s -o /dev/null -w "sleep.${from} to httpbin.foo: %{http_code}\n"; done
sleep.foo to httpbin.foo: 200
sleep.bar to httpbin.foo: 200
sleep.legacy to httpbin.foo: 200
You can also specify a subset of the clients’ request to use ISTIO_MUTUAL
mutual TLS in
DestinationRule
.
After verifying it works by checking Grafana to monitor,
then increase the rollout scope and finally apply to all Istio client services.
Lock down to mutual TLS
After migrating all clients to Istio services and injecting the Envoy sidecar, we can lock down the httpbin.foo
to only accept mutual TLS traffic.
$ cat <<EOF | kubectl apply -n foo -f -
apiVersion: "authentication.istio.io/v1alpha1"
kind: "Policy"
metadata:
name: "example-httpbin-strict"
namespace: foo
spec:
targets:
- name: httpbin
peers:
- mtls:
mode: STRICT
EOF
Now, you should see the request from sleep.legacy
fail.
$ for from in "foo" "bar" "legacy"; do kubectl exec $(kubectl get pod -l app=sleep -n ${from} -o jsonpath={.items..metadata.name}) -c sleep -n ${from} -- curl http://httpbin.foo:8000/ip -s -o /dev/null -w "sleep.${from} to httpbin.foo: %{http_code}\n"; done
sleep.foo to httpbin.foo: 200
sleep.bar to httpbin.foo: 200
sleep.legacy to httpbin.foo: 503
If you can’t migrate all your services to Istio (injecting Envoy sidecar), you have to stay at PERMISSIVE
mode.
However, when configured with PERMISSIVE
mode, no authentication or authorization checks will be performed for plaintext traffic by default.
We recommend you use Istio Authorization to configure different paths with different authorization policies.
Clean up the example
To remove all resources created in this section:
$ kubectl delete ns foo bar legacy
Namespaces foo bar legacy deleted.
Option 2: globally enable mutual TLS for the cluster
This section describes how to apply the configuration to enforce mutual TLS for a cluster. For more details, see the Authentication policy task.
Configure all clients to send mutual TLS traffic
Run the following command to enable all Envoy sidecars to send mutual TLS traffic to the servers.
$ kubectl apply -f - <<EOF
apiVersion: "networking.istio.io/v1alpha3"
kind: "DestinationRule"
metadata:
name: "default"
namespace: "istio-system"
spec:
host: "*.local"
trafficPolicy:
tls:
mode: ISTIO_MUTUAL
EOF
The connections between services should not be interrupted.
Lock down to mutual TLS for the entire cluster
Run the following command to enforce all Envoy sidecars to only receive mutual TLS traffic.
$ kubectl apply -f - <<EOF
apiVersion: "authentication.istio.io/v1alpha1"
kind: "MeshPolicy"
metadata:
name: "default"
spec:
peers:
- mtls: {}
EOF
The connections between services should not be interrupted.
Clean up global mutual TLS configuration
Choose one of the following, depending on the mode you want to switch to:
To switch to
PERMISSIVE
mode for the cluster:$ kubectl apply -f - <<EOF apiVersion: "authentication.istio.io/v1alpha1" kind: "MeshPolicy" metadata: name: "default" spec: peers: - mtls: mode: PERMISSIVE EOF
To disable the global mutual TLS and switch to plaintext only:
$ kubectl delete meshpolicy default $ kubectl delete destinationrule default -n istio-system