Custom CA Integration using Kubernetes CSR
This feature requires Kubernetes version >= 1.18.
This task shows how to provision Workload Certificates using a custom certificate authority that integrates with the Kubernetes CSR API. Different workloads can get their certificates signed from different cert-signers. Each cert-signer is effectively a different CA. It is expected that workloads whose certificates are issued from the same cert-signer can talk MTLS to each other while workloads signed by different signers cannot. This feature leverages Chiron, a lightweight component linked with Istiod that signs certificates using the Kubernetes CSR API.
For this example, we use open-source cert-manager.
Cert-manager has added experimental Support for Kubernetes CertificateSigningRequests
starting with version 1.4.
Deploy Custom CA controller in the Kubernetes cluster
Deploy cert-manager according to the installation doc.
Create three self signed cluster issuers
istio-system
,foo
andbar
for cert-manager. Note: Namespace issuers and other types of issuers can also be used.$ cat <<EOF > ./selfsigned-issuer.yaml apiVersion: cert-manager.io/v1 kind: ClusterIssuer metadata: name: selfsigned-bar-issuer spec: selfSigned: {} --- apiVersion: cert-manager.io/v1 kind: Certificate metadata: name: bar-ca namespace: cert-manager spec: isCA: true commonName: bar secretName: bar-ca-selfsigned issuerRef: name: selfsigned-bar-issuer kind: ClusterIssuer group: cert-manager.io --- apiVersion: cert-manager.io/v1 kind: ClusterIssuer metadata: name: bar spec: ca: secretName: bar-ca-selfsigned --- apiVersion: cert-manager.io/v1 kind: ClusterIssuer metadata: name: selfsigned-foo-issuer spec: selfSigned: {} --- apiVersion: cert-manager.io/v1 kind: Certificate metadata: name: foo-ca namespace: cert-manager spec: isCA: true commonName: foo secretName: foo-ca-selfsigned issuerRef: name: selfsigned-foo-issuer kind: ClusterIssuer group: cert-manager.io --- apiVersion: cert-manager.io/v1 kind: ClusterIssuer metadata: name: foo spec: ca: secretName: foo-ca-selfsigned --- apiVersion: cert-manager.io/v1 kind: ClusterIssuer metadata: name: selfsigned-istio-issuer spec: selfSigned: {} --- apiVersion: cert-manager.io/v1 kind: Certificate metadata: name: istio-ca namespace: cert-manager spec: isCA: true commonName: istio-system secretName: istio-ca-selfsigned issuerRef: name: selfsigned-istio-issuer kind: ClusterIssuer group: cert-manager.io --- apiVersion: cert-manager.io/v1 kind: ClusterIssuer metadata: name: istio-system spec: ca: secretName: istio-ca-selfsigned EOF $ kubectl apply -f ./selfsigned-issuer.yaml
Export root certificates for each cluster issuer
$ export istioca=$(kubectl get clusterissuers istio-system -o jsonpath='{.spec.ca.secretName}' | xargs kubectl get secret -n cert-manager -o jsonpath='{.data.ca\.crt}' | base64 -d)
$ export fooca=$(kubectl get clusterissuers foo -o jsonpath='{.spec.ca.secretName}' | xargs kubectl get secret -n cert-manager -o jsonpath='{.data.ca\.crt}' | base64 -d)
$ export barca=$(kubectl get clusterissuers bar -o jsonpath='{.spec.ca.secretName}' | xargs kubectl get secret -n cert-manager -o jsonpath='{.data.ca\.crt}' | base64 -d)
Deploy Istio with default cert-signer info
Deploy Istio on the cluster using
istioctl
with the following configuration. TheISTIO_META_CERT_SIGNER
is the default cert-signer for workloads.$ cat <<EOF > ./istio.yaml apiVersion: install.istio.io/v1alpha1 kind: IstioOperator spec: meshConfig: defaultConfig: proxyMetadata: ISTIO_META_CERT_SIGNER: istio-system caCertificates: - pem: | $istioca certSigners: - clusterissuers.cert-manager.io/istio-system - pem: | $fooca certSigners: - clusterissuers.cert-manager.io/foo - pem: | $barca certSigners: - clusterissuers.cert-manager.io/bar components: pilot: k8s: env: - name: CERT_SIGNER_DOMAIN value: clusterissuers.cert-manager.io - name: EXTERNAL_CA value: ISTIOD_RA_KUBERNETES_API - name: PILOT_CERT_PROVIDER value: k8s.io/clusterissuers.cert-manager.io/istio-system overlays: - kind: ClusterRole name: istiod-clusterrole-istio-system patches: - path: rules[-1] value: | apiGroups: - certificates.k8s.io resourceNames: - clusterissuers.cert-manager.io/foo - clusterissuers.cert-manager.io/bar - clusterissuers.cert-manager.io/istio-system resources: - signers verbs: - approve EOF $ istioctl install -f ./istio.yaml
Deploy the
proxyconfig-bar.yaml
in thebar
namespace to define cert-signer for workloads in thebar
namespace.$ cat <<EOF > ./proxyconfig-bar.yaml apiVersion: networking.istio.io/v1beta1 kind: ProxyConfig metadata: name: barpc namespace: bar spec: environmentVariables: ISTIO_META_CERT_SIGNER: bar EOF $ kubectl apply -f ./proxyconfig-bar.yaml
Deploy the
proxyconfig-foo.yaml
in the foo namespace to define cert-signer for workloads in thefoo
namespace.$ cat <<EOF > ./proxyconfig-foo.yaml apiVersion: networking.istio.io/v1beta1 kind: ProxyConfig metadata: name: foopc namespace: foo spec: environmentVariables: ISTIO_META_CERT_SIGNER: foo EOF $ kubectl apply -f ./proxyconfig-foo.yaml
Deploy the
httpbin
andsleep
sample application in thefoo
andbar
namespaces.$ kubectl label ns foo istio-injection=enabled $ kubectl label ns bar istio-injection=enabled $ kubectl apply -f samples/httpbin/httpbin.yaml -n foo $ kubectl apply -f samples/sleep/sleep.yaml -n foo $ kubectl apply -f samples/httpbin/httpbin.yaml -n bar $ kubectl apply -f samples/sleep/sleep.yaml -n bar
Verify the network connectivity between httpbin
and sleep
within the same namespace
When the workloads are deployed, they send CSR Requests with related signer info. Istiod forwards the CSR request to the custom CA for signing. The custom CA will use the correct cluster issuer or issuer to sign the cert back. Workloads under foo
namespace will use foo
cluster issuers while workloads under bar
namespace will use the bar
cluster issuers. To verify that they have indeed been signed by correct cluster issuers, We can verify workloads under the same namespace can communicate will while workloads under the different namespace cannot communicate.
Check network connectivity between service
sleep
andhttpbin
in thefoo
namespace.$ export SLEEP_POD_FOO=$(kubectl get pod -n foo -l app=sleep -o jsonpath={.items..metadata.name}) $ kubectl exec -it $SLEEP_POD_FOO -n foo -c sleep curl http://httpbin.foo:8000/html <!DOCTYPE html> <html> <head> </head> <body> <h1>Herman Melville - Moby-Dick</h1> <div> <p> Availing himself of the mild... </p> </div> </body>
Check network connectivity between service
sleep
in thefoo
namespace andhttpbin
in thebar
namespace.$ export SLEEP_POD_FOO=$(kubectl get pod -n foo -l app=sleep -o jsonpath={ .items..metadata.name}) $ kubectl exec -it $SLEEP_POD_FOO -n foo -c sleep curl http://httpbin.bar:8000/html upstream connect error or disconnect/reset before headers. reset reason: connection failure, transport failure reason: TLS error: 268435581:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED
Cleanup
Remove the
istio-system
,foo
andbar
namespaces:$ kubectl delete ns istio-system $ kubectl delete ns foo $ kubectl delete ns bar
Reasons to use this feature
Custom CA Integration - By specifying a Signer name in the Kubernetes CSR Request, this feature allows Istio to integrate with custom Certificate Authorities using the Kubernetes CSR API interface. This does require the custom CA to implement a Kubernetes controller to watch the
CertificateSigningRequest
Resources and act on them.Better multi-tenancy - By specifying a different cert-signer for different workloads, certificates for different tenant’s workloads can be signed by different CAs.