Custom CA Integration using Kubernetes CSR

This feature requires Kubernetes version >= 1.18.

This task shows how to provision Workload Certificates using a custom certificate authority that integrates with the Kubernetes CSR API⁷. Different workloads can get their certificates signed from different cert-signers. Each cert-signer is effectively a different CA. It is expected that workloads whose certificates are issued from the same cert-signer can talk MTLS to each other while workloads signed by different signers cannot. This feature leverages Chiron⁸, a lightweight component linked with Istiod that signs certificates using the Kubernetes CSR API.

For this example, we use open-source cert-manager⁹. Cert-manager has added experimental Support for Kubernetes CertificateSigningRequests¹⁰ starting with version 1.4.

Deploy Custom CA controller in the Kubernetes cluster

1. Deploy cert-manager according to the installation doc¹¹.

2. Create three self signed cluster issuers istio-system, foo and bar for cert-manager. Note: Namespace issuers and other types of issuers can also be used.

   $ cat <<EOF > ./selfsigned-issuer.yaml
   apiVersion: cert-manager.io/v1
   kind: ClusterIssuer
   metadata:
     name: selfsigned-bar-issuer
   spec:
     selfSigned: {}
   ---
   apiVersion: cert-manager.io/v1
   kind: Certificate
   metadata:
     name: bar-ca
     namespace: cert-manager
   spec:
     isCA: true
     commonName: bar
     secretName: bar-ca-selfsigned
     issuerRef:
       name: selfsigned-bar-issuer
       kind: ClusterIssuer
       group: cert-manager.io
   ---
   apiVersion: cert-manager.io/v1
   kind: ClusterIssuer
   metadata:
     name: bar
   spec:
     ca:
       secretName: bar-ca-selfsigned
   ---
   apiVersion: cert-manager.io/v1
   kind: ClusterIssuer
   metadata:
     name: selfsigned-foo-issuer
   spec:
     selfSigned: {}
   ---
   apiVersion: cert-manager.io/v1
   kind: Certificate
   metadata:
     name: foo-ca
     namespace: cert-manager
   spec:
     isCA: true
     commonName: foo
     secretName: foo-ca-selfsigned
     issuerRef:
       name: selfsigned-foo-issuer
       kind: ClusterIssuer
       group: cert-manager.io
   ---
   apiVersion: cert-manager.io/v1
   kind: ClusterIssuer
   metadata:
     name: foo
   spec:
     ca:
       secretName: foo-ca-selfsigned
   ---
   apiVersion: cert-manager.io/v1
   kind: ClusterIssuer
   metadata:
     name: selfsigned-istio-issuer
   spec:
     selfSigned: {}
   ---
   apiVersion: cert-manager.io/v1
   kind: Certificate
   metadata:
     name: istio-ca
     namespace: cert-manager
   spec:
     isCA: true
     commonName: istio-system
     secretName: istio-ca-selfsigned
     issuerRef:
       name: selfsigned-istio-issuer
       kind: ClusterIssuer
       group: cert-manager.io
   ---
   apiVersion: cert-manager.io/v1
   kind: ClusterIssuer
   metadata:
     name: istio-system
   spec:
     ca:
       secretName: istio-ca-selfsigned
   EOF
   $ kubectl apply -f ./selfsigned-issuer.yaml
   

Export root certificates for each cluster issuer

$ export istioca=$(kubectl get clusterissuers istio-system -o jsonpath='{.spec.ca.secretName}' | xargs kubectl get secret -n cert-manager -o jsonpath='{.data.ca\.crt}' | base64 -d)

$ export fooca=$(kubectl get clusterissuers foo -o jsonpath='{.spec.ca.secretName}' | xargs kubectl get secret -n cert-manager -o jsonpath='{.data.ca\.crt}' | base64 -d)

$ export barca=$(kubectl get clusterissuers bar -o jsonpath='{.spec.ca.secretName}' | xargs kubectl get secret -n cert-manager -o jsonpath='{.data.ca\.crt}' | base64 -d)

Deploy Istio with default cert-signer info

1. Deploy Istio on the cluster using istioctl with the following configuration. The ISTIO_META_CERT_SIGNER is the default cert-signer for workloads.

   $ cat <<EOF > ./istio.yaml
   apiVersion: install.istio.io/v1alpha1
   kind: IstioOperator
   spec:
     meshConfig:
       defaultConfig:
         proxyMetadata:
           ISTIO_META_CERT_SIGNER: istio-system
       caCertificates:
       - pem: |
         $istioca
         certSigners:
         - clusterissuers.cert-manager.io/istio-system
       - pem: |
         $fooca
         certSigners:
         - clusterissuers.cert-manager.io/foo
       - pem: |
         $barca
         certSigners:
         - clusterissuers.cert-manager.io/bar
     components:
       pilot:
         k8s:
           env:
           - name: CERT_SIGNER_DOMAIN
             value: clusterissuers.cert-manager.io
           - name: EXTERNAL_CA
             value: ISTIOD_RA_KUBERNETES_API
           - name: PILOT_CERT_PROVIDER
             value: k8s.io/clusterissuers.cert-manager.io/istio-system
           overlays:
             - kind: ClusterRole
               name: istiod-clusterrole-istio-system
               patches:
                 - path: rules[-1]
                   value: |
                     apiGroups:
                     - certificates.k8s.io
                     resourceNames:
                     - clusterissuers.cert-manager.io/foo
                     - clusterissuers.cert-manager.io/bar
                     - clusterissuers.cert-manager.io/istio-system
                     resources:
                     - signers
                     verbs:
                     - approve
   EOF
   $ istioctl install -f ./istio.yaml
   

2. Deploy the proxyconfig-bar.yaml in the bar namespace to define cert-signer for workloads in the bar namespace.

   $ cat <<EOF > ./proxyconfig-bar.yaml
   apiVersion: networking.istio.io/v1beta1
   kind: ProxyConfig
   metadata:
     name: barpc
     namespace: bar
   spec:
     environmentVariables:
       ISTIO_META_CERT_SIGNER: bar
   EOF
   $ kubectl apply  -f ./proxyconfig-bar.yaml
   

3. Deploy the proxyconfig-foo.yaml in the foo namespace to define cert-signer for workloads in the foo namespace.

   $ cat <<EOF > ./proxyconfig-foo.yaml
   apiVersion: networking.istio.io/v1beta1
   kind: ProxyConfig
   metadata:
     name: foopc
     namespace: foo
   spec:
     environmentVariables:
       ISTIO_META_CERT_SIGNER: foo
   EOF
   $ kubectl apply  -f ./proxyconfig-foo.yaml
   

4. Deploy the httpbin and sleep sample application in the foo and bar namespaces.

   $ kubectl label ns foo istio-injection=enabled
   $ kubectl label ns bar istio-injection=enabled
   $ kubectl apply -f samples/httpbin/httpbin.yaml -n foo
   $ kubectl apply -f samples/sleep/sleep.yaml -n foo
   $ kubectl apply -f samples/httpbin/httpbin.yaml -n bar
   $ kubectl apply -f samples/sleep/sleep.yaml -n bar
   

Verify the network connectivity between httpbin and sleep within the same namespace

When the workloads are deployed, they send CSR Requests with related signer info. Istiod forwards the CSR request to the custom CA for signing. The custom CA will use the correct cluster issuer or issuer to sign the cert back. Workloads under foo namespace will use foo cluster issuers while workloads under bar namespace will use the bar cluster issuers. To verify that they have indeed been signed by correct cluster issuers, We can verify workloads under the same namespace can communicate will while workloads under the different namespace cannot communicate.

1. Check network connectivity between service sleep and httpbin in the foo namespace.

   $ export SLEEP_POD_FOO=$(kubectl get pod -n foo -l app=sleep -o jsonpath={.items..metadata.name})
   $ kubectl exec -it $SLEEP_POD_FOO -n foo -c sleep curl http://httpbin.foo:8000/html
   <!DOCTYPE html>
   <html>
     <head>
     </head>
     <body>
         <h1>Herman Melville - Moby-Dick</h1>
   
         <div>
           <p>
             Availing himself of the mild...
           </p>
         </div>
     </body>
   

2. Check network connectivity between service sleep in the foo namespace and httpbin in the bar namespace.

   $ export SLEEP_POD_FOO=$(kubectl get pod -n foo -l app=sleep -o jsonpath={    .items..metadata.name})
   $ kubectl exec -it $SLEEP_POD_FOO -n foo -c sleep curl http://httpbin.bar:8000/html
   upstream connect error or disconnect/reset before headers. reset reason: connection failure, transport failure reason: TLS error: 268435581:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED
   

Cleanup

• Remove the istio-system, foo and bar namespaces:

  $ kubectl delete ns istio-system
  $ kubectl delete ns foo
  $ kubectl delete ns bar
  

Reasons to use this feature

• Custom CA Integration - By specifying a Signer name in the Kubernetes CSR Request, this feature allows Istio to integrate with custom Certificate Authorities using the Kubernetes CSR API interface. This does require the custom CA to implement a Kubernetes controller to watch the CertificateSigningRequest Resources and act on them.

• Better multi-tenancy - By specifying a different cert-signer for different workloads, certificates for different tenant’s workloads can be signed by different CAs.