Custom CA Integration using Kubernetes CSR

This feature requires Kubernetes version >= 1.18.

This task shows how to provision Workload Certificates using a custom certificate authority that integrates with the Kubernetes CSR API. Different workloads can get their certificates signed from different cert-signers. Each cert-signer is effectively a different CA. It is expected that workloads whose certificates are issued from the same cert-signer can talk MTLS to each other while workloads signed by different signers cannot. This feature leverages Chiron, a lightweight component linked with Istiod that signs certificates using the Kubernetes CSR API.

For this example, we use open-source cert-manager. Cert-manager has added experimental Support for Kubernetes CertificateSigningRequests starting with version 1.4.

Deploy Custom CA controller in the Kubernetes cluster

  1. Deploy cert-manager according to the installation doc.

  2. Create three self signed cluster issuers istio-system, foo and bar for cert-manager. Note: Namespace issuers and other types of issuers can also be used.

    $ cat <<EOF > ./selfsigned-issuer.yaml
    apiVersion: cert-manager.io/v1
    kind: ClusterIssuer
    metadata:
      name: selfsigned-bar-issuer
    spec:
      selfSigned: {}
    ---
    apiVersion: cert-manager.io/v1
    kind: Certificate
    metadata:
      name: bar-ca
      namespace: cert-manager
    spec:
      isCA: true
      commonName: bar
      secretName: bar-ca-selfsigned
      issuerRef:
        name: selfsigned-bar-issuer
        kind: ClusterIssuer
        group: cert-manager.io
    ---
    apiVersion: cert-manager.io/v1
    kind: ClusterIssuer
    metadata:
      name: bar
    spec:
      ca:
        secretName: bar-ca-selfsigned
    ---
    apiVersion: cert-manager.io/v1
    kind: ClusterIssuer
    metadata:
      name: selfsigned-foo-issuer
    spec:
      selfSigned: {}
    ---
    apiVersion: cert-manager.io/v1
    kind: Certificate
    metadata:
      name: foo-ca
      namespace: cert-manager
    spec:
      isCA: true
      commonName: foo
      secretName: foo-ca-selfsigned
      issuerRef:
        name: selfsigned-foo-issuer
        kind: ClusterIssuer
        group: cert-manager.io
    ---
    apiVersion: cert-manager.io/v1
    kind: ClusterIssuer
    metadata:
      name: foo
    spec:
      ca:
        secretName: foo-ca-selfsigned
    ---
    apiVersion: cert-manager.io/v1
    kind: ClusterIssuer
    metadata:
      name: selfsigned-istio-issuer
    spec:
      selfSigned: {}
    ---
    apiVersion: cert-manager.io/v1
    kind: Certificate
    metadata:
      name: istio-ca
      namespace: cert-manager
    spec:
      isCA: true
      commonName: istio-system
      secretName: istio-ca-selfsigned
      issuerRef:
        name: selfsigned-istio-issuer
        kind: ClusterIssuer
        group: cert-manager.io
    ---
    apiVersion: cert-manager.io/v1
    kind: ClusterIssuer
    metadata:
      name: istio-system
    spec:
      ca:
        secretName: istio-ca-selfsigned
    EOF
    $ kubectl apply -f ./selfsigned-issuer.yaml
    

Export root certificates for each cluster issuer

$ export istioca=$(kubectl get clusterissuers istio-system -o jsonpath='{.spec.ca.secretName}' | xargs kubectl get secret -n cert-manager -o jsonpath='{.data.ca\.crt}' | base64 -d)

$ export fooca=$(kubectl get clusterissuers foo -o jsonpath='{.spec.ca.secretName}' | xargs kubectl get secret -n cert-manager -o jsonpath='{.data.ca\.crt}' | base64 -d)

$ export barca=$(kubectl get clusterissuers bar -o jsonpath='{.spec.ca.secretName}' | xargs kubectl get secret -n cert-manager -o jsonpath='{.data.ca\.crt}' | base64 -d)

Deploy Istio with default cert-signer info

  1. Deploy Istio on the cluster using istioctl with the following configuration. The ISTIO_META_CERT_SIGNER is the default cert-signer for workloads.

    $ cat <<EOF > ./istio.yaml
    apiVersion: install.istio.io/v1alpha1
    kind: IstioOperator
    spec:
      meshConfig:
        defaultConfig:
          proxyMetadata:
            ISTIO_META_CERT_SIGNER: istio-system
        caCertificates:
        - pem: |
          $istioca
          certSigners:
          - clusterissuers.cert-manager.io/istio-system
        - pem: |
          $fooca
          certSigners:
          - clusterissuers.cert-manager.io/foo
        - pem: |
          $barca
          certSigners:
          - clusterissuers.cert-manager.io/bar
      components:
        pilot:
          k8s:
            env:
            - name: CERT_SIGNER_DOMAIN
              value: clusterissuers.cert-manager.io
            - name: EXTERNAL_CA
              value: ISTIOD_RA_KUBERNETES_API
            - name: PILOT_CERT_PROVIDER
              value: k8s.io/clusterissuers.cert-manager.io/istio-system
            overlays:
              - kind: ClusterRole
                name: istiod-clusterrole-istio-system
                patches:
                  - path: rules[-1]
                    value: |
                      apiGroups:
                      - certificates.k8s.io
                      resourceNames:
                      - clusterissuers.cert-manager.io/foo
                      - clusterissuers.cert-manager.io/bar
                      - clusterissuers.cert-manager.io/istio-system
                      resources:
                      - signers
                      verbs:
                      - approve
    EOF
    $ istioctl install -f ./istio.yaml
    
  2. Deploy the proxyconfig-bar.yaml in the bar namespace to define cert-signer for workloads in the bar namespace.

    $ cat <<EOF > ./proxyconfig-bar.yaml
    apiVersion: networking.istio.io/v1beta1
    kind: ProxyConfig
    metadata:
      name: barpc
      namespace: bar
    spec:
      environmentVariables:
        ISTIO_META_CERT_SIGNER: bar
    EOF
    $ kubectl apply  -f ./proxyconfig-bar.yaml
    
  3. Deploy the proxyconfig-foo.yaml in the foo namespace to define cert-signer for workloads in the foo namespace.

    $ cat <<EOF > ./proxyconfig-foo.yaml
    apiVersion: networking.istio.io/v1beta1
    kind: ProxyConfig
    metadata:
      name: foopc
      namespace: foo
    spec:
      environmentVariables:
        ISTIO_META_CERT_SIGNER: foo
    EOF
    $ kubectl apply  -f ./proxyconfig-foo.yaml
    
  4. Deploy the httpbin and sleep sample application in the foo and bar namespaces.

    $ kubectl label ns foo istio-injection=enabled
    $ kubectl label ns bar istio-injection=enabled
    $ kubectl apply -f samples/httpbin/httpbin.yaml -n foo
    $ kubectl apply -f samples/sleep/sleep.yaml -n foo
    $ kubectl apply -f samples/httpbin/httpbin.yaml -n bar
    $ kubectl apply -f samples/sleep/sleep.yaml -n bar
    

Verify the network connectivity between httpbin and sleep within the same namespace

When the workloads are deployed, they send CSR Requests with related signer info. Istiod forwards the CSR request to the custom CA for signing. The custom CA will use the correct cluster issuer or issuer to sign the cert back. Workloads under foo namespace will use foo cluster issuers while workloads under bar namespace will use the bar cluster issuers. To verify that they have indeed been signed by correct cluster issuers, We can verify workloads under the same namespace can communicate will while workloads under the different namespace cannot communicate.

  1. Check network connectivity between service sleep and httpbin in the foo namespace.

    $ export SLEEP_POD_FOO=$(kubectl get pod -n foo -l app=sleep -o jsonpath={.items..metadata.name})
    $ kubectl exec -it $SLEEP_POD_FOO -n foo -c sleep curl http://httpbin.foo:8000/html
    <!DOCTYPE html>
    <html>
      <head>
      </head>
      <body>
          <h1>Herman Melville - Moby-Dick</h1>
    
          <div>
            <p>
              Availing himself of the mild...
            </p>
          </div>
      </body>
    
  2. Check network connectivity between service sleep in the foo namespace and httpbin in the bar namespace.

    $ export SLEEP_POD_FOO=$(kubectl get pod -n foo -l app=sleep -o jsonpath={    .items..metadata.name})
    $ kubectl exec -it $SLEEP_POD_FOO -n foo -c sleep curl http://httpbin.bar:8000/html
    upstream connect error or disconnect/reset before headers. reset reason: connection failure, transport failure reason: TLS error: 268435581:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED
    

Cleanup

  • Remove the istio-system, foo and bar namespaces:

    $ kubectl delete ns istio-system
    $ kubectl delete ns foo
    $ kubectl delete ns bar
    

Reasons to use this feature

  • Custom CA Integration - By specifying a Signer name in the Kubernetes CSR Request, this feature allows Istio to integrate with custom Certificate Authorities using the Kubernetes CSR API interface. This does require the custom CA to implement a Kubernetes controller to watch the CertificateSigningRequest Resources and act on them.

  • Better multi-tenancy - By specifying a different cert-signer for different workloads, certificates for different tenant’s workloads can be signed by different CAs.

Was this information useful?
Do you have any suggestions for improvement?

Thanks for your feedback!