Direct encrypted traffic from IBM Cloud Kubernetes Service Ingress to Istio Ingress Gateway
Configure the IBM Cloud Kubernetes Service Application Load Balancer to direct traffic to the Istio Ingress gateway with mutual TLS.
In this blog post I show how to configure the Ingress Application Load Balancer (ALB) on IBM Cloud Kubernetes Service (IKS) to direct traffic to the Istio ingress gateway, while securing the traffic between them using mutual TLS authentication.
When you use IKS without Istio, you may control your ingress traffic using the provided ALB. This ingress-traffic routing is configured using a Kubernetes Ingress resource with ALB-specific annotations. IKS provides a DNS domain name, a TLS certificate that matches the domain, and a private key for the certificate. IKS stores the certificates and the private key in a Kubernetes secret.
When you start using Istio in your IKS cluster, the recommended method to send traffic to your Istio enabled workloads is by using the Istio Ingress Gateway instead of using the Kubernetes Ingress. One of the main reasons to use the Istio ingress gateway is the fact the ALB provided by IKS will not be able to communicate directly with the services inside the mesh when you enable STRICT mutual TLS. During your transition to having only Istio ingress gateway as your main entry point, you can continue to use the traditional Ingress for non-Istio services while using the Istio ingress gateway for services that are part of the mesh.
IKS provides a convenient way for clients to access Istio ingress gateway by letting you
register a new DNS subdomain for the
Istio gateway’s IP with an IKS command. The domain is in the following
format:
<cluster_name>-<globally_unique_account_HASH>-0001.<region>.containers.appdomain.cloud
, for example mycluster-a1b2cdef345678g9hi012j3kl4567890-0001.us-south.containers.appdomain.cloud
. In the same way as for the ALB domain,
IKS provides a certificate and a private key, storing them in another Kubernetes secret.
This blog describes how you can chain together the IKS Ingress ALB and the Istio ingress gateway to send traffic to your Istio enabled workloads while being able to continue using the ALB specific features and the ALB subdomain name. You configure the IKS Ingress ALB to direct traffic to the services inside an Istio service mesh through the Istio ingress gateway, while using mutual TLS authentication between the ALB and the gateway. For the mutual TLS authentication, you will configure the ALB and the Istio ingress gateway to use the certificates and keys provided by IKS for the ALB and NLB subdomains. Using certificates provided by IKS saves you the overhead of managing your own certificates for the connection between the ALB and the Istio ingress gateway.
You will use the NLB subdomain certificate as the server certificate for the Istio ingress gateway as intended. The NLB subdomain certificate represents the identity of the server that serves a particular NLB subdomain, in this case, the ingress gateway.
You will use the ALB subdomain certificate as the client certificate in mutual TLS authentication between the ALB and the Istio Ingress. When ALB acts as a server it presents the ALB certificate to the clients so the clients can authenticate the ALB. When ALB acts as a client of the Istio ingress gateway, it presents the same certificate to the Istio ingress gateway, so the Istio ingress gateway could authenticate the ALB.
Traffic to the services without an Istio sidecar can continue to flow as before directly from the ALB.
The diagram below exemplifies the described setting. It shows two services in the cluster, service A
and service B
.
service A
has an Istio sidecar injected and requires mutual TLS. service B
has no Istio sidecar. service B
can
be accessed by clients through the ALB, which directly communicates with service B
. service A
can be also
accessed by clients through the ALB, but in this case the traffic must pass through the Istio ingress gateway. Mutual
TLS authentication between the ALB and the gateway is based on the certificates provided by IKS.
The clients can also access the Istio ingress gateway directly. IKS registers different DNS domains for the ALB and for
the ingress gateway.
Initial setting
Create the
httptools
namespace and enable Istio sidecar injection:$ kubectl create namespace httptools $ kubectl label namespace httptools istio-injection=enabled namespace/httptools created namespace/httptools labeled
Deploy the
httpbin
sample tohttptools
:$ kubectl apply -f @samples/httpbin/httpbin.yaml@ -n httptools service/httpbin created deployment.apps/httpbin created
Create secrets for the ALB and the Istio ingress gateway
IKS generates a TLS certificate and a private key and stores them as a secret in the default namespace when you register
a DNS domain for an external IP by using the ibmcloud ks nlb-dns-create
command. IKS stores the ALB’s
certificate and private key also as a secret in the default namespace. You need these credentials to establish the
identities that the ALB and the Istio ingress gateway will present during the mutual TLS authentication between
them. You will configure the ALB and the Istio ingress gateway to exchange these certificates, to trust the certificates
of one another, and to use their private keys to encrypt and sign the traffic.
Store the name of your cluster in the
CLUSTER_NAME
environment variable:$ export CLUSTER_NAME=<your cluster name>
Store the domain name of your ALB in the
ALB_INGRESS_DOMAIN
environment variable:$ ibmcloud ks cluster get --cluster $CLUSTER_NAME | grep Ingress Ingress Subdomain: <your ALB ingress domain> Ingress Secret: <your ALB secret>
$ export ALB_INGRESS_DOMAIN=<your ALB ingress domain> $ export ALB_SECRET=<your ALB secret>
Store the external IP of your
istio-ingressgateway
service in an environment variable.$ export INGRESS_GATEWAY_IP=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}') $ echo INGRESS_GATEWAY_IP = $INGRESS_GATEWAY_IP
Create a DNS domain and certificates for the IP of the Istio Ingress Gateway service:
$ ibmcloud ks nlb-dns create classic --cluster $CLUSTER_NAME --ip $INGRESS_GATEWAY_IP --secret-namespace istio-system Host name subdomain is created as <some domain>
Store the domain name from the previous command in an environment variable:
$ export INGRESS_GATEWAY_DOMAIN=<the domain from the previous command>
List the registered domain names:
$ ibmcloud ks nlb-dnss --cluster $CLUSTER_NAME Retrieving host names, certificates, IPs, and health check monitors for network load balancer (NLB) pods in cluster <your cluster>... OK Hostname IP(s) Health Monitor SSL Cert Status SSL Cert Secret Name Secret Namespace <your ingress gateway hostname> <your ingress gateway IP> None created <the matching secret name> istio-system ...
Wait until the status of the certificate (the fourth field) of the new domain name becomes
enabled
(initially it ispending
).Store the name of the secret of the new domain name:
$ export INGRESS_GATEWAY_SECRET=<the secret's name as shown in the SSL Cert Secret Name column>
Extract the certificate and the key from the secret provided for the ALB:
$ mkdir alb_certs $ kubectl get secret $ALB_SECRET --namespace=default -o yaml | grep 'tls.key:' | cut -f2 -d: | base64 --decode > alb_certs/client.key $ kubectl get secret $ALB_SECRET --namespace=default -o yaml | grep 'tls.crt:' | cut -f2 -d: | base64 --decode > alb_certs/client.crt $ ls -al alb_certs -rw-r--r-- 1 user staff 3738 Sep 11 07:57 client.crt -rw-r--r-- 1 user staff 1675 Sep 11 07:57 client.key
Download the issuer certificate of the Let’s Encrypt certificate, which is the issuer of the certificates provided by IKS. You specify this certificate as the certificate of a certificate authority to trust, for both the ALB and the Istio ingress gateway.
$ curl https://letsencrypt.org/certs/trustid-x3-root.pem --output trusted.crt
Create a Kubernetes secret to be used by the ALB to establish mutual TLS connection.
$ kubectl create secret generic alb-certs -n istio-system --from-file=trusted.crt --from-file=alb_certs/client.crt --from-file=alb_certs/client.key secret "alb-certs" created
For mutual TLS, a separate Secret named
<tls-cert-secret>-cacert
with acacert
key is needed for the ingress gateway.$ kubectl create -n istio-system secret generic $INGRESS_GATEWAY_SECRET-cacert --from-file=ca.crt=trusted.crt secret/cluster_name-hash-XXXX-cacert created
Configure a mutual TLS ingress gateway
In this section you configure the Istio ingress gateway to perform mutual TLS between external clients and the gateway. You use the certificates and the keys provided to you for the ingress gateway and the ALB.
Define a
Gateway
to allow access on port 443 only, with mutual TLS:$ kubectl apply -n httptools -f - <<EOF apiVersion: networking.istio.io/v1alpha3 kind: Gateway metadata: name: default-ingress-gateway spec: selector: istio: ingressgateway # use istio default ingress gateway servers: - port: number: 443 name: https protocol: HTTPS tls: mode: MUTUAL credentialName: $INGRESS_GATEWAY_SECRET hosts: - "$INGRESS_GATEWAY_DOMAIN" - "httpbin.$ALB_INGRESS_DOMAIN" EOF
Configure routes for traffic entering via the
Gateway
:$ kubectl apply -n httptools -f - <<EOF apiVersion: networking.istio.io/v1alpha3 kind: VirtualService metadata: name: default-ingress spec: hosts: - "$INGRESS_GATEWAY_DOMAIN" - "httpbin.$ALB_INGRESS_DOMAIN" gateways: - default-ingress-gateway http: - match: - uri: prefix: /status route: - destination: port: number: 8000 host: httpbin.httptools.svc.cluster.local EOF
Send a request to
httpbin
by curl, passing as parameters the client certificate (the--cert
option) and the private key (the--key
option):$ curl https://$INGRESS_GATEWAY_DOMAIN/status/418 --cert alb_certs/client.crt --key alb_certs/client.key -=[ teapot ]=- _...._ .' _ _ `. | ."` ^ `". _, \_;`"---"`|// | ;/ \_ _/ `"""`
Remove the directories with the ALB and ingress gateway certificates and keys.
$ rm -r alb_certs trusted.crt
Configure the ALB
You need to configure your Ingress resource to direct traffic to the Istio ingress gateway while using the certificate
stored in the alb-certs
secret. Normally, the ALB decrypts HTTPS requests before forwarding traffic to your apps.
You can configure the ALB to re-encrypt the traffic before it is forwarded to the Istio ingress gateway by using the
ssl-services
annotation on the Ingress resource. This annotation also allows you to specify the certificate stored in
the alb-certs
secret, required for mutual TLS.
Configure the
Ingress
resource for the ALB. You must create theIngress
resource in theistio-system
namespace in order to forward the traffic to the Istio ingress gateway.$ kubectl apply -f - <<EOF apiVersion: extensions/v1beta1 kind: Ingress metadata: name: alb-ingress namespace: istio-system annotations: ingress.bluemix.net/ssl-services: "ssl-service=istio-ingressgateway ssl-secret=alb-certs proxy-ssl-name=$INGRESS_GATEWAY_DOMAIN" spec: tls: - hosts: - httpbin.$ALB_INGRESS_DOMAIN secretName: $ALB_SECRET rules: - host: httpbin.$ALB_INGRESS_DOMAIN http: paths: - path: /status backend: serviceName: istio-ingressgateway servicePort: 443 EOF
Test the ALB ingress:
$ curl https://httpbin.$ALB_INGRESS_DOMAIN/status/418 -=[ teapot ]=- _...._ .' _ _ `. | ."` ^ `". _, \_;`"---"`|// | ;/ \_ _/ `"""`
Congratulations! You configured the IKS Ingress ALB to send encrypted traffic to the Istio ingress gateway. You allocated a host name and certificate for your Istio ingress gateway and used that certificate as the server certificate for Istio ingress gateway. As the client certificate of the ALB you used the certificate provided by IKS for the ALB. Once you had the certificates deployed as Kubernetes secrets, you directed the ingress traffic from the ALB to the Istio ingress gateway for some specific paths and used the certificates for mutual TLS authentication between the ALB and the Istio ingress gateway.
Cleanup
Delete the
Gateway
configuration, theVirtualService
, and the secrets:$ kubectl delete ingress alb-ingress -n istio-system $ kubectl delete virtualservice default-ingress -n httptools $ kubectl delete gateway default-ingress-gateway -n httptools $ kubectl delete secrets alb-certs -n istio-system $ rm -rf alb_certs trusted.crt $ unset CLUSTER_NAME ALB_INGRESS_DOMAIN ALB_SECRET INGRESS_GATEWAY_DOMAIN INGRESS_GATEWAY_SECRET
Shutdown the
httpbin
service:$ kubectl delete -f @samples/httpbin/httpbin.yaml@ -n httptools
Delete the
httptools
namespace:$ kubectl delete namespace httptools