Istio Ingress

This task describes how to configure Istio to expose a service outside of the service mesh cluster. In a Kubernetes environment, the Kubernetes Ingress Resource allows users to specify services that should be exposed outside the cluster. It allows one to define a backend service per virtual host and path.

Once the Istio Ingress specification is defined, traffic entering the cluster is directed through the istio-ingress service. As a result, Istio features, for example, monitoring and route rules, can be applied to the traffic entering the cluster.

The Istio Ingress specification is based on the standard Kubernetes Ingress Resource specification, with the following differences:

1. Istio Ingress specification contains kubernetes.io/ingress.class: istio annotation.

2. All other annotations are ignored.

The following are known limitations of Istio Ingress:

1. Regular expressions in paths are not supported.
2. Fault injection at the Ingress is not supported.

Before you begin

• Setup Istio by following the instructions in the Installation guide.

• Make sure your current directory is the istio directory.

• Start the httpbin sample, which will be used as the destination service to be exposed externally.

  If you installed the Istio-Initializer, do

  kubectl apply -f samples/httpbin/httpbin.yaml
  

  Without the Istio-Initializer:

  kubectl apply -f <(istioctl kube-inject -f samples/httpbin/httpbin.yaml)
  

Configuring ingress (HTTP)

1. Create a basic Ingress specification for the httpbin service

   cat <<EOF | kubectl create -f -
   apiVersion: extensions/v1beta1
   kind: Ingress
   metadata:
     name: simple-ingress
     annotations:
       kubernetes.io/ingress.class: istio
   spec:
     rules:
     - http:
         paths:
         - path: /status/.*
           backend:
             serviceName: httpbin
             servicePort: 8000
         - path: /delay/.*
           backend:
             serviceName: httpbin
             servicePort: 8000
   EOF
   

   /.* is a special Istio notation that is used to indicate a prefix match, specifically a rule match configuration of the form (prefix: /).

Verifying ingress

1. Determine the ingress URL:

   • If your cluster is running in an environment that supports external load balancers, use the ingress’ external address:

     kubectl get ingress simple-ingress -o wide
     

     NAME             HOSTS     ADDRESS                 PORTS     AGE
     simple-ingress   *         130.211.10.121          80        1d
     

     export INGRESS_HOST=130.211.10.121
     

   • If load balancers are not supported, use the ingress controller pod’s hostIP:

     kubectl -n istio-system get po -l istio=ingress -o jsonpath='{.items[0].status.hostIP}'
     

     169.47.243.100
     

     along with the istio-ingress service’s nodePort for port 80:

     kubectl -n istio-system get svc istio-ingress
     

     NAME            CLUSTER-IP     EXTERNAL-IP   PORT(S)                      AGE
     istio-ingress   10.10.10.155   <pending>     80:31486/TCP,443:32254/TCP   32m
     

     export INGRESS_HOST=169.47.243.100:31486
     

2. Access the httpbin service using curl:

   curl -I http://$INGRESS_HOST/status/200
   

   HTTP/1.1 200 OK
   server: envoy
   date: Mon, 29 Jan 2018 04:45:49 GMT
   content-type: text/html; charset=utf-8
   access-control-allow-origin: *
   access-control-allow-credentials: true
   content-length: 0
   x-envoy-upstream-service-time: 48
   

3. Access any other URL that has not been explicitly exposed. You should see a HTTP 404 error

   curl -I http://$INGRESS_HOST/headers
   

   HTTP/1.1 404 Not Found
   date: Mon, 29 Jan 2018 04:45:49 GMT
   server: envoy
   content-length: 0
   

Configuring secure ingress (HTTPS)

1. Generate certificate and key for the ingress

   A private key and certificate can be created for testing using OpenSSL.

   openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /tmp/tls.key -out /tmp/tls.crt -subj "/CN=foo.bar.com"
   

2. Create the secret

   Create the secret istio-ingress-certs in namespace istio-system using kubectl. The Istio Ingress will automatically load the secret.

   Note: the secret must be called istio-ingress-certs in istio-system namespace, for it to be mounted on Istio Ingress.

   kubectl create -n istio-system secret tls istio-ingress-certs --key /tmp/tls.key --cert /tmp/tls.crt
   

3. Create the Ingress specification for the httpbin service

   cat <<EOF | kubectl create -f -
   apiVersion: extensions/v1beta1
   kind: Ingress
   metadata:
     name: secure-ingress
     annotations:
       kubernetes.io/ingress.class: istio
   spec:
     tls:
       - secretName: istio-ingress-certs # currently ignored
     rules:
     - http:
         paths:
         - path: /status/.*
           backend:
             serviceName: httpbin
             servicePort: 8000
         - path: /delay/.*
           backend:
             serviceName: httpbin
             servicePort: 8000
   EOF
   

   Note: Because SNI is not yet supported, Envoy currently only allows a single TLS secret in the ingress. That means the secretName field in ingress resource is not used.

Verifying ingress

1. Determine the ingress URL:

   • If your cluster is running in an environment that supports external load balancers, use the ingress’ external address:

     kubectl get ingress secure-ingress -o wide
     

     NAME             HOSTS     ADDRESS                 PORTS     AGE
     secure-ingress   *         130.211.10.121          80        1d
     

     export INGRESS_HOST=130.211.10.121
     

   • If load balancers are not supported, use the ingress controller pod’s hostIP:

     kubectl -n istio-system get po -l istio=ingress -o jsonpath='{.items[0].status.hostIP}'
     

     169.47.243.100
     

     along with the istio-ingress service’s nodePort for port 443:

     kubectl -n istio-system get svc istio-ingress
     

     NAME            CLUSTER-IP     EXTERNAL-IP   PORT(S)                      AGE
     istio-ingress   10.10.10.155   <pending>     80:31486/TCP,443:32254/TCP   32m
     

     export INGRESS_HOST=169.47.243.100:32254
     

2. Access the httpbin service using curl:

   curl -I -k https://$INGRESS_HOST/status/200
   

   HTTP/1.1 200 OK
   server: envoy
   date: Mon, 29 Jan 2018 04:45:49 GMT
   content-type: text/html; charset=utf-8
   access-control-allow-origin: *
   access-control-allow-credentials: true
   content-length: 0
   x-envoy-upstream-service-time: 96
   

3. Access any other URL that has not been explicitly exposed. You should see a HTTP 404 error

   curl -I -k https://$INGRESS_HOST/headers
   

   HTTP/1.1 404 Not Found
   date: Mon, 29 Jan 2018 04:45:49 GMT
   server: envoy
   content-length: 0
   

4. Configuring RBAC for ingress key/cert

   There are service accounts which can access this ingress key/cert, and this leads to risks of leaking key/cert. We can set up Role-Based Access Control (“RBAC”) to protect it. install/kubernetes/istio.yaml defines ClusterRoles and ClusterRoleBindings which allow service accounts in namespace istio-system to access all secret resources. We need to update or replace these RBAC set up to only allow istio-ingress-service-account to access ingress key/cert.

   We can use kubectl to list all secrets in namespace istio-system that we need to protect using RBAC.

   kubectl get secrets -n istio-system
   

   This produces the following output:

   NAME                                        TYPE                                  DATA      AGE
   istio-ingress-certs                         kubernetes.io/tls                     2         7d
   istio.istio-ingress-service-account         istio.io/key-and-cert                 3         7d
   ......
   

5. Update RBAC set up for istio-pilot-service-account and istio-mixer-istio-service-account

   Record ClusterRole istio-mixer-istio-system and istio-pilot-istio-system. We will refer to these copies when we redefine them to avoid breaking access permissions to other resources.

   kubectl describe ClusterRole istio-mixer-istio-system
   kubectl describe ClusterRole istio-pilot-istio-system
   

   Delete existing ClusterRoleBindings and ClusterRole.

   kubectl delete ClusterRoleBinding istio-pilot-admin-role-binding-istio-system
   kubectl delete ClusterRoleBinding istio-mixer-admin-role-binding-istio-system
   kubectl delete ClusterRole istio-mixer-istio-system
   

   As istio-pilot-istio-system is also bound to istio-ingress-service-account, we will delete istio-pilot-istio-system in next step.

   Create istio-mixer-istio-system.yaml, which allows istio-mixer-service-account to read istio.io/key-and-cert, and istio.io/ca-root types of secret instances. Refer to the recorded copy of istio-mixer-istio-system and add access permissions to other resources.

   kind: ClusterRole
   apiVersion: rbac.authorization.k8s.io/v1beta1
   metadata:
     name: istio-mixer-istio-system
   rules:  
   - apiGroups: [""] # "" indicates the core API group
     resources: ["secrets"]
     resourceNames: ["istio.istio-ca-service-account"]
     verbs: ["get", "list", "watch"]
   - apiGroups: [""] # "" indicates the core API group
     resources: ["secrets"]
     resourceNames: ["istio-ca-secret"]
     verbs: ["get", "list", "watch"]
   ......   
   ---
   kind: ClusterRoleBinding
   apiVersion: rbac.authorization.k8s.io/v1beta1
   metadata:
     name: istio-mixer-admin-role-binding-istio-system
   subjects:
   - kind: ServiceAccount
     name: istio-mixer-service-account
     namespace: istio-system
   roleRef:
     kind: ClusterRole
     name: istio-mixer-istio-system
     apiGroup: rbac.authorization.k8s.io
   

   kubectl apply -f istio-mixer-istio-system.yaml
   

6. Update RBAC set up for istio-pilot-service-account and istio-ingress-service-account

   Delete existing ClusterRoleBinding and ClusterRole.

   kubectl delete clusterrolebinding istio-ingress-admin-role-binding-istio-system
   kubectl delete ClusterRole istio-pilot-istio-system
   

   Create istio-pilot-istio-system.yaml, which allows istio-pilot-service-account to read istio.io/key-and-cert, and istio.io/ca-root types of secret instances. Refer to the recorded copy of istio-pilot-istio-system and add access permissions to other resources.

   kind: ClusterRole
   apiVersion: rbac.authorization.k8s.io/v1beta1
   metadata:
     name: istio-pilot-istio-system
   rules:  
   - apiGroups: [""] # "" indicates the core API group
     resources: ["secrets"]
     resourceNames: ["istio.istio-ca-service-account"]
     verbs: ["get", "list", "watch"]
   - apiGroups: [""] # "" indicates the core API group
     resources: ["secrets"]
     resourceNames: ["istio-ca-secret"]
     verbs: ["get", "list", "watch"]
   ......   
   ---
   kind: ClusterRoleBinding
   apiVersion: rbac.authorization.k8s.io/v1beta1
   metadata:
     name: istio-pilot-admin-role-binding-istio-system
   subjects:
   - kind: ServiceAccount
     name: istio-pilot-service-account
     namespace: istio-system
   roleRef:
     kind: ClusterRole
     name: istio-pilot-istio-system
     apiGroup: rbac.authorization.k8s.io
   

   kubectl apply -f istio-pilot-istio-system.yaml
   

   Create istio-ingress-istio-system.yaml which allows istio-ingress-service-account to read istio-ingress-certs as well as other secret instances. Refer to the recorded copy of istio-pilot-istio-system and add access permissions to other resources.

   kind: ClusterRole
   apiVersion: rbac.authorization.k8s.io/v1
   metadata:
     name: istio-ingress-istio-system
   rules: 
   - apiGroups: [""] # "" indicates the core API group
     resources: ["secrets"]
     resourceNames: ["istio.istio-ca-service-account"]
     verbs: ["get", "list", "watch"]
   - apiGroups: [""] # "" indicates the core API group
     resources: ["secrets"]
     resourceNames: ["istio-ca-secret"]
     verbs: ["get", "list", "watch"]
   ......
   - apiGroups: [""] # "" indicates the core API group
     resources: ["secrets"]
     resourceNames: ["istio-ingress-certs"]
     verbs: ["get", "list", "watch"]
   ---
   kind: ClusterRoleBinding
   apiVersion: rbac.authorization.k8s.io/v1
   metadata:
     name: istio-ingress-admin-role-binding-istio-system
   subjects:
   - kind: ServiceAccount
     name: istio-ingress-service-account
     namespace: istio-system
   roleRef:
     kind: ClusterRole
     name: istio-ingress-istio-system
     apiGroup: rbac.authorization.k8s.io
   

   kubectl apply -f istio-ingress-istio-system.yaml
   

7. Update RBAC set up for istio-ca-service-account

   Record ClusterRole istio-ca-istio-system.

   kubectl describe ClusterRole istio-ca-istio-system
   

   Create istio-ca-istio-system.yaml, which updates existing ClusterRole istio-ca-istio-system that allows istio-ca-service-account to read, create and modify all istio.io/key-and-cert, and istio.io/ca-root types of secrets.

   kind: ClusterRole
   apiVersion: rbac.authorization.k8s.io/v1
   metadata:
    name: istio-ca-istio-system
   rules:
   - apiGroups: [""] # "" indicates the core API group
    resources: ["secrets"]
    resourceNames: ["istio.istio-ca-service-account"]
    verbs: ["get", "list", "watch", "create", "update"]
   - apiGroups: [""] # "" indicates the core API group
    resources: ["secrets"]
    resourceNames: ["istio-ca-secret"]
    verbs: ["get", "list", "watch", "create", "update"]
   ......
   kind: ClusterRoleBinding
   apiVersion: rbac.authorization.k8s.io/v1
   metadata:
     name: istio-ca-role-binding-istio-system
   subjects:
   - kind: ServiceAccount
     name: istio-ca-service-account
     namespace: istio-system
   roleRef:
     kind: ClusterRole
     name: istio-ca-istio-system
     apiGroup: rbac.authorization.k8s.io
   

   kubectl apply -f istio-ca-istio-system.yaml
   

8. Verify that the new ClusterRoles work as expected

   kubectl auth can-i get secret/istio-ingress-certs --as system:serviceaccount:istio-system:istio-ingress-service-account -n istio-system
   

   whose output should be

   yes
   

   In this command, we can replace verb “get” with “list” or “watch”, and the output should always be “yes”. Now let us test with other service accounts.

   kubectl auth can-i get secret/istio-ingress-certs --as system:serviceaccount:istio-system:istio-pilot-service-account -n istio-system
   

   whose output should be

   no - Unknown user "system:serviceaccount:istio-system:istio-pilot-service-account"
   

   In this command, we can replace service account with istio-mixer-service-account, or istio-ca-service-account, we can also replace verb “get” with “watch” or “list”, and the output should look similarly.

   Accessibility to secret resources except istio-ingress-certs should remain the same for istio-ca-service-account, istio-ingress-service-account, istio-pilot-service-account and istio-mixer-service-account.

   kubectl auth can-i get secret/istio-ca-service-account-token-r14xm --as system:serviceaccount:istio-system:istio-ca-service-account -n istio-system
   

   whose output should be

   yes
   

9. Cleanup

   We can delete these newly defined ClusterRoles and ClusterRoleBindings, and restore original ClusterRoles and ClusterRoleBindings according to those recorded copies.

Using Istio Routing Rules with Ingress

Istio’s routing rules can be used to achieve a greater degree of control when routing requests to backend services. For example, the following route rule sets a 4s timeout for all calls to the httpbin service on the /delay URL.

cat <<EOF | istioctl create -f -
apiVersion: config.istio.io/v1alpha2
kind: RouteRule
metadata:
  name: status-route
spec:
  destination:
    name: httpbin
  match:
    # Optionally limit this rule to istio ingress pods only
    source:
      name: istio-ingress
      labels:
        istio: ingress
    request:
      headers:
        uri:
          prefix: /delay/ #must match the path specified in ingress spec
              # if using prefix paths (/delay/.*), omit the .*.
              # if using exact match, use exact: /status
  route:
  - weight: 100
  httpReqTimeout:
    simpleTimeout:
      timeout: 4s
EOF

If you were to make a call to the ingress with the URL http://$INGRESS_HOST/delay/10, you will find that the call returns in 4s instead of the expected 10s delay.

You can use other features of the route rules such as redirects, rewrites, routing to multiple versions, regular expression based match in HTTP headers, websocket upgrades, timeouts, retries, etc. Please refer to the routing rules for more details.

Note 1: Fault injection does not work at the Ingress

Note 2: When matching requests in the routing rule, use the same exact path or prefix as the one used in the Ingress specification.

Understanding ingresses

Ingresses provide gateways for external traffic to enter the Istio service mesh and make the traffic management and policy features of Istio available for edge services.

The servicePort field in the Ingress specification can take a port number (integer) or a name. The port name must follow the Istio port naming conventions (e.g., grpc-*, http2-*, http-*, etc.) in order to function properly. The name used must match the port name in the backend service declaration.

In the preceding steps we created a service inside the Istio service mesh and showed how to expose both HTTP and HTTPS endpoints of the service to external traffic. We also showed how to control the ingress traffic using an Istio route rule.

Cleanup

1. Remove the secret and Ingress Resource definitions.

   kubectl delete ingress simple-ingress secure-ingress 
   kubectl delete -n istio-system secret istio-ingress-certs
   

2. Shutdown the httpbin service.

   kubectl delete -f samples/httpbin/httpbin.yaml
   

What’s next

• Learn more about Ingress Resources.

• Learn more about Routing Rules.