Proxying legacy services using Istio egress gateways
Deploy multiple Istio egress gateways independently to have fine-grained control of egress communication from the mesh.
At Deutsche Telekom Pan-Net, we have embraced Istio as the umbrella to cover our services. Unfortunately, there are services which have not yet been migrated to Kubernetes, or cannot be.
We can set Istio up as a proxy service for these upstream services. This allows us to benefit from capabilities like authorization/authentication, traceability and observability, even while legacy services stand as they are.
At the end of this article there is a hands-on exercise where you can simulate the scenario. In the exercise, an upstream service hosted at https://httpbin.org will be proxied by an Istio egress gateway.
If you are familiar with Istio, one of the methods offered to connect to upstream services is through an egress gateway.
You can deploy one to control all the upstream traffic or you can deploy multiple in order to have fine-grained control and satisfy the single-responsibility principle as this picture shows:
With this model, one egress gateway is in charge of exactly one upstream service.
Although the Operator spec allows you to deploy multiple egress gateways, the manifest can become unmanageable:
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
[...]
spec:
egressGateways:
- name: egressgateway-1
enabled: true
- name: egressgateway-2
enabled: true
[egressgateway-3, egressgateway-4, ...]
- name: egressgateway-N
enabled: true
[...]
As a benefit of decoupling egress getaways from the Operator manifest, you have enabled the possibility of setting up custom readiness probes to have both services (Gateway and upstream Service) aligned.
You can also inject OPA as a sidecar into the pod to perform authorization with complex rules (OPA envoy plugin).
As you can see, your possibilities increase and Istio becomes very extensible.
Let’s look at how you can implement this pattern.
Solution
There are several ways to perform this task, but here you will find how to define multiple Operators and deploy the generated resources.
In the following section you will deploy an egress gateway to connect to an upstream service: httpbin (https://httpbin.org/)
At the end, you will have:
Hands on
Prerequisites
Kind
Save this as config.yaml.
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
kubeadmConfigPatches:
- |
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
metadata:
name: config
apiServer:
extraArgs:
"service-account-issuer": "kubernetes.default.svc"
"service-account-signing-key-file": "/etc/kubernetes/pki/sa.key"
$ kind create cluster --name <my-cluster-name> --config config.yaml
Where <my-cluster-name> is the name for the cluster.
Istio Operator with Istioctl
Install the Operator
$ istioctl operator init --watchedNamespaces=istio-operator
$ kubectl create ns istio-system
Save this as operator.yaml:
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
name: istio-operator
namespace: istio-operator
spec:
profile: default
tag: 1.8.0
meshConfig:
accessLogFile: /dev/stdout
outboundTrafficPolicy:
mode: REGISTRY_ONLY
$ kubectl apply -f operator.yaml
Deploy Egress Gateway
The steps for this task assume:
- The service is installed under the namespace:
httpbin. - The service name is:
http-egress.
Istio 1.8 introduced the possibility to apply overlay configuration, to give fine-grain control over the created resources.
Save this as egress.yaml:
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
profile: empty
tag: 1.8.0
namespace: httpbin
components:
egressGateways:
- name: httpbin-egress
enabled: true
label:
app: istio-egressgateway
istio: egressgateway
custom-egress: httpbin-egress
k8s:
overlays:
- kind: Deployment
name: httpbin-egress
patches:
- path: spec.template.spec.containers[0].readinessProbe
value:
failureThreshold: 30
exec:
command:
- /bin/sh
- -c
- curl http://localhost:15021/healthz/ready && curl https://httpbin.org/status/200
initialDelaySeconds: 1
periodSeconds: 2
successThreshold: 1
timeoutSeconds: 1
values:
gateways:
istio-egressgateway:
runAsRoot: true
Create the namespace where you will install the egress gateway:
$ kubectl create ns httpbin
As it is described in the documentation, you can deploy several Operator resources. However, they have to be pre-parsed and then applied to the cluster.
$ istioctl manifest generate -f egress.yaml | kubectl apply -f -
Istio configuration
Now you will configure Istio to allow connections to the upstream service at https://httpbin.org.
Certificate for TLS
You need a certificate to make a secure connection from outside the cluster to your egress service.
How to generate a certificate is explained in the Istio ingress documentation.
Create and apply one to be used at the end of this article to access the service from outside the cluster (<my-proxied-service-hostname>):
$ kubectl create -n istio-system secret tls <my-secret-name> --key=<key> --cert=<cert>
Where <my-secret-name> is the name used later for the Gateway resource. <key> and <cert> are the files for the certificate. <cert>.
Ingress Gateway
Create a Gateway resource to operate ingress gateway to accept requests.
An example:
apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
name: my-ingressgateway
namespace: istio-system
spec:
selector:
istio: ingressgateway
servers:
- hosts:
- "<my-proxied-service-hostname>"
port:
name: http
number: 80
protocol: HTTP
tls:
httpsRedirect: true
- port:
number: 443
name: https
protocol: https
hosts:
- "<my-proxied-service-hostname>"
tls:
mode: SIMPLE
credentialName: <my-secret-name>
Where <my-proxied-service-hostname> is the hostname to access the service through the my-ingressgateway and <my-secret-name> is the secret which contains the certificate.
Egress Gateway
Create another Gateway object, but this time to operate the egress gateway you have already installed:
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: "httpbin-egress"
namespace: "httpbin"
spec:
selector:
istio: egressgateway
service.istio.io/canonical-name: "httpbin-egress"
servers:
- hosts:
- "<my-proxied-service-hostname>"
port:
number: 80
name: http
protocol: HTTP
Where <my-proxied-service-hostname> is the hostname to access through the my-ingressgateway.
Virtual Service
Create a VirtualService for three use cases:
- Mesh gateway for service-to-service communications within the mesh
- Ingress Gateway for the communication from outside the mesh
- Egress Gateway for the communication to the upstream service
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: "httpbin-egress"
namespace: "httpbin"
spec:
hosts:
- "<my-proxied-service-hostname>"
gateways:
- mesh
- "istio-system/my-ingressgateway"
- "httpbin/httpbin-egress"
http:
- match:
- gateways:
- "istio-system/my-ingressgateway"
- mesh
uri:
prefix: "/"
route:
- destination:
host: "httpbin-egress.httpbin.svc.cluster.local"
port:
number: 80
- match:
- gateways:
- "httpbin/httpbin-egress"
uri:
prefix: "/"
route:
- destination:
host: "httpbin.org"
subset: "http-egress-subset"
port:
number: 443
Where <my-proxied-service-hostname> is the hostname to access through the my-ingressgateway.
Service Entry
Create a ServiceEntry to allow the communication to the upstream service:
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
name: "httpbin-egress"
namespace: "httpbin"
spec:
hosts:
- "httpbin.org"
location: MESH_EXTERNAL
ports:
- number: 443
name: https
protocol: TLS
resolution: DNS
Destination Rule
Create a DestinationRule to allow TLS origination for egress traffic as explained in the documentation
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: "httpbin-egress"
namespace: "httpbin"
spec:
host: "httpbin.org"
subsets:
- name: "http-egress-subset"
trafficPolicy:
loadBalancer:
simple: ROUND_ROBIN
portLevelSettings:
- port:
number: 443
tls:
mode: SIMPLE
Peer Authentication
To secure the service-to-service, you need to enforce mTLS:
apiVersion: "security.istio.io/v1beta1"
kind: "PeerAuthentication"
metadata:
name: "httpbin-egress"
namespace: "httpbin"
spec:
mtls:
mode: STRICT
Test
Verify that your objects were all specified correctly:
$ istioctl analyze --all-namespaces
External access
Test the egress gateway from outside the cluster forwarding the ingressgateway service’s port and calling the service
$ kubectl -n istio-system port-forward svc/istio-ingressgateway 15443:443
$ curl -vvv -k -HHost:<my-proxied-service-hostname> --resolve "<my-proxied-service-hostname>:15443:127.0.0.1" --cacert <cert> "https://<my-proxied-service-hostname>:15443/status/200"
Where <my-proxied-service-hostname> is the hostname to access through the my-ingressgateway and <cert> is the certificate defined for the ingressgateway object. This is due to tls.mode: SIMPLE which does not terminate TLS
Service-to-service access
Test the egress gateway from inside the cluster deploying the sleep service. This is useful when you design failover.
$ kubectl label namespace httpbin istio-injection=enabled --overwrite
$ kubectl apply -n httpbin -f https://raw.githubusercontent.com/istio/istio/release-1.22/samples/sleep/sleep.yaml
$ kubectl -n httpbin "$(kubectl get pod -n httpbin -l app=sleep -o jsonpath={.items..metadata.name})" -- curl -vvv http://<my-proxied-service-hostname>/status/200
Where <my-proxied-service-hostname> is the hostname to access through the my-ingressgateway.
Now it is time to create a second, third and fourth egress gateway pointing to other upstream services.
Final thoughts
Istio might seem complex to configure. But it is definitely worthwhile, due to the huge set of benefits it brings to your services (with an extra Olé! for Kiali).
The way Istio is developed allows us, with minimal effort, to satisfy uncommon requirements like the one presented in this article.
To finish, I just wanted to point out that Istio, as a good cloud native technology, does not require a large team to maintain. For example, our current team is composed of 3 engineers.
To discuss more about Istio and its possibilities, please contact one of us:
