Consuming External Web Services
Egress Rules for HTTPS traffic
In many cases, not all the parts of a microservices-based application reside in a service mesh. Sometimes, the microservices-based applications use functionality provided by legacy systems that reside outside the mesh. We may want to migrate these systems to the service mesh gradually. Until these systems are migrated, they must be accessed by the applications inside the mesh. In other cases, the applications use web services provided by external organizations, often over the World Wide Web.
In this blog post, I modify the Istio Bookinfo Sample Application to fetch book details from an external web service (Google Books APIs). I show how to enable external HTTPS traffic in Istio by using an egress rule. Finally, I explain the current issues related to the egress traffic control in Istio.
Bookinfo sample application with external details web service
Initial setting
To demonstrate the scenario of consuming an external web service, I start with a Kubernetes cluster with Istio installed. Then I deploy Istio Bookinfo Sample Application. This application uses the details microservice to fetch book details, such as the number of pages and the publisher. The original details microservice provides the book details without consulting any external service.
The example commands in this blog post work with Istio version 0.2+, with or without Mutual TLS enabled.
The Bookinfo configuration files required for the scenario of this post appear starting from Istio release version 0.5. The Bookinfo configuration files reside in the samples/bookinfo/kube directory of the Istio release archive.
Here is a copy of the end-to-end architecture of the application from the original Bookinfo Guide.
The Original Bookinfo Application
Bookinfo with details version 2
Let’s add a new version of the details microservice, v2, that fetches the book details from Google Books APIs.
kubectl apply -f <(istioctl kube-inject -f samples/bookinfo/kube/bookinfo-details-v2.yaml)
The updated architecture of the application now looks as follows:
The Bookinfo Application with details V2
Note that the Google Books web service is outside the Istio service mesh, the boundary of which is marked by a dotted line.
Now let’s direct all the traffic destined to the details microservice, to details version v2, using the following route rule:
cat <<EOF | istioctl create -f -
apiVersion: config.istio.io/v1alpha2
kind: RouteRule
metadata:
name: details-v2
namespace: default
spec:
destination:
name: details
route:
- labels:
version: v2
EOF
Let’s access the web page of the application, after determining the ingress IP and port.
Oops… Instead of the book details we have the Error fetching product details message displayed:
The Error Fetching Product Details Message
The good news is that our application did not crash. With a good microservice design, we do not have failure propagation. In our case, the failing details microservice does not cause the productpage microservice to fail. Most of the functionality of the application is still provided, despite the failure in the details microservice. We have graceful service degradation: as you can see, the reviews and the ratings are displayed correctly, and the application is still useful.
So what might have gone wrong? Ah… The answer is that I forgot to enable traffic from inside the mesh to an external service, in this case to the Google Books web service. By default, the Istio sidecar proxies (Envoy proxies) block all the traffic to destinations outside the cluster. To enable such traffic, we must define an egress rule.
Egress rule for Google Books web service
No worries, let’s define an egress rule and fix our application:
cat <<EOF | istioctl create -f -
apiVersion: config.istio.io/v1alpha2
kind: EgressRule
metadata:
name: googleapis
namespace: default
spec:
destination:
service: "*.googleapis.com"
ports:
- port: 443
protocol: https
EOF
Now accessing the web page of the application displays the book details without error:
Book Details Displayed Correctly
Note that our egress rule allows traffic to any domain matching *.googleapis.com, on port 443, using the HTTPS protocol. Let’s assume for the sake of the example that the applications in our Istio service mesh must access multiple subdomains of gooogleapis.com, for example www.googleapis.com and also fcm.googleapis.com. Our rule allows traffic to both www.googleapis.com and fcm.googleapis.com, since they both match *.googleapis.com. This wildcard feature allows us to enable traffic to multiple domains using a single egress rule.
We can query our egress rules:
istioctl get egressrules
and see our new egress rule in the output:
NAME KIND NAMESPACE
googleapis EgressRule.v1alpha2.config.istio.io default
We can delete our egress rule:
istioctl delete egressrule googleapis -n default
and see in the output of istioctl delete that the egress rule is deleted:
Deleted config: egressrule googleapis
Accessing the web page after deleting the egress rule produces the same error that we experienced before, namely Error fetching product details. As we can see, the egress rules are defined dynamically, as many other Istio configuration artifacts. The Istio operators can decide dynamically which domains they allow the microservices to access. They can enable and disable traffic to the external domains on the fly, without redeploying the microservices.
Issues with Istio egress traffic control
TLS origination by Istio
There is a caveat to this story. In HTTPS, all the HTTP details (hostname, path, headers etc.) are encrypted, so Istio cannot know the destination domain of the encrypted requests. Well, Istio could know the destination domain by the SNI (Server Name Indication) field. This feature, however, is not yet implemented in Istio. Therefore, currently Istio cannot perform filtering of HTTPS requests based on the destination domains.
To allow Istio to perform filtering of egress requests based on domains, the microservices must issue HTTP requests. Istio then opens an HTTPS connection to the destination (performs TLS origination). The code of the microservices must be written differently or configured differently, according to whether the microservice runs inside or outside an Istio service mesh. This contradicts the Istio design goal of maximizing transparency. Sometimes we need to compromise…
The diagram below shows how the HTTPS traffic to external services is performed. On the top, a microservice outside an Istio service mesh sends regular HTTPS requests, encrypted end-to-end. On the bottom, the same microservice inside an Istio service mesh must send unencrypted HTTP requests inside a pod, which are intercepted by the sidecar Envoy proxy. The sidecar proxy performs TLS origination, so the traffic between the pod and the external service is encrypted.
HTTPS traffic to external services, from outside vs. from inside an Istio service mesh
Here is how we code this behavior in the the Bookinfo details microservice code, using the Ruby net/http module:
uri = URI.parse('https://www.googleapis.com/books/v1/volumes?q=isbn:' + isbn)
http = Net::HTTP.new(uri.host, uri.port)
...
unless ENV['WITH_ISTIO'] === 'true' then
http.use_ssl = true
end
Note that the port is derived by the URI.parse from the URI’s schema (https://) to be 443, the default HTTPS port. The microservice, when running inside an Istio service mesh, must issue HTTP requests to the port 443, which is the port the external service listens to.
When the WITH_ISTIO environment variable is defined, the request is performed without SSL (plain HTTP).
We set the WITH_ISTIO environment variable to “true” in the Kubernetes deployment spec of details v2, the container section:
env:
- name: WITH_ISTIO
value: "true"
Relation to Istio mutual TLS
Note that the TLS origination in this case is unrelated to the mutual TLS applied by Istio. The TLS origination for the external services will work, whether the Istio mutual TLS is enabled or not. The mutual TLS secures service-to-service communication inside the service mesh and provides each service with a strong identity. In the case of the external services, we have one-way TLS, the same mechanism used to secure communication between a web browser and a web server. TLS is applied to the communication with external services to verify the identity of the external server and to encrypt the traffic.
Malicious microservices threat
Another issue is that the egress rules are currently not a security feature; they only enable traffic to external services. For HTTP-based protocols, the rules are based on domains. Istio does not check that the destination IP of the request matches the Host header. This means that a malicious microservice inside a service mesh could trick Istio to allow traffic to a malicious IP. The attack is to set one of the domains allowed by some existing Egress Rule as the Host header of the malicious request.
Securing egress traffic is currently not supported in Istio and should be performed elsewhere, for example by a firewall or by an additional proxy outside Istio. Right now, we’re working to enable the application of Mixer security policies on the egress traffic and to prevent the attack described above.
No tracing, telemetry and no mixer checks
Note that currently no tracing and telemetry information can be collected for the egress traffic. Mixer policies cannot be applied. We are working to fix this in future Istio releases.
Future work
In my next blog posts I will demonstrate Istio egress rules for TCP traffic and will show examples of combining routing rules and egress rules.
In Istio, we are working on making Istio egress traffic more secure, and in particular on enabling tracing, telemetry, and Mixer checks for the egress traffic.
Conclusion
In this blog post I demonstrated how the microservices in an Istio service mesh can consume external web services via HTTPS. By default, Istio blocks all the traffic to the hosts outside the cluster. To enable such traffic, egress rules must be created for the service mesh. It is possible to access the external sites by HTTPS, however the microservices must issue HTTP requests while Istio will perform TLS origination. Currently, no tracing, telemetry and Mixer checks are enabled for the egress traffic. Egress rules are currently not a security feature, so additional mechanisms are required for securing egress traffic. We’re working to enable logging/telemetry and security policies for the egress traffic in future releases.
To read more about Istio egress traffic control, see Control Egress Traffic Task.