Upcoming networking changes in Istio 1.10

Understanding the upcoming changes to Istio networking, how they may impact your cluster, and what action to take.

Apr 15, 2021 | By John Howard - Google

Background

While Kubernetes networking is customizable, a typical pod’s network will look like this:

A pod's network
A pod's network

An application may choose to bind to either the loopback interface lo (typically binding to 127.0.0.1), or the pods network interface eth0 (typically to the pod’s IP), or both (typically binding to 0.0.0.0).

Binding to lo allows calls such as curl localhost to work from within the pod. Binding to eth0 allows calls to the pod from other pods.

Typically, an application will bind to both. However, applications which have internal logic, such as an admin interface may choose to bind to only lo to avoid access from other pods. Additionally, some applications, typically stateful applications, choose to bind only to eth0.

Current behavior

In Istio prior to release 1.10, the Envoy proxy, running in the same pod as the application, binds to the eth0 interface and redirects all inbound traffic to the lo interface.

A pod's network with Istio today
A pod's network with Istio today

This has two important side effects that cause the behavior to differ from standard Kubernetes:

Applications that bind to both interfaces (which is typical) will not be impacted.

Future behavior

Starting with Istio 1.10, the networking behavior is changed to align with the standard behavior present in Kubernetes.

A pod's network with Istio in the future
A pod's network with Istio in the future

Here we can see that the proxy no longer redirects the traffic to the lo interface, but instead forwards it to the application on eth0. As a result, the standard behavior of Kubernetes is retained, but we still get all the benefits of Istio. This change allows Istio to get closer to its goal of being a drop-in transparent proxy that works with existing workloads with zero configuration. Additionally, it avoids unintended exposure of applications binding only to lo.

Am I impacted?

For new users, this change should only be an improvement. However, if you are an existing user, you may have come to depend on the old behavior, intentionally or accidentally.

To help detect these situations, we have added a check to find pods that will be impacted. You can run the istioctl experimental precheck command to get a report of any pods binding to lo on a port exposed in a Service. This command is available in Istio 1.10+. Without action, these ports will no longer be accessible upon upgrade.

$ istioctl experimental precheck
Error [IST0143] (Pod echo-local-849647c5bd-g9wxf.default) Port 443 is exposed in a Service but listens on localhost. It will not be exposed to other pods.
Error [IST0143] (Pod echo-local-849647c5bd-g9wxf.default) Port 7070 is exposed in a Service but listens on localhost. It will not be exposed to other pods.
Error: Issues found when checking the cluster. Istio may not be safe to install or upgrade.
See https://istio.io/latest/docs/reference/config/analysis for more information about causes and resolutions.

Migration

If you are currently binding to lo, you have a few options: