Debugging Envoy and Istiod
Istio provides two very valuable commands to help diagnose traffic management configuration problems,
the proxy-status
and proxy-config commands. The proxy-status command
allows you to get an overview of your mesh and identify the proxy causing the problem. Then proxy-config can be used
to inspect Envoy configuration and diagnose the issue.
If you want to try the commands described below, you can either:
- Have a Kubernetes cluster with Istio and Bookinfo installed (as described in installation steps1 and Bookinfo installation steps).
OR
- Use similar commands against your own application running in a Kubernetes cluster.
Get an overview of your mesh
The proxy-status command allows you to get an overview of your mesh. If you suspect one of your sidecars isn’t
receiving configuration or is out of sync then proxy-status will tell you this.
If a proxy is missing from this list it means that it is not currently connected to a Istiod instance so will not be receiving any configuration.
SYNCEDmeans that Envoy has acknowledged the last configuration Istiod has sent to it.NOT SENTmeans that Istiod hasn’t sent anything to Envoy. This usually is because Istiod has nothing to send.STALEmeans that Istiod has sent an update to Envoy but has not received an acknowledgement. This usually indicates a networking issue between Envoy and Istiod or a bug with Istio itself.
Retrieve diffs between Envoy and Istiod
The proxy-status command can also be used to retrieve a diff between the configuration Envoy has loaded and the
configuration Istiod would send, by providing a proxy ID. This can help you determine exactly what is out of sync and
where the issue may lie.
Here you can see that the listeners and routes match but the clusters are out of sync.
Deep dive into Envoy configuration
The proxy-config command can be used to see how a given Envoy instance is configured. This can then be used to
pinpoint any issues you are unable to detect by just looking through your Istio configuration and custom resources.
To get a basic summary of clusters, listeners or routes for a given pod use the command as follows (changing clusters
for listeners or routes when required):
In order to debug Envoy you need to understand Envoy clusters/listeners/routes/endpoints and how they all interact.
We will use the proxy-config command with the -o json and filtering flags to follow Envoy as it determines where
to send a request from the productpage pod to the reviews pod at reviews:9080.
If you query the listener summary on a pod you will notice Istio generates the following listeners:
- A listener on
0.0.0.0:15006that receives all inbound traffic to the pod and a listener on0.0.0.0:15001that receives all outbound traffic to the pod, then hands the request over to a virtual listener. - A virtual listener per service IP, per each non-HTTP for outbound TCP/HTTPS traffic.
- A virtual listener on the pod IP for each exposed port for inbound traffic.
- A virtual listener on
0.0.0.0per each HTTP port for outbound HTTP traffic.
- A listener on
From the above summary you can see that every sidecar has a listener bound to
0.0.0.0:15006which is where IP tables routes all inbound pod traffic to and a listener bound to0.0.0.0:15001which is where IP tables routes all outbound pod traffic to. This listener hasuseOriginalDstset to true which means it hands the request over to the listener that best matches the original destination of the request. If it can’t find any matching virtual listeners it sends the request to thePassthroughClusterwhich connects to the destination directly.Our request is an outbound HTTP request to port
9080this means it gets handed off to the0.0.0.0:9080virtual listener. This listener then looks up the route configuration in its configured RDS. In this case it will be looking up route9080in RDS configured by Istiod (via ADS).The
9080route configuration only has a virtual host for each service. Our request is heading to the reviews service so Envoy will select the virtual host to which our request matches a domain. Once matched on domain Envoy looks for the first route that matches the request. In this case we don’t have any advanced routing so there is only one route that matches on everything. This route tells Envoy to send the request to theoutbound|9080||reviews.default.svc.cluster.localcluster.This cluster is configured to retrieve the associated endpoints from Istiod (via ADS). So Envoy will then use the
serviceNamefield as a key to look up the list of Endpoints and proxy the request to one of them.To see the endpoints currently available for this cluster use the
proxy-configendpoints command.
Inspecting bootstrap configuration
So far we have looked at configuration retrieved (mostly) from Istiod, however Envoy requires some bootstrap configuration that includes information like where Istiod can be found. To view this use the following command:
Verifying connectivity to Istiod
Verifying connectivity to Istiod is a useful troubleshooting step. Every proxy container in the service mesh should be able to communicate with Istiod. This can be accomplished in a few simple steps:
Create a
sleeppod:Test connectivity to Istiod using
curl. The following example invokes the v1 registration API using default Istiod configuration parameters and mutual TLS enabled:
You should receive a response listing the “service” and “endpoint” for each service in the mesh.
What Envoy version is Istio using?
To find out the Envoy version used in deployment, you can exec into the container and query the server_info endpoint: