Distributed Tracing FAQ

Istio integrates with distributed tracing systems using Envoy-based tracing. With Envoy-based tracing integration, applications are responsible for forwarding tracing headers for subsequent outgoing requests.

You can find additional information in the Distributed Tracing overview and in the Envoy tracing docs.

Istio enables reporting of trace spans for workload-to-workload communications within a mesh. However, in order for various trace spans to be stitched together for a complete view of the traffic flow, applications must propagate the trace context between incoming and outgoing requests.

In particular, Istio relies on applications to forward the Envoy-generated request ID, and standard headers. These headers include:

• x-request-id
• traceparent
• tracestate

Zipkin users must ensure they propagate the B3 trace headers.

• x-b3-traceid
• x-b3-spanid
• x-b3-parentspanid
• x-b3-sampled
• x-b3-flags
• b3

Header propagation may be accomplished through client libraries, such as OpenTelemetry. It can also be accomplished manually, as documented in the Distributed Tracing task.

For Envoy-based tracing integrations, Envoy (the sidecar proxy) sends tracing information directly to tracing backends on behalf of the applications being proxied.

Envoy:

• generates request IDs and trace headers (i.e., X-B3-TraceId) for requests as they flow through the proxy
• generates trace spans for each request based on request and response metadata (i.e., response time)
• sends the generated trace spans to the tracing backends
• forwards the trace headers to the proxied application

Istio supports OpenTelemetry and compatible backends including Jaeger. Other supported platforms include Zipkin and Apache SkyWalking.

The Istio gateway or sidecar proxy (Envoy) generates the initial headers, if they are not provided by the request.

Although an Istio sidecar will process both inbound and outbound requests for an associated application instance, it has no implicit way of correlating the outbound requests to the inbound request that caused them. The only way this correlation can be achieved is if the application propagates relevant information (i.e. headers) from the inbound request to the outbound requests. Header propagation may be accomplished through client libraries or manually. Further discussion is provided in What is required for distributed tracing with Istio?.

The sampling rate for tracing is set at 1% in the default configuration profile. This means that only 1 out of 100 trace instances captured by Istio will be reported to the tracing backend. The sampling rate in the demo profile is set to 100%. See this section for information on how to set the sampling rate.

If you still do not see any trace data, please confirm that your ports conform to the Istio port naming conventions and that the appropriate container port is exposed (via pod spec, for example) to enable traffic capture by the sidecar proxy (Envoy).

If you only see trace data associated with the egress proxy, but not the ingress proxy, it may still be related to the Istio port naming conventions.

Istio, via Envoy, currently supports a percentage-based sampling strategy for trace generation. Please see this section for more information on how to set this sampling rate.

Istio does not currently provide support for pub/sub and event bus protocols. Any use of those technologies is best-effort and subject to breakage.