Configure tracing with Telemetry API

Istio provides the ability to configure advanced tracing options, such as sampling rate and adding custom tags to reported spans. This task shows you how to customize the tracing options with Telemetry API.

Before you begin

1. Ensure that your applications propagate tracing headers as described here.

2. Follow the tracing installation guide located under Integrations based on your preferred tracing backend to install the appropriate addon and configure your Istio proxies to send traces to the tracing deployment.

Installation

In this example, we will send tracing to zipkin so make sure it is installed:

$ cat <<EOF > ./tracing.yaml
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  meshConfig:
    enableTracing: true
    defaultConfig:
      tracing: {} # disabled MeshConfig tracing options
    extensionProviders:
    # add zipkin provider
    - name: zipkin
      zipkin:
        service: zipkin.istio-system.svc.cluster.local
        port: 9411
EOF
$ istioctl install -f ./tracing.yaml --skip-confirmation

Enable tracing for mesh

Enable tracing by applying the following configuration:

$ kubectl apply -f - <<EOF
apiVersion: telemetry.istio.io/v1
kind: Telemetry
metadata:
  name: mesh-default
  namespace: istio-system
spec:
  tracing:
    - providers:
        - name: "zipkin"
EOF

Customizing Trace sampling

The sampling rate option can be used to control what percentage of requests get reported to your tracing system. This should be configured based upon your traffic in the mesh and the amount of tracing data you want to collect. The default rate is 1%.

$ kubectl apply -f - <<EOF
apiVersion: telemetry.istio.io/v1
kind: Telemetry
metadata:
  name: mesh-default
  namespace: istio-system
spec:
  tracing:
    - providers:
        - name: "zipkin"
      randomSamplingPercentage: 100.00
EOF

Customizing tracing tags

Custom tags can be added to spans based on literals, environmental variables and client request headers in order to provide additional information in spans specific to your environment.

You can customize the tags using any of the three supported options below.

1. Literal represents a static value that gets added to each span.

   apiVersion: telemetry.istio.io/v1
   kind: Telemetry
   metadata:
   name: mesh-default
   namespace: istio-system
   spec:
   tracing:
       - providers:
           - name: "zipkin"
       randomSamplingPercentage: 100.00
       customTags:
         "provider":
           literal:
             value: "zipkin"
   

2. Environmental variables can be used where the value of the custom tag is populated from a workload proxy environment variable.

   apiVersion: telemetry.istio.io/v1
   kind: Telemetry
   metadata:
     name: mesh-default
     namespace: istio-system
   spec:
     tracing:
       - providers:
           - name: "zipkin"
         randomSamplingPercentage: 100.00
         customTags:
           "cluster_id":
             environment:
               name: ISTIO_META_CLUSTER_ID
               defaultValue: Kubernetes # optional
   

3. Client request header option can be used to populate tag value from an incoming client request header.

   apiVersion: telemetry.istio.io/v1
   kind: Telemetry
   metadata:
     name: mesh-default
     namespace: istio-system
   spec:
     tracing:
       - providers:
           - name: "zipkin"
         randomSamplingPercentage: 100.00
         customTags:
           my_tag_header:
             header:
               name: <CLIENT-HEADER>
               defaultValue: <VALUE>      # optional
   

Customizing tracing tag length

By default, the maximum length for the request path included as part of the HttpUrl span tag is 256. To modify this maximum length, add the following to your tracing.yaml file.

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  meshConfig:
    enableTracing: true
    defaultConfig:
      tracing: {} # disabled tracing options via `MeshConfig`
    extensionProviders:
    # add zipkin provider
    - name: zipkin
      zipkin:
        service: zipkin.istio-system.svc.cluster.local
        port: 9411
        maxTagLength: <VALUE>

Verify the results

You can verify the results with Zipkin UI.