Trace Sampling

Istio provides multiple ways to configure trace sampling. In this page you will learn and understand all the different ways sampling can be configured.

Before you begin

1. Ensure that your applications propagate tracing headers as described here.

Available trace sampling configurations

1. Percentage Sampler: A random sampling rate for percentage of requests that will be selected for trace generation.

2. Custom OpenTelemetry Sampler: A custom sampler implementation, that must be paired with the OpenTelemetryTracingProvider.

3. Deploy the OpenTelemetry Collector

   • Create a namespace for the OpenTelemetry Collector:

     $ kubectl create namespace observability
         

   • Deploy the OpenTelemetry Collector. You can use this example configuration as a starting point: otel.yaml

     Zip

     $ kubectl apply -f @samples/open-telemetry/otel.yaml@ -n observability
         

Percentage Sampler

The random sampling rate percentage uses the specified percentage value to pick which requests to sample.

The sampling rate should be in the range of 0.0 to 100.0 with a precision of 0.01. For example, to trace 5 requests out of every 10000, use 0.05 as the value here.

There are three ways you can configure the random sampling rate:

Globally via MeshConfig

Random percentage sampling can be configured globally via MeshConfig.

$ cat <<EOF | istioctl install -y -f -
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  meshConfig:
    enableTracing: true
    defaultConfig:
      tracing:
        sampling: 10
    extensionProviders:
    - name: otel-tracing
      opentelemetry:
        port: 4317
        service: opentelemetry-collector.observability.svc.cluster.local
        resource_detectors:
          environment: {}
EOF

Then enable the tracing provider via Telemetry API. Note we don’t set randomSamplingPercentage here.

$ kubectl apply -f - <<EOF
apiVersion: telemetry.istio.io/v1
kind: Telemetry
metadata:
  name: mesh-default
  namespace: istio-system
spec:
  tracing:
  - providers:
    - name: otel-tracing
EOF

Pod annotation proxy.istio.io/config

You can add the proxy.istio.io/config annotation to your Pod metadata specification to override any mesh-wide sampling settings.

For instance, to override the mesh-wide sampling above, you would add the following to your pod manifest:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: sleep
spec:
  ...
  template:
    metadata:
      ...
      annotations:
        ...
        proxy.istio.io/config: |
          tracing:
            sampling: 20
    spec:
      ...

Telemetry API

The random percentage sampler can also be configured via the Telemetry API. Via the Telemetry API, sampling can be configured on various scopes: mesh-wide, namespace or workload, offering great flexibility. To learn more, please see the Telemetry API documentation.

Install Istio without setting sampling inside defaultConfig:

$ cat <<EOF | istioctl install -y -f -
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  meshConfig:
    enableTracing: true
    extensionProviders:
    - name: otel-tracing
      opentelemetry:
        port: 4317
        service: opentelemetry-collector.observability.svc.cluster.local
        resource_detectors:
          environment: {}
EOF

Then enable the tracing provider via Telemetry API and set the randomSamplingPercentage.

$ kubectl apply -f - <<EOF
apiVersion: telemetry.istio.io/v1
kind: Telemetry
metadata:
   name: otel-demo
spec:
  tracing:
  - providers:
    - name: otel-tracing
    randomSamplingPercentage: 10
EOF

Custom OpenTelemetry Sampler

The OpenTelemetry specification defines the Sampler API. The Sampler API enables building a custom sampler that can perform more intelligent and efficient sampling decisions, such as Probability Sampling.

Such samplers can then be paired with the OpenTelemetryTracingProvider.

Current custom sampler configurations in Istio:

• Dynatrace Sampler

Custom samplers are configured via Meshconfig. Here is an example of configuring the Dynatrace sampler:

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  meshConfig:
    extensionProviders:
    - name: otel-tracing
      opentelemetry:
        port: 443
        service: abc.live.dynatrace.com/api/v2/otlp
        http:
          path: "/api/v2/otlp/v1/traces"
          timeout: 10s
          headers:
            - name: "Authorization"
              value: "Api-Token dt0c01."
        dynatrace_sampler:
          tenant: "abc"
          cluster_id: 123

Order of precedence

With multiple ways of configuring sampling, it is important to understand the order of precedence of each method.

When using the random percentage sampler the order of precedence is:

Telemetry API > Pod Annotation > MeshConfig.

That means, if a value is defined in all of the above, the value on the Telemetry API is the one selected.

When a custom OpenTelemetry sampler is configured, the order of precedence is:

Custom OTel Sampler > (Telemetry API | Pod Annotation | MeshConfig)

That means, if a custom OpenTelemetry sampler is configured, it overrides all the others methods. Additionally, the random percentage value is set to 100 and cannot be changed. This is important because the custom sampler needs to receive 100% of spans to be able to properly perform its decision.

Deploy the Bookinfo Application

Deploy the Bookinfo sample application.

Generating traces using the Bookinfo sample

1. When the Bookinfo application is up and running, access http://$GATEWAY_URL/productpage one or more times to generate trace information.

   To see trace data, you must send requests to your service. The number of requests depends on Istio’s sampling rate and can be configured using the Telemetry API. With the default sampling rate of 1%, you need to send at least 100 requests before the first trace is visible. To send 100 requests to the productpage service, use the following command:

   $ for i in $(seq 1 100); do curl -s -o /dev/null "http://$GATEWAY_URL/productpage"; done
       

Cleanup

1. Remove the Telemetry resource:

   $ kubectl delete telemetry otel-demo
   

2. Remove any istioctl processes that may still be running using control-C or:

   $ istioctl uninstall --purge -y
   

3. Uninstall the OpenTelemetry Collector:

   Zip

   $ kubectl delete -f @samples/open-telemetry/otel.yaml@ -n observability
   $ kubectl delete namespace observability