Trace Sampling
Istio provides multiple ways to configure trace sampling. In this page you will learn and understand all the different ways sampling can be configured.
Before you begin
- Ensure that your applications propagate tracing headers as described here.
Available trace sampling configurations
Percentage Sampler: A random sampling rate for percentage of requests that will be selected for trace generation.
Custom OpenTelemetry Sampler: A custom sampler implementation, that must be paired with the
OpenTelemetryTracingProvider
.Deploy the OpenTelemetry Collector
Create a namespace for the OpenTelemetry Collector:
$ kubectl create namespace observability
Deploy the OpenTelemetry Collector. You can use this example configuration as a starting point:
otel.yaml
$ kubectl apply -f @samples/open-telemetry/otel.yaml@ -n observability
Percentage Sampler
The random sampling rate percentage uses the specified percentage value to pick which requests to sample.
The sampling rate should be in the range of 0.0 to 100.0 with a precision of 0.01. For example, to trace 5 requests out of every 10000, use 0.05 as the value here.
There are three ways you can configure the random sampling rate:
Globally via MeshConfig
Random percentage sampling can be configured globally via MeshConfig
.
$ cat <<EOF | istioctl install -y -f -
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
meshConfig:
enableTracing: true
defaultConfig:
tracing:
sampling: 10
extensionProviders:
- name: otel-tracing
opentelemetry:
port: 4317
service: opentelemetry-collector.observability.svc.cluster.local
resource_detectors:
environment: {}
EOF
Then enable the tracing provider via Telemetry API. Note we don’t set randomSamplingPercentage
here.
$ kubectl apply -f - <<EOF
apiVersion: telemetry.istio.io/v1alpha1
kind: Telemetry
metadata:
name: mesh-default
namespace: istio-system
spec:
tracing:
- providers:
- name: otel-tracing
EOF
Pod annotation proxy.istio.io/config
You can add the proxy.istio.io/config
annotation to your Pod metadata
specification to override any mesh-wide sampling settings.
For instance, to override the mesh-wide sampling above, you would add the following to your pod manifest:
apiVersion: apps/v1
kind: Deployment
metadata:
name: sleep
spec:
...
template:
metadata:
...
annotations:
...
proxy.istio.io/config: |
tracing:
sampling: 20
spec:
...
Telemetry API
The random percentage sampler can also be configured via the Telemetry API. Via the Telemetry API, sampling can be configured on various scopes: mesh-wide, namespace or workload, offering great flexibility. To learn more, please see the Telemetry API documentation.
Install Istio without setting sampling
inside defaultConfig
:
$ cat <<EOF | istioctl install -y -f -
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
meshConfig:
enableTracing: true
extensionProviders:
- name: otel-tracing
opentelemetry:
port: 4317
service: opentelemetry-collector.observability.svc.cluster.local
resource_detectors:
environment: {}
EOF
Then enable the tracing provider via Telemetry API and set the randomSamplingPercentage
.
$ kubectl apply -f - <<EOF
apiVersion: telemetry.istio.io/v1alpha1
kind: Telemetry
metadata:
name: otel-demo
spec:
tracing:
- providers:
- name: otel-tracing
randomSamplingPercentage: 10
EOF
Custom OpenTelemetry Sampler
The OpenTelemetry specification defines the Sampler API. The Sampler API enables building a custom sampler that can perform more intelligent and efficient sampling decisions, such as Probability Sampling.
Such samplers can then be paired with the OpenTelemetryTracingProvider
.
Current custom sampler configurations in Istio:
Custom samplers are configured via Meshconfig
. Here is an example of configuring the Dynatrace sampler:
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
meshConfig:
extensionProviders:
- name: otel-tracing
opentelemetry:
port: 443
service: abc.live.dynatrace.com/api/v2/otlp
http:
path: "/api/v2/otlp/v1/traces"
timeout: 10s
headers:
- name: "Authorization"
value: "Api-Token dt0c01."
dynatrace_sampler:
tenant: "abc"
cluster_id: 123
Order of precedence
With multiple ways of configuring sampling, it is important to understand the order of precedence of each method.
When using the random percentage sampler the order of precedence is:
Telemetry API
> Pod Annotation
> MeshConfig
.
That means, if a value is defined in all of the above, the value on the Telemetry API
is the one selected.
When a custom OpenTelemetry sampler is configured, the order of precedence is:
Custom OTel Sampler
> (Telemetry API
| Pod Annotation
| MeshConfig
)
That means, if a custom OpenTelemetry sampler is configured, it overrides all the others methods.
Additionally, the random percentage value is set to 100
and cannot be changed. This is important
because the custom sampler needs to receive 100% of spans to be able to properly perform its decision.
Deploy the Bookinfo Application
Deploy the Bookinfo sample application.
Generating traces using the Bookinfo sample
When the Bookinfo application is up and running, access
http://$GATEWAY_URL/productpage
one or more times to generate trace information.To see trace data, you must send requests to your service. The number of requests depends on Istio’s sampling rate and can be configured using the Telemetry API. With the default sampling rate of 1%, you need to send at least 100 requests before the first trace is visible. To send 100 requests to the
productpage
service, use the following command:$ for i in $(seq 1 100); do curl -s -o /dev/null "http://$GATEWAY_URL/productpage"; done
Cleanup
Remove the Telemetry resource:
$ kubectl delete telemetry otel-demo
Remove any
istioctl
processes that may still be running using control-C or:$ istioctl uninstall --purge -y
Uninstall the OpenTelemetry Collector:
$ kubectl delete -f @samples/open-telemetry/otel.yaml@ -n observability $ kubectl delete namespace observability