Customizing Istio Metrics with Telemetry API
Telemetry API has been in Istio as a first-class API for quite sometime now.
Previously, users had to configure metrics in the telemetry
section of the Istio configuration.
This task shows you how to customize the metrics that Istio generates with Telemetry API.
Before you begin
Install Istio in your cluster and deploy an application.
Telemetry API can not work together with EnvoyFilter
. For more details please checkout this issue.
Starting with Istio version
1.18
, the PrometheusEnvoyFilter
will not be installed by default, and insteadmeshConfig.defaultProviders
is used to enable it. Telemetry API should be used to further customize the telemetry pipeline.For versions of Istio before
1.18
, you should install with the followingIstioOperator
configuration:apiVersion: install.istio.io/v1alpha1 kind: IstioOperator spec: values: telemetry: enabled: true v2: enabled: false
Override metrics
The metrics
section provides values for the metric dimensions as expressions,
and allows you to remove or override the existing metric dimensions.
You can modify the standard metric definitions using tags_to_remove
or by re-defining a dimension.
Remove
grpc_response_status
tags fromREQUEST_COUNT
metricapiVersion: telemetry.istio.io/v1alpha1 kind: Telemetry metadata: name: remove-tags namespace: istio-system spec: metrics: - providers: - name: prometheus overrides: - match: mode: CLIENT_AND_SERVER metric: REQUEST_COUNT tagOverrides: grpc_response_status: operation: REMOVE
Add custom tags for
REQUEST_COUNT
metricapiVersion: telemetry.istio.io/v1alpha1 kind: Telemetry metadata: name: custom-tags namespace: istio-system spec: metrics: - overrides: - match: metric: REQUEST_COUNT mode: CLIENT tagOverrides: destination_x: value: upstream_peer.labels['app'].value - match: metric: REQUEST_COUNT mode: SERVER tagOverrides: source_x: value: downstream_peer.labels['app'].value providers: - name: prometheus
Disable metrics
Disable all metrics by following configuration:
apiVersion: telemetry.istio.io/v1alpha1 kind: Telemetry metadata: name: remove-all-metrics namespace: istio-system spec: metrics: - providers: - name: prometheus overrides: - disabled: true match: mode: CLIENT_AND_SERVER metric: ALL_METRICS
Disable
REQUEST_COUNT
metrics by following configuration:apiVersion: telemetry.istio.io/v1alpha1 kind: Telemetry metadata: name: remove-request-count namespace: istio-system spec: metrics: - providers: - name: prometheus overrides: - disabled: true match: mode: CLIENT_AND_SERVER metric: REQUEST_COUNT
Disable
REQUEST_COUNT
metrics for client by following configuration:apiVersion: telemetry.istio.io/v1alpha1 kind: Telemetry metadata: name: remove-client namespace: istio-system spec: metrics: - providers: - name: prometheus overrides: - disabled: true match: mode: CLIENT metric: REQUEST_COUNT
Disable
REQUEST_COUNT
metrics for server by following configuration:apiVersion: telemetry.istio.io/v1alpha1 kind: Telemetry metadata: name: remove-server namespace: istio-system spec: metrics: - providers: - name: prometheus overrides: - disabled: true match: mode: SERVER metric: REQUEST_COUNT
Verify the results
Send traffic to the mesh. For the Bookinfo sample, visit http://$GATEWAY_URL/productpage
in your web
browser or issue the following command:
$ curl "http://$GATEWAY_URL/productpage"
Use the following command to verify that Istio generates the data for your new or modified dimensions:
$ istioctl x es "$(kubectl get pod -l app=productpage -o jsonpath='{.items[0].metadata.name}')" -oprom | grep istio_requests_total | grep -v TYPE |grep -v 'reporter="destination"'
$ istioctl x es "$(kubectl get pod -l app=details -o jsonpath='{.items[0].metadata.name}')" -oprom | grep istio_requests_total
For example, in the output, locate the metric istio_requests_total
and
verify it contains your new dimension.