Classifying Metrics Based on Request or Response
It’s useful to visualize telemetry based on the type of requests and responses handled by services in your mesh. For example, a bookseller tracks the number of times book reviews are requested. A book review request has this structure:
GET /reviews/{review_id}
Counting the number of review requests must account for the unbounded element
review_id
. GET /reviews/1
followed by GET /reviews/2
should count as two
requests to get reviews.
Istio lets you create classification rules using the
AttributeGen plugin that groups requests
into a fixed number of logical operations. For example, you can create an operation named
GetReviews
, which is a common way to identify operations using the
Open API Spec operationId
.
This information is injected into request processing as istio_operationId
attribute with
value equal to GetReviews
.
You can use the attribute as a dimension in Istio standard metrics. Similarly,
you can track metrics based on other operations like ListReviews
and
CreateReviews
.
Classify metrics by request
You can classify requests based on their type, for example ListReview
,
GetReview
, CreateReview
.
Create a file, for example
attribute_gen_service.yaml
, and save it with the following contents. This adds theistio.attributegen
plugin. It also creates an attribute,istio_operationId
and populates it with values for the categories to count as metrics.This configuration is service-specific since request paths are typically service-specific.
apiVersion: extensions.istio.io/v1alpha1 kind: WasmPlugin metadata: name: istio-attributegen-filter spec: selector: matchLabels: app: reviews url: https://storage.googleapis.com/istio-build/proxy/attributegen-359dcd3a19f109c50e97517fe6b1e2676e870c4d.wasm imagePullPolicy: Always phase: AUTHN pluginConfig: attributes: - output_attribute: "istio_operationId" match: - value: "ListReviews" condition: "request.url_path == '/reviews' && request.method == 'GET'" - value: "GetReview" condition: "request.url_path.matches('^/reviews/[[:alnum:]]*$') && request.method == 'GET'" - value: "CreateReview" condition: "request.url_path == '/reviews/' && request.method == 'POST'" --- apiVersion: telemetry.istio.io/v1alpha1 kind: Telemetry metadata: name: custom-tags spec: metrics: - overrides: - match: metric: REQUEST_COUNT mode: CLIENT_AND_SERVER tagOverrides: request_operation: value: istio_operationId providers: - name: prometheus
Apply your changes using the following command:
$ kubectl -n istio-system apply -f attribute_gen_service.yaml
After the changes take effect, visit Prometheus and look for the new or changed dimensions, for example
istio_requests_total
inreviews
pods.
Classify metrics by response
You can classify responses using a similar process as requests. Do note that the response_code
dimension already exists by default.
The example below will change how it is populated.
Create a file, for example
attribute_gen_service.yaml
, and save it with the following contents. This adds theistio.attributegen
plugin and generates theistio_responseClass
attribute used by the stats plugin.This example classifies various responses, such as grouping all response codes in the
200
range as a2xx
dimension.apiVersion: extensions.istio.io/v1alpha1 kind: WasmPlugin metadata: name: istio-attributegen-filter spec: selector: matchLabels: app: productpage url: https://storage.googleapis.com/istio-build/proxy/attributegen-359dcd3a19f109c50e97517fe6b1e2676e870c4d.wasm imagePullPolicy: Always phase: AUTHN pluginConfig: attributes: - output_attribute: istio_responseClass match: - value: 2xx condition: response.code >= 200 && response.code <= 299 - value: 3xx condition: response.code >= 300 && response.code <= 399 - value: "404" condition: response.code == 404 - value: "429" condition: response.code == 429 - value: "503" condition: response.code == 503 - value: 5xx condition: response.code >= 500 && response.code <= 599 - value: 4xx condition: response.code >= 400 && response.code <= 499 --- apiVersion: telemetry.istio.io/v1alpha1 kind: Telemetry metadata: name: custom-tags spec: metrics: - overrides: - match: metric: REQUEST_COUNT mode: CLIENT_AND_SERVER tagOverrides: response_code: value: istio_responseClass providers: - name: prometheus
Apply your changes using the following command:
$ kubectl -n istio-system apply -f attribute_gen_service.yaml
Verify the results
Generate metrics by sending traffic to your application.
Visit Prometheus and look for the new or changed dimensions, for example
2xx
. Alternatively, use the following command to verify that Istio generates the data for your new dimension:$ kubectl exec pod-name -c istio-proxy -- curl -sS 'localhost:15000/stats/prometheus' | grep istio_
In the output, locate the metric (e.g.
istio_requests_total
) and verify the presence of the new or changed dimension.
Troubleshooting
If classification does not occur as expected, check the following potential causes and resolutions.
Review the Envoy proxy logs for the pod that has the service on which you applied the configuration change. Check that there are no errors reported by the service in the Envoy proxy logs on the pod, (pod-name
), where you configured classification by using the following command:
$ kubectl logs pod-name -c istio-proxy | grep -e "Config Error" -e "envoy wasm"
Additionally, ensure that there are no Envoy proxy crashes by looking for signs of restarts in the output of the following command:
$ kubectl get pods pod-name
Cleanup
Remove the yaml configuration file.
$ kubectl -n istio-system delete -f attribute_gen_service.yaml