Classifying Metrics Based on Request or Response

It’s useful to visualize telemetry based on the type of requests and responses handled by services in your mesh. For example, a bookseller tracks the number of times book reviews are requested. A book review request has this structure:

GET /reviews/{review_id}

Counting the number of review requests must account for the unbounded element review_id. GET /reviews/1 followed by GET /reviews/2 should count as two requests to get reviews.

Istio lets you create classification rules using the AttributeGen plugin that groups requests into a fixed number of logical operations. For example, you can create an operation named GetReviews, which is a common way to identify operations using the Open API Spec operationId. This information is injected into request processing as istio_operationId attribute with value equal to GetReviews. You can use the attribute as a dimension in Istio standard metrics. Similarly, you can track metrics based on other operations like ListReviews and CreateReviews.

Classify metrics by request

You can classify requests based on their type, for example ListReview, GetReview, CreateReview.

  1. Create a file, for example attribute_gen_service.yaml, and save it with the following contents. This adds the istio.attributegen plugin. It also creates an attribute, istio_operationId and populates it with values for the categories to count as metrics.

    This configuration is service-specific since request paths are typically service-specific.

    apiVersion: extensions.istio.io/v1alpha1
    kind: WasmPlugin
    metadata:
      name: istio-attributegen-filter
    spec:
      selector:
        matchLabels:
          app: reviews
      url: https://storage.googleapis.com/istio-build/proxy/attributegen-359dcd3a19f109c50e97517fe6b1e2676e870c4d.wasm
      imagePullPolicy: Always
      phase: AUTHN
      pluginConfig:
        attributes:
        - output_attribute: "istio_operationId"
          match:
            - value: "ListReviews"
              condition: "request.url_path == '/reviews' && request.method == 'GET'"
            - value: "GetReview"
              condition: "request.url_path.matches('^/reviews/[[:alnum:]]*$') && request.method == 'GET'"
            - value: "CreateReview"
              condition: "request.url_path == '/reviews/' && request.method == 'POST'"
    ---
    apiVersion: telemetry.istio.io/v1alpha1
    kind: Telemetry
    metadata:
      name: custom-tags
    spec:
      metrics:
        - overrides:
            - match:
                metric: REQUEST_COUNT
                mode: CLIENT_AND_SERVER
              tagOverrides:
                request_operation:
                  value: istio_operationId
          providers:
            - name: prometheus
    
  2. Apply your changes using the following command:

    $ kubectl -n istio-system apply -f attribute_gen_service.yaml
    
  3. After the changes take effect, visit Prometheus and look for the new or changed dimensions, for example istio_requests_total in reviews pods.

Classify metrics by response

You can classify responses using a similar process as requests. Do note that the response_code dimension already exists by default. The example below will change how it is populated.

  1. Create a file, for example attribute_gen_service.yaml, and save it with the following contents. This adds the istio.attributegen plugin and generates the istio_responseClass attribute used by the stats plugin.

    This example classifies various responses, such as grouping all response codes in the 200 range as a 2xx dimension.

    apiVersion: extensions.istio.io/v1alpha1
    kind: WasmPlugin
    metadata:
      name: istio-attributegen-filter
    spec:
      selector:
        matchLabels:
          app: productpage
      url: https://storage.googleapis.com/istio-build/proxy/attributegen-359dcd3a19f109c50e97517fe6b1e2676e870c4d.wasm
      imagePullPolicy: Always
      phase: AUTHN
      pluginConfig:
        attributes:
          - output_attribute: istio_responseClass
            match:
              - value: 2xx
                condition: response.code >= 200 && response.code <= 299
              - value: 3xx
                condition: response.code >= 300 && response.code <= 399
              - value: "404"
                condition: response.code == 404
              - value: "429"
                condition: response.code == 429
              - value: "503"
                condition: response.code == 503
              - value: 5xx
                condition: response.code >= 500 && response.code <= 599
              - value: 4xx
                condition: response.code >= 400 && response.code <= 499
    ---
    apiVersion: telemetry.istio.io/v1alpha1
    kind: Telemetry
    metadata:
      name: custom-tags
    spec:
      metrics:
        - overrides:
            - match:
                metric: REQUEST_COUNT
                mode: CLIENT_AND_SERVER
              tagOverrides:
                response_code:
                  value: istio_responseClass
          providers:
            - name: prometheus
    
  2. Apply your changes using the following command:

    $ kubectl -n istio-system apply -f attribute_gen_service.yaml
    

Verify the results

  1. Generate metrics by sending traffic to your application.

  2. Visit Prometheus and look for the new or changed dimensions, for example 2xx. Alternatively, use the following command to verify that Istio generates the data for your new dimension:

    $ kubectl exec pod-name -c istio-proxy -- curl -sS 'localhost:15000/stats/prometheus' | grep istio_
    

    In the output, locate the metric (e.g. istio_requests_total) and verify the presence of the new or changed dimension.

Troubleshooting

If classification does not occur as expected, check the following potential causes and resolutions.

Review the Envoy proxy logs for the pod that has the service on which you applied the configuration change. Check that there are no errors reported by the service in the Envoy proxy logs on the pod, (pod-name), where you configured classification by using the following command:

$ kubectl logs pod-name -c istio-proxy | grep -e "Config Error" -e "envoy wasm"

Additionally, ensure that there are no Envoy proxy crashes by looking for signs of restarts in the output of the following command:

$ kubectl get pods pod-name
Was this information useful?
Do you have any suggestions for improvement?

Thanks for your feedback!