Querying Metrics from Prometheus

This task shows you how to query for Istio Metrics using Prometheus. As part of this task, you will install the Prometheus Istio addon and use the web-based interface for querying metric values.

The Bookinfo sample application is used as the example application throughout this task.

Before you begin

Querying Istio Metrics

  1. To query the metrics provided by Mixer, first install the Prometheus add-on.

    In Kubernetes environments, execute the following command:

    kubectl apply -f install/kubernetes/addons/prometheus.yaml

  2. Verify that the service is running in your cluster.

    In Kubernetes environments, execute the following command:

    kubectl -n istio-system get svc prometheus

    The output will be similar to:

    NAME         CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
prometheus   10.59.241.54   <none>        9090/TCP   2m

  3. Send traffic to the mesh.

    For the Bookinfo sample, visit http://$GATEWAY_URL/productpage in your web browser or issue the following command:

    curl http://$GATEWAY_URL/productpage

    Note: $GATEWAY_URL is the value set in the Bookinfo guide.

  4. Open the Prometheus UI.

    In Kubernetes environments, execute the following command:

    kubectl -n istio-system port-forward $(kubectl -n istio-system get pod -l app=prometheus -o jsonpath='{.items[0].metadata.name}') 9090:9090 &

    Visit http://localhost:9090/graph in your web browser.

  5. Execute a Prometheus query.

    In the “Expression” input box at the top of the web page, enter the text: istio_request_count. Then, click the Execute button.

    The results will be similar to:

Prometheus Query Result

Prometheus Query Result

Other queries to try:

- Total count of all requests to `productpage` service:

  ```
  istio_request_count{destination_service="productpage.default.svc.cluster.local"}
  ```

- Total count of all requests to `v3` of the `reviews` service:

  ```
  istio_request_count{destination_service="reviews.default.svc.cluster.local", destination_version="v3"}
  ```
   
  This query returns the current total count of all requests to the v3 of the reviews service.

- Rate of requests over the past 5 minutes to all `productpage` services:

  ```
  rate(istio_request_count{destination_service=~"productpage.*", response_code="200"}[5m])
  ```

About the Prometheus Add-on

Mixer comes with a built-in Prometheus adapter that exposes an endpoint serving generated metric values. The Prometheus add-on is a Prometheus server that comes pre-configured to scrape Mixer endpoints to collect the exposed metrics. It provides a mechanism for persistent storage and querying of Istio metrics.

The configured Prometheus add-on scrapes three endpoints:

  1. istio-mesh (istio-mixer.istio-system:42422): all Mixer-generated mesh metrics.
  2. mixer (istio-mixer.istio-system:9093): all Mixer-specific metrics. Used to monitor Mixer itself.
  3. envoy (istio-mixer.istio-system:9102): raw stats generated by Envoy (and translated from statsd to prometheus).

For more on querying Prometheus, please read their querying docs.

Cleanup

  • In Kubernetes environments, execute the following command to remove the Prometheus add-on:

    kubectl delete -f install/kubernetes/addons/prometheus.yaml

  • Remove any kubectl port-forward processes that may still be running:

    killall kubectl

  • If you are not planning to explore any follow-on tasks, refer to the Bookinfo cleanup instructions to shutdown the application.

What’s next