days to Istio 1.5

Metrics and Logs FAQ

Can Istio metrics be accessed through REST?

You can collect telemetry data about Istio using Prometheus. And then use Prometheus’s HTTP API to query that data.

How can I control the data being reported by the sidecar?

It is sometimes useful to exclude access to specific URLs from being reported. For example, you might wish to exclude some health-checking URLs. You can skip telemetry reports specific to certain URLs by excluding them using a match clause of the telemetry configuration such as:

match: source.name != "health"
Can the Prometheus adapter be used in non-Kubernetes environments?

You can use docker-compose to install Prometheus. Also, without the Kubernetes API server, components such as Mixer will need local configuration for rules/handlers/instances.

How to figure out what happened to a request in Istio?

You can enable tracing to determine the flow of a request in Istio.

Additionally, you can use the following commands to know more about the state of the mesh:

  • istioctl proxy-config: Retrieve information about proxy configuration when running in Kubernetes:

    # Retrieve information about bootstrap configuration for the Envoy instance in the specified pod.
    $ istioctl proxy-config bootstrap productpage-v1-bb8d5cbc7-k7qbm
    
    # Retrieve information about cluster configuration for the Envoy instance in the specified pod.
    $ istioctl proxy-config cluster productpage-v1-bb8d5cbc7-k7qbm
    
    # Retrieve information about listener configuration for the Envoy instance in the specified pod.
    $ istioctl proxy-config listener productpage-v1-bb8d5cbc7-k7qbm
    
    # Retrieve information about route configuration for the Envoy instance in the specified pod.
    $ istioctl proxy-config route productpage-v1-bb8d5cbc7-k7qbm
    
    # Retrieve information about endpoint configuration for the Envoy instance in the specified pod.
    $ istioctl proxy-config endpoints productpage-v1-bb8d5cbc7-k7qbm
    
    # Try the following to discover more proxy-config commands
    $ istioctl proxy-config --help
    
  • kubectl get: Gets information about different resources in the mesh along with routing configuration:

    # List all virtual services
    $ kubectl get virtualservices
    
  • Mixer access logs: Mixer writes access logs that contain information about requests. You can get them with:

    # Fill <istio namespace> with the namespace of your istio mesh. Ex: istio-system
    $ TELEMETRY_POD=`kubectl get po -n <istio namespace> | grep istio-telemetry | awk '{print $1;}'`
    $ kubectl logs $TELEMETRY_POD -c mixer  -n istio-system  | grep accesslog
    
Can I use Prometheus to scrape application metrics with Istio?

Yes. Istio ships with configuration for Prometheus that enables collection of application metrics when mutual TLS is enabled or disabled.

The kubernetes-pods job collects application metrics from pods in environments without mutual TLS. The kubernetes-pods-istio-secure job collects metrics from application pods when mutual TLS is enabled for Istio.

Both jobs require that the following annotations are added to any deployments from which application metric collection is desired:

  • prometheus.io/scrape: "true"
  • prometheus.io/path: "<metrics path>"
  • prometheus.io/port: "<metrics port>"

A few notes:

  • If the Prometheus pod started before the Istio Citadel pod could generate the required certificates and distribute them to Prometheus, the Prometheus pod will need to be restarted in order to collect from mutual TLS-protected targets.
  • If your application exposes Prometheus metrics on a dedicated port, that port should be added to the service and deployment specifications.