Telemetry

Missing Metrics

The procedures below help you diagnose problems where metrics you are expecting to see reported and not being collected.

The expected flow for metrics is:

  1. Envoy reports attributes from requests asynchronously to Mixer in a batch.

  2. Mixer translates the attributes into instances based on the operator-provided configuration.

  3. Mixer hands the instances to Mixer adapters for processing and backend storage.

  4. The backend storage systems record the metrics data.

The Mixer default installations include a Prometheus adapter and the configuration to generate a default set of metric values and send them to the Prometheus adapter. The Prometheus adapter configuration enables a Prometheus instance to scrape Mixer for metrics.

If the Istio Dashboard or the Prometheus queries don’t show the expected metrics, any step of the flow above may present an issue. The following sections provide instructions to troubleshoot each step.

Verify Mixer is receiving Report calls

Mixer generates metrics to monitor its own behavior. The first step is to check these metrics:

  1. Establish a connection to the Mixer self-monitoring endpoint for the istio-telemetry deployment. In Kubernetes environments, execute the following command:

    $ kubectl -n istio-system port-forward <istio-telemetry pod> 9093 &

  2. Verify successful report calls. On the Mixer self-monitoring endpoint, search for grpc_server_handled_total. You should see something like:

    grpc_server_handled_total{grpc_code="OK",grpc_method="Report",grpc_service="istio.mixer.v1.Mixer",grpc_type="unary"} 68

    If you do not see any data for grpc_server_handled_total with a grpc_method="Report", then Envoy is not calling Mixer to report telemetry.

  3. In this case, ensure you integrated the services properly into the mesh. You can achieve this task with either automatic or manual sidecar injection.

Verify the Mixer rules exist

In Kubernetes environments, issue the following command:

$ kubectl get rules --all-namespaces
NAMESPACE      NAME        AGE
istio-system   kubeattrgenrulerule      13d
istio-system   promhttp                 13d
istio-system   promtcp                  13d
istio-system   stdio                    13d
istio-system   tcpkubeattrgenrulerule   13d

If the output shows no rules named promhttp or promtcp, then the Mixer configuration for sending metric instances to the Prometheus adapter is missing. You must supply the configuration for rules connecting the Mixer metric instances to a Prometheus handler.

For reference, please consult the default rules for Prometheus.

Verify the Prometheus handler configuration exists

  1. In Kubernetes environments, issue the following command:

    $ kubectl get prometheuses.config.istio.io --all-namespaces
NAMESPACE      NAME      AGE
istio-system   handler   13d

  2. If the output shows no configured Prometheus handlers, you must reconfigure Mixer with the appropriate handler configuration.

    For reference, please consult the default handler configuration for Prometheus.

Verify Mixer metric instances configuration exists

  1. In Kubernetes environments, issue the following command:

    $ kubectl get metrics.config.istio.io --all-namespaces
NAMESPACE      NAME              AGE
istio-system   requestcount      13d
istio-system   requestduration   13d
istio-system   requestsize       13d
istio-system   responsesize      13d
istio-system   tcpbytereceived   13d
istio-system   tcpbytesent       13d

  2. If the output shows no configured metric instances, you must reconfigure Mixer with the appropriate instance configuration.

    For reference, please consult the default instances configuration for metrics.

Verify there are no known configuration errors

  1. To establish a connection to the Istio-telemetry self-monitoring endpoint, setup a port-forward to the Istio-telemetry self-monitoring port as described in Verify Mixer is receiving Report calls.

  2. For each of the following metrics, verify that the most up-to-date value is 0:

    • mixer_config_adapter_info_config_error_count

    • mixer_config_handler_validation_error_count

    • mixer_config_instance_config_error_count

    • mixer_config_rule_config_error_count

    • mixer_config_rule_config_match_error_count

    • mixer_config_unsatisfied_action_handler_count

    • mixer_handler_handler_build_failure_count

On the page showing Mixer self-monitoring port, search for each of the metrics listed above. The matching text should look something like the following (using mixer_config_instance_config_error_count):

mixer_config_rule_config_match_error_count{configID="-1"} 0
mixer_config_rule_config_match_error_count{configID="0"} 0
mixer_config_rule_config_match_error_count{configID="1"} 0</td>

Confirm that the metric value with the largest configuration ID is 0. This will verify that Mixer has generated no errors in processing the configuration as supplied.

Verify Mixer is sending metric instances to the Prometheus adapter

  1. Establish a connection to the istio-telemetry self-monitoring endpoint. Setup a port-forward to the istio-telemetry self-monitoring port as described in Verify Mixer is receiving Report calls.

  2. On the Mixer self-monitoring port, search for mixer_runtime_dispatch_count. The output should be similar to:

    mixer_runtime_dispatch_count{adapter="prometheus",error="false",handler="handler.prometheus.istio-system",meshFunction="metric"} 916
mixer_runtime_dispatch_count{adapter="prometheus",error="true",handler="handler.prometheus.istio-system",meshFunction="metric"} 0

  3. Confirm that mixer_runtime_dispatch_count is present with the values:

    adapter="prometheus"
error="false"

    If you can’t find recorded dispatches to the Prometheus adapter, there is likely a configuration issue. Please follow the steps above to ensure everything is configured properly.

    If the dispatches to the Prometheus adapter report errors, check the Mixer logs to determine the source of the error. The most likely cause is a configuration issue for the handler listed in mixer_runtime_dispatch_count.

  4. Check the Mixer logs in a Kubernetes environment with:

    $ kubectl -n istio-system logs <istio-telemetry pod> -c mixer

Verify Prometheus configuration

  1. Connect to the Prometheus UI

  2. Verify you can successfully scrape Mixer through the UI.

  3. In Kubernetes environments, setup port-forwarding with:

    $ kubectl -n istio-system port-forward $(kubectl -n istio-system get pod -l app=prometheus -o jsonpath='{.items[0].metadata.name}') 9090:9090 &

  4. Visit http://localhost:9090/targets

  5. Confirm the target istio-mesh has a status of UP.

  6. Visit http://localhost:9090/config

  7. Confirm an entry exists similar to:

    - job_name: 'istio-mesh'
# Override the global default and scrape targets from this job every 5 seconds.
scrape_interval: 5s
# metrics_path defaults to '/metrics'
# scheme defaults to 'http'.
static_configs:
- targets: ['istio-mixer.istio-system:42422']</td>

Grafana