OpenTelemetry
The Envoy proxies can be configured to export their access logs6 in OpenTelemetry format7.
In this example, the proxies send access logs to an OpenTelemetry collector8, which is configured to print the logs to standard output.
The standard output of the OpenTelemetry collector can then be accessed via the kubectl logs
command.
Before you begin
Setup Istio by following the instructions in the Installation guide9.
Deploy the sleep11 sample app to use as a test source for sending requests. If you have automatic sidecar injection enabled, run the following command to deploy the sample app:
Otherwise, manually inject the sidecar before deploying the
sleep
application with the following command:Set the
SOURCE_POD
environment variable to the name of your source pod:
Start the httpbin13 sample.
If you have enabled automatic sidecar injection, deploy the
httpbin
service:Otherwise, you have to manually inject the sidecar before deploying the
httpbin
application:
Start the otel-collector15 sample.
Enable Envoy’s access logging
To enable access logging, use the Telemetry API17.
Edit MeshConfig
to add an OpenTelemetry provider, named otel
. This involves adding an extension provider stanza:
The final configuration should look something like:
Next, add a Telemetry resource that tells Istio to send access logs to the OpenTelemetry collector.
The above example uses the otel
access log provider, and we do not configure anything other than default settings.
Similar configuration can also be applied on an individual namespace, or to an individual workload, to control logging at a fine grained level.
For more information about using the Telemetry API, see the Telemetry API overview17.
Using Mesh Config
If you used an IstioOperator
CR to install Istio, add the following field to your configuration:
Otherwise, add the equivalent setting to your original istioctl install
command, for example:
Default access log format
Istio will use the following default access log format if accessLogFormat
is not specified:
The following table shows an example using the default access log format for a request sent from sleep
to httpbin
:
Log operator | access log in sleep | access log in httpbin |
---|---|---|
[%START_TIME%] | [2020-11-25T21:26:18.409Z] | [2020-11-25T21:26:18.409Z] |
\"%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%\" | "GET /status/418 HTTP/1.1" | "GET /status/418 HTTP/1.1" |
%RESPONSE_CODE% | 418 | 418 |
%RESPONSE_FLAGS% | - | - |
%RESPONSE_CODE_DETAILS% | via_upstream | via_upstream |
%CONNECTION_TERMINATION_DETAILS% | - | - |
\"%UPSTREAM_TRANSPORT_FAILURE_REASON%\" | "-" | "-" |
%BYTES_RECEIVED% | 0 | 0 |
%BYTES_SENT% | 135 | 135 |
%DURATION% | 4 | 3 |
%RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)% | 4 | 1 |
\"%REQ(X-FORWARDED-FOR)%\" | "-" | "-" |
\"%REQ(USER-AGENT)%\" | "curl/7.73.0-DEV" | "curl/7.73.0-DEV" |
\"%REQ(X-REQUEST-ID)%\" | "84961386-6d84-929d-98bd-c5aee93b5c88" | "84961386-6d84-929d-98bd-c5aee93b5c88" |
\"%REQ(:AUTHORITY)%\" | "httpbin:8000" | "httpbin:8000" |
\"%UPSTREAM_HOST%\" | "10.44.1.27:80" | "127.0.0.1:80" |
%UPSTREAM_CLUSTER% | outbound|8000||httpbin.foo.svc.cluster.local | inbound|8000|| |
%UPSTREAM_LOCAL_ADDRESS% | 10.44.1.23:37652 | 127.0.0.1:41854 |
%DOWNSTREAM_LOCAL_ADDRESS% | 10.0.45.184:8000 | 10.44.1.27:80 |
%DOWNSTREAM_REMOTE_ADDRESS% | 10.44.1.23:46520 | 10.44.1.23:37652 |
%REQUESTED_SERVER_NAME% | - | outbound_.8000_._.httpbin.foo.svc.cluster.local |
%ROUTE_NAME% | default | default |
Test the access log
Send a request from
sleep
tohttpbin
:Check
otel-collector
’s log:
Note that the messages corresponding to the request appear in logs of the Istio proxies of both the source and the destination, sleep
and httpbin
, respectively. You can see in the log the HTTP verb (GET
), the HTTP path (/status/418
), the response code (418
) and other request-related information18.
Cleanup
Shutdown the sleep11 and httpbin13 services:
Disable Envoy’s access logging
Remove, or set to ""
, the meshConfig.extensionProviders
and meshConfig.defaultProviders
setting in your Istio install configuration.