Lightstep
This task shows you how to configure Istio to collect trace spans and send them to Lightstep. Lightstep lets you analyze 100% of unsampled transaction data from large scale production software to produce meaningful distributed traces and metrics that help explain performance behaviors and accelerate root cause analysis. At the end of this task, Istio sends trace spans from the proxies to a Lightstep Satellite pool making them available to the web UI. By default, all HTTP requests are captured (to see end-to-end traces, your code needs to forward OT headers even if it does not join the traces).
If you only want to collect tracing spans directly from Istio (and not add specific instrumentation directly to your code), then you don’t need to configure any tracers, as long as your services forward the HTTP headers generated by traces.
This task uses the Bookinfo sample application as an example.
Before you begin
Ensure you have a Lightstep account. Sign up for a free trial of Lightstep.
If you’re using on-premise Satellites, ensure you have a satellite pool configured with TLS certs and a secure GRPC port exposed. See Install and Configure Satellites for details about setting up satellites.
For Lightstep Public Satellites or Developer Satellites, your satellites are already configured. However you need to download this certificate to a local directory.
Ensure sure you have a Lightstep access token. Access tokens allow your app to communicate with your Lightstep project.
Deploy Istio
How you deploy Istio depends on which type of Satellite you use.
Deploy Istio with On-Premise Satellites
These instructions do not assume TLS. If you are using TLS for your Satellite pool, follow the config for the Public Satellite pool, but
use your own cert and your own pool’s endpoint (host:port
).
You need to deploy Istio with your Satellite address at an address in the format
<Host>:<Port>
, for examplelightstep-satellite.lightstep:9292
. You find this in your configuration file.Deploy Istio with the following configuration parameters specified:
global.proxy.tracer="lightstep"
meshConfig.defaultConfig.tracing.sampling=100
meshConfig.defaultConfig.tracing.lightstep.address="<satellite-address>"
meshConfig.defaultConfig.tracing.lightstep.accessToken="<access-token>"
You can set these parameters using the
--set key=value
syntax when you run the install command. For example:$ istioctl install \ --set global.proxy.tracer="lightstep" \ --set meshConfig.defaultConfig.tracing.sampling=100 \ --set meshConfig.defaultConfig.tracing.lightstep.address="<satellite-address>" \ --set meshConfig.defaultConfig.tracing.lightstep.accessToken="<access-token>" \
Deploy Istio with Public or Developer Mode Satellites
Follow these steps if you’re using the Public or Developer Mode Satellites, or if you’re using on-premise Satellites with a TLS certificate.
Store your satellite pool’s certificate authority certificate as a secret in the default and
istio-system
namespace, the latter for use by the Istio gateways. Download and use this certificate. If you deploy the Bookinfo application in a different namespace, create the secret in that namespace instead.$ CACERT=$(cat Cert_Auth.crt | base64) # Cert_Auth.crt contains the necessary CACert $ NAMESPACE=default
$ cat <<EOF | kubectl apply -f - apiVersion: v1 kind: Secret metadata: name: lightstep.cacert namespace: $NAMESPACE labels: app: lightstep type: Opaque data: cacert.pem: $CACERT EOF
Deploy Istio with the following configuration parameters specified:
global: proxy: tracer: "lightstep" meshConfig: defaultConfig: tracing: lightstep: address: "ingest.lightstep.com:443" accessToken: "<access-token>" sampling: 100 tlsSettings mode: "SIMPLE" # Specifying ca certificate here will mount `lightstep.cacert` secret volume # at all sidecars by default. caCertificates="/etc/lightstep/cacert.pem" components: ingressGateways: # `lightstep.cacert` secret volume needs to be mount at gateways via k8s overlay. - name: istio-ingressgateway enabled: true k8s: overlays: - kind: Deployment name: istio-ingressgateway patches: - path: spec.template.spec.containers[0].volumeMounts[-1] value: | name: lightstep-certs mountPath: /etc/lightstep readOnly: true - path: spec.template.spec.volumes[-1] value: | name: lightstep-certs secret: secretName: lightstep.cacert optional: true
Install and run the Bookinfo app
Follow the instructions to deploy the Bookinfo sample application.
Follow the instructions to create an ingress gateway for the Bookinfo application.
To verify the previous step’s success, confirm that you set
GATEWAY_URL
environment variable in your shell.Send traffic to the sample application.
$ curl http://$GATEWAY_URL/productpage
Visualize trace data
Load the Lightstep web UI. You’ll see the three Bookinfo services listed in the Service Directory.
Navigate to the Explorer view.
Find the query bar at the top. The query bar allows you to interactively filter results by a Service, Operation, and Tag values.
Select
productpage.default
from the Service drop-down list.Click Run. You see something similar to the following:
Click on the first row in the table of example traces below the latency histogram to see the details corresponding to your refresh of the
/productpage
. The page then looks similar to:
The screenshot shows that the trace is comprised of a set of spans. Each span corresponds to a Bookinfo service invoked
during the execution of a /productpage
request.
Two spans in the trace represent every RPC. For example, the call from productpage
to reviews
starts
with the span labeled with the reviews.default.svc.cluster.local:9080/*
operation and the
productpage.default: proxy client
service. This service represents the client-side span of the call. The screenshot shows
that the call took 15.30 ms. The second span is labeled with the reviews.default.svc.cluster.local:9080/*
operation
and the reviews.default: proxy server
service. The second span is a child of the first span and represents the
server-side span of the call. The screenshot shows that the call took 14.60 ms.
Trace sampling
Istio captures traces at a configurable trace sampling percentage. To learn how to modify the trace sampling percentage, visit the Distributed Tracing trace sampling section.
When using Lightstep, we do not recommend reducing the trace sampling percentage below 100%. To handle a high traffic mesh, consider scaling up the size of your satellite pool.
Cleanup
If you are not planning any follow-up tasks, remove the Bookinfo sample application and any Lightstep secrets from your cluster.
To remove the Bookinfo application, refer to the Bookinfo cleanup instructions.
Remove the secret generated for Lightstep:
$ kubectl delete secret lightstep.cacert