Remotely Accessing Telemetry Addons

 7 minute read

This task shows how to configure Istio to expose and access the telemetry addons outside of a cluster.

Configuring remote access

Remote access to the telemetry addons can be configured in a number of different ways. This task covers two basic access methods: secure (via HTTPS) and insecure (via HTTP). The secure method is strongly recommended for any production or sensitive environment. Insecure access is simpler to set up, but will not protect any credentials or data transmitted outside of your cluster.

Option 1: Secure access (HTTPS)

A server certificate is required for secure access. Follow these steps to install and configure server certificates for a domain that you control.

You may use self-signed certificates instead. Visit our Securing Gateways with HTTPS Using Secret Discovery Service task for general information on using self-signed certificates to access in-cluster services.

  1. Install Istio in your cluster and enable the cert-manager flag and configure istio-ingressgateway to use the Secret Discovery Service.

    To install Istio accordingly, use the following Helm installation options:

    • --set gateways.enabled=true
    • --set gateways.istio-ingressgateway.enabled=true
    • --set gateways.istio-ingressgateway.sds.enabled=true
    • --set certmanager.enabled=true
    • --set certmanager.email=mailbox@donotuseexample.com

    To additionally install the telemetry addons, use the following Helm installation options:

    • Grafana: --set grafana.enabled=true
    • Kiali: --set kiali.enabled=true
    • Prometheus: --set prometheus.enabled=true
    • Tracing: --set tracing.enabled=true

  2. Configure the DNS records for your domain.

    1. Get the external IP address of the istio-ingressgateway.

      $ kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}'

      <IP ADDRESS OF CLUSTER INGRESS>

    2. Set an environment variable to hold your target domain.

      $ TELEMETRY_DOMAIN=<your.desired.domain>

    3. Point your desired domain at that external IP address via your domain provider.

      The mechanism for achieving this step varies by provider. Here are a few example documentation links:

    4. Verify that the DNS records are correct.

      $ dig +short $TELEMETRY_DOMAIN

      <IP ADDRESS OF CLUSTER INGRESS>

  3. Generate a server certificate

    $ cat <<EOF | kubectl apply -f -
apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
  name: telemetry-gw-cert
  namespace: istio-system
spec:
  secretName: telemetry-gw-cert
  issuerRef:
    name: letsencrypt
    kind: ClusterIssuer
  commonName: $TELEMETRY_DOMAIN
  dnsNames:
  - $TELEMETRY_DOMAIN
  acme:
    config:
    - http01:
        ingressClass: istio
      domains:
      - $TELEMETRY_DOMAIN
---
EOF

    certificate.certmanager.k8s.io "telemetry-gw-cert" created

  4. Wait until the server certificate is ready.

    $ JSONPATH='{range .items[*]}{@.metadata.name}:{range @.status.conditions[*]}{@.type}={@.status}{end}{end}' && kubectl -n istio-system get certificates -o jsonpath="$JSONPATH"

    telemetry-gw-cert:Ready=True

  5. Apply networking configuration for the telemetry addons.

    1. Apply the following configuration to expose Grafana:

      $ cat <<EOF | kubectl apply -f -
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: grafana-gateway
  namespace: istio-system
spec:
  selector:
    istio: ingressgateway
  servers:
  - port:
      number: 15031
      name: https-grafana
      protocol: HTTPS
    tls:
      mode: SIMPLE
      serverCertificate: sds
      privateKey: sds
      credentialName: telemetry-gw-cert
    hosts:
    - "$TELEMETRY_DOMAIN"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: grafana-vs
  namespace: istio-system
spec:
  hosts:
  - "$TELEMETRY_DOMAIN"
  gateways:
  - grafana-gateway
  http:
  - match:
    - port: 15031
    route:
    - destination:
        host: grafana
        port:
          number: 3000
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: grafana
  namespace: istio-system
spec:
  host: grafana
  trafficPolicy:
    tls:
      mode: DISABLE
---
EOF

      gateway.networking.istio.io "grafana-gateway" configured
virtualservice.networking.istio.io "grafana-vs" configured
destinationrule.networking.istio.io "grafana" configured

    2. Apply the following configuration to expose Kiali:

      $ cat <<EOF | kubectl apply -f -
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: kiali-gateway
  namespace: istio-system
spec:
  selector:
    istio: ingressgateway
  servers:
  - port:
      number: 15029
      name: https-kiali
      protocol: HTTPS
    tls:
      mode: SIMPLE
      serverCertificate: sds
      privateKey: sds
      credentialName: telemetry-gw-cert
    hosts:
    - "$TELEMETRY_DOMAIN"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: kiali-vs
  namespace: istio-system
spec:
  hosts:
  - "$TELEMETRY_DOMAIN"
  gateways:
  - kiali-gateway
  http:
  - match:
    - port: 15029
    route:
    - destination:
        host: kiali
        port:
          number: 20001
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: kiali
  namespace: istio-system
spec:
  host: kiali
  trafficPolicy:
    tls:
      mode: DISABLE
---
EOF

      gateway.networking.istio.io "kiali-gateway" configured
virtualservice.networking.istio.io "kiali-vs" configured
destinationrule.networking.istio.io "kiali" configured

    3. Apply the following configuration to expose Prometheus:

      $ cat <<EOF | kubectl apply -f -
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: prometheus-gateway
  namespace: istio-system
spec:
  selector:
    istio: ingressgateway
  servers:
  - port:
      number: 15030
      name: https-prom
      protocol: HTTPS
    tls:
      mode: SIMPLE
      serverCertificate: sds
      privateKey: sds
      credentialName: telemetry-gw-cert
    hosts:
    - "$TELEMETRY_DOMAIN"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: prometheus-vs
  namespace: istio-system
spec:
  hosts:
  - "$TELEMETRY_DOMAIN"
  gateways:
  - prometheus-gateway
  http:
  - match:
    - port: 15030
    route:
    - destination:
        host: prometheus
        port:
          number: 9090
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: prometheus
  namespace: istio-system
spec:
  host: prometheus
  trafficPolicy:
    tls:
      mode: DISABLE
---
EOF

      gateway.networking.istio.io "prometheus-gateway" configured
virtualservice.networking.istio.io "prometheus-vs" configured
destinationrule.networking.istio.io "prometheus" configured

    4. Apply the following configuration to expose the tracing service:

      $ cat <<EOF | kubectl apply -f -
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: tracing-gateway
  namespace: istio-system
spec:
  selector:
    istio: ingressgateway
  servers:
  - port:
      number: 15032
      name: https-tracing
      protocol: HTTPS
    tls:
      mode: SIMPLE
      serverCertificate: sds
      privateKey: sds
      credentialName: telemetry-gw-cert
    hosts:
    - "$TELEMETRY_DOMAIN"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: tracing-vs
  namespace: istio-system
spec:
  hosts:
  - "$TELEMETRY_DOMAIN"
  gateways:
  - tracing-gateway
  http:
  - match:
    - port: 15032
    route:
    - destination:
        host: tracing
        port:
          number: 80
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: tracing
  namespace: istio-system
spec:
  host: tracing
  trafficPolicy:
    tls:
      mode: DISABLE
---
EOF

      gateway.networking.istio.io "tracing-gateway" configured
virtualservice.networking.istio.io "tracing-vs" configured
destinationrule.networking.istio.io "tracing" configured

  6. Visit the telemetry addons via your browser.

    • Kiali: https://$TELEMETRY_DOMAIN:15029/
    • Prometheus: https://$TELEMETRY_DOMAIN:15030/
    • Grafana: https://$TELEMETRY_DOMAIN:15031/
    • Tracing: https://$TELEMETRY_DOMAIN:15032/

Option 2: Insecure access (HTTP)

  1. Install Istio in your cluster with your desired telemetry addons.

    To additionally install the telemetry addons, use the following Helm installation options:

    • Grafana: --set grafana.enabled=true
    • Kiali: --set kiali.enabled=true
    • Prometheus: --set prometheus.enabled=true
    • Tracing: --set tracing.enabled=true

  2. Apply networking configuration for the telemetry addons.

    1. Apply the following configuration to expose Grafana:

      $ cat <<EOF | kubectl apply -f -
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: grafana-gateway
  namespace: istio-system
spec:
  selector:
    istio: ingressgateway
  servers:
  - port:
      number: 15031
      name: http-grafana
      protocol: HTTP
    hosts:
    - "*"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: grafana-vs
  namespace: istio-system
spec:
  hosts:
  - "*"
  gateways:
  - grafana-gateway
  http:
  - match:
    - port: 15031
    route:
    - destination:
        host: grafana
        port:
          number: 3000
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: grafana
  namespace: istio-system
spec:
  host: grafana
  trafficPolicy:
    tls:
      mode: DISABLE
---
EOF

      gateway.networking.istio.io "grafana-gateway" configured
virtualservice.networking.istio.io "grafana-vs" configured
destinationrule.networking.istio.io "grafana" configured

    2. Apply the following configuration to expose Kiali:

      $ cat <<EOF | kubectl apply -f -
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: kiali-gateway
  namespace: istio-system
spec:
  selector:
    istio: ingressgateway
  servers:
  - port:
      number: 15029
      name: http-kiali
      protocol: HTTP
    hosts:
    - "*"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: kiali-vs
  namespace: istio-system
spec:
  hosts:
  - "*"
  gateways:
  - kiali-gateway
  http:
  - match:
    - port: 15029
    route:
    - destination:
        host: kiali
        port:
          number: 20001
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: kiali
  namespace: istio-system
spec:
  host: kiali
  trafficPolicy:
    tls:
      mode: DISABLE
---
EOF

      gateway.networking.istio.io "kiali-gateway" configured
virtualservice.networking.istio.io "kiali-vs" configured
destinationrule.networking.istio.io "kiali" configured

    3. Apply the following configuration to expose Prometheus:

      $ cat <<EOF | kubectl apply -f -
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: prometheus-gateway
  namespace: istio-system
spec:
  selector:
    istio: ingressgateway
  servers:
  - port:
      number: 15030
      name: http-prom
      protocol: HTTP
    hosts:
    - "*"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: prometheus-vs
  namespace: istio-system
spec:
  hosts:
  - "*"
  gateways:
  - prometheus-gateway
  http:
  - match:
    - port: 15030
    route:
    - destination:
        host: prometheus
        port:
          number: 9090
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: prometheus
  namespace: istio-system
spec:
  host: prometheus
  trafficPolicy:
    tls:
      mode: DISABLE
---
EOF

      gateway.networking.istio.io "prometheus-gateway" configured
virtualservice.networking.istio.io "prometheus-vs" configured
destinationrule.networking.istio.io "prometheus" configured

    4. Apply the following configuration to expose the tracing service:

      $ cat <<EOF | kubectl apply -f -
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: tracing-gateway
  namespace: istio-system
spec:
  selector:
    istio: ingressgateway
  servers:
  - port:
      number: 15032
      name: http-tracing
      protocol: HTTP
    hosts:
    - "*"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: tracing-vs
  namespace: istio-system
spec:
  hosts:
  - "*"
  gateways:
  - tracing-gateway
  http:
  - match:
    - port: 15032
    route:
    - destination:
        host: tracing
        port:
          number: 80
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: tracing
  namespace: istio-system
spec:
  host: tracing
  trafficPolicy:
    tls:
      mode: DISABLE
---
EOF

      gateway.networking.istio.io "tracing-gateway" configured
virtualservice.networking.istio.io "tracing-vs" configured
destinationrule.networking.istio.io "tracing" configured

  3. Visit the telemetry addons via your browser.

    • Kiali: http://<IP ADDRESS OF CLUSTER INGRESS>:15029/
    • Prometheus: http://<IP ADDRESS OF CLUSTER INGRESS>:15030/
    • Grafana: http://<IP ADDRESS OF CLUSTER INGRESS>:15031/
    • Tracing: http://<IP ADDRESS OF CLUSTER INGRESS>:15032/

Cleanup

  • Remove all related Gateways:

    $ kubectl -n istio-system delete gateway grafana-gateway kiali-gateway prometheus-gateway tracing-gateway

    gateway.networking.istio.io "grafana-gateway" deleted
gateway.networking.istio.io "kiali-gateway" deleted
gateway.networking.istio.io "prometheus-gateway" deleted
gateway.networking.istio.io "tracing-gateway" deleted

  • Remove all related Virtual Services:

    $ kubectl -n istio-system delete virtualservice grafana-vs kiali-vs prometheus-vs tracing-vs

    virtualservice.networking.istio.io "grafana-vs" deleted
virtualservice.networking.istio.io "kiali-vs" deleted
virtualservice.networking.istio.io "prometheus-vs" deleted
virtualservice.networking.istio.io "tracing-vs" deleted

  • If installed, remove the gateway certificate:

    $ kubectl -n istio-system delete certificate telemetry-gw-cert

    certificate.certmanager.k8s.io "telemetry-gw-cert" deleted