Frequently Asked Questions

Automatic sidecar injection requires the initializer alpha feature. Run the following command to check if the initializer has been enabled (empty output indicates that initializers are not enabled):

kubectl api-versions | grep admissionregistration

In addition, the Kubernetes API server must be started with the Initializer plugin enabled. Failure to enable the Initializer plugin will result in the following error when trying to create the initializer deployment.

The Deployment “istio-initializer” is invalid: metadata.initializers.pending: Invalid value: “null”: must be non-empty when result is not set

Ensure that your cluster has met the prerequisites for the automatic sidecar injection. If your microservice is deployed in kube-system, kube-public or istio-system namespaces, they are exempted from automatic sidecar injection. Please use a different namespace instead.

Upgrading from Istio 0.1.x to 0.2.x is not supported. You must uninstall Istio v0.1, including pods with Istio sidecars and start with a fresh install of Istio v0.2.

Please ensure all required containers are running: etcd, istio-apiserver, consul, registrator, pilot. If one of them is not running, you may find the {containerID} using docker ps -a and then use docker logs {containerID} to read the logs.

Your kubectl is switched to use the istio context at the end of the istio context-create command. You can use kubectl config get-contexts to obtain the list of contexts and kubectl config use-context {desired-context} to switch to use your desired context.

Please ensure all required containers are running: etcd, istio-apiserver, consul, registrator, istio-pilot. If one of them is not running, you may find the {containerID} using docker ps -a and then use docker logs {containerID} to read the logs.

Your kubectl is switched to use the istio context at the end of the istio context-create command. You can use kubectl config get-contexts to obtain the list of contexts and kubectl config use-context {desired-context} to switch to use your desired context.

The most straightforward way to enable/disable mTLS is by entirely uninstalling and re-installing Istio.

If you are an advanced user and understand the risks you can also do the following:

kubectl edit configmap -n istio-system istio

comment out or uncomment out authPolicy: MUTUAL_TLS to toggle mTLS and then

kubectl delete pods -n istio-system -l istio=pilot

to restart Pilot, after a few seconds (depending on your *RefreshDelay) your Envoy proxies will have picked up the change from Pilot. During that time your services may be unavailable.

We are working on a smoother solution.

This is not supported currently, but will be in the near future.

Starting with release 0.3, you can use service-level annotations to disable (or enable) Istio Auth for particular service-port. The annotation key should be auth.istio.io/{port_number}, and the value should be NONE (to disable), or MUTUAL_TLS (to enable).

Example: disable Istio Auth on port 9080 for service details.

kind: Service
metadata:
name: details
labels:
  app: details
annotations:
  auth.istio.io/9080: NONE

If Istio Auth is enabled, http and tcp health check from kubelet will not work since they do not have Istio Auth issued certs. A workaround is to use a liveness command for health check, e.g., one can install curl in the service pod and curl itself within the pod. The Istio team is actively working on a solution.

An example of readinessProbe:

livenessProbe:
exec:
  command:
  - curl
  - -f
  - http://localhost:8080/healthz # Replace port and URI by your actual health check
initialDelaySeconds: 10
periodSeconds: 5

The Kubernetes API server does not support mutual TLS authentication, so strictly speaking: no. However, if you use version 0.3 or later, see next question to learn how to disable mTLS in upstream config on clients side so they can access API server.

Starting with release 0.3, edit the mtlsExcludedServices list in Istio config map to contain the fully-qualified name of the API server (and any other control services for that matter). The default value of mtlsExcludedServices already contains kubernetes.default.svc.cluster.local, which is the default service name of the Kubernetes API server.

For a quick reference, here are commands to edit Istio configmap and to restart pilot.

kubectl edit configmap -n istio-system istio
kubectl delete pods -n istio-system -l istio=pilot

Note: DO NOT use this approach to disable mTLS for services that are managed by Istio (i.e. using Istio sidecar). Instead, use service-level annotations to overwrite the authentication policy (see above).

Not currently - but we are working on it. At the moment, we only support the kubernetes service account as the principal identity in Istio Auth. We are investigating using JWT together with mutual TLS to support enhanced authentication and authorization.

Yes. The key and certificate distribution in Istio Auth is based on Kubernetes secrets.

Secrets have known security risks. The kubernetes team is working on several features to improve Kubernetes secret security, from secret encryption to node-level access control. And as of version 1.6, Kubernetes introduces RBAC authorization, which can provide fine-grained secrets management.

Mixer provides a rich intermediation layer between the Istio components as well as Istio-based services, and the infrastructure backends used to perform access control checks and telemetry capture. This layer enables operators to have rich insights and control over service behavior without requiring changes to service binaries.

Mixer is designed as a stand-alone component, distinct from Envoy. This has numerous benefits:

• Scalability. The work that Mixer and Envoy do is very different in nature, leading to different scalability requirements. Keeping the components separate enables independent component-appropriate scaling.

• Resource Usage. Istio depends on being able to deploy many instances of its proxy, making it important to minimize the cost of each individual instance. Moving Mixer’s complex logic into a distinct component makes it possible for Envoy to remain svelte and agile.

• Reliability. Mixer and its open-ended extensibility model represents the most complex parts of the data path processing pipeline. By hosting this functionality in Mixer rather than Envoy, it creates distinct failure domains which enables Envoy to continue operating even if Mixer fails, preventing outages.

• Isolation. Mixer provides a level of insulation between Istio and the infrastructure backends. Each Envoy instance can be configured to have a very narrow scope of interaction, limiting the impact of potential attacks.

• Extensibility. It was imperative to design a simple extensibility model to allow Istio to interoperate with as widest breath of backends as possible. Due to its design and language choice, Mixer is inherently easier to extend than Envoy is. The separation of concerns also makes it possible to use Istio policy and telemetry processing with different proxies, just as a mix of Envoy and NGINX.

Envoy implements sophisticated caching, batching, and prefetching, to largely mitigate the latency impact of needing to interact with Mixer on the request path.

Configuration for instances, handlers, and rules is stored as Kubernetes Custom Resources. Configuration may be accessed by using kubectl to query the Kubernetes API server for the resources.

Rules

To see the list of all rules, execute the following:

kubectl get rules --all-namespaces

Output will be similar to:

NAMESPACE      NAME        KIND
default        mongoprom   rule.v1alpha2.config.istio.io
istio-system   promhttp    rule.v1alpha2.config.istio.io
istio-system   promtcp     rule.v1alpha2.config.istio.io
istio-system   stdio       rule.v1alpha2.config.istio.io

To see an individual rule configuration, execute the following:

kubectl -n <namespace> get rules <name> -o yaml

Handlers

Handlers are defined based on Kubernetes Custom Resource Definitions for adapters.

First, identify the list of adapter kinds:

kubectl get crd -listio=mixer-adapter

The output will be similar to:

NAME                           KIND
deniers.config.istio.io        CustomResourceDefinition.v1beta1.apiextensions.k8s.io
listcheckers.config.istio.io   CustomResourceDefinition.v1beta1.apiextensions.k8s.io
memquotas.config.istio.io      CustomResourceDefinition.v1beta1.apiextensions.k8s.io
noops.config.istio.io          CustomResourceDefinition.v1beta1.apiextensions.k8s.io
prometheuses.config.istio.io   CustomResourceDefinition.v1beta1.apiextensions.k8s.io
stackdrivers.config.istio.io   CustomResourceDefinition.v1beta1.apiextensions.k8s.io
statsds.config.istio.io        CustomResourceDefinition.v1beta1.apiextensions.k8s.io
stdios.config.istio.io         CustomResourceDefinition.v1beta1.apiextensions.k8s.io
svcctrls.config.istio.io       CustomResourceDefinition.v1beta1.apiextensions.k8s.io

Then, for each adapter kind in that list, issue the following command:

kubectl get <adapter kind name> --all-namespaces

Output for stdios will be similar to:

NAMESPACE      NAME      KIND
istio-system   handler   stdio.v1alpha2.config.istio.io

To see an individual handler configuration, execute the following:

kubectl -n <namespace> get <adapter kind name> <name> -o yaml

Instances

Instances are defined according to Kubernetes Custom Resource Definitions for instances.

First, identify the list of instance kinds:

kubectl get crd -listio=mixer-instance

The output will be similar to:

NAME                             KIND
checknothings.config.istio.io    CustomResourceDefinition.v1beta1.apiextensions.k8s.io
listentries.config.istio.io      CustomResourceDefinition.v1beta1.apiextensions.k8s.io
logentries.config.istio.io       CustomResourceDefinition.v1beta1.apiextensions.k8s.io
metrics.config.istio.io          CustomResourceDefinition.v1beta1.apiextensions.k8s.io
quotas.config.istio.io           CustomResourceDefinition.v1beta1.apiextensions.k8s.io
reportnothings.config.istio.io   CustomResourceDefinition.v1beta1.apiextensions.k8s.io

Then, for each instance kind in that list, issue the following command:

kubectl get <instance kind name> --all-namespaces

Output for metrics will be similar to:

NAMESPACE      NAME                 KIND
default        mongoreceivedbytes   metric.v1alpha2.config.istio.io
default        mongosentbytes       metric.v1alpha2.config.istio.io
istio-system   requestcount         metric.v1alpha2.config.istio.io
istio-system   requestduration      metric.v1alpha2.config.istio.io
istio-system   requestsize          metric.v1alpha2.config.istio.io
istio-system   responsesize         metric.v1alpha2.config.istio.io
istio-system   tcpbytereceived      metric.v1alpha2.config.istio.io
istio-system   tcpbytesent          metric.v1alpha2.config.istio.io

To see an individual instance configuration, execute the following:

kubectl -n <namespace> get <instance kind name> <name> -o yaml

Please see the Expression Language Reference for the full set of supported attribute expressions.

Mixer exposes a monitoring endpoint (default port: 9093). There are a few useful paths to investigate Mixer performance and audit function:

• /metrics provides Prometheus metrics on the Mixer process as well as gRPC metrics related to API calls and metrics on adapter dispatch.
• /debug/pprof provides an endpoint for profiling data in pprof format.
• /debug/vars provides an endpoint exposing server metrics in JSON format.

Mixer logs can be accessed via a kubectl logs command, as follows:

kubectl -n istio-system logs $(kubectl -n istio-system get pods -listio=mixer -o jsonpath='{.items[0].metadata.name}') mixer

Mixer trace generation is controlled by the command-line flag traceOutput. If the flag value is set to STDOUT or STDERR trace data will be written directly to those locations. If a URL is provided, Mixer will post Zipkin-formatted data to that endpoint (example: http://zipkin:9411/api/v1/spans).

In the 0.2 release, Mixer only supports Zipkin tracing.

Learn how to implement a new adapter for Mixer by consulting the Adapter Developer’s Guide.

Rules can be viewed using istioctl get routerules -o yaml or kubectl get routerules -o yaml.

For the current Envoy sidecar implementation, up to 100 requests may be required for the desired distribution to be observed.

This is an known issue with the current Envoy sidecar implementation. After two seconds of creating the rule, services should become available.

Simple ingress specifications, with host, TLS, and exact path based matches will work out of the box without the need for route rules. However, note that the path used in the ingress resource should not have any . characters.

For example, the following ingress resource matches requests for the example.com host, with /helloworld as the URL.

cat <<EOF | kubectl create -f -
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: simple-ingress
annotations:
  kubernetes.io/ingress.class: istio
spec:
rules:
- host: example.com
  http:
    paths:
    - path: /helloworld
      backend:
        serviceName: myservice
        servicePort: grpc
EOF

However, the following rules will not work because it uses regular expressions in the path and uses ingress.kubernetes.io annotations:

cat <<EOF | kubectl create -f -
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: this-will-not-work
annotations:
  kubernetes.io/ingress.class: istio
  # Ingress annotations other than ingress class will not be honored
  ingress.kubernetes.io/rewrite-target: /
spec:
rules:
- host: example.com
  http:
    paths:
    - path: /hello(.*?)world/
      backend:
        serviceName: myservice
        servicePort: grpc
EOF