Sidecar Injection Problems
The result of sidecar injection was not what I expected
This includes an injected sidecar when it wasn’t expected and a lack of injected sidecar when it was.
Ensure your pod is not in the
kube-system
orkube-public
namespace. Automatic sidecar injection will be ignored for pods in these namespaces.Ensure your pod does not have
hostNetwork: true
in its pod spec. Automatic sidecar injection will be ignored for pods that are on the host network.The sidecar model assumes that the iptables changes required for Envoy to intercept traffic are within the pod. For pods on the host network this assumption is violated, and this can lead to routing failures at the host level.
Check the webhook’s
namespaceSelector
to determine whether the webhook is scoped to opt-in or opt-out for the target namespace.The
namespaceSelector
for opt-in will look like the following:The injection webhook will be invoked for pods created in namespaces with the
istio-injection=enabled
label.The
namespaceSelector
for opt-out will look like the following:The injection webhook will be invoked for pods created in namespaces without the
istio-injection=disabled
label.Verify the application pod’s namespace is labeled properly and (re) label accordingly, e.g.
(repeat for all namespaces in which the injection webhook should be invoked for new pods)
Check default policy
Check the default injection policy in the
istio-sidecar-injector configmap
.Allowed policy values are
disabled
andenabled
. The default policy only applies if the webhook’snamespaceSelector
matches the target namespace. Unrecognized policy causes injection to be disabled completely.Check the per-pod override annotation
The default policy can be overridden with the
sidecar.istio.io/inject
label in the pod template spec’s metadata. The deployment’s metadata is ignored. Label value oftrue
forces the sidecar to be injected while a value offalse
forces the sidecar to not be injected.The following label overrides whatever the default
policy
was to force the sidecar to be injected:
Pods cannot be created at all
Run kubectl describe -n namespace deployment name
on the failing
pod’s deployment. Failure to invoke the injection webhook will
typically be captured in the event log.
x509 certificate related errors
x509: certificate signed by unknown authority
errors are typically
caused by an empty caBundle
in the webhook configuration.
Verify the caBundle
in the mutatingwebhookconfiguration
matches the
root certificate mounted in the istiod
pod.
The CA certificate should match. If they do not, restart the istiod pods.
Errors in deployment status
When automatic sidecar injection is enabled for a pod, and the injection fails for any reason, the pod creation will also fail. In such cases, you can check the deployment status of the pod to identify the error. The errors will also appear in the events of the namespace associated with the deployment.
For example, if the istiod
control plane pod was not running when you tried to deploy your pod, the events would show the following error:
If the istiod pod or endpoints aren’t ready, check the pod logs and status for any indication about why the webhook pod is failing to start and serve traffic.
Automatic sidecar injection fails if the Kubernetes API server has proxy settings
When the Kubernetes API server includes proxy settings such as:
With these settings, Sidecar injection fails. The only related failure log can be found in kube-apiserver
log:
Make sure both pod and service CIDRs are not proxied according to *_proxy
variables. Check the kube-apiserver
files and logs to verify the configuration and whether any requests are being proxied.
One workaround is to remove the proxy settings from the kube-apiserver
manifest, another workaround is to include istio-sidecar-injector.istio-system.svc
or .svc
in the no_proxy
value. Make sure that kube-apiserver
is restarted after each workaround.
An issue4 was filed with Kubernetes related to this and has since been closed. https://github.com/kubernetes/kubernetes/pull/58698#discussion_r1638794435
Limitations for using Tcpdump in pods
Tcpdump doesn’t work in the sidecar pod - the container doesn’t run as root. However any other container in the same pod will see all the packets, since the
network namespace is shared. iptables
will also see the pod-wide configuration.
Communication between Envoy and the app happens on 127.0.0.1, and is not encrypted.
Cluster is not scaled down automatically
Due to the fact that the sidecar container mounts a local storage volume, the
node autoscaler is unable to evict nodes with the injected pods. This is
a known issue6. The workaround is
to add a pod annotation "cluster-autoscaler.kubernetes.io/safe-to-evict": "true"
to the injected pods.
Pod or containers start with network issues if istio-proxy is not ready
Many applications execute commands or checks during startup, which require network connectivity. This can cause application containers to hang or restart if the istio-proxy
sidecar container is not ready.
To avoid this, set holdApplicationUntilProxyStarts
to true
. This causes the sidecar injector to inject the sidecar at the start of the pod’s container list, and configures it to block the start of all other containers until the proxy is ready.
This can be added as a global config option:
or as a pod annotation: