Istio Webhook Management [Experimental]
Istio has two webhooks: Galley and the sidecar injector. By default, these webhooks manage their own configurations. From a security perspective, this default behavior is not recommended because a compromised webhook could then conduct privilege escalation attacks.
This task shows how to use the new istioctl x post-install webhook command to securely manage the configurations of the webhooks.
Getting started
Install Istio with DNS certificates configured and
global.operatorManageWebhooks
set totrue
.$ cat <<EOF > ./istio.yaml apiVersion: install.istio.io/v1alpha1 kind: IstioOperator spec: values: global: operatorManageWebhooks: true certificates: - secretName: dns.istio-galley-service-account dnsNames: [istio-galley.istio-system.svc, istio-galley.istio-system] - secretName: dns.istio-sidecar-injector-service-account dnsNames: [istio-sidecar-injector.istio-system.svc, istio-sidecar-injector.istio-system] EOF $ istioctl manifest apply -f ./istio.yaml
Install
jq
for JSON parsing.
Check webhook certificates
To display the DNS names in the webhook certificates of Galley and the sidecar injector, you need to get the secret from Kubernetes, parse it, decode it, and view the text output with the following commands:
$ kubectl get secret dns.istio-galley-service-account -n istio-system -o json | jq -r '.data["cert-chain.pem"]' | base64 --decode | openssl x509 -in - -text -noout
$ kubectl get secret dns.istio-sidecar-injector-service-account -n istio-system -o json | jq -r '.data["cert-chain.pem"]' | base64 --decode | openssl x509 -in - -text -noout
The output from the above commands should include the DNS names of Galley and the sidecar injector, respectively:
X509v3 Subject Alternative Name:
DNS:istio-galley.istio-system.svc, DNS:istio-galley.istio-system
X509v3 Subject Alternative Name:
DNS:istio-sidecar-injector.istio-system.svc, DNS:istio-sidecar-injector.istio-system
Enable webhook configurations
To generate the
MutatingWebhookConfiguration
andValidatingWebhookConfiguration
configuration files, run the following command.$ istioctl manifest generate > istio.yaml
Open the
istio.yaml
configuration file, search forkind: MutatingWebhookConfiguration
and save theMutatingWebhookConfiguration
of the sidecar injector tosidecar-injector-webhook.yaml
. The following is aMutatingWebhookConfiguration
in an exampleistio.yaml
.apiVersion: admissionregistration.k8s.io/v1beta1 kind: MutatingWebhookConfiguration metadata: name: istio-sidecar-injector labels: app: sidecarInjectorWebhook release: istio webhooks: - name: sidecar-injector.istio.io clientConfig: service: name: istio-sidecar-injector namespace: istio-system path: "/inject" caBundle: "" rules: - operations: [ "CREATE" ] apiGroups: [""] apiVersions: ["v1"] resources: ["pods"] failurePolicy: Fail namespaceSelector: matchLabels: istio-injection: enabled
Open the
istio.yaml
configuration file, search forkind: ValidatingWebhookConfiguration
and save theValidatingWebhookConfiguration
of Galley togalley-webhook.yaml
. The following is aValidatingWebhookConfiguration
in an exampleistio.yaml
(only a part of the configuration is shown to save space).apiVersion: admissionregistration.k8s.io/v1beta1 kind: ValidatingWebhookConfiguration metadata: name: istio-galley labels: app: galley release: istio istio: galley webhooks: - name: pilot.validation.istio.io clientConfig: service: name: istio-galley namespace: istio-system path: "/admitpilot" caBundle: "" rules: - operations: - CREATE - UPDATE apiGroups: - config.istio.io ... SKIPPED failurePolicy: Fail sideEffects: None
Verify that there are no existing webhook configurations for Galley and the sidecar injector. The output of the following two commands should not contain any configurations for Galley and the sidecar injector.
$ kubectl get mutatingwebhookconfiguration $ kubectl get validatingwebhookconfiguration
If there are existing webhook configurations (e.g., from a previous Istio deployment) for Galley and the sidecar injector, delete them using the following commands. Before running these commands, replace the webhook configuration names in the commands with the actual webhook configuration names of Galley and the sidecar injector in your cluster.
$ kubectl delete mutatingwebhookconfiguration SIDECAR-INJECTOR-WEBHOOK-CONFIGURATION-NAME $ kubectl delete validatingwebhookconfiguration GALLEY-WEBHOOK-CONFIGURATION-NAME
Use
istioctl
to enable the webhook configurations:$ istioctl experimental post-install webhook enable --webhook-secret dns.istio-galley-service-account \ --namespace istio-system --validation-path galley-webhook.yaml \ --injection-path sidecar-injector-webhook.yaml
To check that the sidecar injector webhook is working, verify that the webhook injects a sidecar container into an example pod with the following commands:
$ kubectl create namespace test-injection $ kubectl label namespaces test-injection istio-injection=enabled $ kubectl run --generator=run-pod/v1 --image=nginx nginx-app --port=80 -n test-injection $ kubectl get pod -n test-injection
The output from the
get pod
command should show the following. The2/2
value means that the webhook injected a sidecar into the example pod:NAME READY STATUS RESTARTS AGE nginx-app 2/2 Running 0 10s
Check that the validation webhook is working:
$ kubectl create namespace test-validation $ kubectl apply -n test-validation -f - <<EOF apiVersion: networking.istio.io/v1alpha3 kind: Gateway metadata: name: invalid-gateway spec: selector: # DO NOT CHANGE THESE LABELS # The ingressgateway is defined in install/kubernetes/helm/istio/values.yaml # with these labels istio: ingressgateway EOF
The output from the gateway creation command should show the following output. The error in the output indicates that the validation webhook checked the gateway’s configuration YAML file:
Error from server: error when creating "invalid-gateway.yaml": admission webhook "pilot.validation.istio.io" denied the request: configuration is invalid: gateway must have at least one server
Show webhook configurations
If you named the sidecar injector’s configuration
istio-sidecar-injector
and named Galley’s configurationistio-galley-istio-system
, use the following command to show the configurations of these two webhooks:$ istioctl experimental post-install webhook status --validation-config=istio-galley-istio-system --injection-config=istio-sidecar-injector
If you named the sidecar injector’s configuration
istio-sidecar-injector
, use the following command to show the configuration of the sidecar injector:$ istioctl experimental post-install webhook status --validation=false --injection-config=istio-sidecar-injector
If you named Galley’s configuration
istio-galley-istio-system
, show Galley’s configuration with the following command:$ istioctl experimental post-install webhook status --injection=false --validation-config=istio-galley-istio-system
Disable webhook configurations
If you named the sidecar injector’s configuration
istio-sidecar-injector
and named Galley’s configurationistio-galley-istio-system
, use the following command to disable the configurations of these two webhooks:$ istioctl experimental post-install webhook disable --validation-config=istio-galley-istio-system --injection-config=istio-sidecar-injector
If you named the sidecar injector’s configuration
istio-sidecar-injector
, disable the webhook with the following command:$ istioctl experimental post-install webhook disable --validation=false --injection-config=istio-sidecar-injector
If you named Galleys’s configuration
istio-galley-istio-system
, disable the webhook with the following command:$ istioctl experimental post-install webhook disable --injection=false --validation-config=istio-galley-istio-system
Cleanup
You can run the following command to delete the resources created in this tutorial.
$ kubectl delete ns test-injection test-validation
$ kubectl delete -f galley-webhook.yaml
$ kubectl delete -f sidecar-injector-webhook.yaml