Troubleshoot issues with waypoints
This guide describes what to do if you have enrolled a namespace, service or workload in a waypoint proxy, but you are not seeing the expected behavior.
Problems with traffic routing or security policy
To send some requests to the reviews
service via the productpage
service from the sleep
pod:
$ kubectl exec deploy/sleep -- curl -s http://productpage:9080/productpage
To send some requests to the reviews
v2
pod from the sleep
pod:
$ export REVIEWS_V2_POD_IP=$(kubectl get pod -l version=v2,app=reviews -o jsonpath='{.items[0].status.podIP}')
$ kubectl exec deploy/sleep -- curl -s http://$REVIEWS_V2_POD_IP:9080/reviews/1
Requests to the reviews
service should be enforced by the reviews-svc-waypoint
for any L7 policies.
Requests to the reviews
v2
pod should be enforced by the reviews-v2-pod-waypoint
for any L7 policies.
If your L7 configuration isn’t applied, run
istioctl analyze
first to check if your configuration has a validation issue.$ istioctl analyze ✔ No validation issues found when analyzing namespace: default.
Determine which waypoint is implementing the L7 configuration for your service or pod.
If your source calls the destination using the service’s hostname or IP, use the
istioctl experimental ztunnel-config service
command to confirm your waypoint is used by the destination service. Following the example earlier, thereviews
service should use thereviews-svc-waypoint
while all other services in thedefault
namespace should use the namespacewaypoint
.$ istioctl experimental ztunnel-config service NAMESPACE SERVICE NAME SERVICE VIP WAYPOINT default bookinfo-gateway-istio 10.43.164.194 waypoint default bookinfo-gateway-istio 10.43.164.194 waypoint default bookinfo-gateway-istio 10.43.164.194 waypoint default bookinfo-gateway-istio 10.43.164.194 waypoint default details 10.43.160.119 waypoint default kubernetes 10.43.0.1 waypoint default notsleep 10.43.156.147 waypoint default productpage 10.43.172.254 waypoint default ratings 10.43.71.236 waypoint default reviews 10.43.162.105 reviews-svc-waypoint ...
If your source calls the destination using a pod IP, use the
istioctl experimental ztunnel-config workload
command to confirm your waypoint is used by the destination pod. Following the example earlier, thereviews
v2
pod should use thereviews-v2-pod-waypoint
while all other pods in thedefault
namespace should not have any waypoints, because by default a waypoint only handles traffic addressed to services.$ istioctl experimental ztunnel-config workload NAMESPACE POD NAME IP NODE WAYPOINT PROTOCOL default bookinfo-gateway-istio-7c57fc4647-wjqvm 10.42.2.8 k3d-k3s-default-server-0 None TCP default details-v1-698d88b-wwsnv 10.42.2.4 k3d-k3s-default-server-0 None HBONE default notsleep-685df55c6c-nwhs6 10.42.0.9 k3d-k3s-default-agent-0 None HBONE default productpage-v1-675fc69cf-fp65z 10.42.2.6 k3d-k3s-default-server-0 None HBONE default ratings-v1-6484c4d9bb-crjtt 10.42.0.4 k3d-k3s-default-agent-0 None HBONE default reviews-svc-waypoint-c49f9f569-b492t 10.42.2.10 k3d-k3s-default-server-0 None TCP default reviews-v1-5b5d6494f4-nrvfx 10.42.2.5 k3d-k3s-default-server-0 None HBONE default reviews-v2-5b667bcbf8-gj7nz 10.42.0.5 k3d-k3s-default-agent-0 reviews-v2-pod-waypoint HBONE ...
If the value for the pod’s waypoint column isn’t correct, verify your pod is labeled with
istio.io/use-waypoint
and the label’s value is the name of a waypoint that can process workload traffic. For example, if yourreviews
v2
pod uses a waypoint that can only process service traffic, you will not see any waypoint used by that pod. If theistio.io/use-waypoint
label on your pod looks correct verify that the Gateway resource for your waypoint is labeled with a compatible value foristio.io/waypoint-for
. In the case of a pod, suitable values would beall
orworkload
.Check the waypoint’s proxy status via the
istioctl proxy-status
command.$ istioctl proxy-status NAME CLUSTER CDS LDS EDS RDS ECDS ISTIOD VERSION bookinfo-gateway-istio-7c57fc4647-wjqvm.default Kubernetes SYNCED SYNCED SYNCED SYNCED NOT SENT istiod-795d55fc6d-vqtjx 1.23-alpha.75c6eafc5bc8d160b5643c3ea18acb9785855564 reviews-svc-waypoint-c49f9f569-b492t.default Kubernetes SYNCED SYNCED SYNCED NOT SENT NOT SENT istiod-795d55fc6d-vqtjx 1.23-alpha.75c6eafc5bc8d160b5643c3ea18acb9785855564 reviews-v2-pod-waypoint-7f5dbd597-7zzw7.default Kubernetes SYNCED SYNCED NOT SENT NOT SENT NOT SENT istiod-795d55fc6d-vqtjx 1.23-alpha.75c6eafc5bc8d160b5643c3ea18acb9785855564 waypoint-6f7b665c89-6hppr.default Kubernetes SYNCED SYNCED SYNCED NOT SENT NOT SENT istiod-795d55fc6d-vqtjx 1.23-alpha.75c6eafc5bc8d160b5643c3ea18acb9785855564 ...
Enable Envoy’s access log and check the logs of the waypoint proxy after sending some requests:
$ kubectl logs deploy/waypoint
If there is not enough information, you can enable the debug logs for the waypoint proxy:
$ istioctl pc log deploy/waypoint --level debug
Check the envoy configuration for the waypoint via the
istioctl proxy-config
command, which shows all the information related to the waypoint such as clusters, endpoints, listeners, routes and secrets:$ istioctl proxy-config all deploy/waypoint
Refer to the deep dive into Envoy configuration section for more information regarding how to debug Envoy since waypoint proxies are based on Envoy.