Configuring istioctl for a remote cluster

Using a proxy server to support istioctl commands in a mesh with an external control plane.

Mar 25, 2022 | By Frank Budinsky - IBM

When using the istioctl CLI on a remote cluster of an external control plane or a multicluster Istio deployment, some of the commands will not work by default. For example, istioctl proxy-status requires access to the istiod service to retrieve the status and configuration of the proxies it’s managing. If you try running it on a remote cluster, you’ll get an error message like this:

$ istioctl proxy-status
Error: unable to find any Istiod instances

Notice that the error message doesn’t just say that it’s unable to access the istiod service, it specifically mentions its inability to find istiod instances. This is because the istioctl proxy-status implementation needs to retrieve the sync status of not just any single istiod instance, but rather all of them. When there is more than one istiod instance (replica) running, each instance is only connected to a subset of the service proxies running in the mesh. The istioctl command needs to return the status for the entire mesh, not just the subset managed by one of the instances.

In an ordinary Istio installation where the istiod service is running locally on the cluster (i.e., a primary cluster), the command is implemented by simply finding all of the running istiod pods, calling each one in turn, and then aggregating the result before returning it to the user.

CLI with local access to istiod pods
CLI with local access to istiod pods

When using a remote cluster, on the other hand, this is not possible since the istiod instances are running outside of the mesh cluster and not accessible to the mesh user. The instances may not even be deployed using pods on a Kubernetes cluster.

Fortunately, istioctl provides a configuration option to address this issue. You can configure istioctl with the address of an external proxy service that will have access to the istiod instances. Unlike an ordinary load-balancer service, which would delegate incoming requests to one of the instances, this proxy service must instead delegate to all of the istiod instances, aggregate the responses, and then return the combined result.

If the external proxy service is, in fact, running on another Kubernetes cluster, the proxy implementation code can be very similar to the implementation code that istioctl runs in the primary cluster case, i.e., find all of the running istiod pods, call each one in turn, and then aggregate the result.

CLI without local access to istiod pods
CLI without local access to istiod pods

An Istio Ecosystem project that includes an implementation of such an istioctl proxy server can be found here. To try it out, you’ll need two clusters, one of which is configured as a remote cluster using a control plane installed in the other cluster.

Install Istio with a remote cluster topology

To demonstrate istioctl working on a remote cluster, we’ll start by using the external control plane install instructions to set up a single remote cluster mesh with an external control plane running in a separate external cluster.

After completing the installation, we should have two environment variables, CTX_REMOTE_CLUSTER and CTX_EXTERNAL_CLUSTER, containing the context names of the remote (mesh) and external (control plane) clusters, respectively.

We should also have the helloworld and sleep samples running in the mesh, i.e., on the remote cluster:

$ kubectl get pod -n sample --context="${CTX_REMOTE_CLUSTER}"
NAME                             READY   STATUS    RESTARTS   AGE
helloworld-v1-776f57d5f6-tmpkd   2/2     Running   0          10s
sleep-557747455f-v627d           2/2     Running   0          9s

Notice that if you try to run istioctl proxy-status in the remote cluster, you will see the error message described earlier:

$ istioctl proxy-status --context="${CTX_REMOTE_CLUSTER}"
Error: unable to find any Istiod instances

Configure istioctl to use the sample proxy service

To configure istioctl, we first need to deploy the proxy service next to the running istiod pods. In our installation, we’ve deployed the control plane in the external-istiod namespace, so we start the proxy service on the external cluster using the following command:

$ kubectl apply -n external-istiod --context="${CTX_EXTERNAL_CLUSTER}" \
    -f https://raw.githubusercontent.com/istio-ecosystem/istioctl-proxy-sample/main/istioctl-proxy.yaml
service/istioctl-proxy created
serviceaccount/istioctl-proxy created
secret/jwt-cert-key-secret created
deployment.apps/istioctl-proxy created
role.rbac.authorization.k8s.io/istioctl-proxy-role created
rolebinding.rbac.authorization.k8s.io/istioctl-proxy-role created

You can run the following command to confirm that the istioctl-proxy service is running next to istiod:

$ kubectl get po -n external-istiod --context="${CTX_EXTERNAL_CLUSTER}"
NAME                              READY   STATUS    RESTARTS   AGE
istioctl-proxy-664bcc596f-9q8px   1/1     Running   0          15s
istiod-666fb6694d-jklkt           1/1     Running   0          5m31s

The proxy service is a gRPC server that is serving on port 9090:

$ kubectl get svc istioctl-proxy -n external-istiod --context="${CTX_EXTERNAL_CLUSTER}"
NAME             TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
istioctl-proxy   ClusterIP   172.21.127.192   <none>        9090/TCP   11m

Before we can use it, however, we need to expose it outside of the external cluster. There are many ways to do that, depending on the deployment environment. In our setup, we have an ingress gateway running on the external cluster, so we could update it to also expose port 9090, update the associated virtual service to direct port 9090 requests to the proxy service, and then configure istioctl to use the gateway address for the proxy service. This would be a “proper” approach.

However, since this is just a simple demonstration where we have access to both clusters, we will simply port-forward the proxy service to localhost:

$ kubectl port-forward -n external-istiod service/istioctl-proxy 9090:9090 --context="${CTX_EXTERNAL_CLUSTER}"

We now configure istioctl to use localhost:9090 to access the proxy by setting the ISTIOCTL_XDS_ADDRESS environment variable:

$ export ISTIOCTL_XDS_ADDRESS=localhost:9090
$ export ISTIOCTL_ISTIONAMESPACE=external-istiod
$ export ISTIOCTL_PREFER_EXPERIMENTAL=true

Because our control plane is running in the external-istiod namespace, instead of the default istio-system, we also need to set the ISTIOCTL_ISTIONAMESPACE environment variable.

Setting ISTIOCTL_PREFER_EXPERIMENTAL is optional. It instructs istioctl to redirect istioctl command calls to an experimental equivalent, istioctl x command, for any command that has both a stable and experimental implementation. In our case we need to use istioctl x proxy-status, the version that implements the proxy delegation feature.

Run the istioctl proxy-status command

Now that we’re finished configuring istioctl we can try it out by running the proxy-status command again:

$ istioctl proxy-status --context="${CTX_REMOTE_CLUSTER}"
NAME                                                      CDS        LDS        EDS        RDS        ISTIOD         VERSION
helloworld-v1-776f57d5f6-tmpkd.sample                     SYNCED     SYNCED     SYNCED     SYNCED     <external>     1.12.1
istio-ingressgateway-75bfd5668f-lggn4.external-istiod     SYNCED     SYNCED     SYNCED     SYNCED     <external>     1.12.1
sleep-557747455f-v627d.sample                             SYNCED     SYNCED     SYNCED     SYNCED     <external>     1.12.1

As you can see, this time it correctly displays the sync status of all the services running in the mesh. Notice that the ISTIOD column returns the generic value <external>, instead of the instance name (e.g., istiod-666fb6694d-jklkt) that would be displayed if the pod was running locally. In this case, this detail is not available, or needed, by the mesh user. It’s only available on the external cluster for the mesh operator to see.

Summary

In this article, we used a sample proxy server to configure istioctl to work with an external control plane installation. We’ve seen how some of the istioctl CLI commands don’t work out of the box on a remote cluster managed by an external control plane. Commands such as istioctl proxy-status, among others, need access to the istiod service instances managing the mesh, which are unavailable when the control plane is running outside of the mesh cluster. To address this issue, istioctl was configured to delegate to a proxy server, running along side the external control plane, which accesses the istiod instances on its behalf.