Virtual Machines in Single-Network Meshes

8 minute read

This example shows how to integrate a VM or a bare metal host into a single-network Istio mesh deployed on Kubernetes.

Prerequisites

You have already set up Istio on Kubernetes. If you haven’t done so, you can find out how in the Installation guide.
Virtual machines (VMs) must have IP connectivity to the endpoints in the mesh. This typically requires a VPC or a VPN, as well as a container network that provides direct (without NAT or firewall deny) routing to the endpoints. The machine is not required to have access to the cluster IP addresses assigned by Kubernetes.
VMs must have access to a DNS server that resolves names to cluster IP addresses. Options include exposing the Kubernetes DNS server through an internal load balancer, using a Core DNS server, or configuring the IPs in any other DNS server accessible from the VM.

The following instructions:

Assume the expansion VM is running on GCE.
Use Google platform-specific commands for some steps.

Installation steps

Setup consists of preparing the mesh for expansion and installing and configuring each VM.

Preparing the Kubernetes cluster for VMs

The first step when adding non-Kubernetes services to an Istio mesh is to configure the Istio installation itself, and generate the configuration files that let VMs connect to the mesh. Prepare the cluster for the VM with the following commands on a machine with cluster admin privileges:

Create a Kubernetes secret for your generated CA certificates using a command similar to the following. See Certificate Authority (CA) certificates for more details.
The root and intermediate certificate from the samples directory are widely distributed and known. Do not use these certificates in production as your clusters would then be open to security vulnerabilities and compromise.
ZipZipZipZip
```
$ kubectl create namespace istio-system
$ kubectl create secret generic cacerts -n istio-system \
    --from-file=@samples/certs/ca-cert.pem@ \
    --from-file=@samples/certs/ca-key.pem@ \
    --from-file=@samples/certs/root-cert.pem@ \
    --from-file=@samples/certs/cert-chain.pem@
```
For a simple setup, deploy Istio control plane into the cluster
```
$ istioctl manifest apply
```
For further details and customization options, refer to the installation instructions.

Alternatively, the user can create an explicit service of type LoadBalancer and use internal load balancer type. User can also deploy a separate ingress Gateway, with internal load balancer type for both mesh expansion and multicluster. The main requirement is for the exposed address to do TCP load balancing to the Istiod deployment, and for the DNS name associated with the assigned load balancer address to match the certificate provisioned into istiod deployment, defaulting to ‘istiod.istio-system.svc’

Define the namespace the VM joins. This example uses the SERVICE_NAMESPACE environment variable to store the namespace. The value of this variable must match the namespace you use in the configuration files later on, and the identity encoded in the certificates.
```
$ export SERVICE_NAMESPACE="vm"
```

Determine and store the IP address of the Istiod since the VMs access Istiod through this IP address.

$ export IstiodIP=$(kubectl get -n istio-system service istiod -o jsonpath='{.spec.clusterIP}')
$ echo $IstiodIP
10.55.240.12

Generate a cluster.env configuration to deploy in the VMs. This file contains the Kubernetes cluster IP address ranges to intercept and redirect via Envoy. You specify the CIDR range when you install Kubernetes as servicesIpv4Cidr. Replace $MY_ZONE and $MY_PROJECT in the following example commands with the appropriate values to obtain the CIDR after installation:
```
$ ISTIO_SERVICE_CIDR=$(gcloud container clusters describe $K8S_CLUSTER --zone $MY_ZONE --project $MY_PROJECT --format "value(servicesIpv4Cidr)")
$ echo -e "ISTIO_SERVICE_CIDR=$ISTIO_SERVICE_CIDR\n" > cluster.env
```
It is also possible to intercept all traffic, as is done for pods. Depending on vendor and installation mechanism you may use different commands to determine the IP range used for services and pods. Multiple ranges can be specified if the VM is making requests to multiple K8S clusters.
Check the contents of the generated cluster.env file. It should be similar to the following example:
```
$ cat cluster.env
ISTIO_SERVICE_CIDR=10.55.240.0/20
```
If the VM only calls services in the mesh, you can skip this step. Otherwise, add the ports the VM exposes to the cluster.env file with the following command. You can change the ports later if necessary.
```
$ echo "ISTIO_INBOUND_PORTS=3306,8080" >> cluster.env
```
In order to use mesh expansion, the VM must be provisioned with certificates signed by the same root CA as the rest of the mesh.
It is recommended to follow the instructions for “Plugging in External CA Key and Certificates”, and use a separate intermediary CA for provisioning the VM. There are many tools and procedures for managing certificates for VMs - Istio requirement is that the VM will get a certificate with an Istio-compatible SPIFEE SAN, with the correct trust domain, namespace and service account.
As an example, for very simple demo setups, you can also use:
```
$ go run istio.io/istio/security/tools/generate_cert \
      -client -host spiffee://cluster.local/vm/vmname --out-priv key.pem --out-cert cert-chain.pem  -mode citadel
$ kubectl -n istio-system get cm istio-ca-root-cert -o jsonpath='{.data.root-cert\.pem}' > root-cert.pem
```

Setting up the VM

Next, run the following commands on each machine that you want to add to the mesh:

Copy the previously created cluster.env and *.pem files to the VM. For example:

$ export GCE_NAME="your-gce-instance"
$ gcloud compute scp --project=${MY_PROJECT} --zone=${MY_ZONE} {key.pem,cert-chain.pem,cluster.env,root-cert.pem} ${GCE_NAME}:~

Install the Debian package with the Envoy sidecar.

$ gcloud compute ssh --project=${MY_PROJECT} --zone=${MY_ZONE} "${GCE_NAME}"
$ curl -L https://storage.googleapis.com/istio-release/releases/1.5.4/deb/istio-sidecar.deb > istio-sidecar.deb
$ sudo dpkg -i istio-sidecar.deb

Add the IP address of the Istiod to /etc/hosts. Revisit the preparing the cluster section to learn how to obtain the IP address. The following example updates the /etc/hosts file with the Istiod address:
```
$ echo "${IstiodIP} istiod.istio-system.svc" | sudo tee -a /etc/hosts
```

A better options is to configure the DNS resolver of the VM to resolve the address, using a split-DNS server. Using /etc/hosts is an easy to use example. It is also possible to use a real DNS and certificate for Istiod, this is beyond the scope of this document.

Install root-cert.pem, key.pem and cert-chain.pem under /etc/certs/.

$ sudo mkdir -p /etc/certs
$ sudo cp {root-cert.pem,cert-chain.pem,key.pem} /etc/certs

Install root-cert.pem under /var/run/secrets/istio/.
Install cluster.env under /var/lib/istio/envoy/.
```
$ sudo cp cluster.env /var/lib/istio/envoy
```
Transfer ownership of the files in /etc/certs/ , /var/lib/istio/envoy/ and /var/run/secrets/istio/to the Istio proxy.
```
$ sudo chown -R istio-proxy /etc/certs /var/lib/istio/envoy /var/run/secrets/istio/
```
Start Istio using systemctl.
```
$ sudo systemctl start istio
```

Send requests from VM workloads to Kubernetes services

After setup, the machine can access services running in the Kubernetes cluster or on other VMs.

The following example shows accessing a service running in the Kubernetes cluster from a VM using /etc/hosts/, in this case using a service from the Bookinfo example.

First, on the cluster admin machine get the virtual IP address (clusterIP) for the service:
```
$ kubectl get svc productpage -o jsonpath='{.spec.clusterIP}'
10.55.246.247
```

Then on the added VM, add the service name and address to its etc/hosts file. You can then connect to the cluster service from the VM, as in the example below:

$ echo "10.55.246.247 productpage.default.svc.cluster.local" | sudo tee -a /etc/hosts
$ curl -v productpage.default.svc.cluster.local:9080
< HTTP/1.1 200 OK
< content-type: text/html; charset=utf-8
< content-length: 1836
< server: envoy
... html content ...

The server: envoy header indicates that the sidecar intercepted the traffic.

Running services on the added VM

Setup an HTTP server on the VM instance to serve HTTP traffic on port 8080:
```
$ gcloud compute ssh ${GCE_NAME}
$ python -m SimpleHTTPServer 8080
```

Determine the VM instance’s IP address. For example, find the IP address of the GCE instance with the following commands:

$ export GCE_IP=$(gcloud --format="value(networkInterfaces[0].networkIP)" compute instances describe ${GCE_NAME})
$ echo ${GCE_IP}

Add VM services to the mesh
```
$ istioctl experimental add-to-mesh external-service vmhttp ${VM_IP} http:8080 -n ${SERVICE_NAMESPACE}
```
Ensure you have added the istioctl client to your path, as described in the download page.

Deploy a pod running the sleep service in the Kubernetes cluster, and wait until it is ready:

$ kubectl apply -f @samples/sleep/sleep.yaml@
$ kubectl get pod
NAME                             READY     STATUS    RESTARTS   AGE
sleep-88ddbcfdd-rm42k            2/2       Running   0          1s
...

Send a request from the sleep service on the pod to the VM’s HTTP service:

$ kubectl exec -it sleep-88ddbcfdd-rm42k -c sleep -- curl vmhttp.${SERVICE_NAMESPACE}.svc.cluster.local:8080

You should see something similar to the output below.

<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 3.2 Final//EN"><html>
<title>Directory listing for /</title>
<body>
<h2>Directory listing for /</h2>
<hr>
<ul>
<li><a href=".bashrc">.bashrc</a></li>
<li><a href=".ssh/">.ssh/</a></li>
...
</body>

Congratulations! You successfully configured a service running in a pod within the cluster to send traffic to a service running on a VM outside of the cluster and tested that the configuration worked.

Cleanup

Run the following commands to remove the expansion VM from the mesh’s abstract model.

$ istioctl experimental remove-from-mesh -n ${SERVICE_NAMESPACE} vmhttp
Kubernetes Service "vmhttp.vm" has been deleted for external service "vmhttp"
Service Entry "mesh-expansion-vmhttp" has been deleted for external service "vmhttp"

Troubleshooting

The following are some basic troubleshooting steps for common VM-related issues.

When making requests from a VM to the cluster, ensure you don’t run the requests as root or istio-proxy user. By default, Istio excludes both users from interception.

Verify the machine can reach the IP of the all workloads running in the cluster. For example:

$ kubectl get endpoints productpage -o jsonpath='{.subsets[0].addresses[0].ip}'
10.52.39.13

$ curl 10.52.39.13:9080
html output

Check the status of the Istio Agent and sidecar:
```
$ sudo systemctl status istio
```

Check that the processes are running. The following is an example of the processes you should see on the VM if you run ps, filtered for istio:

$ ps aux | grep istio
root      6955  0.0  0.0  49344  3048 ?        Ss   21:32   0:00 su -s /bin/bash -c INSTANCE_IP=10.150.0.5 POD_NAME=demo-vm-1 POD_NAMESPACE=vm exec /usr/local/bin/pilot-agent proxy > /var/log/istio/istio.log istio-proxy
istio-p+  7016  0.0  0.1 215172 12096 ?        Ssl  21:32   0:00 /usr/local/bin/pilot-agent proxy
istio-p+  7094  4.0  0.3  69540 24800 ?        Sl   21:32   0:37 /usr/local/bin/envoy -c /etc/istio/proxy/envoy-rev1.json --restart-epoch 1 --drain-time-s 2 --parent-shutdown-time-s 3 --service-cluster istio-proxy --service-node sidecar~10.150.0.5~demo-vm-1.vm-vm.svc.cluster.local

Check the Envoy access and error logs:

$ tail /var/log/istio/istio.log
$ tail /var/log/istio/istio.err.log