Virtual Machines in Multi-Network Meshes

5 minute read page test

This example provides instructions to integrate a VM or a bare metal host into a multi-network Istio mesh deployed on Kubernetes using gateways. This approach doesn’t require VPN connectivity or direct network access between the VM, the bare metal and the clusters.

Prerequisites

One or more Kubernetes clusters with versions: 1.16, 1.17, 1.18, 1.19.
Virtual machines (VMs) must have IP connectivity to the Ingress gateways in the mesh.
Services in the cluster must be accessible through the Ingress gateway.

Installation steps

Setup consists of preparing the mesh for expansion and installing and configuring each VM.

Preparing your environment

When expanding Istio’s mesh capabilities to VMs across multiple networks (where the VM is in a network where traffic cannot directly route to pods in the Kubernetes cluster, for example), we’ll need to take advantage of Istio’s split-horizon DNS capabilities.

Before we get started, you should prepare a VM and connect it to the Istio control plane through the Ingress Gateway. These steps are detailed in Setup: Install: Virtual Machine Installation.

Note There are a few alterations to that document as follows:

When we create the IstioOperator resource, we need to specify which networks we expect, and how to treat them
When creating the cluster.env we need to specify which network in which the VM belongs
We need to create a Gateway resource to allow application traffic from the VM to the workload items running in the mesh

Installing the Istio Control Plane

When following the Virtual Machine Installation setup guide to install the control plane, we will need to tweak the installation as follows:

Specify the expected networks in the mesh, including the vm-network

$ cat <<EOF> ./vmintegration.yaml
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  values:
    global:
      meshExpansion:
        enabled: true
      multiCluster:
        clusterName: kube-cluster
      network: main-network
      meshNetworks:
        main-network:
          endpoints:
          - fromRegistry: kube-cluster
          gateways:
          - registryServiceName: istio-ingressgateway.istio-system.svc.cluster.local
            port: 443

EOF

Install with the virtual-machine features enabled:
```
$ istioctl install -f ./vmintegration.yaml
```

Specify the network for the VM sidecar

When following the Virtual Machine Installation setup guide for creating the cluster.env file we need to tweak the installation by adding the following entry:

$ echo "ISTIO_META_NETWORK=vm-network" >> cluster.env

Create Gateway for application traffic

The last step is to create a Gateway resource that allows application traffic from the VMs to route correctly.

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: cluster-aware-gateway
  namespace: istio-system
spec:
  selector:
    istio: ingressgateway
  servers:
  - port:
      number: 443
      name: tls
      protocol: TLS
    tls:
      mode: AUTO_PASSTHROUGH
    hosts:
    - "*.local"

Applying this gateway will route any of the traffic from the VM destined for the workloads in the mesh running on *.local

At this point, you can continue with the Setup Virtual Machine documentation.

Verify setup

After setup, the machine can access services running in the Kubernetes cluster or on other VMs. When a service on the VM tries to access a service in the mesh running on Kubernetes, the endpoints (i.e., IPs) for those services will be the ingress gateway on the Kubernetes Cluster. To verify that, on the VM run the following command (assuming you have a service named httpbin on the Kubernetes cluster):

$ curl -v localhost:15000/clusters | grep httpbin

This should show endpoints for httpbin that point to the ingress gateway similar to this:

outbound|8000||httpbin.default.svc.cluster.local::34.72.46.113:443::cx_active::1
outbound|8000||httpbin.default.svc.cluster.local::34.72.46.113:443::cx_connect_fail::0
outbound|8000||httpbin.default.svc.cluster.local::34.72.46.113:443::cx_total::1
outbound|8000||httpbin.default.svc.cluster.local::34.72.46.113:443::rq_active::0

The IP 34.72.46.113 in this case is the ingress gateway public endpoint.

Send requests from VM workloads to Kubernetes services

At this point we should be able to send traffic to httpbin.default.svc.cluster.local and get a response from the server. You may have to set up DNS in /etc/hosts to map the httpbin.default.svc.cluster.local domain name to an IP since the IP will not resolve. In this case, the IP should be an IP that gets routed to the local Istio Proxy sidecar. You can use the IP from the ISTIO_SERVICE_CIDR variable in the cluster.env file you created in the Setup Virtual Machine documentation.

$ curl -v httpbin.default.svc.cluster.local:8000/headers

Running services on the added VM

Setup an HTTP server on the VM instance to serve HTTP traffic on port 8080:
```
$ python -m SimpleHTTPServer 8080
```
Note, you may have to open firewalls to be able to access the 8080 port on your VM

Add VM services to the mesh

Add a service to the Kubernetes cluster into a namespace (in this example, <vm-namespace>) where you prefer to keep resources (like Service, ServiceEntry, WorkloadEntry, ServiceAccount) with the VM services:

$ cat <<EOF | kubectl -n <vm-namespace> apply -f -
apiVersion: v1
kind: Service
metadata:
  name: cloud-vm
  labels:
    app: cloud-vm
spec:
  ports:
  - port: 8080
    name: http-vm
    targetPort: 8080
  selector:
    app: cloud-vm
EOF

Lastly create a workload with the external IP of the VM (substitute VM_IP with the IP of your VM):

$ cat <<EOF | kubectl -n <vm-namespace> apply -f -
apiVersion: networking.istio.io/v1beta1
kind: WorkloadEntry
metadata:
  name: "cloud-vm"
  namespace: "<vm-namespace>"
spec:
  address: "${VM_IP}"
  labels:
    app: cloud-vm
  serviceAccount: "<service-account>"
EOF

Deploy a pod running the sleep service in the Kubernetes cluster, and wait until it is ready:

$ kubectl apply -f @samples/sleep/sleep.yaml@
$ kubectl get pod
NAME                             READY     STATUS    RESTARTS   AGE
sleep-88ddbcfdd-rm42k            2/2       Running   0          1s
...

Send a request from the sleep service on the pod to the VM’s HTTP service:

$ kubectl exec -it sleep-88ddbcfdd-rm42k -c sleep -- curl cloud-vm.${VM_NAMESPACE}.svc.cluster.local:8080

You should see something similar to the output below.

<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 3.2 Final//EN"><html>
<title>Directory listing for /</title>
<body>
<h2>Directory listing for /</h2>
<hr>
<ul>
<li><a href=".bashrc">.bashrc</a></li>
<li><a href=".ssh/">.ssh/</a></li>
...
</body>

Congratulations! You successfully configured a service running in a pod within the cluster to send traffic to a service running on a VM outside of the cluster and tested that the configuration worked.

Cleanup

At this point, you can remove the VM resources from the Kubernetes cluster in the <vm-namespace> namespace.