Istio and Envoy WebAssembly Extensibility, One Year On

One year ago today, in the 1.5 release, we introduced WebAssembly-based extensibility to Istio. Over the course of the year, the Istio, Envoy, and Proxy-Wasm communities have continued our joint efforts to make WebAssembly (Wasm) extensibility stable, reliable, and easy to adopt. Let’s walk through the updates to Wasm support through the Istio 1.9 release, and our plans for the future.

WebAssembly support merged in upstream Envoy

After adding experimental support for Wasm and the WebAssembly for Proxies (Proxy-Wasm) ABI to Istio’s fork of Envoy, we collected some great feedback from our community of early adopters. This, combined with the experience gained from developing core Istio Wasm extensions, helped us mature and stabilize the runtime. These improvements unblocked merging Wasm support directly into Envoy upstream in October 2020, allowing it to become part of all official Envoy releases. This was a significant milestone, since it indicates that:

• The runtime is ready for wider adoption.
• The programming ABI/API, extension configuration API, and runtime behavior, are becoming stable.
• You can expect a larger community of adoption and support moving forward.

wasm-extensions Ecosystem Repository

As an early adopter of the Envoy Wasm runtime, the Istio Extensions and Telemetry working group gained a lot of experience in developing extensions. We built several first-class extensions, including metadata exchange, Prometheus stats, and attribute generation. In order to share our learning more broadly, we created a wasm-extensions repository in the istio-ecosystem organization. This repository serves two purposes:

• It provides canonical example extensions, covering several highly demanded features (such as basic authentication).
• It provides a guide for Wasm extension development, testing, and release. The guide is based on the same build tool chains and test frameworks that are used, maintained and tested by the Istio extensibility team.

The guide currently covers WebAssembly extension development and unit testing with C++, as well as integration testing with a Go test framework, which simulates a real runtime by running a Wasm module with the Istio proxy binary. In the future, we will also add several more canonical extensions, such as an integration with Open Policy Agent, and header manipulation based on JWT tokens.

Wasm module distribution via the Istio Agent

Prior to Istio 1.9, Envoy remote data sources were needed to distribute remote Wasm modules to the proxy. In this example, you can see two EnvoyFilter resources are defined: one to add a remote fetch Envoy cluster, and the other one to inject a Wasm filter into the HTTP filter chain. This method has a drawback: if remote fetch fails, either due to bad configuration or transient error, Envoy will be stuck with the bad configuration. If a Wasm extension is configured as fail closed, a bad remote fetch will stop Envoy from serving. To fix this issue, a fundamental change is needed to the Envoy xDS protocol to make it allow asynchronous xDS responses.

Istio 1.9 provides a reliable distribution mechanism out of the box by leveraging the xDS proxy inside istio-agent and Envoy’s Extension Configuration Discovery Service (ECDS).

istio-agent intercepts the extension config resource update from istiod, reads the remote fetch hint from it, downloads the Wasm module, and rewrites the ECDS configuration with the path of the downloaded Wasm module. If the download fails, istio-agent will reject the ECDS update and prevent a bad configuration reaching Envoy. For more detail, please see our docs on Wasm module distribution.

Istio Wasm SIG and Future Work

Although we have made a lot of progress on Wasm extensibility, there are still many aspects of the project that remain to be completed. In order to consolidate the efforts from various parties and better tackle the challenges ahead, we have formed an Istio WebAssembly SIG, with aim of providing a standard and reliable way for Istio to consume Wasm extensions. Here are some of the things we are working on:

• A first-class extension API: Currently Wasm extensions needs to be injected via Istio’s EnvoyFilter API. A first-class extension API will make using Wasm with Istio easier, and we expect this to be introduced in Istio 1.10.
• Distribution artifacts interoperability: Built on top of Solo.io’s WebAssembly OCI image spec effort, a standard Wasm artifacts format will make it easy to build, pull, publish, and execute.
• Container Storage Interface (CSI) based artifacts distribution: Using istio-agent to distribute modules is easy for adoption, but may not be efficient as each proxy will keep a copy of the Wasm module. As a more efficient solution, with Ephemeral CSI, a DaemonSet will be provided which could configure storage for pods. Working similarly to a CNI plugin, a CSI driver would fetch the Wasm module out-of-band from the xDS flow and mount it inside the rootfs when the pod starts up.

If you would like to join us, the group will meet every other week Tuesdays at 2PM PT. You can find the meeting on the Istio working group calendar.

We look forward to seeing how you will use Wasm to extend Istio!