Configuring Gateway Network Topology [Experimental]

 6 minute read     page test

Forwarding external client attributes (IP address, certificate info) to destination workloads

Many applications require knowing the client IP address and certificate information of the originating request to behave properly. Notable cases include logging and audit tools that require the client IP be populated and security tools, such as Web Application Firewalls (WAF), that need this information to apply rule sets properly. The ability to provide client attributes to services has long been a staple of reverse proxies. To forward these client attributes to destination workloads, proxies use the X-Forwarded-For (XFF) and X-Forwarded-Client-Cert (XFCC) headers.

Today’s networks vary widely in nature, but support for these attributes is a requirement no matter what the network topology is. This information should be preserved and forwarded whether the network uses cloud-based Load Balancers, on-premise Load Balancers, gateways that are exposed directly to the internet, gateways that serve many intermediate proxies, and other deployment topologies not specified.

While Istio provides an ingress gateway, given the varieties of architectures mentioned above, reasonable defaults are not able to be shipped that support the proper forwarding of client attributes to the destination workloads. This becomes ever more vital as Istio multicluster deployment models become more common.

For more information on X-Forwarded-For, see the IETF’s RFC.

Configuring network topologies

Configuration of XFF and XFCC headers can be set globally for all gateway workloads via MeshConfig or per gateway using a pod annotation. For example, to configure globally during install or upgrade when using an IstioOperator custom resource:

spec:
  meshConfig:
    defaultConfig:
      gatewayTopology:
        numTrustedProxies: <VALUE>
        forwardClientCertDetails: <ENUM_VALUE>

You can also configure both of these settings by adding the proxy.istio.io/config annotation to the Pod spec of your Istio ingress gateway.

...
  metadata:
    annotations:
      "proxy.istio.io/config": '{"gatewayTopology" : { "numTrustedProxies": <VALUE>, "forwardClientCertDetails": <ENUM_VALUE> } }'

Configuring X-Forwarded-For Headers

Applications rely on reverse proxies to forward client attributes in a request, such as X-Forward-For header. However, due to the variety of network topologies that Istio can be deployed in, you must set the numTrustedProxies to the number of trusted proxies deployed in front of the Istio gateway proxy, so that the client address can be extracted correctly. This controls the value populated by the ingress gateway in the X-Envoy-External-Address header which can be reliably used by the upstream services to access client’s original IP address.

For example, if you have a cloud based Load Balancer and a reverse proxy in front of your Istio gateway, set numTrustedProxies to 2.

Example using X-Forwarded-For capability with httpbin

  1. Run the following command to create a file named topology.yaml with numTrustedProxies set to 2 and install Istio:

    $ cat <<EOF > topology.yaml
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  meshConfig:
    defaultConfig:
      gatewayTopology:
        numTrustedProxies: 2
EOF
$ istioctl install -f topology.yaml

  2. Create an httpbin namespace:

    $ kubectl create namespace httpbin
namespace/httpbin created

  3. Set the istio-injection label to enabled for sidecar injection:

    $ kubectl label --overwrite namespace httpbin istio-injection=enabled
namespace/httpbin labeled

  4. Deploy httpbin in the httpbin namespace:

    $ kubectl apply -n httpbin -f samples/httpbin/httpbin.yaml

  5. Deploy a gateway associated with httpbin:

    $ kubectl apply -n httpbin -f samples/httpbin/httpbin-gateway.yaml

  6. Set a local GATEWAY_URL environmental variable based on your Istio ingress gateway’s IP address:

    $ export GATEWAY_URL=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')

  7. Run the following curl command to simulate a request with proxy addresses in the X-Forwarded-For header:

    $ curl -s -H 'X-Forwarded-For: 56.5.6.7, 72.9.5.6, 98.1.2.3' "$GATEWAY_URL"/get?show_env=true
{
  "args": {
    "show_env": "true"
  },
  "headers": {
    "Accept": ...
    "Content-Length": ...
    "Host": ...
    "User-Agent": ...
    "X-B3-Parentspanid": ...
    "X-B3-Sampled": ...
    "X-B3-Spanid": ...
    "X-B3-Traceid": ...
    "X-Envoy-Attempt-Count": ...
    "X-Envoy-External-Address": "72.9.5.6",
    "X-Forwarded-Client-Cert": ...
    "X-Forwarded-For": "56.5.6.7, 72.9.5.6, 98.1.2.3,10.244.0.1",
    "X-Forwarded-Proto": ...
    "X-Request-Id": ...
  },
  "origin": "56.5.6.7, 72.9.5.6, 98.1.2.3,10.244.0.1",
  "url": ...
}

The above output shows the request headers that the httpbin workload received. When the Istio gateway received this request, it set the X-Envoy-External-Address header to the second to last (numTrustedProxies: 2) address in the X-Forwarded-For header from your curl command. Additionally, the gateway appends its own IP to the X-Forwarded-For header before forwarding it to the httpbin workload.

Configuring X-Forwarded-Client-Cert Headers

From Envoy’s documentation regarding XFCC:

To configure how XFCC headers are handled, set forwardClientCertDetails in your IstioOperator

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  meshConfig:
    defaultConfig:
      gatewayTopology:
        forwardClientCertDetails: <ENUM_VALUE>

where ENUM_VALUE can be of the following type.

ENUM_VALUE
UNDEFINEDField is not set.
SANITIZEDo not send the XFCC header to the next hop. This is the default value for a gateway.
FORWARD_ONLYWhen the client connection is mTLS (Mutual TLS), forward the XFCC header in the request.
APPEND_FORWARDWhen the client connection is mTLS, append the client certificate information to the request’s XFCC header and forward it.
SANITIZE_SETWhen the client connection is mTLS, reset the XFCC header with the client certificate information and send it to the next hop.
ALWAYS_FORWARD_ONLYAlways forward the XFCC header in the request, regardless of whether the client connection is mTLS.

See the Envoy documentation for examples of using this capability.

PROXY Protocol

The PROXY protocol allows for exchanging and preservation of client attributes across multiple proxies without relying on Layer 7 protocols.

If your external load balancer is configured to use the PROXY protocol, the Istio gateway must also be configured to accept the PROXY protocol. Enabling this requires adding the Envoy Proxy Protocol filter using an EnvoyFilter applied on the gateway workload. For example:

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: proxy-protocol
  namespace: istio-system
spec:
  configPatches:
  - applyTo: LISTENER
    patch:
      operation: MERGE
      value:
        listener_filters:
        - name: envoy.listener.proxy_protocol
        - name: envoy.listener.tls_inspector
  workloadSelector:
    labels:
      istio: ingressgateway

The client IP is retrieved from the PROXY protocol and set (or appended) in the X-Forwarded-For and X-Envoy-External-Address header. When PROXY protocol is used in conjunction with the gatewayTopology configuration, the numTrustedProxies and the received X-Forwarded-For header takes precedence in determining the trusted client addresses.

Was this information useful?
Do you have any suggestions for improvement?

Thanks for your feedback!