Configuration Scoping
In order to program the service mesh, the Istio control plane (Istiod) reads a variety of configurations, including core Kubernetes types like Service
and Node
,
and Istio’s own types like Gateway
.
These are then sent to the data plane (see Architecture for more information).
By default, the control plane will read all configuration in all namespaces. Each proxy instance will receive configuration for all namespaces as well. This includes information about workloads that are not enrolled in the mesh.
This default ensures correct behavior out of the box, but comes with a scalability cost. Each configuration has a cost (in CPU and memory, primarily) to maintain and keep up to date. At large scales, it is critical to limit the configuration scope to avoid excessive resource consumption.
Scoping mechanisms
Istio offers a few tools to help control the scope of a configuration to meet different use cases. Depending on your requirements, these can be used alone or together.
Sidecar
provides a mechanism for specific workloads to import a set of configurationsexportTo
provides a mechanism to export a configuration to a set of workloadsdiscoverySelectors
provides a mechanism to let Istio completely ignore a set of configurations
Sidecar
import
The egress.hosts
field in Sidecar
allows specifying a list of configurations to import.
Only configurations matching the specified criteria will be seen by sidecars impacted by the Sidecar
resource.
For example:
apiVersion: networking.istio.io/v1beta1
kind: Sidecar
metadata:
name: default
spec:
egress:
- hosts:
- "./*" # Import all configuration from our own namespace
- "bookinfo/*" # Import all configuration from the bookinfo namespace
- "external-services/example.com" # Import only 'example.com' from the external-services namespace
exportTo
Istio’s VirtualService
, DestinationRule
, and ServiceEntry
provide a spec.exportTo
field.
Similarly, Service
can be configured with the networking.istio.io/exportTo
annotation.
Unlike Sidecar
which allows a workload owner to control what dependencies it has, exportTo
works in the opposite way, and allows the service owners to control
their own service’s visibility.
For example, this configuration makes the details
Service
only visible to its own namespace, and the client
namespace:
apiVersion: v1
kind: Service
metadata:
name: details
annotations:
networking.istio.io/exportTo: ".,client"
spec: ...
DiscoverySelectors
While the previous controls operate on a workload or service owner level, DiscoverySelectors
provides mesh wide control over configuration visibility.
Discovery selectors allows specifying criteria for which namespaces should be visible to the control plane.
Any namespaces not matching are ignored by the control plane entirely.
This can be configured as part of meshConfig
during installation. For example:
meshConfig:
discoverySelectors:
- matchLabels:
# Allow any namespaces with `istio-discovery=enabled`
istio-discovery: enabled
- matchLabels:
# Allow "kube-system"; Kubernetes automatically adds this label to each namespace
kubernetes.io/metadata.name: kube-system
Frequently asked questions
How can I understand the cost of a certain configuration?
In order to get the best return-on-investment for scoping down configuration, it can be helpful to understand the cost of each object. Unfortunately, there is not a straightforward answer; scalability depends on a large number of factors. However, there are a few general guidelines:
Configuration changes are expensive in Istio, as they require recomputation.
While Endpoints
changes (generally from a Pod scaling up or down) are heavily optimized, most other configurations are fairly expensive.
This can be especially harmful when controllers are constantly making changes to an object (sometimes this happens accidentally!).
Some tools to detect which configurations are changing:
- Istiod will log each change like:
Push debounce stable 1 for config Gateway/default/gateway: ..., full=true
. This shows aGateway
object in thedefault
namespace changed.full=false
would represent and optimized update such asEndpoint
. Note: changes toService
andEndpoints
will all show asServiceEntry
. - Istiod exposes metrics
pilot_k8s_cfg_events
andpilot_k8s_reg_events
for each change. kubectl get <resource> --watch -oyaml --show-managed-fields
can show changes to an object (or objects) to help understand what is changing, and by whom.
Headless services (besides ones declared as HTTP)
scale with the number of instances. This makes large headless services expensive, and a good candidate for exclusion with exportTo
or equivalent.
What happens if I connect to a service outside of my scope?
When connecting to a service that has been excluded through one of the scoping mechanisms, the data plane will not know anything about the destination, so it will be treated as Unmatched traffic.
What about Gateways?
While Gateways will respect exportTo
and DiscoverySelectors
, Sidecar
objects do not impact Gateways.
However, unlike sidecars, gateways do not have configuration for the entire cluster by default.
Instead, each configuration is explicitly attached to the gateway, which mostly avoids this problem.
However, currently part of the data plane configuration (a “cluster”, in Envoy terms), is always sent for the entire cluster, even if it is not referenced explicitly.