install-cni
Install and configure Istio CNI plugin on a node, detect and repair pod which is broken by race condition.
install-cni [flags]
| Flags | Description |
|---|---|
--ambient-enabled | Whether ambient controller is enabled |
--chained-cni-plugin | Whether to install CNI plugin as a chained or standalone |
--cni-conf-name <string> | Name of the CNI configuration file (default ``) |
--cni-event-address <string> | The UDS server address which CNI plugin will forward ambient pod creation events to (default `/var/run/istio-cni/pluginevent.sock`) |
--cni-net-dir <string> | Directory on the host where CNI network plugins are installed (default `/etc/cni/net.d`) |
--cni-network-config <string> | CNI configuration template as a string (default ``) |
--cni-network-config-file <string> | CNI config template as a file (default ``) |
--ctrlz_address <string> | The IP Address to listen on for the ControlZ introspection facility. Use '*' to indicate all addresses. (default `localhost`) |
--ctrlz_port <uint16> | The IP port to use for the ControlZ introspection facility (default `9876`) |
--kube-ca-file <string> | CA file for kubeconfig. Defaults to the same as install-cni pod (default ``) |
--kubecfg-file-name <string> | Name of the kubeconfig file which CNI plugin will use when interacting with API server (default `ZZZ-istio-cni-kubeconfig`) |
--kubeconfig-mode <int> | File mode of the kubeconfig file (default `384`) |
--log-level <string> | Fallback value for log level in CNI config file, if not specified in helm template (default `warn`) |
--log-uds-address <string> | The UDS server address which CNI plugin will copy log output to (default `/var/run/istio-cni/log.sock`) |
--log_as_json | Whether to format output as JSON or in plain console-friendly format |
--log_caller <string> | Comma-separated list of scopes for which to include caller information, scopes can be any of [all, ambient, cni, controllers, default, install, iptables, klog, model, monitoring, repair, spiffe, trustBundle, validation] (default ``) |
--log_output_level <string> | Comma-separated minimum per-scope logging level of messages to output, in the form of <scope>:<level>,<scope>:<level>,... where scope can be one of [all, ambient, cni, controllers, default, install, iptables, klog, model, monitoring, repair, spiffe, trustBundle, validation] and level can be one of [debug, info, warn, error, fatal, none] (default `default:info`) |
--log_rotate <string> | The path for the optional rotating log file (default ``) |
--log_rotate_max_age <int> | The maximum age in days of a log file beyond which the file is rotated (0 indicates no limit) (default `30`) |
--log_rotate_max_backups <int> | The maximum number of log file backups to keep before older files are deleted (0 indicates no limit) (default `1000`) |
--log_rotate_max_size <int> | The maximum size in megabytes of a log file beyond which the file is rotated (default `104857600`) |
--log_stacktrace_level <string> | Comma-separated minimum per-scope logging level at which stack traces are captured, in the form of <scope>:<level>,<scope:level>,... where scope can be one of [all, ambient, cni, controllers, default, install, iptables, klog, model, monitoring, repair, spiffe, trustBundle, validation] and level can be one of [debug, info, warn, error, fatal, none] (default `default:none`) |
--log_target <stringArray> | The set of paths where to output the log. This can be any path as well as the special values stdout and stderr (default `[stdout]`) |
--monitoring-port <int> | HTTP port to serve prometheus metrics (default `15014`) |
--mounted-cni-net-dir <string> | Directory on the container where CNI networks are installed (default `/host/etc/cni/net.d`) |
--repair-broken-pod-label-key <string> | The key portion of the label which will be set by the race repair if label pods is true (default `cni.istio.io/uninitialized`) |
--repair-broken-pod-label-value <string> | The value portion of the label which will be set by the race repair if label pods is true (default `true`) |
--repair-delete-pods | Controller will delete pods when detecting pod broken by race condition |
--repair-enabled | Whether to enable race condition repair or not |
--repair-field-selectors <string> | A set of field selectors in label=value format that will be added to the pod list filters (default ``) |
--repair-init-container-exit-code <int> | Expected exit code for the init container when crash-looping because of CNI misconfiguration (default `126`) |
--repair-init-container-name <string> | The name of the istio init container (will crash-loop if CNI is not configured for the pod) (default `istio-validation`) |
--repair-init-container-termination-message <string> | The expected termination message for the init container when crash-looping because of CNI misconfiguration (default ``) |
--repair-label-pods | Controller will label pods when detecting pod broken by race condition |
--repair-label-selectors <string> | A set of label selectors in label=value format that will be added to the pod list filters (default ``) |
--repair-node-name <string> | The name of the managed node (will manage all nodes if unset) (default ``) |
--repair-sidecar-annotation <string> | An annotation key that indicates this pod contains an istio sidecar. All pods without this annotation will be ignored.The value of the annotation is ignored. (default `sidecar.istio.io/status`) |
--skip-tls-verify | Whether to use insecure TLS in kubeconfig file |
--ztunnel-uds-address <string> | The UDS server address which ztunnel will connect to (default `/var/run/ztunnel/ztunnel.sock`) |
install-cni completion
Generate the autocompletion script for install-cni for the specified shell. See each sub-command's help for details on how to use the generated script.
| Flags | Description |
|---|---|
--ctrlz_address <string> | The IP Address to listen on for the ControlZ introspection facility. Use '*' to indicate all addresses. (default `localhost`) |
--ctrlz_port <uint16> | The IP port to use for the ControlZ introspection facility (default `9876`) |
--log_as_json | Whether to format output as JSON or in plain console-friendly format |
--log_caller <string> | Comma-separated list of scopes for which to include caller information, scopes can be any of [all, ambient, cni, controllers, default, install, iptables, klog, model, monitoring, repair, spiffe, trustBundle, validation] (default ``) |
--log_output_level <string> | Comma-separated minimum per-scope logging level of messages to output, in the form of <scope>:<level>,<scope>:<level>,... where scope can be one of [all, ambient, cni, controllers, default, install, iptables, klog, model, monitoring, repair, spiffe, trustBundle, validation] and level can be one of [debug, info, warn, error, fatal, none] (default `default:info`) |
--log_rotate <string> | The path for the optional rotating log file (default ``) |
--log_rotate_max_age <int> | The maximum age in days of a log file beyond which the file is rotated (0 indicates no limit) (default `30`) |
--log_rotate_max_backups <int> | The maximum number of log file backups to keep before older files are deleted (0 indicates no limit) (default `1000`) |
--log_rotate_max_size <int> | The maximum size in megabytes of a log file beyond which the file is rotated (default `104857600`) |
--log_stacktrace_level <string> | Comma-separated minimum per-scope logging level at which stack traces are captured, in the form of <scope>:<level>,<scope:level>,... where scope can be one of [all, ambient, cni, controllers, default, install, iptables, klog, model, monitoring, repair, spiffe, trustBundle, validation] and level can be one of [debug, info, warn, error, fatal, none] (default `default:none`) |
--log_target <stringArray> | The set of paths where to output the log. This can be any path as well as the special values stdout and stderr (default `[stdout]`) |
install-cni completion bash
Generate the autocompletion script for the bash shell.
This script depends on the 'bash-completion' package. If it is not installed already, you can install it via your OS's package manager.
To load completions in your current shell session:
source <(install-cni completion bash)
To load completions for every new session, execute once:
#### Linux:
install-cni completion bash > /etc/bash_completion.d/install-cni
#### macOS:
install-cni completion bash > $(brew --prefix)/etc/bash_completion.d/install-cni
You will need to start a new shell for this setup to take effect.
install-cni completion bash
| Flags | Description |
|---|---|
--ctrlz_address <string> | The IP Address to listen on for the ControlZ introspection facility. Use '*' to indicate all addresses. (default `localhost`) |
--ctrlz_port <uint16> | The IP port to use for the ControlZ introspection facility (default `9876`) |
--log_as_json | Whether to format output as JSON or in plain console-friendly format |
--log_caller <string> | Comma-separated list of scopes for which to include caller information, scopes can be any of [all, ambient, cni, controllers, default, install, iptables, klog, model, monitoring, repair, spiffe, trustBundle, validation] (default ``) |
--log_output_level <string> | Comma-separated minimum per-scope logging level of messages to output, in the form of <scope>:<level>,<scope>:<level>,... where scope can be one of [all, ambient, cni, controllers, default, install, iptables, klog, model, monitoring, repair, spiffe, trustBundle, validation] and level can be one of [debug, info, warn, error, fatal, none] (default `default:info`) |
--log_rotate <string> | The path for the optional rotating log file (default ``) |
--log_rotate_max_age <int> | The maximum age in days of a log file beyond which the file is rotated (0 indicates no limit) (default `30`) |
--log_rotate_max_backups <int> | The maximum number of log file backups to keep before older files are deleted (0 indicates no limit) (default `1000`) |
--log_rotate_max_size <int> | The maximum size in megabytes of a log file beyond which the file is rotated (default `104857600`) |
--log_stacktrace_level <string> | Comma-separated minimum per-scope logging level at which stack traces are captured, in the form of <scope>:<level>,<scope:level>,... where scope can be one of [all, ambient, cni, controllers, default, install, iptables, klog, model, monitoring, repair, spiffe, trustBundle, validation] and level can be one of [debug, info, warn, error, fatal, none] (default `default:none`) |
--log_target <stringArray> | The set of paths where to output the log. This can be any path as well as the special values stdout and stderr (default `[stdout]`) |
--no-descriptions | disable completion descriptions |
install-cni completion fish
Generate the autocompletion script for the fish shell.
To load completions in your current shell session:
install-cni completion fish | source
To load completions for every new session, execute once:
install-cni completion fish > ~/.config/fish/completions/install-cni.fish
You will need to start a new shell for this setup to take effect.
install-cni completion fish [flags]
| Flags | Description |
|---|---|
--ctrlz_address <string> | The IP Address to listen on for the ControlZ introspection facility. Use '*' to indicate all addresses. (default `localhost`) |
--ctrlz_port <uint16> | The IP port to use for the ControlZ introspection facility (default `9876`) |
--log_as_json | Whether to format output as JSON or in plain console-friendly format |
--log_caller <string> | Comma-separated list of scopes for which to include caller information, scopes can be any of [all, ambient, cni, controllers, default, install, iptables, klog, model, monitoring, repair, spiffe, trustBundle, validation] (default ``) |
--log_output_level <string> | Comma-separated minimum per-scope logging level of messages to output, in the form of <scope>:<level>,<scope>:<level>,... where scope can be one of [all, ambient, cni, controllers, default, install, iptables, klog, model, monitoring, repair, spiffe, trustBundle, validation] and level can be one of [debug, info, warn, error, fatal, none] (default `default:info`) |
--log_rotate <string> | The path for the optional rotating log file (default ``) |
--log_rotate_max_age <int> | The maximum age in days of a log file beyond which the file is rotated (0 indicates no limit) (default `30`) |
--log_rotate_max_backups <int> | The maximum number of log file backups to keep before older files are deleted (0 indicates no limit) (default `1000`) |
--log_rotate_max_size <int> | The maximum size in megabytes of a log file beyond which the file is rotated (default `104857600`) |
--log_stacktrace_level <string> | Comma-separated minimum per-scope logging level at which stack traces are captured, in the form of <scope>:<level>,<scope:level>,... where scope can be one of [all, ambient, cni, controllers, default, install, iptables, klog, model, monitoring, repair, spiffe, trustBundle, validation] and level can be one of [debug, info, warn, error, fatal, none] (default `default:none`) |
--log_target <stringArray> | The set of paths where to output the log. This can be any path as well as the special values stdout and stderr (default `[stdout]`) |
--no-descriptions | disable completion descriptions |
install-cni completion powershell
Generate the autocompletion script for powershell.
To load completions in your current shell session:
install-cni completion powershell | Out-String | Invoke-Expression
To load completions for every new session, add the output of the above command to your powershell profile.
install-cni completion powershell [flags]
| Flags | Description |
|---|---|
--ctrlz_address <string> | The IP Address to listen on for the ControlZ introspection facility. Use '*' to indicate all addresses. (default `localhost`) |
--ctrlz_port <uint16> | The IP port to use for the ControlZ introspection facility (default `9876`) |
--log_as_json | Whether to format output as JSON or in plain console-friendly format |
--log_caller <string> | Comma-separated list of scopes for which to include caller information, scopes can be any of [all, ambient, cni, controllers, default, install, iptables, klog, model, monitoring, repair, spiffe, trustBundle, validation] (default ``) |
--log_output_level <string> | Comma-separated minimum per-scope logging level of messages to output, in the form of <scope>:<level>,<scope>:<level>,... where scope can be one of [all, ambient, cni, controllers, default, install, iptables, klog, model, monitoring, repair, spiffe, trustBundle, validation] and level can be one of [debug, info, warn, error, fatal, none] (default `default:info`) |
--log_rotate <string> | The path for the optional rotating log file (default ``) |
--log_rotate_max_age <int> | The maximum age in days of a log file beyond which the file is rotated (0 indicates no limit) (default `30`) |
--log_rotate_max_backups <int> | The maximum number of log file backups to keep before older files are deleted (0 indicates no limit) (default `1000`) |
--log_rotate_max_size <int> | The maximum size in megabytes of a log file beyond which the file is rotated (default `104857600`) |
--log_stacktrace_level <string> | Comma-separated minimum per-scope logging level at which stack traces are captured, in the form of <scope>:<level>,<scope:level>,... where scope can be one of [all, ambient, cni, controllers, default, install, iptables, klog, model, monitoring, repair, spiffe, trustBundle, validation] and level can be one of [debug, info, warn, error, fatal, none] (default `default:none`) |
--log_target <stringArray> | The set of paths where to output the log. This can be any path as well as the special values stdout and stderr (default `[stdout]`) |
--no-descriptions | disable completion descriptions |
install-cni completion zsh
Generate the autocompletion script for the zsh shell.
If shell completion is not already enabled in your environment you will need to enable it. You can execute the following once:
echo "autoload -U compinit; compinit" >> ~/.zshrc
To load completions in your current shell session:
source <(install-cni completion zsh)
To load completions for every new session, execute once:
#### Linux:
install-cni completion zsh > "${fpath[1]}/_install-cni"
#### macOS:
install-cni completion zsh > $(brew --prefix)/share/zsh/site-functions/_install-cni
You will need to start a new shell for this setup to take effect.
install-cni completion zsh [flags]
| Flags | Description |
|---|---|
--ctrlz_address <string> | The IP Address to listen on for the ControlZ introspection facility. Use '*' to indicate all addresses. (default `localhost`) |
--ctrlz_port <uint16> | The IP port to use for the ControlZ introspection facility (default `9876`) |
--log_as_json | Whether to format output as JSON or in plain console-friendly format |
--log_caller <string> | Comma-separated list of scopes for which to include caller information, scopes can be any of [all, ambient, cni, controllers, default, install, iptables, klog, model, monitoring, repair, spiffe, trustBundle, validation] (default ``) |
--log_output_level <string> | Comma-separated minimum per-scope logging level of messages to output, in the form of <scope>:<level>,<scope>:<level>,... where scope can be one of [all, ambient, cni, controllers, default, install, iptables, klog, model, monitoring, repair, spiffe, trustBundle, validation] and level can be one of [debug, info, warn, error, fatal, none] (default `default:info`) |
--log_rotate <string> | The path for the optional rotating log file (default ``) |
--log_rotate_max_age <int> | The maximum age in days of a log file beyond which the file is rotated (0 indicates no limit) (default `30`) |
--log_rotate_max_backups <int> | The maximum number of log file backups to keep before older files are deleted (0 indicates no limit) (default `1000`) |
--log_rotate_max_size <int> | The maximum size in megabytes of a log file beyond which the file is rotated (default `104857600`) |
--log_stacktrace_level <string> | Comma-separated minimum per-scope logging level at which stack traces are captured, in the form of <scope>:<level>,<scope:level>,... where scope can be one of [all, ambient, cni, controllers, default, install, iptables, klog, model, monitoring, repair, spiffe, trustBundle, validation] and level can be one of [debug, info, warn, error, fatal, none] (default `default:none`) |
--log_target <stringArray> | The set of paths where to output the log. This can be any path as well as the special values stdout and stderr (default `[stdout]`) |
--no-descriptions | disable completion descriptions |
install-cni version
Prints out build version information
install-cni version [flags]
| Flags | Shorthand | Description |
|---|---|---|
--ctrlz_address <string> | The IP Address to listen on for the ControlZ introspection facility. Use '*' to indicate all addresses. (default `localhost`) | |
--ctrlz_port <uint16> | The IP port to use for the ControlZ introspection facility (default `9876`) | |
--log_as_json | Whether to format output as JSON or in plain console-friendly format | |
--log_caller <string> | Comma-separated list of scopes for which to include caller information, scopes can be any of [all, ambient, cni, controllers, default, install, iptables, klog, model, monitoring, repair, spiffe, trustBundle, validation] (default ``) | |
--log_output_level <string> | Comma-separated minimum per-scope logging level of messages to output, in the form of <scope>:<level>,<scope>:<level>,... where scope can be one of [all, ambient, cni, controllers, default, install, iptables, klog, model, monitoring, repair, spiffe, trustBundle, validation] and level can be one of [debug, info, warn, error, fatal, none] (default `default:info`) | |
--log_rotate <string> | The path for the optional rotating log file (default ``) | |
--log_rotate_max_age <int> | The maximum age in days of a log file beyond which the file is rotated (0 indicates no limit) (default `30`) | |
--log_rotate_max_backups <int> | The maximum number of log file backups to keep before older files are deleted (0 indicates no limit) (default `1000`) | |
--log_rotate_max_size <int> | The maximum size in megabytes of a log file beyond which the file is rotated (default `104857600`) | |
--log_stacktrace_level <string> | Comma-separated minimum per-scope logging level at which stack traces are captured, in the form of <scope>:<level>,<scope:level>,... where scope can be one of [all, ambient, cni, controllers, default, install, iptables, klog, model, monitoring, repair, spiffe, trustBundle, validation] and level can be one of [debug, info, warn, error, fatal, none] (default `default:none`) | |
--log_target <stringArray> | The set of paths where to output the log. This can be any path as well as the special values stdout and stderr (default `[stdout]`) | |
--output <string> | -o | One of 'yaml' or 'json'. (default ``) |
--short | -s | Use --short=false to generate full version information |
Environment variables
These environment variables affect the behavior of theinstall-cni command. Please use with caution as these environment variables are experimental and can change anytime.| Variable Name | Type | Default Value | Description |
|---|---|---|---|
AMBIENT_ENABLED | Boolean | false | Whether ambient controller is enabled |
CA_TRUSTED_NODE_ACCOUNTS | String | | If set, the list of service accounts that are allowed to use node authentication for CSRs. Node authentication allows an identity to create CSRs on behalf of other identities, but only if there is a pod running on the same node with that identity. This is intended for use with node proxies. |
CERT_SIGNER_DOMAIN | String | | The cert signer domain info |
CHAINED_CNI_PLUGIN | Boolean | true | Whether to install CNI plugin as a chained or standalone |
CLUSTER_ID | String | Kubernetes | Defines the cluster and service registry that this Istiod instance belongs to |
CNI_CONF_NAME | String | | Name of the CNI configuration file |
CNI_EVENT_ADDRESS | String | /var/run/istio-cni/pluginevent.sock | The UDS server address which CNI plugin will forward ambient pod creation events to |
CNI_NETWORK_CONFIG | String | | CNI configuration template as a string |
CNI_NETWORK_CONFIG_FILE | String | | CNI config template as a file |
CNI_NET_DIR | String | /etc/cni/net.d | Directory on the host where CNI network plugins are installed |
DRY_RUN_FILE_PATH | String | | If provided, StdoutStubDependencies will write the input from stdin to the given file. |
ENABLE_AUTO_SNI | Boolean | true | If enabled, automatically set SNI when `DestinationRules` do not specify the same |
ENABLE_CA_SERVER | Boolean | true | If this is set to false, will not create CA server in istiod. |
ENABLE_DEBUG_ON_HTTP | Boolean | true | If this is set to false, the debug interface will not be enabled, recommended for production |
ENABLE_ENHANCED_RESOURCE_SCOPING | Boolean | false | If enabled, meshConfig.discoverySelectors will limit the CustomResource configurations(like Gateway,VirtualService,DestinationRule,Ingress, etc)that can be processed by pilot. This will also restrict the root-ca certificate distribution. |
ENABLE_EXTERNAL_NAME_ALIAS | Boolean | true | If enabled, ExternalName Services will be treated as simple aliases: anywhere where we would match the concrete service, we also match the ExternalName. In general, this mirrors Kubernetes behavior more closely. However, it means that policies (routes and DestinationRule) cannot be applied to the ExternalName service. If disabled, ExternalName behaves in fairly unexpected manner. Port matters, while it does not in Kubernetes. If it is a TCP port, all traffic on that port will be matched, which can have disastrous consequences. Additionally, the destination is seen as an opaque destination; even if it is another service in the mesh, policies such as mTLS and load balancing will not be used when connecting to it. |
ENABLE_HCM_INTERNAL_NETWORKS | Boolean | false | If enable, endpoints defined in mesh networks will be configured as internal addresses in Http Connection Manager |
ENABLE_LEADER_ELECTION | Boolean | true | If enabled (default), starts a leader election client and gains leadership before executing controllers. If false, it assumes that only one instance of istiod is running and skips leader election. |
ENABLE_MCS_AUTO_EXPORT | Boolean | false | If enabled, istiod will automatically generate Kubernetes Multi-Cluster Services (MCS) ServiceExport resources for every service in the mesh. Services defined to be cluster-local in MeshConfig are excluded. |
ENABLE_MCS_CLUSTER_LOCAL | Boolean | false | If enabled, istiod will treat the host `<svc>.<namespace>.svc.cluster.local` as defined by the Kubernetes Multi-Cluster Services (MCS) spec. In this mode, requests to `cluster.local` will be routed to only those endpoints residing within the same cluster as the client. Requires that both ENABLE_MCS_SERVICE_DISCOVERY and ENABLE_MCS_HOST also be enabled. |
ENABLE_MCS_HOST | Boolean | false | If enabled, istiod will configure a Kubernetes Multi-Cluster Services (MCS) host (<svc>.<namespace>.svc.clusterset.local) for each service exported (via ServiceExport) in at least one cluster. Clients must, however, be able to successfully lookup these DNS hosts. That means that either Istio DNS interception must be enabled or an MCS controller must be used. Requires that ENABLE_MCS_SERVICE_DISCOVERY also be enabled. |
ENABLE_MCS_SERVICE_DISCOVERY | Boolean | false | If enabled, istiod will enable Kubernetes Multi-Cluster Services (MCS) service discovery mode. In this mode, service endpoints in a cluster will only be discoverable within the same cluster unless explicitly exported via ServiceExport. |
ENABLE_MULTICLUSTER_HEADLESS | Boolean | true | If true, the DNS name table for a headless service will resolve to same-network endpoints in any cluster. |
ENABLE_NATIVE_SIDECARS | Boolean | false | If set, used Kubernetes native Sidecar container support. Requires SidecarContainer feature flag. |
ENABLE_OPTIMIZED_CONFIG_REBUILD | Boolean | true | If enabled, pilot will only rebuild config for resources that have changed |
ENABLE_OTEL_BUILTIN_RESOURCE_LABELS | Boolean | false | If enabled, envoy will send builtin labels(e.g. node_name) via OTel sink. |
ENABLE_SELECTOR_BASED_K8S_GATEWAY_POLICY | Boolean | true | If disabled, Gateway API gateways will ignore workloadSelector policies, onlyapplying policies that select the gateway with a targetRef. |
ENABLE_TLS_ON_SIDECAR_INGRESS | Boolean | false | If enabled, the TLS configuration on Sidecar.ingress will take effect |
ENABLE_VTPROTOBUF | Boolean | false | If true, will use optimized vtprotobuf based marshaling |
ENVOY_USER | String | istio-proxy | Envoy proxy username |
EXTERNAL_ISTIOD | Boolean | false | If this is set to true, one Istiod will control remote clusters including CA. |
HOST_PROBE_SNAT_IP | String | 169.254.7.127 | |
INBOUND_INTERCEPTION_MODE | String | | The mode used to redirect inbound connections to Envoy, either "REDIRECT" or "TPROXY" |
INBOUND_TPROXY_MARK | String | | |
INJECTION_WEBHOOK_CONFIG_NAME | String | istio-sidecar-injector | Name of the mutatingwebhookconfiguration to patch, if istioctl is not used. |
IPTABLES_TRACE_LOGGING | Boolean | false | When enable, all iptables actions will be logged. This requires NET_ADMIN privilege and has noisy logs; as a result, this is intended for debugging only |
ISTIOD_CUSTOM_HOST | String | | Custom host name of istiod that istiod signs the server cert. Multiple custom host names are supported, and multiple values are separated by commas. |
ISTIO_AGENT_ENABLE_WASM_REMOTE_LOAD_CONVERSION | Boolean | true | If enabled, Istio agent will intercept ECDS resource update, downloads Wasm module, and replaces Wasm module remote load with downloaded local module file. |
ISTIO_DELTA_XDS | Boolean | false | If enabled, pilot will only send the delta configs as opposed to the state of the world on a Resource Request. This feature uses the delta xds api, but does not currently send the actual deltas. |
ISTIO_DUAL_STACK | Boolean | false | If true, Istio will enable the Dual Stack feature. |
ISTIO_ENABLE_CONTROLLER_QUEUE_METRICS | Boolean | false | If enabled, publishes metrics for queue depth, latency and processing times. |
ISTIO_ENABLE_IPV4_OUTBOUND_LISTENER_FOR_IPV6_CLUSTERS | Boolean | false | If true, pilot will configure an additional IPv4 listener for outbound traffic in IPv6 only clusters, e.g. AWS EKS IPv6 only clusters. |
ISTIO_ENABLE_OPTIMIZED_SERVICE_PUSH | Boolean | true | If enabled, Istiod will not push changes on arbitrary annotation change. |
ISTIO_GPRC_MAXRECVMSGSIZE | Integer | 4194304 | Sets the max receive buffer size of gRPC stream in bytes. |
ISTIO_GPRC_MAXSTREAMS | Integer | 100000 | Sets the maximum number of concurrent grpc streams. |
ISTIO_KUBE_CLIENT_CONTENT_TYPE | String | protobuf | The content type to use for Kubernetes clients. Defaults to protobuf. Valid options: [protobuf, json] |
ISTIO_MULTIROOT_MESH | Boolean | false | If enabled, mesh will support certificates signed by more than one trustAnchor for ISTIO_MUTUAL mTLS |
ISTIO_OUTBOUND_IPV4_LOOPBACK_CIDR | String | 127.0.0.1/32 | IPv4 CIDR range used to identify outbound traffic on loopback interface intended for application container |
ISTIO_OUTBOUND_OWNER_GROUPS | String | * | Comma separated list of groups whose outgoing traffic is to be redirected to Envoy. A group can be specified either by name or by a numeric GID. The wildcard character "*" can be used to configure redirection of traffic from all groups. |
ISTIO_OUTBOUND_OWNER_GROUPS_EXCLUDE | String | | Comma separated list of groups whose outgoing traffic is to be excluded from redirection to Envoy. A group can be specified either by name or by a numeric GID. Only applies when traffic from all groups (i.e. "*") is being redirected to Envoy. |
ISTIO_WATCH_NAMESPACE | String | | If set, limit Kubernetes watches to a single namespace. Warning: only a single namespace can be set. |
ISTIO_WORKLOAD_ENTRY_VALIDATE_IDENTITY | Boolean | true | If enabled, will validate the identity of a workload matches the identity of the WorkloadEntry it is associating with for health checks and auto registration. This flag is added for backwards compatibility only and will be removed in future releases |
JWKS_RESOLVER_INSECURE_SKIP_VERIFY | Boolean | false | If enabled, istiod will skip verifying the certificate of the JWKS server. |
JWT_POLICY | String | third-party-jwt | The JWT validation policy. |
KUBECFG_FILE_NAME | String | ZZZ-istio-cni-kubeconfig | Name of the kubeconfig file which CNI plugin will use when interacting with API server |
KUBECONFIG_MODE | Integer | 384 | File mode of the kubeconfig file |
KUBE_CA_FILE | String | | CA file for kubeconfig. Defaults to the same as install-cni pod |
LABEL_CANONICAL_SERVICES_FOR_MESH_EXTERNAL_SERVICE_ENTRIES | Boolean | false | If enabled, metadata representing canonical services for ServiceEntry resources with a location of mesh_external will be populatedin the cluster metadata for those endpoints. |
LOCAL_CLUSTER_SECRET_WATCHER | Boolean | false | If enabled, the cluster secret watcher will watch the namespace of the external cluster instead of config cluster |
LOG_LEVEL | String | warn | Fallback value for log level in CNI config file, if not specified in helm template |
LOG_UDS_ADDRESS | String | /var/run/istio-cni/log.sock | The UDS server address which CNI plugin will copy log output to |
MCS_API_GROUP | String | multicluster.x-k8s.io | The group to be used for the Kubernetes Multi-Cluster Services (MCS) API. |
MCS_API_VERSION | String | v1alpha1 | The version to be used for the Kubernetes Multi-Cluster Services (MCS) API. |
METRIC_GRACEFUL_DELETION_INTERVAL | Time Duration | 5m0s | Metric expiry graceful deletion interval. No-op if METRIC_ROTATION_INTERVAL is disabled. |
METRIC_ROTATION_INTERVAL | Time Duration | 0s | Metric scope rotation interval, set to 0 to disable the metric scope rotation |
MONITORING_PORT | Integer | 15014 | HTTP port to serve prometheus metrics |
MOUNTED_CNI_NET_DIR | String | /host/etc/cni/net.d | Directory on the container where CNI networks are installed |
MUTEX_PROFILE_FRACTION | Integer | 1000 | If set to a non-zero value, enables mutex profiling a rate of 1/MUTEX_PROFILE_FRACTION events. For example, '1000' will record 0.1% of events. Set to 0 to disable entirely. |
NATIVE_METADATA_EXCHANGE | Boolean | true | If set, uses a native implementation of the HTTP metadata exchange filter |
NODE_NAME | String | | |
PERSIST_OLDEST_FIRST_HEURISTIC_FOR_VIRTUAL_SERVICE_HOST_MATCHING | Boolean | false | If enabled, istiod will persist the oldest first heuristic for subtly conflicting traffic policy selection(such as with overlapping wildcard hosts) |
PILOT_ALLOW_SIDECAR_SERVICE_INBOUND_LISTENER_MERGE | Boolean | false | If set, it allows creating inbound listeners for service ports and sidecar ingress listeners |
PILOT_ANALYSIS_INTERVAL | Time Duration | 10s | If analysis is enabled, pilot will run istio analyzers using this value as interval in seconds Istio Resources |
PILOT_CERT_PROVIDER | String | istiod | The provider of Pilot DNS certificate. |
PILOT_CONVERT_SIDECAR_SCOPE_CONCURRENCY | Integer | 1 | Used to adjust the concurrency of SidecarScope conversions. When istiod is deployed on a multi-core CPU server, increasing this value will help to use the CPU to accelerate configuration push, but it also means that istiod will consume more CPU resources. |
PILOT_DEBOUNCE_AFTER | Time Duration | 100ms | The delay added to config/registry events for debouncing. This will delay the push by at least this interval. If no change is detected within this period, the push will happen, otherwise we'll keep delaying until things settle, up to a max of PILOT_DEBOUNCE_MAX. |
PILOT_DEBOUNCE_MAX | Time Duration | 10s | The maximum amount of time to wait for events while debouncing. If events keep showing up with no breaks for this time, we'll trigger a push. |
PILOT_DISABLE_MX_ALPN | Boolean | false | If true, pilot will not put istio-peer-exchange ALPN into TLS handshake configuration. |
PILOT_DISTRIBUTION_HISTORY_RETENTION | Time Duration | 1m0s | If enabled, Pilot will keep track of old versions of distributed config for this duration. |
PILOT_DRAINING_LABEL | String | istio.io/draining | If not empty, endpoints with the label value present will be sent with status DRAINING. |
PILOT_ENABLE_ALPHA_GATEWAY_API | Boolean | false | If this is set to true, support for alpha APIs in the Kubernetes gateway-api (github.com/kubernetes-sigs/gateway-api) will be enabled. In addition to this being enabled, the gateway-api CRDs need to be installed. |
PILOT_ENABLE_ALPN_FILTER | Boolean | true | If true, pilot will add Istio ALPN filters, required for proper protocol sniffing. |
PILOT_ENABLE_AMBIENT_CONTROLLERS | Boolean | false | If enabled, controllers required for ambient will run. This is required to run ambient mesh. |
PILOT_ENABLE_ANALYSIS | Boolean | false | If enabled, pilot will run istio analyzers and write analysis errors to the Status field of any Istio Resources |
PILOT_ENABLE_CDS_CACHE | Boolean | true | If true, Pilot will cache CDS responses. Note: this depends on PILOT_ENABLE_XDS_CACHE. |
PILOT_ENABLE_CONFIG_DISTRIBUTION_TRACKING | Boolean | false | If enabled, Pilot will assign meaningful nonces to each Envoy configuration message, and allow users to interrogate which envoy has which config from the debug interface. |
PILOT_ENABLE_CROSS_CLUSTER_WORKLOAD_ENTRY | Boolean | true | If enabled, pilot will read WorkloadEntry from other clusters, selectable by Services in that cluster. |
PILOT_ENABLE_EDS_DEBOUNCE | Boolean | true | If enabled, Pilot will include EDS pushes in the push debouncing, configured by PILOT_DEBOUNCE_AFTER and PILOT_DEBOUNCE_MAX. EDS pushes may be delayed, but there will be fewer pushes. By default this is enabled |
PILOT_ENABLE_EDS_FOR_HEADLESS_SERVICES | Boolean | false | If enabled, for headless service in Kubernetes, pilot will send endpoints over EDS, allowing the sidecar to load balance among pods in the headless service. This feature should be enabled if applications access all services explicitly via a HTTP proxy port in the sidecar. |
PILOT_ENABLE_GATEWAY_API | Boolean | true | If this is set to true, support for Kubernetes gateway-api (github.com/kubernetes-sigs/gateway-api) will be enabled. In addition to this being enabled, the gateway-api CRDs need to be installed. |
PILOT_ENABLE_GATEWAY_API_DEPLOYMENT_CONTROLLER | Boolean | true | If this is set to true, gateway-api resources will automatically provision in cluster deployment, services, etc |
PILOT_ENABLE_GATEWAY_API_GATEWAYCLASS_CONTROLLER | Boolean | true | If this is set to true, istiod will create and manage its default GatewayClasses |
PILOT_ENABLE_GATEWAY_API_STATUS | Boolean | true | If this is set to true, gateway-api resources will have status written to them |
PILOT_ENABLE_HBONE | Boolean | false | If enabled, HBONE support can be configured for proxies. Note: proxies must opt in on a per-proxy basis with ENABLE_HBONE to actually get HBONE config, in addition to this flag. |
PILOT_ENABLE_HEADLESS_SERVICE_POD_LISTENERS | Boolean | true | If enabled, for a headless service/stateful set in Kubernetes, pilot will generate an outbound listener for each pod in a headless service. This feature should be disabled if headless services have a large number of pods. |
PILOT_ENABLE_K8S_SELECT_WORKLOAD_ENTRIES | Boolean | true | If enabled, Kubernetes services with selectors will select workload entries with matching labels. It is safe to disable it if you are quite sure you don't need this feature |
PILOT_ENABLE_METADATA_EXCHANGE | Boolean | true | If true, pilot will add metadata exchange filters, which will be consumed by telemetry filter. |
PILOT_ENABLE_MONGO_FILTER | Boolean | true | EnableMongoFilter enables injection of `envoy.filters.network.mongo_proxy` in the filter chain. |
PILOT_ENABLE_MYSQL_FILTER | Boolean | false | EnableMysqlFilter enables injection of `envoy.filters.network.mysql_proxy` in the filter chain. |
PILOT_ENABLE_PERSISTENT_SESSION_FILTER | Boolean | false | If enabled, Istiod sets up persistent session filter for listeners, if services have 'PILOT_PERSISTENT_SESSION_LABEL' set. |
PILOT_ENABLE_QUIC_LISTENERS | Boolean | false | If true, QUIC listeners will be generated wherever there are listeners terminating TLS on gateways if the gateway service exposes a UDP port with the same number (for example 443/TCP and 443/UDP) |
PILOT_ENABLE_RDS_CACHE | Boolean | true | If true, Pilot will cache RDS responses. Note: this depends on PILOT_ENABLE_XDS_CACHE. |
PILOT_ENABLE_REDIS_FILTER | Boolean | false | EnableRedisFilter enables injection of `envoy.filters.network.redis_proxy` in the filter chain. |
PILOT_ENABLE_ROUTE_COLLAPSE_OPTIMIZATION | Boolean | true | If true, Pilot will merge virtual hosts with the same routes into a single virtual host, as an optimization. |
PILOT_ENABLE_SERVICEENTRY_SELECT_PODS | Boolean | true | If enabled, service entries with selectors will select pods from the cluster. It is safe to disable it if you are quite sure you don't need this feature |
PILOT_ENABLE_STATUS | Boolean | false | If enabled, pilot will update the CRD Status field of all istio resources with reconciliation status. |
PILOT_ENABLE_TELEMETRY_LABEL | Boolean | true | If true, pilot will add telemetry related metadata to cluster and endpoint resources, which will be consumed by telemetry filter. |
PILOT_ENABLE_WORKLOAD_ENTRY_AUTOREGISTRATION | Boolean | true | Enables auto-registering WorkloadEntries based on associated WorkloadGroups upon XDS connection by the workload. |
PILOT_ENABLE_WORKLOAD_ENTRY_HEALTHCHECKS | Boolean | true | Enables automatic health checks of WorkloadEntries based on the config provided in the associated WorkloadGroup |
PILOT_ENABLE_XDS_CACHE | Boolean | true | If true, Pilot will cache XDS responses. |
PILOT_ENABLE_XDS_IDENTITY_CHECK | Boolean | true | If enabled, pilot will authorize XDS clients, to ensure they are acting only as namespaces they have permissions for. |
PILOT_ENDPOINT_TELEMETRY_LABEL | Boolean | true | If true, pilot will add telemetry related metadata to Endpoint resource, which will be consumed by telemetry filter. |
PILOT_ENVOY_FILTER_STATS | Boolean | false | If true, Pilot will collect metrics for envoy filter operations. |
PILOT_FILTER_GATEWAY_CLUSTER_CONFIG | Boolean | false | If enabled, Pilot will send only clusters that referenced in gateway virtual services attached to gateway |
PILOT_HTTP10 | Boolean | false | Enables the use of HTTP 1.0 in the outbound HTTP listeners, to support legacy applications. |
PILOT_INSECURE_MULTICLUSTER_KUBECONFIG_OPTIONS | String | | Comma separated list of potentially insecure kubeconfig authentication options that are allowed for multicluster authentication.Support values: all authProviders (`gcp`, `azure`, `exec`, `openstack`), `clientKey`, `clientCertificate`, `tokenFile`, and `exec`. |
PILOT_JWT_ENABLE_REMOTE_JWKS | String | false | Mode of fetching JWKs from JwksUri in RequestAuthentication. Supported value: istiod, false, hybrid, true, envoy. The client fetching JWKs is as following: istiod/false - Istiod; hybrid/true - Envoy and fallback to Istiod if JWKs server is external; envoy - Envoy. |
PILOT_JWT_PUB_KEY_REFRESH_INTERVAL | Time Duration | 20m0s | The interval for istiod to fetch the jwks_uri for the jwks public key. |
PILOT_MAX_REQUESTS_PER_SECOND | Floating-Point | 0 | Limits the number of incoming XDS requests per second. On larger machines this can be increased to handle more proxies concurrently. If set to 0 or unset, the max will be automatically determined based on the machine size |
PILOT_MULTI_NETWORK_DISCOVER_GATEWAY_API | Boolean | false | If true, Pilot will discover labeled Kubernetes gateway objects as multi-network gateways. |
PILOT_PERSISTENT_SESSION_HEADER_LABEL | String | istio.io/persistent-session-header | If not empty, services with this label will use header based persistent sessions |
PILOT_PERSISTENT_SESSION_LABEL | String | istio.io/persistent-session | If not empty, services with this label will use cookie based persistent sessions |
PILOT_PUSH_THROTTLE | Integer | 0 | Limits the number of concurrent pushes allowed. On larger machines this can be increased for faster pushes. If set to 0 or unset, the max will be automatically determined based on the machine size |
PILOT_REMOTE_CLUSTER_TIMEOUT | Time Duration | 30s | After this timeout expires, pilot can become ready without syncing data from clusters added via remote-secrets. Setting the timeout to 0 disables this behavior. |
PILOT_SCOPE_GATEWAY_TO_NAMESPACE | Boolean | false | If enabled, a gateway workload can only select gateway resources in the same namespace. Gateways with same selectors in different namespaces will not be applicable. |
PILOT_SEND_UNHEALTHY_ENDPOINTS | Boolean | false | If enabled, Pilot will include unhealthy endpoints in EDS pushes and even if they are sent Envoy does not use them for load balancing. To avoid, sending traffic to non ready endpoints, enabling this flag, disables panic threshold in Envoy i.e. Envoy does not load balance requests to unhealthy/non-ready hosts even if the percentage of healthy hosts fall below minimum health percentage(panic threshold). |
PILOT_SIDECAR_USE_REMOTE_ADDRESS | Boolean | false | UseRemoteAddress sets useRemoteAddress to true for sidecar outbound listeners. |
PILOT_SKIP_VALIDATE_TRUST_DOMAIN | Boolean | false | Skip validating the peer is from the same trust domain when mTLS is enabled in authentication policy |
PILOT_STATUS_BURST | Integer | 500 | If status is enabled, controls the Burst rate with which status will be updated. See https://godoc.org/k8s.io/client-go/rest#Config Burst |
PILOT_STATUS_MAX_WORKERS | Integer | 100 | The maximum number of workers Pilot will use to keep configuration status up to date. Smaller numbers will result in higher status latency, but larger numbers may impact CPU in high scale environments. |
PILOT_STATUS_QPS | Integer | 100 | If status is enabled, controls the QPS with which status will be updated. See https://godoc.org/k8s.io/client-go/rest#Config QPS |
PILOT_STATUS_UPDATE_INTERVAL | Time Duration | 500ms | Interval to update the XDS distribution status. |
PILOT_TRACE_SAMPLING | Floating-Point | 1 | Sets the mesh-wide trace sampling percentage. Should be 0.0 - 100.0. Precision to 0.01. Default is 1.0. |
PILOT_WORKLOAD_ENTRY_GRACE_PERIOD | Time Duration | 10s | The amount of time an auto-registered workload can remain disconnected from all Pilot instances before the associated WorkloadEntry is cleaned up. |
PILOT_XDS_CACHE_INDEX_CLEAR_INTERVAL | Time Duration | 5s | The interval for xds cache index clearing. |
PILOT_XDS_CACHE_SIZE | Integer | 60000 | The maximum number of cache entries for the XDS cache. |
PILOT_XDS_CACHE_STATS | Boolean | false | If true, Pilot will collect metrics for XDS cache efficiency. |
PILOT_XDS_SEND_TIMEOUT | Time Duration | 0s | The timeout to send the XDS configuration to proxies. After this timeout is reached, Pilot will discard that push. |
POD_NAME | String | | |
REPAIR_BROKEN_POD_LABEL_KEY | String | cni.istio.io/uninitialized | The key portion of the label which will be set by the race repair if label pods is true |
REPAIR_BROKEN_POD_LABEL_VALUE | String | true | The value portion of the label which will be set by the race repair if label pods is true |
REPAIR_DELETE_PODS | Boolean | false | Controller will delete pods when detecting pod broken by race condition |
REPAIR_ENABLED | Boolean | true | Whether to enable race condition repair or not |
REPAIR_FIELD_SELECTORS | String | | A set of field selectors in label=value format that will be added to the pod list filters |
REPAIR_INIT_CONTAINER_EXIT_CODE | Integer | 126 | Expected exit code for the init container when crash-looping because of CNI misconfiguration |
REPAIR_INIT_CONTAINER_NAME | String | istio-validation | The name of the istio init container (will crash-loop if CNI is not configured for the pod) |
REPAIR_INIT_CONTAINER_TERMINATION_MESSAGE | String | | The expected termination message for the init container when crash-looping because of CNI misconfiguration |
REPAIR_LABEL_PODS | Boolean | false | Controller will label pods when detecting pod broken by race condition |
REPAIR_LABEL_SELECTORS | String | | A set of label selectors in label=value format that will be added to the pod list filters |
REPAIR_NODE_NAME | String | | The name of the managed node (will manage all nodes if unset) |
REPAIR_SIDECAR_ANNOTATION | String | sidecar.istio.io/status | An annotation key that indicates this pod contains an istio sidecar. All pods without this annotation will be ignored.The value of the annotation is ignored. |
RESOLVE_HOSTNAME_GATEWAYS | Boolean | true | If true, hostnames in the LoadBalancer addresses of a Service will be resolved at the control plane for use in cross-network gateways. |
REVISION | String | | |
SHARED_MESH_CONFIG | String | | Additional config map to load for shared MeshConfig settings. The standard mesh config will take precedence. |
SKIP_TLS_VERIFY | Boolean | false | Whether to use insecure TLS in kubeconfig file |
STACKDRIVER_AUDIT_LOG | Boolean | false | If enabled, StackDriver audit logging will be enabled. |
SYSTEM_NAMESPACE | String | istio-system | pod's namespace |
TRUSTED_GATEWAY_CIDR | String | | If set, any connections from gateway to Istiod with this CIDR range are treated as trusted for using authentication mechanisms like XFCC. This can only be used when the network where Istiod and the authenticating gateways are running in a trusted/secure network |
UNSAFE_ENABLE_ADMIN_ENDPOINTS | Boolean | false | If this is set to true, dangerous admin endpoints will be exposed on the debug interface. Not recommended for production. |
UNSAFE_PILOT_ENABLE_DELTA_TEST | Boolean | false | If enabled, addition runtime tests for Delta XDS efficiency are added. These checks are extremely expensive, so this should be used only for testing, not production. |
UNSAFE_PILOT_ENABLE_RUNTIME_ASSERTIONS | Boolean | false | If enabled, addition runtime asserts will be performed. These checks are both expensive and panic on failure. As a result, this should be used only for testing. |
USE_CACERTS_FOR_SELF_SIGNED_CA | Boolean | false | If enabled, istiod will use a secret named cacerts to store its self-signed istio-generated root certificate. |
VALIDATION_WEBHOOK_CONFIG_NAME | String | istio-istio-system | If not empty, the controller will automatically patch validatingwebhookconfiguration when the CA certificate changes. Only works in kubernetes environment. |
VERIFY_CERTIFICATE_AT_CLIENT | Boolean | true | If enabled, certificates received by the proxy will be verified against the OS CA certificate bundle. |
VERIFY_SDS_CERTIFICATE | Boolean | true | If enabled, certificates fetched from SDS server will be verified before sending back to proxy. |
XDS_AUTH | Boolean | true | If true, will authenticate XDS clients. |
ZTUNNEL_UDS_ADDRESS | String | /var/run/ztunnel/ztunnel.sock | The UDS server address which ztunnel will connect to |
Exported metrics
| Metric Name | Type | Description |
|---|---|---|
controller_sync_errors_total | Sum | Total number of errorMetric syncing controllers. |
endpoint_no_pod | LastValue | Endpoints without an associated pod. |
istio_build | LastValue | Istio component build info |
istio_cni_install_ready | LastValue | Whether the CNI plugin installation is ready or not |
istio_cni_installs_total | Sum | Total number of CNI plugins installed by the Istio CNI installer |
istio_cni_repair_pods_repaired_total | Sum | Total number of pods repaired by repair controller |
nodeagent_reconcile_events_total | Sum | The total number of node agent reconcile events. |
pilot_conflict_inbound_listener | LastValue | Number of conflicting inbound listeners. |
pilot_conflict_outbound_listener_tcp_over_current_tcp | LastValue | Number of conflicting tcp listeners with current tcp listener. |
pilot_destrule_subsets | LastValue | Duplicate subsets across destination rules for same host |
pilot_dns_cluster_without_endpoints | LastValue | DNS clusters without endpoints caused by the endpoint field in STRICT_DNS type cluster is not set or the corresponding subset cannot select any endpoint |
pilot_duplicate_envoy_clusters | LastValue | Duplicate envoy clusters caused by service entries with same hostname |
pilot_eds_no_instances | LastValue | Number of clusters without instances. |
pilot_endpoint_not_ready | LastValue | Endpoint found in unready state. |
pilot_jwks_resolver_network_fetch_fail_total | Sum | Total number of failed network fetch by pilot jwks resolver |
pilot_jwks_resolver_network_fetch_success_total | Sum | Total number of successfully network fetch by pilot jwks resolver |
pilot_no_ip | LastValue | Pods not found in the endpoint table, possibly invalid. |
pilot_total_rejected_configs | Sum | Total number of configs that Pilot had to reject or ignore. |
pilot_virt_services | LastValue | Total virtual services known to pilot. |
pilot_vservice_dup_domain | LastValue | Virtual services with dup domains. |
provider_lookup_cluster_failures | Sum | Number of times a cluster lookup failed |
xds_cache_dependent_config_size | LastValue | Current size of dependent configs |
xds_cache_evictions | Sum | Total number of xds cache evictions. |
xds_cache_reads | Sum | Total number of xds cache xdsCacheReads. |
xds_cache_size | LastValue | Current size of xds cache |
ztunnel_connected | LastValue | number of connections to ztunnel |