## Clean up <del> Run `kind delete cluster --name psa-with-cluster-pss` and `kind delete cluster --name psa-wo-cluster-pss` to delete the clusters you created. </del> <ins> Now delete the clusters which you created above by running the following command: ```shell kind delete cluster --name psa-with-cluster-pss ``` ```shell kind delete cluster --name psa-wo-cluster-pss ``` </ins> ## {{% heading "whatsnext" %}}


## Clean up <del> Run `kind delete cluster --name psa-ns-level` to delete the cluster created. </del> <ins> Now delete the cluster which you created above by running the following command: ```shell kind delete cluster --name psa-ns-level ``` </ins> ## {{% heading "whatsnext" %}}


<ins> --- title: Common Expression Language in Kubernetes reviewers: - jpbetz - cici37 content_type: concept weight: 35 min-kubernetes-server-version: 1.25 --- <!-- overview --> The [Common Expression Language (CEL)](https://github.com/google/cel-go) is used in the Kubernetes API to declare validation rules, policy rules, and other constraints or conditions. CEL expressions are evaluated directly in the {{< glossary_tooltip text="API server" term_id="kube-apiserver" >}}, making CEL a convenient alternative to out-of-process mechanisms, such as webhooks, for many extensibility use cases. Your CEL expressions continue to execute so long as the control plane's API server component remains available. <!-- body --> ## Language overview The [CEL language](https://github.com/google/cel-spec/blob/master/doc/langdef.md) has a straightforward syntax that is similar to the expressions in C, C++, Java, JavaScript and Go. CEL was designed to be embedded into applications. Each CEL "program" is a single expression that evaluates to a single value. CEL expressions are typically short "one-liners" that inline well into the string fields of Kubernetes API resources. Inputs to a CEL program are "variables". Each Kubernetes API field that contains CEL declares in the API documentation which variables are available to use for that field. For example, in the `x-kubernetes-validations[i].rules` field of CustomResourceDefinitions, the `self` and `oldSelf` variables are available and refer to the previous and current state of the custom resource data to be validated by the CEL expression. Other Kubernetes API fields may declare different variables. See the API documentation of the API fields to learn which variables are available for that field. Example CEL expressions: {{< table caption="Examples of CEL expressions and the purpose of each" >}} | Rule                                                                               | Purpose                                                                           | |------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------| | `self.minReplicas <= self.replicas && self.replicas <= self.maxReplicas`           | Validate that the three fields defining replicas are ordered appropriately        | | `'Available' in self.stateCounts`                                                  | Validate that an entry with the 'Available' key exists in a map                   | | `(self.list1.size() == 0) != (self.list2.size() == 0)`                             | Validate that one of two lists is non-empty, but not both                         | | `self.envars.filter(e, e.name = 'MY_ENV').all(e, e.value.matches('^[a-zA-Z]*$')`   | Validate the 'value' field of a listMap entry where key field 'name' is 'MY_ENV'  | | `has(self.expired) && self.created + self.ttl < self.expired`                      | Validate that 'expired' date is after a 'create' date plus a 'ttl' duration       | | `self.health.startsWith('ok')`                                                     | Validate a 'health' string field has the prefix 'ok'                              | | `self.widgets.exists(w, w.key == 'x' && w.foo < 10)`                               | Validate that the 'foo' property of a listMap item with a key 'x' is less than 10 | | `type(self) == string ? self == '99%' : self == 42`                                | Validate an int-or-string field for both the the int and string cases             | | `self.metadata.name == 'singleton'`                                                | Validate that an object's name matches a specific value (making it a singleton)   | | `self.set1.all(e, !(e in self.set2))`                                              | Validate that two listSets are disjoint                                           | | `self.names.size() == self.details.size() && self.names.all(n, n in self.details)` | Validate the 'details' map is keyed by the items in the 'names' listSet           | ## CEL community libraries Kubernetes CEL expressions have access to the following CEL community libraries: - CEL standard functions, defined in the [list of standard definitions](https://github.com/google/cel-spec/blob/master/doc/langdef.md#list-of-standard-definitions) - CEL standard [macros](https://github.com/google/cel-spec/blob/v0.7.0/doc/langdef.md#macros) - CEL [extended string function library](https://pkg.go.dev/github.com/google/cel-go/ext#Strings) ## Kubernetes CEL libraries In additional to the CEL community libraries, Kubernetes includes CEL libraries that are available everywhere CEL is used in Kubernetes. ### Kubernetes list library The list library includes `indexOf` and `lastIndexOf`, which work similar to the strings functions of the same names. These functions either the first or last positional index of the provided element in the list. The list library also includes `min`, `max` and `sum`. Sum is supported on all number types as well as the duration type. Min and max are supported on all comparable types. `isSorted` is also provided as a convenience function and is supported on all comparable types. Examples: {{< table caption="Examples of CEL expressions using list library functions" >}} | CEL Expression                                                                     | Purpose                                                   | |------------------------------------------------------------------------------------|-----------------------------------------------------------| | `names.isSorted()`                                                                 | Verify that a list of names is kept in alphabetical order | | `items.map(x, x.weight).sum() == 1.0`                                              | Verify that the "weights" of a list of objects sum to 1.0 | | `lowPriorities.map(x, x.priority).max() < highPriorities.map(x, x.priority).min()` | Verify that two sets of priorities do not overlap         | | `names.indexOf('should-be-first') == 1`                                            | Require that the first name in a list if a specific value | See the [Kubernetes List Library](https://pkg.go.dev/k8s.io/apiextensions-apiserver/pkg/apiserver/schema/cel/library#Lists) godoc for more information. ### Kubernetes regex library In addition to the `matches` function provided by the CEL standard library, the regex library provides `find` and `findAll`, enabling a much wider range of regex operations. Examples: {{< table caption="Examples of CEL expressions using regex library functions" >}} | CEL Expression                                              | Purpose                                                  | |-------------------------------------------------------------|----------------------------------------------------------| | `"abc 123".find('[0-9]*')`                                  | Find the first number in a string                        | | `"1, 2, 3, 4".findAll('[0-9]*').map(x, int(x)).sum() < 100` | Verify that the numbers in a string sum to less than 100 | See the [Kubernetes regex library](https://pkg.go.dev/k8s.io/apiextensions-apiserver/pkg/apiserver/schema/cel/library#Regex) godoc for more information. ### Kubernetes URL library To make it easier and safer to process URLs, the following functions have been added: - `isURL(string)` checks if a string is a valid URL according to the [Go's net/url](https://pkg.go.dev/net/url#URL) package. The string must be an absolute URL. - `url(string) URL` converts a string to a URL or results in an error if the string is not a valid URL. Once parsed via the `url` function, the resulting URL object has `getScheme`, `getHost`, `getHostname`, `getPort`, `getEscapedPath` and `getQuery` accessors. Examples: {{< table caption="Examples of CEL expressions using URL library functions" >}} | CEL Expression                                                  | Purpose                                        | |-----------------------------------------------------------------|------------------------------------------------| | `url('https://example.com:80/').getHost()`                      | Get the 'example.com:80' host part of the URL. | | `url('https://example.com/path with spaces/').getEscapedPath()` | Returns '/path%20with%20spaces/'               | See the [Kubernetes URL library](https://pkg.go.dev/k8s.io/apiextensions-apiserver/pkg/apiserver/schema/cel/library#URLs) godoc for more information. ## Type checking CEL is a [gradually typed language](https://github.com/google/cel-spec/blob/master/doc/langdef.md#gradual-type-checking). Some Kubernetes API fields contain fully type checked CEL expressions. For example, [CustomResourceDefinitions Validation Rules](/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/#validation-rules) are fully type checked. Some Kubernetes API fields contain partially type checked CEL expressions. A partially type checked expression is an experessions where some of the variables are statically typed but others are dynamically typed. For example, in the CEL expressions of [ValidatingAdmissionPolicies](/docs/reference/access-authn-authz/validating-admission-policy/) the `request` variable is typed, but the `object` variable is dynamically typed. As a result, an expression containing `request.namex` would fail type checking because the `namex` field is not defined. However, `object.namex` would pass type checking even when the `namex` field is not defined for the resource kinds that `object` refers to, because `object` is dynamically typed. The `has()` macro in CEL may be used in CEL expressions to check if a field of a dynamically typed variable is accessable before attempting to access the field's value. For example: ```cel has(object.namex) ? object.namex == 'special' : request.name == 'special' ``` ## Type system integration {{< table caption="Table showing the relationship between OpenAPIv3 types and CEL types" >}} | OpenAPIv3 type                                     | CEL type                                                                                | |----------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------| | 'object' with Properties                           | object / "message type" (`type(<object>)` evaluates to `selfType<uniqueNumber>.path.to.object.from.self`                     | | 'object' with AdditionalProperties                 | map                                                                                | | 'object' with x-kubernetes-embedded-type           | object / "message type", 'apiVersion', 'kind', 'metadata.name' and 'metadata.generateName' are implicitly included in schema | | 'object' with x-kubernetes-preserve-unknown-fields | object / "message type", unknown fields are NOT accessible in CEL expression                                                 | | x-kubernetes-int-or-string                         | union of int or string,  `self.intOrString < 100 || self.intOrString == '50%'` evaluates to true for both `50` and `"50%"` | | 'array                                             | list                                                                                | | 'array' with x-kubernetes-list-type=map            | list with map based Equality & unique key guarantees                                                                         | | 'array' with x-kubernetes-list-type=set            | list with set based Equality & unique entry guarantees                                                                       | | 'boolean'                                          | boolean                                                                                | | 'number' (all formats)                             | double                                                                                | | 'integer' (all formats)                            | int (64)                                                                                | | _no equivalent_                                    | uint (64)                                                                                | | 'null'                                             | null_type                                                                                | | 'string'                                           | string                                                                                | | 'string' with format=byte (base64 encoded)         | bytes                                                                                | | 'string' with format=date                          | timestamp (google.protobuf.Timestamp)                                                                                | | 'string' with format=datetime                      | timestamp (google.protobuf.Timestamp)                                                                                | | 'string' with format=duration                      | duration (google.protobuf.Duration)                                                                                | Also see: [CEL types](https://github.com/google/cel-spec/blob/v0.6.0/doc/langdef.md#values), [OpenAPI types](https://swagger.io/specification/#data-types), [Kubernetes Structural Schemas](/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/#specifying-a-structural-schema). Equality comparison for arrays with `x-kubernetes-list-type` of `set` or `map` ignores element order. For example `[1, 2] == [2, 1]` if the arrays represent Kubernetes `set` values. Concatenation on arrays with `x-kubernetes-list-type` use the semantics of the list type: - `set`: `X + Y` performs a union where the array positions of all elements in `X` are preserved and non-intersecting elements in `Y` are appended, retaining their partial order. - `map`: `X + Y` performs a merge where the array positions of all keys in `X` are preserved but the values are overwritten by values in `Y` when the key sets of `X` and `Y` intersect. Elements in `Y` with non-intersecting keys are appended, retaining their partial order. ## Escaping Only Kubernetes resource property names of the form `[a-zA-Z_.-/][a-zA-Z0-9_.-/]*` are accessible from CEL. Accessible property names are escaped according to the following rules when accessed in the expression: {{< table caption="Table of CEL identifier escaping rules" >}} | escape sequence   | property name equivalent                                                                     | |-------------------|----------------------------------------------------------------------------------------------| | `__underscores__` | `__`                                                                                | | `__dot__`         | `.`                                                                                | | `__dash__`        | `-`                                                                                | | `__slash__`       | `/`                                                                                | | `__{keyword}__`   | [CEL **RESERVED** keyword](https://github.com/google/cel-spec/blob/v0.6.0/doc/langdef.md#syntax) | When you escape any of CEL's **RESERVED** keywords you need to match the exact property name use the underscore escaping (for example, `int` in the word `sprint` would not be escaped and nor would it need to be). Examples on escaping: {{< table caption="Examples escaped CEL identifiers" >}} | property name | rule with escaped property name   | |---------------|-----------------------------------| | `namespace`   | `self.__namespace__ > 0`          | | `x-prop`      | `self.x__dash__prop > 0`          | | `redact__d`   | `self.redact__underscores__d > 0` | | `string`      | `self.startsWith('kube')`         | ## Resource constraints CEL is non-Turing complete and offers a variety of production safety controls to limit execution time. CEL's _resource constraint_ features provide feedback to developers about expression complexity and help protect the API server from excessive resource consumption during evaluation. CEL's resource constraint features are used to prevent CEL evaluation from consuming excessive API server resources. A key element of the resource constraint features is a _cost unit_ that CEL defines as a way of tracking CPU utilization. Cost units are independant of system load and hardware. Cost units are also deterministic; for any given CEL expression and input data, evaluation of the expression by the CEL interpreter will always result in the same cost. Many of CEL's core operations have fixed costs. The simplest operations, such as comparisons (e.g. `<`) have a cost of 1. Some have a higher fixed cost, for example list literal declarations have a fixed base cost of 40 cost units. Calls to functions implemented in native code approximate cost based on the time complexity of the operation. For example: operations that use regular expressions, such as `match` and `find`, are estimated using an approximated cost of `length(regexString)*length(inputString)`. The approximated cost reflects the worst case time complexity of Go's RE2 implementation. ### Runtime cost budget All CEL expressions evaluated by Kubernetes are constrained by a runtime cost budget. The runtime cost budget is an estimate of actual CPU utilization computed by incrementing a cost unit counter while interpreting a CEL expression. If the CEL interpreter executes too many instructions, the runtime cost budget will be exceeded, execution of the expressions will be halted, and an error will result. Some Kubernetes resources define an additional runtime cost budget that bounds the execution of multiple expressions. If the sum total of the cost of expressions exceed the budget, execution of the expressions will be halted, and an error will result. For example the validation of a custom resource has a _per-validation_ runtime cost budget for all [Validation Rules](https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/#validation-rules) evaluated to validate the custom resource. ### Estimated cost limits For some Kubernetes resources, the API server may also check if worst case estimated running time of CEL expressions would be prohibitively expensive to execute. If so, the API server prevent the CEL expression from being written to API resources by rejecting create or update operations containing the CEL expression to the API resources. This feature offers a stronger assurance that CEL expressions written to the API resource will be evaluate at runtime without exceeding the runtime cost budget. </ins>  No newline at end of file


Different kinds of object can also have different `.status`; again, the API reference pages detail the structure of that `.status` field, and its content for each different type of object. <ins> ## Server side field validation Starting with Kubernetes v1.25, the API server offers server side [field validation](/docs/reference/using-api/api-concepts/#field-validation) that detects unrecognized or duplicate fields in an object. It provides all the functionality of `kubectl --validate` on the server side. The `kubectl` tool uses the `--validate` flag to set the level of field validation. It accepts the values `ignore`, `warn`, and `strict` while also accepting the values `true` (equivalent to `strict`) and `false` (equivalent to `ignore`). The default validation setting for `kubectl` is `--validate=true`. `Strict` : Strict field validation, errors on validation failure `Warn` : Field validation is performed, but errors are exposed as warnings rather than failing the request `Ignore` : No server side field validation is performed When `kubectl` cannot connect to an API server that supports field validation it will fall back to using client-side validation. Kubernetes 1.27 and later versions always offer field validation; older Kubernetes releases might not. If your cluster is older than v1.27, check the documentation for your version of Kubernetes. </ins> ## {{% heading "whatsnext" %}} If you're new to Kubernetes, read more about the following:


Kubernetes provides great primitives for deploying applications to a cluster: it can be as simple as `kubectl create -f app.yaml`. Deploy apps across multiple clusters has never been that simple. How should app workloads be distributed? Should the app resources be replicated into all clusters, replicated into selected clusters, or partitioned into clusters? How is  access to the clusters managed? What happens if some of the resources that a user wants to distribute pre-exist, in some or all of the clusters, in some form? <del> In SIG Multicluster, our journey has revealed that there are multiple possible models to solve these problems and there probably is no single best-fit, all-scenario solution. [Federation](/docs/concepts/cluster-administration/federation/), however, is the single biggest Kubernetes open source sub-project, and has seen the maximum interest and contribution from the community in this problem space. The project initially reused the Kubernetes API to do away with any added usage complexity for an existing Kubernetes user. This approach was not viable, because of the problems summarised below: </del> <ins> In SIG Multicluster, our journey has revealed that there are multiple possible models to solve these problems and there probably is no single best-fit, all-scenario solution. [Kubernetes Cluster Federation (KubeFed for short)](https://github.com/kubernetes-sigs/kubefed), however, is the single biggest Kubernetes open source sub-project, and has seen the maximum interest and contribution from the community in this problem space. The project initially reused the Kubernetes API to do away with any added usage complexity for an existing Kubernetes user. This approach was not viable, because of the problems summarised below: </ins> * Difficulties in re-implementing the Kubernetes API at the cluster level, as federation-specific extensions were stored in annotations. * Limited flexibility in federated types, placement and reconciliation, due to 1:1 emulation of the Kubernetes API.


## {{% heading "prerequisites" %}} <del> These instructions are for Kubernetes {{< skew currentVersion >}}. If you want to check the integrity of components for a different version of Kubernetes, check the documentation for that Kubernetes release. </del> You will need to have the following tools installed: - `cosign` ([install guide](https://docs.sigstore.dev/cosign/installation/)) - `curl` (often provided by your operating system) <ins> - `jq` ([download jq](https://stedolan.github.io/jq/download/)) </ins> ## Verifying binary signatures


done ``` <del> Then verify the blob by using `cosign`: </del> <ins> Then verify the blob by using `cosign verify-blob`: </ins> ```shell <del> cosign verify-blob "$BINARY" --signature "$BINARY".sig --certificate "$BINARY".cert </del> <ins> cosign verify-blob "$BINARY"  --signature "$BINARY".sig  --certificate "$BINARY".cert  --certificate-identity krel-staging@k8s-releng-prod.iam.gserviceaccount.com  --certificate-oidc-issuer https://accounts.google.com </ins> ``` <del> cosign v1.9.0 is required to be able to use the `--certificate` flag. Please use `--cert` for older versions of cosign. </del> {{< note >}} <del> To learn more about keyless signing, please refer to [Keyless Signatures](https://github.com/sigstore/cosign/blob/main/KEYLESS.md#keyless-signatures). </del> <ins> Cosign 2.0 requires the `--certificate-identity` and `--certificate-oidc-issuer` options. To learn more about keyless signing, please refer to [Keyless Signatures](https://docs.sigstore.dev/cosign/keyless). Previous versions of Cosign required that you set `COSIGN_EXPERIMENTAL=1`. For additional information, plase refer to the [sigstore Blog](https://blog.sigstore.dev/cosign-2-0-released/) </ins> {{< /note >}} ## Verifying image signatures


For a complete list of images that are signed please refer to [Releases](/releases/download/). <del> Let's pick one image from this list and verify its signature using </del> <ins> Pick one image from this list and verify its signature using </ins> the `cosign verify` command: ```shell <del> COSIGN_EXPERIMENTAL=1 cosign verify registry.k8s.io/kube-apiserver-amd64:v{{< skew currentPatchVersion >}} </del> <ins> cosign verify registry.k8s.io/kube-apiserver-amd64:v{{< skew currentPatchVersion >}}  --certificate-identity krel-trust@k8s-releng-prod.iam.gserviceaccount.com  --certificate-oidc-issuer https://accounts.google.com  | jq . </ins> ``` <del> {{< note >}} `COSIGN_EXPERIMENTAL=1` is used to allow verification of images signed in `KEYLESS` mode. To learn more about keyless signing, please refer to [Keyless Signatures](https://github.com/sigstore/cosign/blob/main/KEYLESS.md#keyless-signatures) . {{< /note >}} </del> ### Verifying images for all control plane components <del> To verify all signed control plane images, please run this command: </del> <ins> To verify all signed control plane images for the latest stable version (v{{< skew currentPatchVersion >}}), please run the following commands: </ins> ```shell <del> curl -Ls "https://sbom.k8s.io/$(curl -Ls https://dl.k8s.io/release/stable.txt)/release" | grep "SPDXID: SPDXRef-Package-registry.k8s.io" |  grep -v sha256 | cut -d- -f3- | sed 's/-///' | sed 's/-v1/:v1/' > images.txt </del> <ins> curl -Ls "https://sbom.k8s.io/$(curl -Ls https://dl.k8s.io/release/stable.txt)/release"  | grep "SPDXID: SPDXRef-Package-registry.k8s.io"  | grep -v sha256 | cut -d- -f3- | sed 's/-///' | sed 's/-v1/:v1/'  | sort > images.txt </ins> input=images.txt while IFS= read -r image do <del> COSIGN_EXPERIMENTAL=1 cosign verify "$image" </del> <ins> cosign verify "$image"  --certificate-identity krel-trust@k8s-releng-prod.iam.gserviceaccount.com  --certificate-oidc-issuer https://accounts.google.com  | jq . </ins> done < "$input" ``` <del> Once you have verified an image, specify that image by its digest in your Pod manifests as per this example: `registry-url/image-name@sha256:45b23dee08af5e43a7fea6c4cf9c25ccf269ee113168c19722f87876677c5cb2` . </del> <ins> Once you have verified an image, you can specify the image by its digest in your Pod manifests as per this example: ```console registry-url/image-name@sha256:45b23dee08af5e43a7fea6c4cf9c25ccf269ee113168c19722f87876677c5cb2 ``` </ins> For more information, please refer <del> to [Image Pull Policy](/docs/concepts/containers/images/#image-pull-policy) </del> <ins> to the [Image Pull Policy](/docs/concepts/containers/images/#image-pull-policy) </ins> section. ## Verifying Image Signatures with Admission Controller <del> For non-control plane images ( e.g. [conformance image](https://github.com/kubernetes/kubernetes/blob/master/test/conformance/image/README.md)) , signatures can also be verified at deploy time using </del> <ins> For non-control plane images (for example [conformance image](https://github.com/kubernetes/kubernetes/blob/master/test/conformance/image/README.md)), signatures can also be verified at deploy time using </ins> [sigstore policy-controller](https://docs.sigstore.dev/policy-controller/overview) <del> admission controller. To get started with `policy-controller` here are a few helpful resources: </del> <ins> admission controller. Here are some helpful resources to get started with `policy-controller`: </ins> - [Installation](https://github.com/sigstore/helm-charts/tree/main/charts/policy-controller) - [Configuration Options](https://github.com/sigstore/policy-controller/tree/main/config)


Install CNI plugins (required for most pod network): ```bash <del> CNI_PLUGINS_VERSION="v1.1.1" </del> <ins> CNI_PLUGINS_VERSION="v1.2.0" </ins> ARCH="amd64" DEST="/opt/cni/bin" sudo mkdir -p "$DEST"


Install crictl (required for kubeadm / Kubelet Container Runtime Interface (CRI)) ```bash <del> CRICTL_VERSION="v1.25.0" </del> <ins> CRICTL_VERSION="v1.26.0" </ins> ARCH="amd64" curl -L "https://github.com/kubernetes-sigs/cri-tools/releases/download/${CRICTL_VERSION}/crictl-${CRICTL_VERSION}-linux-${ARCH}.tar.gz" | sudo tar -C $DOWNLOAD_DIR -xz ```


## {{% heading "whatsnext" %}} <del> * [Using kubeadm to Create a Cluster](/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/) </del> <ins> * [Using kubeadm to Create a Cluster](/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/) </ins>  No newline at end of file


{{< glossary_definition term_id="kube-controller-manager" length="all" >}} <del> Some types of these controllers are: </del> <ins> There are many different types of controllers. Some examples of them are: </ins> * Node controller: Responsible for noticing and responding when nodes go down. * Job controller: Watches for Job objects that represent one-off tasks, then creates


* EndpointSlice controller: Populates EndpointSlice objects (to provide a link between Services and Pods). * ServiceAccount controller: Create default ServiceAccounts for new namespaces. <ins> The above is not an exhaustive list. </ins> ### cloud-controller-manager {{< glossary_definition term_id="cloud-controller-manager" length="short" >}}


* Etcd's official [documentation](https://etcd.io/docs/). * Several [container runtimes](/docs/setup/production-environment/container-runtimes/) in Kubernetes. * Integrating with cloud providers using [cloud-controller-manager](/docs/concepts/architecture/cloud-controller/). <del> * [kubectl](/docs/reference/generated/kubectl/kubectl-commands) commands. </del> <ins> * [kubectl](/docs/reference/generated/kubectl/kubectl-commands) commands. </ins>  No newline at end of file


each kube-apiserver. You can inspect Leases owned by each kube-apiserver by checking for lease objects in the `kube-system` namespace <del> with the name `kube-apiserver-<sha256-hash>`. Alternatively you can use the label selector `k8s.io/component=kube-apiserver`: </del> <ins> with the name `kube-apiserver-<sha256-hash>`. Alternatively you can use the label selector `apiserver.kubernetes.io/identity=kube-apiserver`: </ins> ```shell <del> kubectl -n kube-system get lease -l k8s.io/component=kube-apiserver </del> <ins> kubectl -n kube-system get lease -l apiserver.kubernetes.io/identity=kube-apiserver </ins> ``` ``` NAME                                        HOLDER                                                                           AGE <del> kube-apiserver-c4vwjftbvpc5os2vvzle4qg27a   kube-apiserver-c4vwjftbvpc5os2vvzle4qg27a_9cbf54e5-1136-44bd-8f9a-1dcd15c346b4   5m33s kube-apiserver-dz2dqprdpsgnm756t5rnov7yka   kube-apiserver-dz2dqprdpsgnm756t5rnov7yka_84f2a85d-37c1-4b14-b6b9-603e62e4896f   4m23s kube-apiserver-fyloo45sdenffw2ugwaz3likua   kube-apiserver-fyloo45sdenffw2ugwaz3likua_c5ffa286-8a9a-45d4-91e7-61118ed58d2e   4m43s </del> <ins> apiserver-07a5ea9b9b072c4a5f3d1c3702        apiserver-07a5ea9b9b072c4a5f3d1c3702_0c8914f7-0f35-440e-8676-7844977d3a05        5m33s apiserver-7be9e061c59d368b3ddaf1376e        apiserver-7be9e061c59d368b3ddaf1376e_84f2a85d-37c1-4b14-b6b9-603e62e4896f        4m23s apiserver-1dfef752bcb36637d2763d1868        apiserver-1dfef752bcb36637d2763d1868_c5ffa286-8a9a-45d4-91e7-61118ed58d2e        4m43s </ins> ``` The SHA256 hash used in the lease name is based on the OS hostname as seen by that API server. Each kube-apiserver should be


hostname used by kube-apisever by checking the value of the `kubernetes.io/hostname` label: ```shell <del> kubectl -n kube-system get lease kube-apiserver-c4vwjftbvpc5os2vvzle4qg27a -o yaml </del> <ins> kubectl -n kube-system get lease apiserver-07a5ea9b9b072c4a5f3d1c3702 -o yaml </ins> ``` ```yaml apiVersion: coordination.k8s.io/v1 kind: Lease metadata: <del> creationTimestamp: "2022-11-30T15:37:15Z" </del> <ins> creationTimestamp: "2023-07-02T13:16:48Z" </ins> labels: <del> k8s.io/component: kube-apiserver kubernetes.io/hostname: kind-control-plane name: kube-apiserver-c4vwjftbvpc5os2vvzle4qg27a </del> <ins> apiserver.kubernetes.io/identity: kube-apiserver kubernetes.io/hostname: master-1 name: apiserver-07a5ea9b9b072c4a5f3d1c3702 </ins> namespace: kube-system <del> resourceVersion: "18171" uid: d6c68901-4ec5-4385-b1ef-2d783738da6c </del> <ins> resourceVersion: "334899" uid: 90870ab5-1ba9-4523-b215-e4d4e662acb1 </ins> spec: <del> holderIdentity: kube-apiserver-c4vwjftbvpc5os2vvzle4qg27a_9cbf54e5-1136-44bd-8f9a-1dcd15c346b4 </del> <ins> holderIdentity: apiserver-07a5ea9b9b072c4a5f3d1c3702_0c8914f7-0f35-440e-8676-7844977d3a05 </ins> leaseDurationSeconds: 3600 <del> renewTime: "2022-11-30T18:04:27.912073Z" </del> <ins> renewTime: "2023-07-04T21:58:48.065888Z" </ins> ``` Expired leases from kube-apiservers that no longer exist are garbage collected by new kube-apiservers after 1 hour.


You should now be able to curl the nginx Service on `<CLUSTER-IP>:<PORT>` from any node in your cluster. Note that the Service IP is completely virtual, it never hits the wire. If you're curious about how this works you can read more <del> about the [service proxy](/docs/concepts/services-networking/service/#virtual-ips-and-service-proxies). </del> <ins> about the [service proxy](/docs/reference/networking/virtual-ips/). </ins> ## Accessing the Service


| Session affinity | Ensures that connections from a particular client are passed to the same Pod each time. | Windows Server 2022 | Set `service.spec.sessionAffinity` to "ClientIP" | | Direct Server Return (DSR) | Load balancing mode where the IP address fixups and the LBNAT occurs at the container vSwitch port directly; service traffic arrives with the source IP set as the originating pod IP. | Windows Server 2019 | Set the following flags in kube-proxy: `--feature-gates="WinDSR=true" --enable-dsr=true` | | Preserve-Destination | Skips DNAT of service traffic, thereby preserving the virtual IP of the target service in packets reaching the backend Pod. Also disables node-node forwarding. | Windows Server, version 1903 | Set `"preserve-destination": "true"` in service annotations and enable DSR in kube-proxy. | <del> | IPv4/IPv6 dual-stack networking | Native IPv4-to-IPv4 in parallel with IPv6-to-IPv6 communications to, from, and within a cluster | Windows Server 2019 | See [IPv4/IPv6 dual-stack](#ipv4ipv6-dual-stack) | </del> <ins> | IPv4/IPv6 dual-stack networking | Native IPv4-to-IPv4 in parallel with IPv6-to-IPv6 communications to, from, and within a cluster | Windows Server 2019 | See [IPv4/IPv6 dual-stack](/docs/concepts/services-networking/dual-stack/#windows-support) | </ins> | Client IP preservation | Ensures that source IP of incoming ingress traffic gets preserved. Also disables node-node forwarding. |  Windows Server 2019  | Set `service.spec.externalTrafficPolicy` to "Local" and enable DSR in kube-proxy | {{< /table >}}


I componenti del Control Plane sono responsabili di tutte le decisioni globali sul cluster (ad esempio, lo scheduling) oltre che a rilevare e rispondere agli eventi del cluster (ad esempio, l'avvio di un nuovo {{< glossary_tooltip text="pod" term_id="pod">}} quando il valore `replicas` di un deployment non è soddisfatto). I componenti della Control Plane possono essere eseguiti su qualsiasi nodo del cluster stesso. Solitamente, per semplicità, gli script di installazione tendono a eseguire tutti i componenti della Control Plane sulla stessa macchina, separando la Control Plane dai workload dell'utente. <del> Vedi [creare un cluster in High-Availability](/docs/admin/high-availability/) per un esempio di un'installazione multi-master. </del> <ins> Vedi [creare un cluster in High-Availability](/docs/setup/production-environment/tools/kubeadm/high-availability/) per un esempio di un'installazione multi-master. </ins> ### kube-apiserver


body.cid-community #navigation-items { padding: 0.25em; <del> width: 100vw; </del> <ins> width: 100%; </ins> max-width: initial; margin-top: 2.5em;


body.cid-community #gallery { display: flex; <del> max-width: 100vw; </del> <ins> max-width: 100%; </ins> gap: 0.75rem; justify-content: center; margin-left: auto;


body.cid-community .community-section#events { <del> width: 100vw; </del> <ins> width: 100%; </ins> max-width: initial; margin-bottom: 0;


} body.cid-community .community-section#values { <del> width: 100vw; </del> <ins> width: 100%; </ins> max-width: initial; background-image: url('/images/community/event-bg.jpg'); color: #fff;


} body.cid-community .community-section#meetups { <del> width: 100vw; </del> <ins> width: 100%; </ins> max-width: initial; margin-top: 0;


background-repeat: no-repeat, repeat; background-size: auto 100%, cover; color: #fff; <del> width: 100vw; </del> /* fallback in case calc() fails */ padding: 5vw; padding-bottom: 1em;


} body.cid-community #videos { <del> width: 100vw; </del> <ins> width: 100%; </ins> max-width: initial; padding: 0.5em 5vw 5% 5vw; /* fallback in case calc() fails */ background-color: #eeeeee;


body.cid-community .community-section.community-frame { <del> width: 100vw; </del> <ins> width: 100%; </ins> } body.cid-community .community-section.community-frame .twittercol1 {


body.cid-community .community-section#meetups p:last-of-type { margin-bottom: 6em; /* extra space for background */ } <ins> } @media only screen and (max-width: 767px) { body.cid-community .community-section h2:before, body.cid-community .community-section h2:after { display: none; } </ins> }  No newline at end of file


## Labels, annotations and taints used on API objects <ins> ### apf.kubernetes.io/autoupdate-spec Type: Annotation Example: `apf.kubernetes.io/autoupdate-spec: "true"` Used on: [`FlowSchema` and `PriorityLevelConfiguration` Objects](/concepts/cluster-administration/flow-control/#defaults) If this annotation is set to true on a FlowSchema or PriorityLevelConfiguration, the `spec` for that object is managed by the kube-apiserver. If the API server does not recognize an APF object, and you annotate it for automatic update, the API server deletes the entire object. Otherwise, the API server does not manage the object spec. For more details, read  [Maintenance of the Mandatory and Suggested Configuration Objects](/docs/concepts/cluster-administration/flow-control/#maintenance-of-the-mandatory-and-suggested-configuration-objects). </ins> ### app.kubernetes.io/component Type: Label


over the package builds. This means that anything before v1.24.0 will only be available in the Google-hosted repository. - There's a dedicated package repository for each Kubernetes minor version. <del> When upgrading to to a different minor release, you must bear in mind that </del> <ins> When upgrading to a different minor release, you must bear in mind that </ins> the package repository details also change. {{< /note >}}


Limiting port ranges of communication | This recommendation may be a bit self-explanatory, but wherever possible you should only expose the ports on your service that are absolutely essential for communication or metric gathering. | 3rd Party Dependency Security | It is a good practice to regularly scan your application's third party libraries for known security vulnerabilities. Each programming language has a tool for performing this check automatically. | Static Code Analysis | Most languages provide a way for a snippet of code to be analyzed for any potentially unsafe coding practices. Whenever possible you should perform checks using automated tooling that can scan codebases for common security errors. Some of the tools can be found at: https://owasp.org/www-community/Source_Code_Analysis_Tools | <del> Dynamic probing attacks | There are a few automated tools that you can run against your service to try some of the well known service attacks. These include SQL injection, CSRF, and XSS. One of the most popular dynamic analysis tools is the [OWASP Zed Attack proxy](https://owasp.org/www-project-zap/) tool. | </del> <ins> Dynamic probing attacks | There are a few automated tools that you can run against your service to try some of the well known service attacks. These include SQL injection, CSRF, and XSS. One of the most popular dynamic analysis tools is the [OWASP Zed Attack proxy](https://www.zaproxy.org/) tool. | </ins> {{< /table >}}


If you haven't worked through the earlier sections, start from Using minikube to create a cluster.

Scaling is accomplished by changing the number of replicas in a Deployment

  NOTE  If you are trying this after the previous section , you may need to start from creating a cluster as the services may have been deleted


To reflect the change on kubeadm nodes you must do the following: - Log in to a kubeadm node - Run `kubeadm upgrade node phase kubelet-config` to download the latest `kubelet-config` <del> ConfigMap contents into the local file `/var/lib/kubelet/config.conf` </del> <ins> ConfigMap contents into the local file `/var/lib/kubelet/config.yaml` </ins> - Edit the file `/var/lib/kubelet/kubeadm-flags.env` to apply additional configuration with flags - Restart the kubelet service with `systemctl restart kubelet`


{{< note >}} During `kubeadm upgrade`, kubeadm downloads the `KubeletConfiguration` from the <del> `kubelet-config` ConfigMap and overwrite the contents of `/var/lib/kubelet/config.conf`. </del> <ins> `kubelet-config` ConfigMap and overwrite the contents of `/var/lib/kubelet/config.yaml`. </ins> This means that node local configuration must be applied either by flags in `/var/lib/kubelet/kubeadm-flags.env` or by manually updating the contents of <del> `/var/lib/kubelet/config.conf` after `kubeadm upgrade`, and then restarting the kubelet. </del> <ins> `/var/lib/kubelet/config.yaml` after `kubeadm upgrade`, and then restarting the kubelet. </ins> {{< /note >}} ### Applying kube-proxy configuration changes


#### Persisting kubelet reconfiguration <del> Any changes to the `KubeletConfiguration` stored in `/var/lib/kubelet/config.conf` will be overwritten on </del> <ins> Any changes to the `KubeletConfiguration` stored in `/var/lib/kubelet/config.yaml` will be overwritten on </ins> `kubeadm upgrade` by downloading the contents of the cluster wide `kubelet-config` ConfigMap. <del> To persist kubelet node specific configuration either the file `/var/lib/kubelet/config.conf` </del> <ins> To persist kubelet node specific configuration either the file `/var/lib/kubelet/config.yaml` </ins> has to be updated manually post-upgrade or the file `/var/lib/kubelet/kubeadm-flags.env` can include flags. The kubelet flags override the associated `KubeletConfiguration` options, but note that some of the flags are deprecated. <del> A kubelet restart will be required after changing `/var/lib/kubelet/config.conf` or </del> <ins> A kubelet restart will be required after changing `/var/lib/kubelet/config.yaml` or </ins> `/var/lib/kubelet/kubeadm-flags.env`. ## {{% heading "whatsnext" %}}


and bare metal workloads. * [Flannel](https://github.com/flannel-io/flannel#deploying-flannel-manually) is an overlay network provider that can be used with Kubernetes. <ins> * [Gateway API](/docs/concepts/services-networking/gateway/) is an open source project managed by the [SIG Network](https://github.com/kubernetes/community/tree/master/sig-network) community and provides an expressive, extensible, and role-oriented API for modeling service networking. </ins> * [Knitter](https://github.com/ZTE/Knitter/) is a plugin to support multiple network interfaces in a Kubernetes pod. * [Multus](https://github.com/k8snetworkplumbingwg/multus-cni) is a Multi plugin for


to be reachable from outside your cluster. - [Ingress](/docs/concepts/services-networking/ingress/) provides extra functionality specifically for exposing HTTP applications, websites and APIs. <ins> - [Gateway API](/docs/concepts/services-networking/gateway/) is an {{<glossary_tooltip text="add-on" term_id="addons">}} that provides an expressive, extensible, and role-oriented family of API kinds for modeling service networking. </ins> - You can also use Services to [publish services only for consumption inside your cluster](/docs/concepts/services-networking/service-traffic-policy/).


<ins> --- title: Gateway API content_type: concept description: >- Gateway API is a family of API kinds that provide dynamic infrastructure provisioning and advanced traffic routing. weight: 55 --- <!-- overview --> Make network services available by using an extensible, role-oriented, protocol-aware configuration mechanism. [Gateway API](https://gateway-api.sigs.k8s.io/) is an {{<glossary_tooltip text="add-on" term_id="addons">}} containing API [kinds](https://gateway-api.sigs.k8s.io/references/spec/) that provide dynamic infrastructure provisioning and advanced traffic routing. <!-- body --> ## Design principles The following principles shaped the design and architecture of Gateway API: * __Role-oriented:__ Gateway API kinds are modeled after organizational roles that are responsible for managing Kubernetes service networking: * __Infrastructure Provider:__ Manages infrastructure that allows multiple isolated clusters to serve multiple tenants, e.g. a cloud provider. * __Cluster Operator:__ Manages clusters and is typically concerned with policies, network access, application permissions, etc. * __Application Developer:__ Manages an application running in a cluster and is typically concerned with application-level configuration and [Service](/docs/concepts/services-networking/service/) composition. * __Portable:__ Gateway API specifications are defined as [custom resources](docs/concepts/extend-kubernetes/api-extension/custom-resources) and are supported by many [implementations](https://gateway-api.sigs.k8s.io/implementations/). * __Expressive:__ Gateway API kinds support functionality for common traffic routing use cases such as header-based matching, traffic weighting, and others that were only possible in [Ingress](/docs/concepts/services-networking/ingress/) by using custom annotations. * __Extensible:__ Gateway allows for custom resources to be linked at various layers of the API. This makes granular customization possible at the appropriate places within the API structure. ## Resource model Gateway API has three stable API kinds: * __GatewayClass:__ Defines a set of gateways with common configuration and managed by a controller that implements the class. * __Gateway:__ Defines an instance of traffic handling infrastructure, such as cloud load balancer. * __HTTPRoute:__ Defines HTTP-specific rules for mapping traffic from a Gateway listener to a representation of backend network endpoints. These endpoints are often represented as a {{<glossary_tooltip text="Service" term_id="service">}}. Gateway API is organized into different API kinds that have interdependent relationships to support the role-oriented nature of organizations. A Gateway object is associated with exactly one GatewayClass; the GatewayClass describes the gateway controller responsible for managing Gateways of this class. One or more route kinds such as HTTPRoute, are then associated to Gateways. A Gateway can filter the routes that may be attached to its `listeners`, forming a bidirectional trust model with routes. The following figure illustrates the relationships of the three stable Gateway API kinds: {{< figure src="/docs/images/gateway-kind-relationships.svg" alt="A figure illustrating the relationships of the three stable Gateway API kinds" class="diagram-medium" >}} ### GatewayClass {#api-kind-gateway-class} Gateways can be implemented by different controllers, often with different configurations. A Gateway must reference a GatewayClass that contains the name of the controller that implements the class. A minimal GatewayClass example: ```yaml apiVersion: gateway.networking.k8s.io/v1 kind: GatewayClass metadata: name: example-class spec: controllerName: example.com/gateway-controller ``` In this example, a controller that has implemented Gateway API is configured to manage GatewayClasses with the controller name `example.com/gateway-controller`. Gateways of this class will be managed by the implementation's controller. See the [GatewayClass](https://gateway-api.sigs.k8s.io/references/spec/#gateway.networking.k8s.io/v1.GatewayClass) reference for a full definition of this API kind. ### Gateway {#api-kind-gateway} A Gateway describes an instance of traffic handling infrastructure. It defines a network endpoint that can be used for processing traffic, i.e. filtering, balancing, splitting, etc. for backends such as a Service. For example, a Gateway may represent a cloud load balancer or an in-cluster proxy server that is configured to accept HTTP traffic. A minimal Gateway resource example: ```yaml apiVersion: gateway.networking.k8s.io/v1 kind: Gateway metadata: name: example-gateway spec: gatewayClassName: example-class listeners: - name: http protocol: HTTP port: 80 ``` In this example, an instance of traffic handling infrastructure is programmed to listen for HTTP traffic on port 80. Since the `addresses` field is unspecified, an address or hostname is assigned to the Gateway by the implementation's controller. This address is used as a network endpoint for processing traffic of backend network endpoints defined in routes. See the [Gateway](https://gateway-api.sigs.k8s.io/references/spec/#gateway.networking.k8s.io/v1.Gateway) reference for a full definition of this API kind. ### HTTPRoute {#api-kind-httproute} The HTTPRoute kind specifies routing behavior of HTTP requests from a Gateway listener to backend network endpoints. For a Service backend, an implementation may represent the backend network endpoint as a Service IP or the backing Endpoints of the Service. An HTTPRoute represents configuration that is applied to the underlying Gateway implementation. For example, defining a new HTTPRoute may result in configuring additional traffic routes in a cloud load balancer or in-cluster proxy server. A minimal HTTPRoute example: ```yaml apiVersion: gateway.networking.k8s.io/v1 kind: HTTPRoute metadata: name: example-httproute spec: parentRefs: - name: example-gateway hostnames: - "www.example.com" rules: - matches: - path: type: PathPrefix value: /login backendRefs: - name: example-svc port: 8080 ``` In this example, HTTP traffic from Gateway `example-gateway` with the Host: header set to `www.example.com` and the request path specified as `/login` will be routed to Service `example-svc` on port `8080`. See the [HTTPRoute](https://gateway-api.sigs.k8s.io/references/spec/#gateway.networking.k8s.io/v1.HTTPRoute) reference for a full definition of this API kind. ## Request flow Here is a simple example of HTTP traffic being routed to a Service by using a Gateway and an HTTPRoute: {{< figure src="/docs/images/gateway-request-flow.svg" alt="A diagram that provides an example of HTTP traffic being routed to a Service by using a Gateway and an HTTPRoute" class="diagram-medium" >}} In this example, the request flow for a Gateway implemented as a reverse proxy is: 1. The client starts to prepare an HTTP request for the URL `http://www.example.com` 2. The client's DNS resolver queries for the destination name and learns a mapping to one or more IP addresses associated with the Gateway. 3. The client sends a request to the Gateway IP address; the reverse proxy receives the HTTP request and uses the Host: header to match a configuration that was derived from the Gateway and attached HTTPRoute. 4. Optionally, the reverse proxy can perform request header and/or path matching based on match rules of the HTTPRoute. 5. Optionally, the reverse proxy can modify the request; for example, to add or emove headers, based on filter rules of the HTTPRoute. 6. Lastly, the reverse proxy forwards the request to one or more backends. ## Conformance Gateway API covers a broad set of features and is widely implemented. This combination requires clear conformance definitions and tests to ensure that the API provides a consistent experience wherever it is used. See the [conformance](https://gateway-api.sigs.k8s.io/concepts/conformance/) documentation to understand details such as release channels, support levels, and running conformance tests. ## Migrating from Ingress Gateway API is the successor to the [Ingress](/docs/concepts/services-networking/ingress/) API. However, it does not include the Ingress kind. As a result, a one-time conversion from your existing Ingress resources to Gateway API resources is necessary. Refer to the [ingress migration](https://gateway-api.sigs.k8s.io/guides/migrating-from-ingress/#migrating-from-ingress) guide for details on migrating Ingress resources to Gateway API resources. ## {{% heading "whatsnext" %}} Instead of Gateway API resources being natively implemented by Kubernetes, the specifications are defined as [Custom Resources](docs/concepts/extend-kubernetes/api-extension/custom-resources) supported by a wide range of [implementations](https://gateway-api.sigs.k8s.io/implementations/). [Install](https://gateway-api.sigs.k8s.io/guides/#installing-gateway-api) the Gateway API CRDs or follow the installation instructions of your selected implementation. After installing an implementation, use the [Getting Started](https://gateway-api.sigs.k8s.io/guides/) guide to help you quickly start working with Gateway API. {{< note >}} Make sure to review the documentation of your selected implementation to understand any caveats. {{< /note >}} Refer to the [API specification](https://gateway-api.sigs.k8s.io/reference/spec/) for additional details of all Gateway API kinds. </ins>


{{< feature-state for_k8s_version="v1.19" state="stable" >}} {{< glossary_definition term_id="ingress" length="all" >}} <ins> {{< note >}} Ingress is frozen. New features are being added to the [Gateway API](/docs/concepts/services-networking/gateway/). {{< /note >}} </ins> <!-- body --> ## Terminology


* Read about [Ingress](/docs/concepts/services-networking/ingress/), which exposes HTTP and HTTPS routes from outside the cluster to Services within your cluster. <del> * Read about [Gateway](https://gateway-api.sigs.k8s.io/), an extension to </del> <ins> * Read about [Gateway](/docs/concepts/services-networking/gateway/), an extension to </ins> Kubernetes that provides more flexibility than Ingress. For more context, read the following:


 <svg id="graph-div" width="100%" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" height="130" style="max-width: 465.6875px;" viewBox="0 0 465.6875 130"><g><g class="output"><g class="clusters"><g class="cluster" id="flowchart-cluster-721" transform="translate(232.84375,65)" style="opacity: 1;"><rect width="449.6875" height="114" x="-224.84375" y="-57"></rect><g class="label" transform="translate(0, -43)" id="graph-divText"><g transform="translate(-24.578125,-12)"><foreignObject width="49.15625" height="24">
cluster
</foreignObject></g></g></g></g><g class="edgePaths"><g class="edgePath LS-gateway LE-gatewayclass" id="L-gateway-gatewayclass" style="opacity: 1;"><path class="path" d="M264.53125,65L268.6979166666667,65C272.8645833333333,65,281.1979166666667,65,289.53125,65C297.8645833333333,65,306.1979166666667,65,310.3645833333333,65L314.53125,65" marker-end="url(#arrowhead1117)" style="fill:none"></path><defs><marker id="arrowhead1117" viewBox="0 0 10 10" refX="9" refY="5" markerUnits="strokeWidth" markerWidth="8" markerHeight="6" orient="auto"><path d="M 0 0 L 10 5 L 0 10 z" class="arrowheadPath" style="stroke-width: 1; stroke-dasharray: 1, 0;"></path></marker></defs></g><g class="edgePath LS-httproute1 LE-gateway" id="L-httproute1-gateway" style="opacity: 1;"><path class="path" d="M132.03125,65L136.19791666666666,65C140.36458333333334,65,148.69791666666666,65,157.03125,65C165.36458333333334,65,173.69791666666666,65,177.86458333333334,65L182.03125,65" marker-end="url(#arrowhead1118)" style="fill:none"></path><defs><marker id="arrowhead1118" viewBox="0 0 10 10" refX="9" refY="5" markerUnits="strokeWidth" markerWidth="8" markerHeight="6" orient="auto"><path d="M 0 0 L 10 5 L 0 10 z" class="arrowheadPath" style="stroke-width: 1; stroke-dasharray: 1, 0;"></path></marker></defs></g></g><g class="edgeLabels"><g class="edgeLabel" transform="" style="opacity: 1;"><g transform="translate(0,0)" class="label"><rect rx="0" ry="0" width="0" height="0"></rect><foreignObject width="0" height="0">

</foreignObject></g></g><g class="edgeLabel" transform="" style="opacity: 1;"><g transform="translate(0,0)" class="label"><rect rx="0" ry="0" width="0" height="0"></rect><foreignObject width="0" height="0">

</foreignObject></g></g></g><g class="nodes"><g class="node k8s" id="flowchart-gatewayclass-717" transform="translate(373.609375,65)" style="opacity: 1;"><rect rx="0" ry="0" x="-59.078125" y="-22" width="118.15625" height="44" class="label-container"></rect><g class="label" transform="translate(0,0)"><g transform="translate(-49.078125,-12)"><foreignObject width="98.15625" height="24">
GatewayClass
</foreignObject></g></g></g><g class="node k8s" id="flowchart-gateway-715" transform="translate(223.28125,65)" style="opacity: 1;"><rect rx="0" ry="0" x="-41.25" y="-22" width="82.5" height="44" class="label-container"></rect><g class="label" transform="translate(0,0)"><g transform="translate(-31.25,-12)"><foreignObject width="62.5" height="24">
Gateway
</foreignObject></g></g></g><g class="node k8s" id="flowchart-httproute1-719" transform="translate(82.515625,65)" style="opacity: 1;"><rect rx="0" ry="0" x="-49.515625" y="-22" width="99.03125" height="44" class="label-container"></rect><g class="label" transform="translate(0,0)"><g transform="translate(-39.515625,-12)"><foreignObject width="79.03125" height="24">
HTTPRoute
</foreignObject></g></g></g></g></g></g></svg>   No newline at end of file


<ins> <svg version="1.1" viewBox="0.0 0.0 821.0 113.0" fill="none" stroke="none" stroke-linecap="square" stroke-miterlimit="10" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns="http://www.w3.org/2000/svg"><clipPath id="p.0"><path d="m0 0l821.0 0l0 113.0l-821.0 0l0 -113.0z" clip-rule="nonzero"/></clipPath><g clip-path="url(#p.0)"><path fill="#000000" fill-opacity="0.0" d="m0 0l821.0 0l0 113.0l-821.0 0z" fill-rule="evenodd"/><path fill="#1155cc" d="m199.1693 28.55643l103.30708 0l0 66.4252l-103.30708 0z" fill-rule="evenodd"/><path stroke="#1155cc" stroke-width="1.0" stroke-linejoin="round" stroke-linecap="butt" d="m199.1693 28.55643l103.30708 0l0 66.4252l-103.30708 0z" fill-rule="evenodd"/><path fill="#ffffff" d="m220.01613 63.78278l0 -2.25l5.8125 0l0 5.312496q-0.84375 0.828125 -2.453125 1.453125q-1.609375 0.625 -3.25 0.625q-2.09375 0 -3.65625 -0.875q-1.5625 -0.890625 -2.34375 -2.515625q-0.78125 -1.6406212 -0.78125 -3.5624962q0 -2.09375 0.875 -3.703125q0.875 -1.625 2.5625 -2.5q1.28125 -0.65625 3.203125 -0.65625q2.484375 0 3.875 1.046875q1.40625 1.03125 1.796875 2.875l-2.671875 0.5q-0.28125 -0.984375 -1.0625 -1.546875q-0.78125 -0.578125 -1.9375 -0.578125q-1.78125 0 -2.828125 1.125q-1.03125 1.125 -1.03125 3.328125q0 2.3749962 1.046875 3.5624962q1.0625 1.1875 2.78125 1.1875q0.84375 0 1.6875 -0.328125q0.859375 -0.328125 1.46875 -0.8125l0 -1.6874962l-3.09375 0zm10.198929 -1.8125l-2.328125 -0.421875q0.40625 -1.40625 1.359375 -2.078125q0.953125 -0.671875 2.84375 -0.671875q1.703125 0 2.546875 0.40625q0.84375 0.40625 1.171875 1.03125q0.34375 0.625 0.34375 2.28125l-0.015625 2.9999962q0 1.265625 0.109375 1.875q0.125 0.609375 0.46875 1.296875l-2.53125 0q-0.109375 -0.25 -0.25 -0.75q-0.0625 -0.234375 -0.09375 -0.3125q-0.65625 0.640625 -1.40625 0.96875q-0.734375 0.3125 -1.59375 0.3125q-1.484375 0 -2.34375 -0.8125q-0.859375 -0.8125 -0.859375 -2.046875q0 -0.828125 0.390625 -1.46875q0.390625 -0.6406212 1.09375 -0.9687462q0.703125 -0.34375 2.03125 -0.609375q1.796875 -0.328125 2.484375 -0.625l0 -0.25q0 -0.75 -0.359375 -1.0625q-0.359375 -0.3125 -1.375 -0.3125q-0.6875 0 -1.078125 0.28125q-0.375 0.265625 -0.609375 0.9375zm3.421875 2.0781212q-0.484375 0.15625 -1.5625 0.390625q-1.0625 0.21875 -1.390625 0.4375q-0.5 0.359375 -0.5 0.90625q0 0.53125 0.40625 0.9375q0.40625 0.390625 1.015625 0.390625q0.703125 0 1.328125 -0.46875q0.46875 -0.34375 0.609375 -0.84375q0.09375 -0.328125 0.09375 -1.25l0 -0.5zm9.485092 -5.031246l0 2.03125l-1.75 0l0 3.9062462q0 1.1875 0.046875 1.390625q0.046875 0.1875 0.21875 0.3125q0.1875 0.125 0.4375 0.125q0.359375 0 1.03125 -0.25l0.21875 2.0q-0.890625 0.375 -2.015625 0.375q-0.703125 0 -1.265625 -0.234375q-0.546875 -0.234375 -0.8125 -0.59375q-0.25 -0.375 -0.34375 -1.0q-0.09375 -0.453125 -0.09375 -1.8125l0 -4.218746l-1.171875 0l0 -2.03125l1.171875 0l0 -1.921875l2.578125 -1.5l0 3.421875l1.75 0zm7.368927 6.593746l2.546875 0.421875q-0.484375 1.40625 -1.546875 2.140625q-1.0625 0.734375 -2.65625 0.734375q-2.515625 0 -3.734375 -1.65625q-0.953125 -1.3125 -0.953125 -3.3281212q0 -2.40625 1.25 -3.765625q1.265625 -1.359375 3.1875 -1.359375q2.15625 0 3.40625 1.421875q1.25 1.421875 1.1875 4.374996l-6.40625 0q0.03125 1.140625 0.609375 1.78125q0.59375 0.625 1.484375 0.625q0.59375 0 1.0 -0.328125q0.421875 -0.328125 0.625 -1.0625zm0.15625 -2.5937462q-0.03125 -1.109375 -0.578125 -1.6875q-0.546875 -0.578125 -1.328125 -0.578125q-0.84375 0 -1.390625 0.609375q-0.546875 0.609375 -0.53125 1.65625l3.828125 0zm6.422592 5.671871l-3.0625 -9.671871l2.484375 0l1.8125 6.343746l1.671875 -6.343746l2.46875 0l1.609375 6.343746l1.859375 -6.343746l2.515625 0l-3.109375 9.671871l-2.453125 0l-1.671875 -6.218746l-1.640625 6.218746l-2.484375 0zm14.620819 -6.718746l-2.328125 -0.421875q0.40625 -1.40625 1.359375 -2.078125q0.953125 -0.671875 2.84375 -0.671875q1.703125 0 2.546875 0.40625q0.84375 0.40625 1.171875 1.03125q0.34375 0.625 0.34375 2.28125l-0.015625 2.9999962q0 1.265625 0.109375 1.875q0.125 0.609375 0.46875 1.296875l-2.53125 0q-0.109375 -0.25 -0.25 -0.75q-0.0625 -0.234375 -0.09375 -0.3125q-0.65625 0.640625 -1.40625 0.96875q-0.734375 0.3125 -1.59375 0.3125q-1.484375 0 -2.34375 -0.8125q-0.859375 -0.8125 -0.859375 -2.046875q0 -0.828125 0.390625 -1.46875q0.390625 -0.6406212 1.09375 -0.9687462q0.703125 -0.34375 2.03125 -0.609375q1.796875 -0.328125 2.484375 -0.625l0 -0.25q0 -0.75 -0.359375 -1.0625q-0.359375 -0.3125 -1.375 -0.3125q-0.6875 0 -1.078125 0.28125q-0.375 0.265625 -0.609375 0.9375zm3.421875 2.0781212q-0.484375 0.15625 -1.5625 0.390625q-1.0625 0.21875 -1.390625 0.4375q-0.5 0.359375 -0.5 0.90625q0 0.53125 0.40625 0.9375q0.40625 0.390625 1.015625 0.390625q0.703125 0 1.328125 -0.46875q0.46875 -0.34375 0.609375 -0.84375q0.09375 -0.328125 0.09375 -1.25l0 -0.5zm3.828827 -5.031246l2.71875 0l2.328125 6.859371l2.25 -6.859371l2.65625 0l-3.421875 9.312496l-0.609375 1.6875q-0.328125 0.84375 -0.640625 1.28125q-0.3125 0.453125 -0.703125 0.71875q-0.390625 0.28125 -0.96875 0.4375q-0.578125 0.15625 -1.3125 0.15625q-0.734375 0 -1.453125 -0.15625l-0.21875 -2.0q0.59375 0.125 1.078125 0.125q0.890625 0 1.3125 -0.53125q0.4375 -0.515625 0.671875 -1.328125l-3.6875 -9.703121z" fill-rule="nonzero"/><path fill="#000000" fill-opacity="0.0" d="m85.531494 61.769028l113.6378 0" fill-rule="evenodd"/><path stroke="#000000" stroke-width="2.0" stroke-linejoin="round" stroke-linecap="butt" stroke-dasharray="2.0,6.0" d="m85.531494 61.769028l101.6378 0" fill-rule="evenodd"/><path fill="#000000" stroke="#000000" stroke-width="2.0" stroke-linecap="butt" d="m187.1693 65.072495l9.076187 -3.3034668l-9.076187 -3.303463z" fill-rule="evenodd"/><path fill="#000000" fill-opacity="0.0" d="m302.47638 61.769028l43.338593 0" fill-rule="evenodd"/><path stroke="#000000" stroke-width="2.0" stroke-linejoin="round" stroke-linecap="butt" d="m302.47638 61.769028l31.338593 0" fill-rule="evenodd"/><path fill="#000000" stroke="#000000" stroke-width="2.0" stroke-linecap="butt" d="m333.81497 65.072495l9.076172 -3.3034668l-9.076172 -3.303463z" fill-rule="evenodd"/><path fill="#d9d9d9" d="m27.899902 43.32703l51.09812 0l0 0c6.6939926 0 12.120544 8.256859 12.120544 18.442207c0 10.185345 -5.426552 18.442204 -12.120544 18.442204l-51.09812 0l0 0c-6.6939926 0 -12.120544 -8.256859 -12.120544 -18.442204c0 -10.1853485 5.426552 -18.442207 12.120544 -18.442207z" fill-rule="evenodd"/><path stroke="#ffffff" stroke-width="1.0" stroke-linejoin="round" stroke-linecap="butt" d="m27.899902 43.32703l51.09812 0l0 0c6.6939926 0 12.120544 8.256859 12.120544 18.442207c0 10.185345 -5.426552 18.442204 -12.120544 18.442204l-51.09812 0l0 0c-6.6939926 0 -12.120544 -8.256859 -12.120544 -18.442204c0 -10.1853485 5.426552 -18.442207 12.120544 -18.442207z" fill-rule="evenodd"/><path fill="#000000" d="m39.219585 65.14236l1.609375 0.21875q-0.265625 1.65625 -1.359375 2.609375q-1.078125 0.9375 -2.671875 0.9375q-1.984375 0 -3.1875 -1.296875q-1.203125 -1.296875 -1.203125 -3.71875q0 -1.578125 0.515625 -2.75q0.515625 -1.171875 1.578125 -1.75q1.0625 -0.59375 2.3125 -0.59375q1.578125 0 2.578125 0.796875q1.0 0.796875 1.28125 2.265625l-1.59375 0.234375q-0.234375 -0.96875 -0.8125 -1.453125q-0.578125 -0.5 -1.390625 -0.5q-1.234375 0 -2.015625 0.890625q-0.78125 0.890625 -0.78125 2.8125q0 1.953125 0.75 2.84375q0.75 0.875 1.953125 0.875q0.96875 0 1.609375 -0.59375q0.65625 -0.59375 0.828125 -1.828125zm2.96875 3.546875l0 -13.359375l1.640625 0l0 13.359375l-1.640625 0zm4.191696 -11.46875l0 -1.890625l1.640625 0l0 1.890625l-1.640625 0zm0 11.46875l0 -9.671875l1.640625 0l0 9.671875l-1.640625 0zm10.769821 -3.109375l1.6875 0.203125q-0.40625 1.484375 -1.484375 2.3125q-1.078125 0.8125 -2.765625 0.8125q-2.125 0 -3.375 -1.296875q-1.234375 -1.3125 -1.234375 -3.671875q0 -2.453125 1.25 -3.796875q1.265625 -1.34375 3.265625 -1.34375q1.9375 0 3.15625 1.328125q1.234375 1.3125 1.234375 3.703125q0 0.15625 0 0.4375l-7.21875 0q0.09375 1.59375 0.90625 2.453125q0.8125 0.84375 2.015625 0.84375q0.90625 0 1.546875 -0.46875q0.640625 -0.484375 1.015625 -1.515625zm-5.390625 -2.65625l5.40625 0q-0.109375 -1.21875 -0.625 -1.828125q-0.78125 -0.953125 -2.03125 -0.953125q-1.125 0 -1.90625 0.765625q-0.765625 0.75 -0.84375 2.015625zm9.141342 5.765625l0 -9.671875l1.46875 0l0 1.375q1.0625 -1.59375 3.0781212 -1.59375q0.875 0 1.609375 0.3125q0.734375 0.3125 1.09375 0.828125q0.375 0.5 0.515625 1.203125q0.09375 0.453125 0.09375 1.59375l0 5.953125l-1.640625 0l0 -5.890625q0 -1.0 -0.203125 -1.484375q-0.1875 -0.5 -0.671875 -0.796875q-0.484375 -0.296875 -1.140625 -0.296875q-1.046875 0 -1.8124962 0.671875q-0.75 0.65625 -0.75 2.515625l0 5.28125l-1.640625 0zm13.953838 -1.46875l0.234375 1.453125q-0.6875 0.140625 -1.234375 0.140625q-0.890625 0 -1.390625 -0.28125q-0.484375 -0.28125 -0.6875 -0.734375q-0.203125 -0.46875 -0.203125 -1.9375l0 -5.578125l-1.203125 0l0 -1.265625l1.203125 0l0 -2.390625l1.625 -0.984375l0 3.375l1.65625 0l0 1.265625l-1.65625 0l0 5.671875q0 0.6875 0.078125 0.890625q0.09375 0.203125 0.28125 0.328125q0.203125 0.109375 0.578125 0.109375q0.265625 0 0.71875 -0.0625z" fill-rule="nonzero"/><path fill="#1155cc" d="m345.82153 28.55643l121.73227 0l0 66.4252l-121.73227 0z" fill-rule="evenodd"/><path stroke="#1155cc" stroke-width="1.0" stroke-linejoin="round" stroke-linecap="butt" d="m345.82153 28.55643l121.73227 0l0 66.4252l-121.73227 0z" fill-rule="evenodd"/><path fill="#ffffff" d="m357.26614 68.689026l0 -13.359371l2.703125 0l0 5.265625l5.28125 0l0 -5.265625l2.703125 0l0 13.359371l-2.703125 0l0 -5.843746l-5.28125 0l0 5.843746l-2.703125 0zm16.472961 0l0 -11.093746l-3.953125 0l0 -2.265625l10.609375 0l0 2.265625l-3.953125 0l0 11.093746l-2.703125 0zm11.395966 0l0 -11.093746l-3.953125 0l0 -2.265625l10.609375 0l0 2.265625l-3.953125 0l0 11.093746l-2.703125 0zm8.395996 0l0 -13.359371l4.328125 0q2.453125 0 3.203125 0.203125q1.140625 0.296875 1.921875 1.3125q0.78125 1.0 0.78125 2.59375q0 1.234375 -0.453125 2.078125q-0.453125 0.828125 -1.140625 1.3125q-0.6875 0.46875 -1.390625 0.625q-0.96875 0.203125 -2.796875 0.203125l-1.765625 0l0 5.031246l-2.6875 0zm2.6875 -11.093746l0 3.78125l1.484375 0q1.59375 0 2.125 -0.203125q0.546875 -0.203125 0.84375 -0.65625q0.3125 -0.453125 0.3125 -1.03125q0 -0.734375 -0.4375 -1.203125q-0.421875 -0.484375 -1.078125 -0.59375q-0.484375 -0.09375 -1.9375 -0.09375l-1.3125 0zm9.756073 11.093746l0 -13.359371l5.6875 0q2.140625 0 3.109375 0.359375q0.96875 0.359375 1.546875 1.28125q0.578125 0.921875 0.578125 2.109375q0 1.5 -0.890625 2.484375q-0.875 0.96875 -2.625 1.234375q0.875 0.5 1.4375 1.125q0.578125 0.6093712 1.53125 2.1562462l1.640625 2.609375l-3.234375 0l-1.9375 -2.90625q-1.046875 -1.5625 -1.4375 -1.9687462q-0.375 -0.40625 -0.8125 -0.546875q-0.421875 -0.15625 -1.34375 -0.15625l-0.546875 0l0 5.578121l-2.703125 0zm2.703125 -7.703121l2.0 0q1.9375 0 2.421875 -0.15625q0.484375 -0.171875 0.75 -0.578125q0.28125 -0.40625 0.28125 -1.0q0 -0.671875 -0.359375 -1.078125q-0.359375 -0.421875 -1.015625 -0.53125q-0.328125 -0.046875 -1.96875 -0.046875l-2.109375 0l0 3.390625zm10.160431 2.734375q0 -1.28125 0.625 -2.46875q0.625 -1.203125 1.78125 -1.828125q1.15625 -0.625 2.578125 -0.625q2.1875 0 3.59375 1.421875q1.40625 1.421875 1.40625 3.609375q0 2.1874962 -1.421875 3.6406212q-1.421875 1.4375 -3.5625 1.4375q-1.328125 0 -2.546875 -0.59375q-1.203125 -0.609375 -1.828125 -1.765625q-0.625 -1.171875 -0.625 -2.8281212zm2.625 0.125q0 1.4531212 0.671875 2.2187462q0.6875 0.75 1.6875 0.75q1.0 0 1.671875 -0.75q0.6875 -0.765625 0.6875 -2.2343712q0 -1.421875 -0.6875 -2.1875q-0.671875 -0.765625 -1.671875 -0.765625q-1.0 0 -1.6875 0.765625q-0.671875 0.765625 -0.671875 2.203125zm15.724121 4.843746l0 -1.453125q-0.53125 0.78125 -1.390625 1.234375q-0.859375 0.4375 -1.8125 0.4375q-0.96875 0 -1.75 -0.421875q-0.765625 -0.4375 -1.125 -1.203125q-0.34375 -0.78125 -0.34375 -2.140625l0 -6.124996l2.5625 0l0 4.4375q0 2.0468712 0.140625 2.5156212q0.140625 0.453125 0.515625 0.71875q0.375 0.265625 0.953125 0.265625q0.65625 0 1.171875 -0.359375q0.515625 -0.359375 0.703125 -0.890625q0.203125 -0.53125 0.203125 -2.6093712l0 -4.078125l2.546875 0l0 9.671871l-2.375 0zm9.474091 -9.671871l0 2.03125l-1.75 0l0 3.9062462q0 1.1875 0.046875 1.390625q0.046875 0.1875 0.21875 0.3125q0.1875 0.125 0.4375 0.125q0.359375 0 1.03125 -0.25l0.21875 2.0q-0.890625 0.375 -2.015625 0.375q-0.703125 0 -1.265625 -0.234375q-0.546875 -0.234375 -0.8125 -0.59375q-0.25 -0.375 -0.34375 -1.0q-0.09375 -0.453125 -0.09375 -1.8125l0 -4.218746l-1.171875 0l0 -2.03125l1.171875 0l0 -1.921875l2.578125 -1.5l0 3.421875l1.75 0zm7.368927 6.593746l2.546875 0.421875q-0.484375 1.40625 -1.546875 2.140625q-1.0625 0.734375 -2.65625 0.734375q-2.515625 0 -3.734375 -1.65625q-0.953125 -1.3125 -0.953125 -3.3281212q0 -2.40625 1.25 -3.765625q1.265625 -1.359375 3.1875 -1.359375q2.15625 0 3.40625 1.421875q1.25 1.421875 1.1875 4.374996l-6.40625 0q0.03125 1.140625 0.609375 1.78125q0.59375 0.625 1.484375 0.625q0.59375 0 1.0 -0.328125q0.421875 -0.328125 0.625 -1.0625zm0.15625 -2.5937462q-0.03125 -1.109375 -0.578125 -1.6875q-0.546875 -0.578125 -1.328125 -0.578125q-0.84375 0 -1.390625 0.609375q-0.546875 0.609375 -0.53125 1.65625l3.828125 0z" fill-rule="nonzero"/><path fill="#d9d9d9" d="m109.42257 42.04987l63.244102 0l0 18.456692l-63.244102 0z" fill-rule="evenodd"/><path stroke="#ffffff" stroke-width="1.0" stroke-linejoin="round" stroke-linecap="butt" d="m109.42257 42.04987l63.244102 0l0 18.456692l-63.244102 0z" fill-rule="evenodd"/><path fill="#000000" d="m127.99311 55.018215l0 -7.625l1.0000076 0l0 3.125l3.96875 0l0 -3.125l1.015625 0l0 7.625l-1.015625 0l0 -3.59375l-3.96875 0l0 3.59375l-1.0000076 0zm9.60186 0l0 -6.734375l-2.515625 0l0 -0.890625l6.046875 0l0 0.890625l-2.53125 0l0 6.734375l-1.0 0zm6.509262 0l0 -6.734375l-2.515625 0l0 -0.890625l6.046875 0l0 0.890625l-2.53125 0l0 6.734375l-1.0 0zm4.571762 0l0 -7.625l2.875 0q0.75 0 1.15625 0.0625q0.5625 0.09375 0.9375 0.359375q0.390625 0.265625 0.609375 0.75q0.234375 0.46875 0.234375 1.03125q0 0.96875 -0.625 1.65625q-0.609375 0.671875 -2.234375 0.671875l-1.953125 0l0 3.09375l-1.0 0zm1.0 -4.0l1.96875 0q0.984375 0 1.390625 -0.359375q0.421875 -0.375 0.421875 -1.03125q0 -0.484375 -0.25 -0.8125q-0.234375 -0.34375 -0.640625 -0.453125q-0.25 -0.078125 -0.9375 -0.078125l-1.953125 0l0 2.734375z" fill-rule="nonzero"/><path fill="#d9d9d9" d="m109.42126 63.026245l63.244102 0l0 18.456696l-63.244102 0z" fill-rule="evenodd"/><path stroke="#ffffff" stroke-width="1.0" stroke-linejoin="round" stroke-linecap="butt" d="m109.42126 63.026245l63.244102 0l0 18.456696l-63.244102 0z" fill-rule="evenodd"/><path fill="#000000" d="m123.959114 75.99459l0 -5.53125l0.84375 0l0 0.84375q0.328125 -0.59375 0.59375 -0.78125q0.28125 -0.1875 0.609375 -0.1875q0.46875 0 0.953125 0.3125l-0.3125 0.859375q-0.34375 -0.203125 -0.6875 -0.203125q-0.3125 0 -0.5625 0.1875q-0.234375 0.1875 -0.34375 0.515625q-0.15625 0.5 -0.15625 1.09375l0 2.890625l-0.9375 0zm7.345497 -1.78125l0.96875 0.125q-0.234375 0.84375 -0.859375 1.3125q-0.609375 0.46875 -1.578125 0.46875q-1.203125 0 -1.9218826 -0.75q-0.703125 -0.75 -0.703125 -2.09375q0 -1.390625 0.71875 -2.15625q0.7187576 -0.78125 1.8593826 -0.78125q1.109375 0 1.8125 0.765625q0.703125 0.75 0.703125 2.125q0 0.078125 0 0.234375l-4.125 0q0.046875 0.921875 0.515625 1.40625q0.46875 0.484375 1.15625 0.484375q0.515625 0 0.875 -0.265625q0.359375 -0.28125 0.578125 -0.875zm-3.078125 -1.515625l3.09375 0q-0.0625 -0.6875 -0.359375 -1.046875q-0.453125 -0.53125 -1.15625 -0.53125q-0.640625 0 -1.09375 0.4375q-0.4375 0.421875 -0.484375 1.140625zm8.738998 5.421875l0 -2.71875q-0.21875 0.3125 -0.609375 0.515625q-0.390625 0.203125 -0.828125 0.203125q-0.984375 0 -1.703125 -0.78125q-0.703125 -0.796875 -0.703125 -2.15625q0 -0.828125 0.28125 -1.484375q0.296875 -0.671875 0.84375 -1.015625q0.546875 -0.34375 1.203125 -0.34375q1.03125 0 1.609375 0.875l0 -0.75l0.84375 0l0 7.65625l-0.9375 0zm-2.875 -4.90625q0 1.0625 0.4375 1.609375q0.453125 0.53125 1.078125 0.53125q0.59375 0 1.015625 -0.5q0.4375 -0.515625 0.4375 -1.546875q0 -1.109375 -0.453125 -1.65625q-0.453125 -0.5625 -1.0625 -0.5625q-0.609375 0 -1.03125 0.515625q-0.421875 0.515625 -0.421875 1.609375zm8.910873 2.78125l0 -0.8125q-0.65625 0.9375 -1.75 0.9375q-0.5 0 -0.921875 -0.1875q-0.421875 -0.1875 -0.625 -0.46875q-0.203125 -0.28125 -0.296875 -0.703125q-0.046875 -0.265625 -0.046875 -0.875l0 -3.421875l0.9375 0l0 3.0625q0 0.734375 0.046875 1.0q0.09375 0.359375 0.375 0.578125q0.296875 0.203125 0.703125 0.203125q0.421875 0 0.796875 -0.203125q0.375 -0.21875 0.515625 -0.59375q0.15625 -0.375 0.15625 -1.078125l0 -2.96875l0.9375 0l0 5.53125l-0.828125 0zm6.0827484 -1.78125l0.96875 0.125q-0.234375 0.84375 -0.859375 1.3125q-0.609375 0.46875 -1.578125 0.46875q-1.203125 0 -1.921875 -0.75q-0.703125 -0.75 -0.703125 -2.09375q0 -1.390625 0.71875 -2.15625q0.71875 -0.78125 1.859375 -0.78125q1.109375 0 1.8125 0.765625q0.703125 0.75 0.703125 2.125q0 0.078125 0 0.234375l-4.125 0q0.046875 0.921875 0.515625 1.40625q0.46875 0.484375 1.15625 0.484375q0.515625 0 0.875 -0.265625q0.359375 -0.28125 0.578125 -0.875zm-3.078125 -1.515625l3.09375 0q-0.0625 -0.6875 -0.359375 -1.046875q-0.453125 -0.53125 -1.15625 -0.53125q-0.640625 0 -1.09375 0.4375q-0.4375 0.421875 -0.484375 1.140625zm4.8483734 1.640625l0.921875 -0.140625q0.078125 0.5625 0.4375 0.859375q0.359375 0.296875 1.0 0.296875q0.640625 0 0.953125 -0.265625q0.3125 -0.265625 0.3125 -0.625q0 -0.3125 -0.28125 -0.5q-0.1875 -0.125 -0.953125 -0.3125q-1.03125 -0.265625 -1.4375 -0.453125q-0.390625 -0.1875 -0.59375 -0.515625q-0.203125 -0.34375 -0.203125 -0.75q0 -0.359375 0.171875 -0.671875q0.171875 -0.328125 0.453125 -0.53125q0.21875 -0.15625 0.59375 -0.265625q0.390625 -0.125 0.8125 -0.125q0.65625 0 1.140625 0.1875q0.5 0.1875 0.734375 0.515625q0.234375 0.3125 0.3125 0.859375l-0.90625 0.125q-0.0625 -0.4375 -0.375 -0.671875q-0.296875 -0.234375 -0.828125 -0.234375q-0.65625 0 -0.9375 0.21875q-0.265625 0.203125 -0.265625 0.484375q0 0.1875 0.109375 0.328125q0.125 0.15625 0.359375 0.25q0.140625 0.0625 0.828125 0.25q1.0 0.265625 1.390625 0.4375q0.390625 0.15625 0.609375 0.484375q0.234375 0.3125 0.234375 0.796875q0 0.46875 -0.28125 0.890625q-0.265625 0.40625 -0.78125 0.640625q-0.515625 0.21875 -1.171875 0.21875q-1.078125 0 -1.640625 -0.4375q-0.5625 -0.453125 -0.71875 -1.34375zm7.75 0.8125l0.125 0.828125q-0.390625 0.09375 -0.703125 0.09375q-0.5 0 -0.78125 -0.15625q-0.28125 -0.171875 -0.40625 -0.4375q-0.109375 -0.265625 -0.109375 -1.109375l0 -3.171875l-0.6875 0l0 -0.734375l0.6875 0l0 -1.359375l0.9375 -0.5625l0 1.921875l0.9375 0l0 0.734375l-0.9375 0l0 3.234375q0 0.390625 0.046875 0.515625q0.046875 0.109375 0.15625 0.1875q0.109375 0.0625 0.328125 0.0625q0.15625 0 0.40625 -0.046875z" fill-rule="nonzero"/><path fill="#000000" fill-opacity="0.0" d="m467.5538 61.769028l116.75589 -0.8503914" fill-rule="evenodd"/><path stroke="#000000" stroke-width="2.0" stroke-linejoin="round" stroke-linecap="butt" d="m467.5538 61.769028l104.756256 -0.76299286" fill-rule="evenodd"/><path fill="#000000" stroke="#000000" stroke-width="2.0" stroke-linecap="butt" d="m572.3341 64.30941l9.05188 -3.3694801l-9.100037 -3.2372704z" fill-rule="evenodd"/><path fill="#d9d9d9" d="m473.99738 52.540684l88.8819 0l0 18.456692l-88.8819 0z" fill-rule="evenodd"/><path stroke="#ffffff" stroke-width="1.0" stroke-linejoin="round" stroke-linecap="butt" d="m473.99738 52.540684l88.8819 0l0 18.456692l-88.8819 0z" fill-rule="evenodd"/><path fill="#000000" d="m490.55234 65.509026l0 -7.624996l3.375 0q1.015625 0 1.546875 0.203125q0.53125 0.203125 0.84375 0.734375q0.328125 0.515625 0.328125 1.140625q0 0.8125 -0.53125 1.375q-0.515625 0.546875 -1.625 0.703125q0.40625 0.1875 0.609375 0.375q0.453125 0.40625 0.84375 1.015625l1.328125 2.0781212l-1.265625 0l-1.015625 -1.5937462q-0.4375 -0.6875 -0.734375 -1.046875q-0.28125 -0.359375 -0.515625 -0.5q-0.21875 -0.15625 -0.453125 -0.203125q-0.171875 -0.046875 -0.5625 -0.046875l-1.171875 0l0 3.3906212l-1.0 0zm1.0 -4.265621l2.171875 0q0.6875 0 1.078125 -0.140625q0.390625 -0.140625 0.59375 -0.453125q0.203125 -0.3125 0.203125 -0.6875q0 -0.546875 -0.390625 -0.890625q-0.390625 -0.34375 -1.234375 -0.34375l-2.421875 0l0 2.515625zm6.2112427 1.5q0 -1.53125 0.84375 -2.265625q0.71875 -0.625 1.734375 -0.625q1.140625 0 1.859375 0.75q0.734375 0.75 0.734375 2.0625q0 1.0625 -0.328125 1.6874962q-0.3125 0.609375 -0.921875 0.953125q-0.609375 0.328125 -1.34375 0.328125q-1.15625 0 -1.875 -0.734375q-0.703125 -0.75 -0.703125 -2.1562462zm0.953125 0q0 1.0625 0.46875 1.5937462q0.46875 0.53125 1.15625 0.53125q0.703125 0 1.15625 -0.53125q0.46875 -0.5312462 0.46875 -1.6249962q0 -1.015625 -0.46875 -1.546875q-0.453125 -0.53125 -1.15625 -0.53125q-0.6875 0 -1.15625 0.53125q-0.46875 0.515625 -0.46875 1.578125zm8.942108 2.7656212l0 -0.8125q-0.65625 0.9375 -1.75 0.9375q-0.5 0 -0.921875 -0.1875q-0.421875 -0.1875 -0.625 -0.46875q-0.203125 -0.28125 -0.296875 -0.703125q-0.046875 -0.265625 -0.046875 -0.8749962l0 -3.421875l0.9375 0l0 3.0625q0 0.734375 0.046875 0.9999962q0.09375 0.359375 0.375 0.578125q0.296875 0.203125 0.703125 0.203125q0.421875 0 0.796875 -0.203125q0.375 -0.21875 0.515625 -0.59375q0.15625 -0.3749962 0.15625 -1.0781212l0 -2.96875l0.9375 0l0 5.531246l-0.828125 0zm4.3483887 -0.84375l0.125 0.828125q-0.390625 0.09375 -0.703125 0.09375q-0.5 0 -0.78125 -0.15625q-0.28125 -0.171875 -0.40625 -0.4375q-0.109375 -0.265625 -0.109375 -1.1093712l0 -3.171875l-0.6875 0l0 -0.734375l0.6875 0l0 -1.359375l0.9375 -0.5625l0 1.921875l0.9375 0l0 0.734375l-0.9375 0l0 3.234375q0 0.3906212 0.046875 0.5156212q0.046875 0.109375 0.15625 0.1875q0.109375 0.0625 0.328125 0.0625q0.15625 0 0.40625 -0.046875zm0.9137573 -5.703121l0 -1.078125l0.9375 0l0 1.078125l-0.9375 0zm0 6.546871l0 -5.531246l0.9375 0l0 5.531246l-0.9375 0zm2.3674927 0l0 -5.531246l0.84375 0l0 0.796875q0.609375 -0.921875 1.75 -0.921875q0.5 0 0.921875 0.1875q0.421875 0.171875 0.625 0.46875q0.21875 0.296875 0.296875 0.6875q0.046875 0.265625 0.046875 0.921875l0 3.3906212l-0.9375 0l0 -3.3593712q0 -0.578125 -0.109375 -0.859375q-0.109375 -0.28125 -0.390625 -0.453125q-0.265625 -0.171875 -0.640625 -0.171875q-0.59375 0 -1.03125 0.390625q-0.4375 0.375 -0.4375 1.4375l0 3.0156212l-0.9375 0zm5.7546387 0.453125l0.90625 0.140625q0.0625 0.421875 0.328125 0.609375q0.34375 0.265625 0.9375 0.265625q0.65625 0 1.0 -0.265625q0.359375 -0.25 0.484375 -0.71875q0.078125 -0.296875 0.078125 -1.203125q-0.625 0.71875 -1.53125 0.71875q-1.140625 0 -1.765625 -0.8125q-0.625 -0.8281212 -0.625 -1.9843712q0 -0.78125 0.28125 -1.453125q0.296875 -0.671875 0.828125 -1.03125q0.546875 -0.375 1.28125 -0.375q0.984375 0 1.609375 0.796875l0 -0.671875l0.875 0l0 4.781246q0 1.296875 -0.265625 1.828125q-0.265625 0.546875 -0.84375 0.859375q-0.5625 0.3125 -1.390625 0.3125q-0.984375 0 -1.59375 -0.453125q-0.609375 -0.4375 -0.59375 -1.34375zm0.78125 -3.3124962q0 1.078125 0.421875 1.5781212q0.4375 0.5 1.078125 0.5q0.65625 0 1.09375 -0.484375q0.4375 -0.4999962 0.4375 -1.5624962q0 -1.015625 -0.453125 -1.53125q-0.453125 -0.515625 -1.09375 -0.515625q-0.609375 0 -1.046875 0.515625q-0.4375 0.5 -0.4375 1.5zm8.262146 2.8593712l0 -5.531246l0.84375 0l0 0.84375q0.328125 -0.59375 0.59375 -0.78125q0.28125 -0.1875 0.609375 -0.1875q0.46875 0 0.953125 0.3125l-0.3125 0.859375q-0.34375 -0.203125 -0.6875 -0.203125q-0.3125 0 -0.5625 0.1875q-0.234375 0.1875 -0.34375 0.515625q-0.15625 0.5 -0.15625 1.09375l0 2.8906212l-0.9375 0zm7.189209 0l0 -0.8125q-0.65625 0.9375 -1.75 0.9375q-0.5 0 -0.921875 -0.1875q-0.421875 -0.1875 -0.625 -0.46875q-0.203125 -0.28125 -0.296875 -0.703125q-0.046875 -0.265625 -0.046875 -0.8749962l0 -3.421875l0.9375 0l0 3.0625q0 0.734375 0.046875 0.9999962q0.09375 0.359375 0.375 0.578125q0.296875 0.203125 0.703125 0.203125q0.421875 0 0.796875 -0.203125q0.375 -0.21875 0.515625 -0.59375q0.15625 -0.3749962 0.15625 -1.0781212l0 -2.96875l0.9375 0l0 5.531246l-0.828125 0zm2.2858887 0l0 -7.624996l0.9375 0l0 7.624996l-0.9375 0zm6.1643066 -1.7812462l0.96875 0.125q-0.234375 0.8437462 -0.859375 1.3124962q-0.609375 0.46875 -1.578125 0.46875q-1.203125 0 -1.921875 -0.75q-0.703125 -0.75 -0.703125 -2.0937462q0 -1.390625 0.71875 -2.15625q0.71875 -0.78125 1.859375 -0.78125q1.109375 0 1.8125 0.765625q0.703125 0.75 0.703125 2.125q0 0.078125 0 0.234375l-4.125 0q0.046875 0.921875 0.515625 1.4062462q0.46875 0.484375 1.15625 0.484375q0.515625 0 0.875 -0.265625q0.359375 -0.28125 0.578125 -0.8749962zm-3.078125 -1.515625l3.09375 0q-0.0625 -0.6875 -0.359375 -1.046875q-0.453125 -0.53125 -1.15625 -0.53125q-0.640625 0 -1.09375 0.4375q-0.4375 0.421875 -0.484375 1.140625z" fill-rule="nonzero"/><path fill="#1155cc" d="m584.3202 35.097115l100.409485 0l0 51.65354l-100.409485 0z" fill-rule="evenodd"/><path stroke="#1155cc" stroke-width="1.0" stroke-linejoin="round" stroke-linecap="butt" d="m584.3202 35.097115l100.409485 0l0 51.65354l-100.409485 0z" fill-rule="evenodd"/><path fill="#ffffff" d="m602.00183 63.500134l2.625 -0.25q0.234375 1.3124962 0.953125 1.9374962q0.734375 0.609375 1.96875 0.609375q1.296875 0 1.953125 -0.546875q0.671875 -0.546875 0.671875 -1.2812462q0 -0.484375 -0.28125 -0.8125q-0.28125 -0.328125 -0.96875 -0.578125q-0.484375 -0.15625 -2.171875 -0.578125q-2.15625 -0.546875 -3.03125 -1.328125q-1.234375 -1.09375 -1.234375 -2.6875q0 -1.015625 0.578125 -1.90625q0.578125 -0.890625 1.65625 -1.34375q1.09375 -0.46875 2.640625 -0.46875q2.515625 0 3.78125 1.109375q1.28125 1.09375 1.34375 2.9375l-2.703125 0.109375q-0.171875 -1.03125 -0.75 -1.46875q-0.5625 -0.453125 -1.703125 -0.453125q-1.171875 0 -1.84375 0.46875q-0.421875 0.3125 -0.421875 0.84375q0 0.46875 0.40625 0.796875q0.5 0.4375 2.46875 0.90625q1.96875 0.453125 2.90625 0.953125q0.953125 0.5 1.484375 1.359375q0.53125 0.859375 0.53125 2.125q0 1.1562462 -0.640625 2.1562462q-0.640625 1.0 -1.8125 1.484375q-1.15625 0.484375 -2.890625 0.484375q-2.53125 0 -3.890625 -1.171875q-1.359375 -1.171875 -1.625 -3.4062462zm18.709167 1.2656212l2.546875 0.421875q-0.484375 1.40625 -1.546875 2.140625q-1.0625 0.734375 -2.65625 0.734375q-2.515625 0 -3.734375 -1.65625q-0.953125 -1.3125 -0.953125 -3.3281212q0 -2.40625 1.25 -3.765625q1.265625 -1.359375 3.1875 -1.359375q2.15625 0 3.40625 1.421875q1.25 1.421875 1.1875 4.375l-6.40625 0q0.03125 1.1406212 0.609375 1.7812462q0.59375 0.625 1.484375 0.625q0.59375 0 1.0 -0.328125q0.421875 -0.328125 0.625 -1.0625zm0.15625 -2.5937462q-0.03125 -1.109375 -0.578125 -1.6875q-0.546875 -0.578125 -1.328125 -0.578125q-0.84375 0 -1.390625 0.609375q-0.546875 0.609375 -0.53125 1.65625l3.828125 0zm7.0788574 5.671871l-2.5625 0l0 -9.671871l2.375 0l0 1.375q0.609375 -0.984375 1.09375 -1.28125q0.484375 -0.3125 1.109375 -0.3125q0.875 0 1.6875 0.484375l-0.796875 2.234375q-0.640625 -0.421875 -1.203125 -0.421875q-0.53125 0 -0.90625 0.296875q-0.375 0.296875 -0.59375 1.078125q-0.203125 0.765625 -0.203125 3.2343712l0 2.984375zm7.463379 0l-3.90625 -9.671871l2.6875 0l1.828125 4.9375l0.53125 1.6406212q0.203125 -0.625 0.265625 -0.8281212q0.125 -0.40625 0.265625 -0.8125l1.84375 -4.9375l2.625 0l-3.84375 9.671871l-2.296875 0zm7.7194824 -10.984371l0 -2.375l2.5625 0l0 2.375l-2.5625 0zm0 10.984371l0 -9.671871l2.5625 0l0 9.671871l-2.5625 0zm13.620789 -6.812496l-2.53125 0.453125q-0.125 -0.75 -0.578125 -1.125q-0.453125 -0.390625 -1.171875 -0.390625q-0.953125 0 -1.53125 0.65625q-0.5625 0.65625 -0.5625 2.203125q0 1.7343712 0.578125 2.4374962q0.578125 0.703125 1.546875 0.703125q0.734375 0 1.203125 -0.40625q0.46875 -0.421875 0.65625 -1.421875l2.515625 0.421875q-0.390625 1.734375 -1.515625 2.625q-1.109375 0.875 -2.96875 0.875q-2.125 0 -3.390625 -1.328125q-1.25 -1.34375 -1.25 -3.7187462q0 -2.390625 1.265625 -3.71875q1.265625 -1.34375 3.421875 -1.34375q1.765625 0 2.796875 0.765625q1.046875 0.75 1.515625 2.3125zm7.5319824 3.7343712l2.546875 0.421875q-0.484375 1.40625 -1.546875 2.140625q-1.0625 0.734375 -2.65625 0.734375q-2.515625 0 -3.734375 -1.65625q-0.953125 -1.3125 -0.953125 -3.3281212q0 -2.40625 1.25 -3.765625q1.265625 -1.359375 3.1875 -1.359375q2.15625 0 3.40625 1.421875q1.25 1.421875 1.1875 4.375l-6.40625 0q0.03125 1.1406212 0.609375 1.7812462q0.59375 0.625 1.484375 0.625q0.59375 0 1.0 -0.328125q0.421875 -0.328125 0.625 -1.0625zm0.15625 -2.5937462q-0.03125 -1.109375 -0.578125 -1.6875q-0.546875 -0.578125 -1.328125 -0.578125q-0.84375 0 -1.390625 0.609375q-0.546875 0.609375 -0.53125 1.65625l3.828125 0z" fill-rule="nonzero"/><path fill="#000000" fill-opacity="0.0" d="m684.7297 60.923885l56.661377 -26.834648" fill-rule="evenodd"/><path stroke="#000000" stroke-width="2.0" stroke-linejoin="round" stroke-linecap="butt" d="m684.7297 60.923885l45.816162 -21.698391" fill-rule="evenodd"/><path fill="#000000" stroke="#000000" stroke-width="2.0" stroke-linecap="butt" d="m731.95984 42.211063l6.7888184 -6.870373l-9.61676 0.8992386z" fill-rule="evenodd"/><path fill="#1155cc" d="m741.3963 15.661417l63.24414 0l0 36.88189l-63.24414 0z" fill-rule="evenodd"/><path stroke="#1155cc" stroke-width="1.0" stroke-linejoin="round" stroke-linecap="butt" d="m741.3963 15.661417l63.24414 0l0 36.88189l-63.24414 0z" fill-rule="evenodd"/><path fill="#ffffff" d="m756.76 41.02236l0 -13.359377l4.328125 0q2.453125 0 3.203125 0.203125q1.140625 0.296875 1.921875 1.3125q0.78125 1.0 0.78125 2.59375q0 1.2343769 -0.453125 2.078127q-0.453125 0.828125 -1.140625 1.3125q-0.6875 0.46875 -1.390625 0.625q-0.96875 0.203125 -2.796875 0.203125l-1.765625 0l0 5.03125l-2.6875 0zm2.6875 -11.093752l0 3.781252l1.484375 0q1.59375 0 2.125 -0.203125q0.546875 -0.203125 0.84375 -0.65625q0.3125 -0.453125 0.3125 -1.0312519q0 -0.734375 -0.4375 -1.203125q-0.421875 -0.484375 -1.078125 -0.59375q-0.484375 -0.09375 -1.9375 -0.09375l-1.3125 0zm9.1466675 6.125002q0 -1.28125 0.625 -2.46875q0.625 -1.203125 1.78125 -1.8281269q1.15625 -0.625 2.578125 -0.625q2.1875 0 3.59375 1.4218769q1.40625 1.421875 1.40625 3.609375q0 2.1875 -1.421875 3.640625q-1.421875 1.4375 -3.5625 1.4375q-1.328125 0 -2.546875 -0.59375q-1.203125 -0.609375 -1.828125 -1.765625q-0.625 -1.171875 -0.625 -2.828125zm2.625 0.125q0 1.453125 0.671875 2.21875q0.6875 0.75 1.6875 0.75q1.0 0 1.671875 -0.75q0.6875 -0.765625 0.6875 -2.234375q0 -1.421875 -0.6875 -2.1875q-0.671875 -0.765625 -1.671875 -0.765625q-1.0 0 -1.6875 0.765625q-0.671875 0.765625 -0.671875 2.203125zm18.239746 4.84375l-2.390625 0l0 -1.421875q-0.59375 0.828125 -1.40625 1.234375q-0.796875 0.40625 -1.609375 0.40625q-1.671875 0 -2.859375 -1.34375q-1.1875 -1.34375 -1.1875 -3.75q0 -2.453125 1.15625 -3.734375q1.15625 -1.2812519 2.921875 -1.2812519q1.625 0 2.8125 1.3437519l0 -4.812502l2.5625 0l0 13.359377zm-6.84375 -5.046875q0 1.546875 0.4375 2.234375q0.609375 1.015625 1.71875 1.015625q0.890625 0 1.5 -0.75q0.625 -0.765625 0.625 -2.25q0 -1.671875 -0.609375 -2.40625q-0.59375 -0.734375 -1.53125 -0.734375q-0.90625 0 -1.53125 0.734375q-0.609375 0.71875 -0.609375 2.15625z" fill-rule="nonzero"/><path fill="#1155cc" d="m741.3963 63.026245l63.24414 0l0 36.88189l-63.24414 0z" fill-rule="evenodd"/><path stroke="#1155cc" stroke-width="1.0" stroke-linejoin="round" stroke-linecap="butt" d="m741.3963 63.026245l63.24414 0l0 36.88189l-63.24414 0z" fill-rule="evenodd"/><path fill="#ffffff" d="m756.76 88.38719l0 -13.359375l4.328125 0q2.453125 0 3.203125 0.203125q1.140625 0.296875 1.921875 1.3125q0.78125 1.0 0.78125 2.59375q0 1.234375 -0.453125 2.078125q-0.453125 0.828125 -1.140625 1.3125q-0.6875 0.46875 -1.390625 0.625q-0.96875 0.203125 -2.796875 0.203125l-1.765625 0l0 5.03125l-2.6875 0zm2.6875 -11.09375l0 3.78125l1.484375 0q1.59375 0 2.125 -0.203125q0.546875 -0.203125 0.84375 -0.65625q0.3125 -0.453125 0.3125 -1.03125q0 -0.734375 -0.4375 -1.203125q-0.421875 -0.484375 -1.078125 -0.59375q-0.484375 -0.09375 -1.9375 -0.09375l-1.3125 0zm9.1466675 6.125q0 -1.28125 0.625 -2.46875q0.625 -1.203125 1.78125 -1.828125q1.15625 -0.625 2.578125 -0.625q2.1875 0 3.59375 1.421875q1.40625 1.421875 1.40625 3.609375q0 2.1875 -1.421875 3.640625q-1.421875 1.4375 -3.5625 1.4375q-1.328125 0 -2.546875 -0.59375q-1.203125 -0.609375 -1.828125 -1.765625q-0.625 -1.171875 -0.625 -2.828125zm2.625 0.125q0 1.453125 0.671875 2.21875q0.6875 0.75 1.6875 0.75q1.0 0 1.671875 -0.75q0.6875 -0.765625 0.6875 -2.234375q0 -1.421875 -0.6875 -2.1875q-0.671875 -0.765625 -1.671875 -0.765625q-1.0 0 -1.6875 0.765625q-0.671875 0.765625 -0.671875 2.203125zm18.239746 4.84375l-2.390625 0l0 -1.421875q-0.59375 0.828125 -1.40625 1.234375q-0.796875 0.40625 -1.609375 0.40625q-1.671875 0 -2.859375 -1.34375q-1.1875 -1.34375 -1.1875 -3.75q0 -2.453125 1.15625 -3.734375q1.15625 -1.28125 2.921875 -1.28125q1.625 0 2.8125 1.34375l0 -4.8125l2.5625 0l0 13.359375zm-6.84375 -5.046875q0 1.546875 0.4375 2.234375q0.609375 1.015625 1.71875 1.015625q0.890625 0 1.5 -0.75q0.625 -0.765625 0.625 -2.25q0 -1.671875 -0.609375 -2.40625q-0.59375 -0.734375 -1.53125 -0.734375q-0.90625 0 -1.53125 0.734375q-0.609375 0.71875 -0.609375 2.15625z" fill-rule="nonzero"/><path fill="#000000" fill-opacity="0.0" d="m684.7297 60.923885l56.661377 20.535435" fill-rule="evenodd"/><path stroke="#000000" stroke-width="2.0" stroke-linejoin="round" stroke-linecap="butt" d="m684.7297 60.923885l45.379517 16.446606" fill-rule="evenodd"/><path fill="#000000" stroke="#000000" stroke-width="2.0" stroke-linecap="butt" d="m728.9836 80.47627l9.65863 -0.013198853l-7.4074097 -6.198372z" fill-rule="evenodd"/></g></svg> </ins>  No newline at end of file


<ins> --- title: Gateway API id: gateway-api date: 2023-10-19 full_link: /docs/concepts/services-networking/gateway/ short_description: > An API for modeling service networking in Kubernetes. aka: tags: - networking - architecture - extension --- A family of API kinds for modeling service networking in Kubernetes. <!--more--> Gateway API provides a family of extensible, role-oriented, protocol-aware API kinds for modeling service networking in Kubernetes. </ins>


 --- title: Autoscaling Workloads description: >- With autoscaling, you can automatically update your workloads in one way or another. This allows your cluster to react to changes in resource demand more elastically and efficiently. content_type: concept weight: 40 ---  In Kubernetes, you can _scale_ a workload depending on the current demand of resources. This allows your cluster to react to changes in resource demand more elastically and efficiently. When you scale a workload, you can either increase or decrease the number of replicas managed by the workload, or adjust the resources available to the replicas in-place. The first approach is referred to as _horizontal scaling_, while the second is referred to as _vertical scaling_. There are manual and automatic ways to scale your workloads, depending on your use case.  ## Scaling workloads manually Kubernetes supports _manual scaling_ of workloads. Horizontal scaling can be done using the `kubectl` CLI. For vertical scaling, you need to _patch_ the resource definition of your workload. See below for examples of both strategies. - **Horizontal scaling**: [Running multiple instances of your app](/docs/tutorials/kubernetes-basics/scale/scale-intro/) - **Vertical scaling**: [Resizing CPU and memory resources assigned to containers](/docs/tasks/configure-pod-container/resize-container-resources) ## Scaling workloads automatically Kubernetes also supports _automatic scaling_ of workloads, which is the focus of this page. The concept of _Autoscaling_ in Kubernetes refers to the ability to automatically update an object that manages a set of Pods (for example a {{< glossary_tooltip text="Deployment" term_id="deployment" >}}. ### Scaling workloads horizontally In Kubernetes, you can automatically scale a workload horizontally using a _HorizontalPodAutoscaler_ (HPA). It is implemented as a Kubernetes API resource and a {{< glossary_tooltip text="controller" term_id="controller" >}} and periodically adjusts the number of {{< glossary_tooltip text="replicas" term_id="replica" >}} in a workload to match observed resource utilization such as CPU or memory usage. There is a [walkthrough tutorial](/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough) of configuring a HorizontalPodAutoscaler for a Deployment. ### Scaling workloads vertically {{< feature-state for_k8s_version="v1.25" state="stable" >}} You can automatically scale a workload vertically using a _VerticalPodAutoscaler_ (VPA). Different to the HPA, the VPA doesn't come with Kubernetes by default, but is a separate project that can be found [on GitHub](https://github.com/kubernetes/autoscaler/tree/9f87b78df0f1d6e142234bb32e8acbd71295585a/vertical-pod-autoscaler). Once installed, it allows you to create {{< glossary_tooltip text="CustomResourceDefinitions" term_id="customresourcedefinition" >}} (CRDs) for your workloads which define _how_ and _when_ to scale the resources of the managed replicas. {{< note >}} You will need to have the [Metrics Server](https://github.com/kubernetes-sigs/metrics-server) installed to your cluster for the HPA to work. {{< /note >}} At the moment, the VPA can operate in four different modes: {{
}} Mode | Description :----|:----------- `Auto` | Currently `Recreate`, might change to in-place updates in the future `Recreate` | The VPA assigns resource requests on pod creation as well as updates them on existing pods by evicting them when the requested resources differ significantly from the new recommendation `Initial` | The VPA only assigns resource requests on pod creation and never changes them later. `Off` | The VPA does not automatically change the resource requirements of the pods. The recommendations are calculated and can be inspected in the VPA object. {{
}} #### Requirements for in-place resizing {{< feature-state for_k8s_version="v1.27" state="alpha" >}} Resizing a workload in-place **without** restarting the {{< glossary_tooltip text="Pods" term_id="pod" >}} or its {{< glossary_tooltip text="Containers" term_id="container" >}} requires Kubernetes version 1.27 or later.
 Additionally, the `InPlaceVerticalScaling` feature gate needs to be enabled. {{< feature-gate-description name="InPlacePodVerticalScaling" >}} ### Autoscaling based on cluster size For workloads that need to be scaled based on the size of the cluster (for example `cluster-dns` or other system components), you can use the _Cluster Proportional Autoscaler_.
 Just like the VPA, it is not part of the Kubernetes core, but hosted in its own repository [on GitHub](https://github.com/kubernetes-sigs/cluster-proportional-autoscaler). The Cluster Proportional Autoscaler watches the number of schedulable {{< glossary_tooltip text="nodes" term_id="node" >}} and cores and scales the number of replicas of the target workload accordingly. ### Event driven Autoscaling It is also possible to scale workloads based on events, for example using the [_Kubernetes Event Driven Autoscaler_ (**KEDA**)](https://keda.sh/). KEDA is a CNCF graduated enabling you to scale your workloads based on the number of events to be processed, for example the amount of messages in a queue. There exists a wide range of adapters for different event sources to choose from. ### Autoscaling based on schedules Another strategy for scaling your workloads is to **schedule** the scaling operations, for example in order to reduce resource consumption during off-peak hours. Similar to event driven autoscaling, such behavior can be achieved using KEDA in conjunction with its [`Cron` scaler](https://keda.sh/docs/2.13/scalers/cron/). The `Cron` scaler allows you to define schedules (and time zones) for scaling your workloads in or out. ## Scaling cluster infrastructure If scaling workloads isn't enough to meet your needs, you can also scale your cluster infrastructure itself. Scaling the cluster infrastructure normally means adding or removing {{< glossary_tooltip text="nodes" term_id="node" >}}. This can be done using one of two available autoscalers: - [**Cluster Autoscaler**](https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler) - [**Karpenter**](https://github.com/kubernetes-sigs/karpenter?tab=readme-ov-file) Both scalers work by watching for pods marked as _unschedulable_ or _underutilized_ nodes and then adding or removing nodes as needed. ## {{% heading "whatsnext" %}} - Learn more about scaling horizontally - [Scale a StatefulSet](/docs/tasks/run-application/scale-stateful-set/) - [HorizontalPodAutoscaler Walkthrough](/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough/) - [Resize Container Resources In-Place](/docs/tasks/configure-pod-container/resize-container-resources/) - [Autoscale the DNS Service in a Cluster](/docs/tasks/administer-cluster/dns-horizontal-autoscaling/)


You can use {{< glossary_tooltip term_id="containerd" text="ContainerD" >}} 1.4.0+ as the container runtime for Kubernetes nodes that run Windows. <del> Learn how to [install ContainerD on a Windows node](/docs/setup/production-environment/container-runtimes/#install-containerd). </del> <ins> Learn how to [install ContainerD on a Windows node](/docs/setup/production-environment/container-runtimes/#containerd). </ins> {{< note >}} There is a [known limitation](/docs/tasks/configure-pod-container/configure-gmsa/#gmsa-limitations) when using GMSA with containerd to access Windows network shares, which requires a


- `kubectl edit cm kubelet-config-x.yy -n kube-system`を呼び出します(`x.yy`はKubernetesのバージョンに置き換えてください)。 - 既存の`cgroupDriver`の値を修正するか、以下のような新しいフィールドを追加します。 <del> ``yaml </del> <ins> ```yaml </ins> cgroupDriver: systemd ```