Question: timeout := 2 * time.Minute framework.Logf("Waiting up to %v for sourceIp test to be executed", timeout) cmd := fmt.Sprintf(`wget -T 30 -qO- %s:%d | grep client_address`, serviceIp, servicePort) <del> // need timeout mechanism because it may takes more times for iptables to be populated </del> <ins> // Need timeout mechanism because it may takes more times for iptables to be populated. </ins> for start := time.Now(); time.Since(start) < timeout; time.Sleep(2) { stdout, err = framework.RunHostCmd(execPod.Namespace, execPod.Name, cmd) if err != nil { framework.Logf("got err: %v, retry until timeout", err) continue } <ins> // Need to check output because wget -q might omit the error. if strings.TrimSpace(stdout) == "" { framework.Logf("got empty stdout, retry until timeout") continue } </ins> break } ExpectNoError(err) <del> // the stdout return from RunHostCmd seems to come with "n", so TrimSpace is needed // desired stdout in this format: client_address=x.x.x.x </del> <ins> // The stdout return from RunHostCmd seems to come with "n", so TrimSpace is needed. // Desired stdout in this format: client_address=x.x.x.x </ins> outputs := strings.Split(strings.TrimSpace(stdout), "=") <ins> sourceIp := "" </ins> if len(outputs) != 2 { <del> // fail the test if output format is unexpected </del> <ins> // Fail the test if output format is unexpected. </ins> framework.Failf("exec pod returned unexpected stdout format: [%v]n", stdout) <del> return execPodIp, "" </del> <ins> } else { sourceIp = outputs[1] </ins> } <del> return execPodIp, outputs[1] </del> <ins> return execPodIp, sourceIp </ins> }

https://k8s- Failed: [] Services should preserve source pod IP for traffic thru service cluster IP {Kubernetes e2e suite} Happened on a presubmit run in .
Sorry, it is a new e2e test I am still working on. It is not merged yet.
Don't worry about it, our flake detection isn't quite perfect yet :)
[FLAKE-PING] This flaky-test issue would love to have more attention...
[FLAKE-PING] This flaky-test issue would love to have more attention.
This new e2e test has been merged. Let's observe it few more days and close this issue if nothing bad happen.
[FLAKE-PING] This flaky-test issue would love to have more attention.
This does't look like a failure in new test, rather its a generic failure with serviceaccounts.
Thanks for the investigation. Yeah I think it is the serviceaccounts issue. I wonder when could we close this flaky issue. Seems like it is stable if serviceaccounts issue is fixed.
You can close it with a reference to the serviceaccounts issue - do you know the # for that one ?
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] Services should preserve source pod IP for traffic thru service cluster IP {Kubernetes e2e suite}
https://k8s- Failed: [] Services should preserve source pod IP for traffic thru service cluster IP {Kubernetes e2e suite}
https://k8s- Failed: [] Services should preserve source pod IP for traffic thru service cluster IP {Kubernetes e2e suite}
https://k8s- Failed: [] Services should preserve source pod IP for traffic thru service cluster IP {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
Have a fix for this, I'm now keeping running tests on my own cluster. Will send out PR once verify the fix works.


Question: package apparmor import ( <del> "bufio" </del> "errors" "fmt" <del> "os" "path" </del> "strings" "github.com/opencontainers/runc/libcontainer/apparmor"

As mentioned the return value of is not being used any longer and the entire method can be removed. The usage of was removed in Now there is no need to have the function and store it's result at all. /sig node /kind cleanup /good-first-issue /help-wanted KEP xref:
This request has been marked as suitable for new contributors. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : As mentioned the return value of is not being used any longer and the entire method can be removed. /kind cleanup /good-first-issue /help-wanted Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/triage accepted
/assign
I am having issues submitting the pull request. I removed that portion of the code under however this is my first time contributing and would like some help submitting this new updated version..
/assign
/assign
/assign
/assign


Question: if oftc.IsDir() { t.Errorf("[%d: %q] expected block device, got unexpected folder: %s", i, tc.name, path) } <del> if !oftc.IsFile() { </del> <ins> if oftc.IsFile() { </ins> t.Errorf("[%d: %q] expected block device, got unexpected file: %s", i, tc.name, path) } if oftc.IsSocket() {

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: This feature is lacking coverage, especially for containerized kubelet environments.
/assign since I'm the owner of this feature.
And just fixes a bug for containerized kubelet when the mount file is an empty file.
Are we going to enable these tests in v1.11? We're getting code freeze soon.
There is a local containerized cluster pull and CI job that we can probably leverage. Right now it only runs conformance tests, we may be able to add new suites to enable other tests. Cc who can point us to the owner for the containerized testing effort
yes, seems to be holding up pretty well - https://k8s-,%20master%20(dev)
In I made pull-kubernetes-local-e2e-containerized to run also all test with string. I'm enabling some tests (subpath!) in , feel free to suggest more tests.
Ref
IIUC, the default host path type is . And when host path type is , we expect container runtime to create the host path if it doesn't exist (docker is doing that today). Is that true? If that is, we should revert part of , because container runtime should still create host path if it does not exist. /cc I tried to enforce , and got a lot of errors , e.g.:
Correct. This is mainly for backwards compatibility.
I see. If so, I'll remove the CRI definition "If the hostPath doesn't exist (e.g. hostPath volume), runtimes should report errors". Thanks!
SGTM. Refer to .
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale will help fix this.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close.


Question: if oftc.IsDir() { t.Errorf("[%d: %q] expected block device, got unexpected folder: %s", i, tc.name, path) } <del> if !oftc.IsFile() { </del> <ins> if oftc.IsFile() { </ins> t.Errorf("[%d: %q] expected block device, got unexpected file: %s", i, tc.name, path) } if oftc.IsSocket() {

If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with .


Question: if oftc.IsDir() { t.Errorf("[%d: %q] expected block device, got unexpected folder: %s", i, tc.name, path) } <del> if !oftc.IsFile() { </del> <ins> if oftc.IsFile() { </ins> t.Errorf("[%d: %q] expected block device, got unexpected file: %s", i, tc.name, path) } if oftc.IsSocket() {

Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: import ( "context" <ins> "fmt" "path" </ins> "strings" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"

[X] test approval issue : [X] test pr : 3 May 2023 [X] two weeks soak end date : 16 May 2023 [X] test promotion pr : [X] remove endpoint form file According to following two APIsnoop queries, there are still 12 APIResources endpoints that are not tested for Conformance. Using a number of existing e2e test practices a new has been created to address these 12 endpoints. The e2e logs for this test are listed below. This query shows the following apiresources endpoints are hit within a short period of running this e2e test. If a test with these calls gets merged, test coverage will go up by 12 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/triage accepted
/open For tracking the rest of the processes
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: func mustBeUnHealthy(t kubeapiservertesting.Logger, checkName, wantBodyContains string, clientConfig *rest.Config, excludes ...string) { t.Helper() var restErr error <del> pollErr := wait.PollImmediate(2*time.Second, 2*time.Minute, func() (bool, error) { </del> <ins> pollErr := wait.PollImmediate(1*time.Second, 5*time.Minute, func() (bool, error) { </ins> body, ok, err := getHealthz(checkName, clientConfig, excludes...) restErr = err if err != nil {

{Failed === RUN TestKMSv2Healthz --- FAIL: TestKMSv2Healthz (167.38s) also shown in " TestKMSv2Healthz": 1, Not sure No response No response /sig auth
/assign /triage accepted Use for poll interval Change the poll timeout to 5m Update the poll error check (check if not nil)


Question: "command": [ "/bin/sh", "-c", <del> "if [ -e /usr/local/bin/migrate-if-needed.sh ]; then /usr/local/bin/migrate-if-needed.sh 1>>/var/log/etcd{{ suffix }}.log 2>&1; fi; /usr/local/bin/etcd --name etcd-{{ hostname }} --listen-peer-urls {{ etcd_protocol }}://{{ hostname }}:{{ server_port }} --initial-advertise-peer-urls {{ etcd_protocol }}://{{ hostname }}:{{ server_port }} --advertise-client-urls http://127.0.0.1:{{ port }} --listen-client-urls http://127.0.0.1:{{ port }} {{ quota_bytes }} --data-dir /var/etcd/data{{ suffix }} --initial-cluster-state {{ cluster_state }} --initial-cluster {{ etcd_cluster }} {{ etcd_creds }} 1>>/var/log/etcd{{ suffix }}.log 2>&1" </del> <ins> "if [ -e /usr/local/bin/migrate-if-needed.sh ]; then /usr/local/bin/migrate-if-needed.sh 1>>/var/log/etcd{{ suffix }}.log 2>&1; fi; exec /usr/local/bin/etcd --name etcd-{{ hostname }} --listen-peer-urls {{ etcd_protocol }}://{{ hostname }}:{{ server_port }} --initial-advertise-peer-urls {{ etcd_protocol }}://{{ hostname }}:{{ server_port }} --advertise-client-urls http://127.0.0.1:{{ port }} --listen-client-urls http://127.0.0.1:{{ port }} {{ quota_bytes }} --data-dir /var/etcd/data{{ suffix }} --initial-cluster-state {{ cluster_state }} --initial-cluster {{ etcd_cluster }} {{ etcd_creds }} 1>>/var/log/etcd{{ suffix }}.log 2>&1" </ins> ], "env": [ { "name": "TARGET_STORAGE",

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: When kube-apiserver's livenessProbe suggests that pod is unhealthy, kubelet tries to kill its container with 30 seconds grace period. The container is killed ungracefully with SIGKILL when grace period passes. This happens each time. What you expected to happen: I expected that SIGTERM will be passed to kube-apiserver which will try to gracefully stop before 30 seconds pass (at least in some cases). How to reproduce it (as minimally and precisely as possible): Inject kube-apiserver failure (e.g. ) Watch kubelet logs for container's death You will see that container is killed after 30 seconds each time Anything else we need to know?: I believe the problem is that in kube-apiserver.manifest uses '/bin/sh -c CMD' as a PID1 process which doesn't pass SIGTERM signals to child process. See Environment: Kubernetes version (use ): v1.7.11
/sig node
Looks like without the pipe command will have to be removed which will make the logs hard to track. There is indeed some which makes kube-apiserver run in background and trapping SIGTERM then send a SIGTERM to apiserver's PID.
I think this is the case for all our components, not just kube-apiserver. - FYI
How about using 'exec'-based solution suggested here: In our case it means prepending 'exec': I did changed this in my cluster and the process seems to meet requirements: fd/1 and fd/2 are redirected to /var/log/kube- container is killed instantaneously If everyone is fine with such a solution, I can prepare PR adding 'exec ' in front of all kubernetes/cluster/saltbase/salt/kube-apiserver/*.manifest using '/bin/sh -c CMD' pattern.
Not sure if understand what you mean. Before there was no logs, so /bin/sh -c wasn't needed. -based solution seems to have all required properties. What is your concern about using it?
Umm. just ignore my previous comment. I just tried prepending and it works fine for me.


Question: "command": [ "/bin/sh", "-c", <del> "if [ -e /usr/local/bin/migrate-if-needed.sh ]; then /usr/local/bin/migrate-if-needed.sh 1>>/var/log/etcd{{ suffix }}.log 2>&1; fi; /usr/local/bin/etcd --name etcd-{{ hostname }} --listen-peer-urls {{ etcd_protocol }}://{{ hostname }}:{{ server_port }} --initial-advertise-peer-urls {{ etcd_protocol }}://{{ hostname }}:{{ server_port }} --advertise-client-urls http://127.0.0.1:{{ port }} --listen-client-urls http://127.0.0.1:{{ port }} {{ quota_bytes }} --data-dir /var/etcd/data{{ suffix }} --initial-cluster-state {{ cluster_state }} --initial-cluster {{ etcd_cluster }} {{ etcd_creds }} 1>>/var/log/etcd{{ suffix }}.log 2>&1" </del> <ins> "if [ -e /usr/local/bin/migrate-if-needed.sh ]; then /usr/local/bin/migrate-if-needed.sh 1>>/var/log/etcd{{ suffix }}.log 2>&1; fi; exec /usr/local/bin/etcd --name etcd-{{ hostname }} --listen-peer-urls {{ etcd_protocol }}://{{ hostname }}:{{ server_port }} --initial-advertise-peer-urls {{ etcd_protocol }}://{{ hostname }}:{{ server_port }} --advertise-client-urls http://127.0.0.1:{{ port }} --listen-client-urls http://127.0.0.1:{{ port }} {{ quota_bytes }} --data-dir /var/etcd/data{{ suffix }} --initial-cluster-state {{ cluster_state }} --initial-cluster {{ etcd_cluster }} {{ etcd_creds }} 1>>/var/log/etcd{{ suffix }}.log 2>&1" </ins> ], "env": [ { "name": "TARGET_STORAGE",

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): No What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): 3 Environment: Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: What happened: All of the salt scripts bundled in kubernetes (for example, ) start up the component in a broken way. They use which means the shell will intercept all signals. So these daemons are not gracefully shutdown when the pod is stopped. This causes serious problems since when apiserver is killed abruptly it doesn't seem to close watches correctly. This was observed in for example. What you expected to happen: Running is incorrect. It should run the binary directly. Otherwise, it needs to perform an so the binary correctly replaces the shell script process. How to reproduce it (as minimally and precisely as possible): Run kube-apiserver inside a container with and try to stop it. You can't - eventually docker will force kill it. Anything else we need to know:
It looks like we changed from running the binaries directly to using in (from a few years ago to enable us to capture logs. or may recall why we had to use a shell since they reviewed that PR.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close.


Question: "command": [ "/bin/sh", "-c", <del> "if [ -e /usr/local/bin/migrate-if-needed.sh ]; then /usr/local/bin/migrate-if-needed.sh 1>>/var/log/etcd{{ suffix }}.log 2>&1; fi; /usr/local/bin/etcd --name etcd-{{ hostname }} --listen-peer-urls {{ etcd_protocol }}://{{ hostname }}:{{ server_port }} --initial-advertise-peer-urls {{ etcd_protocol }}://{{ hostname }}:{{ server_port }} --advertise-client-urls http://127.0.0.1:{{ port }} --listen-client-urls http://127.0.0.1:{{ port }} {{ quota_bytes }} --data-dir /var/etcd/data{{ suffix }} --initial-cluster-state {{ cluster_state }} --initial-cluster {{ etcd_cluster }} {{ etcd_creds }} 1>>/var/log/etcd{{ suffix }}.log 2>&1" </del> <ins> "if [ -e /usr/local/bin/migrate-if-needed.sh ]; then /usr/local/bin/migrate-if-needed.sh 1>>/var/log/etcd{{ suffix }}.log 2>&1; fi; exec /usr/local/bin/etcd --name etcd-{{ hostname }} --listen-peer-urls {{ etcd_protocol }}://{{ hostname }}:{{ server_port }} --initial-advertise-peer-urls {{ etcd_protocol }}://{{ hostname }}:{{ server_port }} --advertise-client-urls http://127.0.0.1:{{ port }} --listen-client-urls http://127.0.0.1:{{ port }} {{ quota_bytes }} --data-dir /var/etcd/data{{ suffix }} --initial-cluster-state {{ cluster_state }} --initial-cluster {{ etcd_cluster }} {{ etcd_creds }} 1>>/var/log/etcd{{ suffix }}.log 2>&1" </ins> ], "env": [ { "name": "TARGET_STORAGE",

Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


Question: logdir := "/var/log" http.ServeFile(resp.ResponseWriter, req.Request, logdir) } <ins> // logFileNameIsTooLong checks filename length, returns true if it's longer than 255. // cause http.ServeFile returns default error code 500 except for NotExist and Forbidden, but we need to separate the real 500 from oversize filename here. func logFileNameIsTooLong(filePath string) bool { _, err := os.Stat(filePath) if err != nil { if e, ok := err.(*os.PathError); ok && e.Err == syscall.ENAMETOOLONG { return true } } return false } </ins>

Hello and I were fuzzing the k8s (via ) and found out following bug. Getting logs from unicode logpath via API causes internal server error. The API endpoint is . Response with non-500 HTTP status code We were fuzzing k8s locally via minikube. minukube ca certificates k8s IP address the IP to as access token and export it as request Kubernetes version (use ):Cloud provider or hardware configuration: minikube with virtual-box driver OS (e.g: ): Ubuntu 18.04.5 LTS (bionic) Kernel (e.g. ): Linux pine 4.15.0-140-generic -Ubuntu SMP Fri Mar 19 14:12:35 UTC 2021 x8664 x8664 x86_64 GNU/Linux Install tools: downloaded precompiled binary minikube version: v1.19.0 (commit: )
/sig api-machinery
/assign I will look into it.
thank you , please keep us posted! /triage accepted
For clarity are you saying that the fuzzed URL is valid or invalid UTF-8 (URL-encoded)?
It's valid UTF-8
Actually, according to my test,this bug comes from http.ServeFile which replies to the request with the contents of the named file or directory. When giving too long(On Linux: The maximum length for a file name is 255 bytes. The maximum combined length of both the file name and path name is 4096 bytes.) filename or pathname to http.ServeFile,it will return code 500(file name too long) rather than 404(no such file or directory). And when filename is not that long, unicode path work correctly too. Here is my simple test below: And since /logs is the only place use http.ServeFile, there is no other api has the same problem. ! So I will add a length check before calling http.ServeFile to solve this. Does that sound good to you?
Maybe 400 - bad request would be more suitable response code. But I'm not a maintainer so it's not up to me to decide.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with .


Question: logdir := "/var/log" http.ServeFile(resp.ResponseWriter, req.Request, logdir) } <ins> // logFileNameIsTooLong checks filename length, returns true if it's longer than 255. // cause http.ServeFile returns default error code 500 except for NotExist and Forbidden, but we need to separate the real 500 from oversize filename here. func logFileNameIsTooLong(filePath string) bool { _, err := os.Stat(filePath) if err != nil { if e, ok := err.(*os.PathError); ok && e.Err == syscall.ENAMETOOLONG { return true } } return false } </ins>

Send feedback to sig-contributor-experience at . /lifecycle stale
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: <ins> /* Copyright 2017 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package gce import ( "testing" "fmt" compute "google.golang.org/api/compute/v1" "google.golang.org/api/googleapi" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/kubernetes/pkg/cloudprovider" ) func TestCreateDisk_Basic(t *testing.T) { /* Arrange */ fakeManager := newFakeManager() projectId := "test-project" gce := GCECloud{ manager:      fakeManager, managedZones: []string{"zone1"}, projectID:    projectId, } diskName := "disk" diskType := DiskTypeSSD zone := "zone1" const sizeGb int64 = 128 tags := make(map[string]string) tags["test-tag"] = "test-value" diskTypeUri := fmt.Sprintf(diskTypeUriTemplate, projectId, zone, diskType) expectedDescription := "{"test-tag":"test-value"}" /* Act */ err := gce.CreateDisk(diskName, diskType, zone, sizeGb, tags) /* Assert */ if err != nil { t.Error(err) } if !fakeManager.createDiskCalled { t.Error("Never called GCE disk create.") } if !fakeManager.doesOpMatch { t.Error("Ops used in WaitForZoneOp does not match what's returned by CreateDisk.") } // Partial check of equality between disk description sent to GCE and parameters of method. diskToCreate := fakeManager.diskToCreate if diskToCreate.Name != diskName { t.Errorf("Expected disk name: %s; Actual: %s", diskName, diskToCreate.Name) } if diskToCreate.Type != diskTypeUri { t.Errorf("Expected disk type: %s; Actual: %s", diskTypeUri, diskToCreate.Type) } if diskToCreate.SizeGb != sizeGb { t.Errorf("Expected disk size: %d; Actual: %d", sizeGb, diskToCreate.SizeGb) } if diskToCreate.Description != expectedDescription { t.Errorf("Expected tag string: %s; Actual: %s", expectedDescription, diskToCreate.Description) } } func TestCreateDisk_DiskAlreadyExists(t *testing.T) { /* Arrange */ fakeManager := newFakeManager() gce := GCECloud{manager: fakeManager, managedZones: []string{"zone1"}} // Inject disk AlreadyExists error. alreadyExistsError := googleapi.ErrorItem{Reason: "alreadyExists"} fakeManager.waitForZoneOpError = &googleapi.Error{ Errors: []googleapi.ErrorItem{alreadyExistsError}, } /* Act */ err := gce.CreateDisk("disk", DiskTypeSSD, "zone1", 128, nil) /* Assert */ if err != nil { t.Error( "Expected success when a disk with the given name already exists, but an error is returned.") } } func TestCreateDisk_WrongZone(t *testing.T) { /* Arrange */ fakeManager := newFakeManager() gce := GCECloud{manager: fakeManager, managedZones: []string{"zone1"}} diskName := "disk" diskType := DiskTypeSSD const sizeGb int64 = 128 /* Act */ err := gce.CreateDisk(diskName, diskType, "zone2", sizeGb, nil) /* Assert */ if err == nil { t.Error("Expected error when zone is not managed, but none returned.") } } func TestCreateDisk_NoManagedZone(t *testing.T) { /* Arrange */ fakeManager := newFakeManager() gce := GCECloud{manager: fakeManager, managedZones: []string{}} diskName := "disk" diskType := DiskTypeSSD const sizeGb int64 = 128 /* Act */ err := gce.CreateDisk(diskName, diskType, "zone1", sizeGb, nil) /* Assert */ if err == nil { t.Error("Expected error when managedZones is empty, but none returned.") } } func TestCreateDisk_BadDiskType(t *testing.T) { /* Arrange */ fakeManager := newFakeManager() gce := GCECloud{manager: fakeManager, managedZones: []string{"zone1"}} diskName := "disk" diskType := "arbitrary-disk" zone := "zone1" const sizeGb int64 = 128 /* Act */ err := gce.CreateDisk(diskName, diskType, zone, sizeGb, nil) /* Assert */ if err == nil { t.Error("Expected error when disk type is not supported, but none returned.") } } func TestCreateDisk_MultiZone(t *testing.T) { /* Arrange */ fakeManager := newFakeManager() gce := GCECloud{manager: fakeManager, managedZones: []string{"zone1", "zone2", "zone3"}} diskName := "disk" diskType := DiskTypeStandard const sizeGb int64 = 128 /* Act & Assert */ for _, zone := range gce.managedZones { diskName = zone + "disk" err := gce.CreateDisk(diskName, diskType, zone, sizeGb, nil) if err != nil { t.Errorf("Error creating disk in zone '%v'; error: "%v"", zone, err) } } } func TestDeleteDisk_Basic(t *testing.T) { /* Arrange */ fakeManager := newFakeManager() gce := GCECloud{manager: fakeManager, managedZones: []string{"zone1"}} diskName := "disk" diskType := DiskTypeSSD zone := "zone1" const sizeGb int64 = 128 gce.CreateDisk(diskName, diskType, zone, sizeGb, nil) /* Act */ err := gce.DeleteDisk(diskName) /* Assert */ if err != nil { t.Error(err) } if !fakeManager.deleteDiskCalled { t.Error("Never called GCE disk delete.") } if !fakeManager.doesOpMatch { t.Error("Ops used in WaitForZoneOp does not match what's returned by DeleteDisk.") } } func TestDeleteDisk_NotFound(t *testing.T) { /* Arrange */ fakeManager := newFakeManager() gce := GCECloud{manager: fakeManager, managedZones: []string{"zone1"}} diskName := "disk" /* Act */ err := gce.DeleteDisk(diskName) /* Assert */ if err != nil { t.Error("Expected successful operation when disk is not found, but an error is returned.") } } func TestDeleteDisk_ResourceBeingUsed(t *testing.T) { /* Arrange */ fakeManager := newFakeManager() gce := GCECloud{manager: fakeManager, managedZones: []string{"zone1"}} diskName := "disk" diskType := DiskTypeSSD zone := "zone1" const sizeGb int64 = 128 gce.CreateDisk(diskName, diskType, zone, sizeGb, nil) fakeManager.resourceInUse = true /* Act */ err := gce.DeleteDisk(diskName) /* Assert */ if err == nil { t.Error("Expected error when disk is in use, but none returned.") } } func TestDeleteDisk_SameDiskMultiZone(t *testing.T) { /* Assert */ fakeManager := newFakeManager() gce := GCECloud{manager: fakeManager, managedZones: []string{"zone1", "zone2", "zone3"}} diskName := "disk" diskType := DiskTypeSSD const sizeGb int64 = 128 for _, zone := range gce.managedZones { gce.CreateDisk(diskName, diskType, zone, sizeGb, nil) } /* Act */ // DeleteDisk will call FakeServiceManager.GetDisk() with all zones, // and FakeServiceManager.GetDisk() always returns a disk, // so DeleteDisk thinks a disk with diskName exists in all zones. err := gce.DeleteDisk(diskName) /* Assert */ if err == nil { t.Error("Expected error when disk is found in multiple zones, but none returned.") } } func TestDeleteDisk_DiffDiskMultiZone(t *testing.T) { /* Arrange */ fakeManager := newFakeManager() gce := GCECloud{manager: fakeManager, managedZones: []string{"zone1"}} diskName := "disk" diskType := DiskTypeSSD const sizeGb int64 = 128 for _, zone := range gce.managedZones { diskName = zone + "disk" gce.CreateDisk(diskName, diskType, zone, sizeGb, nil) } /* Act & Assert */ var err error for _, zone := range gce.managedZones { diskName = zone + "disk" err = gce.DeleteDisk(diskName) if err != nil { t.Errorf("Error deleting disk in zone '%v'; error: "%v"", zone, err) } } } func TestGetAutoLabelsForPD_Basic(t *testing.T) { /* Arrange */ fakeManager := newFakeManager() diskName := "disk" diskType := DiskTypeSSD zone := "us-central1-c" const sizeGb int64 = 128 gce := GCECloud{manager: fakeManager, managedZones: []string{zone}} gce.CreateDisk(diskName, diskType, zone, sizeGb, nil) /* Act */ labels, err := gce.GetAutoLabelsForPD(diskName, zone) /* Assert */ if err != nil { t.Error(err) } if labels[metav1.LabelZoneFailureDomain] != zone { t.Errorf("Failure domain is '%v', but zone is '%v'", labels[metav1.LabelZoneFailureDomain], zone) } if labels[metav1.LabelZoneRegion] != "us-central1" { t.Errorf("Region is '%v', but zone is 'us-central1'", labels[metav1.LabelZoneRegion]) } } func TestGetAutoLabelsForPD_NoZone(t *testing.T) { /* Arrange */ fakeManager := newFakeManager() diskName := "disk" diskType := DiskTypeStandard zone := "europe-west1-d" const sizeGb int64 = 128 gce := GCECloud{manager: fakeManager, managedZones: []string{zone}} gce.CreateDisk(diskName, diskType, zone, sizeGb, nil) /* Act */ labels, err := gce.GetAutoLabelsForPD(diskName, "") /* Assert */ if err != nil { t.Error(err) } if labels[metav1.LabelZoneFailureDomain] != zone { t.Errorf("Failure domain is '%v', but zone is '%v'", labels[metav1.LabelZoneFailureDomain], zone) } if labels[metav1.LabelZoneRegion] != "europe-west1" { t.Errorf("Region is '%v', but zone is 'europe-west1'", labels[metav1.LabelZoneRegion]) } } func TestGetAutoLabelsForPD_DiskNotFound(t *testing.T) { /* Arrange */ fakeManager := newFakeManager() diskName := "disk" zone := "asia-northeast1-a" gce := GCECloud{manager: fakeManager, managedZones: []string{zone}} /* Act */ _, err := gce.GetAutoLabelsForPD(diskName, zone) /* Assert */ if err == nil { t.Error("Expected error when the specified disk does not exist, but none returned.") } } func TestGetAutoLabelsForPD_DiskNotFoundAndNoZone(t *testing.T) { /* Arrange */ fakeManager := newFakeManager() diskName := "disk" gce := GCECloud{manager: fakeManager, managedZones: []string{}} /* Act */ _, err := gce.GetAutoLabelsForPD(diskName, "") /* Assert */ if err == nil { t.Error("Expected error when the specified disk does not exist, but none returned.") } } func TestGetAutoLabelsForPD_DupDisk(t *testing.T) { /* Arrange */ fakeManager := newFakeManager() diskName := "disk" diskType := DiskTypeStandard zone := "us-west1-b" const sizeGb int64 = 128 gce := GCECloud{manager: fakeManager, managedZones: []string{"us-west1-b", "asia-southeast1-a"}} for _, zone := range gce.managedZones { gce.CreateDisk(diskName, diskType, zone, sizeGb, nil) } /* Act */ labels, err := gce.GetAutoLabelsForPD(diskName, zone) /* Assert */ if err != nil { t.Error("Disk name and zone uniquely identifies a disk, yet an error is returned.") } if labels[metav1.LabelZoneFailureDomain] != zone { t.Errorf("Failure domain is '%v', but zone is '%v'", labels[metav1.LabelZoneFailureDomain], zone) } if labels[metav1.LabelZoneRegion] != "us-west1" { t.Errorf("Region is '%v', but zone is 'us-west1'", labels[metav1.LabelZoneRegion]) } } func TestGetAutoLabelsForPD_DupDiskNoZone(t *testing.T) { /* Arrange */ fakeManager := newFakeManager() diskName := "disk" diskType := DiskTypeStandard const sizeGb int64 = 128 gce := GCECloud{manager: fakeManager, managedZones: []string{"us-west1-b", "asia-southeast1-a"}} for _, zone := range gce.managedZones { gce.CreateDisk(diskName, diskType, zone, sizeGb, nil) } /* Act */ _, err := gce.GetAutoLabelsForPD(diskName, "") /* Assert */ if err == nil { t.Error("Expected error when the disk is duplicated and zone is not specified, but none returned.") } } type FakeServiceManager struct { // Common fields shared among tests op                 *compute.Operation // Mocks an operation returned by GCE API calls doesOpMatch        bool disks              map[string]string // zone: diskName waitForZoneOpError error             // Error to be returned by WaitForZoneOp // Fields for TestCreateDisk createDiskCalled bool diskToCreate     *compute.Disk // Fields for TestDeleteDisk deleteDiskCalled bool resourceInUse    bool // Marks the disk as in-use } func newFakeManager() *FakeServiceManager { return &FakeServiceManager{disks: make(map[string]string)} } /** * Upon disk creation, disk info is stored in FakeServiceManager * to be used by other tested methods. */ func (manager *FakeServiceManager) CreateDisk( project string, zone string, disk *compute.Disk) (*compute.Operation, error) { manager.createDiskCalled = true op := &compute.Operation{} manager.op = op manager.diskToCreate = disk manager.disks[zone] = disk.Name return op, nil } /** * Gets disk info stored in the FakeServiceManager. */ func (manager *FakeServiceManager) GetDisk( project string, zone string, diskName string) (*compute.Disk, error) { if manager.disks[zone] == "" { return nil, cloudprovider.DiskNotFound } if manager.resourceInUse { errorItem := googleapi.ErrorItem{Reason: "resourceInUseByAnotherResource"} err := &googleapi.Error{Errors: []googleapi.ErrorItem{errorItem}} return nil, err } disk := &compute.Disk{Name: diskName, Zone: zone, Kind: "compute#disk"} return disk, nil } /** * Disk info is removed from the FakeServiceManager. */ func (manager *FakeServiceManager) DeleteDisk( project string, zone string, disk string) (*compute.Operation, error) { manager.deleteDiskCalled = true op := &compute.Operation{} manager.op = op manager.disks[zone] = "" return op, nil } func (manager *FakeServiceManager) WaitForZoneOp( op *compute.Operation, zone string, mc *metricContext) error { if op == manager.op { manager.doesOpMatch = true } return manager.waitForZoneOpError } </ins>

Unit tests are needed for the GCE cloud provider disk interface.
Assigned to
/assign
GitHub didn't allow me to assign the following users: verult. Note that only can be assigned.. <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. I understand the commands that are listed . </details


Question: } if err := d.writeCachedFile(filename, liveGroups); err != nil { <del> glog.V(3).Infof("failed to write cache to %v due to %v", filename, err) </del> <ins> glog.V(1).Infof("failed to write cache to %v due to %v", filename, err) </ins> } return liveGroups, nil

/kind bug What happened: kubectl became very slow, taking 30s to list pods etc. After going deep down the rabbit hole of debugging the controller components, I ran kubectl -v=8 which showed: What you expected to happen: Since caching is required for kubectl to respond in reasonable time and the issue isn't obvious, kubectl should at least print warnings in default log level. How to reproduce it (as minimally and precisely as possible): I got into the situation by running kubectl with sudo, but just chown'ing should be enough to reproduce this. Anything else we need to know?: Environment: kubectl v1.8.3
/sig cli
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale /lifecycle freeze


Question: import ( "fmt" "io" <ins> "time" </ins> "k8s.io/kubernetes/pkg/admission" "k8s.io/kubernetes/pkg/api"

We should have a single NamespaceLifecycle plugin that enforces the Namespace rules now that all providers are off NamespaceAutoProvision. We may also want to make this no longer a user choice to configure and hard-wire the server to always run this check first to simplify configuration errors. For an example


Question: t.lock.Lock() defer t.lock.Unlock() <ins> t.setHintsLocked(serviceKey, addrType, allocatedHintsByZone) } func (t *TopologyCache) setHintsLocked(serviceKey string, addrType discovery.AddressType, allocatedHintsByZone EndpointZoneInfo) { </ins> _, ok := t.endpointsByService[serviceKey] if !ok { t.endpointsByService[serviceKey] = map[discovery.AddressType]EndpointZoneInfo{}

I noticed that kube-controller-manager terminated with a concurrent map access: No concurrent map access, and no fatal error. No reproducer. I noticed this in an internal CI run. This appears to be an obvious (in retrospect) logic error with a straightforward solution, so I assume that it isn't necessary to provide a reproducer. This is from an OpenShift cluster based on Kubernetes 1.27. AWS. Red Hat Enterprise Linux CoreOS 414.92.-0. No response No response No response
/sig network
/assign
/triage accepted


Question: wantNotFound: true, }, { <ins> name: "deleted token", secrets: []*api.Secret{ { ObjectMeta: metav1.ObjectMeta{ Name:              bootstrapapi.BootstrapTokenSecretPrefix + tokenID, DeletionTimestamp: &now, }, Data: map[string][]byte{ bootstrapapi.BootstrapTokenIDKey:               []byte(tokenID), bootstrapapi.BootstrapTokenSecretKey:           []byte(tokenSecret), bootstrapapi.BootstrapTokenUsageAuthentication: []byte("true"), }, Type: "bootstrap.kubernetes.io/token", }, }, token:        tokenID + "." + tokenSecret, wantNotFound: true, }, { </ins> name: "expired token", secrets: []*api.Secret{ {

Resources can be deleted, but still exist in the API before they are reaped. They should not be valid during this time.
I'll add this to my pile. Thanks for filing :+1: FYI
FYI as well


Question: } } <ins> func TestBackupFinalizerRemoval(t *testing.T) { _, ctx := ktesting.NewTestContext(t) clientset := fake.NewSimpleClientset() sharedInformers := informers.NewSharedInformerFactory(clientset, controller.NoResyncPeriodFunc()) manager := NewController(ctx, sharedInformers.Core().V1().Pods(), sharedInformers.Batch().V1().Jobs(), clientset) manager.podStoreSynced = alwaysReady manager.jobStoreSynced = alwaysReady stopCh := make(chan struct{}) defer close(stopCh) podInformer := sharedInformers.Core().V1().Pods().Informer() go podInformer.Run(stopCh) cache.WaitForCacheSync(stopCh, podInformer.HasSynced) // 1. Create the controller but do not start the workers. // This is done above by initializing the manager and not calling manager.Run() yet. // 2. Create a job. job := newJob(1, 1, 1, batch.NonIndexedCompletion) _, err := clientset.BatchV1().Jobs(job.GetNamespace()).Create(ctx, job, metav1.CreateOptions{}) if err != nil { t.Fatalf("Creating job: %v", err) } pod := newPod("test-pod", job) // pod.Finalizers = append(pod.Finalizers, batch.JobTrackingFinalizer) _, err = clientset.CoreV1().Pods(pod.GetNamespace()).Create(ctx, pod, metav1.CreateOptions{}) if err != nil { t.Fatalf("Creating pod: %v", err) } job.Status.Conditions = append(job.Status.Conditions, batch.JobCondition{ Type:   batch.JobComplete, Status: v1.ConditionTrue, }) clientset.BatchV1().Jobs(job.GetNamespace()).UpdateStatus(ctx, job, metav1.UpdateOptions{}) go manager.Run(context.TODO(), 0) ctx, cancel := context.WithTimeout(context.Background(), wait.ForeverTestTimeout) defer cancel() if err := wait.PollUntilContextTimeout(ctx, 100*time.Millisecond, wait.ForeverTestTimeout, true, func(ctx context.Context) (bool, error) { p, err := clientset.CoreV1().Pods(pod.Namespace).Get(ctx, pod.Name, metav1.GetOptions{}) if err != nil { return false, err } return !hasJobTrackingFinalizer(p), nil }); err != nil { // Check for context deadline exceeded, which would mean the operation timed out if errors.Is(err, context.DeadlineExceeded) { t.Errorf("Timed out waiting for Pod to get the finalizer removed") } else { t.Errorf("Waiting for Pod to get the finalizer removed: %v", err) } } } </ins> func checkJobCompletionLabel(t *testing.T, p *v1.PodTemplateSpec) { t.Helper() labels := p.GetLabels()

When a Job is deleted (via Background cascade), we have a backup logic to remove finalizers from pods This logic could also be applied for any finished Job. While investigating a case similar to , we observed that the job controller didn't delete finalizers from Pods where the Job was deleted with Foreground cascading policy. Since the Job was already marked as Finished, it only makes sense to trigger the backup logic to remove finalizers, so that we can guard against any possible bug where a Job is marked as Finished but not all pod finalizers are removed. Note that I'm not aware of any other bug in tracking with finalizers.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig apps
/remove-kind feature /kind cleanup
/wg batch
I am happy to give this a go.
/assign
Just to check if I have the logic chain right: We are talking about adding this logic If so, I should probably turn the backup-logic code into a function that then apply that in both places? I also would be unsure of where in the unit tests this would be tested, are you able to point me to how this is tested in the deletePod function? I assume we add a pod that is never included in the "uncounted" lists.
I would put it in the event handler, to be more of a "catch all" Yes, that's a good idea. I think you would have to force it quite a bit, as we have fixed all the bugs that we have found. For example, from the test: the controller, but do not start the workers (this will setup the event handlers) a job the pods the job the workers some inspiration here
Alright, thank you :))
Should the issue title be: Remove finalizers from Pods belonging to finished Jobs ?
If someone has set a custom finalizer on a Pod, even one owned by a Job, (when) would we remove that finalizer?
/retitle Remove tracking finalizers from Pods belonging to finished Jobs No, only the finalizer specific to job tracking


Question: pluginNotImplementingScore        = "plugin-not-implementing-score" preFilterPluginName               = "prefilter-plugin" preFilterWithExtensionsPluginName = "prefilter-with-extensions-plugin" <ins> duplicatePluginName               = "duplicate-plugin" </ins> ) // TestScoreWithNormalizePlugin implements ScoreWithNormalizePlugin interface.

There is no reason this should be allowed to happen, especially that we can't configure duplicate plugins differently since plugin configs is a separate list . /sig scheduling
/label help-wanted
The label(s) cannot be applied. These labels are supported: <detailsIn response to : /label help-wanted Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/help-wanted
/help /good-first-issue
/assign
/remove-good-first-issue /remove-help


Question: return errs } <del> // ApplyPolicySourceTo sets cfg.PolicySource from flags passed on the command line in the following precedence order: </del> <ins> // ApplyTo sets cfg.PolicySource from flags passed on the command line in the following precedence order: </ins> // // 1. --use-legacy-policy-config to use a policy file. // 2. --policy-configmap to use a policy config map value.

ci-kubernetes-e2e-gci-gce-alpha-features kubetest.Up Jul 02 2021, 01:26:42 UTC+0200 /sig release
based on testgrid probably cc <img width="1203" alt="Screen Shot 2021-07-02 at 4 24 25 PM" src="https://user-"
/cc working on it
/assign
the scheduler fails to start with this error: it seems the feature gates from the command line does not take effect in the defaulting phrase, do you know how to fix it?
ah, found there is a cycle here: The scheduler options , but the default config's default values depend on the feature gates options. I think this disables other feature gate controlled logic in the defaulting too:
I tried to fix it by updating the component config after the CLI flags (including the feature gates) are parsed: PTAL
fixed the flake. is back green now.
thanks!


Question: "k8s.io/klog" "k8s.io/utils/mount" <del> v1 "k8s.io/api/core/v1" </del> "k8s.io/kubernetes/pkg/volume" volumeutil "k8s.io/kubernetes/pkg/volume/util" )

What happened: An iSCSI PV with is mounted without these options. What you expected to happen: The volume is mounted with . Anything else we need to know?: iSCSI volume is mounted in and passes as : This can be fixed trivially, however, why is it mounted in at all? There is another mount attempt in , with correct mount options: This code is skipped, because the volume is already mounted. Environment: Kubernetes version (use ): today's master, v1.19.0-alpha.0.781+-dirty , do you remember why we mount stuff in ? iSCSI and FibreChannel does so, the others don't.
/sig storage


Question: for _, na := range expectedTree { expectedNumNodes += len(na.nodes) } <del> if nt.NumNodes != expectedNumNodes { t.Errorf("unexpected NodeTree.numNodes. Expected: %v, Got: %v", expectedNumNodes, nt.NumNodes) </del> <ins> if numNodes := nt.NumNodes(); numNodes != expectedNumNodes { t.Errorf("unexpected NodeTree.numNodes. Expected: %v, Got: %v", expectedNumNodes, numNodes) </ins> } if !reflect.DeepEqual(nt.tree, expectedTree) { t.Errorf("The node tree is not the same as expected. Expected: %v, Got: %v", expectedTree, nt.tree)

Description: , part of the is exporting a field which allows to modify its value externally. This also means that this field is not thread-safe. Solution: Expose a method that returns this value in a thread-safe way and do not export the field. /kind bug /sig scheduling


Question: } var fileDescriptor_66649ee9bbcd89d2 = []byte{ <del> // 2095 bytes of a gzipped FileDescriptorProto </del> <ins> // 2085 bytes of a gzipped FileDescriptorProto </ins> 0x1f, 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0xff, 0xd4, 0x1a, 0xcd, 0x6f, 0x1c, 0x57, <del> 0xdd, 0xb3, 0xeb, 0xcf, 0x9f, 0xbf, 0x9f, 0x93, 0xe0, 0x9a, 0x66, 0x37, 0x99, 0x22, 0x70, 0xda, 0x74, 0xb7, 0x71, 0x4b, 0x1b, 0x5a, 0x90, 0xf0, 0xd8, 0x6e, 0xe4, 0x28, 0x1f, 0xce, 0xdb, 0x26, </del> <ins> 0x3d, 0xb3, 0xeb, 0xcf, 0x9f, 0xbf, 0x9f, 0x93, 0xe0, 0x9a, 0x66, 0x37, 0x99, 0x22, 0x70, 0xda, 0x74, 0xb6, 0x71, 0x4b, 0x1b, 0x5a, 0x90, 0xf0, 0xd8, 0x6e, 0xe4, 0x28, 0x1f, 0xce, 0xdb, 0x26, </ins> 0x34, 0x50, 0x4a, 0x9f, 0x67, 0x9f, 0xed, 0xc1, 0xb3, 0x33, 0xd3, 0x99, 0x37, 0x26, 0x16, 0x12, 0x8a, 0xb8, 0x70, 0x8b, 0x7a, 0xe5, 0x80, 0xb8, 0x21, 0x21, 0x0e, 0x70, 0xe0, 0x88, 0x54, 0x24, <del> 0x90, 0xc8, 0x31, 0x08, 0x0e, 0x3d, 0x2d, 0xc9, 0x22, 0xfe, 0x89, 0x5c, 0x40, 0xef, 0xcd, 0x9b, 0xcf, 0x9d, 0xd9, 0xcc, 0x46, 0x91, 0x45, 0x6f, 0x9e, 0xdf, 0xf7, 0xfb, 0x7d, 0xbf, 0xb7, 0x86, 0x8b, 0x87, 0x97, 0xbd, 0x86, 0x61, 0x37, 0x89, 0x63, 0x34, 0x5d, 0xea, 0xd9, 0xbe, 0xab, 0xd3, 0xe6, 0xd1, 0x25, 0x62, 0x3a, 0x07, 0xe4, 0xcd, 0xe6, 0x3e, 0xb5, 0xa8, 0x4b, 0x18, 0x6d, 0x37, 0x1c, 0xd7, 0x66, 0x36, 0x7a, 0x39, 0xa0, 0x6e, 0x10, 0xc7, 0x68, 0x84, 0xd4, 0x8d, 0x90, 0x7a, 0xe5, 0xf5, 0x7d, 0x83, 0x1d, 0xf8, 0xbb, 0x0d, 0xdd, 0xee, 0x34, 0xf7, 0xed, 0x7d, 0xbb, 0x29, 0x98, 0x76, 0xfd, 0x3d, 0xf1, 0x25, 0x3e, 0xc4, 0x5f, 0x81, 0xb0, 0x15, 0x35, 0xa1, 0x5a, 0xb7, 0x5d, 0xae, 0x36, 0xab, 0x70, 0xe5, 0xad, 0x98, 0xa6, 0x43, 0xf4, 0x03, 0xc3, 0xa2, 0xee, 0x71, 0xd3, 0x39, 0xdc, 0x4f, 0xdb, 0x3b, 0x0c, 0x97, 0xd7, 0xec, 0x50, 0x46, 0xf2, 0x74, 0x35, 0x8b, 0xb8, 0x5c, 0xdf, 0x62, 0x46, 0xa7, 0x5f, 0xcd, 0xdb, 0xcf, 0x62, 0xf0, 0xf4, 0x03, 0xda, 0x21, 0x59, 0x3e, 0xf5, 0xbf, 0x0a, 0x2c, 0xac, 0x9b, 0xa6, 0xad, 0x13, 0x66, 0xd8, 0x16, 0xa6, 0x9e, 0x6f, 0x32, 0xf4, 0x23, 0x98, 0x68, 0xd3, 0x23, 0x43, 0xa7, 0xde, 0xb2, 0x72, 0x4e, 0x59, 0x9d, 0x5e, 0x7b, 0xab, 0x31, 0xc8, 0xd9, 0x8d, 0x4d, 0x41, 0x9c, 0x15, 0xa3, 0xcd, 0x3f, 0xec, 0xd6, 0x47, 0x7a, 0xdd, 0xfa, 0x44, 0x80, 0xf7, 0x70, 0x28, 0x15, 0xdd, 0x81, 0x19, 0xcb, 0x6e, 0xd3, 0x16, 0x35, 0xa9, 0xce, 0x6c, 0x77, 0xb9, 0x2a, 0xb4, 0x9c, 0x4b, 0x6a, 0xe1, 0x51, 0x68, 0x1c, 0x5d, 0x6a, 0xdc, 0x48, 0xd0, 0x69, 0x0b, 0xbd, 0x6e, 0x7d, 0x26, 0x09, 0xc1, 0x29, 0x39, 0x68, 0x0d, 0x40, 0xb7, 0x2d, 0xe6, 0xda, 0xa6, 0x49, 0xdd, 0xe5, 0xd1, 0x73, 0xca, 0xea, 0x94, 0x86, 0xa4, 0x15, 0xb0, 0x11, 0x61, 0x70, 0x82, 0x4a, 0x7d, 0x5c, 0x85, 0x69, 0x8d, 0x78, 0x86, 0x1e, 0x58, 0x89, 0x7e, 0x06, 0x40, 0x18, 0x73, 0x8d, 0x5d, 0x9f, 0x89, 0xf3, 0x57, 0x57, 0xa7, 0xd7, 0xbe, 0x35, 0xf8, 0xfc, 0x09, 0xf6, 0xc6, 0x7a, 0xc4, 0xbb, 0x65, 0x31, 0xf7, 0x58, 0x7b, 0x25, 0x54, 0x1f, 0x23, 0x7e, 0xfe, 0xaf, 0xfa, 0xec, 0x2d, 0x9f, 0x98, 0xc6, 0x9e, 0x41, 0xdb, 0x37, 0x48, 0x87, 0xe2, 0x84, 0x46, 0x74, 0x04, 0x93, 0x3a, 0x71, 0x88, 0x6e, 0xb0, 0xe3, 0xe5, 0x8a, 0xd0, 0xfe, 0x4e, 0x79, 0xed, 0x1b, 0x92, 0x33, 0xd0, 0x7d, 0x5e, 0xea, 0x9e, 0x0c, 0xc1, 0xfd, 0x9a, 0x23, 0x5d, 0x2b, 0x26, 0xcc, 0x67, 0x6c, 0x47, 0x0b, 0x50, 0x3d, 0xa4, 0xc7, 0x22, 0x07, 0xa6, 0x30, 0xff, 0x13, 0x6d, 0xc0, 0xd8, 0x11, 0x31, 0x7d, 0xba, 0x5c, 0x11, 0x11, 0x7b, 0xbd, 0x54, 0x5e, 0x84, 0x52, 0x71, 0xc0, 0xfb, 0x6e, 0xe5, 0xb2, 0xb2, 0x72, 0x08, 0xb3, 0x29, 0x5b, 0x73, 0x74, 0x6d, 0xa6, 0x75, 0x35, 0x12, 0xba, 0xa2, 0x14, 0x6f, 0x38, 0x87, 0xfb, 0x69, 0xe5, 0xb7, 0x7c, 0x62, 0x31, 0x83, 0x1d, 0x27, 0x94, 0xa9, 0x57, 0x60, 0x71, 0x63, 0xeb, 0x5a, 0x60, 0x4d, 0x32, 0x57, 0xe8, 0x3d, 0xc7, 0xa5, 0x9e, 0x67, 0xd8, 0x56, 0xa0, 0x37, 0xce, 0x95, 0xad, 0x08, 0x83, 0x13, 0x54, 0xea, 0x11, 0x8c, 0xcb, 0x2c, 0x39, 0x07, 0xa3, 0x16, 0xe9, 0x50, 0xc9, 0x37, 0x23, 0xf9, 0x46, 0x85, 0x4f, 0x05, 0x06, 0x5d, 0x85, 0xb1, 0x5d, 0x1e, 0x19, 0x69, 0xfe, 0x85, 0xd2, 0x41, 0xd4, 0xa6, 0x7a, 0xdd, 0xfa, 0x98, 0x00, 0xe0, 0x40, 0x84, 0xfa, 0xa0, 0x02, 0x67, 0xb3, 0x45, 0xb6, 0x61, 0x5b, 0x7b, 0xc6, 0xbe, 0xef, 0x8a, 0x0f, 0xf4, 0x5d, 0x18, 0x0f, 0x44, 0x4a, 0x8b, 0x56, 0xa5, 0x45, 0xe3, 0x2d, 0x01, 0x7d, 0xda, 0xad, 0x9f, 0xc9, 0xb2, 0x06, 0x18, 0x2c, 0xf9, 0xd0, 0x2a, 0x4c, 0xba, 0xf4, 0x53, 0x9f, 0x7a, 0xcc, 0x13, 0x79, 0x37, 0xa5, 0xcd, 0xf0, 0xd4, 0xc1, 0x12, 0x86, 0x23, 0x2c, 0xba, 0xaf, 0xc0, 0x52, 0x50, 0xc9, 0x29, 0x1b, 0x64, 0x15, 0x5f, 0x2a, 0x93, 0x13, 0x29, 0x46, 0xed, 0xab, 0xd2, 0xd8, 0xa5, 0x1c, 0x24, 0xce, 0x53, 0xa5, 0xfe, 0x47, 0x81, 0x33, 0xf9, 0x5d, 0x07, 0xed, 0xc1, 0x84, 0x2b, 0xfe, 0x0a, 0x8b, 0xf7, 0xbd, 0x32, 0x06, 0xc9, 0x63, 0x16, 0xf7, 0xb0, 0xe0, 0xdb, 0xc3, 0xa1, 0x70, 0xa4, 0xc3, 0xb8, 0x2e, 0x6c, 0x92, 0x55, 0xfa, 0xde, 0x70, 0x3d, 0x32, 0xed, 0x81, 0xb9, 0x30, 0x5c, 0x01, 0x18, 0x4b, 0xd1, 0xea, 0x6f, 0x15, 0x98, 0xcf, 0x54, 0x11, 0xaa, 0x41, 0xd5, 0xb0, 0x98, 0x48, 0xab, 0x6a, 0x10, 0xa3, 0x6d, 0x8b, 0xdd, 0xe1, 0xc9, 0x8e, 0x39, 0x02, 0x9d, 0x87, 0xd1, 0x5d, 0xdb, 0x36, 0x45, 0x38, 0x26, 0xb5, 0xd9, 0x5e, 0xb7, 0x3e, 0xa5, 0xd9, 0xb6, 0x19, 0x50, 0x08, 0x14, 0xfa, 0x06, 0x8c, 0x7b, 0xcc, 0x35, 0xac, 0x7d, 0xd9, 0x23, 0xe7, 0x7b, 0xdd, 0xfa, 0x74, 0x4b, 0x40, 0x02, 0x32, 0x89, 0x46, 0xaf, 0xc2, 0xc4, 0x11, 0x75, 0x45, 0x85, 0x8c, 0x09, 0x4a, 0xd1, 0x81, 0xef, 0x04, 0xa0, 0x80, 0x34, 0x24, 0x50, 0x7f, 0x5f, 0x81, 0x69, 0x19, 0x40, 0x93, 0x18, 0x1d, 0x74, 0x37, 0x91, 0x50, 0x41, 0x24, 0x5e, 0x1b, 0x22, 0x12, 0xda, 0x42, 0xd8, 0xbc, 0x72, 0x32, 0x90, 0xc2, 0xb4, 0x6e, 0x5b, 0x1e, 0x73, 0x89, 0x61, 0xc9, 0x74, 0x4d, 0x37, 0x88, 0x41, 0x89, 0x27, 0xd9, 0xb4, 0x25, 0xa9, 0x60, 0x3a, 0x86, 0x79, 0x38, 0x29, 0x17, 0x7d, 0x1c, 0x85, 0xb8, 0x2a, 0x34, 0xbc, 0x5d, 0x4a, 0x03, 0x3f, 0x7c, 0xb9, 0xe8, 0xfe, 0x4d, 0x81, 0xe5, 0x22, 0xa6, 0x54, 0x3d, 0x2a, 0xcf, 0x55, 0x8f, 0x95, 0x93, 0xab, 0xc7, 0x3f, 0x2b, 0x89, 0xd8, 0x7b, 0x1e, 0xfa, 0x04, 0x26, 0xf9, 0x6a, 0xd3, 0x26, 0x8c, 0xc8, 0x15, 0xe2, 0x8d, 0x41, 0xed, 0xdb, 0x6b, 0x70, 0x6a, 0x3e, 0xee, 0x6f, 0xee, 0xfe, 0x98, 0xea, 0xec, 0x3a, 0x65, 0x24, 0x6e, 0xc6, 0x31, 0x0c, 0x47, 0x52, 0xd1, 0x4d, 0x18, 0xf5, 0x1c, 0xaa, 0x0f, 0x33, 0x88, 0x84, 0x69, 0x2d, 0x87, 0xea, 0x71, 0xbf, 0xe6, 0x5f, 0x58, 0x08, 0x52, 0x7f, 0x95, 0x0c, 0x86, 0xe7, 0xa5, 0x83, 0x51, 0xe4, 0x62, 0xe5, 0xe4, 0x5c, 0xfc, 0x79, 0xd4, 0x0a, 0x84, 0x7d, 0xd7, 0x0c, 0x8f, 0xa1, 0x8f, 0xfa, 0xdc, 0xdc, 0x28, 0xe7, 0x66, 0xce, 0x2d, 0x9c, 0x1c, 0x55, 0x59, 0x08, 0x49, 0xb8, 0xf8, 0x06, 0x8c, 0x19, 0x8c, 0x76, 0xc2, 0xfa, 0xba, 0x50, 0xda, 0xc7, 0xda, 0xac, 0x94, 0x3a, 0xb6, 0xcd, 0xf9, 0x71, 0x20, 0x46, 0xfd, 0x5d, 0x25, 0x75, 0x02, 0xee, 0x7b, 0xf4, 0x43, 0x98, 0xf2, 0xe4, 0x44, 0x0e, 0xbb, 0xc4, 0xc5, 0x32, 0x7a, 0xa2, 0x95, 0x70, 0x51, 0xaa, 0x9a, 0x0a, 0x21, 0x1e, 0x8e, 0x25, 0x26, 0x2a, 0xb8, 0x32, 0x54, 0x05, 0x67, 0xe2, 0x5f, 0x54, 0xc1, 0xe8, 0x2e, 0xcc, 0x7a, 0xbe, 0xc1, 0xc8, 0xae, 0x49, 0xf9, 0x5a, 0xea, 0x95, 0xde, 0x64, 0x17, 0x7b, 0xdd, 0xfa, 0x6c, 0x2b, 0xc9, 0x8a, 0xd3, 0x92, 0x54, 0x17, 0xf2, 0x72, 0x03, 0xfd, 0x00, 0xc6, 0x6d, 0x87, 0x7c, 0xea, 0x53, 0x19, 0xf0, 0x67, 0x2c, 0x87, 0x37, 0x05, 0x6d, 0x5e, 0x06, 0x02, 0x3f, 0x4e, 0x80, 0xc6, 0x52, 0xa4, 0xfa, 0x40, 0x81, 0x85, 0x6c, 0x9f, 0x1c, 0xa2, 0x11, 0xed, 0xc0, 0x5c, 0x87, 0x30, 0xfd, 0x20, 0x9a, 0x55, 0xa2, 0x3a, 0xa7, 0xb4, 0xd5, 0x5e, 0xb7, 0x3e, 0x77, 0x3d, 0x85, 0x79, 0xda, 0xad, 0xa3, 0xf7, 0x7d, 0xd3, 0x3c, 0x4e, 0xaf, 0xa3, 0x19, 0x7e, 0xf5, 0x17, 0x55, 0x98, 0x4d, 0x8d, 0x85, 0x12, 0x8b, 0xd7, 0x3a, 0xcc, 0xb7, 0xe3, 0x38, 0x72, 0x84, 0x34, 0xe3, 0x2b, 0x92, 0x38, 0x99, 0x84, 0x82, 0x2f, 0x4b, 0x9f, 0xce, 0xca, 0xea, 0x0b, 0xcf, 0xca, 0x3b, 0x30, 0x47, 0xa2, 0x45, 0xe0, 0xba, 0xdd, 0xa6, 0x72, 0x0c, 0x37, 0x24, 0xd7, 0xdc, 0x7a, 0x0a, 0xfb, 0xb4, 0x5b, 0x3f, 0x95, 0x5d, 0x1f, 0x38, 0x1c, 0x67, 0xa4, 0xa0, 0x57, 0x60, 0x4c, 0xb7, 0x7d, 0x8b, 0x89, 0x59, 0x5d, 0x8d, 0xab, 0x70, 0x83, 0x03, 0x71, 0x80, 0x43, 0xdf, 0x84, 0x69, 0xd2, 0xee, 0x18, 0xd6, 0xba, 0xae, 0x53, 0xcf, 0x5b, 0x1e, 0x17, 0x5b, 0x42, 0x34, 0x0b, 0xd7, 0x63, 0x14, 0x4e, 0xd2, 0xa9, 0x7f, 0x52, 0xc2, 0x15, 0xb4, 0x60, 0x55, 0x42, 0x17, 0xf8, 0xe2, 0x25, 0x50, 0x32, 0x38, 0x89, 0xdd, 0x49, 0x80, 0x71, 0x88, 0x47, 0x5f, 0x87, 0xf1, 0xb6, 0x6b, 0x1c, 0x51, 0x57, 0x46, 0x26, 0x2a, 0xaf, 0x4d, 0x01, 0xc5, 0x12, 0xcb, 0x83, 0xed, 0x84, 0xab, 0x4c, 0x22, 0xd8, 0x3b, 0xb6, 0x6d, 0x62, 0x81, 0x11, 0x92, 0x84, 0x55, 0xd2, 0x85, 0xb1, 0xa4, 0xc0, 0x56, 0x89, 0x55, 0x3f, 0x82, 0xb9, 0xcc, 0xfe, 0x7f, 0x15, 0xaa, 0x3a, 0x35, 0x65, 0x15, 0x35, 0x07, 0x47, 0xb7, 0xef, 0xf6, 0xa0, 0x4d, 0xf4, 0xba, 0xf5, 0xea, 0xc6, 0xd6, 0x35, 0xcc, 0x85, 0xa8, 0xbf, 0x51, 0xe0, 0xa5, 0xc2, 0x4a, 0x4b, 0x9c, 0x56, 0x19, 0x78, 0x5a, 0x02, 0xe0, 0x10, 0x97, 0x74, 0x28, 0xa3, 0xae, 0x97, 0x33, 0xd8, 0xd2, 0xfd, 0x5c, 0x5e, 0xec, 0x1b, 0x98, 0xfc, 0x64, 0xeb, 0x1e, 0xa3, 0x16, 0xdf, 0xc1, 0xe2, 0x99, 0xb9, 0x13, 0x09, 0xc2, 0x09, 0xa1, 0xea, 0x1f, 0x2b, 0x70, 0x6a, 0xc7, 0x6e, 0xb7, 0xf4, 0x03, 0xda, 0xf6, 0x4d, 0xc3, 0xda, 0xe7, 0x97, 0x62, 0x7a, 0x8f, 0x9d, 0xc0, 0xc0, 0xfe, 0x30, 0x35, 0xb0, 0x9f, 0xd1, 0x88, 0xf3, 0x6c, 0x2c, 0x9a, 0xdc, 0xe8, 0x13, 0xbe, 0xcd, 0x12, 0xe6, 0x87, 0xdd, 0xf7, 0xf2, 0x73, 0xc8, 0x16, 0xfc, 0x71, 0x64, 0x82, 0x6f, 0x2c, 0xe5, 0xaa, 0x7f, 0x57, 0x60, 0x39, 0x8f, 0xed, 0x04, 0x86, 0xf0, 0xf7, 0xd2, 0x43, 0x78, 0x6d, 0xf8, 0xb3, 0x15, 0x4c, 0xe3, 0xcf, 0x0a, 0xce, 0x24, 0xc6, 0xf2, 0x65, 0x98, 0x09, 0xda, 0x15, 0x6d, 0xf3, 0x69, 0x24, 0x13, 0xf7, 0x94, 0x14, 0x34, 0xd3, 0x4a, 0xe0, 0x70, 0x8a, 0x12, 0xbd, 0x0b, 0x73, 0x8e, 0xcd, 0xa8, 0xc5, 0x0c, 0x62, 0x06, 0x23, 0x31, 0xb8, 0x4c, 0x22, 0xde, 0xd7, 0x76, 0x52, 0x18, 0x9c, 0xa1, 0x54, 0x7f, 0xa9, 0xc0, 0x4a, 0x71, 0x74, 0xd0, 0x4f, 0x61, 0x2e, 0x3c, 0xb1, 0xd8, 0x97, 0x4b, 0x5e, 0xf0, 0x70, 0x92, 0x27, 0x96, 0x2d, 0x43, 0x7e, 0x26, 0xec, 0xb9, 0x29, 0x32, 0x0f, 0x67, 0x54, 0xa9, 0xbf, 0xae, 0xc0, 0x6c, 0x8a, 0xe4, 0x04, 0x4a, 0xe6, 0x56, 0xaa, 0x64, 0x9a, 0xc3, 0x1c, 0xb3, 0xa8, 0x56, 0xee, 0x66, 0x6a, 0xe5, 0xd2, 0x30, 0x42, 0x07, 0x17, 0x49, 0x4f, 0x81, 0x5a, 0x8a, 0x9e, 0xef, 0x10, 0x7e, 0x87, 0xba, 0x98, 0xee, 0x51, 0x97, 0x5a, 0x3a, 0x45, 0x17, 0x61, 0x92, 0x38, 0xc6, 0x15, 0xd7, 0xf6, 0x1d, 0x99, 0x52, 0x51, 0xea, 0xaf, 0xef, 0x6c, 0x0b, 0x38, 0x8e, 0x28, 0x38, 0x75, 0x68, 0x91, 0x9c, 0x00, 0x89, 0x3b, 0x61, 0x00, 0xc7, 0x11, 0x45, 0xb4, 0x18, 0x8c, 0x16, 0x2e, 0x06, 0x1a, 0x54, 0x7d, 0xa3, 0x2d, 0x2f, 0xb2, 0x6f, 0x48, 0x82, 0xea, 0xed, 0xed, 0xcd, 0xa7, 0xdd, 0xfa, 0xf9, 0xa2, 0xf7, 0x53, 0x76, 0xec, 0x50, 0xaf, 0x71, 0x7b, 0x7b, 0x13, 0x73, 0x66, 0xf5, 0x2f, 0x0a, 0x2c, 0xa6, 0x0e, 0x79, 0x02, 0x2d, 0x60, 0x27, 0xdd, 0x02, 0x5e, 0x1b, 0x22, 0x64, 0x05, 0xb5, 0x7f, 0x5f, 0x81, 0xb3, 0x03, 0xcb, 0xa2, 0xc4, 0x9a, 0xf5, 0x1d, 0x98, 0xf7, 0xad, 0xf4, 0xf2, 0x1b, 0x54, 0xfa, 0x12, 0x5f, 0xb1, 0x6e, 0xa7, 0x51, 0x38, 0x4b, 0xcb, 0xaf, 0x5b, 0x8b, 0x7d, 0x29, 0x8b, 0x3e, 0xc8, 0xbe, 0x3c, 0x5f, 0x28, 0x7d, 0xe5, 0x1e, 0xf0, 0xdc, 0x9c, 0x7e, 0x16, 0xae, 0x94, 0x7a, 0x16, 0xfe, 0xbc, 0x02, 0x4b, 0x39, 0xd9, 0x8f, 0x3e, 0x06, 0x88, 0xb7, 0xae, 0x9c, 0x60, 0xe7, 0x18, 0xd9, 0xf7, 0xa8, 0x34, 0x27, 0xde, 0x83, 0x63, 0x68, 0x42, 0x22, 0xf2, 0x60, 0xda, 0xa5, 0x1e, 0x75, 0x8f, 0x68, 0xfb, 0x7d, 0xdb, 0x95, 0x21, 0xff, 0xf6, 0x10, 0x21, 0xef, 0xab, 0xba, 0x78, 0xb9, 0xc3, 0xb1, 0x60, 0x9c, 0xd4, 0x82, 0x5a, 0x70, 0xba, 0x4d, 0x49, 0xc2, 0x4c, 0xb1, 0xa6, 0xd1, 0xb6, 0x7c, 0x43, 0x3a, 0x2b, 0x05, 0x9c, 0xde, 0xcc, 0x23, 0xc2, 0xf9, 0xbc, 0xea, 0x3f, 0x15, 0x38, 0x9d, 0xb2, 0xec, 0x03, 0xda, 0x71, 0x4c, 0xc2, 0xe8, 0x09, 0x74, 0xce, 0xbb, 0xa9, 0xce, 0xf9, 0xce, 0x10, 0xee, 0x0b, 0x8d, 0x2c, 0x7c, 0x27, 0xf8, 0x87, 0x02, 0x2f, 0xe5, 0x72, 0x9c, 0x40, 0x27, 0xf8, 0x30, 0xdd, 0x09, 0xde, 0x7c, 0x8e, 0x73, 0x15, 0x74, 0x84, 0x47, 0x45, 0xa7, 0x6a, 0x05, 0x1b, 0xd6, 0x97, 0x6f, 0xd4, 0xa9, 0x7f, 0x50, 0x60, 0x26, 0xa4, 0xe4, 0x37, 0x86, 0x12, 0x3d, 0x6d, 0x0d, 0x40, 0xfe, 0x40, 0x16, 0xbe, 0x9f, 0x55, 0x63, 0xbb, 0xaf, 0x44, 0x18, 0x9c, 0xa0, 0x42, 0x57, 0x01, 0x85, 0x16, 0xb6, 0x4c, 0xb1, 0xfb, 0xf3, 0x1b, 0x58, 0x55, 0xf0, 0xae, 0x48, 0x5e, 0x84, 0xfb, 0x28, 0x70, 0x0e, 0x97, 0xfa, 0x57, 0x25, 0x5e, 0x32, 0x04, 0xf8, 0xff, 0xd5, 0xf3, 0xc2, 0xb8, 0x42, 0xcf, 0x27, 0x87, 0xa4, 0xa0, 0x0c, 0x4b, 0xc3, 0x94, 0x29, 0xfd, 0xe2, 0x4a, 0x23, 0x94, 0xf8, 0x9c, 0x43, 0x52, 0x58, 0x57, 0x50, 0x12, 0x0f, 0xaa, 0x99, 0x53, 0x88, 0x52, 0x28, 0x7b, 0x99, 0xbb, 0x26, 0xaf, 0xae, 0x81, 0x5b, 0x5f, 0x2d, 0x67, 0x0e, 0x4f, 0xd3, 0xdc, 0x6b, 0xee, 0x45, 0x98, 0xb4, 0xec, 0x36, 0x15, 0x8f, 0x19, 0x99, 0x55, 0xe8, 0x86, 0x84, 0xe3, 0x88, 0xa2, 0xef, 0xe7, 0xd5, 0xd1, 0x17, 0xf4, 0xf3, 0x2a, 0x5f, 0xdf, 0x4c, 0xb9, 0xd5, 0x8f, 0x89, 0xc9, 0x10, 0xaf, 0x6f, 0x12, 0x8e, 0x23, 0x0a, 0x74, 0x33, 0x9e, 0xe5, 0xe3, 0x22, 0x26, 0x5f, 0x2b, 0x33, 0xcb, 0x8b, 0xc7, 0xb8, 0xa6, 0x3d, 0x7c, 0x52, 0x1b, 0x79, 0xf4, 0xa4, 0x36, 0xf2, 0xc5, 0x93, 0xda, 0xc8, 0xfd, 0x5e, 0x4d, 0x79, 0xd8, 0xab, 0x29, 0x8f, 0x7a, 0x35, 0xe5, 0x8b, 0x5e, 0x4d, 0x79, 0xdc, 0xab, 0x29, 0x9f, 0xfd, 0xbb, 0x36, 0xf2, 0xfd, 0x97, 0x07, 0xfd, 0x17, 0xc1, 0xff, 0x02, 0x00, 0x00, 0xff, 0xff, 0x4d, 0x8f, 0x9b, 0x77, 0x64, 0x20, 0x00, 0x00, </del> <ins> 0x90, 0xc8, 0x31, 0x08, 0x0e, 0x3d, 0x2d, 0xcd, 0x22, 0xfe, 0x89, 0x5c, 0x40, 0xef, 0xcd, 0x9b, 0x4f, 0xcf, 0x6c, 0x66, 0xa3, 0xca, 0x2a, 0x37, 0xcf, 0xef, 0xfb, 0xfd, 0xbe, 0xdf, 0x5b, 0xc3, 0xa5, 0x83, 0x2b, 0xbe, 0x66, 0x3a, 0x2d, 0xe2, 0x9a, 0x2d, 0x8f, 0xfa, 0x4e, 0xe0, 0x19, 0xb4, 0x75, 0x78, 0x99, 0x58, 0xee, 0x3e, 0x79, 0xbd, 0xb5, 0x47, 0x6d, 0xea, 0x11, 0x46, 0x3b, 0x9a, 0xeb, 0x39, 0xcc, 0x41, 0x2f, 0x86, 0xd4, 0x1a, 0x71, 0x4d, 0x2d, 0xa2, 0xd6, 0x22, 0xea, 0xe5, 0x57, 0xf7, 0x4c, 0xb6, 0x1f, 0xec, 0x68, 0x86, 0xd3, 0x6d, 0xed, 0x39, 0x7b, 0x4e, 0x4b, 0x30, 0xed, 0x04, 0xbb, 0xe2, 0x4b, 0x7c, 0x88, 0xbf, 0x42, 0x61, 0xcb, 0x6a, 0x4a, 0xb5, 0xe1, 0x78, 0x5c, 0x6d, 0x5e, 0xe1, 0xf2, 0x1b, 0x09, 0x4d, 0x97, 0x18, 0xfb, 0xa6, 0x4d, 0xbd, 0xa3, 0x96, 0x7b, 0xb0, 0x97, 0xb5, 0x77, 0x18, 0x2e, 0xbf, 0xd5, 0xa5, 0x8c, 0x14, 0xe9, 0x6a, 0x95, 0x71, 0x79, 0x81, 0xcd, 0xcc, 0xee, 0x71, 0x35, 0x6f, 0x3e, 0x8b, 0xc1, 0x37, 0xf6, 0x69, 0x97, 0xe4, 0xf9, 0xd4, 0xff, 0x2a, 0x30, 0xbf, 0x66, 0x59, 0x8e, 0x41, 0x98, 0xe9, 0xd8, 0x98, 0xfa, 0x81, 0xc5, 0xd0, 0x8f, 0x60, 0xbc, 0x43, 0x0f, 0x4d, 0x83, 0xfa, 0x4b, 0xca, 0x79, 0x65, 0x65, 0x6a, 0xf5, 0x0d, 0x6d, 0x90, 0xb3, 0xb5, 0x0d, 0x41, 0x9c, 0x17, 0xa3, 0xcf, 0x3d, 0xea, 0x35, 0x4f, 0xf5, 0x7b, 0xcd, 0xf1, 0x10, 0xef, 0xe3, 0x48, 0x2a, 0xba, 0x0b, 0xd3, 0xb6, 0xd3, 0xa1, 0x6d, 0x6a, 0x51, 0x83, 0x39, 0xde, 0x52, 0x5d, 0x68, 0x39, 0x9f, 0xd6, 0xc2, 0xa3, 0xa0, 0x1d, 0x5e, 0xd6, 0x6e, 0xa6, 0xe8, 0xf4, 0xf9, 0x7e, 0xaf, 0x39, 0x9d, 0x86, 0xe0, 0x8c, 0x1c, 0xb4, 0x0a, 0x60, 0x38, 0x36, 0xf3, 0x1c, 0xcb, 0xa2, 0xde, 0xd2, 0xc8, 0x79, 0x65, 0x65, 0x52, 0x47, 0xd2, 0x0a, 0x58, 0x8f, 0x31, 0x38, 0x45, 0xa5, 0x7e, 0x5e, 0x87, 0x29, 0x9d, 0xf8, 0xa6, 0x11, 0x5a, 0x89, 0x7e, 0x06, 0x40, 0x18, 0xf3, 0xcc, 0x9d, 0x80, 0x89, 0xf3, 0xd7, 0x57, 0xa6, 0x56, 0xbf, 0x35, 0xf8, 0xfc, 0x29, 0x76, 0x6d, 0x2d, 0xe6, 0xdd, 0xb4, 0x99, 0x77, 0xa4, 0xbf, 0x14, 0xa9, 0x4f, 0x10, 0x3f, 0xff, 0x57, 0x73, 0xe6, 0x76, 0x40, 0x2c, 0x73, 0xd7, 0xa4, 0x9d, 0x9b, 0xa4, 0x4b, 0x71, 0x4a, 0x23, 0x3a, 0x84, 0x09, 0x83, 0xb8, 0xc4, 0x30, 0xd9, 0xd1, 0x52, 0x4d, 0x68, 0x7f, 0xab, 0xba, 0xf6, 0x75, 0xc9, 0x19, 0xea, 0xbe, 0x20, 0x75, 0x4f, 0x44, 0xe0, 0xe3, 0x9a, 0x63, 0x5d, 0xcb, 0x16, 0xcc, 0xe5, 0x6c, 0x47, 0xf3, 0x50, 0x3f, 0xa0, 0x47, 0x22, 0x07, 0x26, 0x31, 0xff, 0x13, 0xad, 0xc3, 0xe8, 0x21, 0xb1, 0x02, 0xba, 0x54, 0x13, 0x11, 0x7b, 0xb5, 0x52, 0x5e, 0x44, 0x52, 0x71, 0xc8, 0xfb, 0x76, 0xed, 0x8a, 0xb2, 0x7c, 0x00, 0x33, 0x19, 0x5b, 0x0b, 0x74, 0x6d, 0x64, 0x75, 0x69, 0x29, 0x5d, 0x71, 0x8a, 0x6b, 0xee, 0xc1, 0x5e, 0x56, 0xf9, 0xed, 0x80, 0xd8, 0xcc, 0x64, 0x47, 0x29, 0x65, 0xea, 0x55, 0x58, 0x58, 0xdf, 0xbc, 0x1e, 0x5a, 0x93, 0xce, 0x15, 0x7a, 0xdf, 0xf5, 0xa8, 0xef, 0x9b, 0x8e, 0x1d, 0xea, 0x4d, 0x72, 0x65, 0x33, 0xc6, 0xe0, 0x14, 0x95, 0x7a, 0x08, 0x63, 0x32, 0x4b, 0xce, 0xc3, 0x88, 0x4d, 0xba, 0x54, 0xf2, 0x4d, 0x4b, 0xbe, 0x11, 0xe1, 0x53, 0x81, 0x41, 0xd7, 0x60, 0x74, 0x87, 0x47, 0x46, 0x9a, 0x7f, 0xb1, 0x72, 0x10, 0xf5, 0xc9, 0x7e, 0xaf, 0x39, 0x2a, 0x00, 0x38, 0x14, 0xa1, 0x3e, 0xac, 0xc1, 0xb9, 0x7c, 0x91, 0xad, 0x3b, 0xf6, 0xae, 0xb9, 0x17, 0x78, 0xe2, 0x03, 0x7d, 0x17, 0xc6, 0x42, 0x91, 0xd2, 0xa2, 0x15, 0x69, 0xd1, 0x58, 0x5b, 0x40, 0x9f, 0xf6, 0x9a, 0x67, 0xf3, 0xac, 0x21, 0x06, 0x4b, 0x3e, 0xb4, 0x02, 0x13, 0x1e, 0xfd, 0x38, 0xa0, 0x3e, 0xf3, 0x45, 0xde, 0x4d, 0xea, 0xd3, 0x3c, 0x75, 0xb0, 0x84, 0xe1, 0x18, 0x8b, 0x1e, 0x28, 0xb0, 0x18, 0x56, 0x72, 0xc6, 0x06, 0x59, 0xc5, 0x97, 0xab, 0xe4, 0x44, 0x86, 0x51, 0xff, 0xaa, 0x34, 0x76, 0xb1, 0x00, 0x89, 0x8b, 0x54, 0xa9, 0xff, 0x51, 0xe0, 0x6c, 0x71, 0xd7, 0x41, 0xbb, 0x30, 0xee, 0x89, 0xbf, 0xa2, 0xe2, 0x7d, 0xa7, 0x8a, 0x41, 0xf2, 0x98, 0xe5, 0x3d, 0x2c, 0xfc, 0xf6, 0x71, 0x24, 0x1c, 0x19, 0x30, 0x66, 0x08, 0x9b, 0x64, 0x95, 0xbe, 0x33, 0x5c, 0x8f, 0xcc, 0x7a, 0x60, 0x36, 0x0a, 0x57, 0x08, 0xc6, 0x52, 0xb4, 0xfa, 0x5b, 0x05, 0xe6, 0x72, 0x55, 0x84, 0x1a, 0x50, 0x37, 0x6d, 0x26, 0xd2, 0xaa, 0x1e, 0xc6, 0x68, 0xcb, 0x66, 0x77, 0x79, 0xb2, 0x63, 0x8e, 0x40, 0x17, 0x60, 0x64, 0xc7, 0x71, 0x2c, 0x11, 0x8e, 0x09, 0x7d, 0xa6, 0xdf, 0x6b, 0x4e, 0xea, 0x8e, 0x63, 0x85, 0x14, 0x02, 0x85, 0xbe, 0x01, 0x63, 0x3e, 0xf3, 0x4c, 0x7b, 0x4f, 0xf6, 0xc8, 0xb9, 0x7e, 0xaf, 0x39, 0xd5, 0x16, 0x90, 0x90, 0x4c, 0xa2, 0xd1, 0xcb, 0x30, 0x7e, 0x48, 0x3d, 0x51, 0x21, 0xa3, 0x82, 0x52, 0x74, 0xe0, 0xbb, 0x21, 0x28, 0x24, 0x8d, 0x08, 0xd4, 0xdf, 0xd7, 0x60, 0x4a, 0x06, 0xd0, 0x22, 0x66, 0x17, 0xdd, 0x4b, 0x25, 0x54, 0x18, 0x89, 0x57, 0x86, 0x88, 0x84, 0x3e, 0x1f, 0x35, 0xaf, 0x82, 0x0c, 0xa4, 0x30, 0x65, 0x38, 0xb6, 0xcf, 0x3c, 0x62, 0xda, 0x32, 0x5d, 0xb3, 0x0d, 0x62, 0x50, 0xe2, 0x49, 0x36, 0x7d, 0x51, 0x2a, 0x98, 0x4a, 0x60, 0x3e, 0x4e, 0xcb, 0x45, 0x1f, 0xc6, 0x21, 0xae, 0x0b, 0x0d, 0x6f, 0x56, 0xd2, 0xc0, 0x0f, 0x5f, 0x2d, 0xba, 0x7f, 0x53, 0x60, 0xa9, 0x8c, 0x29, 0x53, 0x8f, 0xca, 0x73, 0xd5, 0x63, 0xed, 0xe4, 0xea, 0xf1, 0xcf, 0x4a, 0x2a, 0xf6, 0xbe, 0x8f, 0x3e, 0x82, 0x09, 0xbe, 0xda, 0x74, 0x08, 0x23, 0x72, 0x85, 0x78, 0x6d, 0x50, 0xfb, 0xf6, 0x35, 0x4e, 0xcd, 0xc7, 0xfd, 0xad, 0x9d, 0x1f, 0x53, 0x83, 0xdd, 0xa0, 0x8c, 0x24, 0xcd, 0x38, 0x81, 0xe1, 0x58, 0x2a, 0xba, 0x05, 0x23, 0xbe, 0x4b, 0x8d, 0x61, 0x06, 0x91, 0x30, 0xad, 0xed, 0x52, 0x23, 0xe9, 0xd7, 0xfc, 0x0b, 0x0b, 0x41, 0xea, 0xaf, 0xd2, 0xc1, 0xf0, 0xfd, 0x6c, 0x30, 0xca, 0x5c, 0xac, 0x9c, 0x9c, 0x8b, 0x3f, 0x8d, 0x5b, 0x81, 0xb0, 0xef, 0xba, 0xe9, 0x33, 0xf4, 0xc1, 0x31, 0x37, 0x6b, 0xd5, 0xdc, 0xcc, 0xb9, 0x85, 0x93, 0xe3, 0x2a, 0x8b, 0x20, 0x29, 0x17, 0xdf, 0x84, 0x51, 0x93, 0xd1, 0x6e, 0x54, 0x5f, 0x17, 0x2b, 0xfb, 0x58, 0x9f, 0x91, 0x52, 0x47, 0xb7, 0x38, 0x3f, 0x0e, 0xc5, 0xa8, 0xbf, 0xab, 0x65, 0x4e, 0xc0, 0x7d, 0x8f, 0x7e, 0x08, 0x93, 0xbe, 0x9c, 0xc8, 0x51, 0x97, 0xb8, 0x54, 0x45, 0x4f, 0xbc, 0x12, 0x2e, 0x48, 0x55, 0x93, 0x11, 0xc4, 0xc7, 0x89, 0xc4, 0x54, 0x05, 0xd7, 0x86, 0xaa, 0xe0, 0x5c, 0xfc, 0xcb, 0x2a, 0x18, 0xdd, 0x83, 0x19, 0x3f, 0x30, 0x19, 0xd9, 0xb1, 0x28, 0x5f, 0x4b, 0xfd, 0xca, 0x9b, 0xec, 0x42, 0xbf, 0xd7, 0x9c, 0x69, 0xa7, 0x59, 0x71, 0x56, 0x92, 0xea, 0x41, 0x51, 0x6e, 0xa0, 0x1f, 0xc0, 0x98, 0xe3, 0x92, 0x8f, 0x03, 0x2a, 0x03, 0xfe, 0x8c, 0xe5, 0xf0, 0x96, 0xa0, 0x2d, 0xca, 0x40, 0xe0, 0xc7, 0x09, 0xd1, 0x58, 0x8a, 0x54, 0x1f, 0x2a, 0x30, 0x9f, 0xef, 0x93, 0x43, 0x34, 0xa2, 0x6d, 0x98, 0xed, 0x12, 0x66, 0xec, 0xc7, 0xb3, 0x4a, 0x54, 0xe7, 0xa4, 0xbe, 0xd2, 0xef, 0x35, 0x67, 0x6f, 0x64, 0x30, 0x4f, 0x7b, 0x4d, 0xf4, 0x6e, 0x60, 0x59, 0x47, 0xd9, 0x75, 0x34, 0xc7, 0xaf, 0xfe, 0xa2, 0x0e, 0x33, 0x99, 0xb1, 0x50, 0x61, 0xf1, 0x5a, 0x83, 0xb9, 0x4e, 0x12, 0x47, 0x8e, 0x90, 0x66, 0x7c, 0x45, 0x12, 0xa7, 0x93, 0x50, 0xf0, 0xe5, 0xe9, 0xb3, 0x59, 0x59, 0xff, 0xc2, 0xb3, 0xf2, 0x2e, 0xcc, 0x92, 0x78, 0x11, 0xb8, 0xe1, 0x74, 0xa8, 0x1c, 0xc3, 0x9a, 0xe4, 0x9a, 0x5d, 0xcb, 0x60, 0x9f, 0xf6, 0x9a, 0xa7, 0xf3, 0xeb, 0x03, 0x87, 0xe3, 0x9c, 0x14, 0xf4, 0x12, 0x8c, 0x1a, 0x4e, 0x60, 0x33, 0x31, 0xab, 0xeb, 0x49, 0x15, 0xae, 0x73, 0x20, 0x0e, 0x71, 0xe8, 0x9b, 0x30, 0x45, 0x3a, 0x5d, 0xd3, 0x5e, 0x33, 0x0c, 0xea, 0xfb, 0x4b, 0x63, 0x62, 0x4b, 0x88, 0x67, 0xe1, 0x5a, 0x82, 0xc2, 0x69, 0x3a, 0xf5, 0x4f, 0x4a, 0xb4, 0x82, 0x96, 0xac, 0x4a, 0xe8, 0x22, 0x5f, 0xbc, 0x04, 0x4a, 0x06, 0x27, 0xb5, 0x3b, 0x09, 0x30, 0x8e, 0xf0, 0xe8, 0xeb, 0x30, 0xd6, 0xf1, 0xcc, 0x43, 0xea, 0xc9, 0xc8, 0xc4, 0xe5, 0xb5, 0x21, 0xa0, 0x58, 0x62, 0x79, 0xb0, 0xdd, 0x68, 0x95, 0x49, 0x05, 0x7b, 0xdb, 0x71, 0x2c, 0x2c, 0x30, 0x42, 0x92, 0xb0, 0x4a, 0xba, 0x30, 0x91, 0x14, 0xda, 0x2a, 0xb1, 0xea, 0x07, 0x30, 0x9b, 0xdb, 0xff, 0xaf, 0x41, 0xdd, 0xa0, 0x96, 0xac, 0xa2, 0xd6, 0xe0, 0xe8, 0x1e, 0xbb, 0x3d, 0xe8, 0xe3, 0xfd, 0x5e, 0xb3, 0xbe, 0xbe, 0x79, 0x1d, 0x73, 0x21, 0xea, 0x6f, 0x14, 0x78, 0xa1, 0xb4, 0xd2, 0x52, 0xa7, 0x55, 0x06, 0x9e, 0x96, 0x00, 0xb8, 0xc4, 0x23, 0x5d, 0xca, 0xa8, 0xe7, 0x17, 0x0c, 0xb6, 0x6c, 0x3f, 0x97, 0x17, 0x7b, 0x0d, 0x93, 0x9f, 0x6c, 0xde, 0x67, 0xd4, 0xe6, 0x3b, 0x58, 0x32, 0x33, 0xb7, 0x63, 0x41, 0x38, 0x25, 0x54, 0xfd, 0x63, 0x0d, 0x4e, 0x6f, 0x3b, 0x9d, 0xb6, 0xb1, 0x4f, 0x3b, 0x81, 0x65, 0xda, 0x7b, 0xfc, 0x52, 0x4c, 0xef, 0xb3, 0x13, 0x18, 0xd8, 0xef, 0x67, 0x06, 0xf6, 0x33, 0x1a, 0x71, 0x91, 0x8d, 0x65, 0x93, 0x1b, 0x7d, 0xc4, 0xb7, 0x59, 0xc2, 0x82, 0xa8, 0xfb, 0x5e, 0x79, 0x0e, 0xd9, 0x82, 0x3f, 0x89, 0x4c, 0xf8, 0x8d, 0xa5, 0x5c, 0xf5, 0xef, 0x0a, 0x2c, 0x15, 0xb1, 0x9d, 0xc0, 0x10, 0xfe, 0x5e, 0x76, 0x08, 0xaf, 0x0e, 0x7f, 0xb6, 0x92, 0x69, 0xfc, 0x49, 0xc9, 0x99, 0xc4, 0x58, 0xbe, 0x02, 0xd3, 0x61, 0xbb, 0xa2, 0x1d, 0x3e, 0x8d, 0x64, 0xe2, 0x9e, 0x96, 0x82, 0xa6, 0xdb, 0x29, 0x1c, 0xce, 0x50, 0xa2, 0xb7, 0x61, 0xd6, 0x75, 0x18, 0xb5, 0x99, 0x49, 0xac, 0x70, 0x24, 0x86, 0x97, 0x49, 0xc4, 0xfb, 0xda, 0x76, 0x06, 0x83, 0x73, 0x94, 0xea, 0x2f, 0x15, 0x58, 0x2e, 0x8f, 0x0e, 0xfa, 0x29, 0xcc, 0x46, 0x27, 0x16, 0xfb, 0x72, 0xc5, 0x0b, 0x1e, 0x4e, 0xf3, 0x24, 0xb2, 0x65, 0xc8, 0xcf, 0x46, 0x3d, 0x37, 0x43, 0xe6, 0xe3, 0x9c, 0x2a, 0xf5, 0xd7, 0x35, 0x98, 0xc9, 0x90, 0x9c, 0x40, 0xc9, 0xdc, 0xce, 0x94, 0x4c, 0x6b, 0x98, 0x63, 0x96, 0xd5, 0xca, 0xbd, 0x5c, 0xad, 0x5c, 0x1e, 0x46, 0xe8, 0xe0, 0x22, 0xe9, 0x2b, 0xd0, 0xc8, 0xd0, 0xf3, 0x1d, 0x22, 0xe8, 0x52, 0x0f, 0xd3, 0x5d, 0xea, 0x51, 0xdb, 0xa0, 0xe8, 0x12, 0x4c, 0x10, 0xd7, 0xbc, 0xea, 0x39, 0x81, 0x2b, 0x53, 0x2a, 0x4e, 0xfd, 0xb5, 0xed, 0x2d, 0x01, 0xc7, 0x31, 0x05, 0xa7, 0x8e, 0x2c, 0x92, 0x13, 0x20, 0x75, 0x27, 0x0c, 0xe1, 0x38, 0xa6, 0x88, 0x17, 0x83, 0x91, 0xd2, 0xc5, 0x40, 0x87, 0x7a, 0x60, 0x76, 0xe4, 0x45, 0xf6, 0x35, 0x49, 0x50, 0xbf, 0xb3, 0xb5, 0xf1, 0xb4, 0xd7, 0xbc, 0x50, 0xf6, 0x7e, 0xca, 0x8e, 0x5c, 0xea, 0x6b, 0x77, 0xb6, 0x36, 0x30, 0x67, 0x56, 0xff, 0xa2, 0xc0, 0x42, 0xe6, 0x90, 0x27, 0xd0, 0x02, 0xb6, 0xb3, 0x2d, 0xe0, 0x95, 0x21, 0x42, 0x56, 0x52, 0xfb, 0x0f, 0x14, 0x38, 0x37, 0xb0, 0x2c, 0x2a, 0xac, 0x59, 0xdf, 0x81, 0xb9, 0xc0, 0xce, 0x2e, 0xbf, 0x61, 0xa5, 0x2f, 0xf2, 0x15, 0xeb, 0x4e, 0x16, 0x85, 0xf3, 0xb4, 0xfc, 0xba, 0xb5, 0x70, 0x2c, 0x65, 0xd1, 0x7b, 0xf9, 0x97, 0xe7, 0x8b, 0x95, 0xaf, 0xdc, 0x03, 0x9e, 0x9b, 0xb3, 0xcf, 0xc2, 0xb5, 0x4a, 0xcf, 0xc2, 0x9f, 0xd6, 0x60, 0xb1, 0x20, 0xfb, 0xd1, 0x87, 0x00, 0xc9, 0xd6, 0x55, 0x10, 0xec, 0x02, 0x23, 0x8f, 0x3d, 0x2a, 0xcd, 0x8a, 0xf7, 0xe0, 0x04, 0x9a, 0x92, 0x88, 0x7c, 0x98, 0xf2, 0xa8, 0x4f, 0xbd, 0x43, 0xda, 0x79, 0xd7, 0xf1, 0x64, 0xc8, 0xbf, 0x3d, 0x44, 0xc8, 0x8f, 0x55, 0x5d, 0xb2, 0xdc, 0xe1, 0x44, 0x30, 0x4e, 0x6b, 0x41, 0x6d, 0x38, 0xd3, 0xa1, 0x24, 0x65, 0xa6, 0x58, 0xd3, 0x68, 0x47, 0xbe, 0x21, 0x9d, 0x93, 0x02, 0xce, 0x6c, 0x14, 0x11, 0xe1, 0x62, 0x5e, 0xf5, 0x9f, 0x0a, 0x9c, 0xc9, 0x58, 0xf6, 0x1e, 0xed, 0xba, 0x16, 0x61, 0xf4, 0x04, 0x3a, 0xe7, 0xbd, 0x4c, 0xe7, 0x7c, 0x6b, 0x08, 0xf7, 0x45, 0x46, 0x96, 0xbe, 0x13, 0xfc, 0x43, 0x81, 0x17, 0x0a, 0x39, 0x4e, 0xa0, 0x13, 0xbc, 0x9f, 0xed, 0x04, 0xaf, 0x3f, 0xc7, 0xb9, 0x4a, 0x3a, 0xc2, 0xe3, 0xb2, 0x53, 0xb5, 0xc3, 0x0d, 0xeb, 0xff, 0x6f, 0xd4, 0xa9, 0x7f, 0x50, 0x60, 0x3a, 0xa2, 0xe4, 0x37, 0x86, 0x0a, 0x3d, 0x6d, 0x15, 0x40, 0xfe, 0x40, 0x16, 0xbd, 0x9f, 0xd5, 0x13, 0xbb, 0xaf, 0xc6, 0x18, 0x9c, 0xa2, 0x42, 0xd7, 0x00, 0x45, 0x16, 0xb6, 0x2d, 0xb1, 0xfb, 0xf3, 0x1b, 0x58, 0x5d, 0xf0, 0x2e, 0x4b, 0x5e, 0x84, 0x8f, 0x51, 0xe0, 0x02, 0x2e, 0xf5, 0xaf, 0x4a, 0xb2, 0x64, 0x08, 0xf0, 0x97, 0xd5, 0xf3, 0xc2, 0xb8, 0x52, 0xcf, 0xa7, 0x87, 0xa4, 0xa0, 0xfc, 0xd2, 0x0e, 0x49, 0x61, 0x5d, 0x49, 0x49, 0x3c, 0xac, 0xe7, 0x4e, 0x21, 0x4a, 0xa1, 0xea, 0x65, 0xee, 0xba, 0xbc, 0xba, 0x86, 0x6e, 0x7d, 0xb9, 0x9a, 0x39, 0x3c, 0x4d, 0x0b, 0xaf, 0xb9, 0x97, 0x60, 0xc2, 0x76, 0x3a, 0x54, 0x3c, 0x66, 0xe4, 0x56, 0xa1, 0x9b, 0x12, 0x8e, 0x63, 0x8a, 0x63, 0x3f, 0xaf, 0x8e, 0x7c, 0x41, 0x3f, 0xaf, 0xf2, 0xf5, 0xcd, 0x92, 0x5b, 0xfd, 0xa8, 0x98, 0x0c, 0xc9, 0xfa, 0x26, 0xe1, 0x38, 0xa6, 0x40, 0xb7, 0x92, 0x59, 0x3e, 0x26, 0x62, 0xf2, 0xb5, 0x2a, 0xb3, 0xbc, 0x7c, 0x8c, 0xeb, 0xfa, 0xa3, 0x27, 0x8d, 0x53, 0x8f, 0x9f, 0x34, 0x4e, 0x7d, 0xf6, 0xa4, 0x71, 0xea, 0x41, 0xbf, 0xa1, 0x3c, 0xea, 0x37, 0x94, 0xc7, 0xfd, 0x86, 0xf2, 0x59, 0xbf, 0xa1, 0x7c, 0xde, 0x6f, 0x28, 0x9f, 0xfc, 0xbb, 0x71, 0xea, 0xfb, 0x2f, 0x0e, 0xfa, 0x2f, 0x82, 0xff, 0x05, 0x00, 0x00, 0xff, 0xff, 0x23, 0x3d, 0xa6, 0x20, 0x64, 0x20, 0x00, 0x00, </ins> } func (m *AllocationResult) Marshal() (dAtA []byte, err error) {

Listing s on v1.31.0 returns an object with a field instead of the standard field for (list) objects: Listing s on v1.30.3 behaves correctly: This is also propagated to the Swagger spec (which is how I noticed it, since I maintain the Rust bindings for the client API). v1.31.0: v1.30.3: As explained above. v1.31.0 with feature gate enabled and runtime config . With kind, this is: the API server request as in the above curl command in any way you like as long as it lets you see the raw JSON response. No response <details</details<detailskind 0.23.0 </details<details</details<details</details<details</details<details</details
/sig api-machinery
It seems there may be a discrepancy in the API response format between v1.30.3 and v1.31.0. The object is expected to follow standard Kubernetes API conventions, which typically use the field for list objects.
This deviation might cause compatibility issues for API clients that rely on this structure.
If this deviation from the expected API structure wasn't intentional, it should be aligned with the standard conventions.
The presence of instead of could disrupt client integrations, as you've encountered with the Swagger spec and Rust bindings.
cc
/triage accepted
/assign
this must be fixed to json:"metadata" protobuf:"bytes,1,opt,name=metadata"`` the serialization will not work with standard clients
/priority critical-urgent
Shoot, that's clearly a mistake.


3 July 2020 [x] Two weeks soak end date: 20 July 2020 According to this APIsnoop query, there are still an endpoint which is untested. Get the server version (hits api path /version) Check that the major version is current Discover useragents: List endpoints hit by the test: Display endpoint coverage change: If a test with these calls gets merged, test coverage will go up by 1 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen was Merge but the merge was reversed by Issue and Test grid restarted.
Reopened this issue. <detailsIn response to : /reopen was Merge but the merge was reversed by Issue and Test grid restarted. </details
Automatic retest and Merge again


Question: offset, numCandidates := pl.getOffsetAndNumCandidates(int32(len(nodeInfos))) candidates, _ := dryRunPreemption(context.Background(), fwk, state, tt.pod, nodeInfos, nil, offset, numCandidates) s := SelectCandidate(candidates) <ins> if s == nil || len(s.Name()) == 0 { return } </ins> found := false for _, nodeName := range tt.expected { if nodeName == s.Name() {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --Kubernetes 1.19 (confirmed with 1.19.7 and 1.19.10) kube-scheduler crashes and restarts with the following errors in . panic: runtime error: index out of range [0] with length 0 The problem code is line 389. When is nil or is empty, the error happens. If we skip when it's nil or empty, the error is gone. panic: runtime error: invalid memory address or nil poster dereference The problem is does not exist and returns nil sometimes. Simply skipping it won't solve the problem. The scheduler will reach either line 456 or 341. The scheduler works without failures. When there are preemptions in a cluster. Kubernetes version (use ): v1.19.10 or v1.19.7 Cloud provider or hardware configuration: bare metal OS (e.g: ): centos 7 Kernel (e.g. ): Linux 5.4.77-7.el7pie SMP Sat Nov 21 01:16:27 UTC 2020 x8664 x8664 x8664 GNU/Linux Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scheduling
/cc
/assign
Are you running the vanilla scheduler? or a customized one that leverages the utilities in
It's a custom scheduler with some internal out-of-tree plugins (just like ), but it doesn't have any custom postFilter plugins. Which utility functions are you referring to? Thanks.
are those custom plugins filter plugins? do they maintain state?


Question: offset, numCandidates := pl.getOffsetAndNumCandidates(int32(len(nodeInfos))) candidates, _ := dryRunPreemption(context.Background(), fwk, state, tt.pod, nodeInfos, nil, offset, numCandidates) s := SelectCandidate(candidates) <ins> if s == nil || len(s.Name()) == 0 { return } </ins> found := false for _, nodeName := range tt.expected { if nodeName == s.Name() {

If yes, then one hypothesis is that those custom filters filter the node in the filter phase, but not when executed in the preemption phase, and so this will result in a candidate node with no victims?
Yes, it includes filter plugins and uses cycle state.
you need to make sure that those filter plugins play nicely when executed again in the preemption phase in the same cycle: i.e., produce the same result.
Thanks What do you mean by performing filter in the preemption phase? would you mind elaborating it a little?
What additional logic is needed to handle the preemption case?
in the preemption phase we run the filters again to check if removing lower priority pods make the node schedulable: we later add them one by one to reduce the set of victim pods to the absolute minimum: My hypothesis is that somehow your filter returns success when adding all the pods back, and so you end up with zero victims and a candidate node. This shouldn't happen because if the pod fits the node without removing any victims, then we shouldn't be running preemption in the first place. So the theory is that the custom filter is returning false when run in the filter phase, but returns true when executed in the preemption phase perhaps because it makes some assumptions about cyclestate that makes it behave this way (e.g. that a the filter will be executed once per node in a scheduling cycle).
Disabled the filter plugin, but still see the same issues.
are you able to change the scheduler code and run the test again? I can send a patch tomorrow to add some debugging messages to help us root cause the issue.
yes, thanks a lot! really appreciate it!
You are right. We still use a deprecated Predicate extender, which causes the problem. After disabling it, it's working fine. It used to work with 1.18 though. We are retiring it. Thank you so much!!!
If the test cases can help debug scheduler extender Webhook (Predicate) too, I'd like to try it. Thanks again!
How can we improve the preemption logic to prevent bugs and issues in custom Filters or Predicates crashing the scheduler?
for starter, we should check the list len, the scheduler should not panic in all cases.


Question: offset, numCandidates := pl.getOffsetAndNumCandidates(int32(len(nodeInfos))) candidates, _ := dryRunPreemption(context.Background(), fwk, state, tt.pod, nodeInfos, nil, offset, numCandidates) s := SelectCandidate(candidates) <ins> if s == nil || len(s.Name()) == 0 { return } </ins> found := false for _, nodeName := range tt.expected { if nodeName == s.Name() {

Other than that, I think we need to clearly document what the preemption logic does (that it executes the prefilter extensions points and the filter plugins multiple times) and the expectations from filter plugins (that they may get executed more than once in the same cycle)
Is it a predicate extender or preemption extender? If it's a predicate extender, I don't quite think we found the root cause - b/c predicate extenders are not invoked during preemption (). It makes more sense if it's a preemption extender - if the preemption extender mutates the to make it invalid. If it's the case, could you verify it by adding debug info before and after L157 to check the validness of ?
That's always true :)
filed a PR to mitigate it.
It's a predicate extender. I'll debug it more.
Added some debug info. after and . Here's the result. returned 4 candidates, but one of candidates () has zero pod. The candidate happened to be the best candidate and hence caused the "" error. It should have nothing to do with since our extender doesn't support Preemption. Disabling it may change pod placements/candidates selection and bypassed the error. It's like a coincidence. To remove the panic, a simple fix is to check if is empty. If it is, do not add the node to candidates. Alternatively, a change to may work too. Here is the log with debug info.
This looks more promising. Back to digging into the root cause, it seems returns an empty but with . So that implies firstly we have a non-nil , and then in , the is separated into two list: violatingVictims and nonViolatingVictims. Finally the two victims list are tried one by one to run . It looks like returns {fits=true, nil error} for every victim, which is quite abnormal. I think it's problematic that every pod is retrievable and in which we can debug into.
Yes, should not return fit=true with empty pods in the first place, but it's hard to locate the problematic code in this function.
The following code in looks questionable to me. If is empty, should the function return nil, nil? Also, if is true for all extenders, the loop won't do anything and candidates will remain the same.


Question: offset, numCandidates := pl.getOffsetAndNumCandidates(int32(len(nodeInfos))) candidates, _ := dryRunPreemption(context.Background(), fwk, state, tt.pod, nodeInfos, nil, offset, numCandidates) s := SelectCandidate(candidates) <ins> if s == nil || len(s.Name()) == 0 { return } </ins> found := false for _, nodeName := range tt.expected { if nodeName == s.Name() {

Why do we bother creating a new candidates later? Can it just return candidates?
It can prevent from returning candidates with empty pods, but what about ? It's possible to add nodes without victim pods to candidates if returned by contain such nodes. How about adding the following check to CallExtender?
Updated the PR based on the finding and discussions.
Yes, we can use a flag to mark if all extenders don't support preemption. If yes, simply return the candidates.
Technically this can exist due to faulty extender implementation. If we really want to guard it, instead of logging the error and continue, I'm more inclined to return the error immediately as this is a fatal error - the victimsMap cannot be used either for latter extender or further preemptor nominating.
Is my understanding correct? In , if victimsMap is empty, should it return instead of ?
The only case victimsMap can be empty is empty candidates, and so return either doesn't quite matter, right?
You are right.
As long as one extender returns an invalid victimsMap (with empty pods), s returns an error immediately? What about ? Should it return an error immediately if it returnswith empty pods?
I think so. Because the result it returned is faulty, and we don't want to continue based on it The same. We may don't want to continue the scheduling cycle based on faulty result.


Question: } By("Reading value under foo from member with index 2") <del> if v := pet.read(2, "foo"); v != "bar" { framework.Failf("Read unexpected value %v, expected bar under key foo", v) </del> <ins> if err := pollReadWithTimeout(pet, 2, "foo", "bar"); err != nil { framework.Failf("%v", err) </ins> } }) })

https://k8s- Failed: [] PetSet [Slow] [Feature:PetSet] [] Deploy clustered applications [Slow] [Feature:PetSet] should creating a working redis cluster [Feature:PetSet] {Kubernetes e2e suite}
The test failed because it read nothing back from the key for the "member with index 2". Looking at , I notice that it's writing to the "member with index 0" and expecting to read it back from the "member with index 2". I'm curious, is each index supposed to represent a separate redis instance? What is the consistency model here?
/cc
https://k8s- Failed: [] PetSet [Slow] [Feature:PetSet] [] Deploy clustered applications [Slow] [Feature:PetSet] should creating a working redis cluster [Feature:PetSet] {Kubernetes e2e suite}
https://k8s- Failed: [] PetSet [Slow] [Feature:PetSet] [] Deploy clustered applications [Slow] [Feature:PetSet] should creating a working redis cluster [Feature:PetSet] {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
It's master slave redis, you should be able to create the same petset with this: , fyi
https://k8s- Failed: [] PetSet [Slow] [Feature:PetSet] [] Deploy clustered applications [Slow] [Feature:PetSet] should creating a working redis cluster [Feature:PetSet] {Kubernetes e2e suite}
https://k8s- Failed: [] PetSet [Slow] [Feature:PetSet] [] Deploy clustered applications [Slow] [Feature:PetSet] should creating a working redis cluster [Feature:PetSet] {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] PetSet [Slow] [Feature:PetSet] [] Deploy clustered applications [Slow] [Feature:PetSet] should creating a working redis cluster [Feature:PetSet] {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.


Question: } By("Reading value under foo from member with index 2") <del> if v := pet.read(2, "foo"); v != "bar" { framework.Failf("Read unexpected value %v, expected bar under key foo", v) </del> <ins> if err := pollReadWithTimeout(pet, 2, "foo", "bar"); err != nil { framework.Failf("%v", err) </ins> } }) })

https://k8s- Failed: [] PetSet [Slow] [Feature:PetSet] [] Deploy clustered applications [Slow] [Feature:PetSet] should creating a working redis cluster [Feature:PetSet] {Kubernetes e2e suite}
https://k8s- Failed: [] PetSet [Slow] [Feature:PetSet] [] Deploy clustered applications [Slow] [Feature:PetSet] should creating a working redis cluster [Feature:PetSet] {Kubernetes e2e suite}
https://k8s- Failed: [] PetSet [Slow] [Feature:PetSet] [] Deploy clustered applications [Slow] [Feature:PetSet] should creating a working redis cluster [Feature:PetSet] {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.


Question: switch groupVersion { case extensionsv1beta1.SchemeGroupVersion: // no-op for compatibility <del> case appsv1beta2.SchemeGroupVersion: </del> <ins> case appsv1beta2.SchemeGroupVersion, appsv1.SchemeGroupVersion: </ins> // disallow mutation of selector allErrs = append(allErrs, apivalidation.ValidateImmutableField(newDaemonSet.Spec.Selector, oldDaemonSet.Spec.Selector, field.NewPath("spec").Child("selector"))...) default:

/priority failing-test /priority critical-urgent /sig apps This test case has been failing for a while and affects a number of jobs: This is affecting multiple jobs on the testgrid dashboard, and prevents us from cutting 1.9.0-alpha.2 (kubernetes/sig-release). Is there work ongoing to bring this job back to green?
All test failures look similar: . ~Is it due to network issue instead of DaemonSet?~
This seems related to the new DaemonSet. The connection refused error is happening because kube-apiserver is crashing:
needs an update
Ah, well... I guess it's halfway working as intended. We put that there to make the tests fail if we forgot to update it when adding a new version. We had hoped it would fail during presubmit tests though. :(
Sending a fix now
I wonder why the test is not failing on submit queue tests. Do i need to fix deployment and replicaset?
The rest of the group is not merged, and the v1 PR takes the selector mutability into account.
Since this bug crashes apiserver I'd expect it to affect many other e2e tests.
Noted, we need to fix deployment and replicaset when the v1 PR is merged.
Not quite sure how this test isn't failing before but is now consistently failing. I'd expect the commit that triggers this to block the submit queue. The e2e test is still accessing DaemonSet through extensions/v1beta1 endpoints. From the above apiserver log it's hard to tell what updates DaemonSet via apps/v1 before the code panics. I tried to reproduce it locally, and found that garbage collector PUT apps/v1 DaemonSet:
When apps/v1 gets introduced, it may be ordered before apps/beta1 alphabetically. IIRC, there is a code change on the apps/v1 PR that changes the order of API versions such that apps/v1 is ranked before apps/v1beta2 and apps/v1beta2 before apps/v1beta1. Perhaps this problem can be fixed when the PR merges.
Since crashes apiserver and makes debugging harder, which defeats the purpose of using it, we should return error on unsupported versions instead.
The first run with this change is still in progress, but this test already passed: https://k8s- Thanks for the quick fix


Question: nodePortAddresses []string, kernelHandler KernelHandler, ) (*Proxier, error) { <del> // Set the route_localnet sysctl we need for if err := utilproxy.EnsureSysctl(sysctl, sysctlRouteLocalnet, 1); err != nil { return nil, err } </del> // Proxy needs br_netfilter and bridge-nf-call-iptables=1 when containers // are connected to a Linux bridge (but not SDN bridges).  Until most // plugins handle this, log when config is missing

CVSS Rating: In typical clusters: medium (5.4) In clusters where API server insecure port has not been disabled: high (8.8) A security issue was discovered in which allows adjacent hosts to reach TCP and UDP services bound to running on the node or in the node's network namespace. For example, if a cluster administrator runs a TCP service on a node that listens on , because of this bug, that service would be potentially reachable by other hosts on the same LAN as the node, or by containers running on the same node as the service. If the example service on port required no additional authentication (because it assumed that only other localhost processes could reach it), then it could be vulnerable to attacks that make use of this bug. The Kubernetes API Server's default insecure port setting causes the API server to listen on where it will accept requests without authentication. Many Kubernetes installers explicitly disable the API Server's insecure port, but in clusters where it is not disabled, an attacker with access to another system on the same LAN or with control of a container running on the master may be able to reach the API server and execute arbitrary API requests on the cluster. This port is deprecated, and will be removed in Kubernetes v1.20. You may be vulnerable if: You are running a vulnerable version (see below) Your cluster nodes run in an environment where untrusted hosts share the same layer 2 domain (i.e. same LAN) as nodes Your cluster allows untrusted pods to run containers with (the Kubernetes default is to allow this capability). Your nodes (or hostnetwork pods) run any localhost-only services which do not require any further authentication. To list services that are potentially affected, run the following commands on nodes: - On a master node, an lsof entry like this indicates that the API server may be listening with an insecure port: kubelet/kube-proxy v1.18.0-1.18.3 kubelet/kube-proxy v1.17.0-1.17.6 kubelet/kube-proxy <=1.16.10 Prior to upgrading, this vulnerability can be mitigated by manually adding an iptables rule on nodes. This rule will reject traffic to 127.0.0.1 which does not originate on the node. Additionally, if your cluster does not already have the API Server insecure port disabled, we strongly suggest that you disable it.


Question: nodePortAddresses []string, kernelHandler KernelHandler, ) (*Proxier, error) { <del> // Set the route_localnet sysctl we need for if err := utilproxy.EnsureSysctl(sysctl, sysctlRouteLocalnet, 1); err != nil { return nil, err } </del> // Proxy needs br_netfilter and bridge-nf-call-iptables=1 when containers // are connected to a Linux bridge (but not SDN bridges).  Until most // plugins handle this, log when config is missing

Add the following flag to your kubernetes API server command line: Packets on the wire with an IPv4 destination in the range 127.0.0.0/8 and a layer-2 destination MAC address of a node may indicate that an attack is targeting this vulnerability. Although the issue is caused by , the current fix for the issue is in (although future versions may have the fix in instead). We recommend updating both and to be sure the issue is addressed. The following versions contain the fix: kubelet/kube-proxy master - fixed by kubelet/kube-proxy v1.18.4+ - fixed by kubelet/kube-proxy v1.17.7+ - fixed by kubelet/kube-proxy v1.16.11+ - fixed by To upgrade, refer to the documentation: This issue was originally raised in issue which details how the sets which causes the system not to reject traffic to localhost which originates on other hosts. IPv6-only services that bind to a address are not affected. There may be additional attack vectors possible in addition to those fixed by and its cherry-picks. For those attacks to succeed, the target service would need to be UDP and the attack could only rely upon sending UDP datagrams since it wouldn't receive any replies. Finally, the target node would need to have reverse-path filtering disabled for an attack to have any effect. Work is ongoing to determine whether and how this issue should be fixed. See for up-to-date status on this issue. This vulnerability was reported by János Kövér, Ericsson with additional impacts reported by Rory McCune, NCC Group and Yuval Avrahami and Ariel Zelivansky, Palo Alto Networks. /area security /kind bug /committee product-security /sig network /sig node /area kubelet
For insecure port removal:
The original announcement contained an error. The announcement makes it sound like upgrading kube-proxy will address the issue, but that is incorrect. I have edited the issue summary to add the following paragraph in the "Fixed Versions" section: Additionally, I have changed the components in listed versions to say kubelet/kube-proxy instead of kube-proxy.
Could you add numerical CVSS scores to these? "High" is too vague, frequently varies from vendor to vendor. I can calculate a score from the vector myself, but adding the score would save us some work...
Edited.