Question: // } } } <ins> func TestRateLimitedRoundTripper(t *testing.T) { handler := utiltesting.FakeHandler{StatusCode: 200} server := httptest.NewServer(&handler) defer server.Close() method := "GET" path := "/foo/bar" req, err := http.NewRequest(method, server.URL+path, nil) if err != nil { t.Errorf("unexpected error: %v", err) } // TODO(zmerlynn): Validate the rate limiter is actually getting called. client := http.Client{ Transport: &rateLimitedRoundTripper{ rt:      http.DefaultTransport, limiter: flowcontrol.NewFakeAlwaysRateLimiter(), }, } _, err = client.Do(req) if err != nil { t.Errorf("unexpected error: %v", err) } handler.ValidateRequest(t, path, method, nil) } </ins>

On a large cluster, the routecontroller basically makes O(n) requests all at once, and most of them error out with "Rate Limit Exceeded". We then wait for operations to complete on the ones that did stick, and damn the rest, they'll be caught next reconcile (which is a while). In fact, in some cases it appears we're spamming hard enough to get rate limited by the upper-level API DOS protections and seeing something akin to , e.g.: We can do better than this!
c.f. , which is the other leg of this problem
I took a look into code and it seems that we are doing full reconciliation every 10s. If we need to create 2000 routes at the beginning it seems to frequently. I think that we need to: first compute how many routes we have to create based on that try to spread them over some time to reduce possibility of causing 'rate limit exceeded' I will try to put some small PR together as a starting point.
I'm reopening this one, since this doesn't seem to be solved. So basically something strange is happening here. With PR we have 10/s limit on API calls send to GCE. And we are still rate-limitted. However, according to documentation we are allowed for 20/s: So something either doesn't work or is weird.
What is super interesting from the logs of recent run: It starts to creating routes: After 2 minutes from that, we are hitting "Rate limit Exceeded" for the first time: And only after 9 minutes we create the first route ever. The second one is created after 15 minutes, and from that point we create a number of them So it seems that nothing useful is happening for the first 2 minutes, and then between 3m30 and 9m30
OK - so I think that what is happening is that all controllers are sharing GCE interface (and thuse they have common throttling). So if one controller is generating a lot of API calls, then other controllers may be throttled. has a hypothesis that it may be caused by nodecontroller.
One thing my PR didn't address (because it was late): Now we have some obnoxious differences in logging when you go to make a request, because the versus could actually be absorbing a fair amount of ratelimiter.


Question: // } } } <ins> func TestRateLimitedRoundTripper(t *testing.T) { handler := utiltesting.FakeHandler{StatusCode: 200} server := httptest.NewServer(&handler) defer server.Close() method := "GET" path := "/foo/bar" req, err := http.NewRequest(method, server.URL+path, nil) if err != nil { t.Errorf("unexpected error: %v", err) } // TODO(zmerlynn): Validate the rate limiter is actually getting called. client := http.Client{ Transport: &rateLimitedRoundTripper{ rt:      http.DefaultTransport, limiter: flowcontrol.NewFakeAlwaysRateLimiter(), }, } _, err = client.Do(req) if err != nil { t.Errorf("unexpected error: %v", err) } handler.ValidateRequest(t, path, method, nil) } </ins>

It could before because of the operation rate limiter and operation polling anyways, but now could be worse.
- yes I'm aware of that. And probably this is exactly the case here.
I got fooled by that - I guess we should fix it somehow.
One approach to fixing it is a similar approach to the operation polling code, but in the : warn if we ratelimited for more than N seconds (and dump the request or something? not clear how to clarify what we were sending).
I found something else - Kubelet is also building GCE interface: We have 2000 kubelets, so if all of them send multiple requests to GCE, we may hit limits.
Any idea in what circumstances Kubelet will contact cloud provider?
Hmm - it seems it contacts GCE exactly once at the beginning. So it doesn't explain much.
I think it's only here: I've rarely seen GCE ratelimit readonly calls, in practice, and kubelets come up pretty well staggered.
OK - so with logs that I , it is pretty clear that we are very heavily throttled on GCE api calls. In particular, when I was running 1000-node cluster, I see lines where we were throttle for 1m30. I'm pretty sure it's significantly higher in 2000-node cluster. I'm going to investigate a bit more where all those reuqests come from.
Actually - it seems that we have a pretty big problem here. Basically, Kubelet seems to send a lot of requests to GCE. So to clarify: every time Kubelet is updating NodeStatus (so every 10s by default) it is sending an API call to GCE to get it's not addresses: that means, that if we have 2000 nodes, kubelets generate 200 QPS to GCE (which is significantly more than we can afford) We should think how to solve this issue, but I think the options are: increasing QPS quota or calling GCE only once per X statusUpdates.
The second issue is that we are throttled at the controller level too, and this one I'm tryin to understand right now.


Question: // } } } <ins> func TestRateLimitedRoundTripper(t *testing.T) { handler := utiltesting.FakeHandler{StatusCode: 200} server := httptest.NewServer(&handler) defer server.Close() method := "GET" path := "/foo/bar" req, err := http.NewRequest(method, server.URL+path, nil) if err != nil { t.Errorf("unexpected error: %v", err) } // TODO(zmerlynn): Validate the rate limiter is actually getting called. client := http.Client{ Transport: &rateLimitedRoundTripper{ rt:      http.DefaultTransport, limiter: flowcontrol.NewFakeAlwaysRateLimiter(), }, } _, err = client.Do(req) if err != nil { t.Errorf("unexpected error: %v", err) } handler.ValidateRequest(t, path, method, nil) } </ins>

- that said, any rate-limitting on our side, will not solve the kubelet-related problem
Hmm - actually looking into implementation it seems that NodeAddresses are only contacting metadata server, so I'm not longer that convinced...
Also, I think that important thing is that a single call to GCE API can translate to multiple api calls to GCE. In particular - CreateRoute translates to: get instance insert route a bunch of getOperation (until this is finished)
OK - so I think that what is happening here is that since CreateRoute translates to: get instance insert route a bunch of getOperations That means that if we call say 2000 CreateRoute() at the same time, all of them will try to issue: getInstance at the beginning. So we will quickly accumulate 2000 GET requests in the queue. Once they are processed, we generate the "POST route" then for any processed. So It's kind of expected what is happening here. So getting back to the Zach`s PR - I think that throttling POST and GET and OPERATION calls separate is kind of good idea in general.
But I still don't really understand why do we get "RateLimitExceeded". Is that because of Kubelets?
I think that one problem we have at the RouteController level is that if CreateRoute fails, we simply don't react on it. And we will wait at least 5 minutes before even retrying it. We should add some retries at the route controller level too. I will send out some short PR for it.
But I still don't understand why do we get those "RateLimitExceeded" error
OK - so as an update. In GCE, we have a separate rate-limit for number of in-flight CreateRoute call per project. is supposed to address this issue.
OK - so this one is actually fixed. I mean that it is still long, but currently it's purely blocked on GCE.
As a comment - we have an internal bug for it already.


Question: # TODO(pjh): move the logging agent code below into a separate # module; it was put here temporarily to avoid disrupting the file layout in # the K8s release machinery. <del> $LOGGINGAGENT_VERSION = '1.7.3' </del> <ins> $LOGGINGAGENT_VERSION = '1.7.5' </ins> $LOGGINGAGENT_ROOT = 'C:fluent-bit' $LOGGINGAGENT_SERVICE = 'fluent-bit' $LOGGINGAGENT_CMDLINE = '*fluent-bit.exe*'

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --On Windows 20H2 we are observing crashloops with fluent-bit due to a malformed Windows Event Log entry in the Security log. Fluent-bit to not crash loop. Bring up a GCE Windows cluster on Windows 20H2. You may see a loss of logs. Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /sig windows /triage accepted


Question: "//pkg/api:go_default_library", "//pkg/api/v1/ref:go_default_library", "//pkg/kubelet/apis/cri/v1alpha1/runtime:go_default_library", <del> "//pkg/kubelet/events:go_default_library", </del> "//pkg/kubelet/util/format:go_default_library", "//pkg/kubelet/util/ioutils:go_default_library", "//pkg/util/hash:go_default_library",

Is this a BUG REPORT or FEATURE REQUEST?: What happened: Now the and has been consistent in by PR . There is no need to convert v1.ObjectReference to clientv1.ObjectReference in . What you expected to happen: Remove this function and cleanup the codes which call this function. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?:
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <bre.g., for API Machinery<br(2) specifying the label manually: <bre.g., for sig/scalability<br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
Will submit the cleanup after PR merged.
/sig api-machinery
/sub


Question: return err } // Unmount the device from the device mount point. <del> klog.V(4).Infof("rbd: unmouting device mountpoint %s", deviceMountPath) if err = detacher.mounter.Unmount(deviceMountPath); err != nil { </del> <ins> notMnt, err := detacher.mounter.IsLikelyNotMountPoint(deviceMountPath) if err != nil { </ins> return err } <del> klog.V(3).Infof("rbd: successfully umount device mountpath %s", deviceMountPath) </del> <ins> if !notMnt { klog.V(4).Infof("rbd: unmouting device mountpoint %s", deviceMountPath) if err = detacher.mounter.Unmount(deviceMountPath); err != nil { return err } klog.V(3).Infof("rbd: successfully umount device mountpath %s", deviceMountPath) } </ins> <del> klog.V(4).Infof("rbd: detaching device %s", devicePath) err = detacher.manager.DetachDisk(detacher.plugin, deviceMountPath, devicePath) if err != nil { return err </del> <ins> // Get devicePath from deviceMountPath if devicePath is empty if devicePath == "" { rbdImageInfo, err := getRbdImageInfo(deviceMountPath) if err != nil { return err } found := false devicePath, found = getRbdDevFromImageAndPool(rbdImageInfo.pool, rbdImageInfo.name) if !found { klog.Warningf("rbd: can't found devicePath for %v. Device is already unmounted, Image %v, Pool %v", deviceMountPath, rbdImageInfo.pool, rbdImageInfo.name) } } if devicePath != "" { klog.V(4).Infof("rbd: detaching device %s", devicePath) err = detacher.manager.DetachDisk(detacher.plugin, deviceMountPath, devicePath) if err != nil { return err } klog.V(3).Infof("rbd: successfully detach device %s", devicePath) </ins> } <del> klog.V(3).Infof("rbd: successfully detach device %s", devicePath) </del> err = os.Remove(deviceMountPath) if err != nil { return err

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: Pod was deleted successfully but there is still a lock on the ceph image. I find two logs,one for that lock not removed and another for that lock removed. I find that there are little difference between them. the log that lock not removed: and the log that lock removed: What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): v1.6.7 Cloud provider or hardware configuration**: OS (e.g. from /etc/os-release): centos7 Kernel (e.g. ): 3.10.0-327.el7.x86_64 Install tools: Others:
/sig storage
Error 16 is EBUSY. similar to
were you running kubelet with cadvisor on?
/assign
yes, every kubelet run with cadvisor,and i know what code 16 represent for,but i still don't know the reason, is this a bug? How can i avoid this kind of situation from happening,thank you
It is probably related to
We see some other case may cause , kubelet run with nodeexporter and nodeexporter is deployed by k8s daemonset with -v /:/rootfs:ro (mount kubelet host rootfs into nodeexporter container). rbd device is mounted before nodeexporter started, because k8s 1.6.7 not support mount propagation, when the rbd plugin unmount /dev/rbd0 on the host namespace, the mount info will still leave over in node_exporter container namespace, may cause . hope this would help
thanks, does the problem persist if node_exporter stops collecting filesystem metrics?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with .


Question: return err } // Unmount the device from the device mount point. <del> klog.V(4).Infof("rbd: unmouting device mountpoint %s", deviceMountPath) if err = detacher.mounter.Unmount(deviceMountPath); err != nil { </del> <ins> notMnt, err := detacher.mounter.IsLikelyNotMountPoint(deviceMountPath) if err != nil { </ins> return err } <del> klog.V(3).Infof("rbd: successfully umount device mountpath %s", deviceMountPath) </del> <ins> if !notMnt { klog.V(4).Infof("rbd: unmouting device mountpoint %s", deviceMountPath) if err = detacher.mounter.Unmount(deviceMountPath); err != nil { return err } klog.V(3).Infof("rbd: successfully umount device mountpath %s", deviceMountPath) } </ins> <del> klog.V(4).Infof("rbd: detaching device %s", devicePath) err = detacher.manager.DetachDisk(detacher.plugin, deviceMountPath, devicePath) if err != nil { return err </del> <ins> // Get devicePath from deviceMountPath if devicePath is empty if devicePath == "" { rbdImageInfo, err := getRbdImageInfo(deviceMountPath) if err != nil { return err } found := false devicePath, found = getRbdDevFromImageAndPool(rbdImageInfo.pool, rbdImageInfo.name) if !found { klog.Warningf("rbd: can't found devicePath for %v. Device is already unmounted, Image %v, Pool %v", deviceMountPath, rbdImageInfo.pool, rbdImageInfo.name) } } if devicePath != "" { klog.V(4).Infof("rbd: detaching device %s", devicePath) err = detacher.manager.DetachDisk(detacher.plugin, deviceMountPath, devicePath) if err != nil { return err } klog.V(3).Infof("rbd: successfully detach device %s", devicePath) </ins> } <del> klog.V(3).Infof("rbd: successfully detach device %s", devicePath) </del> err = os.Remove(deviceMountPath) if err != nil { return err

Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
I'm experiencing this issue. From what I can read above, if the rbd mount happens before the node_exporter starts; we'll always experience status 16?
We're getting this issue. This is because the mounted volume is mounted on Cadvisor container too. So the /dev/rbdX file is still in use. But still didn't find any solution to fix.
rbd volume will stuck when failed as will always returen
My solution: container use a volumeMount. make a script file to be executed every a minute:


Question: nsFlag := fmt.Sprintf("--namespace=%v", ns) ginkgo.By("executing a command with run and attach with stdin") <del> runOutput := framework.NewKubectlCommand(nsFlag, "run", "run-test", "--image="+busyboxImage, "--restart=OnFailure", "--attach=true", "--stdin", "--", "sh", "-c", "cat && echo 'stdin closed'"). WithStdinData("abcd1234"). </del> <ins> // We wait for a non-empty line so we know kubectl has attached runOutput := framework.NewKubectlCommand(nsFlag, "run", "run-test", "--image="+busyboxImage, "--restart=OnFailure", "--attach=true", "--stdin", "--", "sh", "-c", "while [ -z "$s" ]; do read s; sleep 1; done; echo read:$s && cat && echo 'stdin closed'"). WithStdinData("valuenabcd1234"). </ins> ExecOrDie() <del> g := func(pods []*v1.Pod) sort.Interface { return sort.Reverse(controller.ActivePods(pods)) } runTestPod, _, err := polymorphichelpers.GetFirstPod(f.ClientSet.CoreV1(), ns, "run=run-test", 1*time.Minute, g) gomega.Expect(err).To(gomega.BeNil()) // NOTE: we cannot guarantee our output showed up in the container logs before stdin was closed, so we have // to loop test. err = wait.PollImmediate(time.Second, time.Minute, func() (bool, error) { if !e2epod.CheckPodsRunningReady(c, ns, []string{runTestPod.Name}, 1*time.Second) { e2elog.Failf("Pod %q of Job %q should still be running", runTestPod.Name, "run-test") } logOutput := framework.RunKubectlOrDie(nsFlag, "logs", runTestPod.Name) gomega.Expect(runOutput).To(gomega.ContainSubstring("abcd1234")) gomega.Expect(runOutput).To(gomega.ContainSubstring("stdin closed")) return strings.Contains(logOutput, "abcd1234"), nil }) gomega.Expect(err).To(gomega.BeNil()) </del> <ins> gomega.Expect(runOutput).To(gomega.ContainSubstring("read:value")) gomega.Expect(runOutput).To(gomega.ContainSubstring("abcd1234")) gomega.Expect(runOutput).To(gomega.ContainSubstring("stdin closed")) </ins> gomega.Expect(c.BatchV1().Jobs(ns).Delete("run-test", nil)).To(gomega.BeNil())

The test case "executing a command with run and attach without stdin" is inherently racy. The test runs, and expected to see in the output. The case when the test passes: case 1: Kubernetes creates and starts the container without stdin; The container exits before attach runs; kubectl falls back to get container log, which contains . case 2: Kubernetes creates and starts the container without stdin; kubectl attach before "stdin closed" is printed; kubectl attach contains . The case when the test fails: Kubernetes creates and starts the container without stdin; The container prints "stdin closed", but hasn't exited yet; kubectl attach contains nothing. One example failure: Related issue: Can we modify this test not to check the output? Or use to check output instead? /cc
Uploaded to try to address
/reopen This test is still inherently racy, even after
Fix in (I hope!)
Moving into the current milestone as it was not fixed in 1.14
Hello I'm a bug triage shadow for the 1.16 release cycle and considering this issue is tagged for 1.16, I'd like to check its status. Code freeze is starting on August 29th (about 3 days from now). Do we still target this issue to be fixed for 1.16?
Thanks . PR is available and awaiting review!
What is the PR in question? Is it targeting 1.16 or 1.17?
/reopen This test is having a lot of failures these days
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
is right, it seems to be present in kind jobs now only
/assign let's see if we start to recover a bit of stability :smile:
The rate of failures decreased a lot since last changes Checking the logs of the failures it seems that the problem is that the pod has not finished before checking the output, as Lantao described The test fails Collecting events from namespace The framework logs the pods the kubelet thinks is on the node
note that it's still racy, but it's hopefully somewhat mitigated.
/reopen
Reopened this issue.


Question: nsFlag := fmt.Sprintf("--namespace=%v", ns) ginkgo.By("executing a command with run and attach with stdin") <del> runOutput := framework.NewKubectlCommand(nsFlag, "run", "run-test", "--image="+busyboxImage, "--restart=OnFailure", "--attach=true", "--stdin", "--", "sh", "-c", "cat && echo 'stdin closed'"). WithStdinData("abcd1234"). </del> <ins> // We wait for a non-empty line so we know kubectl has attached runOutput := framework.NewKubectlCommand(nsFlag, "run", "run-test", "--image="+busyboxImage, "--restart=OnFailure", "--attach=true", "--stdin", "--", "sh", "-c", "while [ -z "$s" ]; do read s; sleep 1; done; echo read:$s && cat && echo 'stdin closed'"). WithStdinData("valuenabcd1234"). </ins> ExecOrDie() <del> g := func(pods []*v1.Pod) sort.Interface { return sort.Reverse(controller.ActivePods(pods)) } runTestPod, _, err := polymorphichelpers.GetFirstPod(f.ClientSet.CoreV1(), ns, "run=run-test", 1*time.Minute, g) gomega.Expect(err).To(gomega.BeNil()) // NOTE: we cannot guarantee our output showed up in the container logs before stdin was closed, so we have // to loop test. err = wait.PollImmediate(time.Second, time.Minute, func() (bool, error) { if !e2epod.CheckPodsRunningReady(c, ns, []string{runTestPod.Name}, 1*time.Second) { e2elog.Failf("Pod %q of Job %q should still be running", runTestPod.Name, "run-test") } logOutput := framework.RunKubectlOrDie(nsFlag, "logs", runTestPod.Name) gomega.Expect(runOutput).To(gomega.ContainSubstring("abcd1234")) gomega.Expect(runOutput).To(gomega.ContainSubstring("stdin closed")) return strings.Contains(logOutput, "abcd1234"), nil }) gomega.Expect(err).To(gomega.BeNil()) </del> <ins> gomega.Expect(runOutput).To(gomega.ContainSubstring("read:value")) gomega.Expect(runOutput).To(gomega.ContainSubstring("abcd1234")) gomega.Expect(runOutput).To(gomega.ContainSubstring("stdin closed")) </ins> gomega.Expect(c.BatchV1().Jobs(ns).Delete("run-test", nil)).To(gomega.BeNil())

<detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
any updates here?
nope, totally agree with the description, is racy by design, couldn't find any way or trick to avoid the randomness
EDIT: It's late and I'm actually not sure I'm remembering this correctly, take this comment with a heap of salt: Specific sub-tests were fixed previously I think, but this one sub-case: That one we can't use the "Wait for stdin input to signal we're attached" hack, so it's just racy and broken. We could remove that case maybe. I'm not sure who I'd want to weigh in on that, probably someone more active in node / CLI.
not super common atm but still happening. discussing
/unassign has a fix for this


Question: CreateOrUpdate(ctx context.Context, resourceGroupName string, publicIPAddressName string, parameters network.PublicIPAddress) (resp *http.Response, err error) Delete(ctx context.Context, resourceGroupName string, publicIPAddressName string) (resp *http.Response, err error) Get(ctx context.Context, resourceGroupName string, publicIPAddressName string, expand string) (result network.PublicIPAddress, err error) <ins> GetVirtualMachineScaleSetPublicIPAddress(ctx context.Context, resourceGroupName string, virtualMachineScaleSetName string, virtualmachineIndex string, networkInterfaceName string, IPConfigurationName string, publicIPAddressName string, expand string) (result network.PublicIPAddress, err error) </ins> List(ctx context.Context, resourceGroupName string) (result []network.PublicIPAddress, err error) }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Node external IP of azure VMSS cannot be obtained when cloud-controller-manager is enabled. Corresponding logs are shown as follows:
/sig azure
/milestone v1.16


Question: } } } func TestPrintPod(t *testing.T) { tests := []struct { pod    api.Pod

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: After labeling clusters in a Federation, when you type the field is empty. What you expected to happen: The fiels should contain information about the labels in the cluster How to reproduce it (as minimally and precisely as possible): a federation control pane in any Cloud Provider and any DNS provider your cluster: info aboyut the cluster with the flag: Note: if you describe it, you see the labels being : Environment: Kubernetes version (use ): 1.8.0 Cloud provider or hardware configuration**: anyone OS (e.g. from /etc/os-release): Ubuntu 16.04 Kernel (e.g. ): Linux Install tools: curl Others:
/sig multicluster
I'm working on it.


Question: } }) <del> ginkgo.It("should manage the lifecycle of a ResourceQuota", func() { </del> <ins> /* Release: v1.25 Testname: ResourceQuota, manage lifecycle of a ResourceQuota Description: Attempt to create a ResourceQuota for CPU and Memory quota limits. Creation MUST be successful. Attempt to list all namespaces with a label selector which MUST succeed. One list MUST be found. The ResourceQuota when patched MUST succeed. Given the patching of the ResourceQuota, the fields MUST equal the new values. It MUST succeed at deleting a collection of ResourceQuota via a label selector. */ framework.ConformanceIt("should manage the lifecycle of a ResourceQuota", func() { </ins> client := f.ClientSet ns := f.Namespace.Name

[x] test approval issue : [x] test pr : 7 July 2022 [X] two weeks soak end date : 21 July 2022 [X] test promotion pr : According to this APIsnoop query, there is still some remaining ResourceQuota endpoints which are untested. The Status endpoints will be addressed in another test. listCoreV1ResourceQuotaForAllNamespacespatchCoreV1NamespacedResourceQuotadeleteCoreV1CollectionNamespacedResourceQuota Using a number of existing e2e test practices a new has been created for three ResourceQuota endpoints. The e2e logs for this test are listed below. This query shows the endpoints hit within a short period of running the e2e test If a test with these calls gets merged, test coverage will go up by 3 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/triage accepted
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: ds.network = network.NewPluginManager(plug) klog.Infof("Docker cri networking managed by %v", plug.Name()) <del> // NOTE: cgroup driver is only detectable in docker 1.11+ cgroupDriver := defaultCgroupDriver dockerInfo, err := ds.client.Info() klog.Infof("Docker Info: %+v", dockerInfo) if err != nil { klog.Errorf("Failed to execute Info() call to the Docker client: %v", err) klog.Warningf("Falling back to use the default driver: %q", cgroupDriver) } else if len(dockerInfo.CgroupDriver) == 0 { klog.Warningf("No cgroup driver is set in Docker") klog.Warningf("Falling back to use the default driver: %q", cgroupDriver) } else { cgroupDriver = dockerInfo.CgroupDriver } if len(kubeCgroupDriver) != 0 && kubeCgroupDriver != cgroupDriver { return nil, fmt.Errorf("misconfiguration: kubelet cgroup driver: %q is different from docker cgroup driver: %q", kubeCgroupDriver, cgroupDriver) </del> <ins> // skipping cgroup driver checks for Windows if runtime.GOOS == "linux" { // NOTE: cgroup driver is only detectable in docker 1.11+ cgroupDriver := defaultCgroupDriver dockerInfo, err := ds.client.Info() klog.Infof("Docker Info: %+v", dockerInfo) if err != nil { klog.Errorf("Failed to execute Info() call to the Docker client: %v", err) klog.Warningf("Falling back to use the default driver: %q", cgroupDriver) } else if len(dockerInfo.CgroupDriver) == 0 { klog.Warningf("No cgroup driver is set in Docker") klog.Warningf("Falling back to use the default driver: %q", cgroupDriver) } else { cgroupDriver = dockerInfo.CgroupDriver } if len(kubeCgroupDriver) != 0 && kubeCgroupDriver != cgroupDriver { return nil, fmt.Errorf("misconfiguration: kubelet cgroup driver: %q is different from docker cgroup driver: %q", kubeCgroupDriver, cgroupDriver) } klog.Infof("Setting cgroupDriver to %s", cgroupDriver) ds.cgroupDriver = cgroupDriver </ins> } <del> klog.Infof("Setting cgroupDriver to %s", cgroupDriver) ds.cgroupDriver = cgroupDriver </del> ds.versionCache = cache.NewObjectCache( func() (interface{}, error) { return ds.getDockerVersion()

What happened: I have a Kubernetes Cluster with 2 Windows Server 2019 Core VMs as worker nodes. I upgraded first the cluster, and then the first Windows node following the docs here: Every step ran without issue, but the kubelet service was not started. This was not immediately obvious, since "restart-service kubelet" does not propagate the error. I think "start-service kubelet" would be better advice. The logs said: After changing the value of cgroupDriver manually in from "systemd" to "cgroupfs", the kubelet started successfully. What you expected to happen: One of 2 things: Either the Windows upgrade docs telling me what I need to do to fix this, or the kubeadm not breaking the configuration for windows in the first place. How to reproduce it (as minimally and precisely as possible): a windows node to a Kubernetes v1.19 cluster via kubeadm the cluster via kubeadm the Windows node following the docs Anything else we need to know?: Environment: Kubernetes version (use ): 1.20.1 Cloud provider or hardware configuration: OS (e.g: ): Windows Server 2019 Core Kernel (e.g. ): Install tools: kubeadm Network plugin and version (if this is a network-related bug): Others: /sig windows
this may cause by the default value of defaultCgroupDriver cgroupfs Would you provide the from Windows node?
/assign
kubeadm doesn't do anything about cgroup drivers on Windows because cgroups don't exist on Windows. ideally the kubelet checks about this should be skipped entirely in dockershim for non-Linux.
! However, windows docker return as cgroupfs.
probably 's output is the same structure on all OSes with being a default value. this field should not be checked on Windows at all.
/retitle kubelet: don't check for the Docker cgroup driver on Windows
adding since the PR is favorable. /triage accepted


Question: } func TestValidateDisabledSubpathExpr(t *testing.T) { <del> // Enable feature VolumeSubpathEnvExpansion defer featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, features.VolumeSubpathEnvExpansion, true)() </del> volumes := []core.Volume{ {Name: "abc", VolumeSource: core.VolumeSource{PersistentVolumeClaim: &core.PersistentVolumeClaimVolumeSource{ClaimName: "testclaim1"}}},

/kind cleanup /sig storage /sig node Description: Feature gate VolumeSubpathEnvExpansion was GA in 1.17 and is marked for removal in 1.19 Related Issues: GA Release:
/assign
/cc


Question: return false } bucketID := nextTime.Unix() <ins> t.lock.Lock() defer t.lock.Unlock() </ins> if bucketID < t.startBucketID { bucketID = t.startBucketID }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: I try to backport bookmark feature to our internal branch. When I enable bookmark feature, apiserver leaks memory. I spend some time to dig the root cause and find the bookmarkWatchers of service cacher is keep increasing. I am not sure if it exists at upstream code. Please let me know if it is ok at latest code. What you expected to happen: Avoid memory leaking How to reproduce it (as minimally and precisely as possible): a cluster. not update service. will find kube-apiserver's memory keep increasing. Anything else we need to know?: Kubelet will watch service with bookmark, and add a watcher to bookmarkWatchers. and service init resource version is 0 Without updating service, we cannot update "lastProcessedResourceVersion", it will keep 0, bookmarkTimer branch will always be continued. As a result, it cannot trigger popExpiredWatchers, so the service watcher will keep increasing every watching. Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig scalability PLAT :)
How about making this change ?
- can you also please take a look?
Agree with u. That's same way I fix in our branch. But I wanna hear the suggestion of and .
! With the same bugfix wrote above, we can see the memory not increasing any more.
Given that watch-bookmarks doesn't create any additional go-routines (well, creates 1, but cannot leak them), it has to leak the data. The only data that can be leaked is exactly bookmarkWatchers. So I looked into the code and got into exactly the same conclusion - that we must be leaking watchers for resources that aren't changing. The fix as above makes sense (though we also need a unit test for it).
Thanks a lot.
has the fix by
Thanks for reporting the bug, I will look at this day later. In addition, also need to think carefully about whether need to send a bookmark when there are no events at all.
cc


Question: } func (d *CachedDiscoveryClient) writeCachedFile(filename string, obj runtime.Object) error { <del> if err := os.MkdirAll(filepath.Dir(filename), 0755); err != nil { </del> <ins> if err := os.MkdirAll(filepath.Dir(filename), 0750); err != nil { </ins> return err }

In kubectl v1.8.0+, schema info is cached in the location specified by (defaulting to ), written with world-writeable permissions (rw-rw-rw-). If is specified and pointed at a different location accessible to other users/groups, the written files may be modified by other users/groups and disrupt the kubectl invocation. CVSS score: CVSS:3.0/AV:L/AC:H/PR:L/UI:R/S:U/C:L/I:L/A:N (3.3, low) What versions are affected? kubectl v1.8.0+ What configurations are affected? Invocations that point at world-writeable locations Impact Malformed responses written to the cache directory can disrupt the kubectl invocation Workaround Use the default location in the $HOME directory or point it at a directory that is only accessible to desired users/groups. (original description follows) ==== What happened: The files inside of ".kube/http-cache" are world writeable (rw-rw-rw-). While the default for these files appears to be the home directory, using the "--cache-dir" flag could put these files into a place where world writeable files would allow any user / process to modify the cache files. Modification of the cache files could influence the kubectl utility in a negative way for other users. What you expected to happen: Apply stricter file permissions to the http-cache files. How to reproduce it (as minimally and precisely as possible): Run any generic kubectl command which is successful and then list the cache directory $ kubectl get pods --all-namespaces $ ls -la Anything else we need to know?


Question: } func (d *CachedDiscoveryClient) writeCachedFile(filename string, obj runtime.Object) error { <del> if err := os.MkdirAll(filepath.Dir(filename), 0755); err != nil { </del> <ins> if err := os.MkdirAll(filepath.Dir(filename), 0750); err != nil { </ins> return err }

: I estimate this is a low severity security issue with a CVSS score of "3.3 / CVSS:3.0/AV:L/AC:L/PR:L/UI:N/S:U/C:N/I:L/A:N" - Environment: Linux Kubernetes version (use kubectl version):Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.6", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-02-26T12:49:28Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.6", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-02-26T12:49:28Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: AWS. Running kube api server in hyperkube. OS (e.g: cat /etc/os-release): NAME="CentOS Linux" VERSION="7.1808 (Core)" ID="centos" IDLIKE="rhel fedora" VERSIONID="7" PRETTYNAME="CentOS Linux 7.1808 (Core)" ANSICOLOR="0;31" CPENAME="cpe:/o:centos:centos:7" HOMEURL="" BUGREPORTURL="" CENTOSMANTISBTPROJECT="CentOS-7" CENTOSMANTISBTPROJECTVERSION="7" REDHATSUPPORTPRODUCT="centos" REDHATSUPPORTPRODUCTVERSION="7" OSTREEVERSION=7.1808 Kernel (e.g. uname -a): Linux hackit.internal 3.10.0-862.11.6.el7.x8664 SMP Tue Aug 14 21:49:04 UTC 2018 x8664 x8664 x86_64 GNU/Linux Install tools: Manual installation. Others: n/a
/sig product-security
Closing this issue. <detailsIn response to : if you think this is serious, send an email thanks!


Question: } func (d *CachedDiscoveryClient) writeCachedFile(filename string, obj runtime.Object) error { <del> if err := os.MkdirAll(filepath.Dir(filename), 0755); err != nil { </del> <ins> if err := os.MkdirAll(filepath.Dir(filename), 0750); err != nil { </ins> return err }

</details
Adjusted the CVSS score to to indicate user interaction is required (since they must specify a non-default cache option pointed at a shared writeable location)
There are two places this should be done: PathPerm and FilePerm should be set here: the temp file and final file should have stricter permissions (probably 0660) here:
/cc
/help
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/assign I would like to work on release-1.9, as I just made a hot fix for our team's k8s version of 1.9.
GitHub didn't allow me to assign the following users: yuchengwu. Note that only and repo collaborators can be assigned and that issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign I would like to work on release-1.9, as I just made a hot fix for our team's k8s version of 1.9. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
help fixing this is welcome, though it would only be fixed in releases 1.12+, per
/label official-cve-feed (Related to kubernetes/sig-security)


Question: } service, err := e.serviceLister.Services(namespace).Get(name) if err != nil { <del> // Delete the corresponding endpoint, as the service has been deleted. // TODO: Please note that this will delete an endpoint when a // service is deleted. However, if we're down at the time when // the service is deleted, we will miss that deletion, so this // doesn't completely solve the problem. See #6877. err = e.client.CoreV1().Endpoints(namespace).Delete(name, nil) if err != nil && !errors.IsNotFound(err) { return err } </del> <ins> // Service has been deleted. So no need to do any more operations. </ins> return nil }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: call delete endpoint when service is not found. What you expected to happen: It is no need to do delete in controller. Because in service restStorage, endpoint is deleted when delete service. See How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/area controller-manager /sig apps
/kind bug


Question: kubelet := testKubelet.kubelet kubeClient := testKubelet.fakeKubeClient kubeClient.ReactFn = testclient.NewSimpleFake(&api.NodeList{Items: []api.Node{ <del> {ObjectMeta: api.ObjectMeta{Name: "127.0.0.1"}}, </del> <ins> {ObjectMeta: api.ObjectMeta{Name: testKubeletHostname}}, </ins> }}).ReactFn machineInfo := &cadvisorApi.MachineInfo{ MachineID:      "123",

Looks like some kubelet unit tests perform a lookupIp which ends up depending on the health of the local resolver. Specifically has this failing: Which leads to the test in: And we recently swapped out the node names from testnode to 127.0.0.1, so probably the culprit?
/ref
Interesting. I did swap out the node names in the unit test as part of to be 127.0.0.1, party because I thought that it wouldn't have any problems resolving during unit tests.
/cc


Question: kubelet := testKubelet.kubelet kubeClient := testKubelet.fakeKubeClient kubeClient.ReactFn = testclient.NewSimpleFake(&api.NodeList{Items: []api.Node{ <del> {ObjectMeta: api.ObjectMeta{Name: "127.0.0.1"}}, </del> <ins> {ObjectMeta: api.ObjectMeta{Name: testKubeletHostname}}, </ins> }}).ReactFn machineInfo := &cadvisorApi.MachineInfo{ MachineID:      "123",

cc
What is the environment where you are running the tests? They pass for me on my mac, a linux box, and shippable seems to be passing them as well.
I couldn 't reproduce the failure either.
I'm out right now; i'll try them when i get back. For the record, I run hack/test- Paul Morie wrote:
Ahem FTR This acronym is going to go quad-plat Paul Morie wrote:
I ran on both platforms and saw the tests pass.
I'm still seeing it. Won't have bandwidth to dig in personally until tomorrow.
seeing a similar failure after a git pull
I knew I wasn't crazy
environment: fedora 21, golang: 1.4, docker: 1.6
I think I see the problem. When I set the node name to in the test I was assuming it would parse as a valid ip address and not end up at the LookupIP call. But only the node name (and not the kubelet hostname) is being set (except in my new test where I ) so it's trying to do a lookup. Do you think you could test just changing the default hostname from "testnode" to "127.0.0.1" and see if that fixes it for you?
Trying now
I still get failures but no panic / threaddump.
gist coming
this looks like again?
Good that it isn't panicing any longer. I can dig into the test failures tomorrow (I think it should be relatively simple to resolve).
cool, thanks for looking
Passes in master now for me. Closing.
FYI, they will break again once goes in, and then we can merge my PR to see if that fixes it for you.
:+1:
is in -- are these tests breaking again?
I just rebased which I think will fix them.
They are breaking again.
is on it.


Question: storageframework.BlockVolModeDynamicPV, storageframework.Ext4DynamicPV, storageframework.XfsDynamicPV, <ins> storageframework.NtfsDynamicPV, </ins> } return InitCustomMultiVolumeTestSuite(patterns) }

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --ci-kubernetes-e2e-gci-gce-serial (gce-cos-master-serial) Four tests: Kubernetes e2e suite.[sig-storage] CSI Volumes [Driver: csi-hostpath] [Testpattern: Dynamic PV (default fs)] subPath should unmount if pod is gracefully deleted while kubelet is down [Disruptive][Slow][LinuxOnly] Kubernetes e2e suite.[sig-storage] In-tree Volumes [Driver: hostPath] [Testpattern: Inline-volume (default fs)] subPath should unmount if pod is gracefully deleted while kubelet is down [Disruptive][Slow][LinuxOnly] Kubernetes e2e suite.[sig-storage] In-tree Volumes [Driver: hostPathSymlink] [Testpattern: Inline-volume (default fs)] subPath should unmount if pod is gracefully deleted while kubelet is down [Disruptive][Slow][LinuxOnly] Kubernetes e2e suite.[sig-storage] CSI Volumes [Driver: csi-hostpath] [Testpattern: Dynamic PV (default fs)] subPath should unmount if pod is force deleted while kubelet is down [Disruptive][Slow][LinuxOnly] is failing since June 29th 2021 (but the first failure was different than more recent ones). Commit , infra commit , . Other three tests are failing since July 9th 2021. Commit , infra commit , . Testgrid: https://k8s- Prow history: For CSI Volumes: For In-tree Volumes: There is another open issues for failing tests in this job: Failures reported there are about missing metrics, and were occuring before unmounting problems. So I opened a new issue, even though it's the same SIG. These tests don't look related to me, but I don't have enough context, maybe they somehow are. /sig storage /kind failing-test /milestone 1.22
You must be a member of the GitHub team to set the milestone. If you believe you should be able to issue the /milestone command, please contact your and have them propose you as an additional delegate for this responsibility.


Question: storageframework.BlockVolModeDynamicPV, storageframework.Ext4DynamicPV, storageframework.XfsDynamicPV, <ins> storageframework.NtfsDynamicPV, </ins> } return InitCustomMultiVolumeTestSuite(patterns) }

<detailsIn response to : <!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --ci-kubernetes-e2e-gci-gce-serial (gce-cos-master-serial) Four tests: Testgrid: https://k8s- Prow history: For CSI Volumes: For In-tree Volumes: #### Anything else we need to know: /kind failing-test /milestone 1.22 Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/milestone v1.22
/cc
/cc (my close/reopen yesterday was accidental, sorry)
I can take a look /assign
The non-subpath counterpart of this test is passing: These tests seem to use the default storageclass, whereas failing tests use in-tree hostpath and csi-hostpath
The CSI ones are failing because the hostpath driver never received a NodeUnstage (whereas it did in passing tests), causing the driver to error on volume deletion.
In the failed run kubelet has this error: which seems to have prevented kubelet from triggering a UnmountDevice. This error didn't exist in a successful run.
Hostpath driver can run with and without support for attach. Do you know which mode is enabled for this test?
Between the last passing time and when the test started failing, are there any PRs merged that look suspicious?
According to csi-attacher logs, attach is disabled for both the good run and the failing run. Haven't spotted any obvious PRs so far, but I did find this: in the successful run, the log message for the successful is this: and the failed run: so the reconstructed volume names are somehow different. Checking reconstruction logic
This merged 12 days ago and was a pretty big change to pod lifecycle and could have trickling impact to volume manager. Does the timing of test failure match?
There is also this reconstruction fix that merged yesterday and seems related to outer vs inner spec name. I don't know if that will help fix this issue:
cc
Last successful run started at 7/8 17:49 PT, first failing run started at 7/9 2:23 PT.


Question: storageframework.BlockVolModeDynamicPV, storageframework.Ext4DynamicPV, storageframework.XfsDynamicPV, <ins> storageframework.NtfsDynamicPV, </ins> } return InitCustomMultiVolumeTestSuite(patterns) }

was merged at 7/8 16:48 PT, so it's possible (if there's a delay in picking up master) Also this is in the failing kubelet logs but not in the successful one: I0720 18:20:42. ] "Reconciler sync states: could not find volume information in desired state, clean up the mount points" podName=-e508-45c2-8509- volumeSpecName="pvc--a615-4e89-afe8-"
From , the newly introduced log message here exists in the failing kubelet logs but not the successful one:
Failures starting 07/08: /sig node for visibility
I think caused the failures. Before after kubelet restarts, pod will still be in desired state (even through it is in terminating state), so that clean up mounts can be done completely. I think now, terminating pods will not be back to desired state, but reconstruction to actual state cannot completely clean up mounts. I am testing it about this.
Is there a reason why reconstruction to actual state cannot clean up mounts? Note that a pod can be force deleted at any time, so a terminating pod that is force deleted would still have to be capable of being cleaned up (it won't be in desired state)? (note too that the expectation of the changes in is that terminating and fully terminated and "never seen before" should all completely equivalent from the perspective of other loops in the kubelet) If I understand you correctly, is this an existing bug that we're now triggering consistently?
We have both "grace pod deletion" and "force pod deletion" test cases that are failing.
There were a couple of key changes to volume loops in : Roughly before was "it looks like it's got no running containers", and after my change is "we are going to be stopping all the containers from now on". So this change meant that once a pod starts terminating, we are in "tearing down mode" and so my understanding of this method was that "any pod where containers are being torn down is not in the desired state of the world".


Question: storageframework.BlockVolModeDynamicPV, storageframework.Ext4DynamicPV, storageframework.XfsDynamicPV, <ins> storageframework.NtfsDynamicPV, </ins> } return InitCustomMultiVolumeTestSuite(patterns) }

After a restart, we would definitely not add a pod back that was in terminating state into desired state of the world, but "restart + this change" should be roughly equivalent to "restart and force delete while restart is processing" The other change was in which changed a live CRI read (which was trying to enforce that the dsowp is synchronized with the pod worker) into a threadsafe check against the pod worker directly (the pod worker can provide authoritative answers to "this pod has no running containers" now). The latter change seems less likely to impact you.
Thank you for the details! I think your change is making the right change, but unfortunately some of volume management still have issues, and so the tests rely on some behaviors that is not very right (pod volume is back to desired state even after it is in terminating) The cleanup mount currently does not have the logic to clean up global mount point if the volume plugin sets it up. I am still testing more and will share more details.
is this a release blocker?
Yes, this impacts volume cleanup if kubelet restarts for any reason.
On a kube-up cluster running the latest master against the latest e2e binary, I was able to reproduce the in-tree failures but not the CSI ones for some reason. Will focus on in-tree failures for the time being
/triage accepted
marking release-blocker per note on slack:
/priority critical-urgent (once upon a time we used to not ship a release if any priority/critical-urgent issues were open, not sure if we decided to stop doing that)
We were able to find the trigger for the CSI failures - on 6/29 there was a change in the hostpath driver test manifest, and the driver's is enabled. This makes the driver return an error instead of only logging a warning message on if the volume isn't unstaged properly. IIUC there's a long standing issue that reconstruction doesn't handle node unstage. could you confirm? If that's the case, then these CSI failures are not release blocking. Furthermore, the CSI test with graceful delete started to fail on 7/9 because of since graceful and force deletes are now considered the same from the point of view of the volume manager. cc
When that check was enabled, I don't think we realized it would impact the reconstruction path; it was enabled to validate a different issue.


Question: storageframework.BlockVolModeDynamicPV, storageframework.Ext4DynamicPV, storageframework.XfsDynamicPV, <ins> storageframework.NtfsDynamicPV, </ins> } return InitCustomMultiVolumeTestSuite(patterns) }

Are we running into the issue that reconstruction of staging directories isn't supported () OR is it that our reconstruction path requires drivers to have real mount points ()? Either way since both are known gaps with reconstruction, I think it makes sense to to disable the flag until we can address those issues.
I was looking at this today too, trying to understand the underlying cause. That is why the failures are being reported now, but the part that I didn't understand is why NodeUnstageVolume is not being called for these volumes. In this instance: The DeleteVolume request fails because a staged path is still present: And in the kubelet log, NodeUnpublishVolume is called Then we hit this error: And NodeUnstageVolume is never called for this volume (same for other failures I looked at) so DeleteVolume keeps failing. So I think you're right that it's triggered by reconstruction, and it looks similar to , except in this case the restart happened before NodeUnpublishVolume.
And it's probably not since the test uses a PVC
Will look deeper into why UnmountDevice wasn't issued as well, but some update on the in-tree tests: the force delete test is skipped for the in-tree hostpath CSI driver because of Graceful tests are now failing as well because they behave like force tests due to
I don't think the issue is related to PVC or not. I think it's related to drivers that do not actually mount devices as a separate filesystem than the root.
There are three possible flows during kubelet restart pod gracefully delete case, the pod still exists (it is not considered as terminated in function isPodTerminated(); note that this logic has some changes after , see , so pod volume will be back to desired state, and reconciler will try to set up volume and fill in actual state. When pod volume is finally deleted from desired state, reconciler will trigger teardown with information from actual state to unpublish and unstage volume. pod is forcefully deleted, pod volume will not be in desired state. If volume reconstruction succeeds (be able to reconstruct volume spec), volume will be back into actual state. Then reconciler will trigger teardown with information from actual state to unpublish and unstage volume (this part is similar to case 1).


Question: storageframework.BlockVolModeDynamicPV, storageframework.Ext4DynamicPV, storageframework.XfsDynamicPV, <ins> storageframework.NtfsDynamicPV, </ins> } return InitCustomMultiVolumeTestSuite(patterns) }

last case is reconstruction failed (in this csi-host-path test, reconstruction will fail due to issue related to , it fail to check mount point for csi hostpath driver because it is using a bind mount of the same device ). In this case, reconciler will try to clean up mounts directly, but only published volume path, not the staged volume path (this logic is not implemented due to missing information). As a workaround, we can now disable check-volume-lifecycle if it does not cause other issues. Then we will work on to make sure csi-host-path driver can pass volume mountpoint check during reconstruction.
This issue is now mitigated with merged. There are two longer-term issues that are isolated to hostpath plugins (and maybe other plugins that are similar in behavior, perhaps emptydir and similar?): related to IsLikelyNotMountPoint as mentioned. Because these don't impact the core volume machinery and are not regressions, this issue doesn't need to be release blocking. We'll close this issue once (2) is tracked.
is the IsLikelyNotMountPoint issue
after is merged, two csi-hostpath driver subpath test can pass, and two in-tree driver gracefully delete tests were skipped. https://k8s-
can you please cherry-pick to release-1.22 branch?
Opened cherry-pick, please review:
- what is the plan for reenabling them?
cherry-pick merged!
Cheng and I will work on solving IsLikelyNotMountPoint issue for hostpath drivers. That should allow test to pass and we can enable them.
Can this be removed from the milestone now that the workarounds have been merged, while leaving this issue open to track further work?
We can use existing issue to track the real fix. /close
Closing this issue. <detailsIn response to : We can use existing issue to track the real fix. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: for k, tc := range testcases { tcName := k path := "/apis/" + tc.APIService.Spec.Group + "/" + tc.APIService.Spec.Version + "/foo" <del> called := false </del> <ins> timesCalled := int32(0) </ins> func() { // Cleanup after each test case. backendHandler := http.NewServeMux() backendHandler.Handle(path, websocket.Handler(func(ws *websocket.Conn) { <ins> atomic.AddInt32(&timesCalled, 1) </ins> defer ws.Close() body := make([]byte, 5) ws.Read(body) ws.Write([]byte("hello " + string(body))) <del> called = true </del> })) backendServer := httptest.NewUnstartedServer(backendHandler)

There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig api-machinery


Question: obj.MakeIPTablesUtilChains = boolVar(true) } if obj.IPTablesMasqueradeBit == nil { <del> temp := int32(defaultIPTablesMasqueradeBit) </del> <ins> temp := int32(DefaultIPTablesMasqueradeBit) </ins> obj.IPTablesMasqueradeBit = &temp } if obj.IPTablesDropBit == nil { <del> temp := int32(defaultIPTablesDropBit) </del> <ins> temp := int32(DefaultIPTablesDropBit) </ins> obj.IPTablesDropBit = &temp } if obj.CgroupsPerQOS == nil {

Solution: I will add one.
/cc
xref
Thanks for sending a PR so quickly!
[MILESTONENOTIFIER] Milestone Issue Needs Attention Action required: During code slush, issues in the milestone should be in progress. If this issue is not being actively worked on, please remove it from the milestone. If it is being worked on, please add the label so it can be tracked with other in-flight issues. Note: If this issue is not resolved or labeled as by Wed, Nov 22 it will be moved out of the v1.9 milestone. <details<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details


Question: // This ensures the given VM's Primary NIC's Primary IP Configuration is // participating in the specified LoadBalancer Backend Pool. func (az *Cloud) ensureHostInPool(serviceName string, nodeName types.NodeName, backendPoolID string) error { <ins> var machine compute.VirtualMachine </ins> vmName := mapNodeNameToVMName(nodeName) az.operationPollRateLimiter.Accept() machine, err := az.VirtualMachinesClient.Get(az.ResourceGroup, vmName, "") if err != nil { <del> return err </del> <ins> if az.CloudProviderBackoff { glog.V(2).Infof("ensureHostInPool(%s, %s, %s) backing off", serviceName, nodeName, backendPoolID) machine, err = az.VirtualMachineClientGetWithRetry(az.ResourceGroup, vmName, "") if err != nil { glog.V(2).Infof("ensureHostInPool(%s, %s, %s) abort backoff", serviceName, nodeName, backendPoolID) return err } } else { return err } </ins> } primaryNicID, err := getPrimaryInterfaceID(machine)

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind feature What happened: In certain large cluster scenarios we would like to anticipate kubelet behavior hitting request limits, and backoff in response. E.g., What you expected to happen: This is a request for add'l backoff coverage, so nothing surprising. How to reproduce it (as minimally and precisely as possible): Build a cluster with as many nodes as your subscription will support on azure until kubelet behavior contributes to backend API request limits. Environment: Kubernetes version (use ): and Cloud provider or hardware configuration**: azure OS (e.g. from /etc/os-release): Ubuntu 16.04.2 LTS (Xenial Xerus) Kernel (e.g. ): Linux k8s-pool7--7 4.4.0-83-generic -Ubuntu SMP Mon Jun 26 17:54:43 UTC 2017 x8664 x8664 x86_64 GNU/Linux Install tools: acs-engine Others:
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig azure


Question: // GetPrimaryInterface gets machine primary network interface by node name and vmSet. func (ss *scaleSet) GetPrimaryInterface(nodeName, vmSetName string) (network.Interface, error) { <del> ssName, instanceID, vm, err := ss.getVmssVM(nodeName) </del> <ins> managedByAS, err := ss.isNodeManagedByAvailabilitySet(nodeName) </ins> if err != nil { <del> if err == ErrorNotVmssInstance { glog.V(4).Infof("GetPrimaryInterface: node %q is managed by availability set", nodeName) // Retry with standard type because nodes are not managed by vmss. return ss.availabilitySet.GetPrimaryInterface(nodeName, "") } </del> <ins> glog.Errorf("Failed to check isNodeManagedByAvailabilitySet: %v", err) return network.Interface{}, err } if managedByAS { // vm is managed by availability set. return ss.availabilitySet.GetPrimaryInterface(nodeName, "") } </ins> <del> glog.Errorf("error: ss.GetPrimaryInterface(%s), ss.getCachedVirtualMachine(%s), err=%v", nodeName, nodeName, err) </del> <ins> ssName, instanceID, vm, err := ss.getVmssVM(nodeName) if err != nil { glog.Errorf("error: ss.GetPrimaryInterface(%s), ss.getVmssVM(%s), err=%v", nodeName, nodeName, err) </ins> return network.Interface{}, err }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: VMAS nodes are reporting error in vmss cluster (vmType set to vmss). What you expected to happen: Node should register correctly in the kubernetes cluster. How to reproduce it (as minimally and precisely as possible): Create a VMAS nodes inside vmss cluster. Anything else we need to know?: Environment: Kubernetes version (use ): master Cloud provider or hardware configuration: azure OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig azure


Question: AppArmor:                                    {Default: true, PreRelease: utilfeature.Beta}, DynamicKubeletConfig:                        {Default: false, PreRelease: utilfeature.Alpha}, KubeletConfigFile:                           {Default: false, PreRelease: utilfeature.Alpha}, <del> DynamicVolumeProvisioning:                   {Default: true, PreRelease: utilfeature.Alpha}, </del> ExperimentalHostUserNamespaceDefaultingGate: {Default: false, PreRelease: utilfeature.Beta}, ExperimentalCriticalPodAnnotation:           {Default: false, PreRelease: utilfeature.Alpha}, Accelerators:                                {Default: false, PreRelease: utilfeature.Alpha},

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: kube-apiserver --help still shows this: It looks like the related feature can be turned off. However, dynamic volume provisioning Alpha has retried in 1.6. What you expected to happen: Remove from feature gate to avoid confusion. How to reproduce it (as minimally and precisely as possible): Run Anything else we need to know?: No Environment: Kubernetes version (use ): 1.7+ Cloud provider or hardware configuration**: Any OS (e.g. from /etc/os-release): Any Kernel (e.g. ): Any Install tools: Others:
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig api-machinery
/sig api-machinery
SIG API Machinery doesn't own the contents of the API. /sig storage


Question: tags = ["automanaged"], visibility = ["//visibility:public"], ) <ins> go_test( name = "go_default_test", srcs = ["testutil_test.go"], embed = [":go_default_library"], deps = ["//staging/src/k8s.io/component-base/metrics:go_default_library"], ) </ins>

What would you like to be : Once a metric be deprecated, we should update the unit test accordingly if there is any. We need to work out a way for a unit test freely select codebase version, and also should make sure the unit test won't fail in the subsequent release. Why is this needed: Deprecated also needs to be protected. /sig instrumentation /assign


Question: return GatherAndCompare(registry, expected, metricNames...) } <ins> // NewFakeKubeRegistry creates a fake `KubeRegistry` that takes the input version as `build in version`. // It should only be used in testing scenario especially for the deprecated metrics. // The input version format should be `major.minor.patch`, e.g. '1.18.0'. func NewFakeKubeRegistry(ver string) metrics.KubeRegistry { backup := metrics.BuildVersion defer func() { metrics.BuildVersion = backup }() metrics.BuildVersion = func() apimachineryversion.Info { return apimachineryversion.Info{ GitVersion: fmt.Sprintf("v%s-alpha+1.12345", ver), } } return metrics.NewKubeRegistry() } </ins>


Question: containers, _ := selectContainers(spec.Containers, o.ContainerSelector) if len(containers) != 0 { for i := range containers { <del> if len(o.Limits) != 0 && len(containers[i].Resources.Limits) == 0 { containers[i].Resources.Limits = make(api.ResourceList) } for key, value := range o.ResourceRequirements.Limits { containers[i].Resources.Limits[key] = value } if len(o.Requests) != 0 && len(containers[i].Resources.Requests) == 0 { containers[i].Resources.Requests = make(api.ResourceList) } for key, value := range o.ResourceRequirements.Requests { containers[i].Resources.Requests[key] = value } </del> <ins> containers[i].Resources = o.ResourceRequirements </ins> transformed = true } } else {

from https://k8s-
Really suspect this is
This is putting us into test-go post-commit run is broken
cc:
marked this p0 - if a fix isn't easy let's try to identify what introduced this and revert.
This flake on: 2016-10-30 07:19 PDT First instance: : It is flaking frequently, blocking PRs from getting merged, and slowing the submit queue ( at 75%). As on-call, I'm going to try and identify the cluprit and revert. Can use all help in doing so.
can you investigate whether your additional tests could cause this (by breaking the deployment of those pods, for instance) , Saad Ali wrote:
PR looks most suspect to me. It mucked with , and the errors we are seeing in each of the failures is related to : That PR was merged at Oct 30, 2016, 6:18 AM PDT and the first flake in was at Oct 30, 2016, 7:19 AM PDT. Thoughts? CC
I'm going to initiate a revert of to see if that fixes the issue
IMO we can just change the flaky test to not get pods but some other resources, such as deployments. We can't really expect to clean up pods immediately.
I'm going to proceed with revert (). Please feel free to send a follow up PR to handle this more gracefully.
Going to leave this opened until we have a few hours of runs with the revert. Want to verify, with some degree of certainty, that the revert took care of this flake.
We had 15 green runs of in a row. Considering this flake fixed.


Question: # DO NOT REPORT SECURITY VULNERABILITIES DIRECTLY TO THESE NAMES, FOLLOW THE # INSTRUCTIONS AT https://kubernetes.io/security/ <del> cjcullen joelsmith liggitt philips </del> <ins> cheftako deads2k lavalamp </ins> sttts <del> tallclair </del>

The purpose of the file for each Kubernetes repository is to provide a list of people who can assist the Kubernetes in the event that a security issue related to the repository is discovered or disclosed. As described in the file, those on the list should agree to our . Please update the  file for the sample-controller repository. After finding people who are willing to work in this capacity, you should add them to the list, then remove PSC members (except any PSC member will be working as a security contact for this repository). The list is GitHub usernames, optionally followed by an email address. If no email address is listed, the PSC will use the email address found on git commits made by the listed user. The file may already have people listed who are secuirty contacts. In that case, simply remove any PSC members who aren't also security contacts for the repo. See for more information /area security /committee product-security /kind cleanup /lifecycle frozen /priority important-soon /sig api-machinery


Question: # DO NOT REPORT SECURITY VULNERABILITIES DIRECTLY TO THESE NAMES, FOLLOW THE # INSTRUCTIONS AT https://kubernetes.io/security/ <del> cjcullen joelsmith liggitt philips </del> <ins> cheftako deads2k lavalamp </ins> sttts <del> tallclair </del>

The purpose of the file for each Kubernetes repository is to provide a list of people who can assist the Kubernetes in the event that a security issue related to the repository is discovered or disclosed. As described in the file, those on the list should agree to our . Please update the  file for the sample-apiserver repository. After finding people who are willing to work in this capacity, you should add them to the list, then remove PSC members (except any PSC member will be working as a security contact for this repository). The list is GitHub usernames, optionally followed by an email address. If no email address is listed, the PSC will use the email address found on git commits made by the listed user. The file may already have people listed who are secuirty contacts. In that case, simply remove any PSC members who aren't also security contacts for the repo. See for more information /area security /committee product-security /kind cleanup /lifecycle frozen /priority important-soon /sig api-machinery


Question: # DO NOT REPORT SECURITY VULNERABILITIES DIRECTLY TO THESE NAMES, FOLLOW THE # INSTRUCTIONS AT https://kubernetes.io/security/ <del> cjcullen joelsmith liggitt philips </del> <ins> cheftako deads2k lavalamp </ins> sttts <del> tallclair </del>

The purpose of the file for each Kubernetes repository is to provide a list of people who can assist the Kubernetes in the event that a security issue related to the repository is discovered or disclosed. As described in the file, those on the list should agree to our . Please update the  file for the kube-aggregator repository. After finding people who are willing to work in this capacity, you should add them to the list, then remove PSC members (except any PSC member will be working as a security contact for this repository). The list is GitHub usernames, optionally followed by an email address. If no email address is listed, the PSC will use the email address found on git commits made by the listed user. The file may already have people listed who are secuirty contacts. In that case, simply remove any PSC members who aren't also security contacts for the repo. See for more information /area security /committee product-security /kind cleanup /lifecycle frozen /priority important-soon /sig api-machinery


Question: # DO NOT REPORT SECURITY VULNERABILITIES DIRECTLY TO THESE NAMES, FOLLOW THE # INSTRUCTIONS AT https://kubernetes.io/security/ <del> cjcullen joelsmith liggitt philips </del> <ins> cheftako deads2k lavalamp </ins> sttts <del> tallclair </del>

The purpose of the file for each Kubernetes repository is to provide a list of people who can assist the Kubernetes in the event that a security issue related to the repository is discovered or disclosed. As described in the file, those on the list should agree to our . Please update the  file for the code-generator repository. After finding people who are willing to work in this capacity, you should add them to the list, then remove PSC members (except any PSC member will be working as a security contact for this repository). The list is GitHub usernames, optionally followed by an email address. If no email address is listed, the PSC will use the email address found on git commits made by the listed user. The file may already have people listed who are secuirty contacts. In that case, simply remove any PSC members who aren't also security contacts for the repo. See for more information /area code-generation /area security /committee product-security /kind cleanup /lifecycle frozen /priority important-soon /sig api-machinery


Question: # DO NOT REPORT SECURITY VULNERABILITIES DIRECTLY TO THESE NAMES, FOLLOW THE # INSTRUCTIONS AT https://kubernetes.io/security/ <del> cjcullen joelsmith liggitt philips </del> <ins> cheftako deads2k lavalamp </ins> sttts <del> tallclair </del>

The purpose of the file for each Kubernetes repository is to provide a list of people who can assist the Kubernetes in the event that a security issue related to the repository is discovered or disclosed. As described in the file, those on the list should agree to our . Please update the  file for the client-go repository. After finding people who are willing to work in this capacity, you should add them to the list, then remove PSC members (except any PSC member will be working as a security contact for this repository). The list is GitHub usernames, optionally followed by an email address. If no email address is listed, the PSC will use the email address found on git commits made by the listed user. The file may already have people listed who are secuirty contacts. In that case, simply remove any PSC members who aren't also security contacts for the repo. See for more information /area security /committee product-security /kind cleanup /lifecycle frozen /priority important-soon /sig api-machinery


Question: # DO NOT REPORT SECURITY VULNERABILITIES DIRECTLY TO THESE NAMES, FOLLOW THE # INSTRUCTIONS AT https://kubernetes.io/security/ <del> cjcullen joelsmith liggitt philips </del> <ins> cheftako deads2k lavalamp </ins> sttts <del> tallclair </del>

The purpose of the file for each Kubernetes repository is to provide a list of people who can assist the Kubernetes in the event that a security issue related to the repository is discovered or disclosed. As described in the file, those on the list should agree to our . Please update the  file for the apiserver repository. After finding people who are willing to work in this capacity, you should add them to the list, then remove PSC members (except any PSC member will be working as a security contact for this repository). The list is GitHub usernames, optionally followed by an email address. If no email address is listed, the PSC will use the email address found on git commits made by the listed user. The file may already have people listed who are secuirty contacts. In that case, simply remove any PSC members who aren't also security contacts for the repo. See for more information /area apiserver /area security /committee product-security /kind cleanup /lifecycle frozen /priority important-soon /sig api-machinery


Question: # DO NOT REPORT SECURITY VULNERABILITIES DIRECTLY TO THESE NAMES, FOLLOW THE # INSTRUCTIONS AT https://kubernetes.io/security/ <del> cjcullen joelsmith liggitt philips </del> <ins> cheftako deads2k lavalamp </ins> sttts <del> tallclair </del>

The purpose of the file for each Kubernetes repository is to provide a list of people who can assist the Kubernetes in the event that a security issue related to the repository is discovered or disclosed. As described in the file, those on the list should agree to our . Please update the  file for the apimachinery repository. After finding people who are willing to work in this capacity, you should add them to the list, then remove PSC members (except any PSC member will be working as a security contact for this repository). The list is GitHub usernames, optionally followed by an email address. If no email address is listed, the PSC will use the email address found on git commits made by the listed user. The file may already have people listed who are secuirty contacts. In that case, simply remove any PSC members who aren't also security contacts for the repo. See for more information /area security /committee product-security /kind cleanup /lifecycle frozen /priority important-soon /sig api-machinery


Question: # DO NOT REPORT SECURITY VULNERABILITIES DIRECTLY TO THESE NAMES, FOLLOW THE # INSTRUCTIONS AT https://kubernetes.io/security/ <del> cjcullen joelsmith liggitt philips </del> <ins> cheftako deads2k lavalamp </ins> sttts <del> tallclair </del>

The purpose of the file for each Kubernetes repository is to provide a list of people who can assist the Kubernetes in the event that a security issue related to the repository is discovered or disclosed. As described in the file, those on the list should agree to our . Please update the  file for the apiextensions-apiserver repository. After finding people who are willing to work in this capacity, you should add them to the list, then remove PSC members (except any PSC member will be working as a security contact for this repository). The list is GitHub usernames, optionally followed by an email address. If no email address is listed, the PSC will use the email address found on git commits made by the listed user. The file may already have people listed who are secuirty contacts. In that case, simply remove any PSC members who aren't also security contacts for the repo. See for more information /area security /committee product-security /kind cleanup /lifecycle frozen /priority important-soon /sig api-machinery


Question: } defer CleanupDescriptors(gcmService, projectID) <del> err = CreateAdapter(f.Namespace.Name, adapterDeployment) </del> <ins> err = CreateAdapter(adapterDeployment) </ins> if err != nil { framework.Failf("Failed to set up: %s", err) } <del> defer CleanupAdapter(f.Namespace.Name, adapterDeployment) </del> <ins> defer CleanupAdapter(adapterDeployment) </ins> _, err = kubeClient.RbacV1().ClusterRoleBindings().Create(context.TODO(), HPAPermissions, metav1.CreateOptions{}) if err != nil {

/kind failing-test <!-- If this is a flake, please add: /kind flake --/sig autoscaling Looks like it just started happening today (@ 2020-09-15 03:54:19 +0000 UTC) !
cc the namespace from the provided object "kube-system" does not match the namespace "horizontal-pod-autoscaling-830". You must pass '--namespace=kube-system' to perform this operation. the namespace from the provided object "custom-metrics" does not match the namespace "horizontal-pod-autoscaling-830". You must pass '--namespace=custom-metrics' to perform this operation. the namespace from the provided object "custom-metrics" does not match the namespace "horizontal-pod-autoscaling-830". You must pass '--namespace=custom-metrics' to perform this operation.
related fix:
No I don't think that's the fix for this (that is just a little cleanup). Actually I think we have to delete namespace parameter in line : Because before , the will have no actually effect. But now with the change of , it will have namespace : at the end of the command. But the namespace in yaml file is which is different from . We should not have parameter when applying from file because this test didn't have before, right? Related PR:
/assign


Question: BackendAddressPool: &network.SubResource{ ID: to.StringPtr(lbBackendPoolID), }, <del> LoadDistribution: loadDistribution, FrontendPort:     to.Int32Ptr(port.Port), BackendPort:      to.Int32Ptr(port.Port), EnableFloatingIP: to.BoolPtr(true), </del> <ins> LoadDistribution:    loadDistribution, FrontendPort:        to.Int32Ptr(port.Port), BackendPort:         to.Int32Ptr(port.Port), EnableFloatingIP:    to.BoolPtr(true), DisableOutboundSnat: to.BoolPtr(az.disableLoadBalancerOutboundSNAT()), </ins> }, } if protocol == v1.ProtocolTCP {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: is the recommended way to configure the outbound networks together with Azure standard loadbalancer (SLB). However, for load balancer rules are disabled by default. That means all frontend IPs would be used as the SNAT IPs for virtual machines, according to . This would introduce troubles for various security scenarios, as there's no way to get a stable list of egress IPs. Azure cloud provider should support this scenario, and disable the outbound SNAT when outbound rules are configured by provisioning tools. What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Others: /kind bug /sig azure /priority critical-urgent


Question: assert.Equal(c.groupCalls, 2) } <ins> func TestNewCachedDiscoveryClient_PathPerm(t *testing.T) { assert := assert.New(t) d, err := ioutil.TempDir("", "") assert.NoError(err) os.RemoveAll(d) defer os.RemoveAll(d) c := fakeDiscoveryClient{} cdc := newCachedDiscoveryClient(&c, d, 1*time.Nanosecond) cdc.ServerGroups() err = filepath.Walk(d, func(path string, info os.FileInfo, err error) error { if err != nil { return err } if info.IsDir() { assert.Equal(os.FileMode(0750), info.Mode().Perm()) } else { assert.Equal(os.FileMode(0660), info.Mode().Perm()) } return nil }) assert.NoError(err) } </ins> type fakeDiscoveryClient struct { groupCalls    int resourceCalls int

In kubectl v1.8.0+, schema info is cached in the location specified by (defaulting to ), written with world-writeable permissions (rw-rw-rw-). If is specified and pointed at a different location accessible to other users/groups, the written files may be modified by other users/groups and disrupt the kubectl invocation. CVSS score: CVSS:3.0/AV:L/AC:H/PR:L/UI:R/S:U/C:L/I:L/A:N (3.3, low) What versions are affected? kubectl v1.8.0+ What configurations are affected? Invocations that point at world-writeable locations Impact Malformed responses written to the cache directory can disrupt the kubectl invocation Workaround Use the default location in the $HOME directory or point it at a directory that is only accessible to desired users/groups. (original description follows) ==== What happened: The files inside of ".kube/http-cache" are world writeable (rw-rw-rw-). While the default for these files appears to be the home directory, using the "--cache-dir" flag could put these files into a place where world writeable files would allow any user / process to modify the cache files. Modification of the cache files could influence the kubectl utility in a negative way for other users. What you expected to happen: Apply stricter file permissions to the http-cache files. How to reproduce it (as minimally and precisely as possible): Run any generic kubectl command which is successful and then list the cache directory $ kubectl get pods --all-namespaces $ ls -la Anything else we need to know?


Question: assert.Equal(c.groupCalls, 2) } <ins> func TestNewCachedDiscoveryClient_PathPerm(t *testing.T) { assert := assert.New(t) d, err := ioutil.TempDir("", "") assert.NoError(err) os.RemoveAll(d) defer os.RemoveAll(d) c := fakeDiscoveryClient{} cdc := newCachedDiscoveryClient(&c, d, 1*time.Nanosecond) cdc.ServerGroups() err = filepath.Walk(d, func(path string, info os.FileInfo, err error) error { if err != nil { return err } if info.IsDir() { assert.Equal(os.FileMode(0750), info.Mode().Perm()) } else { assert.Equal(os.FileMode(0660), info.Mode().Perm()) } return nil }) assert.NoError(err) } </ins> type fakeDiscoveryClient struct { groupCalls    int resourceCalls int

: I estimate this is a low severity security issue with a CVSS score of "3.3 / CVSS:3.0/AV:L/AC:L/PR:L/UI:N/S:U/C:N/I:L/A:N" - Environment: Linux Kubernetes version (use kubectl version):Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.6", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-02-26T12:49:28Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.6", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-02-26T12:49:28Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: AWS. Running kube api server in hyperkube. OS (e.g: cat /etc/os-release): NAME="CentOS Linux" VERSION="7.1808 (Core)" ID="centos" IDLIKE="rhel fedora" VERSIONID="7" PRETTYNAME="CentOS Linux 7.1808 (Core)" ANSICOLOR="0;31" CPENAME="cpe:/o:centos:centos:7" HOMEURL="" BUGREPORTURL="" CENTOSMANTISBTPROJECT="CentOS-7" CENTOSMANTISBTPROJECTVERSION="7" REDHATSUPPORTPRODUCT="centos" REDHATSUPPORTPRODUCTVERSION="7" OSTREEVERSION=7.1808 Kernel (e.g. uname -a): Linux hackit.internal 3.10.0-862.11.6.el7.x8664 SMP Tue Aug 14 21:49:04 UTC 2018 x8664 x8664 x86_64 GNU/Linux Install tools: Manual installation. Others: n/a
/sig product-security
Closing this issue. <detailsIn response to : if you think this is serious, send an email thanks!


Question: assert.Equal(c.groupCalls, 2) } <ins> func TestNewCachedDiscoveryClient_PathPerm(t *testing.T) { assert := assert.New(t) d, err := ioutil.TempDir("", "") assert.NoError(err) os.RemoveAll(d) defer os.RemoveAll(d) c := fakeDiscoveryClient{} cdc := newCachedDiscoveryClient(&c, d, 1*time.Nanosecond) cdc.ServerGroups() err = filepath.Walk(d, func(path string, info os.FileInfo, err error) error { if err != nil { return err } if info.IsDir() { assert.Equal(os.FileMode(0750), info.Mode().Perm()) } else { assert.Equal(os.FileMode(0660), info.Mode().Perm()) } return nil }) assert.NoError(err) } </ins> type fakeDiscoveryClient struct { groupCalls    int resourceCalls int

</details
Adjusted the CVSS score to to indicate user interaction is required (since they must specify a non-default cache option pointed at a shared writeable location)
There are two places this should be done: PathPerm and FilePerm should be set here: the temp file and final file should have stricter permissions (probably 0660) here:
/cc
/help
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/assign I would like to work on release-1.9, as I just made a hot fix for our team's k8s version of 1.9.
GitHub didn't allow me to assign the following users: yuchengwu. Note that only and repo collaborators can be assigned and that issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign I would like to work on release-1.9, as I just made a hot fix for our team's k8s version of 1.9. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
help fixing this is welcome, though it would only be fixed in releases 1.12+, per
/label official-cve-feed (Related to kubernetes/sig-security)


Question: // Wraps syscall.Mount() func (mounter *DiskMounter) Mount(source string, target string, fstype string, flags uintptr, data string) error { <ins> glog.V(5).Infof("Mounting %s %s %s %d %s", source, target, fstype, flags, data) </ins> return syscall.Mount(source, target, fstype, flags, data) }

07-02 and 13-02 No response No response /sig
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scalability
I might be related to the npd.
The same test failures happening again on 25.02:
/sig node
/sig node /cc The restart of caused the flake. I also opened to track it in NPD repo.
I missed this log. It is caused by oom. Adding some memory may help. I submitted to fix it. Does mean killing metrics-server?
/remove-sig node Sorry, I misread the log. The NPD found the metric server is OOM. It is about the metrics server, not the NPD. Sorry for disturb. I may add some memory for metric-server and check why metric server memory changed. I heard that metrics server recently released a new version. I will check if it is related.
/sig node
OOM metrics shows the metric server is OOMed. This is odd, that only metrics-server-nanny is with a memory limit. It uses whose command is . metrics-server container has no memory limit, but it got killed by OOM. Does this indicate that the node is OOM? (edited)
The last run failed in node-problem-detector ContainerStateTerminated. I0227 05:03:31. ] PodStartupLatency: perc50: 14m29.018886827s, perc90: 28m44.625518807s, perc99: 32m2.19303496s; threshold 1h0m0s I0227 05:03:32. ]


Question: // Wraps syscall.Mount() func (mounter *DiskMounter) Mount(source string, target string, fstype string, flags uintptr, data string) error { <ins> glog.V(5).Infof("Mounting %s %s %s %d %s", source, target, fstype, flags, data) </ins> return syscall.Mount(source, target, fstype, flags, data) }

Loaded restart count threshold overrides: map[coredns:2 fluentd-gcp:999 konnectivity-agent:2 kube-controller-manager:1 kube-proxy:2 kube-scheduler:1 l7-lb-controller:1 metadata-proxy:2 prometheus-to-sd-exporter:2] I0227 05:03:32. ] ClusterOOMsTracker: gathering cluster OOMs tracking measurement E0227 05:03:32. ] TestMetrics: [action gather failed for SystemPodMetrics measurement: restart counts violation: RestartCount(node-problem-detector-tgl8h, node-problem-detector)=1, want <= 0; RestartCount(node-problem-detector-vpsmz, node-problem-detector)=1, want <= 0] I0227 05:03:32. ] Step "[step: 29] gathering measurements" ended W0227 05:03:32. ] Got errors during step execution: [measurement call TestMetrics - TestMetrics error: [action gather failed for SystemPodMetrics measurement: restart counts violation: RestartCount(node-problem-detector-tgl8h, node-problem-detector)=1, want <= 0; RestartCount(node-problem-detector-vpsmz, node-problem-detector)=1, want <= 0]] I0227 05:03:32. ] Waiting for the chaos monkey subroutine to end... I0227 05:03:32. ] Chaos monkey ended. I0227 05:03:32. ] I0227 05:03:32. ] Stopping DnsLookupLatency probe... I0227 05:03:47. ] Stopping InClusterNetworkLatency probe... I0227 05:04:22. ] Resources cleanup time: 50.539700215s E0227 05:04:22. ]
E0227 05:04:22. ] Test Finished E0227 05:04:22. ] Test: E0227 05:04:22. ] Status: Fail E0227 05:04:22. ]


Question: // Wraps syscall.Mount() func (mounter *DiskMounter) Mount(source string, target string, fstype string, flags uintptr, data string) error { <ins> glog.V(5).Infof("Mounting %s %s %s %d %s", source, target, fstype, flags, data) </ins> return syscall.Mount(source, target, fstype, flags, data) }

Errors: [measurement call TestMetrics - TestMetrics error: [action gather failed for SystemPodMetrics measurement: restart counts violation: RestartCount(node-problem-detector-tgl8h, node-problem-detector)=1, want <= 0; RestartCount(node-problem-detector-vpsmz, node-problem-detector)=1, want <= 0]] E0227 05:04:22. ]
I0227 05:04:22. ]
I0227 05:04:22. ] Running testing/huge- I0227 05:04:22. ]


Question: ### END TEST DEFINITION CUSTOMIZATION ### ####################################################### # Step 1: Start a server which supports both the old and new api versions, # but KUBE_OLD_API_VERSION is the latest (storage) version.

removed this test. We need to continue testing this. This wasn't about testing upgrades, the prefix has nothing to do with upgrades. It's just there so people can share etcd. We maybe don't need to double everything but we do need to make sure that keeps working.
To be honest - I think that what we was testing before didn't give us completely any information. We didn't testing what information we are storing in etcd and where. That said, if we had a hardcoded prefix somewhere in our code, we wouldn't discover it at all, and those tests would still be passing. We should have a test that is directly checking etcd contents. I think it should be possible to write a simple unit test for it. Will try to put something together next week.


Question: "image_id_test.go", "kubelet_test.go", "lifecycle_hook_test.go", <ins> "log_path_test.go", </ins> "memory_eviction_test.go", "mirror_pod_test.go", "resource_usage_test.go",

Cluster logging assumes kubelet stores container log at a specific directory with specific name. Currently the path is: After the CRI work, the path will be: However, we don't have test coverage for this. We don't even know whether the log files are there or not. We should add a node e2e test for this, basically: 1) Create a container which prints logs 2) Check whether the expected log file is created. 3) Check whether the logs are collected in the file as expected. And it would be better to launch a pod to do the check instead of accessing local file system directly, which makes containerizing easier. /cc because you are collecting missing test. /cc
I will try to add a node-e2e test there
I think it's ok to add this for CRI. But there maybe problems on today's interface for rkt. If I'm not wrong, rkt does't store logs at same location with docker. cc/ Is this true?
Thanks! :)
After gets merged, we need to update the test to verify the cri log path.


Question: assert.Equal(t, result, test.expectSocket, "Unexpected result from IsUnixDomainSocket: %v for %s", result, test.label) } } <ins> func TestGetAddressAndDialer(t *testing.T) { tests := []struct { endpoint     string expectError  bool expectedAddr string }{ { endpoint:     "unix:///tmp/s1.sock", expectError:  false, expectedAddr: "/tmp/s1.sock", }, { endpoint:     "unix:///tmp/f6.sock", expectError:  false, expectedAddr: "/tmp/f6.sock", }, { endpoint:    "tcp://localhost:9090", expectError: true, }, { // The misspelling is intentional to make it error endpoint:    "htta://free-test.com", expectError: true, }, { endpoint:    "https://www.youtube.com/", expectError: true, }, { endpoint:    "http://www.baidu.com/", expectError: true, }, } for _, test := range tests { // just test addr and err addr, _, err := GetAddressAndDialer(test.endpoint) if test.expectError { assert.NotNil(t, err, "expected error during parsing %s", test.endpoint) continue } assert.Nil(t, err, "expected no error during parsing %s", test.endpoint) assert.Equal(t, test.expectedAddr, addr) } } </ins>

What would you like to be : Currently, does not have unit test coverage. Unless there's a strong reason not to, we'd like public facing functions to have unit test coverage. Why is this needed: There's a fairly small chance of GetAddressAndDialer being modified, and thus breaking, in the future without this test coverage. However, it should be relatively straightforward to add the test coverage, and could make a good first issue for a new contributor. There's also the benefit that it helps keep the test coverage in pkg/kubelet high :) And we never know, it might help catch a bug some day :) Extra note I think this ticket could be a great candidate for someone who is new to contributing to sig-node/k8s. I'd prefer to please give first dibs for picking this up to any folks who are currently underrepresented in the k8s community. Please reach out to on k8s Slack (or post directly in #sig-node and tag me) if you're interested, and I'll assign you the issue. I'm happy to mentor and be helpful in whatever way I can :)
/sig node
/good-first-issue
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
Hi let's me solve it
Sounds like a plan! Can you please tag and myself for the review?
Hi, I've already created a pr which , please help me complete the review, thanks


Question: readonly KUBE_NODE_BINARIES=("${KUBE_NODE_TARGETS[@]##*/}") readonly KUBE_NODE_BINARIES_WIN=("${KUBE_NODE_BINARIES[@]/%/.exe}") <del> if [[ "${KUBE_FASTBUILD:-}" == "true" ]]; then </del> <ins> if [[ -n "${KUBE_BUILD_PLATFORMS:-}" ]]; then readonly KUBE_SERVER_PLATFORMS=(${KUBE_BUILD_PLATFORMS}) readonly KUBE_NODE_PLATFORMS=(${KUBE_BUILD_PLATFORMS}) readonly KUBE_TEST_PLATFORMS=(${KUBE_BUILD_PLATFORMS}) readonly KUBE_CLIENT_PLATFORMS=(${KUBE_BUILD_PLATFORMS}) elif [[ "${KUBE_FASTBUILD:-}" == "true" ]]; then </ins> readonly KUBE_SERVER_PLATFORMS=(linux/amd64) readonly KUBE_NODE_PLATFORMS=(linux/amd64) if [[ "${KUBE_BUILDER_OS:-}" == "darwin"* ]]; then

hack/build-cross sets it without looking at the current value. And the value has to be an array, which is a bit awkward to use.
This could be improved indeed. Maybe something like and Haven't tested it though... But why would one want to change the value?
I just wanted to build one, so I did the "obvious" thing: hack/dev-build-and-` It failed for multiple reasons :) , Lucas Käldström wrote:
This issue has been a major downer for me. It's much harder right now than it ought to be to articulate the few platforms for which you'd like to build each of server binaries, client binaries, test binaries, etc. Every time I've tried to do this "the right way," I've spent hours looking at scripts and feeling frustrated that build options aren't documented better. Every time, I end up cutting my losses and resorting to temporary build script hacks instead, just to get the job done.
Doesn't fix this issue exactly, but I've found that you can easily build only if you set .
Taking a look at this today, I also thought that setting seems to be the most reasonable.
/sig release