Question: framework.ExpectNoError(verifyServeHostnameServiceUp(cs, ns, host, podNames2, svc2IP, servicePort)) }) <del> ginkgo.It("should be able to preserve UDP traffic when server pod cycles for a NodePort service", func() { serviceName := "clusterip-test" serverPod1Name := "server-1" serverPod2Name := "server-2" ns := f.Namespace.Name nodeIP, err := e2enode.PickIP(cs) // for later framework.ExpectNoError(err) // Create a NodePort service udpJig := e2eservice.NewTestJig(cs, ns, serviceName) ginkgo.By("creating a UDP service " + serviceName + " with type=NodePort in " + ns) udpService, err := udpJig.CreateUDPService(func(svc *v1.Service) { svc.Spec.Type = v1.ServiceTypeNodePort svc.Spec.Ports = []v1.ServicePort{ {Port: 80, Name: "http", Protocol: v1.ProtocolUDP, TargetPort: intstr.FromInt(80)}, } }) framework.ExpectNoError(err) // Add a backend pod to the service ginkgo.By("creating a backend pod for the service " + serviceName) serverPod1 := newAgnhostPod(serverPod1Name, "netexec", fmt.Sprintf("--udp-port=%d", 80)) serverPod1.Labels = udpJig.Labels _, err = cs.CoreV1().Pods(ns).Create(context.TODO(), serverPod1, metav1.CreateOptions{}) ginkgo.By(fmt.Sprintf("checking NodePort service %s on node with public IP %s", serviceName, nodeIP)) framework.ExpectNoError(err) framework.ExpectNoError(e2epod.WaitTimeoutForPodReadyInNamespace(f.ClientSet, serverPod1.Name, f.Namespace.Name, framework.PodStartTimeout)) // Waiting for service to expose endpoint. err = validateEndpointsPorts(cs, ns, serviceName, portsByPodName{serverPod1Name: {80}}) framework.ExpectNoError(err, "failed to validate endpoints for service %s in namespace: %s", serviceName, ns) // Check that the pod reveives the traffic ginkgo.By("Sending UDP traffic to NodePort service " + serviceName + " on node with publicIP " + nodeIP) errorChannel := make(chan error) signal := make(chan struct{}, 1) go continuousEcho(nodeIP, int(udpService.Spec.Ports[0].NodePort), 3*time.Second, 20, signal, errorChannel) // Create a second pod ginkgo.By("creating a second pod for the service " + serviceName) serverPod2 := newAgnhostPod(serverPod2Name, "netexec", fmt.Sprintf("--udp-port=%d", 80)) serverPod2.Labels = udpJig.Labels _, err = cs.CoreV1().Pods(ns).Create(context.TODO(), serverPod2, metav1.CreateOptions{}) framework.ExpectNoError(err) framework.ExpectNoError(e2epod.WaitTimeoutForPodReadyInNamespace(f.ClientSet, serverPod2.Name, f.Namespace.Name, framework.PodStartTimeout)) // and delete the first pod framework.Logf("Cleaning up %s pod", serverPod1Name) err = cs.CoreV1().Pods(ns).Delete(context.TODO(), serverPod1Name, metav1.DeleteOptions{}) framework.ExpectNoError(err, "failed to delete pod: %s on node", serverPod1Name) // Check that the second pod keeps receiving traffic ginkgo.By("Sending UDP traffic to NodePort service " + serviceName + " on node with publicIP " + nodeIP) signal <- struct{}{} // Check that there are no errors err = <-errorChannel framework.ExpectNoError(err, "pod communication failed") }) </del> /* Release : v1.16 Testname: Service, NodePort Service

so we can do things like to reproduce this sort of thing. matt did you get around to the 'try-till-it-fails' scenario?
I haven't been able to get it to fail on kind yet, -- 100 cycles and 0 failures on a three-node kind cluster! I'm letting the loop run for a few hours, will check back in when I have that data. Is it worth running this test on a non-kind environment?
I ran for several hours on my mac, through 1144 iterations no error yet. My conclusion: this test isn't flaky on KinD. Possible next step: run a large number of iterations of the same test on a GKE cluster.
Flakes on kind are also happening:
not many since the portmap fix on the 14th though ( / ) <img width="1111" alt="Screen Shot 2020-07-23 at 11 45 46 PM" src="https://user-"
Agree - it's much better than it was before.
I was analyzing this failure, the problem is that kind jobs does not have verbosity enough in kube-proxy We can see that the packet was never replied, this is the NAT entry from client -nodePort in the node with the client This is the same flow, but in the node with the server and the pod-server-1 seems to be healthy we need more logs :eyeglasses:
We have an open ticket to bump those in CI, it's just been lost in the noise. We should still do that. Antonio Ojea <:
If you are busy just assign it to me, I'll need this verbosity soon :)
It's in the kind repo, shouldn't be too hard to find, we have ~60 open atm. Antonio Ojea <:
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
I made another attempt to reproduce these failures locally on KinD, but still no success. Here's what I did: create a KinD cluster run two instances of sonobuoy at the same time This took ~4 hours to run; all the kube tests passed and sonobuoy reported all successes.


Question: framework.ExpectNoError(verifyServeHostnameServiceUp(cs, ns, host, podNames2, svc2IP, servicePort)) }) <del> ginkgo.It("should be able to preserve UDP traffic when server pod cycles for a NodePort service", func() { serviceName := "clusterip-test" serverPod1Name := "server-1" serverPod2Name := "server-2" ns := f.Namespace.Name nodeIP, err := e2enode.PickIP(cs) // for later framework.ExpectNoError(err) // Create a NodePort service udpJig := e2eservice.NewTestJig(cs, ns, serviceName) ginkgo.By("creating a UDP service " + serviceName + " with type=NodePort in " + ns) udpService, err := udpJig.CreateUDPService(func(svc *v1.Service) { svc.Spec.Type = v1.ServiceTypeNodePort svc.Spec.Ports = []v1.ServicePort{ {Port: 80, Name: "http", Protocol: v1.ProtocolUDP, TargetPort: intstr.FromInt(80)}, } }) framework.ExpectNoError(err) // Add a backend pod to the service ginkgo.By("creating a backend pod for the service " + serviceName) serverPod1 := newAgnhostPod(serverPod1Name, "netexec", fmt.Sprintf("--udp-port=%d", 80)) serverPod1.Labels = udpJig.Labels _, err = cs.CoreV1().Pods(ns).Create(context.TODO(), serverPod1, metav1.CreateOptions{}) ginkgo.By(fmt.Sprintf("checking NodePort service %s on node with public IP %s", serviceName, nodeIP)) framework.ExpectNoError(err) framework.ExpectNoError(e2epod.WaitTimeoutForPodReadyInNamespace(f.ClientSet, serverPod1.Name, f.Namespace.Name, framework.PodStartTimeout)) // Waiting for service to expose endpoint. err = validateEndpointsPorts(cs, ns, serviceName, portsByPodName{serverPod1Name: {80}}) framework.ExpectNoError(err, "failed to validate endpoints for service %s in namespace: %s", serviceName, ns) // Check that the pod reveives the traffic ginkgo.By("Sending UDP traffic to NodePort service " + serviceName + " on node with publicIP " + nodeIP) errorChannel := make(chan error) signal := make(chan struct{}, 1) go continuousEcho(nodeIP, int(udpService.Spec.Ports[0].NodePort), 3*time.Second, 20, signal, errorChannel) // Create a second pod ginkgo.By("creating a second pod for the service " + serviceName) serverPod2 := newAgnhostPod(serverPod2Name, "netexec", fmt.Sprintf("--udp-port=%d", 80)) serverPod2.Labels = udpJig.Labels _, err = cs.CoreV1().Pods(ns).Create(context.TODO(), serverPod2, metav1.CreateOptions{}) framework.ExpectNoError(err) framework.ExpectNoError(e2epod.WaitTimeoutForPodReadyInNamespace(f.ClientSet, serverPod2.Name, f.Namespace.Name, framework.PodStartTimeout)) // and delete the first pod framework.Logf("Cleaning up %s pod", serverPod1Name) err = cs.CoreV1().Pods(ns).Delete(context.TODO(), serverPod1Name, metav1.DeleteOptions{}) framework.ExpectNoError(err, "failed to delete pod: %s on node", serverPod1Name) // Check that the second pod keeps receiving traffic ginkgo.By("Sending UDP traffic to NodePort service " + serviceName + " on node with publicIP " + nodeIP) signal <- struct{}{} // Check that there are no errors err = <-errorChannel framework.ExpectNoError(err, "pod communication failed") }) </del> /* Release : v1.16 Testname: Service, NodePort Service

Would love to help out in reproing this, I'm at a loss for good next step so any suggestions welcome!
that's the problem with these flakes, they are not easy to repro, and with this low rate of failures it may be caused for combination of running tests, environmental problems (cpu, memory, ...) , ... I always start for checking the current failures and try to find patterns
Let's take a look to one occurrence The tests fails because can't connect to the first pod If we check the code of the tests, we can see that it always uses the port as source port, we can use that to identify the nodePort assigned interesting thing here, we are source nating the pod ip, and we don't see any other conntrack entry The test verifies it has endpoints however, kube-proxy is using endpointlsices since this version but it has no entries so the iptables rules to forward the traffic are not there Seems we have to adapt this function in the e2e framework to use endpointslices cc: EDIT seems that this function is being used by all the network tests that can cause flakes if the endpoint is updated before the endpointslice (we switched recently kube-proxy to use the later)
Great catch. Ideally this function should validate if both Endpoint and appropriate EndpointSlice objects(s) exist.
Actually - let me take it and send a PR for it.
is out for review
Interesting failures here It fails to connect to the first pod
Hmm - I took a quick look at kube-proxy logs and these look fine to me....
I clicked too fast, sorry ... I was looking at the wrong service and I though I've found something
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.


Question: framework.ExpectNoError(verifyServeHostnameServiceUp(cs, ns, host, podNames2, svc2IP, servicePort)) }) <del> ginkgo.It("should be able to preserve UDP traffic when server pod cycles for a NodePort service", func() { serviceName := "clusterip-test" serverPod1Name := "server-1" serverPod2Name := "server-2" ns := f.Namespace.Name nodeIP, err := e2enode.PickIP(cs) // for later framework.ExpectNoError(err) // Create a NodePort service udpJig := e2eservice.NewTestJig(cs, ns, serviceName) ginkgo.By("creating a UDP service " + serviceName + " with type=NodePort in " + ns) udpService, err := udpJig.CreateUDPService(func(svc *v1.Service) { svc.Spec.Type = v1.ServiceTypeNodePort svc.Spec.Ports = []v1.ServicePort{ {Port: 80, Name: "http", Protocol: v1.ProtocolUDP, TargetPort: intstr.FromInt(80)}, } }) framework.ExpectNoError(err) // Add a backend pod to the service ginkgo.By("creating a backend pod for the service " + serviceName) serverPod1 := newAgnhostPod(serverPod1Name, "netexec", fmt.Sprintf("--udp-port=%d", 80)) serverPod1.Labels = udpJig.Labels _, err = cs.CoreV1().Pods(ns).Create(context.TODO(), serverPod1, metav1.CreateOptions{}) ginkgo.By(fmt.Sprintf("checking NodePort service %s on node with public IP %s", serviceName, nodeIP)) framework.ExpectNoError(err) framework.ExpectNoError(e2epod.WaitTimeoutForPodReadyInNamespace(f.ClientSet, serverPod1.Name, f.Namespace.Name, framework.PodStartTimeout)) // Waiting for service to expose endpoint. err = validateEndpointsPorts(cs, ns, serviceName, portsByPodName{serverPod1Name: {80}}) framework.ExpectNoError(err, "failed to validate endpoints for service %s in namespace: %s", serviceName, ns) // Check that the pod reveives the traffic ginkgo.By("Sending UDP traffic to NodePort service " + serviceName + " on node with publicIP " + nodeIP) errorChannel := make(chan error) signal := make(chan struct{}, 1) go continuousEcho(nodeIP, int(udpService.Spec.Ports[0].NodePort), 3*time.Second, 20, signal, errorChannel) // Create a second pod ginkgo.By("creating a second pod for the service " + serviceName) serverPod2 := newAgnhostPod(serverPod2Name, "netexec", fmt.Sprintf("--udp-port=%d", 80)) serverPod2.Labels = udpJig.Labels _, err = cs.CoreV1().Pods(ns).Create(context.TODO(), serverPod2, metav1.CreateOptions{}) framework.ExpectNoError(err) framework.ExpectNoError(e2epod.WaitTimeoutForPodReadyInNamespace(f.ClientSet, serverPod2.Name, f.Namespace.Name, framework.PodStartTimeout)) // and delete the first pod framework.Logf("Cleaning up %s pod", serverPod1Name) err = cs.CoreV1().Pods(ns).Delete(context.TODO(), serverPod1Name, metav1.DeleteOptions{}) framework.ExpectNoError(err, "failed to delete pod: %s on node", serverPod1Name) // Check that the second pod keeps receiving traffic ginkgo.By("Sending UDP traffic to NodePort service " + serviceName + " on node with publicIP " + nodeIP) signal <- struct{}{} // Check that there are no errors err = <-errorChannel framework.ExpectNoError(err, "pod communication failed") }) </del> /* Release : v1.16 Testname: Service, NodePort Service

just saw it as well
can you please add links to the job with the failures, I just merged new code to improve the logging 2 days ago
ok, here there is a new occurrence and with the new debug logs is easier to follow has unreplied conntrack entries: looking at the other side The iptables rules are there but no conntrack entries related to 10.244.2.32
In I revert PR with debug info as I think that it likely caused significant memory growth in scalability tests, but we have one run that has finished without being killed by OOMKiller (I increased memory limit for that jobs as a mitigation). Maybe logs from this run will be useful for debugging:
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
Two more flakes today:
It seems that the packet never gets DNATed in the receiving host, so it does not reach the pod in the node. There is the same test that is using ClusterIP instead of NodePort that is more reliable. This is being a tough one
/triage accepted
Yesterday's flake - I don't think I'm seeing the log messages from the latest log PR you made ? Or are the prow logs not logging at a high-enough level?
Logs are there, please bear with me: The error on the test is: click on the job failure and you'll see a larger output where you can see the server pod name and get the node where the pod is running: and the client-pod name and its node: and the flow we are looking for: with the node-name we can get the kube-proxy logs there, clicking in artifacts: and you can see the "new" conntrack log entries : and in the client: wow, there is a kernel bug there , can it be related? :thinking:
wohoo - some progress :) I'm not an expert here, but I think it can be? Adding here - they may be able to confirm/reject the hypothesis.
This kernel panic may be a coincidence, maybe this is the reason
Adding some recent failures:
Spotted another flake today:
Another flake in the 5k job:


Question: import ( "fmt" <ins> "strings" </ins> "testing" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"

What happened: No endpoints created in the EndpointSlice for my service due to the port on the service being too long. Logs show: EndpointSlice created has no endpoint: What you expected to happen: Service that works with Endpoints should work with EndpointSlice. If this is intentional, it should be clearly documented How to reproduce it (as minimally and precisely as possible): Create a service with a long port name Environment: Kubernetes version (use ): 1.16 Cloud provider or hardware configuration: kind
/sig networking
The label(s) cannot be applied. These labels are supported: <detailsIn response to : /sig networking Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig network These SIGs are my best guesses for this issue. Please comment if I am incorrect about one. I am a bot run by vllry.
For a rough measure of impact, Istio had 5 ports that were over this limit (and many more that were right at 15 characters), so I suspect it may not be terribly rare.
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
This needs to be resolved before endpointslices graduate from alpha
/assign
/remove-triage unresolved


Question: framework.ExpectNoError(err, "found a pod(s)") }) <del> ginkgo.It("should run through the lifecycle of Pods and PodStatus", func() { </del> <ins> /* Release: v1.20 Testname: Pods, completes the lifecycle of a Pod and the PodStatus Description: A Pod is created with a static label which MUST succeed. It MUST succeed when patching the label and the pod data. When checking and replacing the PodStatus it MUST succeed. It MUST succeed when deleting the Pod. */ framework.ConformanceIt("should run through the lifecycle of Pods and PodStatus", func() { </ins> podResource := schema.GroupVersionResource{Group: "", Version: "v1", Resource: "pods"} testNamespaceName := f.Namespace.Name testPodName := "pod-test"

[x] Original approval issue: - Reverted due to flakes [x] Two weeks soak end date: 20 November 2020 This issue is created to allow edit by /wip /hold According to this APIsnoop query, there are still some remaining Pod endpoints which are untested. a Pod with a static label the Pod with a new Label and updated data the Pod to ensure it's patched the Pod's status the Pod's status Ready condition to False the Pod's status to check if Ready is False all Pods in all Namespaces find the Pod(1) ensure that the Pod is found and is patched Namespaced Pod(1) via a Collection with a LabelSelector the Pod to check that it's deleted Discover useragents: List endpoints hit by the test: Display endpoint coverage change: If a test with these calls gets merged, test coverage will go up by 4 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/reopen Test PR Merge Promotion PR to be created next
Reopened this issue. <detailsIn response to : /reopen Test PR Merge Promotion PR to be created next Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
reverted by due to flaky test
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
/reopen PR on test grid for 2 weeks before promotion to conformance
Reopened this issue. <detailsIn response to : /reopen PR on test grid for 2 weeks before promotion to conformance Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/triage accepted


Question: removed := map[int]bool{} for i := 0; i < 5; i++ { if i != 0 { <del> klog.V(3).Infof("Attempt %v failed to kill all unwanted process. Retyring", i) </del> <ins> klog.V(3).Infof("Attempt %v failed to kill all unwanted process from cgroup: %v. Retyring", i, podCgroup) </ins> } errlist = []error{} for _, pid := range pidsToKill { if _, ok := removed[pid]; ok { continue } <del> klog.V(3).Infof("Attempt to kill process with pid: %v", pid) </del> <ins> klog.V(3).Infof("Attempt to kill process with pid: %v from cgroup: %v", pid, podCgroup) </ins> if err := m.killOnePid(pid); err != nil { <del> klog.V(3).Infof("failed to kill process with pid: %v", pid) </del> <ins> klog.V(3).Infof("failed to kill process with pid: %v from cgroup: %v", pid, podCgroup) </ins> errlist = append(errlist, err) } else { removed[pid] = true } } if len(errlist) == 0 { <del> klog.V(3).Infof("successfully killed all unwanted processes.") </del> <ins> klog.V(3).Infof("successfully killed all unwanted processes from cgroup: %v", podCgroup) </ins> return nil } }

What happened: When debugging timeouts related to the stopping of cgroups, the current logs from -tryKillingCgroupProcesses do not provide any correlation between the PIDs which were killed and the cgroup they belonged to. What you expected to happen: Log the cgroup name along with the PID How to reproduce it (as minimally and precisely as possible): N/A Anything else we need to know?: N/A Environment: Kubernetes version (use ): master
/sig node


Question: return f.Err } <del> // DeleteTCPLoadBalancer is a test-spy implementation of TCPLoadBalancer.DeleteTCPLoadBalancer. </del> <ins> // EnsureTCPLoadBalancerDeleted is a test-spy implementation of TCPLoadBalancer.EnsureTCPLoadBalancerDeleted. </ins> // It adds an entry "delete" into the internal method call record. <del> func (f *FakeCloud) DeleteTCPLoadBalancer(name, region string) error { </del> <ins> func (f *FakeCloud) EnsureTCPLoadBalancerDeleted(name, region string) error { </ins> f.addCall("delete") return f.Err }

All of our external load balancer tests failed for the better part of a day due to the GCE project that our tests run in hitting its quota for number of target pools. While the target pools have been deleted, the fact that we hit our quota indicates that there must be a leak somewhere, and we need to plug it.
For reference, the previous issue was
Could the leakage be related to ? The quota exhaustion occurred about 22 hrs after that.
I'd guess that this is due to GetTCPLoadBalancer returning that the load balancer doesn't exist if the forwarding rule is gone, ignoring the target pool, while the service controller considers its deleting done if the cloud provider says that the load balancer doesn't exist. I that logic to the cloud provider because the other implementations of DeleteTCPLoadBalancer weren't/won't be idempotent in the sense of returning ok if the LB doesn't exist. To fix it, we'll either need to make sure all Delete implementations are idempotent and remove the Get call in the controller, or add a method along the lines of IsTCPLoadBalancerDeleted to ask the cloud provider whether all pieces of it have been completely cleaned up. We can't make GetTCPLoadBalancer check that because it's relied on to tell us whether or not the balancer is fully created.
Copying over my commentary from on why the GKE e2e tests are leaking so many forwarding rules and target pools vs the GCE tests:
Is there a way for the services test to wait for the service controller to clean up the external resources before finishing (without just doing a sleep)?
Not without polling GCE directly for whether the resources still exist, although that may be worth it as a workaround. What would be perfect is that clean-up-all-cloud-provider-resources API method that we talked about a couple months ago.
Can we not perhaps use GCE namespaces or projects as a cheap way of getting this here?
I don't believe so. We can't use projects due to the fact that projects can't be programatically created (they're intended to be long-lived). I'm not aware of anything along the lines of a GCE namespace other than networks, which won't work because target pools and forwarding rules unfortunately don't seem to be scoped to networks.


Question: return f.Err } <del> // DeleteTCPLoadBalancer is a test-spy implementation of TCPLoadBalancer.DeleteTCPLoadBalancer. </del> <ins> // EnsureTCPLoadBalancerDeleted is a test-spy implementation of TCPLoadBalancer.EnsureTCPLoadBalancerDeleted. </ins> // It adds an entry "delete" into the internal method call record. <del> func (f *FakeCloud) DeleteTCPLoadBalancer(name, region string) error { </del> <ins> func (f *FakeCloud) EnsureTCPLoadBalancerDeleted(name, region string) error { </ins> f.addCall("delete") return f.Err }

The target pool leak in the code might be fixed, but the forwarding rule + target pool leak from deleting GKE clusters too quickly is certainly still happening. Reopening for the sake of that half of the issue.
Can we just add a short sleep to jenkins after the last test has run before it deletes anything?
Yes, you could add a sleep to Ginkgo's AfterEachSuite(): But I'd be concerned about what the upper bound on the sleep period is. Can we get the tests and code in sync with a 1 minute latency here? (e.g. if the controller garbage collects every 60 seconds, then the test only needs to sleep for 60 sec?) And add a BEEG comment to the test's sleep config noting that increasing the sleep time will not improve things, otherwise I'm pretty sure that someone noting flakes in this test in future will think "maybe I should just increase the sleep period a little bit."
Actually, thinking about this a little more, it would be better to put the "sleep" logic in the k8s master teardown procedure (i.e. not in the e2e tests or in Jenkins), because otherwise users are going to run into the same resource leakage problems as we are. So master deletion blocks on cleanup of backend cloud resources.
There's no way to block master deletion from the master itself. The test framework (or user) just pulls the rug out from underneath the master. See
Robby's right -- is the real solution to this, but until we have that the best we can do is to have the wait happen elsewhere. The options seem to be: GKE itself Ginkgo's post-testing logic Jenkins' teardown logic I don't like putting it in GKE, because arbitrarily slowing down our API makes for a poor experience for the majority of people that don't still have resources being deleted. I'm not eager to put it in Gingko either, since won't that sleep also take effect whenever anyone runs an e2e test locally against a dev cluster? Putting it in the Jenkins logic makes the most sense to me. I'll send a PR and cc you.
We may also want to change the kube-down logic to bring down the master and then nodes in parallel to both speed it up and more closely resemble GKE's teardown logic.
I've deleted the old forwarding rules and target pools from the project.


Question: return f.Err } <del> // DeleteTCPLoadBalancer is a test-spy implementation of TCPLoadBalancer.DeleteTCPLoadBalancer. </del> <ins> // EnsureTCPLoadBalancerDeleted is a test-spy implementation of TCPLoadBalancer.EnsureTCPLoadBalancerDeleted. </ins> // It adds an entry "delete" into the internal method call record. <del> func (f *FakeCloud) DeleteTCPLoadBalancer(name, region string) error { </del> <ins> func (f *FakeCloud) EnsureTCPLoadBalancerDeleted(name, region string) error { </ins> f.addCall("delete") return f.Err }

I'll periodically check to see whether anything is leaked after this point. I apologize if the deletions affected any running tests - everything looked to only be from GKE and parallel e2e runs.


Question: kubelet.masterServiceNamespace = metav1.NamespaceDefault kubelet.serviceLister = testServiceLister{} kubelet.serviceHasSynced = func() bool { return true } <ins> kubelet.nodeHasSynced = func() bool { return true } </ins> kubelet.nodeLister = testNodeLister{ nodes: []*v1.Node{ {

What happened: A deployment is created that uses a after the cluster master is created but before the cluster worker nodes are fully initialized. During the initialization of the worker nodes, the nodes are temporarily available for scheduling without the necessary label to match the deployment's node selector. Depending on how long the nodes are available for scheduling without the necessary node labels for the deployment, the scheduler will start to spam with cluster with pods in status. This spamming stops once the worker nodes are fully initialized and the pods are scheduled successful. The exact timing of all of this is TBD. What you expected to happen: Pods in status are annoying and require manual cleanup. It would be preferred that the scheduler avoid this an simply issue warning events. How to reproduce it (as minimally and precisely as possible): See issue description. Anything else we need to know?: No. Environment: Kubernetes version (use ): 1.18.3 Cloud provider or hardware configuration: IBM Cloud Kubernetes Service OS (e.g: ): Ubuntu 18.04.4 LTS Kernel (e.g. ): 4.15.0-96-generic Install tools: Managed service install Network plugin and version (if this is a network-related bug): N/A Others: N/A
Related to and
/sig scheduling
by spamming are you referring to scheduler logs? also, why are the nodes not labelled from the get go?
Just to confirm, all the pods in question are still in Pending state, yes? If the pods are in pending state, all of them will get scheduled eventually once the nodes get their labels.
The pods are not in state. They are actually in state. The pods eventually get scheduled but we are left with anywhere from a few to dozens of these state pods. Also, we are looking at labeling the nodes via kubelet flags to see if that helps.
/remove-kind bug
After we add some retry logic and wait time in this problem can be solved: As you can see this function returns initial node as the first request failed.
I also get hit by this, when single controller node gets rebooted and scheduler starts as static container before kubelet becomes ready.


Question: kubelet.masterServiceNamespace = metav1.NamespaceDefault kubelet.serviceLister = testServiceLister{} kubelet.serviceHasSynced = func() bool { return true } <ins> kubelet.nodeHasSynced = func() bool { return true } </ins> kubelet.nodeLister = testNodeLister{ nodes: []*v1.Node{ {

Can be cleaned up with:
/sig node
the spam cycle is this loop: "pod create, pod schedule, worker reject, pod failed" -- we don't have a lot of protection in higher level workload controllers to handle pods that fail at kubelet admission time versus pods that just dont create due to quota/admission. in general, is the spam causing any other stability issues? is pod gc pruning the pods?
I wonder if we can do a trick in node lister to mitigate any race condition here...
we could maybe do something similar to to ensure node lister has synced...
hacking on option to ensure if kubelet has a node lister with a valid kube client, we wait for it to sync at least once here which should mitigate this issue. see:
We noticed the problem on the OLM pod which is managed by the OLM operator. Right or wrong, the operator does not like seeing the pod go to status and no longer reconciles the pod. The end result is that manual intervention (i.e. pod deletion) is required to fix the problem.
sure... i would have thought the pod is failed in that instance and reached a terminal state. either way, i am looking at what i can do to mitigate this.
The problem persists in our environment as well, and the workaround cited above will not do anymore. Is there something in the works for this one?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


Question: return fmt.Errorf("PersistentVolumeClaims %v not all in phase %s within %v", pvcNames, phase, timeout) } <ins> // findAvailableNamespaceName random namespace name starting with baseName. func findAvailableNamespaceName(baseName string, c clientset.Interface) (string, error) { var name string err := wait.PollImmediate(Poll, 30*time.Second, func() (bool, error) { name = fmt.Sprintf("%v-%v", baseName, randomSuffix()) _, err := c.CoreV1().Namespaces().Get(name, metav1.GetOptions{}) if err == nil { // Already taken return false, nil } if apierrs.IsNotFound(err) { return true, nil } Logf("Unexpected error while getting namespace: %v", err) return false, nil }) return name, err } </ins> // CreateTestingNS should be used by every test, note that we append a common prefix to the provided test name. // Please see NewFramework instead of using this directly. func CreateTestingNS(baseName string, c clientset.Interface, labels map[string]string) (*v1.Namespace, error) {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: In the test failed due to namespace leakage: we created e2e-tests-load-30-nodepods-1-pfn9m, but never got response from api server (got 0-length response instead) we retried and created e2e-tests-load-30-nodepods-1-m9tq5 e2e-tests-load-30-nodepods-1-m9tq5 to f.namespacesToDelete and leaked -pfn9m next test case failed because e2e-tests-load-30-nodepods-1-pfn9m wasn't deleted. What you expected to happen: While it's unclear why we got 0-length response instead ( investigates that), we always can get error while the namespace was created (e.g. connection reset, timeout, io error). We should handle this case in retrying correctly. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): v1.14.0-alpha.0.793+ Cloud provider or hardware configuration: gce OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
There are no sig labels on this issue. Please add a sig label by either: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. See the . The in method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc


Question: endpointsAvailableForLB := !allEndpointsTerminating && !allEndpointsNonServing proxier.deleteExistingLoadBalancer(hns, svcInfo.winProxyOptimization, &svcInfo.hnsID, sourceVip, Enum(svcInfo.Protocol()), uint16(svcInfo.targetPort), uint16(svcInfo.Port()), hnsEndpoints, queriedLoadBalancers) <del> if endpointsAvailableForLB { </del> <ins> // clusterIPEndpoints is the endpoint list used for creating ClusterIP loadbalancer. clusterIPEndpoints := hnsEndpoints if svcInfo.internalTrafficLocal { // Take local endpoints for clusterip loadbalancer when internal traffic policy is local. clusterIPEndpoints = hnsLocalEndpoints } </ins> <del> // clusterIPEndpoints is the endpoint list used for creating ClusterIP loadbalancer. clusterIPEndpoints := hnsEndpoints if svcInfo.internalTrafficLocal { // Take local endpoints for clusterip loadbalancer when internal traffic policy is local. clusterIPEndpoints = hnsLocalEndpoints } </del> <ins> if len(clusterIPEndpoints) > 0 { </ins> // If all endpoints are terminating, then no need to create Cluster IP LoadBalancer // Cluster IP LoadBalancer creation

I am tracking a changed behavior in networking with kubernetes 1.26. If I create a service of type externalTrafficPolicy: Local, and there are no local endpoints in a node, kube-proxy does not create hns loadbalancers for the clusterIP of that service. With kubernetes 1.25, it creates a loadbalancer for clusterIP (but not externalIP). This is the relevant log message: Looks like this comes with a result from a new function isAllEndpointsTerminating introduced in kubernetes 1.26. In Linux, we can still access clusterIP even if there are no local endpoints for the service. Thoughts? The end result is that in Windows, clients cannot access the clusterIP of such a service with there are no local endpoints.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig windows
cc
Does this issue occur when there are no local endpoints for a given service? If there is a local endpoint, then you can reach the service again, right?
, Also let us know, do you have any InternalTrafficPolicy configured ?
Does this issue occur when there are no local endpoints for a given service? If there is a local endpoint, then you can reach the service again, right? That is correct , Also let us know, do you have any InternalTrafficPolicy configured ? We do not have that configured


Question: } mountComplete := false <del> err = wait.Poll(5*time.Second, 10*time.Minute, func() (bool, error) { </del> <ins> err = wait.PollImmediate(1*time.Second, 2*time.Minute, func() (bool, error) { </ins> err := b.mounter.MountSensitive(source, dir, "cifs", mountOptions, sensitiveMountOptions) mountComplete = true return true, err

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: When starting pods with AFS volumes, there is an initial delay of five seconds until the pod is transitioning from the "Scheduled" state. The reason for this is that currently the volume mounting happens inside a wait.Poll which will initially wait a specified interval (currently 5 seconds) before execution. Event log for comparison with and without AFS volume attached (seconds since event): What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /assign /priority important-soon /sig cloud-provider /area provider/azure
see


Question: klog.V(4).Info(log("attacher.Attach finished OK with VolumeAttachment object [%s]", attachID)) <ins> // TODO(71164): In 1.15, return empty devicePath </ins> return attachID, nil } <del> func (c *csiAttacher) WaitForAttach(spec *volume.Spec, attachID string, pod *v1.Pod, timeout time.Duration) (string, error) { </del> <ins> func (c *csiAttacher) WaitForAttach(spec *volume.Spec, _ string, pod *v1.Pod, timeout time.Duration) (string, error) { </ins> source, err := getCSISourceFromSpec(spec) if err != nil { klog.Error(log("attacher.WaitForAttach failed to extract CSI volume source: %v", err)) return "", err } <ins> attachID := getAttachmentName(source.VolumeHandle, source.Driver, string(c.plugin.host.GetNodeName())) </ins> skip, err := c.plugin.skipAttach(source.Driver) if err != nil { klog.Error(log("attacher.Attach failed to find if driver is attachable: %v", err))

The test flakes very occasionally. Below are the findings of the circumstances of the flake. During two pods are created in quick succession. The test flakes when the second Pod fails to come up. The second pod fails to come up with error: Upon exploring the logs we discovered that there was only 1 Attach call made at the beginning, and one Detach call upon timeout of the test [See timestamps]: We tracked down the error to the likely culprit in func : based on the error message returned. This is likely caused by (string) being empty [uncomfirmed], which is what the Kubelet would get from the DevicePath field on the Node object. This DevicePath field is normally on Attach and removed on Detach of the volume. The DevicePath is clearly present on the Node when the first pod is being brought up. But it is possible that it is somehow missing or not able to be retrieved by the time the second Pod is starting up (and continues to be unavailable exhausting the 15m timeout on the test) even though no Detach operation was performed on the volume in the meantime (according to logs). We are currently unable to reproduce this flake on command and did all analysis based on logs from the two flakes we saw naturally occur in the 100s of runs performed of the test. This flake was originally seen in and the test logs can be found here: https://k8s- Environment: GCE Kubernetes version (use ):Cloud provider or hardware configuration: GCP OS (e.g. from /etc/os-release):Kernel (e.g. ): /kind flake /kind bug /sig storage /cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale A check in triage shows that this failing test is still a problem, and in fact has become more of a problem since October 4th. Has there been any progress on a fix?


Question: klog.V(4).Info(log("attacher.Attach finished OK with VolumeAttachment object [%s]", attachID)) <ins> // TODO(71164): In 1.15, return empty devicePath </ins> return attachID, nil } <del> func (c *csiAttacher) WaitForAttach(spec *volume.Spec, attachID string, pod *v1.Pod, timeout time.Duration) (string, error) { </del> <ins> func (c *csiAttacher) WaitForAttach(spec *volume.Spec, _ string, pod *v1.Pod, timeout time.Duration) (string, error) { </ins> source, err := getCSISourceFromSpec(spec) if err != nil { klog.Error(log("attacher.WaitForAttach failed to extract CSI volume source: %v", err)) return "", err } <ins> attachID := getAttachmentName(source.VolumeHandle, source.Driver, string(c.plugin.host.GetNodeName())) </ins> skip, err := c.plugin.skipAttach(source.Driver) if err != nil { klog.Error(log("attacher.Attach failed to find if driver is attachable: %v", err))

/priority important-soon
/assign for triage
We should find the root cause and fix for 1.13
/assign Patrick can you PTAL at what is causing flakiness in this test.
I will have a look. Right now the log output of the sidecar containers and the CSI driver are not captured, which makes debugging a bit hard. In PR I am adding code which includes those message in the test log. Let me finish that PR and once it is merged, debugging the next test flake might provide some better insight into what went wrong.
Hey, this issue is considered a blocker for 1.13beta. Is there anything we can do to get it resolved sooner, so that it doesn't delay beta next week?
is trying to instrument the e2e tests to get logs from the csi driver containers so that we can debug. He has tried running the test locally ~100 times and didn't see it flake.
here's one recent one you can check out:
Michelle Au <: I had a look, but something is odd about that one. The error for the second pod is there: But under there is no further information about node bootstrap-e2e-gce-mz-minion-group-3kn6. Shouldn't that be available?
"Patrick Ohly" Here's a better example where the kublet log with the error is available: That one also has the log of the external-attacher, but that doesn't tell me much. I'm now comparing a "bad" run with a "good" local run. One observation is that in the "good" run, operationExecutor.UnmountDevice is completed before the second usage, including the "Volume detached for volume" step in Then operationExecutor.VerifyControllerAttachedVolume is done again for the second pod, exactly as it was done for the first: In the "bad" run, operationExecutor.MountVolume is invoked before that "Volume detached for volume" and then runs with an empty device path, without going through VerifyControllerAttachedVolume again: So to me it looks like a race between two conflicting operations. Does that make sense? How to fix it is less obvious. The relevant code seems to be this in Any suggestions?
This seems inline with the observations that made initially. We should trace where DevicePath is getting set.


Question: klog.V(4).Info(log("attacher.Attach finished OK with VolumeAttachment object [%s]", attachID)) <ins> // TODO(71164): In 1.15, return empty devicePath </ins> return attachID, nil } <del> func (c *csiAttacher) WaitForAttach(spec *volume.Spec, attachID string, pod *v1.Pod, timeout time.Duration) (string, error) { </del> <ins> func (c *csiAttacher) WaitForAttach(spec *volume.Spec, _ string, pod *v1.Pod, timeout time.Duration) (string, error) { </ins> source, err := getCSISourceFromSpec(spec) if err != nil { klog.Error(log("attacher.WaitForAttach failed to extract CSI volume source: %v", err)) return "", err } <ins> attachID := getAttachmentName(source.VolumeHandle, source.Driver, string(c.plugin.host.GetNodeName())) </ins> skip, err := c.plugin.skipAttach(source.Driver) if err != nil { klog.Error(log("attacher.Attach failed to find if driver is attachable: %v", err))

Could there be a race with AD controller or external attacher if they try to detach the volume? is another race around this area. Unclear if it's the same yet.
Michelle Au <: No idea. Even if it isn't, it's at least related to the same piece of code. Probably worth fixing both together. Can we get someone who has worked in this area to fix these issues? Today is the first day that I've ever seen this code, so I'm a bit afraid that it'll take me a long time to get it right, and I'm worried about regressions.
I've asked to help investigate
It is not a blocker for beta or freeze. Can be addressed in stabilization. Let's keep the 1.13 tag on it. And based on progress by stabilization we can reassess if it is even a 1.13 blocker (at this point I don't believe it is).
The in the actual state of the world gets reset during unmounting:
That removes the device path looks wrong to me. I've submitted PR which changes that. Let's see how much that breaks...
/assign The conclusion in the CSI working group was that while my analysis and the resulting fix might be correct, further debugging and a second pair of eyes would be good. Let me reassign, because I have done what I could for this issue.
The RT considers this a blocker now, because flakes are happening often enough to interfere with getting clear signal on several jobs. Even if the underlying problem isn't that serious.
/assign I will help triage as well.
Ack Thanks
Instead of which may have negative side effects for non-CSI volumes, let's do this instead: Modify the CSI method in to ignore the passed in (which is populated from devicePath. Instead let's have that method generate the from scratch, e.g. attachID := getAttachmentName(volID, driverName, string(nodeName)) This will remove any CSI dependency on devicePath and not effect other volume plugins.
I am trying to twist my head around the code. IMO it affects also AWS that uses devicePath stored in and it should be reproducible there too. So far I was unsuccessful. In the logs it's clearly visible that UnmountDevice succeeded. So why kubelet continues with MountVolume instead of MountDevice? That's wrong.


Question: klog.V(4).Info(log("attacher.Attach finished OK with VolumeAttachment object [%s]", attachID)) <ins> // TODO(71164): In 1.15, return empty devicePath </ins> return attachID, nil } <del> func (c *csiAttacher) WaitForAttach(spec *volume.Spec, attachID string, pod *v1.Pod, timeout time.Duration) (string, error) { </del> <ins> func (c *csiAttacher) WaitForAttach(spec *volume.Spec, _ string, pod *v1.Pod, timeout time.Duration) (string, error) { </ins> source, err := getCSISourceFromSpec(spec) if err != nil { klog.Error(log("attacher.WaitForAttach failed to extract CSI volume source: %v", err)) return "", err } <ins> attachID := getAttachmentName(source.VolumeHandle, source.Driver, string(c.plugin.host.GetNodeName())) </ins> skip, err := c.plugin.skipAttach(source.Driver) if err != nil { klog.Error(log("attacher.Attach failed to find if driver is attachable: %v", err))

A workaround in CSI volume plugin won't really help here, it will work only for drivers without .
I finally reproduced it on AWS with a long sleep in : Steps to reproduce (with the hack above): a pod with a volume the pod. Wait until is called. a new pod with the same volume. - checks that the volume is still attached and the device is still mounted and enqueues an operation to mount the volume. When finishes, starts and 1) uses empty and 2) calls on volume that's not DeviceMounted.
Fix: Any idea how to test it for real? Fake CSI plugin with slow NodeUnstage would be the best, however we don't have one.
I'm going to work on a targeted fix for CSI. We can continue to pursue fixing aws in-tree separately.
PR to fix csi is here:
what are the final set of PRs pending for this? , and ?
Only is needed to fix this particular test failure.
can you please confirm if we are going to fix this using or ?
I discussed with offline, he has some suggestions for that I am investigating. I think we will want both PRs eventually, although only one of them will be needed to fix the CSI test for now. Since it's his off hours, I will continue working on today.
Are we talking 1.13 for as well?
Reopening this since we are seeing consistent failures in latest runs. We also see one more test fail in the job with same error. Failing job https://k8s- Failing Tests [sig-storage] [Serial] CSI Volumes CSI plugin test using CSI driver: gcePD [Serial] should provision storage [sig-storage] [Serial] CSI Volumes CSI plugin test using CSI driver: hostPath [Serial] should provision storage Error This is now 1.13 release blocking.
/priority critical-urgent /kind failing-test
/remove-priority important-soon
I think this is a different issue and could be due to the fact that CSI is intentionally not backwards compatible between 1.12 and 1.13. We may need to enhance the test to figure out what version the cluster is on and launch appropriate versions of the CSI driver.
"intentionally not backwards compatible" -- is there a writeup for the release notes and release announcement for this change?


Question: klog.V(4).Info(log("attacher.Attach finished OK with VolumeAttachment object [%s]", attachID)) <ins> // TODO(71164): In 1.15, return empty devicePath </ins> return attachID, nil } <del> func (c *csiAttacher) WaitForAttach(spec *volume.Spec, attachID string, pod *v1.Pod, timeout time.Duration) (string, error) { </del> <ins> func (c *csiAttacher) WaitForAttach(spec *volume.Spec, _ string, pod *v1.Pod, timeout time.Duration) (string, error) { </ins> source, err := getCSISourceFromSpec(spec) if err != nil { klog.Error(log("attacher.WaitForAttach failed to extract CSI volume source: %v", err)) return "", err } <ins> attachID := getAttachmentName(source.VolumeHandle, source.Driver, string(c.plugin.host.GetNodeName())) </ins> skip, err := c.plugin.skipAttach(source.Driver) if err != nil { klog.Error(log("attacher.Attach failed to find if driver is attachable: %v", err))

Yes has the release note
thanks, I'll also call it to the Release Notes lead's attention
what does it mean from the user impact perspective for CSI to not be backward compatible with 1.12? can you please point me to any discussions on this behavior for 1.13?
Kubernetes v1.13 speaks CSI v1.0 and no longer understands CSI v0.x. Any CSI drivers deployed on previous versions of Kubernetes speaking CSI v0.x must removed, updated to be CSI 1.0 compatible, and redeployed on the on Kubernetes v1.13. If only the master is upgraded to 1.13 (and nodes are 1.12 or earlier) older drivers will continue to work. When nodes are upgraded to 1.13, then older drivers will stop working. I will capture this in detail in the "Action Required" section.
Im tracking these upgrade test failures elsewhere: They are not related to this original issue, the flaky test referenced by this issue should have been solved by and for non CSI /close
Closing this issue. <detailsIn response to : Im tracking these upgrade test failures elsewhere: They are not related to this original issue, the flaky test referenced by this issue should have been solved by /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
as FYI to call this out clearly in the Release notes.


Question: "fmt" "io" "os" <ins> "strconv" </ins> "github.com/GoogleCloudPlatform/kubernetes/pkg/api" cmdutil "github.com/GoogleCloudPlatform/kubernetes/pkg/kubectl/cmd/util"

I an ELB to the Kibana cluster log viewer but it is not reported by which, aside from having a URL, should check each cluster service and if it has an external IP it should report that instead of the master proxy address.
cc


Question: } ) <ins> func TestConflictingCurrentContext(t *testing.T) { commandLineFile, _ := ioutil.TempFile("", "") defer os.Remove(commandLineFile.Name()) envVarFile, _ := ioutil.TempFile("", "") defer os.Remove(envVarFile.Name()) mockCommandLineConfig := clientcmdapi.Config{ CurrentContext: "any-context-value", } mockEnvVarConfig := clientcmdapi.Config{ CurrentContext: "a-different-context", } WriteToFile(mockCommandLineConfig, commandLineFile.Name()) WriteToFile(mockEnvVarConfig, envVarFile.Name()) loadingRules := ClientConfigLoadingRules{ CommandLinePath: commandLineFile.Name(), EnvVarPath:      envVarFile.Name(), } mergedConfig, err := loadingRules.Load() if err != nil { t.Errorf("Unexpected error: %v", err) } if mergedConfig.CurrentContext != mockCommandLineConfig.CurrentContext { t.Errorf("expected %v, got %v", mockCommandLineConfig.CurrentContext, mergedConfig.CurrentContext) } } </ins> func TestResolveRelativePaths(t *testing.T) { pathResolutionConfig1 := clientcmdapi.Config{ AuthInfos: map[string]clientcmdapi.AuthInfo{

I've been pulling my hair out trying get kubectl talk to my local cluster. It seems that kubectl merges the local file with values pulled down from the cloud. I've done the following: The I just set is silently overridden. Questions: is this documented anywhere? should kubectl merge in my GCE cluster when I set my cloud-provider to something else? should kubectl override my previously set current-context?
workaround is to use flag to override the config file value.
but not a good UX long term
Someone else hit this as well: . Better fix is to have the local-up script set a context, which I will try to add today. As for the overwriting behavior, it's documented in , but based on what you describe, the intended load order is not working; local .kubeconfig should overwrite global. cc
That sounds infuriating. I'll take a look this morning.


Question: UpdateFake(obj *metav1.PartialObjectMetadata, opts metav1.UpdateOptions, subresources ...string) (*metav1.PartialObjectMetadata, error) } <ins> // NewTestScheme creates a unique Scheme for each test. func NewTestScheme() *runtime.Scheme { return runtime.NewScheme() } </ins> // NewSimpleMetadataClient creates a new client that will use the provided scheme and respond with the // provided objects when requests are made. It will track actions made to the client which can be checked // with GetActions().

Because of this scheme mutation, this leads to concurrent use of the same scheme to cause the program to panic. It would be ideal to do this in some concurrent safe manner or add a warning or similar
/sig api-machinery
/assign
/assign /triage accepted


Question: cache.ResourceEventHandlerFuncs{UpdateFunc: vs.syncNodeZoneLabels}, zoneLabelsResyncPeriod, ) <ins> nodeLister := informerFactory.Core().V1().Nodes().Lister() vs.nodeManager.SetNodeLister(nodeLister) </ins> klog.V(4).Infof("Node informers in vSphere cloud provider initialized") }

I am enabling CSI migration on a vSphere cluster. A node was fully drained, all its Pods are deleted, but just before KCM detaches in-tree vSphere volume volumes from it, KCM restarts. A new KCM starts, sees attached volumes in that are not needed by any Pod on the node and calls vSphere in-tree volume plugin . The in-tree vSphere volume plugin + cloud provider marks the volume as detached right away, without reaching to vCenter API. The volume remains attached forever. I don't think it's related to CSI migration, IMO a regular node drain + KCM restart at the same time can cause the same issue. KCM logs: The in-tree cloud provider has a and from it. It apparently has not processed all nodes at the time A/D controller tries to detach the node. The in-tree vSphere volume plugin + cloud provider detaches the volume in vCenter. That's really tricky, it's very hard / impossible to reproduce on a real cluster. I had to inject errors to Detach() in-tree volume call, so they fail until KCM is restarted. Newly started KCM would detach the volume just fine (no error would be injected there), but marks the volume as detached right away. For the record (and search-ability), the CSI driver that tries to attach the volume to another node complains either: Or: cc It could be somewhat related to , however, not all cases of can be related to KCM restart. As I mentioned above, it's very hard/impossible to synchronize KCM restart with draining a node in a real cluster. <details</details<detailsvSphere 7.0.3 </details<details</details<detailsOpenShift </details<detailsCRI-O </details<details</details
/sig storage
Tried just moving a Pod between nodes without CSI migration involved + restarting KCM when the volume is still attached to the old node. In-tree volume plugin on the new node says:
/assign
What CSI driver version and K8s version are you testing with?
Kubernetes is 1.25 and CSI driver is 2.7, still, I think it's not releated to the CSI driver, it's KCM who marks the in-tree volume as detached due to not populated node cache.


Question: cache.ResourceEventHandlerFuncs{UpdateFunc: vs.syncNodeZoneLabels}, zoneLabelsResyncPeriod, ) <ins> nodeLister := informerFactory.Core().V1().Nodes().Lister() vs.nodeManager.SetNodeLister(nodeLister) </ins> klog.V(4).Infof("Node informers in vSphere cloud provider initialized") }

/assign
/triage accepted
I found an easier way how to reproduce this without extra code in KCM: a StatefulSet with 1 Pod that uses 1 PVC. the node where the Pod runs. down kube-controller-manager to zero (not really possible for me) or kill all kube-controller-manager Pods. This will cause new KCM processes to wait for leader election. In my env. this gives me ~30 seconds for the next step. Delete the Pod. A new one will be scheduled on a different node (because the current node is cordoned) + there is no KCM running at this time that would detach the volume from the old node. KCM wins its leader election, sees a volume attached to a node where it is not needed -it calls of the volume plugin. VSphere cloud provider does not have its caches populated yet (= this bug) -the volume is not detached, but returns success.
dangling attach does not help here. The volume is still attached to the old node even after ~10 minutes. fails with and not with . I can see there is a lot of dangling attach code in the cloud provider, but I did not debug further.


Question: // Status of containers in the pod. ContainerStatuses []*ContainerStatus // Status of the pod sandbox. <ins> // Only for kuberuntime now, other runtime may keep it nil. </ins> SandboxStatuses []*runtimeapi.PodSandboxStatus }

What happened: It's most visible in the test <img width="323" alt="Screen Shot 2020-05-20 at 6 09 10 PM" src="https://user-"It shows up in kind, kube- and kops jobs The regression was introduced somewhere within , possible culprits are: (FYI (FYI (FYI There are more tests affected, just not as prominently (visually inspecting ) [sig-apps] StatefulSet [] Basic StatefulSet functionality [StatefulSetBasic] should perform rolling updates and roll backs of template modifications with PVCs [sig-apps] StatefulSet [] Basic StatefulSet functionality [StatefulSetBasic] should perform rolling updates and roll backs of template modifications [Conformance] [sig-storage] In-tree Volumes [Driver: nfs] [Testpattern: Inline-volume (default fs)] volumes should store data [sig-storage] In-tree Volumes [Driver: nfs] [Testpattern: Pre-provisioned PV (default fs)] volumes should store data I wasn't sure whether to tag this as kind/regression or not, I'll leave that up to or someone /sig node /sig api-machimery /sig scheduling
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : What happened: It's most visible in the test (FYI (FYI (FYI StatefulSet [] Basic StatefulSet functionality [StatefulSetBasic] should perform rolling updates and roll backs of template modifications with PVCs [sig-apps] StatefulSet [] Basic StatefulSet functionality [StatefulSetBasic] should perform rolling updates and roll backs of template modifications [Conformance] [sig-storage] In-tree Volumes [Driver: nfs] [Testpattern: Inline-volume (default fs)] volumes should store data [sig-storage] In-tree Volumes [Driver: nfs] [Testpattern: Pre-provisioned PV (default fs)] volumes should store data /sig api-machimery /sig scheduling Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository.


Question: // Status of containers in the pod. ContainerStatuses []*ContainerStatus // Status of the pod sandbox. <ins> // Only for kuberuntime now, other runtime may keep it nil. </ins> SandboxStatuses []*runtimeapi.PodSandboxStatus }

</details
/cc
will increase the time for deletion of pod and it should be expected because currently pod is deleted from apiserver before removing the sandbox container but after this fix pod will be deleted from the apiserver only after deleting the sandbox container.
/remove-sig scheduling doesn't affect scheduling running time at all.
Yeah, is almost definitely the culprit. Last time we changed that code path, it also caused a regression. The degradation in performance is expected, but a 5x slowdown is definitely larger than it should be.
do we know if that's because the status used by the path is stale, or if removing sandboxes is actually that slow?
if there's not a quick fix, should we revert and bring it back in once the root issue is understood?
I think a revert is appropriate in this case.
ok, I'll open it
open at
I agree with that 5x is something which shouldn't happen and i can take a look on why thats happening and that might require changing the way how the sandbox is currently deleted which is only during GC.


Question: metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/util/wait" "k8s.io/client-go/kubernetes/fake" <del> "k8s.io/kubernetes/pkg/apis/apps" </del> kubepod "k8s.io/kubernetes/pkg/kubelet/pod" "k8s.io/kubernetes/pkg/kubelet/prober/results" "k8s.io/kubernetes/pkg/kubelet/status"

The metric used to report acutal pod in the label , but now I see the podname prefix only. The full pod name in the label, so I can trace the metric to the failing pod. Check raw api and verify the actual pod name with uids. <details</detailsAzure AKS <details</detailsNo response No response No response_
/sig node
/triage accepted /priority important-soon /cc As discussed this is a change in functionaloty that wasn't highlighted in release notes. Need to decide on what do do with it. I understand it's a change of functionality. what is the side effect of this for you? Does the label without last part not informational enough?
I've been using the metric in my custom grafana dashboard that shows CPU, RAM, IO usage of pods with graph of increases in probes. If the apps are stable, the probe graph looks steadily. I can see if some irregularities in system metrics are influencing probes of respective pod. Also, if readiness probe is failing, the pod is removed from service, but is not restarted so I must have knowledge of pods with failing readiness probes. I attach the screenshot of my graph. ! So I'd welcome the possibility to pair the probes with exact pod and thank you for looking into it.
are we thinking that this resulted in an undesirable change? Spologies for being a bit late to reply, I'm returning from a brief paternity leave staycation.
This issue is labeled with but has not been updated in over 90 days, and should be re-triaged. Important-soon issues must be staffed and worked on either currently, or very soon, ideally in time for the next release. You can: Confirm that this issue is still relevant with (org members only) Deprioritize it with or Close this issue with For more details on the triage process, see /remove-triage accepted
Well this is awkward... I still believe the actual behavior does not respect the intended idea...
This issue makes it hard to link probe metrics with right pods. The value of pod label in and are also inconsistent with how it is used in other kubelet metrics, see: vs Definitely a bug, regression since 1.25.
I can't get anyone to look at the fix/revert: you may have better luck
/milestone v1.28
discussion in
You've pushed it through, congrats!


Question: expectedErrMsg:      fmt.Errorf("failure of getting instance metadata"), }, { <del> name:                "NodeAddresses should report error if VMSS instanceID is invalid", nodeName:            "vm123456", metadataName:        "vmss_$123", vmType:              vmTypeVMSS, useInstanceMetadata: true, expectedErrMsg:      fmt.Errorf("failed to parse VMSS instanceID %q: strconv.ParseInt: parsing %q: invalid syntax", "$123", "$123"), }, { </del> name:                "NodeAddresses should report error if cloud.vmSet is nil", nodeName:            "vm1", vmType:              vmTypeStandard,

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: If computer name prefix does not match VMSS name, then Azure cloud provider will use VMSS API instead of using metadata service data. This leads to throttling, because every affected kubelet is making API requests. What you expected to happen: Azure cloud provider should use metadata service data instead of calling VMSS API when instance is current. How to reproduce it (as minimally and precisely as possible): Set computer name prefix of the VMSS to something different than default. You should observe Azure provider making requests to the VMSS API rather than using metadata service, which leads to throttling if you have multiple kubelets with this bug. Anything else we need to know?: This is bug was introduced by - returns false even if instance is current, because it expects node name to match VMSS name + instance id, however node name is derived from the computer-name-prefix which can be different than VMSS name. Node name should be matched against , not against . Environment: Kubernetes version (use ): 1.18.x and higher Cloud provider or hardware configuration: azure /sig cloud-provider
/sig cloud-provider
/area provider/azure
CC
/remove-sig cloud-provider /sig cloud-provider (trying to remove the label)
/sig cloud-provider
Thanks for the reporting. Unfortunately, computer name prefix is not included in IMDS, so it can't be used without VMSS API for constructing the NodeName. Would check what could we do to workaround the issue.
Can you memorize the computer name prefix in the first request to the Azure API and then use it for the isCurrentInstance check?
opened to fix the issue. Would you like to have a look?
I can confirm that your change should fix the problem I reported. Do you know if your fix will be to 1.18.x?
is working on the cherry pick
Thank you for quick fix!


Question: OOMScoreAdj:        config.OOMScoreAdj, ConfigSyncPeriod:   config.ConfigSyncPeriod.Duration, HealthzServer:      healthzServer, <ins> UseEndpointSlices:  false, </ins> }, nil }

After this () has been merged and EnpointSlice feature is now enabled by default, we noticed Kube-proxy will not work on Windows. Only message I can see in the logs is: /sig windows
/cc /cc /cc
How about separating the above check into two if statements so that the log can pinpoint which field is false ?
It seems long term, EndpointSlice support should be so that is kept in sync with other implementations.
cc
Thanks for reporting this! Windows proxiers were still intended to be reading from Endpoints, but apparently that's not working as intended. EndpointSlices are moving back to disabled by default for 1.17, so that should fix the failing tests for now. I'll work on an actual fix for this as well, but it may not make it in until after KubeCon. /assign /sig network /priority important-soon
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/remove-triage unresolved
/assign cc
GitHub didn't allow me to assign the following users: jocelynberrendonner. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign cc Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
: Please let me know if you need any help on the fix! I'll be more than happy to review/help test the fix on Windows.


Question: }) It("should serve a basic image on each replica with a private image", func() { <del> switch testContext.Provider { case "gce", "gke": ServeImageOrFail(c, "private", "gcr.io/_b_k8s_authenticated_test/serve_hostname:1.1") default: </del> <ins> if !providerIs("gce", "gke") { </ins> By(fmt.Sprintf("Skipping private variant, which is only supported for providers gce and gke (not %s)", testContext.Provider)) <ins> return </ins> } <ins> ServeImageOrFail(c, "private", "gcr.io/_b_k8s_authenticated_test/serve_hostname:1.1") </ins> }) })

I noticed while running e2e tests locally that some tests were apparently being incorrectly skipped: I checked, and this is also true for our Jenkins runs. Currently it looks like and have this bug, though only the former actually skips the test, I think. The issue appears to be that the code inside the top-level in these files is being run before the testContext struct is filled out (and before is even called), and so testContext.Provider is empty. I'm not immediately sure what the proper fix is; I'm also worried that there might be other things broken by the fact that these blocks are called before initialization is complete. We also probably investigate if there is a better way to mark these tests as skipped; right now, there's no indication in the test report (, Jenkins test report, etc) that these tests were skipped.
I merged and which tried to disable these tests for non-{gke,gce} but apparently didn't follow-up and verify that they were still running on our platforms. /cc
Seems easy enough to fix. I'll have a PR out to fix this before Monday.


Question: DeleteLocalData     bool Selector            string PodSelector         string <del> Out                 io.Writer ErrOut              io.Writer </del> <ins> // DisableEviction forces drain to use delete rather than evict DisableEviction bool Out    io.Writer ErrOut io.Writer </ins> // TODO(justinsb): unnecessary? DryRun bool

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Allow to have an additional flag (e.g. or ) to allow a to drain pods off a node, even if they violate PodDisruptionBudgets Why is this needed: There are times where you want to drain pods off a node even if violating a PDB. For example, if you are running a one-worker node kubernetes cluster (not minikube), and you want to upgrade it. Or, like me, you are running a test cluster and want to scale down to zero nodes when you're not going to be using the cluster for a while in order to save costs. If that worker node has a deployment with a PDB, then will get stuck as it cannot get rid of the pod while respecting the PDB, which is expected; but in this case, it is a hindrance. Adding the switch will allow users to explicitly acknowledge that they want to ignore the PDBs (and accept the corresponding downtime for an application/service)
/sig node
+1
+1 Our company offers Kubernetes as PaaS. We manage Kubernetes, including IaaS OS updates and if a customer does not specify the PDB correctly, we are not able to drain the node for the OS updates. A flag would be werry helpfull for our use case.
+1
/active
I have created to address this. This is a pretty simple implementation, but I think it achieves the desired results. A suitable workflow would be: 1) Configure you component to use drain as normal 2) After some timeout, run drain with --disable-eviction option.
I'll wait until the PR is merged and released before testing, but if I understand right, it would require me do this, right? If so, I think it would be better if we did it a different way and have one call, something like this: Where implies that if the drain is still running after this time, switch to the mode without having to call kubectl again
I think that would require a sizeable refactor or add a great deal of complexity to the code, the same could be achieved with In practical terms, I'm focused on automatic node remediation via destroying the node and getting a new one. The timeout would be very large, the drain library is blocking, so I want it to fail quickly and requeue.


Question: DeleteLocalData     bool Selector            string PodSelector         string <del> Out                 io.Writer ErrOut              io.Writer </del> <ins> // DisableEviction forces drain to use delete rather than evict DisableEviction bool Out    io.Writer ErrOut io.Writer </ins> // TODO(justinsb): unnecessary? DryRun bool

I'm going to be performing the calculation of when it's appropriate to evict/delete outside of the drain library, and I suspect most will be as well, so there's not really a great incentive to do it in the drain library itself IMO.
That's all good, I have no issue using the flag as long as does error out and return a non-zero code as expected (there's been a few times where it seems to error out, but still return 0)


Question: // RemoveContainer removes the container. func (ds *dockerService) RemoveContainer(_ context.Context, r *runtimeapi.RemoveContainerRequest) (*runtimeapi.RemoveContainerResponse, error) { <del> ds.performPlatformSpecificContainerForContainer(r.ContainerId) </del> // Ideally, log lifecycle should be independent of container lifecycle. // However, docker will remove container log after container is removed, // we can't prevent that now, so we also clean up the symlink here.

What would you like to be : Periodically remove cleanupInfos for old containers from dockershim's map. One way of doing that would be to just forget about s older than say an hour, as they should always be short-lived. Why is this needed: introduced a new in-memory map for the dockershim, that could potentially (in pathological cases) cause memory leaks - for containers that use GMSA cred specs, get created successfully, but then never get started nor removed. This might be okay for the alpha release of the GMSA feature, but should be addressed in the beta. See
/assign wk8
/sig windows
/milestone v1.15
got moved out. let's move this issue with it as well /milestone v1.16
Hello! I'm bug triage lead for the 1.16 release cycle and considering this issue is tagged for 1.16, but not updated for a long time, I'd like to check its status. The code freeze is starting on August 29th (about 1.5 weeks from now), which means that there should be a PR ready (and merged) until then. Do we still target this issue to be fixed for 1.16?
: yup we'd love to have this in 1.16, still waiting for kind reviewers on :) ...


Question: t.Fatalf("unexpected proxy user: %v", authUser) } } <ins> // The channel must be closed before any of the connections are closed close(closed) </ins> }) } })

TestRoundTripSocks5AndNewConnection Disabled by No response No response No response /sig
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
there is a race shutting down the socks5 server


Question: network_mode: openvswitch node_ip: $MINION_IP etcd_servers: $MASTER_IP <ins> api_servers: $MASTER_IP </ins> networkInterfaceName: eth1 apiservers: $MASTER_IP roles:

is sent as a fix. cc It's a test failure reported by the publishing robot.
The line it fails at:
It looks like a vendor copy vs gopath copy issue.
One possible fix is removing from the vendor/ directory. I'm testing locally.
Confirmed removing from the vendor/ directory fixed the test in robot.
We need to add the "issue filer" feature to the publishing robot, we can't rely on to report robot issues everytime :P I'm just not sure how long the publishing robot will be needed (i.e., when we will get rid of the staging/ directory) so not sure whether we should invest more time on the robot.


Question: "Foo": defaultbinder.New, } cases := []struct { <del> name         string opts         []Option wantErr      string wantProfiles []string </del> <ins> name          string opts          []Option wantErr       string wantProfiles  []string wantExtenders []string </ins> }{ { name: "valid out-of-tree registry",

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --here is supposed to be set through , but is not called from anywhere except tests, so always returns . should be set through here. Am I missing something? N/A None Kubernetes version (use ): master () Cloud provider or hardware configuration: N/A OS (e.g: ): N/A Kernel (e.g. ): N/A Install tools: N/A Network plugin and version (if this is a network-related bug): N/A Others: None
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scheduling
Thanks for pointing that out. The plumbing of extenders to the framework impact calling extenders on preemption. That seems to be a path not covered by tests end to end. Can you submit a PR to fix this? as you mentioned we need to pass the extenders on profile creation. We need to patch the fix back to v1.21 and v1.20 (the release where this was refactored). /cc
I have submitted a PR for a fix.


Question: } t.Logf("rc._internal -> rc.v1") <del> data, err = runtime.Encode(defaultGroup.Codec(), rc) </del> <ins> data, err = runtime.Encode(defaultCodec, rc) </ins> if err != nil { t.Fatalf("unexpected encoding error: %v", err) }

Execute make test WHAT=./pkg/api/testing/ And it failed with information: /kind failing-test
/sig api-machinery
what's the urgency of fixing this? Is it a release blocker?
i don't see it as a release blocker. but the problem is in client-go, so it's important to get fixed.


Question: addNodes(manager.nodeStore, 0, 1, nil) addPods(manager.podStore, "node-0", simpleDaemonSetLabel, ds, 1) addPods(manager.podStore, "node-1", simpleDaemonSetLabel, ds, 1) <ins> podScheduledUsingAffinity := newPod("pod1-node-3", "", simpleDaemonSetLabel, ds) podScheduledUsingAffinity.Spec.Affinity = &v1.Affinity{ NodeAffinity: &v1.NodeAffinity{ RequiredDuringSchedulingIgnoredDuringExecution: &v1.NodeSelector{ NodeSelectorTerms: []v1.NodeSelectorTerm{ { MatchFields: []v1.NodeSelectorRequirement{ { Key:      schedulerapi.NodeFieldSelectorKeyNodeName, Operator: v1.NodeSelectorOpIn, Values:   []string{"node-2"}, }, }, }, }, }, }, } manager.podStore.Add(podScheduledUsingAffinity) </ins> if f { syncAndValidateDaemonSets(t, manager, ds, podControl, 0, 1, 0) } else {

What happened: Occasionally we get a Daemonset stuck with less Current replicas than Desired, causing a Daemonset sync tends to fix it, e.g (add a new node). This happens to all our Daemonsets. I'm using fluentd as an example. There are no Daemonsets pods in pending nor are there any errors when describing the Daemonset What you expected to happen: The Daemonset Pod to be scheduled How to reproduce it (as minimally and precisely as possible): Reproducing this state is very tricky, I've had the most success by terminating nodes in our ASG causing a new one to come up. However I think I've identified why this is happening. The issue appears to be related to daemonset Pod deletion, a Pod gets deleted causing to trigger, this lowers the expectations by as expected. The delete pod event then triggers a sync of the daemonset, in which it finds which involves fetching pods from the informer cache This is somehow returning the deleted pod. You can see in the logs here thinks that it needs to delete the pod that was deleted. It is also setting the expectation back to after the delete pod event previously lowered it by 1. At this point you're stuck as the function can't get past this line As the expectation can not be satisfied as syncNodes set it back to and there's nothing it to get it back to . So I believe this is what's causing us to get less replicas than expected, the function is stuck due to the expectation so it can't perform a full sync to schedule the new pods. There's also not much to trigger a new sync once the expectation has expired until a new node comes up or one goes down. Environment: Kubernetes version (use ):Cloud provider or hardware configuration: AWS OS (e.g: ):Kernel (e.g. ):
/sig apps
I investigated this as much as I could without getting some external input, it feels like it's some kind of race condition in the informer cache. If anyone has ideas of if places to dig deeper that would be handy.
Reiterating the mentions to trigger a notification: <detailsIn response to : I investigated this as much as I could without getting some external input, it feels like it's some kind of race condition in the informer cache. If anyone has ideas of if places to dig deeper that would be handy.


Question: addNodes(manager.nodeStore, 0, 1, nil) addPods(manager.podStore, "node-0", simpleDaemonSetLabel, ds, 1) addPods(manager.podStore, "node-1", simpleDaemonSetLabel, ds, 1) <ins> podScheduledUsingAffinity := newPod("pod1-node-3", "", simpleDaemonSetLabel, ds) podScheduledUsingAffinity.Spec.Affinity = &v1.Affinity{ NodeAffinity: &v1.NodeAffinity{ RequiredDuringSchedulingIgnoredDuringExecution: &v1.NodeSelector{ NodeSelectorTerms: []v1.NodeSelectorTerm{ { MatchFields: []v1.NodeSelectorRequirement{ { Key:      schedulerapi.NodeFieldSelectorKeyNodeName, Operator: v1.NodeSelectorOpIn, Values:   []string{"node-2"}, }, }, }, }, }, }, } manager.podStore.Add(podScheduledUsingAffinity) </ins> if f { syncAndValidateDaemonSets(t, manager, ds, podControl, 0, 1, 0) } else {

Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This problem is definitely related to node termination, it's trying to delete the already deleted pod from the node that's been terminated. The problem seems to go away if is set, I think although this kind of race condition may have always been possible, it's being frequently caused via Which is finding the non-existent pod in the pod cache and seeing that it is assigned to a non-running node, resulting in a request to delete the non-existent pod. cc
Reiterating the mentions to trigger a notification: <detailsIn response to : This problem is definitely related to node termination, it's trying to delete the already deleted pod from the node that's been terminated. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign could you please take a look at this? It seems that it is related to the code you in .
Thanks for reporting the bug. Bug was introduced in which deleting pods without nodes. There is race condition between removing pod from cache and listing pods for DaemonSet. Bug occurs during following scenario: is deleted. for node is deleted by PodGCController. handler is executed by informer () set is to queue by delete handler () is processed and pods are taken from cache before deleted pod is removed. for deleted pod is not found so we add deleted pod to podsToDelete list (). are set to del:1() succeeds as pod is not found(). are never met so we need to wait till expectations expire. As fix I will send a change to so only pods that can't be deleted by PodGCController are deleted by DaemonSet controller.


Question: ID:               volumeV3.ID, Name:             volumeV3.Name, Status:           volumeV3.Status, <ins> Size:             volumeV3.Size, </ins> } if len(volumeV3.Attachments) > 0 {

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: patched out the Size field (was PR ), it will cause issue again. See also and . What you expected to happen: PVC resizes of Cinder volumes successfully. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/kind bug /sig storage /sig openstack
/assign


Question: // +build !ignore_autogenerated /* <del> Copyright 2016 The Kubernetes Authors. </del> <ins> Copyright 2017 The Kubernetes Authors. </ins> Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License.

What happened: Latest kubelet + Windows Docker version 1.14.0-dev refused to start any containers, error output: "invalid option: Windows does not support MemorySwap" How to reproduce it (as minimally and precisely as possible): It's always reproducible with Windows Docker 1.14.0-dev + latest kubelet code base. Anything else do we need to know: It didn't happen for Windows Docker 1.12.2-cs2-ws-beta. The reason behind the issue was because new code was for Windows Docker to check Resources parameters, it doesn't accept a non-zero value for MemorySwap: But kubelet was default to set an unlimited (-1) value. The issue was gone in my setup after applied a private fix to disable MemorySwap by set it to zero for Windows.


Question: <ins> apiVersion: v1beta3 kind: Namespace metadata: name: quota-example </ins>

Resource quota example is missing readme. Was it once there and was accidentally deleted?
I wrote something up in Origin repo I need to repurpose for Kubernetes. Sent from my iPhone


Question: func TestPriorityQueue_AddIfNotPresent(t *testing.T) { q := NewPriorityQueue(nil) <del> q.unschedulableQ.addOrUpdate(&highPriNominatedPod) </del> <ins> addOrUpdateUnschedulablePod(q, &highPriNominatedPod) </ins> q.AddIfNotPresent(&highPriNominatedPod) // Must not add anything. q.AddIfNotPresent(&medPriorityPod) q.AddIfNotPresent(&unschedulablePod)

Which jobs are failing: pull-kubernetes-bazel-test Which test(s) are failing: bazel test //pkg/scheduler/internal/queue:godefaulttest Since when has it been failing: Testgrid link: Reason for failure: WARNING: DATA RACE Read at 0x00c000230db0 by goroutine 47: runtime.mapiterinit() +0x0 (PriorityQueue).flushUnschedulableQLeftover() +0x17a (PriorityQueue).flushUnschedulableQLeftover-fm() +0x41 () +0x61 () +0xce () +0x5a Previous write at 0x00c000230db0 by goroutine 45: runtime.mapassignfaststr() +0x0 (UnschedulablePodsMap).addOrUpdate() +0x30e () +0x601 testing.tRunner() +0x162 Goroutine 47 (running) created at: (PriorityQueue).run() +0x172 () +0xe0b () +0x50 () +0x5a5 testing.tRunner() +0x162 Goroutine 45 (running) created at: testing.(T).Run() +0x659 testing.runTests.func1() +0xa8 testing.tRunner() +0x162 testing.runTests() +0x4ee testing.(M).Run() +0x2ee () bazel-out/k8- +0x2fb Anything else we need to know: I'm very sorry about that. UnschedulablePodsMap is not thread-safe. In , lock will be acquired before operating UnschedulablePodsMap. But in , lock isn‘t be acquired. In the previous case, only a single thread runs so it's ok. But after add goroutine , there will be data race. So I need to acquire lock before operating UnschedulablePodsMap.
/sig testing /sig scheduling


Question: }, }, { <ins> name: "UnschedulableAndUnschedulableFilters", plugins: []*TestPlugin{ { name: "TestPlugin1", inj:  injectedResult{FilterStatus: int(framework.Unschedulable)}, }, { name: "TestPlugin2", inj:  injectedResult{FilterStatus: int(framework.Unschedulable)}, }, }, wantStatus: framework.NewStatus(framework.Unschedulable, injectFilterReason).WithFailedPlugin("TestPlugin1"), wantStatusMap: framework.PluginToStatus{ "TestPlugin1": framework.NewStatus(framework.Unschedulable, injectFilterReason).WithFailedPlugin("TestPlugin1"), }, }, { </ins> name: "SuccessAndSuccessFilters", plugins: []*TestPlugin{ {

changed the default behavior of scheduler framework's - from stopping at the first failed filter to always running all filters. This change breaks Cluster Autoscaler at scale. Cluster Autoscaler imports the scheduler framework to simulate filters on artificial nodes. In a (pretty common) one-pod-per-node scenario, the number of calls is ~quadratic wrt. the number of nodes needed for unschedulable pods - it can be a lot of calls. Before , in such scenario the vast majority of calls would stop at failing Fit or NodeAffinity filters, and not run the rest of them (e.g. PV-related filters). After the change, every call to runs all filters, which completely breaks Cluster Autoscaler at scale (an example scenario of 300 1-pod-per-node pods and 200 node groups takes 10min in 1.24, and ~50s in <1.22). Can we get the old behavior back? Or re-introduce the config option so that Cluster Autoscaler can set it accordingly? Ideally, we'd also like to get this backported as far as possible (the change was introduced in 1.24).
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scheduling
This seems to be a regression. I would be interested in adding a toggle flag that allows one to toggle on that behavior if this was done on purpose.
That PR wasn't supposed to change the behavior actually. Scheduler is always defaulted to stop upon the first Filter failure, and it's just we offered an option to continue go through other Filter plugins upon failures, and that option was removed recently. Let me check the code to see if there is a regression.
When removed the following changes: It should have kept . I will submit a fix along with test, and backport to 1.24, 1.25, 1.26. Thanks for reporting!


Question: }, }, { <ins> name: "UnschedulableAndUnschedulableFilters", plugins: []*TestPlugin{ { name: "TestPlugin1", inj:  injectedResult{FilterStatus: int(framework.Unschedulable)}, }, { name: "TestPlugin2", inj:  injectedResult{FilterStatus: int(framework.Unschedulable)}, }, }, wantStatus: framework.NewStatus(framework.Unschedulable, injectFilterReason).WithFailedPlugin("TestPlugin1"), wantStatusMap: framework.PluginToStatus{ "TestPlugin1": framework.NewStatus(framework.Unschedulable, injectFilterReason).WithFailedPlugin("TestPlugin1"), }, }, { </ins> name: "SuccessAndSuccessFilters", plugins: []*TestPlugin{ {

cc /assign /kind bug /kind regression /priority critical-urgent /triage/accepted
Thanks Kuba, this is indeed a regression on the scheduler end.
/open for patches
/re-open /reopen
Reopened this issue. <detailsIn response to : /re-open /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Thanks for the reporting, and thanks for the fix.
Really sorry about this regression, and hope not to happen again. I think there's no e2e tests between kubernetes scheduler and cluster-autoscaler, so can we also have testcases of pods unschedulable in scheduling perf-test, maybe this can also help to prevent this happening again. UT benchmark test also makes sense, but it's not graphical.
Thanks for the quick fix! Unfortunately I'm afraid this kind of issue would only be caught in a large-scale test scenario, which is not cheap to run continuously.
For this particular issue, UT ( in the PR) is able to catch it. And hence doesn't need to be that complicated to test with cluster-autoscaler. However, standing from a broader angle, why we didn't catch the regression in perf test is b/c the triggering actor is unschedulable Pods, instead of schedulable Pods (which are the happy path most perf tests covered). So going forward, I'd like to see more perf tests to mixin unschedulable Pods, and hence more likely to surface this kind of issue. (we have preemptio preemption test suite but doesn't seem adequate)
This is not for the particular issue, but keeping an eye on whether there's any performance degradation between two kubernetes components, like we do for scheduling performance. but it's actually out of the scope of scheduler. I'll add it as one of the goals of
Actually, the schedulerperf has a test case . And, the schedulerperf collects the metric (), which has the label in it (). But, the problem is currently we don't distinguish the metric values by label on each data; we just collect/aggregate it regardless the label.


Question: }, }, { <ins> name: "UnschedulableAndUnschedulableFilters", plugins: []*TestPlugin{ { name: "TestPlugin1", inj:  injectedResult{FilterStatus: int(framework.Unschedulable)}, }, { name: "TestPlugin2", inj:  injectedResult{FilterStatus: int(framework.Unschedulable)}, }, }, wantStatus: framework.NewStatus(framework.Unschedulable, injectFilterReason).WithFailedPlugin("TestPlugin1"), wantStatusMap: framework.PluginToStatus{ "TestPlugin1": framework.NewStatus(framework.Unschedulable, injectFilterReason).WithFailedPlugin("TestPlugin1"), }, }, { </ins> name: "SuccessAndSuccessFilters", plugins: []*TestPlugin{ {

(We can distinguish the metric values by any labels in the schedulerperf as we do in the metric ()) I'd say just distinguishing the happy path and unhappy path with label in metric should be enough to observe the particular performance degradation in the unhappy path.
could you help share the perf-dash link where this metric () is exposed?
Hm, cannot see any scheduler metrics result from the current perf-dash I see some configuration that seems to be to parse the schedulerperf result file though. https://perf- TBH, I haven't visited the perf-dash so much, so not sure whether it's not working for a long time, or got broken recently. Also, another thing, I noticed the perf-dash doesn't show all test cases. All test cases after haven't been executed.
So... probably, something wrong happens in  job (probably in the test case) and the schedulerperf is failing. The schedulerperf doesn't make the result file when fails, so I'm guessing that's why we cannot see any scheduler metrics results from the perf-dash. Okay, I found it. Let me open another issue for this.
cc is this actually a regression, and is it resolved yet?
It's resolved by , and also cherry-picked to 1.24.x ~ 1.26.x. /close
Closing this issue. <detailsIn response to : It's resolved by , and also cherry-picked to 1.24.x ~ 1.26.x. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: } }() err = r.cmd.Wait() <del> stopped <- true </del> <ins> close(stopped) </ins> if exiterr, ok := err.(*exec.ExitError); ok { klog.Infof("etcd server stopped (signal: %s)", exiterr.Error()) // stopped

It's at line 111 When the server stopped before timeout, the first goroutine will be block on because the receiver in the second goroutine is not exist. Changing to a buffered one may fix it. no goroutine leaks run related test with leak checker No response <detailsdev </details<detailsunknown </details<details</details<details</details<details</details<details</details
/sig api-machinery
/assign /triage accepted
/cc
It should close the channel rather than write to it, that's the easiest fix


Question: - mkdir -p /home/kubernetes/containerized_mounter/rootfs - mount --bind /home/kubernetes/containerized_mounter/ /home/kubernetes/containerized_mounter/ - mount -o remount, exec /home/kubernetes/containerized_mounter/ <del> - wget https://storage.googleapis.com/kubernetes-release/gci-mounter/mounter.tar -O /tmp/mounter.tar </del> <ins> - wget https://dl.k8s.io/gci-mounter/mounter.tar -O /tmp/mounter.tar </ins> - tar xvf /tmp/mounter.tar -C /home/kubernetes/containerized_mounter/rootfs - mkdir -p /home/kubernetes/containerized_mounter/rootfs/var/lib/kubelet - mount --rbind /var/lib/kubelet /home/kubernetes/containerized_mounter/rootfs/var/lib/kubelet

We have a new vanity url, , which redirects to the GCS release bucket - e.g. Right now, the https endpoint doesn't work, since isn't in the list of domains on the certificate. Once it works, we should switch all references over to the nicer vanity URL. x-ref
So what it required to make this issues actionable? Where to add this for the certificate?
the issue tracking that is
This needs to be triaged as a release-blocker or not for 1.5
not a release blocker.
Is it appropriate to move this to the next milestone or clear the 1.5 milestone? (and remove the non-release-blocker tag as well)
can this be closed?
I forgot to get to this in 1.6, but it'd still be good to do.
do you want this in v1.7? is there a sig to own this?
doesn't need to block 1.7, no.
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list .
/sig release