Question: serviceHasSynced cache.InformerSynced // nodeLister knows how to list nodes nodeLister corelisters.NodeLister <ins> // nodeHasSynced indicates whether nodes have been sync'd at least once. // Check this before trusting a response from the node lister. nodeHasSynced cache.InformerSynced </ins> // a list of node labels to register nodeLabels map[string]string

What happened: A deployment is created that uses a after the cluster master is created but before the cluster worker nodes are fully initialized. During the initialization of the worker nodes, the nodes are temporarily available for scheduling without the necessary label to match the deployment's node selector. Depending on how long the nodes are available for scheduling without the necessary node labels for the deployment, the scheduler will start to spam with cluster with pods in status. This spamming stops once the worker nodes are fully initialized and the pods are scheduled successful. The exact timing of all of this is TBD. What you expected to happen: Pods in status are annoying and require manual cleanup. It would be preferred that the scheduler avoid this an simply issue warning events. How to reproduce it (as minimally and precisely as possible): See issue description. Anything else we need to know?: No. Environment: Kubernetes version (use ): 1.18.3 Cloud provider or hardware configuration: IBM Cloud Kubernetes Service OS (e.g: ): Ubuntu 18.04.4 LTS Kernel (e.g. ): 4.15.0-96-generic Install tools: Managed service install Network plugin and version (if this is a network-related bug): N/A Others: N/A
Related to and
/sig scheduling
by spamming are you referring to scheduler logs? also, why are the nodes not labelled from the get go?
Just to confirm, all the pods in question are still in Pending state, yes? If the pods are in pending state, all of them will get scheduled eventually once the nodes get their labels.
The pods are not in state. They are actually in state. The pods eventually get scheduled but we are left with anywhere from a few to dozens of these state pods. Also, we are looking at labeling the nodes via kubelet flags to see if that helps.
/remove-kind bug
After we add some retry logic and wait time in this problem can be solved: As you can see this function returns initial node as the first request failed.
I also get hit by this, when single controller node gets rebooted and scheduler starts as static container before kubelet becomes ready.


Question: serviceHasSynced cache.InformerSynced // nodeLister knows how to list nodes nodeLister corelisters.NodeLister <ins> // nodeHasSynced indicates whether nodes have been sync'd at least once. // Check this before trusting a response from the node lister. nodeHasSynced cache.InformerSynced </ins> // a list of node labels to register nodeLabels map[string]string

Can be cleaned up with:
/sig node
the spam cycle is this loop: "pod create, pod schedule, worker reject, pod failed" -- we don't have a lot of protection in higher level workload controllers to handle pods that fail at kubelet admission time versus pods that just dont create due to quota/admission. in general, is the spam causing any other stability issues? is pod gc pruning the pods?
I wonder if we can do a trick in node lister to mitigate any race condition here...
we could maybe do something similar to to ensure node lister has synced...
hacking on option to ensure if kubelet has a node lister with a valid kube client, we wait for it to sync at least once here which should mitigate this issue. see:
We noticed the problem on the OLM pod which is managed by the OLM operator. Right or wrong, the operator does not like seeing the pod go to status and no longer reconciles the pod. The end result is that manual intervention (i.e. pod deletion) is required to fix the problem.
sure... i would have thought the pod is failed in that instance and reached a terminal state. either way, i am looking at what i can do to mitigate this.
The problem persists in our environment as well, and the workaround cited above will not do anymore. Is there something in the works for this one?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


Question: echo "Prepare kube-addons manifests and start kube addon manager" local -r src_dir="${KUBE_HOME}/kube-manifests/kubernetes/gci-trusty" local -r dst_dir="/etc/kubernetes/addons" <ins> # prep the additional bindings that are particular to e2e users and groups setup-addon-manifests "addons" "e2e-rbac-bindings" </ins> # Set up manifests of other addons. if [[ "${ENABLE_CLUSTER_MONITORING:-}" == "influxdb" ]] ||  [[ "${ENABLE_CLUSTER_MONITORING:-}" == "google" ]] ||

Typically, the dns and dashboard is in crash loop. Ref cc
if you check the file, does it contain lines like ?
Previous comment is ref: from I'll copy that to container-linux as well.


Question: "context" "encoding/json" "fmt" <ins> "mime" </ins> "net/http" "net/url" "sort"

We are using (and ) to interact with Kubernetes clusters behind a couple of proxy layers. One of these proxies completes the headers to include character encoding details. This was working fine until recently, when client-go/kubectl have started to fail when interacting with the discovery API. It seems that a recent change has made kubectl more restrictive of the content-type that it expects. I suspect that it is related to this commit: For example, this is an error from a kubectl command: We can of course try to fix it on our end, removing that charset information. However, it is a valid content-type and the way kubectl is handling the content-type there seems like it is relying on implementation details of kube-apiserver (that it does not add metadata to the content-type). I propose to fix this issue in , by properly parsing and handling the content-type.
cc /kind regression /kind bug /sig api-machinery /priority important-soon looks like the switch statement in downloadLegacy() needs to be more tolerant of the variety of content-types it can get from a server (including text/plain or no content-type if old servers could have returned that...)
/transfer kubernetes/kubernetes
Something went wrong or the destination repo kubernetes/kubernetes/kubernetes does not exist. <detailsIn response to : /transfer kubernetes/kubernetes Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/transfer kubernetes
Yep, the string comparison for the content-type is flimsy, let's address that. Thanks for the report!
/assign
/triage accepted
/cc
The fix is being cherry-picked back to: : :


Question: echo ${short_hash:0:5} } <ins> # Pedantically kill, wait-on and remove a container. The -f -v options # to rm don't actually seem to get the job done, so force kill the # container, wait to ensure it's stopped, then try the remove. This is # a workaround for bug https://github.com/docker/docker/issues/3968. function kube::build::destroy_container() { "${DOCKER[@]}" kill "$1" >/dev/null 2>&1 || true "${DOCKER[@]}" wait "$1" >/dev/null 2>&1 || true "${DOCKER[@]}" rm -f -v "$1" >/dev/null 2>&1 || true } </ins> # --------------------------------------------------------------------------- # Building

Github automatically adds a artifact to every release, which sounds very similar to the artifact that we generate as the release artifact. Assuming we don't have control over the naming or presence of the autogenerated source code artifact, I think we should consider renaming our published artifact to better disambiguate between the two.
We distribute most of our artifacts outside of GitHub. We should probably stop uploading this one entirely.
cc
We've got a thread going on w/ Tim. /assign
can you please link so folks can follow the bread crumbs? ( cc )
-- It's currently sensitive. Will share once there's an outcome.
ACK. thanks for the heads up
I think it's safe to discuss openly at this point. We had a report that the sha512 hashes on our release artifacts were incorrect. We have confirmed that the mismatch was due to a bug in the release pipeline, and not malicious. This issue here was just a point of confusion that came out of that discussion.
thanks !
/assign
Going one step back since I think the exists for historical reasons: The client and server binaries were part of this bundle, which is not the case any more: So it seems that is mostly a collection of licenses and scripts. Do we really need it or may we cleanup some code paths and remove it completely?
+1 to put the word out and remove it completely if no one depends on it.
I'm not confident that CI is not using these versioned scripts, in fact i'm fairly sure that it is. However it would not be pulling it from github.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale I think this remains as relevant as ever.
/milestone v1.23
/triage accepted
Hi and k8s 1.23 bug triage shadow here.


Question: echo ${short_hash:0:5} } <ins> # Pedantically kill, wait-on and remove a container. The -f -v options # to rm don't actually seem to get the job done, so force kill the # container, wait to ensure it's stopped, then try the remove. This is # a workaround for bug https://github.com/docker/docker/issues/3968. function kube::build::destroy_container() { "${DOCKER[@]}" kill "$1" >/dev/null 2>&1 || true "${DOCKER[@]}" wait "$1" >/dev/null 2>&1 || true "${DOCKER[@]}" rm -f -v "$1" >/dev/null 2>&1 || true } </ins> # --------------------------------------------------------------------------- # Building

There hasn't been any activity or linked PRs for this issue. Just wanted to check in to see if this is still on track for 1.23.
/milestone v1.24
Hi I am part of the bug triage team for v1.24. We noticed that this issue hasn't been updated for a while. Just wanted to check if there is any work planned that would go into this.
this one keeps moving down the milestones. i don't see any activity since november. So kick the can down the road /milestone v1.25
Hi My name Hossein Salahi from Bug-Triage team. I am just checking to ensure if everything is on track for K8s 1.25. Thanks!
That's a question for or
any remarks on that?
the release 1.25 bug triage Lead. I'm reaching out to see the status of this issue and if we are still targeting the current release. Also, kindly if you can update the priority so we can track it in the future.
this one keeps moving down the milestones yet again!. i don't see any activity since i moved this down the line. So kick the can down the road: /milestone clear (instead of v1.26 as this has been moving down the line for a few releases now)
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
What's the current status of this issue? Is there anything that we can do to move this forward?
We got new conversations going on during Kubecon on whether we should publish our release binaries on GitHub. Given that and I think we should rethink now what we publish on the release pages. I think we should start publishing all binaries for one. It is a common practice everywhere and I think users largely expect them to be there.


Question: echo ${short_hash:0:5} } <ins> # Pedantically kill, wait-on and remove a container. The -f -v options # to rm don't actually seem to get the job done, so force kill the # container, wait to ensure it's stopped, then try the remove. This is # a workaround for bug https://github.com/docker/docker/issues/3968. function kube::build::destroy_container() { "${DOCKER[@]}" kill "$1" >/dev/null 2>&1 || true "${DOCKER[@]}" wait "$1" >/dev/null 2>&1 || true "${DOCKER[@]}" rm -f -v "$1" >/dev/null 2>&1 || true } </ins> # --------------------------------------------------------------------------- # Building

let's scope the issue to the actual "Kubernetes Source Code" tarball. We can still remove it from what I can see from the discussion.
I agree with Removing the "Kubernetes Source Code" tarball should be as easy as removing this from krel: However, there are a few questions: is it safe to do so? I believe yes, because this "Kubernetes Source Code" tarball is pointing to the GCS bucket, and we'll still continue publishing that tarball to the bucket The link is also available in the changelog, so it's easy to find do we need any deprecation period? Considering the above two points, I don't think so. If you folks agree with that, I can go ahead and create a PR to update krel.
I agree on that, let's drop an additional note on k-dev that others are aware of the change.
/assign
Update: An announcement to proceed with the removal of the "Kubernetes Source Code" artifact has been sent to k-dev: PR to remove this artifact from being published: Removal will go into effect starting the January patch releases.


Question: "k8s.io/client-go/tools/cache" "k8s.io/client-go/tools/record" cloudprovider "k8s.io/cloud-provider" <ins> csilibplugins "k8s.io/csi-translation-lib/plugins" </ins> proxyutil "k8s.io/kubernetes/pkg/proxy/util" . "k8s.io/kubernetes/pkg/volume" "k8s.io/kubernetes/pkg/volume/util/hostutil"

After the cluster was updated to enable PD CSI Driver, there was a race condition where AD Controller (kube-controller-manager) started to BEFORE the PD CSI Driver was up, thus AD Controller gave us the error Then AD Controller triggered the disk detach because the disk/volume was not populated in the desired state of world. Unfortunately the disk was in use during this detach and got corrupted. Since cluster upgrades can cause the controller manager and CSI drivers to start at different times, the AD controller needs to be more conservative when detaching We have not been able to reproduce this, it came up from a customer. /sig storage /assign <detailsk8s 1.23, CSI migration enabled </details<detailsGKE </details<details</details<details</details<details</details<details</details
/triage accepted
I think there's potentially two changes we should do: 1) for the initial DSW population, we should skip the check that if the csi driver is not installed. 2) in general, we should not skip volumes during dsw initial population if an error is returned. If we had some temporary api server error then that would cause volumes to not get to dsw and similar behavior happen. cc
for a repro, maybe you can try something like: a cluster with csi migration enabled (control plane + nodes) a pod with intree-volume and get it running the csi driver node daemonset. verify the driver is removed from CSINode kube-controller-manager
cc
/cc was trying to repro this, but on GKE with the managed driver which made it complicated.
Yeah maybe gce kube-up would be easier for a repro
If CSINode says that it has migrated GCP PD volume plugin to CSI, then IMO A/D controller should always attach GCP PD volumes to it using CSI, regardless if the CSI driver is present or not. In-tree volume plugin in the kubelet is disabled and can't mount any in-tree volumes anyway. To flip the migration feature gate / upgrade / downgrade kubelet, the cluster admin must drain the node first to make sure everything is detached. A/D controller should not see change of CSINode migration annotations with any volume attached to it.
/priority important-soon
Another option could be to process these volumes anyways but add them to DSOW with migration status as "uncertain", so as they can be re-processed later.


Question: "k8s.io/client-go/tools/cache" "k8s.io/client-go/tools/record" cloudprovider "k8s.io/cloud-provider" <ins> csilibplugins "k8s.io/csi-translation-lib/plugins" </ins> proxyutil "k8s.io/kubernetes/pkg/proxy/util" . "k8s.io/kubernetes/pkg/volume" "k8s.io/kubernetes/pkg/volume/util/hostutil"

But surely it looks like might be easiest unless it has any downsides.
There wasn't a node update in this case---there was a control plane update which restarted the PD CSI driver, and it looks like the CSIDriver resource was updated or changed in some way which raced with the kube-controller-manager restart.
Is there anyway we can know if the csi driver is up and running(like a health check ping) before populating DSW?
We use the CSINode registration to determine that. The discussion here I think is concluding that we should not use that as a criteria for adding the volume to DSW. We only need to check that CSI migration is enabled on the node.
+1 we need to be very resilient against races, I think we have to assume a driver is in process of being installed at any point in the AD controller reconcilliation.
/assign sunnylovestiramisu
Reproduced with the steps: In /var/log/kube-controller-
Tested with the and doing the reproduce steps above. The error of does not show up anymore. The error message still get logged in the kube-controller- for debugging purpose: Ran Ran On the GCP console, the disk is used by node: Ran I redeploy the pd-csi-driver and then make sure it is up and running. Ran


Question: // ResourceConfiguration stores per resource configuration. type ResourceConfiguration struct { <del> // resources is a list of kubernetes resources which have to be encrypted. </del> <ins> // resources is a list of kubernetes resources which have to be encrypted. The resource names are derived from `resource` or `resource.group` of the group/version/resource. // eg: pandas.awesome.bears.example is a custom resource with 'group': awesome.bears.example, 'resource': pandas) </ins> Resources []string // providers is a list of transformers to be used for reading and writing the resources to disk. // eg: aesgcm, aescbc, secretbox, identity.

Update the godoc on the encryption config API on how to specify group/resources to be encrypted ref: godoc has the details na No response_ na na <details</details<details</details<details</details<details</details
/sig auth /triage accepted
/assign Cool-Coder174
Thanks for volunteering for this LMK if you need any help with it.
Just for clarification, I just need to add documentation on how to configure encryption on groups/resources?
You can update godoc to mention that custom resources can now be to the list of to be encrypted as well and how to add the of the resource to the list. e.g. from this test
/assign /unassign Release v1.26 deadlines are here so we need to get this completed ASAP.
/milestone v1.26
I apologize for the slow actions. I know time and contributions are valuable. May I request a walk through. This is my first time ever contributing and I really want to learn. I have been reading some of the code base on my spare time.
this feature is on the harder side of things to get started with. Reading the blog post and the associated documentation is probably the best starting point:
& what is the status on this issue? 1.26 is coming up! cc
/milestone v1.27


Question: registry = "k8s.gcr.io/build-image", repository = "debian-base", # Ensure the digests above are updated to match a new tag <del> tag = "buster-v1.3.0",  # ignored, but kept here for documentation </del> <ins> tag = "buster-v1.4.0",  # ignored, but kept here for documentation </ins> ) container_pull(

The based container images for are shipped with the wrong architecture within the manifest if not amd64: Refers to We updated the images a few minutes ago to buster-v1.3.0, which are now ready to be used as base for the multi-architecture images. If the images contain the right architectures, then it may magically happen that the kube-proxy images are fixed, too. This is because we already use the build syntax for it: /cc
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig release


Question: <ins> /* Copyright 2020 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package network import ( "fmt" "strings" "time" "github.com/onsi/ginkgo" v1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/util/intstr" clientset "k8s.io/client-go/kubernetes" "k8s.io/kubernetes/test/e2e/framework" e2enode "k8s.io/kubernetes/test/e2e/framework/node" e2epod "k8s.io/kubernetes/test/e2e/framework/pod" e2eservice "k8s.io/kubernetes/test/e2e/framework/service" e2eskipper "k8s.io/kubernetes/test/e2e/framework/skipper" ) const ( serviceName = "svc-udp" podClient   = "pod-client" podBackend1 = "pod-server-1" podBackend2 = "pod-server-2" srcPort     = 12345 ) // Linux NAT uses conntrack to perform NAT, everytime a new // flow is seen, a connection is created in the conntrack table, and it // is being used by the NAT module. // Each entry in the conntrack table has associated a timeout, that removes // the connection once it expires. // UDP is a connectionless protocol, so the conntrack module tracking functions // are not very advanced. // It uses a short timeout (30 sec by default) that is renewed if there are new flows // matching the connection. Otherwise it expires the entry. // This behaviour can cause issues in Kubernetes when one entry on the conntrack table // is never expired because the sender does not stop sending traffic, but the pods or // endpoints were deleted, blackholing the traffic // In order to mitigate this problem, Kubernetes delete the stale entries: // - when an endpoint is removed // - when a service goes from no endpoints to new endpoint // Ref: https://api.semanticscholar.org/CorpusID:198903401 // Boye, Magnus. “Netfilter Connection Tracking and NAT Implementation.” (2012). var _ = SIGDescribe("Conntrack", func() { fr := framework.NewDefaultFramework("conntrack") type nodeInfo struct { name   string nodeIP string } var ( cs                             clientset.Interface ns                             string clientNodeInfo, serverNodeInfo nodeInfo ) ginkgo.BeforeEach(func() { cs = fr.ClientSet ns = fr.Namespace.Name nodes, err := e2enode.GetBoundedReadySchedulableNodes(cs, 2) framework.ExpectNoError(err) if len(nodes.Items) < 2 { e2eskipper.Skipf( "Test requires >= 2 Ready nodes, but there are only %v nodes", len(nodes.Items)) } ips := e2enode.CollectAddresses(nodes, v1.NodeInternalIP) clientNodeInfo = nodeInfo{ name:   nodes.Items[0].Name, nodeIP: ips[0], } serverNodeInfo = nodeInfo{ name:   nodes.Items[1].Name, nodeIP: ips[1], } }) ginkgo.It("should be able to preserve UDP traffic when server pod cycles for a NodePort service", func() { // Create a NodePort service udpJig := e2eservice.NewTestJig(cs, ns, serviceName) ginkgo.By("creating a UDP service " + serviceName + " with type=NodePort in " + ns) udpService, err := udpJig.CreateUDPService(func(svc *v1.Service) { svc.Spec.Type = v1.ServiceTypeNodePort svc.Spec.Ports = []v1.ServicePort{ {Port: 80, Name: "udp", Protocol: v1.ProtocolUDP, TargetPort: intstr.FromInt(80)}, } }) framework.ExpectNoError(err) // Create a pod in one node to create the UDP traffic against the NodePort service every 5 seconds ginkgo.By("creating a client pod for probing the service " + serviceName) clientPod := newAgnhostPod(podClient, "") clientPod.Spec.NodeName = clientNodeInfo.name cmd := fmt.Sprintf(`date; for i in $(seq 1 300); do echo "$(date) Try: ${i}"; echo hostname | nc -u -w 5 -p %d %s %d; echo; done`, srcPort, serverNodeInfo.nodeIP, udpService.Spec.Ports[0].NodePort) clientPod.Spec.Containers[0].Command = []string{"/bin/sh", "-c", cmd} clientPod.Spec.Containers[0].Name = podClient fr.PodClient().CreateSync(clientPod) // Read the client pod logs logs, err := e2epod.GetPodLogs(cs, ns, podClient, podClient) framework.ExpectNoError(err) framework.Logf("Pod client logs: %s", logs) // Add a backend pod to the service in the other node ginkgo.By("creating a backend pod " + podBackend1 + " for the service " + serviceName) serverPod1 := newAgnhostPod(podBackend1, "netexec", fmt.Sprintf("--udp-port=%d", 80)) serverPod1.Labels = udpJig.Labels serverPod1.Spec.NodeName = serverNodeInfo.name fr.PodClient().CreateSync(serverPod1) // Waiting for service to expose endpoint. err = validateEndpointsPorts(cs, ns, serviceName, portsByPodName{podBackend1: {80}}) framework.ExpectNoError(err, "failed to validate endpoints for service %s in namespace: %s", serviceName, ns) // Check that the pod receives the traffic // UDP conntrack entries timeout is 30 sec by default ginkgo.By("checking client pod connected to the backend 1 on Node IP " + serverNodeInfo.nodeIP) time.Sleep(30 * time.Second) logs, err = e2epod.GetPodLogs(cs, ns, podClient, podClient) framework.ExpectNoError(err) framework.Logf("Pod client logs: %s", logs) if !strings.Contains(string(logs), podBackend1) { framework.Failf("Failed to connecto to backend 1") } // Create a second pod ginkgo.By("creating a second backend pod " + podBackend2 + " for the service " + serviceName) serverPod2 := newAgnhostPod(podBackend2, "netexec", fmt.Sprintf("--udp-port=%d", 80)) serverPod2.Labels = udpJig.Labels serverPod2.Spec.NodeName = serverNodeInfo.name fr.PodClient().CreateSync(serverPod2) // and delete the first pod framework.Logf("Cleaning up %s pod", podBackend1) fr.PodClient().DeleteSync(podBackend1, metav1.DeleteOptions{}, framework.DefaultPodDeletionTimeout) // Check that the second pod keeps receiving traffic // UDP conntrack entries timeout is 30 sec by default ginkgo.By("checking client pod connected to the backend 2 on Node IP " + serverNodeInfo.nodeIP) time.Sleep(30 * time.Second) logs, err = e2epod.GetPodLogs(cs, ns, podClient, podClient) framework.ExpectNoError(err) framework.Logf("Pod client logs: %s", logs) if !strings.Contains(string(logs), podBackend2) { framework.Failf("Failed to connecto to backend 2") } }) ginkgo.It("should be able to preserve UDP traffic when server pod cycles for a ClusterIP service", func() { // Create a NodePort service udpJig := e2eservice.NewTestJig(cs, ns, serviceName) ginkgo.By("creating a UDP service " + serviceName + " with type=ClusterIP in " + ns) udpService, err := udpJig.CreateUDPService(func(svc *v1.Service) { svc.Spec.Type = v1.ServiceTypeClusterIP svc.Spec.Ports = []v1.ServicePort{ {Port: 80, Name: "udp", Protocol: v1.ProtocolUDP, TargetPort: intstr.FromInt(80)}, } }) framework.ExpectNoError(err) // Create a pod in one node to create the UDP traffic against the NodePort service every 5 seconds ginkgo.By("creating a client pod for probing the service " + serviceName) clientPod := newAgnhostPod(podClient, "") clientPod.Spec.NodeName = clientNodeInfo.name cmd := fmt.Sprintf(`date; for i in $(seq 1 300); do echo "$(date) Try: ${i}"; echo hostname | nc -u -w 5 -p %d %s %d; echo; done`, srcPort, udpService.Spec.ClusterIP, udpService.Spec.Ports[0].Port) clientPod.Spec.Containers[0].Command = []string{"/bin/sh", "-c", cmd} clientPod.Spec.Containers[0].Name = podClient fr.PodClient().CreateSync(clientPod) // Read the client pod logs logs, err := e2epod.GetPodLogs(cs, ns, podClient, podClient) framework.ExpectNoError(err) framework.Logf("Pod client logs: %s", logs) // Add a backend pod to the service in the other node ginkgo.By("creating a backend pod " + podBackend1 + " for the service " + serviceName) serverPod1 := newAgnhostPod(podBackend1, "netexec", fmt.Sprintf("--udp-port=%d", 80)) serverPod1.Labels = udpJig.Labels serverPod1.Spec.NodeName = serverNodeInfo.name fr.PodClient().CreateSync(serverPod1) // Waiting for service to expose endpoint. err = validateEndpointsPorts(cs, ns, serviceName, portsByPodName{podBackend1: {80}}) framework.ExpectNoError(err, "failed to validate endpoints for service %s in namespace: %s", serviceName, ns) // Check that the pod receives the traffic // UDP conntrack entries timeout is 30 sec by default ginkgo.By("checking client pod connected to the backend 1 on Node IP " + serverNodeInfo.nodeIP) time.Sleep(30 * time.Second) logs, err = e2epod.GetPodLogs(cs, ns, podClient, podClient) framework.ExpectNoError(err) framework.Logf("Pod client logs: %s", logs) if !strings.Contains(string(logs), podBackend1) { framework.Failf("Failed to connecto to backend 1") } // Create a second pod ginkgo.By("creating a second backend pod " + podBackend2 + " for the service " + serviceName) serverPod2 := newAgnhostPod(podBackend2, "netexec", fmt.Sprintf("--udp-port=%d", 80)) serverPod2.Labels = udpJig.Labels serverPod2.Spec.NodeName = serverNodeInfo.name fr.PodClient().CreateSync(serverPod2) // and delete the first pod framework.Logf("Cleaning up %s pod", podBackend1) fr.PodClient().DeleteSync(podBackend1, metav1.DeleteOptions{}, framework.DefaultPodDeletionTimeout) // Check that the second pod keeps receiving traffic // UDP conntrack entries timeout is 30 sec by default ginkgo.By("checking client pod connected to the backend 2 on Node IP " + serverNodeInfo.nodeIP) time.Sleep(30 * time.Second) logs, err = e2epod.GetPodLogs(cs, ns, podClient, podClient) framework.ExpectNoError(err) framework.Logf("Pod client logs: %s", logs) if !strings.Contains(string(logs), podBackend2) { framework.Failf("Failed to connecto to backend 2") } }) }) </ins>

Which jobs are flaking: ci-kubernetes-e2e-gce-scale-correctness Which test(s) are flaking: [sig-network] Services should be able to preserve UDP traffic when server pod cycles for a NodePort service Testgrid link: https://k8s- Reason for failure: /sig network /assign
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
CRI-O CI is also hitting this failure. We have been unable to reproduce locally. our CI runs on aws
hmm, I only see 1 failure out of 4 and seems that the connectivity didn't work at all
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
/assign
GitHub didn't allow me to assign the following users: JacobTanenbaum. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc Saw the test flaking recently: Wondering if this is just a timeout issue.


Question: <ins> /* Copyright 2020 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package network import ( "fmt" "strings" "time" "github.com/onsi/ginkgo" v1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/util/intstr" clientset "k8s.io/client-go/kubernetes" "k8s.io/kubernetes/test/e2e/framework" e2enode "k8s.io/kubernetes/test/e2e/framework/node" e2epod "k8s.io/kubernetes/test/e2e/framework/pod" e2eservice "k8s.io/kubernetes/test/e2e/framework/service" e2eskipper "k8s.io/kubernetes/test/e2e/framework/skipper" ) const ( serviceName = "svc-udp" podClient   = "pod-client" podBackend1 = "pod-server-1" podBackend2 = "pod-server-2" srcPort     = 12345 ) // Linux NAT uses conntrack to perform NAT, everytime a new // flow is seen, a connection is created in the conntrack table, and it // is being used by the NAT module. // Each entry in the conntrack table has associated a timeout, that removes // the connection once it expires. // UDP is a connectionless protocol, so the conntrack module tracking functions // are not very advanced. // It uses a short timeout (30 sec by default) that is renewed if there are new flows // matching the connection. Otherwise it expires the entry. // This behaviour can cause issues in Kubernetes when one entry on the conntrack table // is never expired because the sender does not stop sending traffic, but the pods or // endpoints were deleted, blackholing the traffic // In order to mitigate this problem, Kubernetes delete the stale entries: // - when an endpoint is removed // - when a service goes from no endpoints to new endpoint // Ref: https://api.semanticscholar.org/CorpusID:198903401 // Boye, Magnus. “Netfilter Connection Tracking and NAT Implementation.” (2012). var _ = SIGDescribe("Conntrack", func() { fr := framework.NewDefaultFramework("conntrack") type nodeInfo struct { name   string nodeIP string } var ( cs                             clientset.Interface ns                             string clientNodeInfo, serverNodeInfo nodeInfo ) ginkgo.BeforeEach(func() { cs = fr.ClientSet ns = fr.Namespace.Name nodes, err := e2enode.GetBoundedReadySchedulableNodes(cs, 2) framework.ExpectNoError(err) if len(nodes.Items) < 2 { e2eskipper.Skipf( "Test requires >= 2 Ready nodes, but there are only %v nodes", len(nodes.Items)) } ips := e2enode.CollectAddresses(nodes, v1.NodeInternalIP) clientNodeInfo = nodeInfo{ name:   nodes.Items[0].Name, nodeIP: ips[0], } serverNodeInfo = nodeInfo{ name:   nodes.Items[1].Name, nodeIP: ips[1], } }) ginkgo.It("should be able to preserve UDP traffic when server pod cycles for a NodePort service", func() { // Create a NodePort service udpJig := e2eservice.NewTestJig(cs, ns, serviceName) ginkgo.By("creating a UDP service " + serviceName + " with type=NodePort in " + ns) udpService, err := udpJig.CreateUDPService(func(svc *v1.Service) { svc.Spec.Type = v1.ServiceTypeNodePort svc.Spec.Ports = []v1.ServicePort{ {Port: 80, Name: "udp", Protocol: v1.ProtocolUDP, TargetPort: intstr.FromInt(80)}, } }) framework.ExpectNoError(err) // Create a pod in one node to create the UDP traffic against the NodePort service every 5 seconds ginkgo.By("creating a client pod for probing the service " + serviceName) clientPod := newAgnhostPod(podClient, "") clientPod.Spec.NodeName = clientNodeInfo.name cmd := fmt.Sprintf(`date; for i in $(seq 1 300); do echo "$(date) Try: ${i}"; echo hostname | nc -u -w 5 -p %d %s %d; echo; done`, srcPort, serverNodeInfo.nodeIP, udpService.Spec.Ports[0].NodePort) clientPod.Spec.Containers[0].Command = []string{"/bin/sh", "-c", cmd} clientPod.Spec.Containers[0].Name = podClient fr.PodClient().CreateSync(clientPod) // Read the client pod logs logs, err := e2epod.GetPodLogs(cs, ns, podClient, podClient) framework.ExpectNoError(err) framework.Logf("Pod client logs: %s", logs) // Add a backend pod to the service in the other node ginkgo.By("creating a backend pod " + podBackend1 + " for the service " + serviceName) serverPod1 := newAgnhostPod(podBackend1, "netexec", fmt.Sprintf("--udp-port=%d", 80)) serverPod1.Labels = udpJig.Labels serverPod1.Spec.NodeName = serverNodeInfo.name fr.PodClient().CreateSync(serverPod1) // Waiting for service to expose endpoint. err = validateEndpointsPorts(cs, ns, serviceName, portsByPodName{podBackend1: {80}}) framework.ExpectNoError(err, "failed to validate endpoints for service %s in namespace: %s", serviceName, ns) // Check that the pod receives the traffic // UDP conntrack entries timeout is 30 sec by default ginkgo.By("checking client pod connected to the backend 1 on Node IP " + serverNodeInfo.nodeIP) time.Sleep(30 * time.Second) logs, err = e2epod.GetPodLogs(cs, ns, podClient, podClient) framework.ExpectNoError(err) framework.Logf("Pod client logs: %s", logs) if !strings.Contains(string(logs), podBackend1) { framework.Failf("Failed to connecto to backend 1") } // Create a second pod ginkgo.By("creating a second backend pod " + podBackend2 + " for the service " + serviceName) serverPod2 := newAgnhostPod(podBackend2, "netexec", fmt.Sprintf("--udp-port=%d", 80)) serverPod2.Labels = udpJig.Labels serverPod2.Spec.NodeName = serverNodeInfo.name fr.PodClient().CreateSync(serverPod2) // and delete the first pod framework.Logf("Cleaning up %s pod", podBackend1) fr.PodClient().DeleteSync(podBackend1, metav1.DeleteOptions{}, framework.DefaultPodDeletionTimeout) // Check that the second pod keeps receiving traffic // UDP conntrack entries timeout is 30 sec by default ginkgo.By("checking client pod connected to the backend 2 on Node IP " + serverNodeInfo.nodeIP) time.Sleep(30 * time.Second) logs, err = e2epod.GetPodLogs(cs, ns, podClient, podClient) framework.ExpectNoError(err) framework.Logf("Pod client logs: %s", logs) if !strings.Contains(string(logs), podBackend2) { framework.Failf("Failed to connecto to backend 2") } }) ginkgo.It("should be able to preserve UDP traffic when server pod cycles for a ClusterIP service", func() { // Create a NodePort service udpJig := e2eservice.NewTestJig(cs, ns, serviceName) ginkgo.By("creating a UDP service " + serviceName + " with type=ClusterIP in " + ns) udpService, err := udpJig.CreateUDPService(func(svc *v1.Service) { svc.Spec.Type = v1.ServiceTypeClusterIP svc.Spec.Ports = []v1.ServicePort{ {Port: 80, Name: "udp", Protocol: v1.ProtocolUDP, TargetPort: intstr.FromInt(80)}, } }) framework.ExpectNoError(err) // Create a pod in one node to create the UDP traffic against the NodePort service every 5 seconds ginkgo.By("creating a client pod for probing the service " + serviceName) clientPod := newAgnhostPod(podClient, "") clientPod.Spec.NodeName = clientNodeInfo.name cmd := fmt.Sprintf(`date; for i in $(seq 1 300); do echo "$(date) Try: ${i}"; echo hostname | nc -u -w 5 -p %d %s %d; echo; done`, srcPort, udpService.Spec.ClusterIP, udpService.Spec.Ports[0].Port) clientPod.Spec.Containers[0].Command = []string{"/bin/sh", "-c", cmd} clientPod.Spec.Containers[0].Name = podClient fr.PodClient().CreateSync(clientPod) // Read the client pod logs logs, err := e2epod.GetPodLogs(cs, ns, podClient, podClient) framework.ExpectNoError(err) framework.Logf("Pod client logs: %s", logs) // Add a backend pod to the service in the other node ginkgo.By("creating a backend pod " + podBackend1 + " for the service " + serviceName) serverPod1 := newAgnhostPod(podBackend1, "netexec", fmt.Sprintf("--udp-port=%d", 80)) serverPod1.Labels = udpJig.Labels serverPod1.Spec.NodeName = serverNodeInfo.name fr.PodClient().CreateSync(serverPod1) // Waiting for service to expose endpoint. err = validateEndpointsPorts(cs, ns, serviceName, portsByPodName{podBackend1: {80}}) framework.ExpectNoError(err, "failed to validate endpoints for service %s in namespace: %s", serviceName, ns) // Check that the pod receives the traffic // UDP conntrack entries timeout is 30 sec by default ginkgo.By("checking client pod connected to the backend 1 on Node IP " + serverNodeInfo.nodeIP) time.Sleep(30 * time.Second) logs, err = e2epod.GetPodLogs(cs, ns, podClient, podClient) framework.ExpectNoError(err) framework.Logf("Pod client logs: %s", logs) if !strings.Contains(string(logs), podBackend1) { framework.Failf("Failed to connecto to backend 1") } // Create a second pod ginkgo.By("creating a second backend pod " + podBackend2 + " for the service " + serviceName) serverPod2 := newAgnhostPod(podBackend2, "netexec", fmt.Sprintf("--udp-port=%d", 80)) serverPod2.Labels = udpJig.Labels serverPod2.Spec.NodeName = serverNodeInfo.name fr.PodClient().CreateSync(serverPod2) // and delete the first pod framework.Logf("Cleaning up %s pod", podBackend1) fr.PodClient().DeleteSync(podBackend1, metav1.DeleteOptions{}, framework.DefaultPodDeletionTimeout) // Check that the second pod keeps receiving traffic // UDP conntrack entries timeout is 30 sec by default ginkgo.By("checking client pod connected to the backend 2 on Node IP " + serverNodeInfo.nodeIP) time.Sleep(30 * time.Second) logs, err = e2epod.GetPodLogs(cs, ns, podClient, podClient) framework.ExpectNoError(err) framework.Logf("Pod client logs: %s", logs) if !strings.Contains(string(logs), podBackend2) { framework.Failf("Failed to connecto to backend 2") } }) }) </ins>

Maybe we can increase the timeout and/or switch to a TCP based test:
the test should use UDP, and UDP is unreliable by definition ... however is curious that most of the failures happen because it is not able to communicate with the first pod I think that is something that should be investigated, I've analyzed some occurences and it works fine for the second pod, however, the UDP losses should be totally random
Hi everyone, yeah so.. collecting some data on this... using a two node kind cluster, locally, with the antrea plugin, it passes using a 4 node AWS cluster with the antrea CNI plugin, it failed using a two node kidn cluster locally with calico, it passes So I guess the best reproducer is to run these tests, for now, in the cloud.... Now that I have a reproducer for this maybe can live debug it some time next wk with one of yall :)
bear in mind that this test is connecting to a nodePort in the node IP, the test is functionally correct, it may be flaky due to the unreliable UDP nature or due to the pod scheduling, but if is failing constantly it has to be environmental.
Second failure after two weeks of green 5k runs:
yep, :/ https://k8s-|UDP /assign
/reopen I'm afraid this is going to be challenging :smile: The difference now is that we are testing NodePort and ClusterIP If both fail, we can blame the test for sure, If only NodePort fails, we need to check the job, if GCE or KIND If only ClusterIP fails, ... :thinking:
Reopened this issue. <detailsIn response to : /reopen I'm afraid this is going to be challenging :smile: Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
We're hitting this really frequently in 5k-node tests. So I started looking a little bit into that.


Question: <ins> /* Copyright 2020 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package network import ( "fmt" "strings" "time" "github.com/onsi/ginkgo" v1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/util/intstr" clientset "k8s.io/client-go/kubernetes" "k8s.io/kubernetes/test/e2e/framework" e2enode "k8s.io/kubernetes/test/e2e/framework/node" e2epod "k8s.io/kubernetes/test/e2e/framework/pod" e2eservice "k8s.io/kubernetes/test/e2e/framework/service" e2eskipper "k8s.io/kubernetes/test/e2e/framework/skipper" ) const ( serviceName = "svc-udp" podClient   = "pod-client" podBackend1 = "pod-server-1" podBackend2 = "pod-server-2" srcPort     = 12345 ) // Linux NAT uses conntrack to perform NAT, everytime a new // flow is seen, a connection is created in the conntrack table, and it // is being used by the NAT module. // Each entry in the conntrack table has associated a timeout, that removes // the connection once it expires. // UDP is a connectionless protocol, so the conntrack module tracking functions // are not very advanced. // It uses a short timeout (30 sec by default) that is renewed if there are new flows // matching the connection. Otherwise it expires the entry. // This behaviour can cause issues in Kubernetes when one entry on the conntrack table // is never expired because the sender does not stop sending traffic, but the pods or // endpoints were deleted, blackholing the traffic // In order to mitigate this problem, Kubernetes delete the stale entries: // - when an endpoint is removed // - when a service goes from no endpoints to new endpoint // Ref: https://api.semanticscholar.org/CorpusID:198903401 // Boye, Magnus. “Netfilter Connection Tracking and NAT Implementation.” (2012). var _ = SIGDescribe("Conntrack", func() { fr := framework.NewDefaultFramework("conntrack") type nodeInfo struct { name   string nodeIP string } var ( cs                             clientset.Interface ns                             string clientNodeInfo, serverNodeInfo nodeInfo ) ginkgo.BeforeEach(func() { cs = fr.ClientSet ns = fr.Namespace.Name nodes, err := e2enode.GetBoundedReadySchedulableNodes(cs, 2) framework.ExpectNoError(err) if len(nodes.Items) < 2 { e2eskipper.Skipf( "Test requires >= 2 Ready nodes, but there are only %v nodes", len(nodes.Items)) } ips := e2enode.CollectAddresses(nodes, v1.NodeInternalIP) clientNodeInfo = nodeInfo{ name:   nodes.Items[0].Name, nodeIP: ips[0], } serverNodeInfo = nodeInfo{ name:   nodes.Items[1].Name, nodeIP: ips[1], } }) ginkgo.It("should be able to preserve UDP traffic when server pod cycles for a NodePort service", func() { // Create a NodePort service udpJig := e2eservice.NewTestJig(cs, ns, serviceName) ginkgo.By("creating a UDP service " + serviceName + " with type=NodePort in " + ns) udpService, err := udpJig.CreateUDPService(func(svc *v1.Service) { svc.Spec.Type = v1.ServiceTypeNodePort svc.Spec.Ports = []v1.ServicePort{ {Port: 80, Name: "udp", Protocol: v1.ProtocolUDP, TargetPort: intstr.FromInt(80)}, } }) framework.ExpectNoError(err) // Create a pod in one node to create the UDP traffic against the NodePort service every 5 seconds ginkgo.By("creating a client pod for probing the service " + serviceName) clientPod := newAgnhostPod(podClient, "") clientPod.Spec.NodeName = clientNodeInfo.name cmd := fmt.Sprintf(`date; for i in $(seq 1 300); do echo "$(date) Try: ${i}"; echo hostname | nc -u -w 5 -p %d %s %d; echo; done`, srcPort, serverNodeInfo.nodeIP, udpService.Spec.Ports[0].NodePort) clientPod.Spec.Containers[0].Command = []string{"/bin/sh", "-c", cmd} clientPod.Spec.Containers[0].Name = podClient fr.PodClient().CreateSync(clientPod) // Read the client pod logs logs, err := e2epod.GetPodLogs(cs, ns, podClient, podClient) framework.ExpectNoError(err) framework.Logf("Pod client logs: %s", logs) // Add a backend pod to the service in the other node ginkgo.By("creating a backend pod " + podBackend1 + " for the service " + serviceName) serverPod1 := newAgnhostPod(podBackend1, "netexec", fmt.Sprintf("--udp-port=%d", 80)) serverPod1.Labels = udpJig.Labels serverPod1.Spec.NodeName = serverNodeInfo.name fr.PodClient().CreateSync(serverPod1) // Waiting for service to expose endpoint. err = validateEndpointsPorts(cs, ns, serviceName, portsByPodName{podBackend1: {80}}) framework.ExpectNoError(err, "failed to validate endpoints for service %s in namespace: %s", serviceName, ns) // Check that the pod receives the traffic // UDP conntrack entries timeout is 30 sec by default ginkgo.By("checking client pod connected to the backend 1 on Node IP " + serverNodeInfo.nodeIP) time.Sleep(30 * time.Second) logs, err = e2epod.GetPodLogs(cs, ns, podClient, podClient) framework.ExpectNoError(err) framework.Logf("Pod client logs: %s", logs) if !strings.Contains(string(logs), podBackend1) { framework.Failf("Failed to connecto to backend 1") } // Create a second pod ginkgo.By("creating a second backend pod " + podBackend2 + " for the service " + serviceName) serverPod2 := newAgnhostPod(podBackend2, "netexec", fmt.Sprintf("--udp-port=%d", 80)) serverPod2.Labels = udpJig.Labels serverPod2.Spec.NodeName = serverNodeInfo.name fr.PodClient().CreateSync(serverPod2) // and delete the first pod framework.Logf("Cleaning up %s pod", podBackend1) fr.PodClient().DeleteSync(podBackend1, metav1.DeleteOptions{}, framework.DefaultPodDeletionTimeout) // Check that the second pod keeps receiving traffic // UDP conntrack entries timeout is 30 sec by default ginkgo.By("checking client pod connected to the backend 2 on Node IP " + serverNodeInfo.nodeIP) time.Sleep(30 * time.Second) logs, err = e2epod.GetPodLogs(cs, ns, podClient, podClient) framework.ExpectNoError(err) framework.Logf("Pod client logs: %s", logs) if !strings.Contains(string(logs), podBackend2) { framework.Failf("Failed to connecto to backend 2") } }) ginkgo.It("should be able to preserve UDP traffic when server pod cycles for a ClusterIP service", func() { // Create a NodePort service udpJig := e2eservice.NewTestJig(cs, ns, serviceName) ginkgo.By("creating a UDP service " + serviceName + " with type=ClusterIP in " + ns) udpService, err := udpJig.CreateUDPService(func(svc *v1.Service) { svc.Spec.Type = v1.ServiceTypeClusterIP svc.Spec.Ports = []v1.ServicePort{ {Port: 80, Name: "udp", Protocol: v1.ProtocolUDP, TargetPort: intstr.FromInt(80)}, } }) framework.ExpectNoError(err) // Create a pod in one node to create the UDP traffic against the NodePort service every 5 seconds ginkgo.By("creating a client pod for probing the service " + serviceName) clientPod := newAgnhostPod(podClient, "") clientPod.Spec.NodeName = clientNodeInfo.name cmd := fmt.Sprintf(`date; for i in $(seq 1 300); do echo "$(date) Try: ${i}"; echo hostname | nc -u -w 5 -p %d %s %d; echo; done`, srcPort, udpService.Spec.ClusterIP, udpService.Spec.Ports[0].Port) clientPod.Spec.Containers[0].Command = []string{"/bin/sh", "-c", cmd} clientPod.Spec.Containers[0].Name = podClient fr.PodClient().CreateSync(clientPod) // Read the client pod logs logs, err := e2epod.GetPodLogs(cs, ns, podClient, podClient) framework.ExpectNoError(err) framework.Logf("Pod client logs: %s", logs) // Add a backend pod to the service in the other node ginkgo.By("creating a backend pod " + podBackend1 + " for the service " + serviceName) serverPod1 := newAgnhostPod(podBackend1, "netexec", fmt.Sprintf("--udp-port=%d", 80)) serverPod1.Labels = udpJig.Labels serverPod1.Spec.NodeName = serverNodeInfo.name fr.PodClient().CreateSync(serverPod1) // Waiting for service to expose endpoint. err = validateEndpointsPorts(cs, ns, serviceName, portsByPodName{podBackend1: {80}}) framework.ExpectNoError(err, "failed to validate endpoints for service %s in namespace: %s", serviceName, ns) // Check that the pod receives the traffic // UDP conntrack entries timeout is 30 sec by default ginkgo.By("checking client pod connected to the backend 1 on Node IP " + serverNodeInfo.nodeIP) time.Sleep(30 * time.Second) logs, err = e2epod.GetPodLogs(cs, ns, podClient, podClient) framework.ExpectNoError(err) framework.Logf("Pod client logs: %s", logs) if !strings.Contains(string(logs), podBackend1) { framework.Failf("Failed to connecto to backend 1") } // Create a second pod ginkgo.By("creating a second backend pod " + podBackend2 + " for the service " + serviceName) serverPod2 := newAgnhostPod(podBackend2, "netexec", fmt.Sprintf("--udp-port=%d", 80)) serverPod2.Labels = udpJig.Labels serverPod2.Spec.NodeName = serverNodeInfo.name fr.PodClient().CreateSync(serverPod2) // and delete the first pod framework.Logf("Cleaning up %s pod", podBackend1) fr.PodClient().DeleteSync(podBackend1, metav1.DeleteOptions{}, framework.DefaultPodDeletionTimeout) // Check that the second pod keeps receiving traffic // UDP conntrack entries timeout is 30 sec by default ginkgo.By("checking client pod connected to the backend 2 on Node IP " + serverNodeInfo.nodeIP) time.Sleep(30 * time.Second) logs, err = e2epod.GetPodLogs(cs, ns, podClient, podClient) framework.ExpectNoError(err) framework.Logf("Pod client logs: %s", logs) if !strings.Contains(string(logs), podBackend2) { framework.Failf("Failed to connecto to backend 2") } }) }) </ins>

I don't have full understanding of what exactly is happening, but I already see a couple things that may be causing issues: when creating the first pod, you wait for corresponding endpoint for that being created; but that doesn't mean that kube-proxy was actually programmed correctly at that time (in general it isn't); given iptables programming is rate-limitted, it may easily happen that actually programming happens 10s+ later That explains some delays, but it doesn't explain e.g. this run: where iptables were programmed at but the test didn't respond until If someone more familiar with networking stuff could look into it, it would be great. - can you help triaging ^^ for deletions it's even worse, because you don't even wait for endpoints to be there (i.e. Endpoints can still contain the old pod when the test start pinging to look for the second one) So I claim that timeouts are visibly to low in this test, but I suspect it's not everything. I will send out the first PR that tries to improve it in couple minutes.
thanks wojtek, have been slammed but Ill try to collect more data on it this wknd as well.
It seems that my PR didn't really help much. We should dig deeper, but I would need someone with more networking expertise to look a bit into
Actually - it seems only failures from one group are on non . So maybe that's actually related. The non-kind related ones look like:
OK - for this one, I actually know what is happening - I will send out a PR with a fix for this too.
for the KIND jobs already increased the resource requests on some of the jobs, there are issues on KIND jobs with kube-proxy not able to take the iptables lock and not being able to create the iptables rules for the services until the next resync
is to fix the flakes I mentioned above; all other flakes seem to be kind-oriented to me (which I'm not super familiar with).
We did not adjust the CPU on all jobs yet, pending how to manage that on prow ... I do see just a few other flakes at the bottom of the triage dashboard that look like timeouts instead.
Actually - almost all failures are effectively timeout. The top one that is is actually also a timeout. Didn't look into kind failures, but this may easily be related to iptables issues.


Question: <ins> /* Copyright 2020 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package network import ( "fmt" "strings" "time" "github.com/onsi/ginkgo" v1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/util/intstr" clientset "k8s.io/client-go/kubernetes" "k8s.io/kubernetes/test/e2e/framework" e2enode "k8s.io/kubernetes/test/e2e/framework/node" e2epod "k8s.io/kubernetes/test/e2e/framework/pod" e2eservice "k8s.io/kubernetes/test/e2e/framework/service" e2eskipper "k8s.io/kubernetes/test/e2e/framework/skipper" ) const ( serviceName = "svc-udp" podClient   = "pod-client" podBackend1 = "pod-server-1" podBackend2 = "pod-server-2" srcPort     = 12345 ) // Linux NAT uses conntrack to perform NAT, everytime a new // flow is seen, a connection is created in the conntrack table, and it // is being used by the NAT module. // Each entry in the conntrack table has associated a timeout, that removes // the connection once it expires. // UDP is a connectionless protocol, so the conntrack module tracking functions // are not very advanced. // It uses a short timeout (30 sec by default) that is renewed if there are new flows // matching the connection. Otherwise it expires the entry. // This behaviour can cause issues in Kubernetes when one entry on the conntrack table // is never expired because the sender does not stop sending traffic, but the pods or // endpoints were deleted, blackholing the traffic // In order to mitigate this problem, Kubernetes delete the stale entries: // - when an endpoint is removed // - when a service goes from no endpoints to new endpoint // Ref: https://api.semanticscholar.org/CorpusID:198903401 // Boye, Magnus. “Netfilter Connection Tracking and NAT Implementation.” (2012). var _ = SIGDescribe("Conntrack", func() { fr := framework.NewDefaultFramework("conntrack") type nodeInfo struct { name   string nodeIP string } var ( cs                             clientset.Interface ns                             string clientNodeInfo, serverNodeInfo nodeInfo ) ginkgo.BeforeEach(func() { cs = fr.ClientSet ns = fr.Namespace.Name nodes, err := e2enode.GetBoundedReadySchedulableNodes(cs, 2) framework.ExpectNoError(err) if len(nodes.Items) < 2 { e2eskipper.Skipf( "Test requires >= 2 Ready nodes, but there are only %v nodes", len(nodes.Items)) } ips := e2enode.CollectAddresses(nodes, v1.NodeInternalIP) clientNodeInfo = nodeInfo{ name:   nodes.Items[0].Name, nodeIP: ips[0], } serverNodeInfo = nodeInfo{ name:   nodes.Items[1].Name, nodeIP: ips[1], } }) ginkgo.It("should be able to preserve UDP traffic when server pod cycles for a NodePort service", func() { // Create a NodePort service udpJig := e2eservice.NewTestJig(cs, ns, serviceName) ginkgo.By("creating a UDP service " + serviceName + " with type=NodePort in " + ns) udpService, err := udpJig.CreateUDPService(func(svc *v1.Service) { svc.Spec.Type = v1.ServiceTypeNodePort svc.Spec.Ports = []v1.ServicePort{ {Port: 80, Name: "udp", Protocol: v1.ProtocolUDP, TargetPort: intstr.FromInt(80)}, } }) framework.ExpectNoError(err) // Create a pod in one node to create the UDP traffic against the NodePort service every 5 seconds ginkgo.By("creating a client pod for probing the service " + serviceName) clientPod := newAgnhostPod(podClient, "") clientPod.Spec.NodeName = clientNodeInfo.name cmd := fmt.Sprintf(`date; for i in $(seq 1 300); do echo "$(date) Try: ${i}"; echo hostname | nc -u -w 5 -p %d %s %d; echo; done`, srcPort, serverNodeInfo.nodeIP, udpService.Spec.Ports[0].NodePort) clientPod.Spec.Containers[0].Command = []string{"/bin/sh", "-c", cmd} clientPod.Spec.Containers[0].Name = podClient fr.PodClient().CreateSync(clientPod) // Read the client pod logs logs, err := e2epod.GetPodLogs(cs, ns, podClient, podClient) framework.ExpectNoError(err) framework.Logf("Pod client logs: %s", logs) // Add a backend pod to the service in the other node ginkgo.By("creating a backend pod " + podBackend1 + " for the service " + serviceName) serverPod1 := newAgnhostPod(podBackend1, "netexec", fmt.Sprintf("--udp-port=%d", 80)) serverPod1.Labels = udpJig.Labels serverPod1.Spec.NodeName = serverNodeInfo.name fr.PodClient().CreateSync(serverPod1) // Waiting for service to expose endpoint. err = validateEndpointsPorts(cs, ns, serviceName, portsByPodName{podBackend1: {80}}) framework.ExpectNoError(err, "failed to validate endpoints for service %s in namespace: %s", serviceName, ns) // Check that the pod receives the traffic // UDP conntrack entries timeout is 30 sec by default ginkgo.By("checking client pod connected to the backend 1 on Node IP " + serverNodeInfo.nodeIP) time.Sleep(30 * time.Second) logs, err = e2epod.GetPodLogs(cs, ns, podClient, podClient) framework.ExpectNoError(err) framework.Logf("Pod client logs: %s", logs) if !strings.Contains(string(logs), podBackend1) { framework.Failf("Failed to connecto to backend 1") } // Create a second pod ginkgo.By("creating a second backend pod " + podBackend2 + " for the service " + serviceName) serverPod2 := newAgnhostPod(podBackend2, "netexec", fmt.Sprintf("--udp-port=%d", 80)) serverPod2.Labels = udpJig.Labels serverPod2.Spec.NodeName = serverNodeInfo.name fr.PodClient().CreateSync(serverPod2) // and delete the first pod framework.Logf("Cleaning up %s pod", podBackend1) fr.PodClient().DeleteSync(podBackend1, metav1.DeleteOptions{}, framework.DefaultPodDeletionTimeout) // Check that the second pod keeps receiving traffic // UDP conntrack entries timeout is 30 sec by default ginkgo.By("checking client pod connected to the backend 2 on Node IP " + serverNodeInfo.nodeIP) time.Sleep(30 * time.Second) logs, err = e2epod.GetPodLogs(cs, ns, podClient, podClient) framework.ExpectNoError(err) framework.Logf("Pod client logs: %s", logs) if !strings.Contains(string(logs), podBackend2) { framework.Failf("Failed to connecto to backend 2") } }) ginkgo.It("should be able to preserve UDP traffic when server pod cycles for a ClusterIP service", func() { // Create a NodePort service udpJig := e2eservice.NewTestJig(cs, ns, serviceName) ginkgo.By("creating a UDP service " + serviceName + " with type=ClusterIP in " + ns) udpService, err := udpJig.CreateUDPService(func(svc *v1.Service) { svc.Spec.Type = v1.ServiceTypeClusterIP svc.Spec.Ports = []v1.ServicePort{ {Port: 80, Name: "udp", Protocol: v1.ProtocolUDP, TargetPort: intstr.FromInt(80)}, } }) framework.ExpectNoError(err) // Create a pod in one node to create the UDP traffic against the NodePort service every 5 seconds ginkgo.By("creating a client pod for probing the service " + serviceName) clientPod := newAgnhostPod(podClient, "") clientPod.Spec.NodeName = clientNodeInfo.name cmd := fmt.Sprintf(`date; for i in $(seq 1 300); do echo "$(date) Try: ${i}"; echo hostname | nc -u -w 5 -p %d %s %d; echo; done`, srcPort, udpService.Spec.ClusterIP, udpService.Spec.Ports[0].Port) clientPod.Spec.Containers[0].Command = []string{"/bin/sh", "-c", cmd} clientPod.Spec.Containers[0].Name = podClient fr.PodClient().CreateSync(clientPod) // Read the client pod logs logs, err := e2epod.GetPodLogs(cs, ns, podClient, podClient) framework.ExpectNoError(err) framework.Logf("Pod client logs: %s", logs) // Add a backend pod to the service in the other node ginkgo.By("creating a backend pod " + podBackend1 + " for the service " + serviceName) serverPod1 := newAgnhostPod(podBackend1, "netexec", fmt.Sprintf("--udp-port=%d", 80)) serverPod1.Labels = udpJig.Labels serverPod1.Spec.NodeName = serverNodeInfo.name fr.PodClient().CreateSync(serverPod1) // Waiting for service to expose endpoint. err = validateEndpointsPorts(cs, ns, serviceName, portsByPodName{podBackend1: {80}}) framework.ExpectNoError(err, "failed to validate endpoints for service %s in namespace: %s", serviceName, ns) // Check that the pod receives the traffic // UDP conntrack entries timeout is 30 sec by default ginkgo.By("checking client pod connected to the backend 1 on Node IP " + serverNodeInfo.nodeIP) time.Sleep(30 * time.Second) logs, err = e2epod.GetPodLogs(cs, ns, podClient, podClient) framework.ExpectNoError(err) framework.Logf("Pod client logs: %s", logs) if !strings.Contains(string(logs), podBackend1) { framework.Failf("Failed to connecto to backend 1") } // Create a second pod ginkgo.By("creating a second backend pod " + podBackend2 + " for the service " + serviceName) serverPod2 := newAgnhostPod(podBackend2, "netexec", fmt.Sprintf("--udp-port=%d", 80)) serverPod2.Labels = udpJig.Labels serverPod2.Spec.NodeName = serverNodeInfo.name fr.PodClient().CreateSync(serverPod2) // and delete the first pod framework.Logf("Cleaning up %s pod", podBackend1) fr.PodClient().DeleteSync(podBackend1, metav1.DeleteOptions{}, framework.DefaultPodDeletionTimeout) // Check that the second pod keeps receiving traffic // UDP conntrack entries timeout is 30 sec by default ginkgo.By("checking client pod connected to the backend 2 on Node IP " + serverNodeInfo.nodeIP) time.Sleep(30 * time.Second) logs, err = e2epod.GetPodLogs(cs, ns, podClient, podClient) framework.ExpectNoError(err) framework.Logf("Pod client logs: %s", logs) if !strings.Contains(string(logs), podBackend2) { framework.Failf("Failed to connecto to backend 2") } }) }) </ins>

After my second deflaking PR, we seem to be in the stage where: tests are failing on dozens of different things (probably iptables-issues related) tests are failing only on NodePort tests on "Failed to connecto to backend 1" which it would be great if someone else could look into (kube-proxy seemed to be programmed correctly based on logs)
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
This has failed again - Linking to specific part of log where timeout occurred .... we see this
So reviewing the above logs note how in STEP: checking client pod connected to the backend 1 on Node IP 10.138.0.5 28 attempts are made all in the 25th second of the 2 minute at 11am Should this Step back off and wait a few seconds in between tries? So looking at this code we see the following
From reading ... the intention expressed in the test code looks good to me. It is not clear to me why there is a mismatch between the 28 tries in the POD Client Logs and the expressed intention running the logContainsFn(podBackend1) every 5 seconds still digging ....
we are trying to catch one failure in this PR that has additional debugging, it dumps the conntrack table of the nodes on a failure
for those following the issue, we've extra debugging to the test so it dumps the whole conntrack table once it finish. We should look that there is NO stale entries, connections from the pod client to the deleted endpoint) and there is conntrack entries to the new endpoint from the pod client.
- we have runs with the conntrack dump now, e.g. this one: would you be able to take a look into it?
not yet :/ , but I think tomorrow can have some time
cc
For anyone reproducing this issue btw, you need a multinode k8s cluster (it isnt doable on kind i guess bc less likeliehood of network failures when running local) -
Curious about two potential hypotheses to fix this, have we looked into either of these? 1) can we confirm that is turned on/off? 2) is it possible that scaling the number of coredns replicas could help ?


Question: <ins> /* Copyright 2020 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package network import ( "fmt" "strings" "time" "github.com/onsi/ginkgo" v1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/util/intstr" clientset "k8s.io/client-go/kubernetes" "k8s.io/kubernetes/test/e2e/framework" e2enode "k8s.io/kubernetes/test/e2e/framework/node" e2epod "k8s.io/kubernetes/test/e2e/framework/pod" e2eservice "k8s.io/kubernetes/test/e2e/framework/service" e2eskipper "k8s.io/kubernetes/test/e2e/framework/skipper" ) const ( serviceName = "svc-udp" podClient   = "pod-client" podBackend1 = "pod-server-1" podBackend2 = "pod-server-2" srcPort     = 12345 ) // Linux NAT uses conntrack to perform NAT, everytime a new // flow is seen, a connection is created in the conntrack table, and it // is being used by the NAT module. // Each entry in the conntrack table has associated a timeout, that removes // the connection once it expires. // UDP is a connectionless protocol, so the conntrack module tracking functions // are not very advanced. // It uses a short timeout (30 sec by default) that is renewed if there are new flows // matching the connection. Otherwise it expires the entry. // This behaviour can cause issues in Kubernetes when one entry on the conntrack table // is never expired because the sender does not stop sending traffic, but the pods or // endpoints were deleted, blackholing the traffic // In order to mitigate this problem, Kubernetes delete the stale entries: // - when an endpoint is removed // - when a service goes from no endpoints to new endpoint // Ref: https://api.semanticscholar.org/CorpusID:198903401 // Boye, Magnus. “Netfilter Connection Tracking and NAT Implementation.” (2012). var _ = SIGDescribe("Conntrack", func() { fr := framework.NewDefaultFramework("conntrack") type nodeInfo struct { name   string nodeIP string } var ( cs                             clientset.Interface ns                             string clientNodeInfo, serverNodeInfo nodeInfo ) ginkgo.BeforeEach(func() { cs = fr.ClientSet ns = fr.Namespace.Name nodes, err := e2enode.GetBoundedReadySchedulableNodes(cs, 2) framework.ExpectNoError(err) if len(nodes.Items) < 2 { e2eskipper.Skipf( "Test requires >= 2 Ready nodes, but there are only %v nodes", len(nodes.Items)) } ips := e2enode.CollectAddresses(nodes, v1.NodeInternalIP) clientNodeInfo = nodeInfo{ name:   nodes.Items[0].Name, nodeIP: ips[0], } serverNodeInfo = nodeInfo{ name:   nodes.Items[1].Name, nodeIP: ips[1], } }) ginkgo.It("should be able to preserve UDP traffic when server pod cycles for a NodePort service", func() { // Create a NodePort service udpJig := e2eservice.NewTestJig(cs, ns, serviceName) ginkgo.By("creating a UDP service " + serviceName + " with type=NodePort in " + ns) udpService, err := udpJig.CreateUDPService(func(svc *v1.Service) { svc.Spec.Type = v1.ServiceTypeNodePort svc.Spec.Ports = []v1.ServicePort{ {Port: 80, Name: "udp", Protocol: v1.ProtocolUDP, TargetPort: intstr.FromInt(80)}, } }) framework.ExpectNoError(err) // Create a pod in one node to create the UDP traffic against the NodePort service every 5 seconds ginkgo.By("creating a client pod for probing the service " + serviceName) clientPod := newAgnhostPod(podClient, "") clientPod.Spec.NodeName = clientNodeInfo.name cmd := fmt.Sprintf(`date; for i in $(seq 1 300); do echo "$(date) Try: ${i}"; echo hostname | nc -u -w 5 -p %d %s %d; echo; done`, srcPort, serverNodeInfo.nodeIP, udpService.Spec.Ports[0].NodePort) clientPod.Spec.Containers[0].Command = []string{"/bin/sh", "-c", cmd} clientPod.Spec.Containers[0].Name = podClient fr.PodClient().CreateSync(clientPod) // Read the client pod logs logs, err := e2epod.GetPodLogs(cs, ns, podClient, podClient) framework.ExpectNoError(err) framework.Logf("Pod client logs: %s", logs) // Add a backend pod to the service in the other node ginkgo.By("creating a backend pod " + podBackend1 + " for the service " + serviceName) serverPod1 := newAgnhostPod(podBackend1, "netexec", fmt.Sprintf("--udp-port=%d", 80)) serverPod1.Labels = udpJig.Labels serverPod1.Spec.NodeName = serverNodeInfo.name fr.PodClient().CreateSync(serverPod1) // Waiting for service to expose endpoint. err = validateEndpointsPorts(cs, ns, serviceName, portsByPodName{podBackend1: {80}}) framework.ExpectNoError(err, "failed to validate endpoints for service %s in namespace: %s", serviceName, ns) // Check that the pod receives the traffic // UDP conntrack entries timeout is 30 sec by default ginkgo.By("checking client pod connected to the backend 1 on Node IP " + serverNodeInfo.nodeIP) time.Sleep(30 * time.Second) logs, err = e2epod.GetPodLogs(cs, ns, podClient, podClient) framework.ExpectNoError(err) framework.Logf("Pod client logs: %s", logs) if !strings.Contains(string(logs), podBackend1) { framework.Failf("Failed to connecto to backend 1") } // Create a second pod ginkgo.By("creating a second backend pod " + podBackend2 + " for the service " + serviceName) serverPod2 := newAgnhostPod(podBackend2, "netexec", fmt.Sprintf("--udp-port=%d", 80)) serverPod2.Labels = udpJig.Labels serverPod2.Spec.NodeName = serverNodeInfo.name fr.PodClient().CreateSync(serverPod2) // and delete the first pod framework.Logf("Cleaning up %s pod", podBackend1) fr.PodClient().DeleteSync(podBackend1, metav1.DeleteOptions{}, framework.DefaultPodDeletionTimeout) // Check that the second pod keeps receiving traffic // UDP conntrack entries timeout is 30 sec by default ginkgo.By("checking client pod connected to the backend 2 on Node IP " + serverNodeInfo.nodeIP) time.Sleep(30 * time.Second) logs, err = e2epod.GetPodLogs(cs, ns, podClient, podClient) framework.ExpectNoError(err) framework.Logf("Pod client logs: %s", logs) if !strings.Contains(string(logs), podBackend2) { framework.Failf("Failed to connecto to backend 2") } }) ginkgo.It("should be able to preserve UDP traffic when server pod cycles for a ClusterIP service", func() { // Create a NodePort service udpJig := e2eservice.NewTestJig(cs, ns, serviceName) ginkgo.By("creating a UDP service " + serviceName + " with type=ClusterIP in " + ns) udpService, err := udpJig.CreateUDPService(func(svc *v1.Service) { svc.Spec.Type = v1.ServiceTypeClusterIP svc.Spec.Ports = []v1.ServicePort{ {Port: 80, Name: "udp", Protocol: v1.ProtocolUDP, TargetPort: intstr.FromInt(80)}, } }) framework.ExpectNoError(err) // Create a pod in one node to create the UDP traffic against the NodePort service every 5 seconds ginkgo.By("creating a client pod for probing the service " + serviceName) clientPod := newAgnhostPod(podClient, "") clientPod.Spec.NodeName = clientNodeInfo.name cmd := fmt.Sprintf(`date; for i in $(seq 1 300); do echo "$(date) Try: ${i}"; echo hostname | nc -u -w 5 -p %d %s %d; echo; done`, srcPort, udpService.Spec.ClusterIP, udpService.Spec.Ports[0].Port) clientPod.Spec.Containers[0].Command = []string{"/bin/sh", "-c", cmd} clientPod.Spec.Containers[0].Name = podClient fr.PodClient().CreateSync(clientPod) // Read the client pod logs logs, err := e2epod.GetPodLogs(cs, ns, podClient, podClient) framework.ExpectNoError(err) framework.Logf("Pod client logs: %s", logs) // Add a backend pod to the service in the other node ginkgo.By("creating a backend pod " + podBackend1 + " for the service " + serviceName) serverPod1 := newAgnhostPod(podBackend1, "netexec", fmt.Sprintf("--udp-port=%d", 80)) serverPod1.Labels = udpJig.Labels serverPod1.Spec.NodeName = serverNodeInfo.name fr.PodClient().CreateSync(serverPod1) // Waiting for service to expose endpoint. err = validateEndpointsPorts(cs, ns, serviceName, portsByPodName{podBackend1: {80}}) framework.ExpectNoError(err, "failed to validate endpoints for service %s in namespace: %s", serviceName, ns) // Check that the pod receives the traffic // UDP conntrack entries timeout is 30 sec by default ginkgo.By("checking client pod connected to the backend 1 on Node IP " + serverNodeInfo.nodeIP) time.Sleep(30 * time.Second) logs, err = e2epod.GetPodLogs(cs, ns, podClient, podClient) framework.ExpectNoError(err) framework.Logf("Pod client logs: %s", logs) if !strings.Contains(string(logs), podBackend1) { framework.Failf("Failed to connecto to backend 1") } // Create a second pod ginkgo.By("creating a second backend pod " + podBackend2 + " for the service " + serviceName) serverPod2 := newAgnhostPod(podBackend2, "netexec", fmt.Sprintf("--udp-port=%d", 80)) serverPod2.Labels = udpJig.Labels serverPod2.Spec.NodeName = serverNodeInfo.name fr.PodClient().CreateSync(serverPod2) // and delete the first pod framework.Logf("Cleaning up %s pod", podBackend1) fr.PodClient().DeleteSync(podBackend1, metav1.DeleteOptions{}, framework.DefaultPodDeletionTimeout) // Check that the second pod keeps receiving traffic // UDP conntrack entries timeout is 30 sec by default ginkgo.By("checking client pod connected to the backend 2 on Node IP " + serverNodeInfo.nodeIP) time.Sleep(30 * time.Second) logs, err = e2epod.GetPodLogs(cs, ns, podClient, podClient) framework.ExpectNoError(err) framework.Logf("Pod client logs: %s", logs) if !strings.Contains(string(logs), podBackend2) { framework.Failf("Failed to connecto to backend 2") } }) }) </ins>

3) Would simulating this in kind by overloading coredns be a possibility as a way to make a more reproducible bug model?
regarding 1) kubeadm does not ship nodeLocalDNSCache I don't think GCE kube-up does by default either
ok, so now im curious about (2) - i.e. what if we just bumped coredns replica count?
interesting, :thinking: why do you think is a DNS thing? the netcat command that polls the services use the IP addresses not the hostname. I think that it has to be related to some iptables/conntrack race or similar,at least in KIND jobs we know that we have an iptables problem that affects network tests, that has to be fixed before this because it affect this test You can instrument the test locally with this and dump the state of iptables and conntrack on failure if you are able to reproduce it
i figured DNS bc i believe its known that conntrack and DNS at scale can get very messy, right ?
one other curious thing about this test is that we only create 2 pods in the recycling event for the svc. that likely means that any real error is hidden by chance..... If we created a 3rd pod, then we might potentially double the failure rate thus making the flakyness go away and expose the real underlying issue .
There's an invocation of dumpConntrack(cs) here [sig-network] Conntrack should be able to preserve UDP traffic when server pod cycles for a ClusterIP service
is updating the logging to run in an infinite loop ..
This is improving in KIND with the new changes we rolled out yesterday, let's see how it evolves this week
- which changes ? Are you saying this is no longer reproducible on Kind?
there was a lot of noise in the CI signal, especially in KIND jobs due to the CNI portmap bug we finally solved yesterday now we should have a better signal and we can go to the bottom because probably the failure will be "legit" and not a consequence, so this comment should apply
Hi ! I ran this a few times on kind and it passed, but I see , so I'm going to try the loop approach you suggested. Here's what I'm using to test:
quick note, <- potential motivation to possibly have runnable on some subset of tests in upstream eventually, i.e.


Question: <ins> /* Copyright 2020 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package network import ( "fmt" "strings" "time" "github.com/onsi/ginkgo" v1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/util/intstr" clientset "k8s.io/client-go/kubernetes" "k8s.io/kubernetes/test/e2e/framework" e2enode "k8s.io/kubernetes/test/e2e/framework/node" e2epod "k8s.io/kubernetes/test/e2e/framework/pod" e2eservice "k8s.io/kubernetes/test/e2e/framework/service" e2eskipper "k8s.io/kubernetes/test/e2e/framework/skipper" ) const ( serviceName = "svc-udp" podClient   = "pod-client" podBackend1 = "pod-server-1" podBackend2 = "pod-server-2" srcPort     = 12345 ) // Linux NAT uses conntrack to perform NAT, everytime a new // flow is seen, a connection is created in the conntrack table, and it // is being used by the NAT module. // Each entry in the conntrack table has associated a timeout, that removes // the connection once it expires. // UDP is a connectionless protocol, so the conntrack module tracking functions // are not very advanced. // It uses a short timeout (30 sec by default) that is renewed if there are new flows // matching the connection. Otherwise it expires the entry. // This behaviour can cause issues in Kubernetes when one entry on the conntrack table // is never expired because the sender does not stop sending traffic, but the pods or // endpoints were deleted, blackholing the traffic // In order to mitigate this problem, Kubernetes delete the stale entries: // - when an endpoint is removed // - when a service goes from no endpoints to new endpoint // Ref: https://api.semanticscholar.org/CorpusID:198903401 // Boye, Magnus. “Netfilter Connection Tracking and NAT Implementation.” (2012). var _ = SIGDescribe("Conntrack", func() { fr := framework.NewDefaultFramework("conntrack") type nodeInfo struct { name   string nodeIP string } var ( cs                             clientset.Interface ns                             string clientNodeInfo, serverNodeInfo nodeInfo ) ginkgo.BeforeEach(func() { cs = fr.ClientSet ns = fr.Namespace.Name nodes, err := e2enode.GetBoundedReadySchedulableNodes(cs, 2) framework.ExpectNoError(err) if len(nodes.Items) < 2 { e2eskipper.Skipf( "Test requires >= 2 Ready nodes, but there are only %v nodes", len(nodes.Items)) } ips := e2enode.CollectAddresses(nodes, v1.NodeInternalIP) clientNodeInfo = nodeInfo{ name:   nodes.Items[0].Name, nodeIP: ips[0], } serverNodeInfo = nodeInfo{ name:   nodes.Items[1].Name, nodeIP: ips[1], } }) ginkgo.It("should be able to preserve UDP traffic when server pod cycles for a NodePort service", func() { // Create a NodePort service udpJig := e2eservice.NewTestJig(cs, ns, serviceName) ginkgo.By("creating a UDP service " + serviceName + " with type=NodePort in " + ns) udpService, err := udpJig.CreateUDPService(func(svc *v1.Service) { svc.Spec.Type = v1.ServiceTypeNodePort svc.Spec.Ports = []v1.ServicePort{ {Port: 80, Name: "udp", Protocol: v1.ProtocolUDP, TargetPort: intstr.FromInt(80)}, } }) framework.ExpectNoError(err) // Create a pod in one node to create the UDP traffic against the NodePort service every 5 seconds ginkgo.By("creating a client pod for probing the service " + serviceName) clientPod := newAgnhostPod(podClient, "") clientPod.Spec.NodeName = clientNodeInfo.name cmd := fmt.Sprintf(`date; for i in $(seq 1 300); do echo "$(date) Try: ${i}"; echo hostname | nc -u -w 5 -p %d %s %d; echo; done`, srcPort, serverNodeInfo.nodeIP, udpService.Spec.Ports[0].NodePort) clientPod.Spec.Containers[0].Command = []string{"/bin/sh", "-c", cmd} clientPod.Spec.Containers[0].Name = podClient fr.PodClient().CreateSync(clientPod) // Read the client pod logs logs, err := e2epod.GetPodLogs(cs, ns, podClient, podClient) framework.ExpectNoError(err) framework.Logf("Pod client logs: %s", logs) // Add a backend pod to the service in the other node ginkgo.By("creating a backend pod " + podBackend1 + " for the service " + serviceName) serverPod1 := newAgnhostPod(podBackend1, "netexec", fmt.Sprintf("--udp-port=%d", 80)) serverPod1.Labels = udpJig.Labels serverPod1.Spec.NodeName = serverNodeInfo.name fr.PodClient().CreateSync(serverPod1) // Waiting for service to expose endpoint. err = validateEndpointsPorts(cs, ns, serviceName, portsByPodName{podBackend1: {80}}) framework.ExpectNoError(err, "failed to validate endpoints for service %s in namespace: %s", serviceName, ns) // Check that the pod receives the traffic // UDP conntrack entries timeout is 30 sec by default ginkgo.By("checking client pod connected to the backend 1 on Node IP " + serverNodeInfo.nodeIP) time.Sleep(30 * time.Second) logs, err = e2epod.GetPodLogs(cs, ns, podClient, podClient) framework.ExpectNoError(err) framework.Logf("Pod client logs: %s", logs) if !strings.Contains(string(logs), podBackend1) { framework.Failf("Failed to connecto to backend 1") } // Create a second pod ginkgo.By("creating a second backend pod " + podBackend2 + " for the service " + serviceName) serverPod2 := newAgnhostPod(podBackend2, "netexec", fmt.Sprintf("--udp-port=%d", 80)) serverPod2.Labels = udpJig.Labels serverPod2.Spec.NodeName = serverNodeInfo.name fr.PodClient().CreateSync(serverPod2) // and delete the first pod framework.Logf("Cleaning up %s pod", podBackend1) fr.PodClient().DeleteSync(podBackend1, metav1.DeleteOptions{}, framework.DefaultPodDeletionTimeout) // Check that the second pod keeps receiving traffic // UDP conntrack entries timeout is 30 sec by default ginkgo.By("checking client pod connected to the backend 2 on Node IP " + serverNodeInfo.nodeIP) time.Sleep(30 * time.Second) logs, err = e2epod.GetPodLogs(cs, ns, podClient, podClient) framework.ExpectNoError(err) framework.Logf("Pod client logs: %s", logs) if !strings.Contains(string(logs), podBackend2) { framework.Failf("Failed to connecto to backend 2") } }) ginkgo.It("should be able to preserve UDP traffic when server pod cycles for a ClusterIP service", func() { // Create a NodePort service udpJig := e2eservice.NewTestJig(cs, ns, serviceName) ginkgo.By("creating a UDP service " + serviceName + " with type=ClusterIP in " + ns) udpService, err := udpJig.CreateUDPService(func(svc *v1.Service) { svc.Spec.Type = v1.ServiceTypeClusterIP svc.Spec.Ports = []v1.ServicePort{ {Port: 80, Name: "udp", Protocol: v1.ProtocolUDP, TargetPort: intstr.FromInt(80)}, } }) framework.ExpectNoError(err) // Create a pod in one node to create the UDP traffic against the NodePort service every 5 seconds ginkgo.By("creating a client pod for probing the service " + serviceName) clientPod := newAgnhostPod(podClient, "") clientPod.Spec.NodeName = clientNodeInfo.name cmd := fmt.Sprintf(`date; for i in $(seq 1 300); do echo "$(date) Try: ${i}"; echo hostname | nc -u -w 5 -p %d %s %d; echo; done`, srcPort, udpService.Spec.ClusterIP, udpService.Spec.Ports[0].Port) clientPod.Spec.Containers[0].Command = []string{"/bin/sh", "-c", cmd} clientPod.Spec.Containers[0].Name = podClient fr.PodClient().CreateSync(clientPod) // Read the client pod logs logs, err := e2epod.GetPodLogs(cs, ns, podClient, podClient) framework.ExpectNoError(err) framework.Logf("Pod client logs: %s", logs) // Add a backend pod to the service in the other node ginkgo.By("creating a backend pod " + podBackend1 + " for the service " + serviceName) serverPod1 := newAgnhostPod(podBackend1, "netexec", fmt.Sprintf("--udp-port=%d", 80)) serverPod1.Labels = udpJig.Labels serverPod1.Spec.NodeName = serverNodeInfo.name fr.PodClient().CreateSync(serverPod1) // Waiting for service to expose endpoint. err = validateEndpointsPorts(cs, ns, serviceName, portsByPodName{podBackend1: {80}}) framework.ExpectNoError(err, "failed to validate endpoints for service %s in namespace: %s", serviceName, ns) // Check that the pod receives the traffic // UDP conntrack entries timeout is 30 sec by default ginkgo.By("checking client pod connected to the backend 1 on Node IP " + serverNodeInfo.nodeIP) time.Sleep(30 * time.Second) logs, err = e2epod.GetPodLogs(cs, ns, podClient, podClient) framework.ExpectNoError(err) framework.Logf("Pod client logs: %s", logs) if !strings.Contains(string(logs), podBackend1) { framework.Failf("Failed to connecto to backend 1") } // Create a second pod ginkgo.By("creating a second backend pod " + podBackend2 + " for the service " + serviceName) serverPod2 := newAgnhostPod(podBackend2, "netexec", fmt.Sprintf("--udp-port=%d", 80)) serverPod2.Labels = udpJig.Labels serverPod2.Spec.NodeName = serverNodeInfo.name fr.PodClient().CreateSync(serverPod2) // and delete the first pod framework.Logf("Cleaning up %s pod", podBackend1) fr.PodClient().DeleteSync(podBackend1, metav1.DeleteOptions{}, framework.DefaultPodDeletionTimeout) // Check that the second pod keeps receiving traffic // UDP conntrack entries timeout is 30 sec by default ginkgo.By("checking client pod connected to the backend 2 on Node IP " + serverNodeInfo.nodeIP) time.Sleep(30 * time.Second) logs, err = e2epod.GetPodLogs(cs, ns, podClient, podClient) framework.ExpectNoError(err) framework.Logf("Pod client logs: %s", logs) if !strings.Contains(string(logs), podBackend2) { framework.Failf("Failed to connecto to backend 2") } }) }) </ins>

so we can do things like to reproduce this sort of thing. matt did you get around to the 'try-till-it-fails' scenario?
I haven't been able to get it to fail on kind yet, -- 100 cycles and 0 failures on a three-node kind cluster! I'm letting the loop run for a few hours, will check back in when I have that data. Is it worth running this test on a non-kind environment?
I ran for several hours on my mac, through 1144 iterations no error yet. My conclusion: this test isn't flaky on KinD. Possible next step: run a large number of iterations of the same test on a GKE cluster.
Flakes on kind are also happening:
not many since the portmap fix on the 14th though ( / ) <img width="1111" alt="Screen Shot 2020-07-23 at 11 45 46 PM" src="https://user-"
Agree - it's much better than it was before.
I was analyzing this failure, the problem is that kind jobs does not have verbosity enough in kube-proxy We can see that the packet was never replied, this is the NAT entry from client -nodePort in the node with the client This is the same flow, but in the node with the server and the pod-server-1 seems to be healthy we need more logs :eyeglasses:
We have an open ticket to bump those in CI, it's just been lost in the noise. We should still do that. Antonio Ojea <:
If you are busy just assign it to me, I'll need this verbosity soon :)
It's in the kind repo, shouldn't be too hard to find, we have ~60 open atm. Antonio Ojea <:
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
I made another attempt to reproduce these failures locally on KinD, but still no success. Here's what I did: create a KinD cluster run two instances of sonobuoy at the same time This took ~4 hours to run; all the kube tests passed and sonobuoy reported all successes.