Question: kubectl get pods "${kube_flags[@]}" -lname=redis-master | grep -q 'redis-master' [ ! $(delete pods --all pods -l name=redis-master) ]   # not --all and label selector together kubectl delete --all pods "${kube_flags[@]}" # --all remove all the pods <ins> kubectl create -f examples/guestbook/redis-master.json "${kube_flags[@]}" kubectl create -f examples/redis/redis-proxy.yaml "${kube_flags[@]}" kubectl get pods redis-master redis-proxy "${kube_flags[@]}" kubectl delete pods redis-master redis-proxy # delete multiple pods at once howmanypods="$(kubectl get pods  -o template -t "{{ len .items }}" "${kube_flags[@]}")" [ "$howmanypods" -eq 0 ] kubectl create -f examples/guestbook/redis-master.json "${kube_flags[@]}" kubectl create -f examples/redis/redis-proxy.yaml "${kube_flags[@]}" kubectl stop pods redis-master redis-proxy # stop multiple pods at once </ins> howmanypods="$(kubectl get pods  -o template -t "{{ len .items }}" "${kube_flags[@]}")" [ "$howmanypods" -eq 0 ]

Currently, it could not delete multiple pods at once, need to delete them one by one, it's very inconvenient if I have tons of pods to delete. Unlike docker, docker can multiple stop/rm containers/images at once.
This is possible using label selectors as of , which is better if you have tons of pods.
I agree that using label selector is better to delete all of pods under the label. But to delete multile pods at once is still usable: I just want to delete 1/2 pods under the label, label selector will not work selector could not span over other selectors or pods Another reason is docker rm/stop x y z works, I will wonder why kubectl cannot. This is not urgent, but I hope it can be implemented.
I agree. Thanks for the suggestion.
Would this be a suitable task for someone new to k8s development? If so, I'd be willing to take a look at this, if someone could provide a couple of hints where to start.
Further to my last message, I have a functional work in progress I would like to share, if anyone is interested?
PRs are welcome. See
Yes, this is a reasonable place to start. Since its in the client, you won't get much orientation to the core server code, but it will be a good place to orient to the client side API. There us also prior art that you can model off of. The kubectl create ... command already supports multiple resources. A good command to adapter would be the 'label' command. You can find it in Many thanks in advance! Brendan On Feb 19, 2015 8:21 AM, "Eric Tune" wrote:


Question: func (c *connection) RemoveStreams(streams ...httpstream.Stream) { c.streamLock.Lock() for _, stream := range streams { <del> delete(c.streams, stream.Identifier()) </del> <ins> // It may be possible that the provided stream is nil if timed out. if stream != nil { delete(c.streams, stream.Identifier()) } </ins> } c.streamLock.Unlock() }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --Since upgrading to Kubernetes v1.19.11 (from v1.19.10) we see occasional panics in kubelet related to portforward streams: This issue seem to be introduced with should not panic and crash. We have not been able to successfully force this to happen, but it has happened 41 times in the last 2 weeks across 7 clusters (~100 nodes). Kubernetes version (use ):Cloud provider or hardware configuration: Self managed clusters on AWS. OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /sig node /cc
Hey thank you for the report! :pray: Uhh, thats unfortunate. The panic occurs there on L101: It looks like because works. The stack trace indicates the same: A on is a no-op, so would not be harmful. does also not seem to be , a crash in would have been visible in the stack trace. It may be possible that something assigned to the stream interface. Which means a check for will return , but accessing its fields is causing a segmentation fault. This does not explain why the crash is not visible in the trace.
With a quick look at the streaming code it's likely to be triggered by a timeout in waiting for streams: If the above case is hit then it goes down to , and if the timeout occurs before then it means that either or And then it attempts to remove the streams where at least one is : So it probably makes sense to check for either before passing to or inside the method where it crashes. This theory lines up with our kubelet logs, where I can see this just before the panic (didn't notice before):
Thank you for the summary working on a fix in
/triage accepted /assign
do you use dockershim within your cluster?
Yes, we still use dockershim.
Good, that's also why the panic happens within the kubelet.
Just to make sure I understand the scope of this issue, this only affects things like or other things that use the API, correct?
Exactly :+1:


Question: return nil, fmt.Errorf("failed to start container %q: %v", r.ContainerId, err) } <del> ds.performPlatformSpecificContainerForContainer(r.ContainerId) </del> return &runtimeapi.StartContainerResponse{}, nil }

What would you like to be : Periodically remove cleanupInfos for old containers from dockershim's map. One way of doing that would be to just forget about s older than say an hour, as they should always be short-lived. Why is this needed: introduced a new in-memory map for the dockershim, that could potentially (in pathological cases) cause memory leaks - for containers that use GMSA cred specs, get created successfully, but then never get started nor removed. This might be okay for the alpha release of the GMSA feature, but should be addressed in the beta. See
/assign wk8
/sig windows
/milestone v1.15
got moved out. let's move this issue with it as well /milestone v1.16
Hello! I'm bug triage lead for the 1.16 release cycle and considering this issue is tagged for 1.16, but not updated for a long time, I'd like to check its status. The code freeze is starting on August 29th (about 1.5 weeks from now), which means that there should be a PR ready (and merged) until then. Do we still target this issue to be fixed for 1.16?
: yup we'd love to have this in 1.16, still waiting for kind reviewers on :) ...


Question: import ( corev1 "k8s.io/api/core/v1" discovery "k8s.io/api/discovery/v1alpha1" <ins> apiequality "k8s.io/apimachinery/pkg/api/equality" </ins> "k8s.io/apimachinery/pkg/api/errors" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" corev1client "k8s.io/client-go/kubernetes/typed/core/v1"

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: EndpointSlice for masters was not created after enabling EndpointSlice on a pre-existing cluster. Only those EndpointSlices managed by kube-controller-manager were created. What you expected to happen: EndpointSlice for masters should be created after enabling EndpointSlice on a pre-existing cluster, the same as the EndpointSlices managed by kube-controller-manager. How to reproduce it (as minimally and precisely as possible): kubeadm to create a cluster: the EndpointSlice feature by editing the kube-apiserver, kube-controller-manager, kube-proxy manifest kube-apiserver and kube-controller-manager is restarted, the EndpointSlice in namespace will be missing. Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools:Network plugin and version (if this is a network-related bug): Others:
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/remove-triage unresolved /assign


Question: fs.BoolVar(&o.config.IPVS.StrictARP, "ipvs-strict-arp", o.config.IPVS.StrictARP, "Enable strict ARP by setting arp_ignore to 1 and arp_announce to 2") fs.BoolVar(&o.config.IPTables.MasqueradeAll, "masquerade-all", o.config.IPTables.MasqueradeAll, "If using the pure iptables proxy, SNAT all traffic sent via Service cluster IPs (this not commonly needed)") <del> fs.BoolVar(&o.config.EnableProfiling, "profiling", o.config.EnableProfiling, "If true enables profiling via web interface on /debug/pprof handler.") </del> <ins> fs.BoolVar(&o.config.EnableProfiling, "profiling", o.config.EnableProfiling, "If true enables profiling via web interface on /debug/pprof handler. This parameter is ignored if a config file is specified by --config.") </ins> fs.Float32Var(&o.config.ClientConnection.QPS, "kube-api-qps", o.config.ClientConnection.QPS, "QPS to use while talking with kubernetes apiserver") <del> fs.Var(&o.config.DetectLocalMode, "detect-local-mode", "Mode to use to detect local traffic") </del> <ins> fs.Var(&o.config.DetectLocalMode, "detect-local-mode", "Mode to use to detect local traffic. This parameter is ignored if a config file is specified by --config.") </ins> } // NewOptions returns initialized Options

Start service with a config file without specifying the argument in the config file, instead, pass the value via the CLI args, the value is just ignored, for example, I have a config file as this, NOTE: I have not specified the in the config file, but give a value for the via the CLI, the proxy mode is still set as after the service is started. The reason for this is that CLI args is parsed before the loading of the config file, if the config file haven't specified the or leave it as empty, then it will be use the default value instead. Two choices from my side, Add a note somewhere, maybe where the option is defined, just say that the CLI args will be ignored when the config file is provided regardless the option is specified in the config file or not (this should be noted for lots of options). Merge the value passed from CLI with the config file if the option has an empty value, but we need to consider the backward compatibility. and check which mode is effective. No response_ vanilla deployment with code in the upstream. <details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig network Hey Tim, WDYT? /assign
Heh, this is a recurrent topic :smile: , you did a great analysis: I think that explains here why merging flags and config overriding defaults is not a desirable option (the whole thread is worth reading) I'm in favor of documenting the behavior
thanks for showing the earlier thread, I have created a simple PR to document this in the CLI option.


Question: package priorities import ( <ins> "os/exec" </ins> "reflect" "sort" "strconv"

I just noticed you forgot to register the image locality priority function in in I also missed it when I reviewed your PR. I have fixed this in . But it would be great if you could figure out a mechanism that would ensure that all predicate/priority functions that are tested (you can assume they will always live under ) are registered, and fail the test if not. That will allow us to prevent this mistake in the future.
Thanks! Will fix this soon!
Seems need to wait for in before I send PR? Otherwise this test will always fail ...


Question: opencontrail_kubernetes_tag: '$(echo "$OPENCONTRAIL_KUBERNETES_TAG" | sed -e "s/'/''/g")' opencontrail_public_subnet: '$(echo "$OPENCONTRAIL_PUBLIC_SUBNET" | sed -e "s/'/''/g")' e2e_storage_test_environment: '$(echo "$E2E_STORAGE_TEST_ENVIRONMENT" | sed -e "s/'/''/g")' <ins> enable_hostpath_provisioner: '$(echo "$ENABLE_HOSTPATH_PROVISIONER" | sed -e "s/'/''/g")' </ins> EOF cat <<EOF >/etc/salt/minion.d/log-level-debug.conf

feature request Enables easier testing for pv/pvc, and all derivatives like pet sets. The only alternative is to use manually created pv of local type. I have time to work on it myself and created this feature to collect feedback/ideas on what provider should be enabled by default.
Since this was merged should we close this issue?


Question: <ins> {% if pillar.get('enable_node_autoscaler', '').lower() == 'true' %} </ins> {% set params = pillar['autoscaler_mig_config'] -%} { "kind": "Pod",

For some reason they stopped working and we even don't create manifest file for cluster autoscaler. I used the following command to start the cluster:
It's possible it's related to


Question: // newAgnhostPod returns a pod that uses the agnhost image. The image's binary supports various subcommands // that behave the same, no matter the underlying OS. func newAgnhostPod(name string, args ...string) *v1.Pod { <ins> zero := int64(0) </ins> return &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name: name, }, Spec: v1.PodSpec{ <ins> TerminationGracePeriodSeconds: &zero, </ins> Containers: []v1.Container{ { Name:  "agnhost",

Which jobs are flaking: ci-kubernetes-e2e-gce-scale-correctness Which test(s) are flaking: [sig-network] Services should be able to preserve UDP traffic when server pod cycles for a NodePort service Testgrid link: https://k8s- Reason for failure: /sig network /assign
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
CRI-O CI is also hitting this failure. We have been unable to reproduce locally. our CI runs on aws
hmm, I only see 1 failure out of 4 and seems that the connectivity didn't work at all
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
/assign
GitHub didn't allow me to assign the following users: JacobTanenbaum. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc Saw the test flaking recently: Wondering if this is just a timeout issue.


Question: // newAgnhostPod returns a pod that uses the agnhost image. The image's binary supports various subcommands // that behave the same, no matter the underlying OS. func newAgnhostPod(name string, args ...string) *v1.Pod { <ins> zero := int64(0) </ins> return &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name: name, }, Spec: v1.PodSpec{ <ins> TerminationGracePeriodSeconds: &zero, </ins> Containers: []v1.Container{ { Name:  "agnhost",

Maybe we can increase the timeout and/or switch to a TCP based test:
the test should use UDP, and UDP is unreliable by definition ... however is curious that most of the failures happen because it is not able to communicate with the first pod I think that is something that should be investigated, I've analyzed some occurences and it works fine for the second pod, however, the UDP losses should be totally random
Hi everyone, yeah so.. collecting some data on this... using a two node kind cluster, locally, with the antrea plugin, it passes using a 4 node AWS cluster with the antrea CNI plugin, it failed using a two node kidn cluster locally with calico, it passes So I guess the best reproducer is to run these tests, for now, in the cloud.... Now that I have a reproducer for this maybe can live debug it some time next wk with one of yall :)
bear in mind that this test is connecting to a nodePort in the node IP, the test is functionally correct, it may be flaky due to the unreliable UDP nature or due to the pod scheduling, but if is failing constantly it has to be environmental.
Second failure after two weeks of green 5k runs:
yep, :/ https://k8s-|UDP /assign
/reopen I'm afraid this is going to be challenging :smile: The difference now is that we are testing NodePort and ClusterIP If both fail, we can blame the test for sure, If only NodePort fails, we need to check the job, if GCE or KIND If only ClusterIP fails, ... :thinking:
Reopened this issue. <detailsIn response to : /reopen I'm afraid this is going to be challenging :smile: Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
We're hitting this really frequently in 5k-node tests. So I started looking a little bit into that.


Question: // newAgnhostPod returns a pod that uses the agnhost image. The image's binary supports various subcommands // that behave the same, no matter the underlying OS. func newAgnhostPod(name string, args ...string) *v1.Pod { <ins> zero := int64(0) </ins> return &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name: name, }, Spec: v1.PodSpec{ <ins> TerminationGracePeriodSeconds: &zero, </ins> Containers: []v1.Container{ { Name:  "agnhost",

I don't have full understanding of what exactly is happening, but I already see a couple things that may be causing issues: when creating the first pod, you wait for corresponding endpoint for that being created; but that doesn't mean that kube-proxy was actually programmed correctly at that time (in general it isn't); given iptables programming is rate-limitted, it may easily happen that actually programming happens 10s+ later That explains some delays, but it doesn't explain e.g. this run: where iptables were programmed at but the test didn't respond until If someone more familiar with networking stuff could look into it, it would be great. - can you help triaging ^^ for deletions it's even worse, because you don't even wait for endpoints to be there (i.e. Endpoints can still contain the old pod when the test start pinging to look for the second one) So I claim that timeouts are visibly to low in this test, but I suspect it's not everything. I will send out the first PR that tries to improve it in couple minutes.
thanks wojtek, have been slammed but Ill try to collect more data on it this wknd as well.
It seems that my PR didn't really help much. We should dig deeper, but I would need someone with more networking expertise to look a bit into
Actually - it seems only failures from one group are on non . So maybe that's actually related. The non-kind related ones look like:
OK - for this one, I actually know what is happening - I will send out a PR with a fix for this too.
for the KIND jobs already increased the resource requests on some of the jobs, there are issues on KIND jobs with kube-proxy not able to take the iptables lock and not being able to create the iptables rules for the services until the next resync
is to fix the flakes I mentioned above; all other flakes seem to be kind-oriented to me (which I'm not super familiar with).
We did not adjust the CPU on all jobs yet, pending how to manage that on prow ... I do see just a few other flakes at the bottom of the triage dashboard that look like timeouts instead.
Actually - almost all failures are effectively timeout. The top one that is is actually also a timeout. Didn't look into kind failures, but this may easily be related to iptables issues.


Question: // newAgnhostPod returns a pod that uses the agnhost image. The image's binary supports various subcommands // that behave the same, no matter the underlying OS. func newAgnhostPod(name string, args ...string) *v1.Pod { <ins> zero := int64(0) </ins> return &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name: name, }, Spec: v1.PodSpec{ <ins> TerminationGracePeriodSeconds: &zero, </ins> Containers: []v1.Container{ { Name:  "agnhost",

After my second deflaking PR, we seem to be in the stage where: tests are failing on dozens of different things (probably iptables-issues related) tests are failing only on NodePort tests on "Failed to connecto to backend 1" which it would be great if someone else could look into (kube-proxy seemed to be programmed correctly based on logs)
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
This has failed again - Linking to specific part of log where timeout occurred .... we see this
So reviewing the above logs note how in STEP: checking client pod connected to the backend 1 on Node IP 10.138.0.5 28 attempts are made all in the 25th second of the 2 minute at 11am Should this Step back off and wait a few seconds in between tries? So looking at this code we see the following
From reading ... the intention expressed in the test code looks good to me. It is not clear to me why there is a mismatch between the 28 tries in the POD Client Logs and the expressed intention running the logContainsFn(podBackend1) every 5 seconds still digging ....
we are trying to catch one failure in this PR that has additional debugging, it dumps the conntrack table of the nodes on a failure
for those following the issue, we've extra debugging to the test so it dumps the whole conntrack table once it finish. We should look that there is NO stale entries, connections from the pod client to the deleted endpoint) and there is conntrack entries to the new endpoint from the pod client.
- we have runs with the conntrack dump now, e.g. this one: would you be able to take a look into it?
not yet :/ , but I think tomorrow can have some time
cc
For anyone reproducing this issue btw, you need a multinode k8s cluster (it isnt doable on kind i guess bc less likeliehood of network failures when running local) -
Curious about two potential hypotheses to fix this, have we looked into either of these? 1) can we confirm that is turned on/off? 2) is it possible that scaling the number of coredns replicas could help ?


Question: // newAgnhostPod returns a pod that uses the agnhost image. The image's binary supports various subcommands // that behave the same, no matter the underlying OS. func newAgnhostPod(name string, args ...string) *v1.Pod { <ins> zero := int64(0) </ins> return &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name: name, }, Spec: v1.PodSpec{ <ins> TerminationGracePeriodSeconds: &zero, </ins> Containers: []v1.Container{ { Name:  "agnhost",

3) Would simulating this in kind by overloading coredns be a possibility as a way to make a more reproducible bug model?
regarding 1) kubeadm does not ship nodeLocalDNSCache I don't think GCE kube-up does by default either
ok, so now im curious about (2) - i.e. what if we just bumped coredns replica count?
interesting, :thinking: why do you think is a DNS thing? the netcat command that polls the services use the IP addresses not the hostname. I think that it has to be related to some iptables/conntrack race or similar,at least in KIND jobs we know that we have an iptables problem that affects network tests, that has to be fixed before this because it affect this test You can instrument the test locally with this and dump the state of iptables and conntrack on failure if you are able to reproduce it
i figured DNS bc i believe its known that conntrack and DNS at scale can get very messy, right ?
one other curious thing about this test is that we only create 2 pods in the recycling event for the svc. that likely means that any real error is hidden by chance..... If we created a 3rd pod, then we might potentially double the failure rate thus making the flakyness go away and expose the real underlying issue .
There's an invocation of dumpConntrack(cs) here [sig-network] Conntrack should be able to preserve UDP traffic when server pod cycles for a ClusterIP service
is updating the logging to run in an infinite loop ..
This is improving in KIND with the new changes we rolled out yesterday, let's see how it evolves this week
- which changes ? Are you saying this is no longer reproducible on Kind?
there was a lot of noise in the CI signal, especially in KIND jobs due to the CNI portmap bug we finally solved yesterday now we should have a better signal and we can go to the bottom because probably the failure will be "legit" and not a consequence, so this comment should apply
Hi ! I ran this a few times on kind and it passed, but I see , so I'm going to try the loop approach you suggested. Here's what I'm using to test:
quick note, <- potential motivation to possibly have runnable on some subset of tests in upstream eventually, i.e.


Question: // newAgnhostPod returns a pod that uses the agnhost image. The image's binary supports various subcommands // that behave the same, no matter the underlying OS. func newAgnhostPod(name string, args ...string) *v1.Pod { <ins> zero := int64(0) </ins> return &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name: name, }, Spec: v1.PodSpec{ <ins> TerminationGracePeriodSeconds: &zero, </ins> Containers: []v1.Container{ { Name:  "agnhost",

so we can do things like to reproduce this sort of thing. matt did you get around to the 'try-till-it-fails' scenario?
I haven't been able to get it to fail on kind yet, -- 100 cycles and 0 failures on a three-node kind cluster! I'm letting the loop run for a few hours, will check back in when I have that data. Is it worth running this test on a non-kind environment?
I ran for several hours on my mac, through 1144 iterations no error yet. My conclusion: this test isn't flaky on KinD. Possible next step: run a large number of iterations of the same test on a GKE cluster.
Flakes on kind are also happening:
not many since the portmap fix on the 14th though ( / ) <img width="1111" alt="Screen Shot 2020-07-23 at 11 45 46 PM" src="https://user-"
Agree - it's much better than it was before.
I was analyzing this failure, the problem is that kind jobs does not have verbosity enough in kube-proxy We can see that the packet was never replied, this is the NAT entry from client -nodePort in the node with the client This is the same flow, but in the node with the server and the pod-server-1 seems to be healthy we need more logs :eyeglasses:
We have an open ticket to bump those in CI, it's just been lost in the noise. We should still do that. Antonio Ojea <:
If you are busy just assign it to me, I'll need this verbosity soon :)
It's in the kind repo, shouldn't be too hard to find, we have ~60 open atm. Antonio Ojea <:
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
I made another attempt to reproduce these failures locally on KinD, but still no success. Here's what I did: create a KinD cluster run two instances of sonobuoy at the same time This took ~4 hours to run; all the kube tests passed and sonobuoy reported all successes.


Question: // newAgnhostPod returns a pod that uses the agnhost image. The image's binary supports various subcommands // that behave the same, no matter the underlying OS. func newAgnhostPod(name string, args ...string) *v1.Pod { <ins> zero := int64(0) </ins> return &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name: name, }, Spec: v1.PodSpec{ <ins> TerminationGracePeriodSeconds: &zero, </ins> Containers: []v1.Container{ { Name:  "agnhost",

Would love to help out in reproing this, I'm at a loss for good next step so any suggestions welcome!
that's the problem with these flakes, they are not easy to repro, and with this low rate of failures it may be caused for combination of running tests, environmental problems (cpu, memory, ...) , ... I always start for checking the current failures and try to find patterns
Let's take a look to one occurrence The tests fails because can't connect to the first pod If we check the code of the tests, we can see that it always uses the port as source port, we can use that to identify the nodePort assigned interesting thing here, we are source nating the pod ip, and we don't see any other conntrack entry The test verifies it has endpoints however, kube-proxy is using endpointlsices since this version but it has no entries so the iptables rules to forward the traffic are not there Seems we have to adapt this function in the e2e framework to use endpointslices cc: EDIT seems that this function is being used by all the network tests that can cause flakes if the endpoint is updated before the endpointslice (we switched recently kube-proxy to use the later)
Great catch. Ideally this function should validate if both Endpoint and appropriate EndpointSlice objects(s) exist.
Actually - let me take it and send a PR for it.
is out for review
Interesting failures here It fails to connect to the first pod
Hmm - I took a quick look at kube-proxy logs and these look fine to me....
I clicked too fast, sorry ... I was looking at the wrong service and I though I've found something
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.


Question: // newAgnhostPod returns a pod that uses the agnhost image. The image's binary supports various subcommands // that behave the same, no matter the underlying OS. func newAgnhostPod(name string, args ...string) *v1.Pod { <ins> zero := int64(0) </ins> return &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name: name, }, Spec: v1.PodSpec{ <ins> TerminationGracePeriodSeconds: &zero, </ins> Containers: []v1.Container{ { Name:  "agnhost",

just saw it as well
can you please add links to the job with the failures, I just merged new code to improve the logging 2 days ago
ok, here there is a new occurrence and with the new debug logs is easier to follow has unreplied conntrack entries: looking at the other side The iptables rules are there but no conntrack entries related to 10.244.2.32
In I revert PR with debug info as I think that it likely caused significant memory growth in scalability tests, but we have one run that has finished without being killed by OOMKiller (I increased memory limit for that jobs as a mitigation). Maybe logs from this run will be useful for debugging:
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
Two more flakes today:
It seems that the packet never gets DNATed in the receiving host, so it does not reach the pod in the node. There is the same test that is using ClusterIP instead of NodePort that is more reliable. This is being a tough one
/triage accepted
Yesterday's flake - I don't think I'm seeing the log messages from the latest log PR you made ? Or are the prow logs not logging at a high-enough level?
Logs are there, please bear with me: The error on the test is: click on the job failure and you'll see a larger output where you can see the server pod name and get the node where the pod is running: and the client-pod name and its node: and the flow we are looking for: with the node-name we can get the kube-proxy logs there, clicking in artifacts: and you can see the "new" conntrack log entries : and in the client: wow, there is a kernel bug there , can it be related? :thinking:
wohoo - some progress :) I'm not an expert here, but I think it can be? Adding here - they may be able to confirm/reject the hypothesis.
This kernel panic may be a coincidence, maybe this is the reason
Adding some recent failures:
Spotted another flake today:
Another flake in the 5k job:


Question: // NewDelayingQueueWithCustomClock constructs a new named workqueue // with ability to inject real or fake clock for testing purposes func NewDelayingQueueWithCustomClock(clock clock.Clock, name string) DelayingInterface { <ins> return newDelayingQueue(clock, NewNamed(name), name) } func newDelayingQueue(clock clock.Clock, q Interface, name string) *delayingType { </ins> ret := &delayingType{ <del> Interface:       NewNamed(name), </del> <ins> Interface:       q, </ins> clock:           clock, heartbeat:       clock.NewTicker(maxWait), stopCh:          make(chan struct{}),

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Current constructors: Could we export a new constructor like to inject custom queue. Why is this needed: In my project, i want to create a with a custom queue based on workqueue.Interface but have no way to do that.
/sig api-machinery
thanks for the issue! We've reviewed it in the bug triage, and the consensus was that to ask you to send a PR with this and discuss it there. Would you be able to do it? thanks!
Yes, i will do that.


Question: t.Errorf("Service must be namespace scoped") } oldService := makeValidService() <del> newService := makeValidService() </del> <ins> oldService.Spec.Type = api.ServiceTypeLoadBalancer </ins> oldService.ResourceVersion = "4" <del> newService.ResourceVersion = "4" </del> <ins> oldService.Spec.SessionAffinity = "None" newService := oldService.DeepCopy() </ins> newService.Spec.SessionAffinity = "ClientIP" newService.Status = api.ServiceStatus{ LoadBalancer: api.LoadBalancerStatus{

xref I think this may be one of the few cases where we can actually change the behavior in a "breaking" way. User creates service with , is empty Controller or user updates the status subresource, sets (not wiped or validated, should be) User updates in any way, is wiped It's clear that the is not meant to be set when is not "LoadBalancer". The question is: 1) Fail validation? 2) Silently discard the value? 3) Warn and then discard the value? 4) Just warn and leave it. Because we allowed it in the past, but it is so volatile, I lean towards 3. and - the people most familair with this type. Edit: PrepareForUpdate() runs before warnings. So: (1) is easy (2) is easy (4) is slightly less easy than (2) (3) is slightly less easy than (4)
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
OMG change it to clusterIP as expected Status is wiped update status as ClusterIP ... and if I edit the service back to loadbalancer it keeps the status??
/kind bug
/retitle Service can be set when !=
We talked about this one at the SIG Network meeting today. suggested we use the feature gate for deprecation option to phase the current (problematic) functionality out.
/triage accept
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /triage accept Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: // +listType=map // +listMapKey=name // +optional <del> Variables []Variable `json:"variables" patchStrategy:"merge" patchMergeKey:"name" protobuf:"bytes,7,rep,name=variables"` </del> <ins> Variables []Variable `json:"variables,omitempty" patchStrategy:"merge" patchMergeKey:"name" protobuf:"bytes,7,rep,name=variables"` </ins> } type MatchCondition v1.MatchCondition

spec.variables should be marked omitempty. I'm trying to create a validating admission policy that uses in v1.27 as follows: I'm setting : to "Strict" in CreateOptions so that the server will reject requests with unknown fields but it complains that the don't exist in v1.27 and it does make sense because this field was introduced in v1.28: I'm using a 1.28 client.
/sig api-machinery
/assign
/assign /triage accepted
Thanks for raising this! /cc


Question: }, }, { <del> name: "StorageCapacity changed", </del> <ins> name: "TokenRequests invalidated", </ins> modify: func(new *storage.CSIDriver) { <del> new.Spec.StorageCapacity = &notStorageCapacity </del> <ins> new.Spec.TokenRequests = []storage.TokenRequest{{Audience: gcp}, {Audience: gcp}} </ins> }, }, { <del> name: "TokenRequests invalidated", </del> <ins> name: "invalid nil StorageCapacity", </ins> modify: func(new *storage.CSIDriver) { <del> new.Spec.TokenRequests = []storage.TokenRequest{{Audience: gcp}, {Audience: gcp}} </del> <ins> new.Spec.StorageCapacity = nil </ins> }, }, }

Up to 1.21, that field is immutable. That means in practice, a CSIDriver object has to be deleted and then re-created with a different value instead. Mutation via delete/created is unnecessarily complex and as the only consumer of that field (the kube-scheduler) can handle mutation just fine (it'll always ask the informer cache for the current value), it's better to allow mutation.
/sig storage /assign
This is independent of allowing other fields to become mutable, but that might also make sense.
/triage accepted


Question: return iface.Name, nil } <del> func setUpAllInterfaces() error { interfaces, err := net.Interfaces() if err != nil { return err } for _, netIf := range interfaces { setUpInterface(netIf.Name) // ignore errors } return nil } </del> func setUpInterface(ifName string) error { glog.V(3).Infof("Enabling hairpin on interface %s", ifName) ifPath := path.Join(sysfsNetPath, ifName)

From another issue, said: DockerManager.SyncPod() calls the network Plugin and then sets hairpin mode unilaterally. This is wrong. We should either pass the bool down to here or else we should get rid of the default no-op plugin and use the non-existence of a plugin as a flag to control this. This should be a relatively easy one for people who want to contribute but don't know where to start.
I volunteer myself to do this. Actually I have done something about kubernetes network plugin so guess I can do this. I will start to do this when I am on the office tomorrow.
Awesome. I'd advise starting with a sketch so we can all make sure it is a good approach, before jumping in with both feet. , TonyAdo wrote:
ok, I will start with a sketch as soon as possible.
I thought about this for a while: if we pass the bool down to here, of course we can check this flag, if it is set, we should set veth hairpin mode, but problem is that if user set this flag, but still use a custom network plugin just like you mentioned the ovs plugin, I am afraid we will still enable hairpin mode in all interfaces on this host, since we cannot find a peer interface. Is that correct?
Yeah, I think the better fix is to use the existence of a network plugin as an indicator. It sort of makes --configure-cbr0 redundant , TonyAdo wrote:
ok, I will try to get rid of the default no-op plugin and use the non-existence of a plugin as a flag to control this.
I decided to use the existence of a network plugin as an indicator, also I will get ride of the no-op network plugin, I am not going to add new flag to kubelet structure, if the user don't specify network plugin name, the networkPlugin within kubelet will be which just means non-existence of a plugin, and we have to check the existence in the docker manager to avoid nil pointer exception also for setting, I think if a custom network plugin exists, kubelet should not do this. How about this? Should I open a PR?
fyi, saw a hairpin pr today that might be related. ymmv.
what do you think about this?
What you described sounds right to me.


Question: return iface.Name, nil } <del> func setUpAllInterfaces() error { interfaces, err := net.Interfaces() if err != nil { return err } for _, netIf := range interfaces { setUpInterface(netIf.Name) // ignore errors } return nil } </del> func setUpInterface(ifName string) error { glog.V(3).Infof("Enabling hairpin on interface %s", ifName) ifPath := path.Join(sysfsNetPath, ifName)

I think the only way to see if there is some detail I am missing is to try it and see if anything pops up that would make this not work. , TonyAdo wrote:
I need to check which part is responsible of what, but what if the hairpin package was just a utility package that should be called by the plugin if it needs to set this up? As long as it is single and simple function call, doing so would avoid adding extra logic here and fix the issue. This is valid only if the aim is to have a "linux bridge plugin", an "OVS plugin", etc.
actually kubelet will call after get called, I think to make the network plugin can call hairpin package is not that easy, of course, I agree that network plugin should setup hairpin mode if it really needs, kubelet should not set hairpin mode unilaterally. Besides, kubelet is not going to provide an "linux bridge plugin" or "OVS plugin".
then the validity conditions of my comment are not met, so I see any simpler way. Thanks for checking! :)
still what do you think about disable hairpin completely and leave this to network plugin?
or if there is a custom network plugin, we don't setup hairpin mode, only do this in default.
Completely? It will break my use case (and anyone using Apache Kafka).
Only by default is what I understood from the previous comments, and LGTM.
Really sorry about that, but I really want to know are you using a kubelet network plugin in your use case? If no, I think we will still setup hairpin, as in default we will do this.
I don't use any plugin, so it is ok if hairpin is set up by default.
ok, I think this will not break your use case.
It's not really the network plugin that needs hairpinmode though, it's the pure-iptables-based kube-proxy code. It's possible the proxying code could change again in the future in such a way that hairpinmode wouldn't be needed any more. So it seems wrong to put that code into the plugin. makes it so that we set hairpinmode on the veth if it's attached to a bridge, but just silently do nothing if it's not.


Question: return iface.Name, nil } <del> func setUpAllInterfaces() error { interfaces, err := net.Interfaces() if err != nil { return err } for _, netIf := range interfaces { setUpInterface(netIf.Name) // ignore errors } return nil } </del> func setUpInterface(ifName string) error { glog.V(3).Infof("Enabling hairpin on interface %s", ifName) ifPath := path.Join(sysfsNetPath, ifName)

(Yes, we currently set hairpinmode even if you're using the userspace proxier, but AFAIK it's not needed in that case.)
IMHO... The proxy has the responsibility of service objects, while the kubelet is responsible of pod objects, including their networking. Also, the responsibility of hairpin must be at the same place as the code that creates the network interface, otherwise you have code managing a single object (the pod interface) spread over multiple programs, which will eventually lead to complex problems and a system harder to understand even for the end-user. Thus, it doesn't seems wrong to me that the kubelet does whatever is required to make the pod work in the environment of the cluster. While its true that the hairpin requirement is only for the proxy in iptables mode, a switch to this mode can happen at any time in the life of the cluster. And when it happens, its probably because the userland proxy becomes unresponsive, which means in a situation where the administrator is in a hurry to find a solution. Currently, you just change the proxy mode and restart the proxies. There's little chance the administrator will think about hairpin or restarting the kubelet in this case, so the kubelet should be reacting to this; which means the proxy and the kubelet of a given host should be able to communicate (AFAIK, this doesn't exist now). And all this is currently only to avoid setting up hairpin on the pod's port of standard Linux bridge.
Feel sure I'm missing something but... The CNI bridge plugin now has a flag, so kubenet can use that. Having done that, we can set false when the plugin name is non-blank.
Is this fixed now?
I apologise for leaving a somewhat vague answer, but I have been in that code recently. I don't think it's fixed.
, Bryan Boreham wrote:
Thanks for the PR. CNI Bridge plugin is expecting . Not . Hence the bridge plugin did not recognize it.
EDIT 2016-12-09: The chain below was broken by , so the issue can be closed again (remain closed) I suggest this should be re-opened. The chain of calls is: sets to , e.g. when using CNI calls something goes wrong and it sets the mode on everything. An example of "something goes wrong":


Question: } } <del> // RecommendedConfigOverrideFlags is a convenience method to return recommended flag names prefixed with a string of your choosing func RecommendedConfigOverrideFlags(prefix string) ConfigOverrideFlags { return ConfigOverrideFlags{ AuthOverrideFlags:    RecommendedAuthOverrideFlags(prefix), ClusterOverrideFlags: RecommendedClusterOverrideFlags(prefix), ContextOverrideFlags: RecommendedContextOverrideFlags(prefix), CurrentContext: FlagInfo{prefix + FlagContext, "", "", "The name of the kubeconfig context to use"}, Timeout:        FlagInfo{prefix + FlagTimeout, "", "0", "The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests."}, } } </del> // RecommendedContextOverrideFlags is a convenience method to return recommended flag names prefixed with a string of your choosing func RecommendedContextOverrideFlags(prefix string) ContextOverrideFlags { return ContextOverrideFlags{

Found in openshift, but I'm pretty sure the codepaths are the same. That means the token is getting ignored and in-cluster config is being used in preference to flags.
Is a similar issue?
Yep, kind of regression.
Sigh.
Yes, most likely the same cause.
I'm on it.
Unit test :)
Probably will want to cherrypick this into 1.5.
WIP in


Question: "//vendor/golang.org/x/oauth2/google:go_default_library", "//vendor/google.golang.org/api/compute/v1:go_default_library", "//vendor/google.golang.org/api/dns/v1:go_default_library", <ins> "//vendor/google.golang.org/api/googleapi:go_default_library", </ins> "//vendor/gopkg.in/gcfg.v1:go_default_library", ], )

/kind bug What happened: walteraa [6:55 AM] Hi, The controller manager of the federation using Google cloud DNS is crashing, the log showed is that: the hyperkube command is that: I have created this in the Google DNS Someone knows why the DNS zone query is getting the 404 error?? :disappointed: What you expected to happen: It should have created the missing domain. How to reproduce it (as minimally and precisely as possible): As per above. Anything else we need to know?: Environment: Kubernetes version (use ): 1.7.4 Cloud provider or hardware configuration**: OpenStack (irrelevant) OS (e.g. from /etc/os-release): irrelevant Kernel (e.g. ): irrelevant Install tools: irrelevant Others: irrelevant
The bug is related to this: The above code assumes that the provider code returns no error but an empty list if the domain does not exist, but instead the provider returns a 404. Need to fix the provider and/or the above code.
/sig federation
I am unlikely to have time to work on this soon, so if anyone else wants to, go right ahead.
/assign
An additional information: I have created a which is based on the credential JSON file created through instruction. After that, I have configured the Federation controller manager to mount this secret as a volume in the container and set the environment variable to the JSON file inside this volume. Before doing that, I was getting an authentication error so the google DNS provider couldn't be instantiated, and after doing the instructions that I mentioned before, this authentication error was solved but now I'm getting 404 error when Google DNS provider tries access the specified managed zone. The file that I have generated follow this structure:


Question: return ret } <del> // continuousEcho() uses the same connection for multiple requests, made to run as a goroutine so that // manipulations can be made to the service and backend pods while a connection is ongoing // it starts by sending a series of packets to establish conntrack entries and waits for a signal to keep // sending packts. It returns an error if the number of failed attempts is >= 5 func continuousEcho(host string, port int, timeout time.Duration, maxAttempts int, signal chan struct{}, errorChannel chan error) { defer ginkgo.GinkgoRecover() const threshold = 10 // Sanity check inputs, because it has happened.  These are the only things // that should hard fail the test - they are basically ASSERT()s. if host == "" { errorChannel <- fmt.Errorf("Got empty host for continuous echo (%s)", host) return } if port == 0 { errorChannel <- fmt.Errorf("Got port ==0 for continuous echo (%d)", port) return } hostPort := net.JoinHostPort(host, strconv.Itoa(port)) url := fmt.Sprintf("udp://%s", hostPort) ret := UDPPokeResult{} con, err := net.Dial("udp", hostPort) if err != nil { ret.Status = UDPError ret.Error = err errorChannel <- fmt.Errorf("Connection to %q failed: %v", url, err) return } numErrors := 0 bufsize := len(strconv.Itoa(maxAttempts)) + 1 var buf = make([]byte, bufsize) for i := 0; i < maxAttempts; i++ { if i == threshold { framework.Logf("Continuous echo waiting for signal to continue") <-signal if numErrors == threshold { errorChannel <- fmt.Errorf("continuous echo was not able to communicate with initial server pod") return } } time.Sleep(1 * time.Second) err = con.SetDeadline(time.Now().Add(timeout)) if err != nil { ret.Status = UDPError ret.Error = err framework.Logf("Continuous echo (%q): %v", url, err) numErrors++ continue } myRequest := fmt.Sprintf("echo %d", i) _, err = con.Write([]byte(fmt.Sprintf("%sn", myRequest))) if err != nil { ret.Error = err neterr, ok := err.(net.Error) if ok && neterr.Timeout() { ret.Status = UDPTimeout } else if strings.Contains(err.Error(), "connection refused") { ret.Status = UDPRefused } else { ret.Status = UDPError } numErrors++ framework.Logf("Continuous echo (%q): %v - %d errors seen so far", url, err, numErrors) continue } err = con.SetDeadline(time.Now().Add(timeout)) if err != nil { ret.Status = UDPError ret.Error = err numErrors++ framework.Logf("Continuous echo (%q): %v - %d errors seen so far", url, err, numErrors) continue } n, err := con.Read(buf) if err != nil { ret.Error = err neterr, ok := err.(net.Error) if ok && neterr.Timeout() { ret.Status = UDPTimeout } else if strings.Contains(err.Error(), "connection refused") { ret.Status = UDPRefused } else { ret.Status = UDPError } numErrors++ framework.Logf("Continuous echo (%q): %v - %d errors seen so far", url, err, numErrors) continue } ret.Response = buf[0:n] if string(ret.Response) != fmt.Sprintf("%d", i) { ret.Status = UDPBadResponse ret.Error = fmt.Errorf("response does not match expected string: %q", string(ret.Response)) framework.Logf("Continuous echo (%q): %v", url, ret.Error) numErrors++ continue } ret.Status = UDPSuccess framework.Logf("Continuous echo(%q): success", url) } err = nil if numErrors >= threshold { err = fmt.Errorf("Too many Errors in continuous echo") } errorChannel <- err } </del> // testReachableUDP tests that the given host serves UDP on the given port. func testReachableUDP(host string, port int, timeout time.Duration) { pollfn := func() (bool, error) {

Which jobs are flaking: ci-kubernetes-e2e-gce-scale-correctness Which test(s) are flaking: [sig-network] Services should be able to preserve UDP traffic when server pod cycles for a NodePort service Testgrid link: https://k8s- Reason for failure: /sig network /assign
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
CRI-O CI is also hitting this failure. We have been unable to reproduce locally. our CI runs on aws
hmm, I only see 1 failure out of 4 and seems that the connectivity didn't work at all
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
/assign
GitHub didn't allow me to assign the following users: JacobTanenbaum. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc Saw the test flaking recently: Wondering if this is just a timeout issue.


Question: return ret } <del> // continuousEcho() uses the same connection for multiple requests, made to run as a goroutine so that // manipulations can be made to the service and backend pods while a connection is ongoing // it starts by sending a series of packets to establish conntrack entries and waits for a signal to keep // sending packts. It returns an error if the number of failed attempts is >= 5 func continuousEcho(host string, port int, timeout time.Duration, maxAttempts int, signal chan struct{}, errorChannel chan error) { defer ginkgo.GinkgoRecover() const threshold = 10 // Sanity check inputs, because it has happened.  These are the only things // that should hard fail the test - they are basically ASSERT()s. if host == "" { errorChannel <- fmt.Errorf("Got empty host for continuous echo (%s)", host) return } if port == 0 { errorChannel <- fmt.Errorf("Got port ==0 for continuous echo (%d)", port) return } hostPort := net.JoinHostPort(host, strconv.Itoa(port)) url := fmt.Sprintf("udp://%s", hostPort) ret := UDPPokeResult{} con, err := net.Dial("udp", hostPort) if err != nil { ret.Status = UDPError ret.Error = err errorChannel <- fmt.Errorf("Connection to %q failed: %v", url, err) return } numErrors := 0 bufsize := len(strconv.Itoa(maxAttempts)) + 1 var buf = make([]byte, bufsize) for i := 0; i < maxAttempts; i++ { if i == threshold { framework.Logf("Continuous echo waiting for signal to continue") <-signal if numErrors == threshold { errorChannel <- fmt.Errorf("continuous echo was not able to communicate with initial server pod") return } } time.Sleep(1 * time.Second) err = con.SetDeadline(time.Now().Add(timeout)) if err != nil { ret.Status = UDPError ret.Error = err framework.Logf("Continuous echo (%q): %v", url, err) numErrors++ continue } myRequest := fmt.Sprintf("echo %d", i) _, err = con.Write([]byte(fmt.Sprintf("%sn", myRequest))) if err != nil { ret.Error = err neterr, ok := err.(net.Error) if ok && neterr.Timeout() { ret.Status = UDPTimeout } else if strings.Contains(err.Error(), "connection refused") { ret.Status = UDPRefused } else { ret.Status = UDPError } numErrors++ framework.Logf("Continuous echo (%q): %v - %d errors seen so far", url, err, numErrors) continue } err = con.SetDeadline(time.Now().Add(timeout)) if err != nil { ret.Status = UDPError ret.Error = err numErrors++ framework.Logf("Continuous echo (%q): %v - %d errors seen so far", url, err, numErrors) continue } n, err := con.Read(buf) if err != nil { ret.Error = err neterr, ok := err.(net.Error) if ok && neterr.Timeout() { ret.Status = UDPTimeout } else if strings.Contains(err.Error(), "connection refused") { ret.Status = UDPRefused } else { ret.Status = UDPError } numErrors++ framework.Logf("Continuous echo (%q): %v - %d errors seen so far", url, err, numErrors) continue } ret.Response = buf[0:n] if string(ret.Response) != fmt.Sprintf("%d", i) { ret.Status = UDPBadResponse ret.Error = fmt.Errorf("response does not match expected string: %q", string(ret.Response)) framework.Logf("Continuous echo (%q): %v", url, ret.Error) numErrors++ continue } ret.Status = UDPSuccess framework.Logf("Continuous echo(%q): success", url) } err = nil if numErrors >= threshold { err = fmt.Errorf("Too many Errors in continuous echo") } errorChannel <- err } </del> // testReachableUDP tests that the given host serves UDP on the given port. func testReachableUDP(host string, port int, timeout time.Duration) { pollfn := func() (bool, error) {

Maybe we can increase the timeout and/or switch to a TCP based test:
the test should use UDP, and UDP is unreliable by definition ... however is curious that most of the failures happen because it is not able to communicate with the first pod I think that is something that should be investigated, I've analyzed some occurences and it works fine for the second pod, however, the UDP losses should be totally random
Hi everyone, yeah so.. collecting some data on this... using a two node kind cluster, locally, with the antrea plugin, it passes using a 4 node AWS cluster with the antrea CNI plugin, it failed using a two node kidn cluster locally with calico, it passes So I guess the best reproducer is to run these tests, for now, in the cloud.... Now that I have a reproducer for this maybe can live debug it some time next wk with one of yall :)
bear in mind that this test is connecting to a nodePort in the node IP, the test is functionally correct, it may be flaky due to the unreliable UDP nature or due to the pod scheduling, but if is failing constantly it has to be environmental.
Second failure after two weeks of green 5k runs:
yep, :/ https://k8s-|UDP /assign
/reopen I'm afraid this is going to be challenging :smile: The difference now is that we are testing NodePort and ClusterIP If both fail, we can blame the test for sure, If only NodePort fails, we need to check the job, if GCE or KIND If only ClusterIP fails, ... :thinking:
Reopened this issue. <detailsIn response to : /reopen I'm afraid this is going to be challenging :smile: Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
We're hitting this really frequently in 5k-node tests. So I started looking a little bit into that.


Question: return ret } <del> // continuousEcho() uses the same connection for multiple requests, made to run as a goroutine so that // manipulations can be made to the service and backend pods while a connection is ongoing // it starts by sending a series of packets to establish conntrack entries and waits for a signal to keep // sending packts. It returns an error if the number of failed attempts is >= 5 func continuousEcho(host string, port int, timeout time.Duration, maxAttempts int, signal chan struct{}, errorChannel chan error) { defer ginkgo.GinkgoRecover() const threshold = 10 // Sanity check inputs, because it has happened.  These are the only things // that should hard fail the test - they are basically ASSERT()s. if host == "" { errorChannel <- fmt.Errorf("Got empty host for continuous echo (%s)", host) return } if port == 0 { errorChannel <- fmt.Errorf("Got port ==0 for continuous echo (%d)", port) return } hostPort := net.JoinHostPort(host, strconv.Itoa(port)) url := fmt.Sprintf("udp://%s", hostPort) ret := UDPPokeResult{} con, err := net.Dial("udp", hostPort) if err != nil { ret.Status = UDPError ret.Error = err errorChannel <- fmt.Errorf("Connection to %q failed: %v", url, err) return } numErrors := 0 bufsize := len(strconv.Itoa(maxAttempts)) + 1 var buf = make([]byte, bufsize) for i := 0; i < maxAttempts; i++ { if i == threshold { framework.Logf("Continuous echo waiting for signal to continue") <-signal if numErrors == threshold { errorChannel <- fmt.Errorf("continuous echo was not able to communicate with initial server pod") return } } time.Sleep(1 * time.Second) err = con.SetDeadline(time.Now().Add(timeout)) if err != nil { ret.Status = UDPError ret.Error = err framework.Logf("Continuous echo (%q): %v", url, err) numErrors++ continue } myRequest := fmt.Sprintf("echo %d", i) _, err = con.Write([]byte(fmt.Sprintf("%sn", myRequest))) if err != nil { ret.Error = err neterr, ok := err.(net.Error) if ok && neterr.Timeout() { ret.Status = UDPTimeout } else if strings.Contains(err.Error(), "connection refused") { ret.Status = UDPRefused } else { ret.Status = UDPError } numErrors++ framework.Logf("Continuous echo (%q): %v - %d errors seen so far", url, err, numErrors) continue } err = con.SetDeadline(time.Now().Add(timeout)) if err != nil { ret.Status = UDPError ret.Error = err numErrors++ framework.Logf("Continuous echo (%q): %v - %d errors seen so far", url, err, numErrors) continue } n, err := con.Read(buf) if err != nil { ret.Error = err neterr, ok := err.(net.Error) if ok && neterr.Timeout() { ret.Status = UDPTimeout } else if strings.Contains(err.Error(), "connection refused") { ret.Status = UDPRefused } else { ret.Status = UDPError } numErrors++ framework.Logf("Continuous echo (%q): %v - %d errors seen so far", url, err, numErrors) continue } ret.Response = buf[0:n] if string(ret.Response) != fmt.Sprintf("%d", i) { ret.Status = UDPBadResponse ret.Error = fmt.Errorf("response does not match expected string: %q", string(ret.Response)) framework.Logf("Continuous echo (%q): %v", url, ret.Error) numErrors++ continue } ret.Status = UDPSuccess framework.Logf("Continuous echo(%q): success", url) } err = nil if numErrors >= threshold { err = fmt.Errorf("Too many Errors in continuous echo") } errorChannel <- err } </del> // testReachableUDP tests that the given host serves UDP on the given port. func testReachableUDP(host string, port int, timeout time.Duration) { pollfn := func() (bool, error) {

I don't have full understanding of what exactly is happening, but I already see a couple things that may be causing issues: when creating the first pod, you wait for corresponding endpoint for that being created; but that doesn't mean that kube-proxy was actually programmed correctly at that time (in general it isn't); given iptables programming is rate-limitted, it may easily happen that actually programming happens 10s+ later That explains some delays, but it doesn't explain e.g. this run: where iptables were programmed at but the test didn't respond until If someone more familiar with networking stuff could look into it, it would be great. - can you help triaging ^^ for deletions it's even worse, because you don't even wait for endpoints to be there (i.e. Endpoints can still contain the old pod when the test start pinging to look for the second one) So I claim that timeouts are visibly to low in this test, but I suspect it's not everything. I will send out the first PR that tries to improve it in couple minutes.
thanks wojtek, have been slammed but Ill try to collect more data on it this wknd as well.
It seems that my PR didn't really help much. We should dig deeper, but I would need someone with more networking expertise to look a bit into
Actually - it seems only failures from one group are on non . So maybe that's actually related. The non-kind related ones look like:
OK - for this one, I actually know what is happening - I will send out a PR with a fix for this too.
for the KIND jobs already increased the resource requests on some of the jobs, there are issues on KIND jobs with kube-proxy not able to take the iptables lock and not being able to create the iptables rules for the services until the next resync
is to fix the flakes I mentioned above; all other flakes seem to be kind-oriented to me (which I'm not super familiar with).
We did not adjust the CPU on all jobs yet, pending how to manage that on prow ... I do see just a few other flakes at the bottom of the triage dashboard that look like timeouts instead.
Actually - almost all failures are effectively timeout. The top one that is is actually also a timeout. Didn't look into kind failures, but this may easily be related to iptables issues.


Question: return ret } <del> // continuousEcho() uses the same connection for multiple requests, made to run as a goroutine so that // manipulations can be made to the service and backend pods while a connection is ongoing // it starts by sending a series of packets to establish conntrack entries and waits for a signal to keep // sending packts. It returns an error if the number of failed attempts is >= 5 func continuousEcho(host string, port int, timeout time.Duration, maxAttempts int, signal chan struct{}, errorChannel chan error) { defer ginkgo.GinkgoRecover() const threshold = 10 // Sanity check inputs, because it has happened.  These are the only things // that should hard fail the test - they are basically ASSERT()s. if host == "" { errorChannel <- fmt.Errorf("Got empty host for continuous echo (%s)", host) return } if port == 0 { errorChannel <- fmt.Errorf("Got port ==0 for continuous echo (%d)", port) return } hostPort := net.JoinHostPort(host, strconv.Itoa(port)) url := fmt.Sprintf("udp://%s", hostPort) ret := UDPPokeResult{} con, err := net.Dial("udp", hostPort) if err != nil { ret.Status = UDPError ret.Error = err errorChannel <- fmt.Errorf("Connection to %q failed: %v", url, err) return } numErrors := 0 bufsize := len(strconv.Itoa(maxAttempts)) + 1 var buf = make([]byte, bufsize) for i := 0; i < maxAttempts; i++ { if i == threshold { framework.Logf("Continuous echo waiting for signal to continue") <-signal if numErrors == threshold { errorChannel <- fmt.Errorf("continuous echo was not able to communicate with initial server pod") return } } time.Sleep(1 * time.Second) err = con.SetDeadline(time.Now().Add(timeout)) if err != nil { ret.Status = UDPError ret.Error = err framework.Logf("Continuous echo (%q): %v", url, err) numErrors++ continue } myRequest := fmt.Sprintf("echo %d", i) _, err = con.Write([]byte(fmt.Sprintf("%sn", myRequest))) if err != nil { ret.Error = err neterr, ok := err.(net.Error) if ok && neterr.Timeout() { ret.Status = UDPTimeout } else if strings.Contains(err.Error(), "connection refused") { ret.Status = UDPRefused } else { ret.Status = UDPError } numErrors++ framework.Logf("Continuous echo (%q): %v - %d errors seen so far", url, err, numErrors) continue } err = con.SetDeadline(time.Now().Add(timeout)) if err != nil { ret.Status = UDPError ret.Error = err numErrors++ framework.Logf("Continuous echo (%q): %v - %d errors seen so far", url, err, numErrors) continue } n, err := con.Read(buf) if err != nil { ret.Error = err neterr, ok := err.(net.Error) if ok && neterr.Timeout() { ret.Status = UDPTimeout } else if strings.Contains(err.Error(), "connection refused") { ret.Status = UDPRefused } else { ret.Status = UDPError } numErrors++ framework.Logf("Continuous echo (%q): %v - %d errors seen so far", url, err, numErrors) continue } ret.Response = buf[0:n] if string(ret.Response) != fmt.Sprintf("%d", i) { ret.Status = UDPBadResponse ret.Error = fmt.Errorf("response does not match expected string: %q", string(ret.Response)) framework.Logf("Continuous echo (%q): %v", url, ret.Error) numErrors++ continue } ret.Status = UDPSuccess framework.Logf("Continuous echo(%q): success", url) } err = nil if numErrors >= threshold { err = fmt.Errorf("Too many Errors in continuous echo") } errorChannel <- err } </del> // testReachableUDP tests that the given host serves UDP on the given port. func testReachableUDP(host string, port int, timeout time.Duration) { pollfn := func() (bool, error) {

After my second deflaking PR, we seem to be in the stage where: tests are failing on dozens of different things (probably iptables-issues related) tests are failing only on NodePort tests on "Failed to connecto to backend 1" which it would be great if someone else could look into (kube-proxy seemed to be programmed correctly based on logs)
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
This has failed again - Linking to specific part of log where timeout occurred .... we see this
So reviewing the above logs note how in STEP: checking client pod connected to the backend 1 on Node IP 10.138.0.5 28 attempts are made all in the 25th second of the 2 minute at 11am Should this Step back off and wait a few seconds in between tries? So looking at this code we see the following
From reading ... the intention expressed in the test code looks good to me. It is not clear to me why there is a mismatch between the 28 tries in the POD Client Logs and the expressed intention running the logContainsFn(podBackend1) every 5 seconds still digging ....
we are trying to catch one failure in this PR that has additional debugging, it dumps the conntrack table of the nodes on a failure
for those following the issue, we've extra debugging to the test so it dumps the whole conntrack table once it finish. We should look that there is NO stale entries, connections from the pod client to the deleted endpoint) and there is conntrack entries to the new endpoint from the pod client.
- we have runs with the conntrack dump now, e.g. this one: would you be able to take a look into it?
not yet :/ , but I think tomorrow can have some time
cc
For anyone reproducing this issue btw, you need a multinode k8s cluster (it isnt doable on kind i guess bc less likeliehood of network failures when running local) -
Curious about two potential hypotheses to fix this, have we looked into either of these? 1) can we confirm that is turned on/off? 2) is it possible that scaling the number of coredns replicas could help ?


Question: return ret } <del> // continuousEcho() uses the same connection for multiple requests, made to run as a goroutine so that // manipulations can be made to the service and backend pods while a connection is ongoing // it starts by sending a series of packets to establish conntrack entries and waits for a signal to keep // sending packts. It returns an error if the number of failed attempts is >= 5 func continuousEcho(host string, port int, timeout time.Duration, maxAttempts int, signal chan struct{}, errorChannel chan error) { defer ginkgo.GinkgoRecover() const threshold = 10 // Sanity check inputs, because it has happened.  These are the only things // that should hard fail the test - they are basically ASSERT()s. if host == "" { errorChannel <- fmt.Errorf("Got empty host for continuous echo (%s)", host) return } if port == 0 { errorChannel <- fmt.Errorf("Got port ==0 for continuous echo (%d)", port) return } hostPort := net.JoinHostPort(host, strconv.Itoa(port)) url := fmt.Sprintf("udp://%s", hostPort) ret := UDPPokeResult{} con, err := net.Dial("udp", hostPort) if err != nil { ret.Status = UDPError ret.Error = err errorChannel <- fmt.Errorf("Connection to %q failed: %v", url, err) return } numErrors := 0 bufsize := len(strconv.Itoa(maxAttempts)) + 1 var buf = make([]byte, bufsize) for i := 0; i < maxAttempts; i++ { if i == threshold { framework.Logf("Continuous echo waiting for signal to continue") <-signal if numErrors == threshold { errorChannel <- fmt.Errorf("continuous echo was not able to communicate with initial server pod") return } } time.Sleep(1 * time.Second) err = con.SetDeadline(time.Now().Add(timeout)) if err != nil { ret.Status = UDPError ret.Error = err framework.Logf("Continuous echo (%q): %v", url, err) numErrors++ continue } myRequest := fmt.Sprintf("echo %d", i) _, err = con.Write([]byte(fmt.Sprintf("%sn", myRequest))) if err != nil { ret.Error = err neterr, ok := err.(net.Error) if ok && neterr.Timeout() { ret.Status = UDPTimeout } else if strings.Contains(err.Error(), "connection refused") { ret.Status = UDPRefused } else { ret.Status = UDPError } numErrors++ framework.Logf("Continuous echo (%q): %v - %d errors seen so far", url, err, numErrors) continue } err = con.SetDeadline(time.Now().Add(timeout)) if err != nil { ret.Status = UDPError ret.Error = err numErrors++ framework.Logf("Continuous echo (%q): %v - %d errors seen so far", url, err, numErrors) continue } n, err := con.Read(buf) if err != nil { ret.Error = err neterr, ok := err.(net.Error) if ok && neterr.Timeout() { ret.Status = UDPTimeout } else if strings.Contains(err.Error(), "connection refused") { ret.Status = UDPRefused } else { ret.Status = UDPError } numErrors++ framework.Logf("Continuous echo (%q): %v - %d errors seen so far", url, err, numErrors) continue } ret.Response = buf[0:n] if string(ret.Response) != fmt.Sprintf("%d", i) { ret.Status = UDPBadResponse ret.Error = fmt.Errorf("response does not match expected string: %q", string(ret.Response)) framework.Logf("Continuous echo (%q): %v", url, ret.Error) numErrors++ continue } ret.Status = UDPSuccess framework.Logf("Continuous echo(%q): success", url) } err = nil if numErrors >= threshold { err = fmt.Errorf("Too many Errors in continuous echo") } errorChannel <- err } </del> // testReachableUDP tests that the given host serves UDP on the given port. func testReachableUDP(host string, port int, timeout time.Duration) { pollfn := func() (bool, error) {

3) Would simulating this in kind by overloading coredns be a possibility as a way to make a more reproducible bug model?
regarding 1) kubeadm does not ship nodeLocalDNSCache I don't think GCE kube-up does by default either
ok, so now im curious about (2) - i.e. what if we just bumped coredns replica count?
interesting, :thinking: why do you think is a DNS thing? the netcat command that polls the services use the IP addresses not the hostname. I think that it has to be related to some iptables/conntrack race or similar,at least in KIND jobs we know that we have an iptables problem that affects network tests, that has to be fixed before this because it affect this test You can instrument the test locally with this and dump the state of iptables and conntrack on failure if you are able to reproduce it
i figured DNS bc i believe its known that conntrack and DNS at scale can get very messy, right ?
one other curious thing about this test is that we only create 2 pods in the recycling event for the svc. that likely means that any real error is hidden by chance..... If we created a 3rd pod, then we might potentially double the failure rate thus making the flakyness go away and expose the real underlying issue .
There's an invocation of dumpConntrack(cs) here [sig-network] Conntrack should be able to preserve UDP traffic when server pod cycles for a ClusterIP service
is updating the logging to run in an infinite loop ..
This is improving in KIND with the new changes we rolled out yesterday, let's see how it evolves this week
- which changes ? Are you saying this is no longer reproducible on Kind?
there was a lot of noise in the CI signal, especially in KIND jobs due to the CNI portmap bug we finally solved yesterday now we should have a better signal and we can go to the bottom because probably the failure will be "legit" and not a consequence, so this comment should apply
Hi ! I ran this a few times on kind and it passed, but I see , so I'm going to try the loop approach you suggested. Here's what I'm using to test:
quick note, <- potential motivation to possibly have runnable on some subset of tests in upstream eventually, i.e.


Question: return ret } <del> // continuousEcho() uses the same connection for multiple requests, made to run as a goroutine so that // manipulations can be made to the service and backend pods while a connection is ongoing // it starts by sending a series of packets to establish conntrack entries and waits for a signal to keep // sending packts. It returns an error if the number of failed attempts is >= 5 func continuousEcho(host string, port int, timeout time.Duration, maxAttempts int, signal chan struct{}, errorChannel chan error) { defer ginkgo.GinkgoRecover() const threshold = 10 // Sanity check inputs, because it has happened.  These are the only things // that should hard fail the test - they are basically ASSERT()s. if host == "" { errorChannel <- fmt.Errorf("Got empty host for continuous echo (%s)", host) return } if port == 0 { errorChannel <- fmt.Errorf("Got port ==0 for continuous echo (%d)", port) return } hostPort := net.JoinHostPort(host, strconv.Itoa(port)) url := fmt.Sprintf("udp://%s", hostPort) ret := UDPPokeResult{} con, err := net.Dial("udp", hostPort) if err != nil { ret.Status = UDPError ret.Error = err errorChannel <- fmt.Errorf("Connection to %q failed: %v", url, err) return } numErrors := 0 bufsize := len(strconv.Itoa(maxAttempts)) + 1 var buf = make([]byte, bufsize) for i := 0; i < maxAttempts; i++ { if i == threshold { framework.Logf("Continuous echo waiting for signal to continue") <-signal if numErrors == threshold { errorChannel <- fmt.Errorf("continuous echo was not able to communicate with initial server pod") return } } time.Sleep(1 * time.Second) err = con.SetDeadline(time.Now().Add(timeout)) if err != nil { ret.Status = UDPError ret.Error = err framework.Logf("Continuous echo (%q): %v", url, err) numErrors++ continue } myRequest := fmt.Sprintf("echo %d", i) _, err = con.Write([]byte(fmt.Sprintf("%sn", myRequest))) if err != nil { ret.Error = err neterr, ok := err.(net.Error) if ok && neterr.Timeout() { ret.Status = UDPTimeout } else if strings.Contains(err.Error(), "connection refused") { ret.Status = UDPRefused } else { ret.Status = UDPError } numErrors++ framework.Logf("Continuous echo (%q): %v - %d errors seen so far", url, err, numErrors) continue } err = con.SetDeadline(time.Now().Add(timeout)) if err != nil { ret.Status = UDPError ret.Error = err numErrors++ framework.Logf("Continuous echo (%q): %v - %d errors seen so far", url, err, numErrors) continue } n, err := con.Read(buf) if err != nil { ret.Error = err neterr, ok := err.(net.Error) if ok && neterr.Timeout() { ret.Status = UDPTimeout } else if strings.Contains(err.Error(), "connection refused") { ret.Status = UDPRefused } else { ret.Status = UDPError } numErrors++ framework.Logf("Continuous echo (%q): %v - %d errors seen so far", url, err, numErrors) continue } ret.Response = buf[0:n] if string(ret.Response) != fmt.Sprintf("%d", i) { ret.Status = UDPBadResponse ret.Error = fmt.Errorf("response does not match expected string: %q", string(ret.Response)) framework.Logf("Continuous echo (%q): %v", url, ret.Error) numErrors++ continue } ret.Status = UDPSuccess framework.Logf("Continuous echo(%q): success", url) } err = nil if numErrors >= threshold { err = fmt.Errorf("Too many Errors in continuous echo") } errorChannel <- err } </del> // testReachableUDP tests that the given host serves UDP on the given port. func testReachableUDP(host string, port int, timeout time.Duration) { pollfn := func() (bool, error) {

so we can do things like to reproduce this sort of thing. matt did you get around to the 'try-till-it-fails' scenario?
I haven't been able to get it to fail on kind yet, -- 100 cycles and 0 failures on a three-node kind cluster! I'm letting the loop run for a few hours, will check back in when I have that data. Is it worth running this test on a non-kind environment?
I ran for several hours on my mac, through 1144 iterations no error yet. My conclusion: this test isn't flaky on KinD. Possible next step: run a large number of iterations of the same test on a GKE cluster.
Flakes on kind are also happening:
not many since the portmap fix on the 14th though ( / ) <img width="1111" alt="Screen Shot 2020-07-23 at 11 45 46 PM" src="https://user-"
Agree - it's much better than it was before.
I was analyzing this failure, the problem is that kind jobs does not have verbosity enough in kube-proxy We can see that the packet was never replied, this is the NAT entry from client -nodePort in the node with the client This is the same flow, but in the node with the server and the pod-server-1 seems to be healthy we need more logs :eyeglasses:
We have an open ticket to bump those in CI, it's just been lost in the noise. We should still do that. Antonio Ojea <:
If you are busy just assign it to me, I'll need this verbosity soon :)
It's in the kind repo, shouldn't be too hard to find, we have ~60 open atm. Antonio Ojea <:
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
I made another attempt to reproduce these failures locally on KinD, but still no success. Here's what I did: create a KinD cluster run two instances of sonobuoy at the same time This took ~4 hours to run; all the kube tests passed and sonobuoy reported all successes.


Question: return ret } <del> // continuousEcho() uses the same connection for multiple requests, made to run as a goroutine so that // manipulations can be made to the service and backend pods while a connection is ongoing // it starts by sending a series of packets to establish conntrack entries and waits for a signal to keep // sending packts. It returns an error if the number of failed attempts is >= 5 func continuousEcho(host string, port int, timeout time.Duration, maxAttempts int, signal chan struct{}, errorChannel chan error) { defer ginkgo.GinkgoRecover() const threshold = 10 // Sanity check inputs, because it has happened.  These are the only things // that should hard fail the test - they are basically ASSERT()s. if host == "" { errorChannel <- fmt.Errorf("Got empty host for continuous echo (%s)", host) return } if port == 0 { errorChannel <- fmt.Errorf("Got port ==0 for continuous echo (%d)", port) return } hostPort := net.JoinHostPort(host, strconv.Itoa(port)) url := fmt.Sprintf("udp://%s", hostPort) ret := UDPPokeResult{} con, err := net.Dial("udp", hostPort) if err != nil { ret.Status = UDPError ret.Error = err errorChannel <- fmt.Errorf("Connection to %q failed: %v", url, err) return } numErrors := 0 bufsize := len(strconv.Itoa(maxAttempts)) + 1 var buf = make([]byte, bufsize) for i := 0; i < maxAttempts; i++ { if i == threshold { framework.Logf("Continuous echo waiting for signal to continue") <-signal if numErrors == threshold { errorChannel <- fmt.Errorf("continuous echo was not able to communicate with initial server pod") return } } time.Sleep(1 * time.Second) err = con.SetDeadline(time.Now().Add(timeout)) if err != nil { ret.Status = UDPError ret.Error = err framework.Logf("Continuous echo (%q): %v", url, err) numErrors++ continue } myRequest := fmt.Sprintf("echo %d", i) _, err = con.Write([]byte(fmt.Sprintf("%sn", myRequest))) if err != nil { ret.Error = err neterr, ok := err.(net.Error) if ok && neterr.Timeout() { ret.Status = UDPTimeout } else if strings.Contains(err.Error(), "connection refused") { ret.Status = UDPRefused } else { ret.Status = UDPError } numErrors++ framework.Logf("Continuous echo (%q): %v - %d errors seen so far", url, err, numErrors) continue } err = con.SetDeadline(time.Now().Add(timeout)) if err != nil { ret.Status = UDPError ret.Error = err numErrors++ framework.Logf("Continuous echo (%q): %v - %d errors seen so far", url, err, numErrors) continue } n, err := con.Read(buf) if err != nil { ret.Error = err neterr, ok := err.(net.Error) if ok && neterr.Timeout() { ret.Status = UDPTimeout } else if strings.Contains(err.Error(), "connection refused") { ret.Status = UDPRefused } else { ret.Status = UDPError } numErrors++ framework.Logf("Continuous echo (%q): %v - %d errors seen so far", url, err, numErrors) continue } ret.Response = buf[0:n] if string(ret.Response) != fmt.Sprintf("%d", i) { ret.Status = UDPBadResponse ret.Error = fmt.Errorf("response does not match expected string: %q", string(ret.Response)) framework.Logf("Continuous echo (%q): %v", url, ret.Error) numErrors++ continue } ret.Status = UDPSuccess framework.Logf("Continuous echo(%q): success", url) } err = nil if numErrors >= threshold { err = fmt.Errorf("Too many Errors in continuous echo") } errorChannel <- err } </del> // testReachableUDP tests that the given host serves UDP on the given port. func testReachableUDP(host string, port int, timeout time.Duration) { pollfn := func() (bool, error) {

Would love to help out in reproing this, I'm at a loss for good next step so any suggestions welcome!
that's the problem with these flakes, they are not easy to repro, and with this low rate of failures it may be caused for combination of running tests, environmental problems (cpu, memory, ...) , ... I always start for checking the current failures and try to find patterns
Let's take a look to one occurrence The tests fails because can't connect to the first pod If we check the code of the tests, we can see that it always uses the port as source port, we can use that to identify the nodePort assigned interesting thing here, we are source nating the pod ip, and we don't see any other conntrack entry The test verifies it has endpoints however, kube-proxy is using endpointlsices since this version but it has no entries so the iptables rules to forward the traffic are not there Seems we have to adapt this function in the e2e framework to use endpointslices cc: EDIT seems that this function is being used by all the network tests that can cause flakes if the endpoint is updated before the endpointslice (we switched recently kube-proxy to use the later)
Great catch. Ideally this function should validate if both Endpoint and appropriate EndpointSlice objects(s) exist.
Actually - let me take it and send a PR for it.
is out for review
Interesting failures here It fails to connect to the first pod
Hmm - I took a quick look at kube-proxy logs and these look fine to me....
I clicked too fast, sorry ... I was looking at the wrong service and I though I've found something
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.


Question: return ret } <del> // continuousEcho() uses the same connection for multiple requests, made to run as a goroutine so that // manipulations can be made to the service and backend pods while a connection is ongoing // it starts by sending a series of packets to establish conntrack entries and waits for a signal to keep // sending packts. It returns an error if the number of failed attempts is >= 5 func continuousEcho(host string, port int, timeout time.Duration, maxAttempts int, signal chan struct{}, errorChannel chan error) { defer ginkgo.GinkgoRecover() const threshold = 10 // Sanity check inputs, because it has happened.  These are the only things // that should hard fail the test - they are basically ASSERT()s. if host == "" { errorChannel <- fmt.Errorf("Got empty host for continuous echo (%s)", host) return } if port == 0 { errorChannel <- fmt.Errorf("Got port ==0 for continuous echo (%d)", port) return } hostPort := net.JoinHostPort(host, strconv.Itoa(port)) url := fmt.Sprintf("udp://%s", hostPort) ret := UDPPokeResult{} con, err := net.Dial("udp", hostPort) if err != nil { ret.Status = UDPError ret.Error = err errorChannel <- fmt.Errorf("Connection to %q failed: %v", url, err) return } numErrors := 0 bufsize := len(strconv.Itoa(maxAttempts)) + 1 var buf = make([]byte, bufsize) for i := 0; i < maxAttempts; i++ { if i == threshold { framework.Logf("Continuous echo waiting for signal to continue") <-signal if numErrors == threshold { errorChannel <- fmt.Errorf("continuous echo was not able to communicate with initial server pod") return } } time.Sleep(1 * time.Second) err = con.SetDeadline(time.Now().Add(timeout)) if err != nil { ret.Status = UDPError ret.Error = err framework.Logf("Continuous echo (%q): %v", url, err) numErrors++ continue } myRequest := fmt.Sprintf("echo %d", i) _, err = con.Write([]byte(fmt.Sprintf("%sn", myRequest))) if err != nil { ret.Error = err neterr, ok := err.(net.Error) if ok && neterr.Timeout() { ret.Status = UDPTimeout } else if strings.Contains(err.Error(), "connection refused") { ret.Status = UDPRefused } else { ret.Status = UDPError } numErrors++ framework.Logf("Continuous echo (%q): %v - %d errors seen so far", url, err, numErrors) continue } err = con.SetDeadline(time.Now().Add(timeout)) if err != nil { ret.Status = UDPError ret.Error = err numErrors++ framework.Logf("Continuous echo (%q): %v - %d errors seen so far", url, err, numErrors) continue } n, err := con.Read(buf) if err != nil { ret.Error = err neterr, ok := err.(net.Error) if ok && neterr.Timeout() { ret.Status = UDPTimeout } else if strings.Contains(err.Error(), "connection refused") { ret.Status = UDPRefused } else { ret.Status = UDPError } numErrors++ framework.Logf("Continuous echo (%q): %v - %d errors seen so far", url, err, numErrors) continue } ret.Response = buf[0:n] if string(ret.Response) != fmt.Sprintf("%d", i) { ret.Status = UDPBadResponse ret.Error = fmt.Errorf("response does not match expected string: %q", string(ret.Response)) framework.Logf("Continuous echo (%q): %v", url, ret.Error) numErrors++ continue } ret.Status = UDPSuccess framework.Logf("Continuous echo(%q): success", url) } err = nil if numErrors >= threshold { err = fmt.Errorf("Too many Errors in continuous echo") } errorChannel <- err } </del> // testReachableUDP tests that the given host serves UDP on the given port. func testReachableUDP(host string, port int, timeout time.Duration) { pollfn := func() (bool, error) {

just saw it as well
can you please add links to the job with the failures, I just merged new code to improve the logging 2 days ago
ok, here there is a new occurrence and with the new debug logs is easier to follow has unreplied conntrack entries: looking at the other side The iptables rules are there but no conntrack entries related to 10.244.2.32
In I revert PR with debug info as I think that it likely caused significant memory growth in scalability tests, but we have one run that has finished without being killed by OOMKiller (I increased memory limit for that jobs as a mitigation). Maybe logs from this run will be useful for debugging:
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
Two more flakes today:
It seems that the packet never gets DNATed in the receiving host, so it does not reach the pod in the node. There is the same test that is using ClusterIP instead of NodePort that is more reliable. This is being a tough one
/triage accepted
Yesterday's flake - I don't think I'm seeing the log messages from the latest log PR you made ? Or are the prow logs not logging at a high-enough level?
Logs are there, please bear with me: The error on the test is: click on the job failure and you'll see a larger output where you can see the server pod name and get the node where the pod is running: and the client-pod name and its node: and the flow we are looking for: with the node-name we can get the kube-proxy logs there, clicking in artifacts: and you can see the "new" conntrack log entries : and in the client: wow, there is a kernel bug there , can it be related? :thinking:
wohoo - some progress :) I'm not an expert here, but I think it can be? Adding here - they may be able to confirm/reject the hypothesis.
This kernel panic may be a coincidence, maybe this is the reason
Adding some recent failures:
Spotted another flake today:
Another flake in the 5k job:


Question: return fmt.Errorf("waiting for kubelet timed out") } <del> func RestartApiserver(c discovery.ServerVersionInterface) error { </del> <ins> func RestartApiserver(cs clientset.Interface) error { </ins> // TODO: Make it work for all providers. if !ProviderIs("gce", "gke", "aws") { return fmt.Errorf("unsupported provider: %s", TestContext.Provider) } if ProviderIs("gce", "aws") { <del> return sshRestartMaster() </del> <ins> initialRestartCount, err := getApiserverRestartCount(cs) if err != nil { return fmt.Errorf("failed to get apiserver's restart count: %v", err) } if err := sshRestartMaster(); err != nil { return fmt.Errorf("failed to restart apiserver: %v", err) } return waitForApiserverRestarted(cs, initialRestartCount) </ins> } // GKE doesn't allow ssh access, so use a same-version master // upgrade to teardown/recreate master. <del> v, err := c.ServerVersion() </del> <ins> v, err := cs.Discovery().ServerVersion() </ins> if err != nil { return err }

This test is failing in both gce serial suite: http://k8s- /sig network /priority failing-test /priority critical-urgent /kind bug /status approved-for-milestone cc /assign PTAL
xref
/milestone v1.10
this also fails in http://k8s-
Failing to connect to Services at all. I can't even ping pod IPs:
So I can ping pods same-node and cross-node right up until it kills the apiserver. Rebooting fixes a given node. The apiserver demonstrably comes back - I can query it for info...
- I could use help - have to shift attention soon.
FWIW, I am testing on a Calico-enabled node
I'll take a look at this as well.
Weird, now even after a reboot, this condition is sticky. I have a cluster in this state, if you want to poke.
ugh, now it is resolved.
I just repro'ed without the e2e - have a cluster, proved ping works, killed apiserver ping fails.
Interesting - docker shows the pods not running, but apiserver says they are alive.
Update: docker sees no running pods. API server thinks kubelet is running them. kubelet is getting "" from apiserver, even after kubelet restart.
Update and recap: Bring up a cluster Run some pods Prove that pods are running and services work pkill -9 apiserver Wait. Watch kubelets "lose" pods that were assigned to them Observe that pods are not running and services do not work Observe that apiserver says the pods are running pkill -9 kubelet Observe that it is still broken kubectl delete one pod Observe that kubelet is fixed for that one pod, but not others Q: Why is kubelet not getting a full listing when it restarts?
What exactly does it mean for it to be fixed for a deleted pod?
Kubelet started running the pod that was deleted - it got the update for that one.
The cluster I was testing was a week or 2 old. I built a cluster from HEAD and I can not repro - said the same. Is this test running against head of master?
I also ran the test manually against my new cluster and it passed
FWIW I was also unable to reproduce this yesterday on a cluster from HEAD.


Question: return fmt.Errorf("waiting for kubelet timed out") } <del> func RestartApiserver(c discovery.ServerVersionInterface) error { </del> <ins> func RestartApiserver(cs clientset.Interface) error { </ins> // TODO: Make it work for all providers. if !ProviderIs("gce", "gke", "aws") { return fmt.Errorf("unsupported provider: %s", TestContext.Provider) } if ProviderIs("gce", "aws") { <del> return sshRestartMaster() </del> <ins> initialRestartCount, err := getApiserverRestartCount(cs) if err != nil { return fmt.Errorf("failed to get apiserver's restart count: %v", err) } if err := sshRestartMaster(); err != nil { return fmt.Errorf("failed to restart apiserver: %v", err) } return waitForApiserverRestarted(cs, initialRestartCount) </ins> } // GKE doesn't allow ssh access, so use a same-version master // upgrade to teardown/recreate master. <del> v, err := c.ServerVersion() </del> <ins> v, err := cs.Discovery().ServerVersion() </ins> if err != nil { return err }

Did some investigations and had some findings for this specific failure: Traced down a bit on the history, the first occurrence was https://k8s- From the , seems suspicious because it modified . Also found reported a relevant symptom: The fix () for that was merged a day ago and may explained why we can't repro from HEAD anymore.
nice, is this issue reproducible without the fix ()?
Sorry should have had confirmed this, bringing up a cluster now.
Quick update: I was able to repro the same failure () from HEAD with reverted. Though this test is now flaky on some other errors: https://k8s- https://k8s- They all seem to failed while communicating with apiserver.
Thanks ! So fixed the issue and I am curious about what introduced the issue. Do you happen to know when did this test start to fail? Just trying to find out what went wrong in between. We have some other possibly related issues so finding out the root cause of this one would be super helpful.
/assign
The first occurrence (for that exact failure) I noticed was https://k8s-, which happened on 2018-02-15 09:49 PST.
You all rock! I see a couple greens on it now, but still mostly red?
The test is now flaky mostly on creating an exec pod after restarting apiserver. Some sample failures: To elaborate a bit more, a snippet of the test procedure is: ssh to master and (sending a SIGTERM) Keep polling on /healthz on apiserver until 200 is returned. Create an exec pod for later steps (and failed on various erros --- EOF, connection refused, apiserver terminating). According to above test failures, my suspicion is that the test is assuming the apiserver is restarted too soon. It is possible that apiserver captured SIGTERM and did a graceful termination, so that polling on /healthz may succeed even in the termination phase. I'm not surprised trying to create a pod via a terminating apiserver will hit those errors. Could anyone confirme the SIGTERM theory?
Spoke with folks, sounds like after SIGTERM apiserver does wait a bit for remaining requests to finish. I will take a stab to refine the test. cc
Sent .
Test is now passing on gce but failing on gke.


Question: return fmt.Errorf("waiting for kubelet timed out") } <del> func RestartApiserver(c discovery.ServerVersionInterface) error { </del> <ins> func RestartApiserver(cs clientset.Interface) error { </ins> // TODO: Make it work for all providers. if !ProviderIs("gce", "gke", "aws") { return fmt.Errorf("unsupported provider: %s", TestContext.Provider) } if ProviderIs("gce", "aws") { <del> return sshRestartMaster() </del> <ins> initialRestartCount, err := getApiserverRestartCount(cs) if err != nil { return fmt.Errorf("failed to get apiserver's restart count: %v", err) } if err := sshRestartMaster(); err != nil { return fmt.Errorf("failed to restart apiserver: %v", err) } return waitForApiserverRestarted(cs, initialRestartCount) </ins> } // GKE doesn't allow ssh access, so use a same-version master // upgrade to teardown/recreate master. <del> v, err := c.ServerVersion() </del> <ins> v, err := cs.Discovery().ServerVersion() </ins> if err != nil { return err }

Apparently we don't have a way to trigger apiserver restart on gke at this moment. Sending a PR to skip this test on gke.
[MILESTONENOTIFIER] Milestone Issue Needs Attention Action required: During code freeze, issues in the milestone should be in progress. If this issue is not being actively worked on, please remove it from the milestone. If it is being worked on, please add the label so it can be tracked with other in-flight issues. Note: This issue is marked as , and must be updated every 1 day during code freeze. Example update: <details<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
Commented too quick. On gke we do a same-version master upgrade. Will send another fix :/