Question: isReady, err := testutils.PodRunningReady(p) framework.ExpectNoError(err) framework.ExpectEqual(isReady, true, "pod should be ready") <del> readyIn := readyTime.Sub(startedTime) framework.Logf("Container started at %v, pod became ready at %v, %v after startupProbe succeeded", startedTime, readyTime, readyIn) if readyIn < 0 { framework.Failf("Pod became ready before startupProbe succeeded") } if readyIn > 5*time.Second { framework.Failf("Pod became ready in %v, more than 5s after startupProbe succeeded. It means that the delay readiness probes were not initiated immediately after startup finished.", readyIn) } </del> }) /*

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Seems to be impacting a few KIND jobs on the master branch the most, but also assorted other jobs see the triage link above Reported by Sample failure: /sig node <!-- Please see the deflaking doc () for more guidance! --
xref probing tests
/assign ref /triage accepted
Thanks Elana, still trying today... Does anyone know if we can use the kubelet logs in e2e tests?
Not very portably in cluster e2e tests at least. Maybe in node_e2e instead.
Have you considered that the issue is not with the test but with the underlying code in kubelet like detailed in this issue Like detailed in the issue I can reliably reproduce (on k8s 1.18) the startup probe passing within a few seconds, but the container is only marked as ready after about a minute.
there is already a fix in the 1.21 release for that (part of a bigger PR): And actually this test was to verify that part of the code, but unfortunately sometimes it doesn't respond as quick as it should (but never on my PC, only in ci/cd tests)...
Some recent flakes:
one of the most common flakes in kind jobs currently failures like: Mar 25 17:03:42.508: Pod became ready in 42.007286524s, more than 5s after startupProbe succeeded. It means that the delay readiness probes were not initiated immediately after startup finished.
Maybe we could speed up the review on
Thank you I will take a look. I'm making a pass through flakes tonight trying to identify if anything is on kind's end
I don't think it's kind... since running on my PC with kind it works 100%, I think the issue is either the resources... or contention in the queue as mentioned to in the comment
/milestone v1.22
commented on PR with the suggested fix
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository.


Question: isReady, err := testutils.PodRunningReady(p) framework.ExpectNoError(err) framework.ExpectEqual(isReady, true, "pod should be ready") <del> readyIn := readyTime.Sub(startedTime) framework.Logf("Container started at %v, pod became ready at %v, %v after startupProbe succeeded", startedTime, readyTime, readyIn) if readyIn < 0 { framework.Failf("Pod became ready before startupProbe succeeded") } if readyIn > 5*time.Second { framework.Failf("Pod became ready in %v, more than 5s after startupProbe succeeded. It means that the delay readiness probes were not initiated immediately after startup finished.", readyIn) } </del> }) /*

</details
Yeah...
Thanks for your great work in keeping things in check
FYI
I have (finally) understood what's happening, and I can even explain why the deflake made things worse... Please have a look at the sleep before the worker starts to process probes: So, you have to remember that we have one worker per pod and per probe, and since we have these probe settings for the test: We have (PeriodSeconds=10) starting sometimes immediately (if the Kubelet has started since more than 10s) or after a quick sleep of rand10s. On the contrary, will likely sleep most of the time for rand60s. This causes a delay in the processing of notifications: If we are lucky, this delay is less than 5s and the test passes: Now you can understand why the deflaked settings were actually worse:
The reason I didn't spot these was that I usually start my cluster once for all and iterate on the tests... so the kubelet had plenty of time to start and I never ended in that sleep.
For the solution, I suggest we either remove the sleep completely, or we find something that would make all 3 probe workers start at the same time to avoid sync issues... For the record, I was able to spot this issue thanks to the following log error (because the channel was overflowing): What do you think ?
I have found another issue in that code: If has already ran (and failed), we go to L245 and don't probe via and therefore wait for the next tick, making the test fail.
This seems consistent with what I found when I was adding e2es for the new probe-level termination grace period and why I did my timing math the way I did.
/reopen I don't think this is fixed, I just hit it again and gubernator is still showing plenty of flakes:
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
how to check the exact commit used for each test?


Question: isReady, err := testutils.PodRunningReady(p) framework.ExpectNoError(err) framework.ExpectEqual(isReady, true, "pod should be ready") <del> readyIn := readyTime.Sub(startedTime) framework.Logf("Container started at %v, pod became ready at %v, %v after startupProbe succeeded", startedTime, readyTime, readyIn) if readyIn < 0 { framework.Failf("Pod became ready before startupProbe succeeded") } if readyIn > 5*time.Second { framework.Failf("Pod became ready in %v, more than 5s after startupProbe succeeded. It means that the delay readiness probes were not initiated immediately after startup finished.", readyIn) } </del> }) /*

For instance, I am not sure (from the name) that should include my latest fix...
In your PR the fix wasn't there:
you may send cherry-pick to 1.21 for which is using kubelet 1.21, not 1.22.
is using kubelet. Cherry-pick to 1.20 may be needed as well.
I have found this one that failed even with the fix: corresponding to:
I don't know what could explain this atm... but it seems we still have delay propagating container statuses.
In the job artifacts there are multiple places with this info including the clone records/logs, and In testgrid it's a row.
yeah afaict since the tests ran on my PR after your fix was merged into master, it showed the flake was still present :( Prow will always use the latest master when running CI.
Sorry for doubting what you said... I was probably in deny and hoped this annoyance was fixed once for all :(
Plan: we'll see if Clayton's pod lifecycle refactor and/or the jitter refactor for probes affects this.
Clayton's PR jitter refactor
since isn't (likely) coming to 1.22 we could cut off 90% of flakes by increasing to 15s delay:
it seems it fails even after merging so let's continue our discussion on slack
this is not a regression, our suggestion is not block release on it. feel free to remove the milestone
/milestone v1.23
In Kubelet maybe, but in the flakiness of the e2e tests it was. Unfortunately we can't easily explore the current state :/
In the release blocking jobs the frequency is not horrible now, so unblocking is probably fine, it is not as bad as it was. Outside of the release cycle this is still the flakiest test in some jobs, see e.g. where it's the only flake source we have on record currently.
if we end up needing it, we have a timing/period adjustment PR we can take that should help with the flakiness with


Question: // unset or false and it can be flipped later when storage // capacity information has been published. // <del> // This field is immutable. </del> <ins> // This field was immutable in Kubernetes <= 1.22 and now is mutable. </ins> // // This is a beta field and only available when the CSIStorageCapacity // feature is enabled. The default is false.

Up to 1.21, that field is immutable. That means in practice, a CSIDriver object has to be deleted and then re-created with a different value instead. Mutation via delete/created is unnecessarily complex and as the only consumer of that field (the kube-scheduler) can handle mutation just fine (it'll always ask the informer cache for the current value), it's better to allow mutation.
/sig storage /assign
This is independent of allowing other fields to become mutable, but that might also make sense.
/triage accepted


Question: kubelet.kubeClient = fakeKubeClient kubelet.os = kubecontainer.FakeOS{} <del> kubelet.hostname = "testnode" </del> <ins> kubelet.hostname = testKubeletHostname </ins> kubelet.runtimeUpThreshold = maxWaitForContainerRuntime kubelet.networkPlugin, _ = network.InitNetworkPlugin([]network.NetworkPlugin{}, "", network.NewFakeHost(nil)) if tempDir, err := ioutil.TempDir("/tmp", "kubelet_test."); err != nil {

Looks like some kubelet unit tests perform a lookupIp which ends up depending on the health of the local resolver. Specifically has this failing: Which leads to the test in: And we recently swapped out the node names from testnode to 127.0.0.1, so probably the culprit?
/ref
Interesting. I did swap out the node names in the unit test as part of to be 127.0.0.1, party because I thought that it wouldn't have any problems resolving during unit tests.
/cc


Question: kubelet.kubeClient = fakeKubeClient kubelet.os = kubecontainer.FakeOS{} <del> kubelet.hostname = "testnode" </del> <ins> kubelet.hostname = testKubeletHostname </ins> kubelet.runtimeUpThreshold = maxWaitForContainerRuntime kubelet.networkPlugin, _ = network.InitNetworkPlugin([]network.NetworkPlugin{}, "", network.NewFakeHost(nil)) if tempDir, err := ioutil.TempDir("/tmp", "kubelet_test."); err != nil {

cc
What is the environment where you are running the tests? They pass for me on my mac, a linux box, and shippable seems to be passing them as well.
I couldn 't reproduce the failure either.
I'm out right now; i'll try them when i get back. For the record, I run hack/test- Paul Morie wrote:
Ahem FTR This acronym is going to go quad-plat Paul Morie wrote:
I ran on both platforms and saw the tests pass.
I'm still seeing it. Won't have bandwidth to dig in personally until tomorrow.
seeing a similar failure after a git pull
I knew I wasn't crazy
environment: fedora 21, golang: 1.4, docker: 1.6
I think I see the problem. When I set the node name to in the test I was assuming it would parse as a valid ip address and not end up at the LookupIP call. But only the node name (and not the kubelet hostname) is being set (except in my new test where I ) so it's trying to do a lookup. Do you think you could test just changing the default hostname from "testnode" to "127.0.0.1" and see if that fixes it for you?
Trying now
I still get failures but no panic / threaddump.
gist coming
this looks like again?
Good that it isn't panicing any longer. I can dig into the test failures tomorrow (I think it should be relatively simple to resolve).
cool, thanks for looking
Passes in master now for me. Closing.
FYI, they will break again once goes in, and then we can merge my PR to see if that fixes it for you.
:+1:
is in -- are these tests breaking again?
I just rebased which I think will fix them.
They are breaking again.
is on it.


Question: <ins> ntp: pkg: - installed ntp-service: service: - running - name: ntp - watch: - pkg: ntp </ins>

At least on AWS, if the clock drifts 300 seconds (which it will), then we get the rather cryptic message: This is because AWS includes the timestamp in the request. Installing ntpd would stop this drifting from happening. If the drift does happen, the likely symptom is that EC2 calls would stop working: attach/detach volumes on the minions, instance sync and routing configuration on the master. I don't believe that GCE has the same problem.


Question: // InstanceExistsByProviderID returns true if the instance with the given provider id still exists and is running. // If false is returned with no error, the instance will be immediately deleted by the cloud controller manager. func (vs *VSphere) InstanceExistsByProviderID(ctx context.Context, providerID string) (bool, error) { <del> var nodeName string nodes, err := vs.nodeManager.GetNodeDetails() </del> <ins> nodeName, err := vs.GetNodeNameFromProviderID(providerID) </ins> if err != nil { <del> glog.Errorf("Error while obtaining Kubernetes node nodeVmDetail details. error : %+v", err) </del> <ins> glog.Errorf("Error while getting nodename for providerID %s", providerID) </ins> return false, err } <del> for _, node := range nodes { // ProviderID is UUID for nodes v1.9.3+ if node.VMUUID == GetUUIDFromProviderID(providerID) || node.NodeName == providerID { nodeName = node.NodeName break } } if nodeName == "" { msg := fmt.Sprintf("Error while obtaining Kubernetes nodename for providerID %s.", providerID) return false, errors.New(msg) } </del> _, err = vs.InstanceID(ctx, convertToK8sType(nodeName)) if err == nil { return true, nil

As planned for 1.12 in kubernetes/features: After , the next step is to add the logic to the reconciler to detach the volumes when the nodeShutdown taint is applied. /assign /sig storage /kind feature looping
How do you know that node is Shutdown? is it a planned shutdown or a unplanned shutdown?
AFAIK regardless of whether it's planned or not, it's detected by the cloud provider. In case of bare-metal node it doesn't happen automatically, as there is no universal way to reliably determine if a node is powered off, so the node must be tainted as by some external source.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
any update on this issue?
, I am currently working with sig-node team on handling node shut down. I will make a proposal and discuss in sig-node meeting next Tuesday. Will post the update soon.
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten /kind bug /priority critical-urgent /assign /sig node
Those labels are not set on the issue: <detailsIn response to : /remove-kind feature Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
any update on this? is it going to be part of 1.15?
Stale issues rot after 30d of inactivity. Mark the issue as fresh with .


Question: // InstanceExistsByProviderID returns true if the instance with the given provider id still exists and is running. // If false is returned with no error, the instance will be immediately deleted by the cloud controller manager. func (vs *VSphere) InstanceExistsByProviderID(ctx context.Context, providerID string) (bool, error) { <del> var nodeName string nodes, err := vs.nodeManager.GetNodeDetails() </del> <ins> nodeName, err := vs.GetNodeNameFromProviderID(providerID) </ins> if err != nil { <del> glog.Errorf("Error while obtaining Kubernetes node nodeVmDetail details. error : %+v", err) </del> <ins> glog.Errorf("Error while getting nodename for providerID %s", providerID) </ins> return false, err } <del> for _, node := range nodes { // ProviderID is UUID for nodes v1.9.3+ if node.VMUUID == GetUUIDFromProviderID(providerID) || node.NodeName == providerID { nodeName = node.NodeName break } } if nodeName == "" { msg := fmt.Sprintf("Error while obtaining Kubernetes nodename for providerID %s.", providerID) return false, errors.New(msg) } </del> _, err = vs.InstanceID(ctx, convertToK8sType(nodeName)) if err == nil { return true, nil

Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
/priority important-longterm
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale /lifecycle frozen
/triage accepted /kind feature
This issue has not been updated in over 1 year, and should be re-triaged. You can: Confirm that this issue is still relevant with (org members only) Close this issue with For more details on the triage process, see /remove-triage accepted
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I think this can be closed now as we have non-graceful node shutdown.
/close
Closing this issue. <detailsIn response to : /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: . "github.com/onsi/gomega" ) <del> func runLivenessTest(c *client.Client, podDescr *api.Pod) { </del> <ins> func runLivenessTest(c *client.Client, podDescr *api.Pod, expectRestart bool) { </ins> ns := "e2e-test-" + string(util.NewUUID()) By(fmt.Sprintf("Creating pod %s in namespace %s", podDescr.Name, ns))

As far as I can tell, container liveness probes of type exec flat out don't work against clusters built at head. I had the kubelet always log the output of the exec'ed command, and it's always empty no matter what I exec, causing the probe to fail since it requires the output to be equal to the string "ok". If you want to test it, try adding a liveness probe of this form to any container: I also tried these lovely forms, to more closely match the liveness example under examples/: On the plus side, this may be an opportunistic time to switch over to
The reason that the e2e test for exec liveness probes hasn't caught this is that it only tests that containers are restarted when they fail liveness probes. It doesn't test that containers aren't restarted when their liveness checks should be succeeding. The e2e test's exec'ed commands also only ever produced empty output, rather than the expected "fail" output.
Is it possible to include this in a new pre-release, since it fixes a bug in health checking? I see it merged in master and I'm wondering what the policy is of releasing new code in a separate (0.16.*) release.


Question: import ( "fmt" <ins> "reflect" "sort" </ins> "strings" "testing"

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: I created several Workloads which require up to ten ports for both UDP and TCP protocol. Now my logs get spammed with . What you expected to happen: should consider -combination instead of only . How to reproduce it (as minimally and precisely as possible): Deploy any service with a node-port with both protocols, e.g.: Anything else we need to know?: This was already acknowledged by here: However, as this thread addresses a different bug and seems kind of stuck, i'd like to suggest that this might be addressed in a new context. Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Deployed via Rancher v2.4.3
/sig api-machinery
/remove-sig api-machinery /sig network
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
How did you get into this situation? The allocator should not have granted the same node port twice?
Oh right, that issue. The only real way out is to de-allocate the NodePorts and then re-allocate them, I think
As this happens for all of my ports in the said Production Application (the one where my log gets spammed) as well as complete fresh installation (where i verified the bug), i think there's no "way out" except fixing the bug. As said, for me it's necessary to assign both protocols to the same port. This might be a niche-use-case, but checking the it's actually very likely for ports to allow tcp AND udp. I'm not sure about the history of this, but it might be that issue was the origin, which was adressed in PR ("Allow services which use same port, different protocol to use the same nodePort for both"). But this PR doesn't contain the adjustment of the said , so still considers the behaviour as invalid, while it's actually allowed and supported.


Question: import ( "fmt" <ins> "reflect" "sort" </ins> "strings" "testing"

/assign
Ahh, I thought you meant that you had accidentally been allocated the same nodePorts and I was very confused when you said "all of them". I have reproduced this issue, and I can see it by inspection. The obvious fix would be to collect ports split by protocol, but I think that's not right since we don't store it split. An even simpler fix would be for to build a . Then for each protocol, for each port, build that contains all the port numbers, but remove any duplicates ONLY IF they are duplicated across protocols. Duplicates in the same protocol should be retained (to report an error later). I don't have time to tackle this tonight (and write a test), but it seems like a good bug.
perhaps I can fix this problem /assign
things may be a little more complicated. Right now the is not aware of the protocol of the port so I guess simply rewriting function to return is not enough, the port allocator will still return on same node port even with different protocol. perhaps we need to refactor the to make it protocol-aware.
I don't think we need to do make the allocator be protocol aware. IIRC we allow you to use the same nodeport on different protocols ONLY within a single service. This means (I think) that we can do hyper-local deduplication as we collect the nodeports for each service. If we find the same nodeport number with different protocols, just remove the duplicates because it is allowed. I.e. a service that looks like: return - deduped. if ANY OTHER service tried to use it would be an error at the "rebuilt" allocator Likewise if a single Service claims to use the same nodeport number more than once on the same protocol, it should be an error: return - NOT deduped.
obviously, we want a great unit test here :P
ah got it! thanks for your detailed explanation!
Hi! I have filed a PR to resolve this issue, could you take a look?


Question: return &serverList[0], nil } <del> func findAddrs(netblob interface{}) []string { // Run-time types for the win :( ret := []string{} list, ok := netblob.([]interface{}) if !ok { return ret } for _, item := range list { props, ok := item.(map[string]interface{}) if !ok { continue } tmp, ok := props["addr"] </del> <ins> func getAddressesByName(client *gophercloud.ServiceClient, name string) ([]api.NodeAddress, error) { srv, err := getServerByName(client, name) if err != nil { return nil, err } addrs := []api.NodeAddress{} for network, netblob := range srv.Addresses { list, ok := netblob.([]interface{}) </ins> if !ok { continue } <del> addr, ok := tmp.(string) if !ok { continue </del> <ins> for _, item := range list { var addressType api.NodeAddressType props, ok := item.(map[string]interface{}) if !ok { continue } extIPType, ok := props["OS-EXT-IPS:type"] if (ok && extIPType == "floating") || (!ok && network == "public") { addressType = api.NodeExternalIP } else { addressType = api.NodeInternalIP } tmp, ok := props["addr"] if !ok { continue } addr, ok := tmp.(string) if !ok { continue } api.AddToNodeAddresses(&addrs, api.NodeAddress{ Type:    addressType, Address: addr, }, ) </ins> } <del> ret = append(ret, addr) </del> } <del> return ret </del> <ins> // AccessIPs are usually duplicates of "public" addresses. if srv.AccessIPv4 != "" { api.AddToNodeAddresses(&addrs, api.NodeAddress{ Type:    api.NodeExternalIP, Address: srv.AccessIPv4, }, ) } if srv.AccessIPv6 != "" { api.AddToNodeAddresses(&addrs, api.NodeAddress{ Type:    api.NodeExternalIP, Address: srv.AccessIPv6, }, ) } return addrs, nil </ins> } <del> func getAddressByName(api *gophercloud.ServiceClient, name string) (string, error) { srv, err := getServerByName(api, name) </del> <ins> func getAddressByName(client *gophercloud.ServiceClient, name string) (string, error) { addrs, err := getAddressesByName(client, name) </ins> if err != nil { return "", err <ins> } else if len(addrs) == 0 { return "", ErrNoAddressFound </ins> } <del> var s string if s == "" { if tmp := findAddrs(srv.Addresses["private"]); len(tmp) >= 1 { s = tmp[0] } } if s == "" { if tmp := findAddrs(srv.Addresses["public"]); len(tmp) >= 1 { s = tmp[0] </del> <ins> for _, addr := range addrs { if addr.Type == api.NodeInternalIP { return addr.Address, nil </ins> } } <del> if s == "" { s = srv.AccessIPv4 } if s == "" { s = srv.AccessIPv6 } if s == "" { return "", ErrNoAddressFound } return s, nil </del> <ins> return addrs[0].Address, nil </ins> } // Implementation of Instances.CurrentNodeName

From we can find that, hardcode "private" and "public", seems only support that network with such name. , it fetch the first ip. usually, it can include both v6 and v4 ip. what's the meaning to just fetch the first one ? is it v4 or v6 ?
ping and
Yes. Zoom out a level or two in the code, and you'll see we're trying to either a) setup an openstack loadbalancer to point to the minion node (a single address) using , or b) report to kubernetes all the addresses we know about the node and whether they are "internal" or "external" in . All we have at the starting point is the server name (from kubernetes), and we need to make up the rest of this information somehow. (I agree (a) should reuse the NodeAddresses() function from (b), but the latter is a newer function and the code hasn't yet been refactored in that way. This doesn't really change the substance of this discussion.) Re (1): The logic as currently implemented is: Assume any network named "private" is "internal" Assume any network named "public" is "external" Assume the AccessIPs (IPv4 and IPv6) are "external" Any other network addresses are ignored. I implemented the first two ("private" and "public") following the Rackspace public cloud precedent, and on the basis that users in other OpenStack clouds can use these names if they want to encourage an address other than the Access addresses. For the (a) use case, we just take the first address we find in order of the above; for (b) we report that full list. I preferred "private" to "public" since (again, at least on Rackspace public cloud), the traffic on "private" network is sometimes cheaper/free. I agree this isn't great, and please suggest something better if you have a use case in mind. Perhaps we should pull a list of network names that should be considered "internal" and "external" from the provider config? Perhaps I could assume that provider networks are public and non-provider networks are internal? I don't want to add too much configuration here if there's a good algorithm that will Just Work.


Question: return &serverList[0], nil } <del> func findAddrs(netblob interface{}) []string { // Run-time types for the win :( ret := []string{} list, ok := netblob.([]interface{}) if !ok { return ret } for _, item := range list { props, ok := item.(map[string]interface{}) if !ok { continue } tmp, ok := props["addr"] </del> <ins> func getAddressesByName(client *gophercloud.ServiceClient, name string) ([]api.NodeAddress, error) { srv, err := getServerByName(client, name) if err != nil { return nil, err } addrs := []api.NodeAddress{} for network, netblob := range srv.Addresses { list, ok := netblob.([]interface{}) </ins> if !ok { continue } <del> addr, ok := tmp.(string) if !ok { continue </del> <ins> for _, item := range list { var addressType api.NodeAddressType props, ok := item.(map[string]interface{}) if !ok { continue } extIPType, ok := props["OS-EXT-IPS:type"] if (ok && extIPType == "floating") || (!ok && network == "public") { addressType = api.NodeExternalIP } else { addressType = api.NodeInternalIP } tmp, ok := props["addr"] if !ok { continue } addr, ok := tmp.(string) if !ok { continue } api.AddToNodeAddresses(&addrs, api.NodeAddress{ Type:    addressType, Address: addr, }, ) </ins> } <del> ret = append(ret, addr) </del> } <del> return ret </del> <ins> // AccessIPs are usually duplicates of "public" addresses. if srv.AccessIPv4 != "" { api.AddToNodeAddresses(&addrs, api.NodeAddress{ Type:    api.NodeExternalIP, Address: srv.AccessIPv4, }, ) } if srv.AccessIPv6 != "" { api.AddToNodeAddresses(&addrs, api.NodeAddress{ Type:    api.NodeExternalIP, Address: srv.AccessIPv6, }, ) } return addrs, nil </ins> } <del> func getAddressByName(api *gophercloud.ServiceClient, name string) (string, error) { srv, err := getServerByName(api, name) </del> <ins> func getAddressByName(client *gophercloud.ServiceClient, name string) (string, error) { addrs, err := getAddressesByName(client, name) </ins> if err != nil { return "", err <ins> } else if len(addrs) == 0 { return "", ErrNoAddressFound </ins> } <del> var s string if s == "" { if tmp := findAddrs(srv.Addresses["private"]); len(tmp) >= 1 { s = tmp[0] } } if s == "" { if tmp := findAddrs(srv.Addresses["public"]); len(tmp) >= 1 { s = tmp[0] </del> <ins> for _, addr := range addrs { if addr.Type == api.NodeInternalIP { return addr.Address, nil </ins> } } <del> if s == "" { s = srv.AccessIPv4 } if s == "" { s = srv.AccessIPv6 } if s == "" { return "", ErrNoAddressFound } return s, nil </del> <ins> return addrs[0].Address, nil </ins> } // Implementation of Instances.CurrentNodeName

I think in practice it's probably ok to continue to prefer "internal" to "external" for loadbalancer setup, but I can contrive hypothetical cases that need more flexibility here too. Again, if you have a concrete need for such a thing, please file a bug so we have some idea of the flexibility required. Complicating all this is that Kubernetes never actually defined what an "internal" address was. just says "Generally the IP address of the node that is routable only within the cluster" which isn't a sufficiently strict definition. Looking at where Kubernetes actually uses it: just uses the first address reported (preferring InternalIPs to ExternalIPs) and the Kubernetes master configures an "SSH Tunnel Check" which just uses the first ExternalIP ... which kind of makes it pointless to ever even bother reporting anything beyond the first address. In both cases, it's assumed that this address is reachable by the Kubernetes master nodes, which makes the Internal vs External distinction a bit pointless. Re (2): is meant to return more than one IP, and the loadbalancer setup can only use a single IP for each node (or would unnecessarily track/healthceck multiple paths to the same backend for multiple addresses). If you're seeing single addresses returned by then you should file a bug. All the above treats IPv4 and IPv6 as equally important, if the OpenStack cloud has both, and the "first" address used for the loadbalancer will be an arbitrary choice between the two protocols (not quite true: will use AccessIPv4 before AccessIPv6). I think that's how it should be (the rest of the code should be protocol agnostic), but I wouldn't be surprised if there's some part of Kubernetes that doesn't support IPv6 properly. There certainly used to be lots of places that assumed IPv4 in Kubernetes, and GCE doesn't support IPv6 iirc so it gets less testing - if you've found something that breaks with IPv6, please file bugs. Now this bug: what would you like done here? Would you like the current situation to be documented further? Is the "private"/"public" naming thing too restrictive for your usecase? Something else?


Question: return &serverList[0], nil } <del> func findAddrs(netblob interface{}) []string { // Run-time types for the win :( ret := []string{} list, ok := netblob.([]interface{}) if !ok { return ret } for _, item := range list { props, ok := item.(map[string]interface{}) if !ok { continue } tmp, ok := props["addr"] </del> <ins> func getAddressesByName(client *gophercloud.ServiceClient, name string) ([]api.NodeAddress, error) { srv, err := getServerByName(client, name) if err != nil { return nil, err } addrs := []api.NodeAddress{} for network, netblob := range srv.Addresses { list, ok := netblob.([]interface{}) </ins> if !ok { continue } <del> addr, ok := tmp.(string) if !ok { continue </del> <ins> for _, item := range list { var addressType api.NodeAddressType props, ok := item.(map[string]interface{}) if !ok { continue } extIPType, ok := props["OS-EXT-IPS:type"] if (ok && extIPType == "floating") || (!ok && network == "public") { addressType = api.NodeExternalIP } else { addressType = api.NodeInternalIP } tmp, ok := props["addr"] if !ok { continue } addr, ok := tmp.(string) if !ok { continue } api.AddToNodeAddresses(&addrs, api.NodeAddress{ Type:    addressType, Address: addr, }, ) </ins> } <del> ret = append(ret, addr) </del> } <del> return ret </del> <ins> // AccessIPs are usually duplicates of "public" addresses. if srv.AccessIPv4 != "" { api.AddToNodeAddresses(&addrs, api.NodeAddress{ Type:    api.NodeExternalIP, Address: srv.AccessIPv4, }, ) } if srv.AccessIPv6 != "" { api.AddToNodeAddresses(&addrs, api.NodeAddress{ Type:    api.NodeExternalIP, Address: srv.AccessIPv6, }, ) } return addrs, nil </ins> } <del> func getAddressByName(api *gophercloud.ServiceClient, name string) (string, error) { srv, err := getServerByName(api, name) </del> <ins> func getAddressByName(client *gophercloud.ServiceClient, name string) (string, error) { addrs, err := getAddressesByName(client, name) </ins> if err != nil { return "", err <ins> } else if len(addrs) == 0 { return "", ErrNoAddressFound </ins> } <del> var s string if s == "" { if tmp := findAddrs(srv.Addresses["private"]); len(tmp) >= 1 { s = tmp[0] } } if s == "" { if tmp := findAddrs(srv.Addresses["public"]); len(tmp) >= 1 { s = tmp[0] </del> <ins> for _, addr := range addrs { if addr.Type == api.NodeInternalIP { return addr.Address, nil </ins> } } <del> if s == "" { s = srv.AccessIPv4 } if s == "" { s = srv.AccessIPv6 } if s == "" { return "", ErrNoAddressFound } return s, nil </del> <ins> return addrs[0].Address, nil </ins> } // Implementation of Instances.CurrentNodeName

Hi, if we talked about openstack, I expect it not bind to rackspace, although, the kubernetes called RK SDK, But from my understanding, seems RK itself SDK not bind network to "private" and "pubic". User can define public network with their names, like site-network, for example. or "testnet", And for private, it should be seem. For nova list, the addresses, it can have "OS-EXT-IPS:type":"fixed" or "OS-EXT-IPS:type":"floating" for floating, I think what you means "Public", for fixed, I think its what you means "Private". In other words, we'd better to find related information from other fields in nova or neutron infromation. Instead of bind to "private" and "public" name.
I this issue and wrote a patch that implements what suggests. I will send a PR soon
Glad to see some improvements here. And wish to use it if implemented


Question: } trap cleanup EXIT <ins> goarch=$(go env GOARCH) </ins> kind build node-image --image "$tag" "$(pwd)" <del> curl -L --silent https://github.com/kind-ci/containerd-nightlies/releases/download/$containerd/$containerd-linux-amd64.tar.gz | tar -C "$tmpdir" -vzxf - curl -L --silent https://github.com/kind-ci/containerd-nightlies/releases/download/$containerd/runc.amd64 >"$tmpdir/runc" </del> <ins> curl -L --silent https://github.com/kind-ci/containerd-nightlies/releases/download/$containerd/$containerd-linux-"$goarch".tar.gz | tar -C "$tmpdir" -vzxf - curl -L --silent https://github.com/kind-ci/containerd-nightlies/releases/download/$containerd/runc."$goarch" >"$tmpdir/runc" </ins> cat >"$tmpdir/Dockerfile" <<EOF FROM $tag

I am trying to bring up a kind cluster with containerd in order to run e2e test on mac + kind. Not sure whether this is a bug or I missed some prerequisites. Any help would be greatly appreciated :) building the image dra/node:latest works as expected:bringing up the kind cluster failed (seems that kubelet is unhealthy and healthchecked failed) below the control plane logs: below the logs of one worker: successfully bring up the kind cluster with containerd Tried on M2 macbook pro I followed NONE Darwin MacBook-Pro-de- 22.1.0 Darwin Kernel Version 22.1.0: Sun Oct 9 20:19:12 PDT 2022; root:xnu-8792.41.9~3/RELEASEARM64T6020 arm64 arm Darwin No response No response
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
what type of mac you're using? for m1/m2 mac, you need to replace the linux-amd64 with linux-arm64(your log seems like using m1/m2 mac which same as the problem I've met before)
thanks I am going to try that
ping me if you can run e2e on mac. I setup the cluster but failed to run the e2e test..
can you please share the exact command you tried and the failure ?
If you use a newer version of kind you might not need to clobber the included containerd, not sure what version is required And yes, on M1/M2 you need to use arm binaries/images.
this is the command that I tried !


Question: } trap cleanup EXIT <ins> goarch=$(go env GOARCH) </ins> kind build node-image --image "$tag" "$(pwd)" <del> curl -L --silent https://github.com/kind-ci/containerd-nightlies/releases/download/$containerd/$containerd-linux-amd64.tar.gz | tar -C "$tmpdir" -vzxf - curl -L --silent https://github.com/kind-ci/containerd-nightlies/releases/download/$containerd/runc.amd64 >"$tmpdir/runc" </del> <ins> curl -L --silent https://github.com/kind-ci/containerd-nightlies/releases/download/$containerd/$containerd-linux-"$goarch".tar.gz | tar -C "$tmpdir" -vzxf - curl -L --silent https://github.com/kind-ci/containerd-nightlies/releases/download/$containerd/runc."$goarch" >"$tmpdir/runc" </ins> cat >"$tmpdir/Dockerfile" <<EOF FROM $tag

It worked for me just fine on M2 machine with this obvious fix for the build script: Here is a test output:
I'll submit a PR with the fix. Updating the documentation would be also nice to have. I hope and can do that.
DRA tests fail on kind 0.18.0. Which Containerd version is used there?
It was the latest 1.6.x at release time. 1.6 is their LTS branch before 2.0 comes which has some tricky changes for us to deal with (notably the CRI hosts config deprecation which will break all our local registry users :/)
Unfortunately 1.6 doesn't have CDI support, which DRA depends on. The version used in the build script is 1.6.0-830-g34d078e99, but it was probably built from containerd master after 1.6 was released.
thanks for providing the details. you could run the e2e as shown by and find the result


Question: VolumeMounts:  volumeMounts, LivenessProbe: componentProbe(10252, "/healthz"), Resources:     componentResources("200m"), <ins> Env:           getProxyEnvVars(), </ins> }, volumes...), kubeScheduler: componentPod(api.Container{ Name:          kubeScheduler,

Is this a request for help? No What keywords did you search in Kubernetes issues before filing this one? http proxy, httpproxy, httpsproxy, AWS, kubeadm, kubectl, cloud-provider Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT Kubernetes version (use ): v1.4.4 Environment: Cloud provider or hardware configuration: kubeadm on EC2 instances on AWS OS (e.g. from /etc/os-release): Ubuntu 16.04.1 LTS Kernel (e.g. ): Linux ip-10-0-133-99 4.4.0-45-generic -Ubuntu SMP Wed Oct 19 14:12:37 UTC 2016 x8664 x8664 x8664 GNU/Linux Install tools: kubeadm Others: httpproxy, httpsproxy and noproxy set in /etc/environment, Docker config as well on startup scripts that run What happened: won't complete and gets stuck on it's first deployment. When one looks through the logs it shows that the pods are downloading fine but we are stuck due to the kube-controller-manager continually crashing as it can't contact AWS. Elsewhere in kubernetes I see references to and if the AWS cloud provider could use that it would be able to connect as well. From the POD if I manually set httpproxy and httpsproxy and no_proxy I am able to use the AWS CLI without any issues. What you expected to happen: I would expect to successfully complete and use the environment proxies set up on the system so as not to depend on a default Internet Gateway being connected to the VPC just for AWS access. How to reproduce it (as minimally and precisely as possible): Set up a public VPC with Squid set up as a proxy Set up a private VPC peered to the public VPC Set up a master node and run Set up a regular node and run will never fully initialize. If you remove the --cloud-provider aws` then it will complete and nodes will register. Anything else do we need to know:
what do you have in proxy variables ? (env | grep -i proxy) it tries to connect to (tcp 176.32.118.53:443) directly. You're expecting to connect there via proxy ? then do you have something like in your no_proxy ?


Question: VolumeMounts:  volumeMounts, LivenessProbe: componentProbe(10252, "/healthz"), Resources:     componentResources("200m"), <ins> Env:           getProxyEnvVars(), </ins> }, volumes...), kubeScheduler: componentPod(api.Container{ Name:          kubeScheduler,

Unfortunately on the pod itself for the the environment variables aren't passed through. As this is created using I don't have control over the pod definition. So basically none of the *_proxy variables are set inside the pod in the docker image, however if I and export the variables and try to wget that URL it works. Basically I'm hoping we can update to respect and pass the environment variables down to the system pods if they exist, eg, add: Without it I can't stand up a cluster with AWS integration on a completely private VPC and for such a small change it would make a rather big difference.
ok, so request is actually to pass proxy variables into containers. interesting. I'll check how it can be done. BTW, for noproxy value in your example: things like 10.0.0.0/8 would not work on 99% of software that respects noproxy variable. it must be always exact IP match.
you're right that what I'm actually asking for is to pass proxy variables through either from a commandline setting on kubeadm or pick it up from /etc/environment or shell environment that launches kubeadm. I didn't realize it till I dug in to try and think about how one might go about fixing this. At first I thought we would just pick up the variables from the system environment but then I realized kube-controller-manager is deployed in a container and so we'd need to update the yaml file that launches it to contain the environment variables. As I'd like to go to production with this soon hopefully this is a small change that can be .
This issue was moved to kubernetes/kubeadm


Question: dial = (&net.Dialer{Timeout: 30 * time.Second, KeepAlive: 30 * time.Second}).DialContext } d := connrotation.NewDialer(dial) <del> a.onRotate = d.CloseAll </del> <ins> a.mu.Lock() defer a.mu.Unlock() a.onRotateList = append(a.onRotateList, d.CloseAll) onRotateListLength := len(a.onRotateList) if onRotateListLength > onRotateListWarningLength { klog.Warningf("constructing many client instances from the same exec auth config can cause performance problems during cert rotation and can exhaust available network connections; %d clients constructed calling %q", onRotateListLength, a.cmd) } </ins> c.Dial = d.DialContext return nil

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Data race in client-go with below sample code: Data race: What you expected to happen: No data race How to reproduce it (as minimally and precisely as possible): Using above given sample code Kubernetes version (use ):Cloud provider or hardware configuration:OS (e.g: ): OSX
/sig api-machinery
I'd like to take it. /assign
/remove-sig api-machinery /sig auth cli
Bug triage for release 1.17 here. Since this issue has been tagged for the milestone, we would like to check it's status. Code freeze is on Nov. 14. Will this issue be resolved before then?
Another reminder here that code freeze is in less than two weeks (Nov. 14). We will need to move this issue out of the milestone if isn't going to be resolved before then.


Question: fs.StringVar(&o.ConfigFile, "config", o.ConfigFile, "The path to the configuration file.") fs.StringVar(&o.WriteConfigTo, "write-config-to", o.WriteConfigTo, "If set, write the default configuration values to this file and exit.") fs.StringVar(&o.config.ClientConnection.Kubeconfig, "kubeconfig", o.config.ClientConnection.Kubeconfig, "Path to kubeconfig file with authorization information (the master location can be overridden by the master flag).") <del> fs.StringVar(&o.config.ClusterCIDR, "cluster-cidr", o.config.ClusterCIDR, "The CIDR range of pods in the cluster. When configured, traffic sent to a Service cluster IP from outside this range will be masqueraded and traffic sent from pods to an external LoadBalancer IP will be directed to the respective cluster IP instead") </del> <ins> fs.StringVar(&o.config.ClusterCIDR, "cluster-cidr", o.config.ClusterCIDR, "The CIDR range of pods in the cluster. When configured, traffic sent to a Service cluster IP from outside this range will be masqueraded and traffic sent from pods to an external LoadBalancer IP will be directed to the respective cluster IP instead. This parameter is ignored if a config file is specified by --config.") </ins> fs.StringVar(&o.config.ClientConnection.ContentType, "kube-api-content-type", o.config.ClientConnection.ContentType, "Content type of requests sent to apiserver.") fs.StringVar(&o.master, "master", o.master, "The address of the Kubernetes API server (overrides any value in kubeconfig)") fs.StringVar(&o.hostnameOverride, "hostname-override", o.hostnameOverride, "If non-empty, will use this string as identification instead of the actual hostname.")

Start service with a config file without specifying the argument in the config file, instead, pass the value via the CLI args, the value is just ignored, for example, I have a config file as this, NOTE: I have not specified the in the config file, but give a value for the via the CLI, the proxy mode is still set as after the service is started. The reason for this is that CLI args is parsed before the loading of the config file, if the config file haven't specified the or leave it as empty, then it will be use the default value instead. Two choices from my side, Add a note somewhere, maybe where the option is defined, just say that the CLI args will be ignored when the config file is provided regardless the option is specified in the config file or not (this should be noted for lots of options). Merge the value passed from CLI with the config file if the option has an empty value, but we need to consider the backward compatibility. and check which mode is effective. No response_ vanilla deployment with code in the upstream. <details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig network Hey Tim, WDYT? /assign
Heh, this is a recurrent topic :smile: , you did a great analysis: I think that explains here why merging flags and config overriding defaults is not a desirable option (the whole thread is worth reading) I'm in favor of documenting the behavior
thanks for showing the earlier thread, I have created a simple PR to document this in the CLI option.


Question: <del> /* Copyright 2023 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package plugin import ( "context" "fmt" "net" "os" "path/filepath" "sync" "testing" "github.com/stretchr/testify/assert" "google.golang.org/grpc" drapb "k8s.io/kubelet/pkg/apis/dra/v1alpha4" "k8s.io/kubernetes/test/utils/ktesting" ) const ( v1alpha4Version = "v1alpha4" ) type fakeV1alpha4GRPCServer struct { drapb.UnimplementedNodeServer } var _ drapb.NodeServer = &fakeV1alpha4GRPCServer{} func (f *fakeV1alpha4GRPCServer) NodePrepareResources(ctx context.Context, in *drapb.NodePrepareResourcesRequest) (*drapb.NodePrepareResourcesResponse, error) { return &drapb.NodePrepareResourcesResponse{Claims: map[string]*drapb.NodePrepareResourceResponse{"claim-uid": { Devices: []*drapb.Device{ { RequestNames: []string{"test-request"}, CDIDeviceIDs: []string{"test-cdi-id"}, }, }, }}}, nil } func (f *fakeV1alpha4GRPCServer) NodeUnprepareResources(ctx context.Context, in *drapb.NodeUnprepareResourcesRequest) (*drapb.NodeUnprepareResourcesResponse, error) { return &drapb.NodeUnprepareResourcesResponse{}, nil } type tearDown func() func setupFakeGRPCServer(version string) (string, tearDown, error) { p, err := os.MkdirTemp("", "dra_plugin") if err != nil { return "", nil, err } closeCh := make(chan struct{}) addr := filepath.Join(p, "server.sock") teardown := func() { close(closeCh) os.RemoveAll(addr) } listener, err := net.Listen("unix", addr) if err != nil { teardown() return "", nil, err } s := grpc.NewServer() switch version { case v1alpha4Version: fakeGRPCServer := &fakeV1alpha4GRPCServer{} drapb.RegisterNodeServer(s, fakeGRPCServer) default: return "", nil, fmt.Errorf("unsupported version: %s", version) } go func() { go s.Serve(listener) <-closeCh s.GracefulStop() }() return addr, teardown, nil } func TestGRPCConnIsReused(t *testing.T) { tCtx := ktesting.Init(t) addr, teardown, err := setupFakeGRPCServer(v1alpha4Version) if err != nil { t.Fatal(err) } defer teardown() reusedConns := make(map[*grpc.ClientConn]int) wg := sync.WaitGroup{} m := sync.Mutex{} p := &Plugin{ backgroundCtx:     tCtx, endpoint:          addr, clientCallTimeout: defaultClientCallTimeout, } conn, err := p.getOrCreateGRPCConn() defer func() { err := conn.Close() if err != nil { t.Error(err) } }() if err != nil { t.Fatal(err) } // ensure the plugin we are using is registered draPlugins.add("dummy-plugin", p) defer draPlugins.delete("dummy-plugin") // we call `NodePrepareResource` 2 times and check whether a new connection is created or the same is reused for i := 0; i < 2; i++ { wg.Add(1) go func() { defer wg.Done() client, err := NewDRAPluginClient("dummy-plugin") if err != nil { t.Error(err) return } req := &drapb.NodePrepareResourcesRequest{ Claims: []*drapb.Claim{ { Namespace: "dummy-namespace", UID:       "dummy-uid", Name:      "dummy-claim", }, }, } _, err = client.NodePrepareResources(tCtx, req) assert.NoError(t, err) client.mutex.Lock() conn := client.conn client.mutex.Unlock() m.Lock() defer m.Unlock() reusedConns[conn]++ }() } wg.Wait() // We should have only one entry otherwise it means another gRPC connection has been created if len(reusedConns) != 1 { t.Errorf("expected length to be 1 but got %d", len(reusedConns)) } if counter, ok := reusedConns[conn]; ok && counter != 2 { t.Errorf("expected counter to be 2 but got %d", counter) } } func TestNewDRAPluginClient(t *testing.T) { for _, test := range []struct { description string setup       func(string) tearDown pluginName  string shouldError bool }{ { description: "plugin name is empty", setup: func(_ string) tearDown { return func() {} }, pluginName:  "", shouldError: true, }, { description: "plugin name not found in the list", setup: func(_ string) tearDown { return func() {} }, pluginName:  "plugin-name-not-found-in-the-list", shouldError: true, }, { description: "plugin exists", setup: func(name string) tearDown { draPlugins.add(name, &Plugin{}) return func() { draPlugins.delete(name) } }, pluginName: "dummy-plugin", }, } { t.Run(test.description, func(t *testing.T) { teardown := test.setup(test.pluginName) defer teardown() client, err := NewDRAPluginClient(test.pluginName) if test.shouldError { assert.Nil(t, client) assert.Error(t, err) } else { assert.NotNil(t, client) assert.Nil(t, err) } }) } } func TestNodeUnprepareResources(t *testing.T) { for _, test := range []struct { description   string serverSetup   func(string) (string, tearDown, error) serverVersion string request       *drapb.NodeUnprepareResourcesRequest }{ { description:   "server supports v1alpha4", serverSetup:   setupFakeGRPCServer, serverVersion: v1alpha4Version, request:       &drapb.NodeUnprepareResourcesRequest{}, }, } { t.Run(test.description, func(t *testing.T) { tCtx := ktesting.Init(t) addr, teardown, err := setupFakeGRPCServer(test.serverVersion) if err != nil { t.Fatal(err) } defer teardown() p := &Plugin{ backgroundCtx:     tCtx, endpoint:          addr, clientCallTimeout: defaultClientCallTimeout, } conn, err := p.getOrCreateGRPCConn() defer func() { err := conn.Close() if err != nil { t.Error(err) } }() if err != nil { t.Fatal(err) } draPlugins.add("dummy-plugin", p) defer draPlugins.delete("dummy-plugin") client, err := NewDRAPluginClient("dummy-plugin") if err != nil { t.Fatal(err) } _, err = client.NodeUnprepareResources(tCtx, test.request) if err != nil { t.Fatal(err) } }) } } </del>

The PR above moved everything related to into because that was where most of the code already was, but didn't renamed the files to avoid a larger diff and merge conflicts. Once is merged, we can do the renaming. Code clarity. /sig node
/triage accepted /priority important-soon
/assign
a PR can be created for this now.
will do


Question: ) const ( <del> // When these values are updated, also update test/e2e/framework/util.go </del> <ins> // When these values are updated, also update test/utils/image/manifest.go </ins> defaultPodSandboxImageName    = "gcr.io/google_containers/pause" <del> defaultPodSandboxImageVersion = "3.0" </del> <ins> defaultPodSandboxImageVersion = "3.1" </ins> // From pkg/kubelet/rkt/rkt.go to avoid circular import defaultRktAPIServiceEndpoint = "localhost:15441" )

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: the pause-amd64:3.0 image on has an empty schema2 manifest, which prevents is from being pulled by proxy/caches which only support V2 registries. What you expected to happen: Being able to pull ` using a V2 manifest compatible client. Anything else we need to know?: The proxy/cahce is requesting this
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
After talking with , he mentionned the following
AFAIK, does not support well. After inspecting the image, you can find is still using manifest v1.
/cc FYI
Im not sure which exact version x schéma X is needed. What I know is that we can download through the proxy/cache all other Kubernetes kube-system images, only the pause image is giving us trouble. Therefore, I would say that having the image available using the same format as the other kube-system images would be the request here.
Yeah, we haven't updated the pause image in a while, since it doesn't really ever change. I'm not sure whether rebuilding + pushing over the existing tag is safe, or if we should bump to or something. thoughts?
If we do rebuilding/updating the image, that would help and a bunch of similar issues.
it's better/save to go with new tag
I didn't realize there were changes to the pause binary that were never actually deployed. It seems like we should probably just do that (and bump the tag).
that sounds great. Any idea when the existing 3.0 tag could be pushed again?
We shouldn't push over the 3.0 tag; we should push a 3.1 or 4.0 tag instead. I'm not sure who owns the pause container who should drive this. Maybe sig-node and/or
It seems like it should be sig-node. I'll look into it.
The new container has been built, can you test with
Its working! thanks! I was able to pull it through our artifactory proxy , Lee Verberne <:
excellent, thanks for checking. /assign


Question: ) const ( <del> // When these values are updated, also update test/e2e/framework/util.go </del> <ins> // When these values are updated, also update test/utils/image/manifest.go </ins> defaultPodSandboxImageName    = "gcr.io/google_containers/pause" <del> defaultPodSandboxImageVersion = "3.0" </del> <ins> defaultPodSandboxImageVersion = "3.1" </ins> // From pkg/kubelet/rkt/rkt.go to avoid circular import defaultRktAPIServiceEndpoint = "localhost:15441" )

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: pause does not reap zombie processes using the default pause container provided by What you expected to happen: should include the up to date pause code to reap zombie processes when using docker = 1.13.1 found here How to reproduce it (as minimally and precisely as possible): a.out will leave a zombie process that is not cleaned up Anything else we need to know?: Building the pause container from master solves this issue Related: Environment: Kubernetes version (use ): 1.7.0+ Cloud provider or hardware configuration**: N/A OS (e.g. from /etc/os-release): CentOS 7 Kernel (e.g. ): 4.10.13+ Install tools: Others: docker = 1.13.1
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
i get a simliar problem when i run crond service in pod container, every day it makes some [anacron] zombie progresses UID PID PPID C STIME TTY TIME CMD root 1 0 0 Nov14 ? 00:00:00 /pause root 7 0 0 Nov14 ? 00:00:00 /usr/local/bin/dumb-init /usr/lo root 13 7 0 Nov14 ? 00:00:00 nginx: master process /usr/local nobody 21 13 0 Nov14 ? 00:00:01 nginx: worker process nobody 22 13 0 Nov14 ? 00:00:07 nginx: worker process nobody 23 13 0 Nov14 ? 00:00:07 nginx: worker process nobody 24 13 0 Nov14 ? 00:00:05 nginx: worker process root 77 1 0 Nov14 ? 00:00:00 [anacron] root 1 0 00:01 ? 00:00:00 [anacron] <defunctroot 1 0 01:01 ? 00:00:00 [anacron] <defunctroot 1 0 02:01 ?


Question: ) const ( <del> // When these values are updated, also update test/e2e/framework/util.go </del> <ins> // When these values are updated, also update test/utils/image/manifest.go </ins> defaultPodSandboxImageName    = "gcr.io/google_containers/pause" <del> defaultPodSandboxImageVersion = "3.0" </del> <ins> defaultPodSandboxImageVersion = "3.1" </ins> // From pkg/kubelet/rkt/rkt.go to avoid circular import defaultRktAPIServiceEndpoint = "localhost:15441" )

00:00:00 [anacron] <defunctroot 1 0 03:01 ? 00:00:00 [anacron] <defunctroot 0 0 11:49 ? 00:00:00 /bin/bash root 0 1 19:15 ? 00:00:00 /bin/bash root 0 19:15 ? 00:00:00 ps -ef !
Do we know why this was never resolved / was never deployed?
I just fought through this issue with a rabbitmq helm chart. I had to custom build a pause container from master to fix it. The rabbitmq helm chart uses for its liveness checks. It's a thin wrapper around a call to the rabbitmq-server daemon, which is not PID 1 (/pause is PID 1). The rabbitmq-server process forks some children to do the work and then they exit. Since the pause-amd64:3.0 is not reaping, those processes stick around as zombies. The liveness check runs every 5 seconds. After about 4 hours, there are about 27K zombies and the node begins failing to launch new containers because it's starved for resources. The pause container built from current master fixes that behavior, no zombies at all in the container.
The plan is to build a new version of pause-amd64 and then update the image referenced by Kubernetes. This means the default image will change in the next Kubernetes release. We can make the new image available once the freeze lifts and we can merge .
Looks like this made it into 1.10


Question: e := exec.New() hostIfName, err := findPairInterfaceOfContainerInterface(e, containerInterfaceName, containerDesc, nsenterArgs) if err != nil { <del> glog.Infof("Unable to find pair interface, setting up all interfaces: %v", err) return setUpAllInterfaces() </del> <ins> return err </ins> } return setUpInterface(hostIfName) }

From another issue, said: DockerManager.SyncPod() calls the network Plugin and then sets hairpin mode unilaterally. This is wrong. We should either pass the bool down to here or else we should get rid of the default no-op plugin and use the non-existence of a plugin as a flag to control this. This should be a relatively easy one for people who want to contribute but don't know where to start.
I volunteer myself to do this. Actually I have done something about kubernetes network plugin so guess I can do this. I will start to do this when I am on the office tomorrow.
Awesome. I'd advise starting with a sketch so we can all make sure it is a good approach, before jumping in with both feet. , TonyAdo wrote:
ok, I will start with a sketch as soon as possible.
I thought about this for a while: if we pass the bool down to here, of course we can check this flag, if it is set, we should set veth hairpin mode, but problem is that if user set this flag, but still use a custom network plugin just like you mentioned the ovs plugin, I am afraid we will still enable hairpin mode in all interfaces on this host, since we cannot find a peer interface. Is that correct?
Yeah, I think the better fix is to use the existence of a network plugin as an indicator. It sort of makes --configure-cbr0 redundant , TonyAdo wrote:
ok, I will try to get rid of the default no-op plugin and use the non-existence of a plugin as a flag to control this.
I decided to use the existence of a network plugin as an indicator, also I will get ride of the no-op network plugin, I am not going to add new flag to kubelet structure, if the user don't specify network plugin name, the networkPlugin within kubelet will be which just means non-existence of a plugin, and we have to check the existence in the docker manager to avoid nil pointer exception also for setting, I think if a custom network plugin exists, kubelet should not do this. How about this? Should I open a PR?
fyi, saw a hairpin pr today that might be related. ymmv.
what do you think about this?
What you described sounds right to me.


Question: e := exec.New() hostIfName, err := findPairInterfaceOfContainerInterface(e, containerInterfaceName, containerDesc, nsenterArgs) if err != nil { <del> glog.Infof("Unable to find pair interface, setting up all interfaces: %v", err) return setUpAllInterfaces() </del> <ins> return err </ins> } return setUpInterface(hostIfName) }

I think the only way to see if there is some detail I am missing is to try it and see if anything pops up that would make this not work. , TonyAdo wrote:
I need to check which part is responsible of what, but what if the hairpin package was just a utility package that should be called by the plugin if it needs to set this up? As long as it is single and simple function call, doing so would avoid adding extra logic here and fix the issue. This is valid only if the aim is to have a "linux bridge plugin", an "OVS plugin", etc.
actually kubelet will call after get called, I think to make the network plugin can call hairpin package is not that easy, of course, I agree that network plugin should setup hairpin mode if it really needs, kubelet should not set hairpin mode unilaterally. Besides, kubelet is not going to provide an "linux bridge plugin" or "OVS plugin".
then the validity conditions of my comment are not met, so I see any simpler way. Thanks for checking! :)
still what do you think about disable hairpin completely and leave this to network plugin?
or if there is a custom network plugin, we don't setup hairpin mode, only do this in default.
Completely? It will break my use case (and anyone using Apache Kafka).
Only by default is what I understood from the previous comments, and LGTM.
Really sorry about that, but I really want to know are you using a kubelet network plugin in your use case? If no, I think we will still setup hairpin, as in default we will do this.
I don't use any plugin, so it is ok if hairpin is set up by default.
ok, I think this will not break your use case.
It's not really the network plugin that needs hairpinmode though, it's the pure-iptables-based kube-proxy code. It's possible the proxying code could change again in the future in such a way that hairpinmode wouldn't be needed any more. So it seems wrong to put that code into the plugin. makes it so that we set hairpinmode on the veth if it's attached to a bridge, but just silently do nothing if it's not.


Question: e := exec.New() hostIfName, err := findPairInterfaceOfContainerInterface(e, containerInterfaceName, containerDesc, nsenterArgs) if err != nil { <del> glog.Infof("Unable to find pair interface, setting up all interfaces: %v", err) return setUpAllInterfaces() </del> <ins> return err </ins> } return setUpInterface(hostIfName) }

(Yes, we currently set hairpinmode even if you're using the userspace proxier, but AFAIK it's not needed in that case.)
IMHO... The proxy has the responsibility of service objects, while the kubelet is responsible of pod objects, including their networking. Also, the responsibility of hairpin must be at the same place as the code that creates the network interface, otherwise you have code managing a single object (the pod interface) spread over multiple programs, which will eventually lead to complex problems and a system harder to understand even for the end-user. Thus, it doesn't seems wrong to me that the kubelet does whatever is required to make the pod work in the environment of the cluster. While its true that the hairpin requirement is only for the proxy in iptables mode, a switch to this mode can happen at any time in the life of the cluster. And when it happens, its probably because the userland proxy becomes unresponsive, which means in a situation where the administrator is in a hurry to find a solution. Currently, you just change the proxy mode and restart the proxies. There's little chance the administrator will think about hairpin or restarting the kubelet in this case, so the kubelet should be reacting to this; which means the proxy and the kubelet of a given host should be able to communicate (AFAIK, this doesn't exist now). And all this is currently only to avoid setting up hairpin on the pod's port of standard Linux bridge.
Feel sure I'm missing something but... The CNI bridge plugin now has a flag, so kubenet can use that. Having done that, we can set false when the plugin name is non-blank.
Is this fixed now?
I apologise for leaving a somewhat vague answer, but I have been in that code recently. I don't think it's fixed.
, Bryan Boreham wrote:
Thanks for the PR. CNI Bridge plugin is expecting . Not . Hence the bridge plugin did not recognize it.
EDIT 2016-12-09: The chain below was broken by , so the issue can be closed again (remain closed) I suggest this should be re-opened. The chain of calls is: sets to , e.g. when using CNI calls something goes wrong and it sets the mode on everything. An example of "something goes wrong":


Question: framework.ExpectNoError(verifyServeHostnameServiceUp(cs, ns, host, podNames2, svc2IP, servicePort)) }) <del> ginkgo.It("should be able to preserve UDP traffic when server pod cycles for a NodePort service", func() { serviceName := "clusterip-test" serverPod1Name := "server-1" serverPod2Name := "server-2" ns := f.Namespace.Name nodeIP, err := e2enode.PickIP(cs) // for later framework.ExpectNoError(err) // Create a NodePort service udpJig := e2eservice.NewTestJig(cs, ns, serviceName) ginkgo.By("creating a UDP service " + serviceName + " with type=NodePort in " + ns) udpService, err := udpJig.CreateUDPService(func(svc *v1.Service) { svc.Spec.Type = v1.ServiceTypeNodePort svc.Spec.Ports = []v1.ServicePort{ {Port: 80, Name: "http", Protocol: v1.ProtocolUDP, TargetPort: intstr.FromInt(80)}, } }) framework.ExpectNoError(err) // Add a backend pod to the service ginkgo.By("creating a backend pod for the service " + serviceName) serverPod1 := newAgnhostPod(serverPod1Name, "netexec", fmt.Sprintf("--udp-port=%d", 80)) serverPod1.Labels = udpJig.Labels _, err = cs.CoreV1().Pods(ns).Create(context.TODO(), serverPod1, metav1.CreateOptions{}) ginkgo.By(fmt.Sprintf("checking NodePort service %s on node with public IP %s", serviceName, nodeIP)) framework.ExpectNoError(err) framework.ExpectNoError(e2epod.WaitTimeoutForPodReadyInNamespace(f.ClientSet, serverPod1.Name, f.Namespace.Name, framework.PodStartTimeout)) // Waiting for service to expose endpoint. err = validateEndpointsPorts(cs, ns, serviceName, portsByPodName{serverPod1Name: {80}}) framework.ExpectNoError(err, "failed to validate endpoints for service %s in namespace: %s", serviceName, ns) // Check that the pod reveives the traffic ginkgo.By("Sending UDP traffic to NodePort service " + serviceName + " on node with publicIP " + nodeIP) errorChannel := make(chan error) signal := make(chan struct{}, 1) go continuousEcho(nodeIP, int(udpService.Spec.Ports[0].NodePort), 3*time.Second, 20, signal, errorChannel) // Create a second pod ginkgo.By("creating a second pod for the service " + serviceName) serverPod2 := newAgnhostPod(serverPod2Name, "netexec", fmt.Sprintf("--udp-port=%d", 80)) serverPod2.Labels = udpJig.Labels _, err = cs.CoreV1().Pods(ns).Create(context.TODO(), serverPod2, metav1.CreateOptions{}) framework.ExpectNoError(err) framework.ExpectNoError(e2epod.WaitTimeoutForPodReadyInNamespace(f.ClientSet, serverPod2.Name, f.Namespace.Name, framework.PodStartTimeout)) // and delete the first pod framework.Logf("Cleaning up %s pod", serverPod1Name) err = cs.CoreV1().Pods(ns).Delete(context.TODO(), serverPod1Name, metav1.DeleteOptions{}) framework.ExpectNoError(err, "failed to delete pod: %s on node", serverPod1Name) // Check that the second pod keeps receiving traffic ginkgo.By("Sending UDP traffic to NodePort service " + serviceName + " on node with publicIP " + nodeIP) signal <- struct{}{} // Check that there are no errors err = <-errorChannel framework.ExpectNoError(err, "pod communication failed") }) </del> /* Release : v1.16 Testname: Service, NodePort Service

Which jobs are flaking: ci-kubernetes-e2e-gce-scale-correctness Which test(s) are flaking: [sig-network] Services should be able to preserve UDP traffic when server pod cycles for a NodePort service Testgrid link: https://k8s- Reason for failure: /sig network /assign
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
CRI-O CI is also hitting this failure. We have been unable to reproduce locally. our CI runs on aws
hmm, I only see 1 failure out of 4 and seems that the connectivity didn't work at all
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
/assign
GitHub didn't allow me to assign the following users: JacobTanenbaum. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc Saw the test flaking recently: Wondering if this is just a timeout issue.


Question: framework.ExpectNoError(verifyServeHostnameServiceUp(cs, ns, host, podNames2, svc2IP, servicePort)) }) <del> ginkgo.It("should be able to preserve UDP traffic when server pod cycles for a NodePort service", func() { serviceName := "clusterip-test" serverPod1Name := "server-1" serverPod2Name := "server-2" ns := f.Namespace.Name nodeIP, err := e2enode.PickIP(cs) // for later framework.ExpectNoError(err) // Create a NodePort service udpJig := e2eservice.NewTestJig(cs, ns, serviceName) ginkgo.By("creating a UDP service " + serviceName + " with type=NodePort in " + ns) udpService, err := udpJig.CreateUDPService(func(svc *v1.Service) { svc.Spec.Type = v1.ServiceTypeNodePort svc.Spec.Ports = []v1.ServicePort{ {Port: 80, Name: "http", Protocol: v1.ProtocolUDP, TargetPort: intstr.FromInt(80)}, } }) framework.ExpectNoError(err) // Add a backend pod to the service ginkgo.By("creating a backend pod for the service " + serviceName) serverPod1 := newAgnhostPod(serverPod1Name, "netexec", fmt.Sprintf("--udp-port=%d", 80)) serverPod1.Labels = udpJig.Labels _, err = cs.CoreV1().Pods(ns).Create(context.TODO(), serverPod1, metav1.CreateOptions{}) ginkgo.By(fmt.Sprintf("checking NodePort service %s on node with public IP %s", serviceName, nodeIP)) framework.ExpectNoError(err) framework.ExpectNoError(e2epod.WaitTimeoutForPodReadyInNamespace(f.ClientSet, serverPod1.Name, f.Namespace.Name, framework.PodStartTimeout)) // Waiting for service to expose endpoint. err = validateEndpointsPorts(cs, ns, serviceName, portsByPodName{serverPod1Name: {80}}) framework.ExpectNoError(err, "failed to validate endpoints for service %s in namespace: %s", serviceName, ns) // Check that the pod reveives the traffic ginkgo.By("Sending UDP traffic to NodePort service " + serviceName + " on node with publicIP " + nodeIP) errorChannel := make(chan error) signal := make(chan struct{}, 1) go continuousEcho(nodeIP, int(udpService.Spec.Ports[0].NodePort), 3*time.Second, 20, signal, errorChannel) // Create a second pod ginkgo.By("creating a second pod for the service " + serviceName) serverPod2 := newAgnhostPod(serverPod2Name, "netexec", fmt.Sprintf("--udp-port=%d", 80)) serverPod2.Labels = udpJig.Labels _, err = cs.CoreV1().Pods(ns).Create(context.TODO(), serverPod2, metav1.CreateOptions{}) framework.ExpectNoError(err) framework.ExpectNoError(e2epod.WaitTimeoutForPodReadyInNamespace(f.ClientSet, serverPod2.Name, f.Namespace.Name, framework.PodStartTimeout)) // and delete the first pod framework.Logf("Cleaning up %s pod", serverPod1Name) err = cs.CoreV1().Pods(ns).Delete(context.TODO(), serverPod1Name, metav1.DeleteOptions{}) framework.ExpectNoError(err, "failed to delete pod: %s on node", serverPod1Name) // Check that the second pod keeps receiving traffic ginkgo.By("Sending UDP traffic to NodePort service " + serviceName + " on node with publicIP " + nodeIP) signal <- struct{}{} // Check that there are no errors err = <-errorChannel framework.ExpectNoError(err, "pod communication failed") }) </del> /* Release : v1.16 Testname: Service, NodePort Service

Maybe we can increase the timeout and/or switch to a TCP based test:
the test should use UDP, and UDP is unreliable by definition ... however is curious that most of the failures happen because it is not able to communicate with the first pod I think that is something that should be investigated, I've analyzed some occurences and it works fine for the second pod, however, the UDP losses should be totally random
Hi everyone, yeah so.. collecting some data on this... using a two node kind cluster, locally, with the antrea plugin, it passes using a 4 node AWS cluster with the antrea CNI plugin, it failed using a two node kidn cluster locally with calico, it passes So I guess the best reproducer is to run these tests, for now, in the cloud.... Now that I have a reproducer for this maybe can live debug it some time next wk with one of yall :)
bear in mind that this test is connecting to a nodePort in the node IP, the test is functionally correct, it may be flaky due to the unreliable UDP nature or due to the pod scheduling, but if is failing constantly it has to be environmental.
Second failure after two weeks of green 5k runs:
yep, :/ https://k8s-|UDP /assign
/reopen I'm afraid this is going to be challenging :smile: The difference now is that we are testing NodePort and ClusterIP If both fail, we can blame the test for sure, If only NodePort fails, we need to check the job, if GCE or KIND If only ClusterIP fails, ... :thinking:
Reopened this issue. <detailsIn response to : /reopen I'm afraid this is going to be challenging :smile: Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
We're hitting this really frequently in 5k-node tests. So I started looking a little bit into that.


Question: framework.ExpectNoError(verifyServeHostnameServiceUp(cs, ns, host, podNames2, svc2IP, servicePort)) }) <del> ginkgo.It("should be able to preserve UDP traffic when server pod cycles for a NodePort service", func() { serviceName := "clusterip-test" serverPod1Name := "server-1" serverPod2Name := "server-2" ns := f.Namespace.Name nodeIP, err := e2enode.PickIP(cs) // for later framework.ExpectNoError(err) // Create a NodePort service udpJig := e2eservice.NewTestJig(cs, ns, serviceName) ginkgo.By("creating a UDP service " + serviceName + " with type=NodePort in " + ns) udpService, err := udpJig.CreateUDPService(func(svc *v1.Service) { svc.Spec.Type = v1.ServiceTypeNodePort svc.Spec.Ports = []v1.ServicePort{ {Port: 80, Name: "http", Protocol: v1.ProtocolUDP, TargetPort: intstr.FromInt(80)}, } }) framework.ExpectNoError(err) // Add a backend pod to the service ginkgo.By("creating a backend pod for the service " + serviceName) serverPod1 := newAgnhostPod(serverPod1Name, "netexec", fmt.Sprintf("--udp-port=%d", 80)) serverPod1.Labels = udpJig.Labels _, err = cs.CoreV1().Pods(ns).Create(context.TODO(), serverPod1, metav1.CreateOptions{}) ginkgo.By(fmt.Sprintf("checking NodePort service %s on node with public IP %s", serviceName, nodeIP)) framework.ExpectNoError(err) framework.ExpectNoError(e2epod.WaitTimeoutForPodReadyInNamespace(f.ClientSet, serverPod1.Name, f.Namespace.Name, framework.PodStartTimeout)) // Waiting for service to expose endpoint. err = validateEndpointsPorts(cs, ns, serviceName, portsByPodName{serverPod1Name: {80}}) framework.ExpectNoError(err, "failed to validate endpoints for service %s in namespace: %s", serviceName, ns) // Check that the pod reveives the traffic ginkgo.By("Sending UDP traffic to NodePort service " + serviceName + " on node with publicIP " + nodeIP) errorChannel := make(chan error) signal := make(chan struct{}, 1) go continuousEcho(nodeIP, int(udpService.Spec.Ports[0].NodePort), 3*time.Second, 20, signal, errorChannel) // Create a second pod ginkgo.By("creating a second pod for the service " + serviceName) serverPod2 := newAgnhostPod(serverPod2Name, "netexec", fmt.Sprintf("--udp-port=%d", 80)) serverPod2.Labels = udpJig.Labels _, err = cs.CoreV1().Pods(ns).Create(context.TODO(), serverPod2, metav1.CreateOptions{}) framework.ExpectNoError(err) framework.ExpectNoError(e2epod.WaitTimeoutForPodReadyInNamespace(f.ClientSet, serverPod2.Name, f.Namespace.Name, framework.PodStartTimeout)) // and delete the first pod framework.Logf("Cleaning up %s pod", serverPod1Name) err = cs.CoreV1().Pods(ns).Delete(context.TODO(), serverPod1Name, metav1.DeleteOptions{}) framework.ExpectNoError(err, "failed to delete pod: %s on node", serverPod1Name) // Check that the second pod keeps receiving traffic ginkgo.By("Sending UDP traffic to NodePort service " + serviceName + " on node with publicIP " + nodeIP) signal <- struct{}{} // Check that there are no errors err = <-errorChannel framework.ExpectNoError(err, "pod communication failed") }) </del> /* Release : v1.16 Testname: Service, NodePort Service

I don't have full understanding of what exactly is happening, but I already see a couple things that may be causing issues: when creating the first pod, you wait for corresponding endpoint for that being created; but that doesn't mean that kube-proxy was actually programmed correctly at that time (in general it isn't); given iptables programming is rate-limitted, it may easily happen that actually programming happens 10s+ later That explains some delays, but it doesn't explain e.g. this run: where iptables were programmed at but the test didn't respond until If someone more familiar with networking stuff could look into it, it would be great. - can you help triaging ^^ for deletions it's even worse, because you don't even wait for endpoints to be there (i.e. Endpoints can still contain the old pod when the test start pinging to look for the second one) So I claim that timeouts are visibly to low in this test, but I suspect it's not everything. I will send out the first PR that tries to improve it in couple minutes.
thanks wojtek, have been slammed but Ill try to collect more data on it this wknd as well.
It seems that my PR didn't really help much. We should dig deeper, but I would need someone with more networking expertise to look a bit into
Actually - it seems only failures from one group are on non . So maybe that's actually related. The non-kind related ones look like:
OK - for this one, I actually know what is happening - I will send out a PR with a fix for this too.
for the KIND jobs already increased the resource requests on some of the jobs, there are issues on KIND jobs with kube-proxy not able to take the iptables lock and not being able to create the iptables rules for the services until the next resync
is to fix the flakes I mentioned above; all other flakes seem to be kind-oriented to me (which I'm not super familiar with).
We did not adjust the CPU on all jobs yet, pending how to manage that on prow ... I do see just a few other flakes at the bottom of the triage dashboard that look like timeouts instead.
Actually - almost all failures are effectively timeout. The top one that is is actually also a timeout. Didn't look into kind failures, but this may easily be related to iptables issues.


Question: framework.ExpectNoError(verifyServeHostnameServiceUp(cs, ns, host, podNames2, svc2IP, servicePort)) }) <del> ginkgo.It("should be able to preserve UDP traffic when server pod cycles for a NodePort service", func() { serviceName := "clusterip-test" serverPod1Name := "server-1" serverPod2Name := "server-2" ns := f.Namespace.Name nodeIP, err := e2enode.PickIP(cs) // for later framework.ExpectNoError(err) // Create a NodePort service udpJig := e2eservice.NewTestJig(cs, ns, serviceName) ginkgo.By("creating a UDP service " + serviceName + " with type=NodePort in " + ns) udpService, err := udpJig.CreateUDPService(func(svc *v1.Service) { svc.Spec.Type = v1.ServiceTypeNodePort svc.Spec.Ports = []v1.ServicePort{ {Port: 80, Name: "http", Protocol: v1.ProtocolUDP, TargetPort: intstr.FromInt(80)}, } }) framework.ExpectNoError(err) // Add a backend pod to the service ginkgo.By("creating a backend pod for the service " + serviceName) serverPod1 := newAgnhostPod(serverPod1Name, "netexec", fmt.Sprintf("--udp-port=%d", 80)) serverPod1.Labels = udpJig.Labels _, err = cs.CoreV1().Pods(ns).Create(context.TODO(), serverPod1, metav1.CreateOptions{}) ginkgo.By(fmt.Sprintf("checking NodePort service %s on node with public IP %s", serviceName, nodeIP)) framework.ExpectNoError(err) framework.ExpectNoError(e2epod.WaitTimeoutForPodReadyInNamespace(f.ClientSet, serverPod1.Name, f.Namespace.Name, framework.PodStartTimeout)) // Waiting for service to expose endpoint. err = validateEndpointsPorts(cs, ns, serviceName, portsByPodName{serverPod1Name: {80}}) framework.ExpectNoError(err, "failed to validate endpoints for service %s in namespace: %s", serviceName, ns) // Check that the pod reveives the traffic ginkgo.By("Sending UDP traffic to NodePort service " + serviceName + " on node with publicIP " + nodeIP) errorChannel := make(chan error) signal := make(chan struct{}, 1) go continuousEcho(nodeIP, int(udpService.Spec.Ports[0].NodePort), 3*time.Second, 20, signal, errorChannel) // Create a second pod ginkgo.By("creating a second pod for the service " + serviceName) serverPod2 := newAgnhostPod(serverPod2Name, "netexec", fmt.Sprintf("--udp-port=%d", 80)) serverPod2.Labels = udpJig.Labels _, err = cs.CoreV1().Pods(ns).Create(context.TODO(), serverPod2, metav1.CreateOptions{}) framework.ExpectNoError(err) framework.ExpectNoError(e2epod.WaitTimeoutForPodReadyInNamespace(f.ClientSet, serverPod2.Name, f.Namespace.Name, framework.PodStartTimeout)) // and delete the first pod framework.Logf("Cleaning up %s pod", serverPod1Name) err = cs.CoreV1().Pods(ns).Delete(context.TODO(), serverPod1Name, metav1.DeleteOptions{}) framework.ExpectNoError(err, "failed to delete pod: %s on node", serverPod1Name) // Check that the second pod keeps receiving traffic ginkgo.By("Sending UDP traffic to NodePort service " + serviceName + " on node with publicIP " + nodeIP) signal <- struct{}{} // Check that there are no errors err = <-errorChannel framework.ExpectNoError(err, "pod communication failed") }) </del> /* Release : v1.16 Testname: Service, NodePort Service

After my second deflaking PR, we seem to be in the stage where: tests are failing on dozens of different things (probably iptables-issues related) tests are failing only on NodePort tests on "Failed to connecto to backend 1" which it would be great if someone else could look into (kube-proxy seemed to be programmed correctly based on logs)
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
This has failed again - Linking to specific part of log where timeout occurred .... we see this
So reviewing the above logs note how in STEP: checking client pod connected to the backend 1 on Node IP 10.138.0.5 28 attempts are made all in the 25th second of the 2 minute at 11am Should this Step back off and wait a few seconds in between tries? So looking at this code we see the following
From reading ... the intention expressed in the test code looks good to me. It is not clear to me why there is a mismatch between the 28 tries in the POD Client Logs and the expressed intention running the logContainsFn(podBackend1) every 5 seconds still digging ....
we are trying to catch one failure in this PR that has additional debugging, it dumps the conntrack table of the nodes on a failure
for those following the issue, we've extra debugging to the test so it dumps the whole conntrack table once it finish. We should look that there is NO stale entries, connections from the pod client to the deleted endpoint) and there is conntrack entries to the new endpoint from the pod client.
- we have runs with the conntrack dump now, e.g. this one: would you be able to take a look into it?
not yet :/ , but I think tomorrow can have some time
cc
For anyone reproducing this issue btw, you need a multinode k8s cluster (it isnt doable on kind i guess bc less likeliehood of network failures when running local) -
Curious about two potential hypotheses to fix this, have we looked into either of these? 1) can we confirm that is turned on/off? 2) is it possible that scaling the number of coredns replicas could help ?


Question: framework.ExpectNoError(verifyServeHostnameServiceUp(cs, ns, host, podNames2, svc2IP, servicePort)) }) <del> ginkgo.It("should be able to preserve UDP traffic when server pod cycles for a NodePort service", func() { serviceName := "clusterip-test" serverPod1Name := "server-1" serverPod2Name := "server-2" ns := f.Namespace.Name nodeIP, err := e2enode.PickIP(cs) // for later framework.ExpectNoError(err) // Create a NodePort service udpJig := e2eservice.NewTestJig(cs, ns, serviceName) ginkgo.By("creating a UDP service " + serviceName + " with type=NodePort in " + ns) udpService, err := udpJig.CreateUDPService(func(svc *v1.Service) { svc.Spec.Type = v1.ServiceTypeNodePort svc.Spec.Ports = []v1.ServicePort{ {Port: 80, Name: "http", Protocol: v1.ProtocolUDP, TargetPort: intstr.FromInt(80)}, } }) framework.ExpectNoError(err) // Add a backend pod to the service ginkgo.By("creating a backend pod for the service " + serviceName) serverPod1 := newAgnhostPod(serverPod1Name, "netexec", fmt.Sprintf("--udp-port=%d", 80)) serverPod1.Labels = udpJig.Labels _, err = cs.CoreV1().Pods(ns).Create(context.TODO(), serverPod1, metav1.CreateOptions{}) ginkgo.By(fmt.Sprintf("checking NodePort service %s on node with public IP %s", serviceName, nodeIP)) framework.ExpectNoError(err) framework.ExpectNoError(e2epod.WaitTimeoutForPodReadyInNamespace(f.ClientSet, serverPod1.Name, f.Namespace.Name, framework.PodStartTimeout)) // Waiting for service to expose endpoint. err = validateEndpointsPorts(cs, ns, serviceName, portsByPodName{serverPod1Name: {80}}) framework.ExpectNoError(err, "failed to validate endpoints for service %s in namespace: %s", serviceName, ns) // Check that the pod reveives the traffic ginkgo.By("Sending UDP traffic to NodePort service " + serviceName + " on node with publicIP " + nodeIP) errorChannel := make(chan error) signal := make(chan struct{}, 1) go continuousEcho(nodeIP, int(udpService.Spec.Ports[0].NodePort), 3*time.Second, 20, signal, errorChannel) // Create a second pod ginkgo.By("creating a second pod for the service " + serviceName) serverPod2 := newAgnhostPod(serverPod2Name, "netexec", fmt.Sprintf("--udp-port=%d", 80)) serverPod2.Labels = udpJig.Labels _, err = cs.CoreV1().Pods(ns).Create(context.TODO(), serverPod2, metav1.CreateOptions{}) framework.ExpectNoError(err) framework.ExpectNoError(e2epod.WaitTimeoutForPodReadyInNamespace(f.ClientSet, serverPod2.Name, f.Namespace.Name, framework.PodStartTimeout)) // and delete the first pod framework.Logf("Cleaning up %s pod", serverPod1Name) err = cs.CoreV1().Pods(ns).Delete(context.TODO(), serverPod1Name, metav1.DeleteOptions{}) framework.ExpectNoError(err, "failed to delete pod: %s on node", serverPod1Name) // Check that the second pod keeps receiving traffic ginkgo.By("Sending UDP traffic to NodePort service " + serviceName + " on node with publicIP " + nodeIP) signal <- struct{}{} // Check that there are no errors err = <-errorChannel framework.ExpectNoError(err, "pod communication failed") }) </del> /* Release : v1.16 Testname: Service, NodePort Service

3) Would simulating this in kind by overloading coredns be a possibility as a way to make a more reproducible bug model?
regarding 1) kubeadm does not ship nodeLocalDNSCache I don't think GCE kube-up does by default either
ok, so now im curious about (2) - i.e. what if we just bumped coredns replica count?
interesting, :thinking: why do you think is a DNS thing? the netcat command that polls the services use the IP addresses not the hostname. I think that it has to be related to some iptables/conntrack race or similar,at least in KIND jobs we know that we have an iptables problem that affects network tests, that has to be fixed before this because it affect this test You can instrument the test locally with this and dump the state of iptables and conntrack on failure if you are able to reproduce it
i figured DNS bc i believe its known that conntrack and DNS at scale can get very messy, right ?
one other curious thing about this test is that we only create 2 pods in the recycling event for the svc. that likely means that any real error is hidden by chance..... If we created a 3rd pod, then we might potentially double the failure rate thus making the flakyness go away and expose the real underlying issue .
There's an invocation of dumpConntrack(cs) here [sig-network] Conntrack should be able to preserve UDP traffic when server pod cycles for a ClusterIP service
is updating the logging to run in an infinite loop ..
This is improving in KIND with the new changes we rolled out yesterday, let's see how it evolves this week
- which changes ? Are you saying this is no longer reproducible on Kind?
there was a lot of noise in the CI signal, especially in KIND jobs due to the CNI portmap bug we finally solved yesterday now we should have a better signal and we can go to the bottom because probably the failure will be "legit" and not a consequence, so this comment should apply
Hi ! I ran this a few times on kind and it passed, but I see , so I'm going to try the loop approach you suggested. Here's what I'm using to test:
quick note, <- potential motivation to possibly have runnable on some subset of tests in upstream eventually, i.e.