Question: } } <ins> func TestMul(t *testing.T) { tests := []struct { a        Quantity b        int64 expected Quantity ok       bool }{ {decQuantity(10, 0, DecimalSI), 10, decQuantity(100, 0, DecimalSI), true}, {decQuantity(10, 0, DecimalSI), 1, decQuantity(10, 0, DecimalSI), true}, {decQuantity(10, 0, BinarySI), 1, decQuantity(10, 0, BinarySI), true}, {Quantity{Format: DecimalSI}, 50, decQuantity(0, 0, DecimalSI), true}, {decQuantity(50, 0, DecimalSI), 0, decQuantity(0, 0, DecimalSI), true}, {Quantity{Format: DecimalSI}, 0, decQuantity(0, 0, DecimalSI), true}, {decQuantity(10, 0, DecimalSI), -10, decQuantity(-100, 0, DecimalSI), true}, {decQuantity(-10, 0, DecimalSI), 1, decQuantity(-10, 0, DecimalSI), true}, {decQuantity(10, 0, BinarySI), -1, decQuantity(-10, 0, BinarySI), true}, {decQuantity(-50, 0, DecimalSI), 0, decQuantity(0, 0, DecimalSI), true}, {decQuantity(-50, 0, DecimalSI), -50, decQuantity(2500, 0, DecimalSI), true}, {Quantity{Format: DecimalSI}, -50, decQuantity(0, 0, DecimalSI), true}, {decQuantity(mostPositive, 0, DecimalSI), 0, decQuantity(0, 1, DecimalSI), true}, {decQuantity(mostPositive, 0, DecimalSI), 1, decQuantity(mostPositive, 0, DecimalSI), true}, {decQuantity(mostPositive, 0, DecimalSI), -1, decQuantity(-mostPositive, 0, DecimalSI), true}, {decQuantity(mostPositive/2, 0, DecimalSI), 2, decQuantity((mostPositive/2)*2, 0, DecimalSI), true}, {decQuantity(mostPositive/-2, 0, DecimalSI), -2, decQuantity((mostPositive/2)*2, 0, DecimalSI), true}, {decQuantity(mostPositive, 0, DecimalSI), 2, bigDecQuantity(big.NewInt(0).Mul(bigMostPositive, big.NewInt(2)), 0, DecimalSI), false}, {decQuantity(mostPositive, 0, DecimalSI), 10, decQuantity(mostPositive, 1, DecimalSI), false}, {decQuantity(mostPositive, 0, DecimalSI), -10, decQuantity(-mostPositive, 1, DecimalSI), false}, {decQuantity(mostNegative, 0, DecimalSI), 0, decQuantity(0, 1, DecimalSI), true}, {decQuantity(mostNegative, 0, DecimalSI), 1, decQuantity(mostNegative, 0, DecimalSI), true}, {decQuantity(mostNegative, 0, DecimalSI), -1, bigDecQuantity(big.NewInt(0).Add(bigMostPositive, big.NewInt(1)), 0, DecimalSI), false}, {decQuantity(mostNegative/2, 0, DecimalSI), 2, decQuantity(mostNegative, 0, DecimalSI), true}, {decQuantity(mostNegative/-2, 0, DecimalSI), -2, decQuantity(mostNegative, 0, DecimalSI), true}, {decQuantity(mostNegative, 0, DecimalSI), 2, bigDecQuantity(big.NewInt(0).Mul(bigMostNegative, big.NewInt(2)), 0, DecimalSI), false}, {decQuantity(mostNegative, 0, DecimalSI), 10, decQuantity(mostNegative, 1, DecimalSI), false}, {decQuantity(mostNegative, 0, DecimalSI), -10, bigDecQuantity(big.NewInt(0).Add(bigMostPositive, big.NewInt(1)), 1, DecimalSI), false}, } for i, test := range tests { if ok := test.a.Mul(test.b); test.ok != ok { t.Errorf("[%d] Expected ok: %t, got ok: %t", i, test.ok, ok) } if test.a.Cmp(test.expected) != 0 { t.Errorf("[%d] Expected %q, got %q", i, test.expected.AsDec().String(), test.a.AsDec().String()) } } } </ins> func TestAddSubRoundTrip(t *testing.T) { for k := -10; k <= 10; k++ { q := Quantity{Format: DecimalSI}

I'd like to add a multiplication method for the resource.Quantity. Currently, the resource package provides helper methods to calculate Quantity, such as and : Batch workloads often replicate the same PodTemplate, and controllers calculate the resources needed for all containers. In that case, It is inconvenient to call the function as many times as podTemplate, so I would like to create a helper function for multiplication.
/cc /sig api-machinary /wg batch
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /cc /sig api-machinary /wg batch </details
/sig api-machinery /wg batch
Do you have an example of usage for these utilities? Just curious?
/triage accepted
Sorry for the late response. Taking k-sigs/kueue as an example, I assumed the below: It helps the controllers for batch workloads such as k-sigs/kueue, k-sigs/jobset, kubeflow/mpi-operator, and kubeflow/training-operator. I'm thinking of implementing this function once the code freeze term is finished.
/assign
Hi, may i take this up?
I took this one, and working on this now.


Question: if err != nil { t.Errorf("unexpected error: %v", err) } <del> if !reflect.DeepEqual(fakeDocker.Removed, test.expectedRemoved) { t.Errorf("expected: %v, got: %v", test.expectedRemoved, fakeDocker.Removed) } </del> <ins> verifyStringArrayEqualsAnyOrder(t, test.expectedRemoved, fakeDocker.Removed) </ins> } }

Successfully ran containers with "onFailure" restartPolicy are restarting after purging oldest docker containers. Kubelet purges/removes oldest containers periodically from docker. After purging, we loose the exit code and history of previously ran containers and restart containers as if they were never run.
Thanks & I can help fix this. Let me know. Thanks
Thanks for reporting Just to make sure I understand the issue: You ran some containers with an "onFailure" restartPolicy. When the Kubelet purged old containers the container was restarted (unexpectedly).
Ah, after reading again I think the issue is that after the containers are purged, when the container exits successfully it get restarted although it shouldn't. Is that correct?
After reading through the code I found a bug in the implementation that is most likely causing this error. Should have a PR soon.
Yes. Correct. After initial container run, it should never be run again, since it already ran to completion. Will wait for your PR.
PR is out. This should fix the issue. Feel free to test if you can :)


Question: fi mkfs -t ext4 ${block_devices[0]} <del> echo "${block_devices[0]}  /mnt/ephemeral  ext4     noatime  0 0" >> /etc/fstab </del> <ins> echo "${block_devices[0]}  /mnt/ephemeral  ext4     noatime,nofail  0 0" >> /etc/fstab </ins> mkdir -p /mnt/ephemeral mount /mnt/ephemeral

On AWS we (will) now run our bootstrap script on every boot. The function to initialize ephemeral volumes doesn't tolerate being re-run, and we probably shouldn't re-run it anyway for speed. This does preserve the contents, which may be a good thing to save user's data (it doesn't belong on ephemeral drives, but...). On the other hand, if the ephemeral drives are corrupted, we lose the opportunity to reformat them and recover. Arguably, if the ephemeral storage is corrupted the right thing to do is to terminate the instance entirely.


Question: // matches NodeInfo.node and the pod is not found in the pods list. Otherwise, // returns true. func (n *NodeInfo) Filter(pod *v1.Pod) bool { <del> pFullName := util.GetPodFullName(pod) </del> if pod.Spec.NodeName != n.node.Name { return true } for _, p := range n.pods { <del> if util.GetPodFullName(p) == pFullName { </del> <ins> if p.Name == pod.Name && p.Namespace == pod.Namespace { </ins> return true } }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --It is not a secret that . In my opinion the main reason for the slowness is the flexibility of the inter-pod affinity/anti-affinity rules. However, in most practical scenarios users do not need all the flexibility. For example, people often do not use arbitrary labels in topology key of the affinity rules. They often use node, zone, or region as the topology. Node in particular is used often. That's why I am proposing a fast path for the case that topology key is Node. Today, the affinity/anti-affinity predicate does not have any distinction for topology keys and in the cluster regardless. The logic should recognize "" and scan only pods on the same node for this topology key. Is this a BUG REPORT or FEATURE REQUEST?: FEATURE REQUEST /kind feature /sig scheduling cc/
ref/
/assign
GitHub didn't allow me to assign the following users: misterikkit. Note that only can be assigned. <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc
We found some other simple optimizations that had a significant impact on the affinity predicate. I will upload PRs for those as well.
Reopening this as the performance of anti-affinity still doesn't seem to be satisfactory enough on large clusters (i.e 5k nodes) - and it's causing them to fail to startup. Ref: issue I can help with evaluating any performance improvements if needed.
The change sent improved performance of the predicate, but apparently it is not enough to make our large scale scalability tests happy.
This issue is not about general performance improvment of affinity/anti-affinity. It was to implement a particular improvment which is already implemented. There are other performance improvement issues. I am going to close this one.
Sounds good. Thanks.
How come with kops 1.10.0 I still see all kube-dns pods on the same node because the kube-dns deployment doesn't specify any anti-affinity rule?
I think you are correct. you can double-check with the option to . I have a v1.9 cluster in front of me, and kube-dns has no affinity rules. (I didn't use kops though)


Question: // TODO: the name/domain name pair should ideally be checked for naming convention // as done in kube-dns federation flags check. // https://github.com/kubernetes/dns/blob/master/pkg/dns/federation/federation.go <del> util.FedDomainMapKey: fmt.Sprintf("%s=%s", name, dnsZoneName), </del> <ins> // TODO v2: Until kube-dns can handle trailing periods we strip them all. //          See https://github.com/kubernetes/dns/issues/67 util.FedDomainMapKey: fmt.Sprintf("%s=%s", name, strings.TrimRight(dnsZoneName, ".")), </ins> }, }, Spec: extensions.DeploymentSpec{

Looking at our latest e2e CI , they appear to be due to the fact that the "federations value" for kube-dns install a domain with a trailing dot. The trailing period does not appear when I bring up the cluster manually with I suspect that somewhere we are reconstructing a FEDERATIONSDOMAINMAP when it is unset and a trailing period gets there.
is this blocking for 1.6?
Fixed by


Question: if isHTEnabled() { cpuList = cpuset.MustParse(getCPUSiblingList(0)).ToSlice() cpu1 = cpuList[1] <ins> } else if isMultiNUMA() { cpuList = cpuset.MustParse(getCoreSiblingList(0)).ToSlice() cpu1 = cpuList[1] </ins> } expAllowedCPUsListRegex = fmt.Sprintf("^%dn$", cpu1) err = f.PodClient().MatchContainerOutput(pod1.Name, pod1.Spec.Containers[0].Name, expAllowedCPUsListRegex)

Which jobs are failing: N/A - AFAICT CI jobs works fine Which test(s) are failing: Since when has it been failing: no historical record on CI Testgrid link: N/A Reason for failure: I believe the tests make too many assumptions around the hardware, and the code is actually OK. to be confirmed. Anything else we need to know:
/sig node /sig testing
also affects the recently topology manager e2e tests. My PR uses a different approach to check resource alignment so is not affected. /cc
I'll work on a PR to fix this issue ASAP
Thanks for opening this issue. We knew about this, just had not opened an issue to track it, so thanks for doing that! Deepthi has been looing at CPU Manager tests, not sure if she has some PR in progress or not. I'll make sure to forward this to her (can find her user id here ....) in case she has something in progress


Question: <ins> // +build integration,!no-etcd /* Copyright 2015 Google Inc. All rights reserved. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package integration import ( "bufio" "fmt" "net/http" "net/http/httptest" "testing" "github.com/GoogleCloudPlatform/kubernetes/pkg/api" "github.com/GoogleCloudPlatform/kubernetes/pkg/api/testapi" "github.com/GoogleCloudPlatform/kubernetes/pkg/client" "github.com/GoogleCloudPlatform/kubernetes/pkg/labels" "github.com/golang/glog" "github.com/golang/protobuf/proto" prometheuspb "github.com/prometheus/client_model/go" ) const scrapeRequestHeader = "application/vnd.google.protobuf;proto=io.prometheus.client.MetricFamily;encoding=compact-text" func init() { requireEtcd() } func scrapeMetrics(s *httptest.Server) ([]*prometheuspb.MetricFamily, error) { req, err := http.NewRequest("GET", s.URL+"/metrics", nil) if err != nil { return nil, fmt.Errorf("Unable to create http request: %v", err) } // Ask the prometheus exporter for its text protocol buffer format, since it's // much easier to parse than its plain-text format. Don't use the serialized // proto representation since it uses a non-standard varint delimiter between // metric families. req.Header.Add("Accept", scrapeRequestHeader) client := &http.Client{} resp, err := client.Do(req) if err != nil { return nil, fmt.Errorf("Unable to contact metrics endpoint of master: %v", err) } defer resp.Body.Close() if resp.StatusCode != 200 { return nil, fmt.Errorf("Non-200 response trying to scrape metrics from master: %v", resp) } // Each line in the response body should contain all the data for a single metric. var metrics []*prometheuspb.MetricFamily scanner := bufio.NewScanner(resp.Body) for scanner.Scan() { var metric prometheuspb.MetricFamily if err := proto.UnmarshalText(scanner.Text(), &metric); err != nil { return nil, fmt.Errorf("Failed to unmarshal line of metrics response: %v", err) } glog.Infof("Got metric %q", metric.GetName()) metrics = append(metrics, &metric) } return metrics, nil } func checkForExpectedMetrics(t *testing.T, metrics []*prometheuspb.MetricFamily, expectedMetrics []string) { foundMetrics := make(map[string]bool) for _, metric := range metrics { foundMetrics[metric.GetName()] = true } for _, expected := range expectedMetrics { if _, found := foundMetrics[expected]; !found { t.Errorf("Master metrics did not include expected metric %q", expected) } } } func TestMasterProcessMetrics(t *testing.T) { _, s := runAMaster(t) defer s.Close() metrics, err := scrapeMetrics(s) if err != nil { t.Fatal(err) } checkForExpectedMetrics(t, metrics, []string{ "process_start_time_seconds", "process_cpu_seconds_total", "process_goroutines", "process_open_fds", "process_resident_memory_bytes", }) } func TestApiserverMetrics(t *testing.T) { _, s := runAMaster(t) defer s.Close() // Make a request to the apiserver to ensure there's at least one data point // for the metrics we're expecting -- otherwise, they won't be exported. client := client.NewOrDie(&client.Config{Host: s.URL, Version: testapi.Version()}) if _, err := client.Pods(api.NamespaceDefault).List(labels.Everything()); err != nil { t.Fatalf("unexpected error getting pods: %v", err) } metrics, err := scrapeMetrics(s) if err != nil { t.Fatal(err) } checkForExpectedMetrics(t, metrics, []string{ "apiserver_request_count", "apiserver_request_latencies", }) } </ins>

In 0.13.2, the apiserver has automatically exposed metrics about the process, like start time, memory usage, processor usage, etc. This stopped working sometime between then and when 0.14.0 was cut. They're still correctly exported from the kubelet, interestingly enough. An e2e or integration test is also needed to catch this if it happens again.


Question: } } } <ins> func TestInlineReadOnly(t *testing.T) { g := NewGCEPersistentDiskCSITranslator() pv, err := g.TranslateInTreeInlineVolumeToCSI(&v1.Volume{ VolumeSource: v1.VolumeSource{ GCEPersistentDisk: &v1.GCEPersistentDiskVolumeSource{ PDName:   "foo", ReadOnly: true, }, }, }) if err != nil { t.Fatalf("Failed to translate in tree inline volume to CSI: %v", err) } if pv == nil || pv.Spec.PersistentVolumeSource.CSI == nil { t.Fatal("PV or volume source unexpectedly nil") } if !pv.Spec.PersistentVolumeSource.CSI.ReadOnly { t.Error("PV readonly value not true") } ams := pv.Spec.AccessModes if len(ams) != 1 { t.Errorf("got am %v, expected length of 1", ams) } if ams[0] != v1.ReadOnlyMany { t.Errorf("got am %v, expected access mode of ReadOnlyMany", ams[0]) } } </ins>

When running CSI migration tests focussing on multi-attach scearnios with GCE PD inline volumes that are read-only, the following error pops up in the logs and the second attachment takes 6 mins: For read-only inline volumes, the above warning/error around is unexpected with CSI migration enabled and needs to be root-caused/resolved. What happened: CSI migration tests for read-only inline GCE PD volumes occasionally fails with the above messages in the logs when executing tests like What you expected to happen: The tests and multi-attach scenarios to execute smoothly with no warnings in the logs. How to reproduce it (as minimally and precisely as possible): Execute with focus set to and run a few times. Anything else we need to know?: This has been observed with CSI migration enabled and the tests exercising read-only inline volume scenarios. Environment: Kubernetes version (use ): v1.16 Cloud provider or hardware configuration: GCE OS (e.g: ): Kernel (e.g. ): Install tools: kube- Network plugin and version (if this is a network-related bug): Others:
/cc
/sig storage
Looks like hard coding access modes to ReadWriteOnce during translation for GCE PD at is causing isMultiAttachForbidden to return leading down to We need to set the access mode to ReadOnlyMany for in-tree inline volumes like in GCE PD (as well as others like EBS that support multi-attach) depending on instead of always setting it to ReadWriteOnce.
I can pick up this fix /assign


Question: <ins> /* Copyright 2014 Google Inc. All rights reserved. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package yaml import ( "bufio" "bytes" "encoding/json" "fmt" "io" "testing" ) func TestSplitYAMLDocument(t *testing.T) { testCases := []struct { input  string atEOF  bool expect string adv    int }{ {"foo", true, "foo", 3}, {"fo", false, "", 0}, {"---", true, "---", 3}, {"---n", true, "---n", 4}, {"---n", false, "", 0}, {"n---n", false, "", 5}, {"n---n", true, "", 5}, {"abcn---ndef", true, "abc", 8}, {"def", true, "def", 3}, {"", true, "", 0}, } for i, testCase := range testCases { adv, token, err := splitYAMLDocument([]byte(testCase.input), testCase.atEOF) if err != nil { t.Errorf("%d: unexpected error: %v", i, err) continue } if adv != testCase.adv { t.Errorf("%d: advance did not match: %d %d", i, testCase.adv, adv) } if testCase.expect != string(token) { t.Errorf("%d: token did not match: %q %q", i, testCase.expect, string(token)) } } } func TestScanYAML(t *testing.T) { s := bufio.NewScanner(bytes.NewReader([]byte(`--- stuff: 1 --- `))) s.Split(splitYAMLDocument) if !s.Scan() { t.Fatalf("should have been able to scan") } t.Logf("scan: %s", s.Text()) if !s.Scan() { t.Fatalf("should have been able to scan") } t.Logf("scan: %s", s.Text()) if s.Scan() { t.Fatalf("scan should have been done") } if s.Err() != nil { t.Fatalf("err should have been nil: %v", s.Err()) } } func TestDecodeYAML(t *testing.T) { s := NewYAMLToJSONDecoder(bytes.NewReader([]byte(`--- stuff: 1 --- `))) obj := generic{} if err := s.Decode(&obj); err != nil { t.Fatalf("unexpected error: %v", err) } if fmt.Sprintf("%#v", obj) != `yaml.generic{"stuff":1}` { t.Errorf("unexpected object: %#v", obj) } obj = generic{} if err := s.Decode(&obj); err != nil { t.Fatalf("unexpected error: %v", err) } if len(obj) != 0 { t.Fatalf("unexpected object: %#v", obj) } obj = generic{} if err := s.Decode(&obj); err != io.EOF { t.Fatalf("unexpected error: %v", err) } } type generic map[string]interface{} func TestYAMLOrJSONDecoder(t *testing.T) { testCases := []struct { input  string buffer int isJSON bool err    bool out    []generic }{ {` {"1":2}{"3":4}`, 2, true, false, []generic{ {"1": 2}, {"3": 4}, }}, {" n{}", 3, true, false, []generic{ {}, }}, {" na: b", 2, false, false, []generic{ {"a": "b"}, }}, {` n{"a": "b"}`, 2, false, true, []generic{ {"a": "b"}, }}, {`   {"a":"b"}`, 100, true, false, []generic{ {"a": "b"}, }}, {"", 1, false, false, []generic{}}, {"foo: barn---nbaz: biz", 100, false, false, []generic{ {"foo": "bar"}, {"baz": "biz"}, }}, {"foo: barn---n", 100, false, false, []generic{ {"foo": "bar"}, }}, {"foo: barn---", 100, false, false, []generic{ {"foo": "bar"}, }}, {"foo: barn--", 100, false, true, []generic{ {"foo": "bar"}, }}, {"foo: barn-", 100, false, true, []generic{ {"foo": "bar"}, }}, {"foo: barn", 100, false, false, []generic{ {"foo": "bar"}, }}, } for i, testCase := range testCases { decoder := NewYAMLOrJSONDecoder(bytes.NewReader([]byte(testCase.input)), testCase.buffer) objs := []generic{} var err error for { out := make(generic) err = decoder.Decode(&out) if err != nil { break } objs = append(objs, out) } if err != io.EOF { switch { case testCase.err && err == nil: t.Errorf("%d: unexpected non-error", i) continue case !testCase.err && err != nil: t.Errorf("%d: unexpected error: %v", i, err) continue case err != nil: continue } } switch decoder.decoder.(type) { case *YAMLToJSONDecoder: if testCase.isJSON { t.Errorf("%d: expected JSON decoder, got YAML", i) } case *json.Decoder: if !testCase.isJSON { t.Errorf("%d: expected YAML decoder, got JSON", i) } } if fmt.Sprintf("%#v", testCase.out) != fmt.Sprintf("%#v", objs) { t.Errorf("%d: objects were not equal: n%#vn%#v", i, testCase.out, objs) } } } </ins>

If I stuff into a file and use that after -f, it works fine. cc
What's on stdin?
Bug, I'm only accepting json because I naively reused watch partitioner and we don't have test cases. Will fix tonight.
If that's the problem, why did it work when I gave it that file not over stdin?
Different code path. Stdin supports streaming json a la watch (multiple json objects in a stream), so you can do Unfortunately I missed yaml input and didn't use a segmenting yaml decoder (returning a stream of yaml objects).
Ah! Thanks for explanation. I think it would be good for us to ask for e2e runs before merging kubectl changes, since there's not quite enough coverage in test- yet.
Yeah, I'm slacking on that for these changes.
Fix incoming.
Fix provided. More tests to test-cmd... :)


Question: actualEvent := <-createEvent validateEvent(strconv.Itoa(index), actualEvent, item.expect, t) } <del> logWatcher1.Stop() logWatcher2.Stop() </del> <ins> logWatcher.Stop() </ins> } sinkWatcher.Stop() }

I wasn't able to find automatically create issue for this, but I may not know what to search for. Is our bot creating issues for unit test failures? - this failure looks to me as something in client testing libraries.
If it's exposed in JUnit output, then the bot should be making issues for it.
It's not exposed in junit due to :(
Lots of failures in , probably all this bug.
Yeah this seems to be happening a lot. Raising priority.
I thought was reverted but it's not. I'll revert it.
golang 1.6 doesn't fix the race:
Sent .


Question: }, }, ) <ins> s.endpointWorkerMap = make(map[string]bool) s.serviceWorkerMap = make(map[string]bool) s.endpointWorkerDoneChan = make(chan string, maxNoOfClusters) s.serviceWorkerDoneChan = make(chan string, maxNoOfClusters) </ins> return s }

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): No Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Environment: Cloud provider or hardware configuration: GCP OS (e.g. from /etc/os-release):Kernel (e.g. ):Install tools: Others: What happened: Memory usage of federation-control-manager keeps on increasing to very large size overtime. What you expected to happen: Memory usage is stable How to reproduce it (as minimally and precisely as possible): Register one or more cluster to federation control plane and watch the top output for the federation-control-manager Also can check the pprof, here is the one such output after federation-control-manager is run for an hour. We can see there are around goroutines created in an hour which are leaked and would consume memory. Anything else do we need to know: I do have a fix for the issue and would be happy to submit it, if this issue is accepted.
plz check the issue
feel free to just submit a pull request, we don't really have a concept of an issue being "accepted" :)
This looks like a legitimate bug. could you please clarify what version of the federated service controller you are running? The image path and version of the container, or a git hash of the source should be sufficient.
i am taking the latest code from master branch.


Question: } func (stats *seatDemandStats) update(obs fq.IntegratorResults) { <ins> stats.highWatermark = obs.Max if obs.Duration <= 0 { return } if math.IsNaN(obs.Deviation) { obs.Deviation = 0 } </ins> stats.avg = obs.Average stats.stdDev = obs.Deviation <del> stats.highWatermark = obs.Max </del> envelope := obs.Average + obs.Deviation stats.smoothed = math.Max(envelope, seatDemandSmoothingCoefficient*stats.smoothed+(1-seatDemandSmoothingCoefficient)*envelope) }

hi, we have 3 master nodes , one of nodes errors output this err kubectl -n kube-system logs kube-apiserver-loshu-k8s-master02 k8s version lb proxy haproxy+keeaplive on-premise ubuntu 22.04 kubeadm containerd calico
/sig api-machinery for triage
/cc
Yes, looks like a real bug, an oversight in logic. Will fix.
Will this fix be backported to v1.26?
/cc hello, I want to know what the scenario of this problem is, what if I reproduce it? thanks
I have k8s 1.26 and recently got same error.
/triage accepted
I have observed the same error on v1.27.6
I have observed the same error on v1.28.3
I have observed the same error on v1.28.4
Try to update to the latest patch version.
it seems fixed in See vs
yep, it's been restored, but I want to know the reason it happens


Question: require ( github.com/blang/semver v3.5.0+incompatible <ins> github.com/go-logr/logr v0.1.0 </ins> github.com/google/go-cmp v0.4.0 github.com/konsorten/go-windows-terminal-sequences v1.0.2 // indirect github.com/matttproud/golang_protobuf_extensions v1.0.2-0.20181231171920-c182affec369 // indirect

What would you like to be : As part of should add logging configuration allowing to change logging implementation. Things to implement: --logging-format - flag to pick between available implementations. Defaults to "text" using unchanged klog. LoggingConfig - structure storing component configuration for logging LogFormatRegistry - Used for registration of additional log formats. New formats should be registered by providing unique name and implementation of interface LoggingConfig.Complete method should configure klog by passing logr interface implementation into klog.SetLogger method Why is this needed: Subtask of /help /wg component-standard
The klog 1.0.0 version looks like no SetLogger method, should update klog version in vendor?
yes, forgot about that. I remember saying that it will require some changes, but shouldn't be to hard. I will help with it if needed.
/assign /remove-help
Klog was upgraded to v2.


Question: "[Flaky]", }, { <del> "a disruptive test case that is invalid [Disruptive]", "[Disruptive]", }, { </del> "a feature test case that is invalid [Feature:Awesome]", "[Feature:Awesome]", },

What would you like to be : Our tooling that makes the list of conformance tests needs to allow conformance tests to be marked as disruptive. Why is this needed: To test default, conformance behavior which may be disruptive (e.g. taint eviction) ref /sig testing /area conformance


Question: "[Flaky]", }, { <del> "a disruptive test case that is invalid [Disruptive]", "[Disruptive]", }, { </del> "a feature test case that is invalid [Feature:Awesome]", "[Feature:Awesome]", },

What happened: Ran a conformance test run (using Sonobuoy) on One of the taint based tests a NoExecute taint to a node which evicted all workloads including the Sonobuoy workload itself. If this had been a prod workload, numerous prod workloads could have been impacted. What you expected to happen: Test run as normal with reasonable results. Able to run conformance tests with minimal impact on a running cluster. How to reproduce it (as minimally and precisely as possible): on a v1.16.0 cluster. Anything else we need to know?: Environment: Kubernetes version (use ): /area conformance
/sig testing /sig architecture /sig scheduling
Summarizing the many conversations on the PR and in Slack regarding this: We wish we had identified the lack of a tag sooner and could have taken better action. Since the release is 2 days away and our own tooling currently prohibits us from marking tests as both and , we've decided to let the release proceed with this "bug". Risk: Someone running the conformance tests may unexpectedly have their workloads evicted Mitigation: Sonobuoy v0.16.0 will provide the capabilities to the user to run a "conformance-lite" (name TBD) run if they'd like to run the conformance tests against a cluster with prod workloads. This way end-users can still be reasonably sure of their cluster's continued well-being. This mode will manually skip the tests in question until they can be more appropriately marked as Sonobuoy will also provide, consistent with past behavior, the means to run the full conformance suite including these disruptive tests. Eventual solution: Sonobuoy will, most likely, need to continue offering a "non-disruptive-conformance" mode to aid in end user testing but the regexp will change and should be able to just skip tests K8s tooling needs to be tweaked to allow for tests to be marked as The tag should be to these tests.


Question: }) }) <ins> var _ = common.SIGDescribe("DNS HostNetwork", func() { f := framework.NewDefaultFramework("hostnetworkdns") f.NamespacePodSecurityEnforceLevel = admissionapi.LevelPrivileged ginkgo.It("should resolve DNS of partial qualified names for services on hostNetwork pods with dnsPolicy: ClusterFirstWithHostNet [LinuxOnly]", func() { // Create a test headless service. ginkgo.By("Creating a test headless service") testServiceSelector := map[string]string{ "dns-test": "true", } headlessService := e2eservice.CreateServiceSpec(dnsTestServiceName, "", true, testServiceSelector) _, err := f.ClientSet.CoreV1().Services(f.Namespace.Name).Create(context.TODO(), headlessService, metav1.CreateOptions{}) framework.ExpectNoError(err, "failed to create headless service: %s", dnsTestServiceName) regularServiceName := "test-service-2" regularService := e2eservice.CreateServiceSpec(regularServiceName, "", false, testServiceSelector) regularService, err = f.ClientSet.CoreV1().Services(f.Namespace.Name).Create(context.TODO(), regularService, metav1.CreateOptions{}) framework.ExpectNoError(err, "failed to create regular service: %s", regularServiceName) // All the names we need to be able to resolve. // for headless service. namesToResolve := []string{ headlessService.Name, fmt.Sprintf("%s.%s", headlessService.Name, f.Namespace.Name), fmt.Sprintf("%s.%s.svc", headlessService.Name, f.Namespace.Name), fmt.Sprintf("_http._tcp.%s.%s.svc", headlessService.Name, f.Namespace.Name), fmt.Sprintf("_http._tcp.%s.%s.svc", regularService.Name, f.Namespace.Name), } // TODO: Validate both IPv4 and IPv6 families for dual-stack wheezyProbeCmd, wheezyFileNames := createProbeCommand(namesToResolve, nil, regularService.Spec.ClusterIP, "wheezy", f.Namespace.Name, framework.TestContext.ClusterDNSDomain, framework.TestContext.ClusterIsIPv6()) jessieProbeCmd, jessieFileNames := createProbeCommand(namesToResolve, nil, regularService.Spec.ClusterIP, "jessie", f.Namespace.Name, framework.TestContext.ClusterDNSDomain, framework.TestContext.ClusterIsIPv6()) ginkgo.By("Running these commands on wheezy: " + wheezyProbeCmd + "n") ginkgo.By("Running these commands on jessie: " + jessieProbeCmd + "n") // Run a pod which probes DNS and exposes the results by HTTP. ginkgo.By("creating a pod to probe DNS") pod := createDNSPod(f.Namespace.Name, wheezyProbeCmd, jessieProbeCmd, dnsTestPodHostName, dnsTestServiceName) pod.ObjectMeta.Labels = testServiceSelector pod.Spec.HostNetwork = true pod.Spec.DNSPolicy = v1.DNSClusterFirstWithHostNet validateDNSResults(f, pod, append(wheezyFileNames, jessieFileNames...)) }) }) </ins>

There are not e2e tests for some DNSPolicies see: /help /sig network
This request has been marked as needing help from a contributor. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /sig network Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign


Question: if err != nil { t.Fatal(err) } <del> if res.StatusCode != http.StatusGatewayTimeout { t.Errorf("got res.StatusCde %d; expected %d", res.StatusCode, http.StatusServiceUnavailable) </del> <ins> if actual, expected := res.StatusCode, http.StatusGatewayTimeout; actual != expected { t.Errorf("got status code %d; expected %d", actual, expected) </ins> } res.Body.Close() }

Similar to and easily reproducible. The handler occasionally completes and sends a response before timing out. /kind failing-test /kind flake /sig api-machinery
/assign
/assign Thank you very much for fixing these tests.
/triage accepted


Question: http_archive( name = "io_kubernetes_build", <del> sha256 = "66a44fd5f6357268340d66fbd8a502065445a7c022732fe5f6fd84d9a20f75a3", strip_prefix = "repo-infra-e8f2f7c3decf03e1fde9f30d249e39b8328aa8b0", urls = mirror("https://github.com/kubernetes/repo-infra/archive/e8f2f7c3decf03e1fde9f30d249e39b8328aa8b0.tar.gz"), </del> <ins> sha256 = "21160531ea8a9a4001610223ad815622bf60671d308988c7057168a495a7e2e8", strip_prefix = "repo-infra-b4bc4f1552c7fc1d4654753ca9b0e5e13883429f", urls = mirror("https://github.com/kubernetes/repo-infra/archive/b4bc4f1552c7fc1d4654753ca9b0e5e13883429f.tar.gz"), </ins> ) http_archive(

/kind bug What happened: If you start with a minimal Debian image and attempt to install kubernetes- as an artifact from a Bazel build, it will fail with: This has been a known issue in the past, and there are workarounds for CI in place, but it's still unexpected behaviour () What you expected to happen: kubernetes- installs. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): master Cloud provider or hardware configuration: n/a OS (e.g. from /etc/os-release): debian Kernel (e.g. ): n/a Install tools: Others: /sig cluster-lifecycle
hm, this might be a bug in our "fast" tar builder (). if I revert to the bazel-supplied version: then the tarball looks more like we'd expect:
I'll dig a bit more. I started with the assumption that this was a Bazel bug needing a workaround.
cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale /assign This should probably still be fixed.
/assign
/lifecycle active
Once is ok & merged, referenced will need updating in
Oops. Didn't realise this issue was getting spammed by my bazel testing.
yep, better not include references in commits. is this resolved?
Yes, fixed this.
we still need to bump the kubernetes/repo-infra dependency in WORKSPACE though. I'll open a PR to do that for kubernetes/kubernetes.


Question: return bo1.Before(bo2) } <ins> // NumUnschedulablePods returns the number of unschedulable pods exist in the SchedulingQueue. func (p *PriorityQueue) NumUnschedulablePods() int { p.lock.RLock() defer p.lock.RUnlock() return len(p.unschedulableQ.pods) } </ins> // UnschedulablePodsMap holds pods that cannot be scheduled. This data structure // is used to implement unschedulableQ. type UnschedulablePodsMap struct {

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Today, the scheduler re-processes unschedulable nodes . Node updates are sent by Kubelet frequently (every 10 seconds) and often times they don't change any of the node properties that could potentially make them feasible. We should be smarter about these updates and move pods to the schedulable queue only when certain properties of nodes change that could make them feasible. These properties include node allocatable, node conditions, taints, labels, .... Why is this needed: Improves performance of the scheduler and avoids doing unnecessary attempts to schedule pods. This is particularly helpful when a cluster has a lot of unschedulable pods. <!-- DO NOT EDIT BELOW THIS LINE --/kind feature This is not a feature really. It is performance optimization, but in the absence of such a label, I need to use "feature" here. /sig scheduling
I can help to fix it. /assign mlmhl
The main reason for this problem is / update their status too frequently and most of these updates are heartbeats without any meaningful change. Recently sig-node introduces a new to alleviate this problem. With this mechanism, kubelet will split heartbeat and node status updates into two independent operations, and reduce the frequency of status updates to once per 1 minute(or more). In the long run this mechanism can indeed alleviate our problems, but at present it does not help us much as this is still an alpha feature and the node status updates frequency is still a bit high(per 1 minute right now).
Maybe we can try comparing old and new nodes after stripping their heartbeat info, like this:
I think that had sent earlier is more efficient. However, that PR does not filter out . In Wei's code, the heartbeat time is removed correctly. I think we should fix this issue and resubmit that PR.
could you resubmit a PR to: disregard in conditions comparison: and (optionally) disregard timestamp in taints comparison:
Thanks for all your advice, a new PR is submitted: This PR does not include your second suggestion(disregard timestamp in taints comparison) as I'm not sure if can be changed when node updating.


Question: return bo1.Before(bo2) } <ins> // NumUnschedulablePods returns the number of unschedulable pods exist in the SchedulingQueue. func (p *PriorityQueue) NumUnschedulablePods() int { p.lock.RLock() defer p.lock.RUnlock() return len(p.unschedulableQ.pods) } </ins> // UnschedulablePodsMap holds pods that cannot be scheduled. This data structure // is used to implement unschedulableQ. type UnschedulablePodsMap struct {

If it can be changed, I will add this filter later, otherwise I think we can keep the current implementation to make the compare operation simpler and faster. What's more, just as I said in the PR's description, I'm not sure how much side effects this change has on scheduler's resource consumption, as it always copy node's conditions before comparison.
I am thinking whether we could move this logic(whether node became more schedulable) to kubelet, kubelet reports it through heartbeats.


Question: return nil, fmt.Errorf("container not found (%s)", o.ContainerName) } <del> if len(o.SuggestedCmdUsage) > 0 { </del> <ins> if o.EnableSuggestedCmdUsage { </ins> fmt.Fprintf(o.ErrOut, "Defaulting container name to %s.n", pod.Spec.Containers[0].Name) <del> fmt.Fprintf(o.ErrOut, "%sn", o.SuggestedCmdUsage) </del> <ins> fmt.Fprintf(o.ErrOut, "Use '%s describe pod/%s -n %s' to see all of the containers in this pod.n", o.ParentCommandName, o.PodName, o.Namespace) </ins> } klog.V(4).Infof("defaulting container name to %s", pod.Spec.Containers[0].Name)

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: The pod name is missing in suggested cmd usage when executing : What you expected to happen: The suggested cmd usage should be a complete cmd like the one got from How to reproduce it (as minimally and precisely as possible): Use to attach any pod/deployment/service, the PodName will be missing in the suggested cmd usage like above. Anything else we need to know?: It works fine for kubectl < v1.12 Environment: Kubernetes version (use ): v1.13.1 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig cli /assign


Question: <ins> /* Copyright 2021 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package options import ( "net" "testing" "k8s.io/apiserver/pkg/server" "k8s.io/client-go/rest" ) func TestEmptyMainCert(t *testing.T) { secureServingInfo := &server.SecureServingInfo{} var loopbackClientConfig *rest.Config s := (&SecureServingOptions{ BindAddress: net.ParseIP("127.0.0.1"), }).WithLoopback() ln, err := net.Listen("tcp", "127.0.0.1:0") if err != nil { t.Fatalf("failed to listen on 127.0.0.1:0") } defer ln.Close() s.Listener = ln s.BindPort = ln.Addr().(*net.TCPAddr).Port if err := s.ApplyTo(&secureServingInfo, &loopbackClientConfig); err != nil { t.Errorf("unexpected error: %v", err) } if loopbackClientConfig == nil { t.Errorf("unexpected empty loopbackClientConfig") } if e, a := 1, len(secureServingInfo.SNICerts); e != a { t.Errorf("expected %d SNICert, got %d", e, a) } } </ins>

The code block does these actions: the self-signed cert for loopback connection the loopbackClientConfig the self-signed cert to the If the main cert () is empty, the loopbackClientConfig will not be created due to this check, loopbackClientConfig shouldn't be left empty, because we have created the SNICert, it's just that we haven't executed action 3 to add the SNICert to the list. Switching the order of action 2 and action 3 will make the loopbackClientConfig successfully created. /kind bug /sig auth api-machinery


Question: func (e *transformTest) getEncryptionOptions() []string { if e.transformerConfig != "" { <del> return []string{"--experimental-encryption-provider-config", path.Join(e.configDir, encryptionConfigFileName)} </del> <ins> return []string{"--encryption-provider-config", path.Join(e.configDir, encryptionConfigFileName)} </ins> } return nil

Ref Ref ref /sig auth /kind feature /milestone v1.11 /priority important-longterm /status approved-for-milestone cc
What's need to be done here? I can think only about renaming to and updating the docs. What else?
What's required to promote encryption at rest with static configs to beta?
an example of promoting flags is here: If there's a feature gate, we're going to want to enable it by default too.
/assign
submitted PRs for that: and
[MILESTONENOTIFIER] Milestone Removed From Issue Important: Code freeze is in effect and only issues with may remain in the v1.11 milestone. <details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
Will this make it to Beta in 1.12? From sig-auth on 7/11, it sounded like this was missing a release shepherd. Is anyone owning pushing this forward?
I'll do my best to push this. edit: I did not realize that 1.12 feature freeze is today (or so I heard), I don't think I'll be able to make it :slightlyfrowningface:
Ack - that's what I was wondering (with the feature freeze). Thanks
- Is this making it to the 1.12 ?
The current plan is 1.13


Question: if terminated { p.PodSandboxStatus.State = runtimeapi.PodSandboxState_SANDBOX_NOTREADY } <del> p.PodSandboxStatus.Id = string(uuid.NewUUID()) </del> <ins> p.PodSandboxStatus.Id = strings.ReplaceAll(string(uuid.NewUUID()), "-", "") </ins> return p }

Since upgrading from Kubernetes to we no longer get memory metrics from the endpoint on . Example: In our previous version, we would get a response like the following that included utilisation metrics: Hit the endpoint of on a cluster running version . Kubernetes version (use ): Cloud provider or hardware configuration: Provisioned on AWS with OS (e.g: ): Flatcar Container Linux by Kinvolk 2765.2.6 (Oklo) Kernel (e.g. ): 5.10.43-flatcar
/sig node
Hi AlexLastwhich container runtimes are you using? containerd?
Hi AlexLast, I have test on v1.22 with containerd, I did not reproduce this issue. If memory limit is set in the container spec of the pod, /stats/summary will report availableBytes, if memory limit is not set, /stats/summary will not be reported. just like this: ! ! code is here:
/cc We have reports of missing memory metrics on 1.20.8 and 1.21.2 now... was there a cadvisor or similar change that happened?
also /cc as an FYI
If you are using cri-o, it looks like a similar issue to and (possibly an issue with cadvisor/the way cri-o integrates with cadvisor). This would also apply to the dockershim
I'm not able to repro on GKE v1.20.8-gke.700 on containerd
Can you please provide more info about your environment, container runtime etc and possible repro steps? Thanks!
/triage needs-information
Hi, First of all, I am sorry to cut in on you. I have the same issue with Kubelet, in my case with Kubeadm. Here you are the issue: Here you are my installation: Thanks in advance
^ quick summary of the above, appears to be single-node ubuntu (not sure what kernel?) containerd 1.4.6 k8s 1.21.2 kubeadm installer
It is one node. It is only for making easier where is the problem .


Question: if terminated { p.PodSandboxStatus.State = runtimeapi.PodSandboxState_SANDBOX_NOTREADY } <del> p.PodSandboxStatus.Id = string(uuid.NewUUID()) </del> <ins> p.PodSandboxStatus.Id = strings.ReplaceAll(string(uuid.NewUUID()), "-", "") </ins> return p }

Kubernetes version: 1.21.2 Cloud provider or hardware configuration: Bare metal (VMware) OS: Ubuntu 18.04.5 LTS Kernel: 4.15.0-147-generic Install tools: Kubeadm Network plugin and version (if this is a network-related bug): Weave 2.8.1 CRI: Containerd 1.4.6
Sure, we're setting the following flags for : Version: We deploy our clusters using , these missing metrics are for all pods in the cluster, including pods with memory requests and limits set.
Hitting the same issue on such kind of environments Then I tested and found no issue on environment
Besides memory metrics, we noticed a loss of network in pods. I commented at the block of container and the block of pod The loss of pods occurs at the same time as loss of pods
I created cluster as follows on 1.21.2 with kubetest containerd 1.4.6 on ubuntu. This is on older version of ubuntu 18.04 LTS on kernel I don't see memory metrics being lost here.
Can we have more details on exact kernel/OS image being used here? I can't seem to repro this on 1.21.2 with same containerd/ ubuntu version as mentioned in
Hi, may I know your output of network metrics in pods?
Hi, I have tested with two kernel versions: P.S: I have just updated my in order to make easier the test. You can use it (it is necessary to install and ). Thanks in advance
We tested with two kernel versions too.
I was able to repro finally -- it looks like missing key here is that you must be using the systemd cgroup driver. With this patch and starting cluster as follows:
Hi do you mean that if using systemd cgroup driver, this issue would happen. In my environment, both containerd and kubelet are using cgroupfs driver, but the issue still exists containerd config dump
Hi, Here you are my Containerd & Kubelet config: ` As far as I know, systemd cgroup driver should be configured in order to have a proper perfomance in Kubernetes. Is it a bug with Kubelet?


Question: if terminated { p.PodSandboxStatus.State = runtimeapi.PodSandboxState_SANDBOX_NOTREADY } <del> p.PodSandboxStatus.Id = string(uuid.NewUUID()) </del> <ins> p.PodSandboxStatus.Id = strings.ReplaceAll(string(uuid.NewUUID()), "-", "") </ins> return p }

Thanks in advance
Hi thanks for the response, I've found the following in the , so will have a go at setting that parameter
Thanks for all the info! I was only able to repro this on systemd cgroup driver as mentioned in It did not repro for me when using cgroupfs. This will need some additional investigation to figure out the root cause and why memory metrics are not being reported correctly and see if this is a kubelet or cAdvisor issue. /cc
Hi, Thanks . By the way, there is a new issue reporting the same behaviour:
After setting kubelet's log verbosity level to 10, I found logs like Is this related? And I checked that according the log, all the pod sandboxID are missing
Thanks I was able to confirm that switching to fixed the missing metrics. From a bit of research, it's my understanding that using the driver on a node with is not recommended - can you confirm?
By stepping into the code, I found that the container info returned by this func is nil, so that the will always be false. Thus will always be . That explains the last log has expected fields. But because of 1), this func cannot be called. There's really a label-related issue on cadvisor: (Thanks With the workaround, I can recover the missed metrics
/remove-triage needs-information /triage accepted
Thanks for digging deeper into this I'm still a bit confused on the relationship between this and systemd cgroup driver. Do the labels being missing have something to do with systemd cgroup driver being used? Also are you using a different socket path for containerd i.e. not the default as mentioned in ?
We didn't use the default but used and not specified
Interesting! In the test I made in the container-runtime-endpoint is set to default (i.e. ).
With debugging, I found that on kubernetes 1.21.2 when using different cgroup driver, the cgroupPath format of containers are different. : systemd + containerd:cgroupfs : systemd + containerd: systemd : cgroupfs + containerd: cgroupfs The would return a map using the path.Base of cgroupPath Later, we use containerID to check to add metrics. So only in the third case, the key of are . That's how the cgroup driver impacts the metrics.


Question: if terminated { p.PodSandboxStatus.State = runtimeapi.PodSandboxState_SANDBOX_NOTREADY } <del> p.PodSandboxStatus.Id = string(uuid.NewUUID()) </del> <ins> p.PodSandboxStatus.Id = strings.ReplaceAll(string(uuid.NewUUID()), "-", "") </ins> return p }

To sum up, and are two different cases that caused this issue. For the case, I think it's a bug we need fix: we should make sure the key of always be BTW do you know why the cgroupPath format are different when using different cgroup driver?
I believe it's expected that the cgroup naming will differ when using systemd cgroup driver vs cgroupfs. Kubelet internally has some logic for parsing and converting between the two different formats, e.g. and I left some comments on your proposed PR: Thanks for helping look into this issue!
/priority important-soon


Question: func (rs *REST) Update(ctx context.Context, name string, objInfo rest.UpdatedObjectInfo, createValidation rest.ValidateObjectFunc, updateValidation rest.ValidateObjectUpdateFunc, forceAllowCreate bool, options *metav1.UpdateOptions) (runtime.Object, bool, error) { oldObj, err := rs.services.Get(ctx, name, &metav1.GetOptions{}) if err != nil { <ins> // Support create on update, if forced to. if forceAllowCreate { obj, err := objInfo.UpdatedObject(ctx, nil) if err != nil { return nil, false, err } createdObj, err := rs.Create(ctx, obj, createValidation, &metav1.CreateOptions{DryRun: options.DryRun}) if err != nil { return nil, false, err } return createdObj, true, nil } </ins> return nil, false, err } oldService := oldObj.(*api.Service)

What happened: When running the apiserver returns an error: with being: What you expected to happen: The object being created if not found. How to reproduce it (as minimally and precisely as possible): get etcd up get apiserver up run kubectl apply (built from master as well) Anything else we need to know?: Everything built directly from master Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
I think this is caused by We do need to fix that before alpha is released. The issue is that services use a custom storage implementation which doesn't support create-on-update and the first line of the update function checks that the object exists.
Okay, thanks for looking into it . Anything I can do here?


Question: pred := storage.Everything pred.AllowWatchBookmarks = true <del> ctx, _ := context.WithTimeout(context.Background(), 2*time.Second) </del> <ins> ctx, _ := context.WithTimeout(context.Background(), 3*time.Second) </ins> watcher, err := cacher.WatchList(ctx, "pods/ns", "0", pred) if err != nil { t.Fatalf("Unexpected error: %v", err)

Currently, (best-effort) watch bookmarks are sent just right before the given watch is supposed to timeout: In we're adjusting how the size of watch cache is determined and make it dynamic and more duration-oriented (rather than based on some magic constants) - we're targeting to always keep 5 minutes of history in watch cache. With that change, we should get back to our bookmark policy: change it to (additionally to right before timeout) also send bookmarks every N seconds (setting N to slightly less than expected watch cache history period, so slightly less than X minutes we will use there) scale-test this change (to see if it doesn't introduce non-negligible overhead) if so permanently introduce this change /sig scalability api-machinery
/cc
/cc
/cc
I'm reverting as it broke bookmarks. I'm going to fix and re-submit for review hopefully later this week.


Question: // in CLOSE_WAIT state will remain in the conntrack // table. (e.g. '60s'). Must be greater than 0 to set. TCPCloseWaitTimeout *metav1.Duration `json:"tcpCloseWaitTimeout"` <ins> // tcpBeLiberal, if true, kube-proxy will configure conntrack // to run in liberal mode for TCP connections and packets with // out-of-window sequence numbers won't be marked INVALID. TCPBeLiberal bool `json:"tcpBeLiberal"` </ins> // udpTimeout is how long an idle UDP conntrack entry in // UNREPLIED state will remain in the conntrack table // (e.g. '30s'). Must be greater than 0 to set.

What happened: When kube-proxy is configured with a cluster-cidr range, it drops packets with an INVALID conntrack state for all sources on the KUBE-FORWARD chain. This is meant to prevent spurious retransmits in long-lived TCP connections to a service IP, but it prevents clusters from having asymmetrical routing on private subnets with services outside of a cluster. What you expected to happen: Packets on the KUBE-FORWARD chain with an INVALID conntrack state should be dropped for only sources inside of the cluster-cidr if set. How to reproduce it (as minimally and precisely as possible): N/A Anything else we need to know?: Change to drop INVALID conntrack packets was in Backing out that commit and compiling is my current workaround. Environment: Kubernetes version (use ): 1.17.2 Cloud provider or hardware configuration: OS (e.g: ):Kernel (e.g. ):
/sig network
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
I am reading but I don't fully grok what caused the problem in the first place, or why we think "only drop if it was in-cluster" is a correct fix? Edit, OK I get the original problem, but still don't understand why packets from outside the pod range would not exhibit , thereby requiring yet another revisit of this problem. Is this a conntrack bug? Should it be applying the DNAT?
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
I can try to give a bit more context to our setup, but the knowledge I have around this all dates back to Feb, unfortuately, so I may have forgotten details. We were replacing our mesos cluster with a kubernetes cluster in Feb and using a custom ingress solution (HAProxy w/an outside process configuring it for pod ips / mesos ip-ports) for both. The ingress nodes lived in the private subnet, but outside of the kubernetes cluster cidr range. When we upgraded from kubernetes 1.14 to kubernetes 1.15 (the first version with PR ) our ingress could no longer talk to any pod ips.


Question: // in CLOSE_WAIT state will remain in the conntrack // table. (e.g. '60s'). Must be greater than 0 to set. TCPCloseWaitTimeout *metav1.Duration `json:"tcpCloseWaitTimeout"` <ins> // tcpBeLiberal, if true, kube-proxy will configure conntrack // to run in liberal mode for TCP connections and packets with // out-of-window sequence numbers won't be marked INVALID. TCPBeLiberal bool `json:"tcpBeLiberal"` </ins> // udpTimeout is how long an idle UDP conntrack entry in // UNREPLIED state will remain in the conntrack table // (e.g. '30s'). Must be greater than 0 to set.

I discovered that the KUBE-FORWARD chain had a rule to drop invalid packets, which was in Compiling kubernetes 1.15 without this commit fixed our problems, and we have been carrying that patch forward ever since. I would like to get rid of our forked kubernetes, hence this ticket. When I was writing this ticket up, I also found also mentions the same issue with the iptables dropping INVALID conntrack state packets near the end, regarding asymmetrical routing. Edit, OK I get the original problem, but still don't understand why packets from outside the pod range would not exhibit , thereby requiring yet another revisit of this problem. I think you are correct, I can imagine packets from outside of a cluster could exhibit the same behavior as packets inside the cluster and cause spurious resets. However, I am wondering if kube-proxy should be managing packets from outside of the cluster on a node... That seems outside of the scope of kube-proxy.
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
I took a stab at implementing what I think is the correct behavior for kube-proxy in .
/assign
I think that another option is to set
I don't understand how that will help me here. The packets with a ctstate INVALID are not generated due to conntrack saturation (), but due to asymmetric routing in our network. See for an example of a similar setup.
are the packets on your asymmetric routing setup marked as invalid if you set ? it should only mark the ones that are out of window per the description
/triage accepted
I'd love to pin this down, but I don't have the means to set up a repro. Is there a way I can set up a way to experiment?
We were bit by this too after revisiting an old service that had been unused since the heady days of pre-, and would also like to see the proposed patch merged (or a configuration option to prevent insertion of the INVALID state drop rule - either suits us). I unfortunately don't have any useful input as to setting up for reproduction. In our case we have a pretty odd anycast set up with a small pool of (public) addresses managed by calico.


Question: // in CLOSE_WAIT state will remain in the conntrack // table. (e.g. '60s'). Must be greater than 0 to set. TCPCloseWaitTimeout *metav1.Duration `json:"tcpCloseWaitTimeout"` <ins> // tcpBeLiberal, if true, kube-proxy will configure conntrack // to run in liberal mode for TCP connections and packets with // out-of-window sequence numbers won't be marked INVALID. TCPBeLiberal bool `json:"tcpBeLiberal"` </ins> // udpTimeout is how long an idle UDP conntrack entry in // UNREPLIED state will remain in the conntrack table // (e.g. '30s'). Must be greater than 0 to set.

However I do seem to have a dirty workaround for small clusters. By setting notrack (ctstate UNTRACKED?) in the raw table for packets in our specific ranges that would otherwise match ctstate INVALID, the packets aren't dropped. Which may be of help to someone else who finds this issue.
We are seeing something very similar on large file downloads and our issue exactly fits this one's subject. These RESET (corrected after are happening intermittently and I am seeing similar behavior mentioned in the blog mentioned below. I am unsure if someone has found a workaround for this issue or not, so yeah my +1 to this issue being looked up by the community.
We're using as a solution for the connection reset we were seeing and might fix this issue.
Let me try to avoid that this thread derails and see if I can get current status: -proxy install rules to drop INVALID cstate connections, this solves the ~REJECT~ RESET problems are some scenarios, with asymmetric routing per example, that this rules may drop unrelated traffic. The request is to remove the rule that drops those INVALID states or to try to restrict it, and that can cause that people start to have the ~REJECT~ RESET problems and drops again. If we have a reproducer we can be more precise in the final solution, but removing one rule without any strong evidence in kube-proxy is very risky As a workaround for the asymetric routing problem, I suggested to set the nfconntracktcpbeliberal on the kernel: I think that will solve the problems, it will be good if someone can test it and provide feedback
No sadly, its my bad, it is not REJECT, it is RESET that's happening and conntrack has nothing to do with it, mostly (I hope). Kubeproxy is doing what is intended but specifically when there are external agents in play, the packet doesn't sends an ACK back to the response of SYN-ACK. Our case, we hit A to initiate a download, A calls B to get the action completed to generate download data for the user finally via their browser. A and B both are on k8s, we saw A starts and a complete cycle goes for sometime until there multiple parallel downloads requested where few gets completed and few gets a RESET.


Question: // in CLOSE_WAIT state will remain in the conntrack // table. (e.g. '60s'). Must be greater than 0 to set. TCPCloseWaitTimeout *metav1.Duration `json:"tcpCloseWaitTimeout"` <ins> // tcpBeLiberal, if true, kube-proxy will configure conntrack // to run in liberal mode for TCP connections and packets with // out-of-window sequence numbers won't be marked INVALID. TCPBeLiberal bool `json:"tcpBeLiberal"` </ins> // udpTimeout is how long an idle UDP conntrack entry in // UNREPLIED state will remain in the conntrack table // (e.g. '30s'). Must be greater than 0 to set.

On primary debugging, it appears that RESET was getting invoked by the pod but pod is just sending a RST-ACK not RST itself. The missing packet is never found in the default route, unless we see that there is a chance where ACK is missed and we found some lines in tcpdump with missing sequence for ACKs. Thats why i doubt my issue is related here. One more weird coincidence is few downloads have spurious transmission.
Thankyou, I will take a look, but mostly I would like to wait before using it, debug more to understand the actual cause and then act on it.
sorry , I wanted to say RESET, thanks for catching it. If you think is different and you can provide information and how to reproduce it, I suggest you to open a new issue
I will sure do, when it is different. For now I would like to watch and submit my findings if it is related. PS: sysctl -w net.netfilter.nfconntracktcpbeliberal=1 is already implemented and we are testing if it does something helpful. so far it doesn't looks like it. thanks for your help.
/area kube-proxy
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle rotten
I have the feeling we are also affected by this with a standard calico installation: every node announces a /122 IPv6 network to the upstream router the upstream router uses ECMP to route the packet back packets that are sent from* the pod and returned via asymmetric path from outside seem to be dropped
and if you set ?
Same behaviour with set on all nodes, the outgoing pod traffic does not return. And I instantly see packets going back into the drop rule:
:thinking:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.


Question: // in CLOSE_WAIT state will remain in the conntrack // table. (e.g. '60s'). Must be greater than 0 to set. TCPCloseWaitTimeout *metav1.Duration `json:"tcpCloseWaitTimeout"` <ins> // tcpBeLiberal, if true, kube-proxy will configure conntrack // to run in liberal mode for TCP connections and packets with // out-of-window sequence numbers won't be marked INVALID. TCPBeLiberal bool `json:"tcpBeLiberal"` </ins> // udpTimeout is how long an idle UDP conntrack entry in // UNREPLIED state will remain in the conntrack table // (e.g. '30s'). Must be greater than 0 to set.

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hi, are you still working on this? Same problem happened in our clusters, just because of asymmetrical routing.
/reopen
Reopened this issue. <detailsIn response to : /reopen </details
Also the command suggested by is helping us, but I still think this should be a bug of kube-proxy.
The problem here is: what happens if the solutions are exclusive or we regress? The addition of this rule fixed a legit bug that was documented and there is also a nice blog explaining the details Removing this rule without a repro and understanding what is the root cause of this issue is not really the best way to proceed. I think that people affected should try to find the root cause and explain the problem or just provide a way to reproduce it, so we can fix it
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at .


Question: // in CLOSE_WAIT state will remain in the conntrack // table. (e.g. '60s'). Must be greater than 0 to set. TCPCloseWaitTimeout *metav1.Duration `json:"tcpCloseWaitTimeout"` <ins> // tcpBeLiberal, if true, kube-proxy will configure conntrack // to run in liberal mode for TCP connections and packets with // out-of-window sequence numbers won't be marked INVALID. TCPBeLiberal bool `json:"tcpBeLiberal"` </ins> // udpTimeout is how long an idle UDP conntrack entry in // UNREPLIED state will remain in the conntrack table // (e.g. '30s'). Must be greater than 0 to set.

/close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The same problem happened in our cluster, the same as because of asymmetrical routing. the command suggested by doesn't work for us, and I delete the iptables rule and add new rule(refer to ), it works I think that makes sense I'll put up a reproduction of the gist link later on /reopen
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The main problem is how to be completely sure we don't regress, this comment from Tim is spot on
I'm trying to check that non-local connections will still reset without the INVALID Rule
I think the patch in is wrong; it changes the rule from "if INVALID then drop" to "if INVALID and from a pod, then drop" so that it doesn't affect non-kubernetes traffic, but I think the rule we actually need is "if INVALID and to a service, then drop". In particular, I believe we need to make sure that the rule applies to traffic with a cluster-external source and an in-cluster destination. It's possible that changing it to "if INVALID and to a pod, then drop" would also work. I skimmed and some linked bugs elsewhere but I was unable to confirm whether that rule would cover the reporters' use cases.
I think that is only INVALID and to a pod
In my case, cluster-external source to service is both ok with the INVALID rule or without the INVALID rule. It seems the rule doesn't affect on access service.


Question: // in CLOSE_WAIT state will remain in the conntrack // table. (e.g. '60s'). Must be greater than 0 to set. TCPCloseWaitTimeout *metav1.Duration `json:"tcpCloseWaitTimeout"` <ins> // tcpBeLiberal, if true, kube-proxy will configure conntrack // to run in liberal mode for TCP connections and packets with // out-of-window sequence numbers won't be marked INVALID. TCPBeLiberal bool `json:"tcpBeLiberal"` </ins> // udpTimeout is how long an idle UDP conntrack entry in // UNREPLIED state will remain in the conntrack table // (e.g. '30s'). Must be greater than 0 to set.

I add this rule but it doesn't work
So, based on that, the bug happens when a reply packet fails to get un-DNAT'ed, causing the client to then send a RST to the server, which matches the source/destination IP that the server knows of. But this could happen if you had a service with external IPs too; in that case the reply packet would be from the external endpoint IP to the node IP (because the packet would have been masqueraded), and if conntrack didn't unmap it correctly then the node would send a RST to the external IP, breaking the connection in just the same way. In this case there is no good way kube-proxy can recognize the packet as being its own, other than by using conntrack, which presumably won't work if the packet is INVALID?
It seems that there is no better way to solve this for external sources accessing the cluster CIDR, is it right? If so, Can we have a flag to decide whether to create The invalid rule? It would be nice that It's up to the user to decide whether to create the rule.
There's related discussion in . I think we should create whatever iptables rules we can that will fix up our own traffic without interfering with other traffic, and also document the sysctl that admins can use to change conntrack behavior to work around the problem. So probably, we should change the "drop if INVALID" rule to "drop if INVALID and to a pod", and also address the problem in , but not try to solve the problem with external IPs I mentioned in because we don't have a good way of fixing that without potentially breaking other things. (And it's kind of way more of an edge case anyway.) Note that can currently output rules for "from a pod", but not for "to a pod", so that will need some tweaking. (FWIW will also eventually need the ability to output nftables rules rather than iptables rules so that's another thing to think about if someone is going to change its API.)
So in the absence of a way to fix the problem with external IPs, can we have a flag to decide whether to create this rule or not? If true (default), create the drop if INVALID and to a pod rule. Otherwise, We do not create the rule.


Question: // in CLOSE_WAIT state will remain in the conntrack // table. (e.g. '60s'). Must be greater than 0 to set. TCPCloseWaitTimeout *metav1.Duration `json:"tcpCloseWaitTimeout"` <ins> // tcpBeLiberal, if true, kube-proxy will configure conntrack // to run in liberal mode for TCP connections and packets with // out-of-window sequence numbers won't be marked INVALID. TCPBeLiberal bool `json:"tcpBeLiberal"` </ins> // udpTimeout is how long an idle UDP conntrack entry in // UNREPLIED state will remain in the conntrack table // (e.g. '30s'). Must be greater than 0 to set.

I believe that even without this rule we can solve this problem by way of sysctl, right? I think we should do the following things We need a flag to disable the rule We should change the "drop if INVALID" rule to "drop if INVALID and to a pod" We document sysctl as a way to solve the problem.
If the rule is "drop if INVALID and to a pod", we shouldn't need a flag to disable it; it won't affect non-pod traffic, and it won't interfere with users that want to set the sysctl instead. (Setting the sysctl will cause packets to not be marked as INVALID so the rule we add just won't get hit.)
OK, drop if INVALID and to a pod is help us identify cluster traffic, right? We only drop cluster traffic if it's invalid, but this will not solve the problem of external IP. So we need a flag to disable the rule is drop if INVALID?
The same problem happened in our cluster, the same as because of asymmetrical routing None of the above methods solve the problem in our ENV. Are there any other workarounds?
Can you execute the following command on each node? I think this is a workaround.
Thank you for your reply I also tried removing this rule. However, k8s will reconfigure the deleted rules to the system at some point in time.
I worded that badly: I meant "external IP" as in "the endpoints are cluster-external IPs rather than pod IPs", not as in . So that would apply only to people using kube-proxy to load-balance traffic from clients outside the cluster to servers outside the cluster, which is a pretty weird use case. And nobody ever reported the packet RST bug for that case anyway. So I was arguing that we can just ignore that case. So: change the existing "drop if invalid" rule to "drop if invalid and to a pod" document the sysctl We don't need an option to disable the drop rule, because it won't affect non-k8s traffic, and it will automatically become a no-op if someone decides to use the sysctl instead.
Thank you for your response. I'd love to take this one. But I have a question, How to define k8s traffic?


Question: // in CLOSE_WAIT state will remain in the conntrack // table. (e.g. '60s'). Must be greater than 0 to set. TCPCloseWaitTimeout *metav1.Duration `json:"tcpCloseWaitTimeout"` <ins> // tcpBeLiberal, if true, kube-proxy will configure conntrack // to run in liberal mode for TCP connections and packets with // out-of-window sequence numbers won't be marked INVALID. TCPBeLiberal bool `json:"tcpBeLiberal"` </ins> // udpTimeout is how long an idle UDP conntrack entry in // UNREPLIED state will remain in the conntrack table // (e.g. '30s'). Must be greater than 0 to set.

If a pod in a cluster, we call it an underlay pod, its IP is not allocated from cluster CIDR, this is a real underlay IP, when it accesses other underlay pods in the cluster or pods that allocate IP from cluster CIDR(We call it overlay pod), is this k8s traffic? In our cluster, multiple types of CNI pods are implemented through multus running in a cluster, because of the problem of asymmetrical routing and the rule that makes them inaccessible, it seems that the above method will not solve this problem.
For kube-proxy, this is done by the . If you are in a cluster where there is no simple way to discriminate pod from non-pod traffic, then you'd have to run kube-proxy with , and then kube-proxy will see that returns and so it won't try to distinguish local from non-local traffic, and will do some things (like internal/external traffic policy) slightly differently as a result. So in this case, if there was no functioning localDetector, it would also not create the "drop invalid" rule
Thanks for the details, this is probably the case we want.
/assign
I did not understand how you could get asymmetric routing in K8s, but reading up more carefully I saw: If I understand this correctly, this is a load-balancer outside K8s that load-balance directly to POD addresses. To get asymmetric routing some kind of Direct Server Return (DSR) must also be used. So, to get asymmetric routing: from an external source is directed to an external LB LB is configured with POD-IPs as targets and selects one packet is routed to a K8s node, e.g. with ECMP. But is may not be the node where the target POD is executing packet is handled by the POD, but the response packet may be directed to some other GW This will give asymmetric routing. But it's not the every-day K8s network setup for sure. Given the above it's fairly easy to reproduce the problem. You don't even have to setup a load-balancer. The biggest problem may be that you need two ways in to the cluster. Example: ! I setup direct routes that ensures asymmetric routing, which can be checked with on vm-201 and vm-202.


Question: // in CLOSE_WAIT state will remain in the conntrack // table. (e.g. '60s'). Must be greater than 0 to set. TCPCloseWaitTimeout *metav1.Duration `json:"tcpCloseWaitTimeout"` <ins> // tcpBeLiberal, if true, kube-proxy will configure conntrack // to run in liberal mode for TCP connections and packets with // out-of-window sequence numbers won't be marked INVALID. TCPBeLiberal bool `json:"tcpBeLiberal"` </ins> // udpTimeout is how long an idle UDP conntrack entry in // UNREPLIED state will remain in the conntrack table // (e.g. '30s'). Must be greater than 0 to set.

From I can now directly ping POD-IPs, but due to the problem in this issue, I can't make a TCP connection: The POD with address runs on . A similar setup as above can be used to verify a PR. Now, there is a simple work-around: use proxy-mode=ipvs The update with the rule never made it to the ipvs proxier (a rather common case :cry: ), but in this case it may save your day. The same as above with proxy-mode=ipvs; The server on 5001 responds with the POD name.
Yeah, Our production environment also solved this problem by switching to IPVS mode. In IPVS mode, there is no the DROP rule.
The sysctl solves the problem in , but not the problem in this issue. Those problems are kind of related, but rather distant.
The same affects outgoing traffic if packets are not masqueraded to the node-ip, as should be a common setup for ipv6, please see For Calico I think you can setup BGP for clusterCIDRs and route incoming packets to PODs to the correct K8s node, instead of using ECMP. This would solve the problem, and save the extra hop.
I met the same issue. KUBE-FORWARD drop my packages.
temptatively solution
The issue fixed by , I'll close it, you can feel free to open it if have any questions. /close
Closing this issue. <detailsIn response to : The issue fixed by , I'll close it, you can feel free to open it if have any questions. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: Stop() // Returns a chan which will receive all the events. If an error occurs <del> // or Stop() is called, this channel will be closed, in which case the // watch should be completely cleaned up. </del> <ins> // or Stop() is called, the implementation will close this channel and // release any resources used by the watch. </ins> ResultChan() <-chan Event }

The docs for the Watch interface are a little ambiguous: "If an error occurs or Stop() is called, this channel will be closed, in which case the watch should be completely cleaned up." That can be read two ways: When the channel is closed, the watch is already completely cleaned up When the channel is closed, the caller must call Stop() to clean up resources. If it's the first, can I suggest we rephrase to "If an error occurs or Stop() is called, this channel will be closed. It is not required to call Stop() to free resources when the channel has been closed." If it's the second, can I suggest we rephrase to "If an error occurs or Stop() is called, this channel will be closed. On observing the closed channel, the caller must call Stop() to free resources." /sig apimachinery
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : The docs for the Watch interface are a little ambiguous: </details
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : The docs for the Watch interface are a little ambiguous: </details
/assign Mike would you be interested in taking a look?
OK, I will take a look.


Question: // AUTO-GENERATED FUNCTIONS START HERE. DO NOT EDIT. var map_Endpoint = map[string]string{ "":                   "Endpoint represents a single logical "backend" implementing a service.", <del> "addresses":          "addresses of this endpoint. The contents of this field are interpreted according to the corresponding EndpointSlice addressType field. Consumers must handle different types of addresses in the context of their own capabilities. This must contain at least one address but no more than 100.", </del> <ins> "addresses":          "addresses of this endpoint. The contents of this field are interpreted according to the corresponding EndpointSlice addressType field. Consumers must handle different types of addresses in the context of their own capabilities. This must contain at least one address but no more than 100. These are all assumed to be fungible and clients may choose to only use the first element. Refer to: https://issue.k8s.io/106267", </ins> "conditions":         "conditions contains information about the current status of the endpoint.", "hostname":           "hostname of this endpoint. This field may be used by consumers of endpoints to distinguish endpoints from each other (e.g. in DNS names). Multiple endpoints which use the same hostname should be considered fungible (e.g. multiple A values in DNS). Must be lowercase and pass DNS Label (RFC 1123) validation.", "targetRef":          "targetRef is a reference to a Kubernetes object that represents this endpoint.",

An can contain up to 1000 , which can each contain up to 100 . Or at least, that's what the docs say, but actually, kube-proxy (or more specifically, ) only looks at the first Address in each Endpoint: (This is compatible with how works, since it would only generate multi-Address Endpoints if a pod had multiple of the same address family, which is not allowed. But it's not compatible with manually-generated EndpointSlices.) Either all are used, or else the docs state that only is used and the field is an array solely for historical reasons.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig network /priority important-longterm /cc
All addresses in the slice must be of the same type, do you mean creating an Slice with different address types in the Addresses field?
I'm not sure if/how kube-proxy should handle an endpoint with more than 1 address. I think the goal of kube-proxy is to give each endpoint an equivalent probability of being selected. To extend that logic here, I think we'd need to take the probability assigned to an endpoint and divide it by the number of addresses attached to that endpoint. So if an endpoint has a 50% probability of being selected, and it provides 50 addresses, each should end up with a 1% probability of being selected. Of course with the way we need to do descending probabilities, the logic may get a bit complicated. This isn't as clear as it should be, but I don't think kube-proxy or any consumer of the EndpointSlice API needs to commit to supporting every value specified by that API. For example, many may filter by address type, Service, or some other means. I think it's also valid for a consumer to say they only support a single address per family per endpoint. So all of that to say, I'd support either documenting that kube-proxy only selects the first address in an endpoint, or updating kube-proxy to support more than one address per endpoint.


Question: // AUTO-GENERATED FUNCTIONS START HERE. DO NOT EDIT. var map_Endpoint = map[string]string{ "":                   "Endpoint represents a single logical "backend" implementing a service.", <del> "addresses":          "addresses of this endpoint. The contents of this field are interpreted according to the corresponding EndpointSlice addressType field. Consumers must handle different types of addresses in the context of their own capabilities. This must contain at least one address but no more than 100.", </del> <ins> "addresses":          "addresses of this endpoint. The contents of this field are interpreted according to the corresponding EndpointSlice addressType field. Consumers must handle different types of addresses in the context of their own capabilities. This must contain at least one address but no more than 100. These are all assumed to be fungible and clients may choose to only use the first element. Refer to: https://issue.k8s.io/106267", </ins> "conditions":         "conditions contains information about the current status of the endpoint.", "hostname":           "hostname of this endpoint. This field may be used by consumers of endpoints to distinguish endpoints from each other (e.g. in DNS names). Multiple endpoints which use the same hostname should be considered fungible (e.g. multiple A values in DNS). Must be lowercase and pass DNS Label (RFC 1123) validation.", "targetRef":          "targetRef is a reference to a Kubernetes object that represents this endpoint.",

Given the number of changes currently in flight for kube-proxy, documentation seems like a safer initial approach to me.
/assign
can I have this editing only the first address(Address[0]) in Endpoint.Address to be used. The use of an array in this field is based on historical reasons. Isn't that right
thanks for volunteering to help with this one! I know has some big kube-proxy changes/fixes in flight right now, so it would be good to be sure you don't conflict with him. Otherwise, as long as isn't already working on this, I think you're free to have it.
I don't think we agreed that the array is for historical reasons only. We aren't currently using it, but we aren't ready to say that we won't use it in the future at this point. So, the docs should warn that kube-proxy currently ignores additional addresses, but it shouldn't imply that it's actually wrong to have additional addresses I'm not
thanks i will work on this
It was for a reason, but I think that reason was mixed-family slices (before we decided to break it up into single-family slices). What would it mean to have multiple addresses here? Are they assumed to be equivalent? Or do they have different context (like a split-horizon)? If the latter, there isn't any metadata here (i.e. it's not a struct like to allow new fields). I am inclined to believe it was meant as equivalent, and as such it's probably OK to document that assumption clearly. That said, I don't see how anyone could use multiple addresses in a meaningful way - they are either assumed equivalent or they lack enough metadata to distinguish their differences. ?? If we do document the equivalence, it should be generically worded. E.g. "These addresses are assumed to be equivalent and some consumers may choose to ignore all but the first value."
if it was meant as equivalent, why put multiple same addresses into Addresses? I don't understand what the purpose of this is.
The first version of the API had both IP families in a single slice, so it made sense to have multiple addresses. We changed that, though we didn't immediately remove the old support. So my best guess is that we should have converted this to a singular value when we did that, but we missed it.


Question: // AUTO-GENERATED FUNCTIONS START HERE. DO NOT EDIT. var map_Endpoint = map[string]string{ "":                   "Endpoint represents a single logical "backend" implementing a service.", <del> "addresses":          "addresses of this endpoint. The contents of this field are interpreted according to the corresponding EndpointSlice addressType field. Consumers must handle different types of addresses in the context of their own capabilities. This must contain at least one address but no more than 100.", </del> <ins> "addresses":          "addresses of this endpoint. The contents of this field are interpreted according to the corresponding EndpointSlice addressType field. Consumers must handle different types of addresses in the context of their own capabilities. This must contain at least one address but no more than 100. These are all assumed to be fungible and clients may choose to only use the first element. Refer to: https://issue.k8s.io/106267", </ins> "conditions":         "conditions contains information about the current status of the endpoint.", "hostname":           "hostname of this endpoint. This field may be used by consumers of endpoints to distinguish endpoints from each other (e.g. in DNS names). Multiple endpoints which use the same hostname should be considered fungible (e.g. multiple A values in DNS). Must be lowercase and pass DNS Label (RFC 1123) validation.", "targetRef":          "targetRef is a reference to a Kubernetes object that represents this endpoint.",

On Wed, Nov 24, 2021, 5:45 PM Cyclinder
so we should change the API doc? Or It's just that documenting it will only use the first address in the , like you said above
I'd like some of the others to comment, but it's this a US holiday, so it might be a few days..
So maybe we should just declare that the array is essentially a bug, and only the first address is used? Maybe something should warn if you ever set additional addresses? (FTR, the context that I noticed this in was that I was adding unit tests, and was creating EndpointSlices with a single Endpoint and multiple Addresses, which doesn't work; you need multiple Endpoints with a single Address each.)
I think the simplest and most compat approach is to say "these are all assumed to be fungible and clients may choose to only use the first element". Thoughts?
I think the simplest and most compat approach is to say "these are all assumed to be fungible and clients may choose to only use the first element". Big +1 to that approach. I don't think it's necessarily invalid to set multiple addresses here, there may actually be some use cases for it, but adding this kind of statement should help clear things up significantly.
hmm as we push for multi homed pod this will make a lot of sense. There might also be a chance for community to provide a drop-in replacement for proxy that does much more than we do now. I think we should update the docs and don't modify anything for now.
Note that this API doesn't include per-IP metadata, so a multi-homed pod would be challenging with this mechanic. I think it's maybe time to reconsider multi-net more holistically. On Mon, Dec 13, 2021, 10:24 AM Khaled Henidak (Kal) <
:thinking:
If they're fungible, then "clients may choose to use only one of the elements" (or "only a subset"?). No reason to call out the first one specifically.
That's fair On Tue, Dec 14, 2021, 6:09 AM Dan Winship
I would like to add to this issue that it would makes sense to have kube-proxy reduce the subset of endpoints in the endpointslice based on the label of the node and the & attributes of the endpoint.


Question: // AUTO-GENERATED FUNCTIONS START HERE. DO NOT EDIT. var map_Endpoint = map[string]string{ "":                   "Endpoint represents a single logical "backend" implementing a service.", <del> "addresses":          "addresses of this endpoint. The contents of this field are interpreted according to the corresponding EndpointSlice addressType field. Consumers must handle different types of addresses in the context of their own capabilities. This must contain at least one address but no more than 100.", </del> <ins> "addresses":          "addresses of this endpoint. The contents of this field are interpreted according to the corresponding EndpointSlice addressType field. Consumers must handle different types of addresses in the context of their own capabilities. This must contain at least one address but no more than 100. These are all assumed to be fungible and clients may choose to only use the first element. Refer to: https://issue.k8s.io/106267", </ins> "conditions":         "conditions contains information about the current status of the endpoint.", "hostname":           "hostname of this endpoint. This field may be used by consumers of endpoints to distinguish endpoints from each other (e.g. in DNS names). Multiple endpoints which use the same hostname should be considered fungible (e.g. multiple A values in DNS). Must be lowercase and pass DNS Label (RFC 1123) validation.", "targetRef":          "targetRef is a reference to a Kubernetes object that represents this endpoint.",

Taking only into consideration the first endpoint which is in the same zone of the node or has no zone dependency and is ready for serving. & Would it make sense to reopen this issue?
I don't follow entirely what you are suggesting, can you elaborate?
kube-proxy should understand which zone a node is in and prefer endpoints in the same zone.
but that is unrelated to this issue, topology is considered See KEPs related, topology-aware-routing was replaced by topology-aware-hints
thank you for clarifying. It seems ambiguous to me to have a zone attribute and a hint attribute. I fail to implement the hinting on a service in v1.23.8, but this is urelated to this issue.
heh, don't worry, this topic is complex and we went through different solutions, you have more historical context on the KEPs I linked above and this thread in the mailing list


Question: } // NodeAllocatableRoot returns the literal cgroup path for the node allocatable cgroup <del> func NodeAllocatableRoot(cgroupRoot, cgroupDriver string) string { </del> <ins> func NodeAllocatableRoot(cgroupRoot string, cgroupsPerQOS bool, cgroupDriver string) string { </ins> return "" }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: k8s 1.17.2 config kubelet with those parameters cgroupDriver: cgroupfs cgroupRoot: /docker cgroupsPerQOS: false kubelet log always prints those E0303 17:25:03. ] Failed to get system container stats for "/docker": failed to get cgroup stats for "/docker": failed to get container info for "/docker": unknown container "/docker" E0303 17:25:03. ] eviction manager: failed to construct signal: "allocatableMemory.available" error: system container "pods" not found in metrics What you expected to happen: kubelet does not print those error information How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig node