Question: return nameservers, searches, options, utilerrors.NewAggregate(allErrors) } <ins> // Reads a resolv.conf-like file and returns the DNS config options from it. // Returns an empty DNSConfig if the given resolverConfigFile is an empty string. func getDNSConfig(resolverConfigFile string) (*runtimeapi.DNSConfig, error) { var hostDNS, hostSearch, hostOptions []string // Get host DNS settings if resolverConfigFile != "" { f, err := os.Open(resolverConfigFile) if err != nil { klog.ErrorS(err, "Could not open resolv conf file.") return nil, err } defer f.Close() hostDNS, hostSearch, hostOptions, err = parseResolvConf(f) if err != nil { err := fmt.Errorf("Encountered error while parsing resolv conf file. Error: %w", err) klog.ErrorS(err, "Could not parse resolv conf file.") return nil, err } } return &runtimeapi.DNSConfig{ Servers:  hostDNS, Searches: hostSearch, Options:  hostOptions, }, nil } </ins> func getPodDNSType(pod *v1.Pod) (podDNSType, error) { dnsPolicy := pod.Spec.DNSPolicy switch dnsPolicy {

When a --resolv-conf (DNS resolver config) is passed to kubelet on a Windows node, the config content is used as part of sandbox initialization (through getHostDNSConfig()). Now with the recent changes (), it seems now to fail with an error like: E0315 10:22:16. 4764 ] "Error syncing pod, skipping" err="failed to "CreatePodSandbox" for "img-pull-(-b40a-459d-9033-)" with CreatePodSandboxError: "Failed to generate sandbox config for pod "img-pull-(-b40a-459d-9033-)": Unexpected resolver config value: "C:etckubernetescni". Expected "" or "Host"."" pod="img-puller-3842/img-pull-" podUID=-b40a-459d-9033- This is showing on all the runs of GCE testgrid (latest): Reading/using the passed resolv config on Windows Run a Windows pool with DNS resolv config being set as part of kubelet params. No response_ 27 alpha.3 1.27 beta.0 master <detailsGCE </details<details</details<details</details<details</details<details</details
/triage accepted /milestone v1.27 /sig node /sig network
/sig windows /cc
/priority criticial-urgent as this is likely a release blocking
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /priority criticial-urgent Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/priority critical-urgent
Hmm, so you're setting the [1]. This PR introduced the capability to read the DNS config options from the Windows host registries while removing the capability to read from a file, introducing a regression. I've sent a PR addressing this issue: [1]
Is the file you are pointing to a UNIX-style or something else?
Was the file changing the behavior of the node?


Question: return nameservers, searches, options, utilerrors.NewAggregate(allErrors) } <ins> // Reads a resolv.conf-like file and returns the DNS config options from it. // Returns an empty DNSConfig if the given resolverConfigFile is an empty string. func getDNSConfig(resolverConfigFile string) (*runtimeapi.DNSConfig, error) { var hostDNS, hostSearch, hostOptions []string // Get host DNS settings if resolverConfigFile != "" { f, err := os.Open(resolverConfigFile) if err != nil { klog.ErrorS(err, "Could not open resolv conf file.") return nil, err } defer f.Close() hostDNS, hostSearch, hostOptions, err = parseResolvConf(f) if err != nil { err := fmt.Errorf("Encountered error while parsing resolv conf file. Error: %w", err) klog.ErrorS(err, "Could not parse resolv conf file.") return nil, err } } return &runtimeapi.DNSConfig{ Servers:  hostDNS, Searches: hostSearch, Options:  hostOptions, }, nil } </ins> func getPodDNSType(pod *v1.Pod) (podDNSType, error) { dnsPolicy := pod.Spec.DNSPolicy switch dnsPolicy {

Yeah, what used on GCE/GKE, seems to be a Linux-like conf, eg: nameserver 10.128.0.1 search c.<project_name.internal google.internal ` We haven't tried to remove from kubelet to see the behavior.
This Issue/PR has been marked as release blocker. Please have the fix merged by eod today otherwise the feature has to be reverted. For the Issue/PR owner: please have a revert/feature disable PR open today in case the fix failed to get in. Thank you! See the -- Kubernetes 1.27 release manager
I opened up a revert PR in case this PR does not merge by code freeze.
Thanks for the resolution! both 2019 and 2022 grids seems to be back to green now!
Thanks for confirming! I opened to ensure gets updated too


Question: // if nodeAffinity.RequiredDuringSchedulingRequiredDuringExecution != nil { // 	nodeSelectorTerms := nodeAffinity.RequiredDuringSchedulingRequiredDuringExecution.NodeSelectorTerms // 	glog.V(10).Infof("Match for RequiredDuringSchedulingRequiredDuringExecution node selector terms %+v", nodeSelectorTerms) <del> // 	nodeAffinityMatches = NodeMatchesNodeSelectorTerms(node, nodeSelectorTerms) </del> <ins> // 	nodeAffinityMatches = nodeMatchesNodeSelectorTerms(node, nodeSelectorTerms) </ins> // } // Match node selector for requiredDuringSchedulingIgnoredDuringExecution. if nodeAffinity.RequiredDuringSchedulingIgnoredDuringExecution != nil { nodeSelectorTerms := nodeAffinity.RequiredDuringSchedulingIgnoredDuringExecution.NodeSelectorTerms glog.V(10).Infof("Match for RequiredDuringSchedulingIgnoredDuringExecution node selector terms %+v", nodeSelectorTerms) <del> nodeAffinityMatches = nodeAffinityMatches && NodeMatchesNodeSelectorTerms(node, nodeSelectorTerms) </del> <ins> nodeAffinityMatches = nodeAffinityMatches && nodeMatchesNodeSelectorTerms(node, nodeSelectorTerms) </ins> } }

I just noticed you forgot to register the image locality priority function in in I also missed it when I reviewed your PR. I have fixed this in . But it would be great if you could figure out a mechanism that would ensure that all predicate/priority functions that are tested (you can assume they will always live under ) are registered, and fail the test if not. That will allow us to prevent this mistake in the future.
Thanks! Will fix this soon!
Seems need to wait for in before I send PR? Otherwise this test will always fail ...


Question: } } <ins> func TestAttacherWaitForAttach(t *testing.T) { tests := []struct { name             string driver           string makeAttachment   func() *storage.VolumeAttachment expectedAttachID string expectError      bool }{ { name:   "successful attach", driver: "attachable", makeAttachment: func() *storage.VolumeAttachment { testAttachID := getAttachmentName("test-vol", "attachable", "node") successfulAttachment := makeTestAttachment(testAttachID, "node", "test-pv") successfulAttachment.Status.Attached = true return successfulAttachment }, expectedAttachID: getAttachmentName("test-vol", "attachable", "node"), expectError:      false, }, { name:        "failed attach", driver:      "attachable", expectError: true, }, } for _, test := range tests { t.Run(test.name, func(t *testing.T) { plug, _, tmpDir, _ := newTestWatchPlugin(t, nil) defer os.RemoveAll(tmpDir) attacher, err := plug.NewAttacher() if err != nil { t.Fatalf("failed to create new attacher: %v", err) } csiAttacher := attacher.(*csiAttacher) spec := volume.NewSpecFromPersistentVolume(makeTestPV("test-pv", 10, test.driver, "test-vol"), false) if test.makeAttachment != nil { attachment := test.makeAttachment() _, err = csiAttacher.k8s.StorageV1beta1().VolumeAttachments().Create(attachment) if err != nil { t.Fatalf("failed to create VolumeAttachment: %v", err) } gotAttachment, err := csiAttacher.k8s.StorageV1beta1().VolumeAttachments().Get(attachment.Name, meta.GetOptions{}) if err != nil { t.Fatalf("failed to get created VolumeAttachment: %v", err) } t.Logf("created test VolumeAttachment %+v", gotAttachment) } attachID, err := csiAttacher.WaitForAttach(spec, "", nil, time.Second) if err != nil && !test.expectError { t.Errorf("Unexpected error: %s", err) } if err == nil && test.expectError { t.Errorf("Expected error, got none") } if attachID != test.expectedAttachID { t.Errorf("Expected attachID %q, got %q", test.expectedAttachID, attachID) } }) } } </ins> func TestAttacherWaitForVolumeAttachment(t *testing.T) { nodeName := "test-node" testCases := []struct {

The test flakes very occasionally. Below are the findings of the circumstances of the flake. During two pods are created in quick succession. The test flakes when the second Pod fails to come up. The second pod fails to come up with error: Upon exploring the logs we discovered that there was only 1 Attach call made at the beginning, and one Detach call upon timeout of the test [See timestamps]: We tracked down the error to the likely culprit in func : based on the error message returned. This is likely caused by (string) being empty [uncomfirmed], which is what the Kubelet would get from the DevicePath field on the Node object. This DevicePath field is normally on Attach and removed on Detach of the volume. The DevicePath is clearly present on the Node when the first pod is being brought up. But it is possible that it is somehow missing or not able to be retrieved by the time the second Pod is starting up (and continues to be unavailable exhausting the 15m timeout on the test) even though no Detach operation was performed on the volume in the meantime (according to logs). We are currently unable to reproduce this flake on command and did all analysis based on logs from the two flakes we saw naturally occur in the 100s of runs performed of the test. This flake was originally seen in and the test logs can be found here: https://k8s- Environment: GCE Kubernetes version (use ):Cloud provider or hardware configuration: GCP OS (e.g. from /etc/os-release):Kernel (e.g. ): /kind flake /kind bug /sig storage /cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale A check in triage shows that this failing test is still a problem, and in fact has become more of a problem since October 4th. Has there been any progress on a fix?


Question: } } <ins> func TestAttacherWaitForAttach(t *testing.T) { tests := []struct { name             string driver           string makeAttachment   func() *storage.VolumeAttachment expectedAttachID string expectError      bool }{ { name:   "successful attach", driver: "attachable", makeAttachment: func() *storage.VolumeAttachment { testAttachID := getAttachmentName("test-vol", "attachable", "node") successfulAttachment := makeTestAttachment(testAttachID, "node", "test-pv") successfulAttachment.Status.Attached = true return successfulAttachment }, expectedAttachID: getAttachmentName("test-vol", "attachable", "node"), expectError:      false, }, { name:        "failed attach", driver:      "attachable", expectError: true, }, } for _, test := range tests { t.Run(test.name, func(t *testing.T) { plug, _, tmpDir, _ := newTestWatchPlugin(t, nil) defer os.RemoveAll(tmpDir) attacher, err := plug.NewAttacher() if err != nil { t.Fatalf("failed to create new attacher: %v", err) } csiAttacher := attacher.(*csiAttacher) spec := volume.NewSpecFromPersistentVolume(makeTestPV("test-pv", 10, test.driver, "test-vol"), false) if test.makeAttachment != nil { attachment := test.makeAttachment() _, err = csiAttacher.k8s.StorageV1beta1().VolumeAttachments().Create(attachment) if err != nil { t.Fatalf("failed to create VolumeAttachment: %v", err) } gotAttachment, err := csiAttacher.k8s.StorageV1beta1().VolumeAttachments().Get(attachment.Name, meta.GetOptions{}) if err != nil { t.Fatalf("failed to get created VolumeAttachment: %v", err) } t.Logf("created test VolumeAttachment %+v", gotAttachment) } attachID, err := csiAttacher.WaitForAttach(spec, "", nil, time.Second) if err != nil && !test.expectError { t.Errorf("Unexpected error: %s", err) } if err == nil && test.expectError { t.Errorf("Expected error, got none") } if attachID != test.expectedAttachID { t.Errorf("Expected attachID %q, got %q", test.expectedAttachID, attachID) } }) } } </ins> func TestAttacherWaitForVolumeAttachment(t *testing.T) { nodeName := "test-node" testCases := []struct {

/priority important-soon
/assign for triage
We should find the root cause and fix for 1.13
/assign Patrick can you PTAL at what is causing flakiness in this test.
I will have a look. Right now the log output of the sidecar containers and the CSI driver are not captured, which makes debugging a bit hard. In PR I am adding code which includes those message in the test log. Let me finish that PR and once it is merged, debugging the next test flake might provide some better insight into what went wrong.
Hey, this issue is considered a blocker for 1.13beta. Is there anything we can do to get it resolved sooner, so that it doesn't delay beta next week?
is trying to instrument the e2e tests to get logs from the csi driver containers so that we can debug. He has tried running the test locally ~100 times and didn't see it flake.
here's one recent one you can check out:
Michelle Au <: I had a look, but something is odd about that one. The error for the second pod is there: But under there is no further information about node bootstrap-e2e-gce-mz-minion-group-3kn6. Shouldn't that be available?
"Patrick Ohly" Here's a better example where the kublet log with the error is available: That one also has the log of the external-attacher, but that doesn't tell me much. I'm now comparing a "bad" run with a "good" local run. One observation is that in the "good" run, operationExecutor.UnmountDevice is completed before the second usage, including the "Volume detached for volume" step in Then operationExecutor.VerifyControllerAttachedVolume is done again for the second pod, exactly as it was done for the first: In the "bad" run, operationExecutor.MountVolume is invoked before that "Volume detached for volume" and then runs with an empty device path, without going through VerifyControllerAttachedVolume again: So to me it looks like a race between two conflicting operations. Does that make sense? How to fix it is less obvious. The relevant code seems to be this in Any suggestions?
This seems inline with the observations that made initially. We should trace where DevicePath is getting set.


Question: } } <ins> func TestAttacherWaitForAttach(t *testing.T) { tests := []struct { name             string driver           string makeAttachment   func() *storage.VolumeAttachment expectedAttachID string expectError      bool }{ { name:   "successful attach", driver: "attachable", makeAttachment: func() *storage.VolumeAttachment { testAttachID := getAttachmentName("test-vol", "attachable", "node") successfulAttachment := makeTestAttachment(testAttachID, "node", "test-pv") successfulAttachment.Status.Attached = true return successfulAttachment }, expectedAttachID: getAttachmentName("test-vol", "attachable", "node"), expectError:      false, }, { name:        "failed attach", driver:      "attachable", expectError: true, }, } for _, test := range tests { t.Run(test.name, func(t *testing.T) { plug, _, tmpDir, _ := newTestWatchPlugin(t, nil) defer os.RemoveAll(tmpDir) attacher, err := plug.NewAttacher() if err != nil { t.Fatalf("failed to create new attacher: %v", err) } csiAttacher := attacher.(*csiAttacher) spec := volume.NewSpecFromPersistentVolume(makeTestPV("test-pv", 10, test.driver, "test-vol"), false) if test.makeAttachment != nil { attachment := test.makeAttachment() _, err = csiAttacher.k8s.StorageV1beta1().VolumeAttachments().Create(attachment) if err != nil { t.Fatalf("failed to create VolumeAttachment: %v", err) } gotAttachment, err := csiAttacher.k8s.StorageV1beta1().VolumeAttachments().Get(attachment.Name, meta.GetOptions{}) if err != nil { t.Fatalf("failed to get created VolumeAttachment: %v", err) } t.Logf("created test VolumeAttachment %+v", gotAttachment) } attachID, err := csiAttacher.WaitForAttach(spec, "", nil, time.Second) if err != nil && !test.expectError { t.Errorf("Unexpected error: %s", err) } if err == nil && test.expectError { t.Errorf("Expected error, got none") } if attachID != test.expectedAttachID { t.Errorf("Expected attachID %q, got %q", test.expectedAttachID, attachID) } }) } } </ins> func TestAttacherWaitForVolumeAttachment(t *testing.T) { nodeName := "test-node" testCases := []struct {

Could there be a race with AD controller or external attacher if they try to detach the volume? is another race around this area. Unclear if it's the same yet.
Michelle Au <: No idea. Even if it isn't, it's at least related to the same piece of code. Probably worth fixing both together. Can we get someone who has worked in this area to fix these issues? Today is the first day that I've ever seen this code, so I'm a bit afraid that it'll take me a long time to get it right, and I'm worried about regressions.
I've asked to help investigate
It is not a blocker for beta or freeze. Can be addressed in stabilization. Let's keep the 1.13 tag on it. And based on progress by stabilization we can reassess if it is even a 1.13 blocker (at this point I don't believe it is).
The in the actual state of the world gets reset during unmounting:
That removes the device path looks wrong to me. I've submitted PR which changes that. Let's see how much that breaks...
/assign The conclusion in the CSI working group was that while my analysis and the resulting fix might be correct, further debugging and a second pair of eyes would be good. Let me reassign, because I have done what I could for this issue.
The RT considers this a blocker now, because flakes are happening often enough to interfere with getting clear signal on several jobs. Even if the underlying problem isn't that serious.
/assign I will help triage as well.
Ack Thanks
Instead of which may have negative side effects for non-CSI volumes, let's do this instead: Modify the CSI method in to ignore the passed in (which is populated from devicePath. Instead let's have that method generate the from scratch, e.g. attachID := getAttachmentName(volID, driverName, string(nodeName)) This will remove any CSI dependency on devicePath and not effect other volume plugins.
I am trying to twist my head around the code. IMO it affects also AWS that uses devicePath stored in and it should be reproducible there too. So far I was unsuccessful. In the logs it's clearly visible that UnmountDevice succeeded. So why kubelet continues with MountVolume instead of MountDevice? That's wrong.


Question: } } <ins> func TestAttacherWaitForAttach(t *testing.T) { tests := []struct { name             string driver           string makeAttachment   func() *storage.VolumeAttachment expectedAttachID string expectError      bool }{ { name:   "successful attach", driver: "attachable", makeAttachment: func() *storage.VolumeAttachment { testAttachID := getAttachmentName("test-vol", "attachable", "node") successfulAttachment := makeTestAttachment(testAttachID, "node", "test-pv") successfulAttachment.Status.Attached = true return successfulAttachment }, expectedAttachID: getAttachmentName("test-vol", "attachable", "node"), expectError:      false, }, { name:        "failed attach", driver:      "attachable", expectError: true, }, } for _, test := range tests { t.Run(test.name, func(t *testing.T) { plug, _, tmpDir, _ := newTestWatchPlugin(t, nil) defer os.RemoveAll(tmpDir) attacher, err := plug.NewAttacher() if err != nil { t.Fatalf("failed to create new attacher: %v", err) } csiAttacher := attacher.(*csiAttacher) spec := volume.NewSpecFromPersistentVolume(makeTestPV("test-pv", 10, test.driver, "test-vol"), false) if test.makeAttachment != nil { attachment := test.makeAttachment() _, err = csiAttacher.k8s.StorageV1beta1().VolumeAttachments().Create(attachment) if err != nil { t.Fatalf("failed to create VolumeAttachment: %v", err) } gotAttachment, err := csiAttacher.k8s.StorageV1beta1().VolumeAttachments().Get(attachment.Name, meta.GetOptions{}) if err != nil { t.Fatalf("failed to get created VolumeAttachment: %v", err) } t.Logf("created test VolumeAttachment %+v", gotAttachment) } attachID, err := csiAttacher.WaitForAttach(spec, "", nil, time.Second) if err != nil && !test.expectError { t.Errorf("Unexpected error: %s", err) } if err == nil && test.expectError { t.Errorf("Expected error, got none") } if attachID != test.expectedAttachID { t.Errorf("Expected attachID %q, got %q", test.expectedAttachID, attachID) } }) } } </ins> func TestAttacherWaitForVolumeAttachment(t *testing.T) { nodeName := "test-node" testCases := []struct {

A workaround in CSI volume plugin won't really help here, it will work only for drivers without .
I finally reproduced it on AWS with a long sleep in : Steps to reproduce (with the hack above): a pod with a volume the pod. Wait until is called. a new pod with the same volume. - checks that the volume is still attached and the device is still mounted and enqueues an operation to mount the volume. When finishes, starts and 1) uses empty and 2) calls on volume that's not DeviceMounted.
Fix: Any idea how to test it for real? Fake CSI plugin with slow NodeUnstage would be the best, however we don't have one.
I'm going to work on a targeted fix for CSI. We can continue to pursue fixing aws in-tree separately.
PR to fix csi is here:
what are the final set of PRs pending for this? , and ?
Only is needed to fix this particular test failure.
can you please confirm if we are going to fix this using or ?
I discussed with offline, he has some suggestions for that I am investigating. I think we will want both PRs eventually, although only one of them will be needed to fix the CSI test for now. Since it's his off hours, I will continue working on today.
Are we talking 1.13 for as well?
Reopening this since we are seeing consistent failures in latest runs. We also see one more test fail in the job with same error. Failing job https://k8s- Failing Tests [sig-storage] [Serial] CSI Volumes CSI plugin test using CSI driver: gcePD [Serial] should provision storage [sig-storage] [Serial] CSI Volumes CSI plugin test using CSI driver: hostPath [Serial] should provision storage Error This is now 1.13 release blocking.
/priority critical-urgent /kind failing-test
/remove-priority important-soon
I think this is a different issue and could be due to the fact that CSI is intentionally not backwards compatible between 1.12 and 1.13. We may need to enhance the test to figure out what version the cluster is on and launch appropriate versions of the CSI driver.
"intentionally not backwards compatible" -- is there a writeup for the release notes and release announcement for this change?


Question: } } <ins> func TestAttacherWaitForAttach(t *testing.T) { tests := []struct { name             string driver           string makeAttachment   func() *storage.VolumeAttachment expectedAttachID string expectError      bool }{ { name:   "successful attach", driver: "attachable", makeAttachment: func() *storage.VolumeAttachment { testAttachID := getAttachmentName("test-vol", "attachable", "node") successfulAttachment := makeTestAttachment(testAttachID, "node", "test-pv") successfulAttachment.Status.Attached = true return successfulAttachment }, expectedAttachID: getAttachmentName("test-vol", "attachable", "node"), expectError:      false, }, { name:        "failed attach", driver:      "attachable", expectError: true, }, } for _, test := range tests { t.Run(test.name, func(t *testing.T) { plug, _, tmpDir, _ := newTestWatchPlugin(t, nil) defer os.RemoveAll(tmpDir) attacher, err := plug.NewAttacher() if err != nil { t.Fatalf("failed to create new attacher: %v", err) } csiAttacher := attacher.(*csiAttacher) spec := volume.NewSpecFromPersistentVolume(makeTestPV("test-pv", 10, test.driver, "test-vol"), false) if test.makeAttachment != nil { attachment := test.makeAttachment() _, err = csiAttacher.k8s.StorageV1beta1().VolumeAttachments().Create(attachment) if err != nil { t.Fatalf("failed to create VolumeAttachment: %v", err) } gotAttachment, err := csiAttacher.k8s.StorageV1beta1().VolumeAttachments().Get(attachment.Name, meta.GetOptions{}) if err != nil { t.Fatalf("failed to get created VolumeAttachment: %v", err) } t.Logf("created test VolumeAttachment %+v", gotAttachment) } attachID, err := csiAttacher.WaitForAttach(spec, "", nil, time.Second) if err != nil && !test.expectError { t.Errorf("Unexpected error: %s", err) } if err == nil && test.expectError { t.Errorf("Expected error, got none") } if attachID != test.expectedAttachID { t.Errorf("Expected attachID %q, got %q", test.expectedAttachID, attachID) } }) } } </ins> func TestAttacherWaitForVolumeAttachment(t *testing.T) { nodeName := "test-node" testCases := []struct {

Yes has the release note
thanks, I'll also call it to the Release Notes lead's attention
what does it mean from the user impact perspective for CSI to not be backward compatible with 1.12? can you please point me to any discussions on this behavior for 1.13?
Kubernetes v1.13 speaks CSI v1.0 and no longer understands CSI v0.x. Any CSI drivers deployed on previous versions of Kubernetes speaking CSI v0.x must removed, updated to be CSI 1.0 compatible, and redeployed on the on Kubernetes v1.13. If only the master is upgraded to 1.13 (and nodes are 1.12 or earlier) older drivers will continue to work. When nodes are upgraded to 1.13, then older drivers will stop working. I will capture this in detail in the "Action Required" section.
Im tracking these upgrade test failures elsewhere: They are not related to this original issue, the flaky test referenced by this issue should have been solved by and for non CSI /close
Closing this issue. <detailsIn response to : Im tracking these upgrade test failures elsewhere: They are not related to this original issue, the flaky test referenced by this issue should have been solved by /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
as FYI to call this out clearly in the Release notes.


Question: package gce import ( <ins> "net/http" "net/http/httptest" </ins> "reflect" "testing" compute "google.golang.org/api/compute/v1" <ins> "k8s.io/kubernetes/pkg/util/flowcontrol" </ins> "k8s.io/kubernetes/pkg/util/rand" <ins> utiltesting "k8s.io/kubernetes/pkg/util/testing" </ins> ) func TestGetRegion(t *testing.T) {

On a large cluster, the routecontroller basically makes O(n) requests all at once, and most of them error out with "Rate Limit Exceeded". We then wait for operations to complete on the ones that did stick, and damn the rest, they'll be caught next reconcile (which is a while). In fact, in some cases it appears we're spamming hard enough to get rate limited by the upper-level API DOS protections and seeing something akin to , e.g.: We can do better than this!
c.f. , which is the other leg of this problem
I took a look into code and it seems that we are doing full reconciliation every 10s. If we need to create 2000 routes at the beginning it seems to frequently. I think that we need to: first compute how many routes we have to create based on that try to spread them over some time to reduce possibility of causing 'rate limit exceeded' I will try to put some small PR together as a starting point.
I'm reopening this one, since this doesn't seem to be solved. So basically something strange is happening here. With PR we have 10/s limit on API calls send to GCE. And we are still rate-limitted. However, according to documentation we are allowed for 20/s: So something either doesn't work or is weird.
What is super interesting from the logs of recent run: It starts to creating routes: After 2 minutes from that, we are hitting "Rate limit Exceeded" for the first time: And only after 9 minutes we create the first route ever. The second one is created after 15 minutes, and from that point we create a number of them So it seems that nothing useful is happening for the first 2 minutes, and then between 3m30 and 9m30
OK - so I think that what is happening is that all controllers are sharing GCE interface (and thuse they have common throttling). So if one controller is generating a lot of API calls, then other controllers may be throttled. has a hypothesis that it may be caused by nodecontroller.
One thing my PR didn't address (because it was late): Now we have some obnoxious differences in logging when you go to make a request, because the versus could actually be absorbing a fair amount of ratelimiter.


Question: package gce import ( <ins> "net/http" "net/http/httptest" </ins> "reflect" "testing" compute "google.golang.org/api/compute/v1" <ins> "k8s.io/kubernetes/pkg/util/flowcontrol" </ins> "k8s.io/kubernetes/pkg/util/rand" <ins> utiltesting "k8s.io/kubernetes/pkg/util/testing" </ins> ) func TestGetRegion(t *testing.T) {

It could before because of the operation rate limiter and operation polling anyways, but now could be worse.
- yes I'm aware of that. And probably this is exactly the case here.
I got fooled by that - I guess we should fix it somehow.
One approach to fixing it is a similar approach to the operation polling code, but in the : warn if we ratelimited for more than N seconds (and dump the request or something? not clear how to clarify what we were sending).
I found something else - Kubelet is also building GCE interface: We have 2000 kubelets, so if all of them send multiple requests to GCE, we may hit limits.
Any idea in what circumstances Kubelet will contact cloud provider?
Hmm - it seems it contacts GCE exactly once at the beginning. So it doesn't explain much.
I think it's only here: I've rarely seen GCE ratelimit readonly calls, in practice, and kubelets come up pretty well staggered.
OK - so with logs that I , it is pretty clear that we are very heavily throttled on GCE api calls. In particular, when I was running 1000-node cluster, I see lines where we were throttle for 1m30. I'm pretty sure it's significantly higher in 2000-node cluster. I'm going to investigate a bit more where all those reuqests come from.
Actually - it seems that we have a pretty big problem here. Basically, Kubelet seems to send a lot of requests to GCE. So to clarify: every time Kubelet is updating NodeStatus (so every 10s by default) it is sending an API call to GCE to get it's not addresses: that means, that if we have 2000 nodes, kubelets generate 200 QPS to GCE (which is significantly more than we can afford) We should think how to solve this issue, but I think the options are: increasing QPS quota or calling GCE only once per X statusUpdates.
The second issue is that we are throttled at the controller level too, and this one I'm tryin to understand right now.


Question: package gce import ( <ins> "net/http" "net/http/httptest" </ins> "reflect" "testing" compute "google.golang.org/api/compute/v1" <ins> "k8s.io/kubernetes/pkg/util/flowcontrol" </ins> "k8s.io/kubernetes/pkg/util/rand" <ins> utiltesting "k8s.io/kubernetes/pkg/util/testing" </ins> ) func TestGetRegion(t *testing.T) {

- that said, any rate-limitting on our side, will not solve the kubelet-related problem
Hmm - actually looking into implementation it seems that NodeAddresses are only contacting metadata server, so I'm not longer that convinced...
Also, I think that important thing is that a single call to GCE API can translate to multiple api calls to GCE. In particular - CreateRoute translates to: get instance insert route a bunch of getOperation (until this is finished)
OK - so I think that what is happening here is that since CreateRoute translates to: get instance insert route a bunch of getOperations That means that if we call say 2000 CreateRoute() at the same time, all of them will try to issue: getInstance at the beginning. So we will quickly accumulate 2000 GET requests in the queue. Once they are processed, we generate the "POST route" then for any processed. So It's kind of expected what is happening here. So getting back to the Zach`s PR - I think that throttling POST and GET and OPERATION calls separate is kind of good idea in general.
But I still don't really understand why do we get "RateLimitExceeded". Is that because of Kubelets?
I think that one problem we have at the RouteController level is that if CreateRoute fails, we simply don't react on it. And we will wait at least 5 minutes before even retrying it. We should add some retries at the route controller level too. I will send out some short PR for it.
But I still don't understand why do we get those "RateLimitExceeded" error
OK - so as an update. In GCE, we have a separate rate-limit for number of in-flight CreateRoute call per project. is supposed to address this issue.
OK - so this one is actually fixed. I mean that it is still long, but currently it's purely blocked on GCE.
As a comment - we have an internal bug for it already.


Question: if err != nil { assert.NoErrorf(t, err, "unable to create temp file") } stdoutBuf := &bytes.Buffer{} stderrBuf := &bytes.Buffer{} containerID := "fake-container-id"

Runs k8s with master golang: test Failure was observed on Aug 25th, it is passing after that. However it can be reproduced with stress tool. No response No response It is failing on both ppc and x86 architectures. Steps to reproduce: up golang tip version. clone kubernetes master /sig testing
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
it is a sig node test, sig testing charter is not about doing or maintain tests, we need to remove that old mindset of having a QA department, that is not how software development works or should work :) Besides, thanks for spotting it and submittn a patch /sig node /remove-sig testing
Flaky in
merging this PR will fix the issue


Question: } // writeLogs writes logs into stdout, stderr. <del> func (w *logWriter) write(msg *logMessage) error { </del> <ins> func (w *logWriter) write(msg *logMessage, addPrefix bool) error { </ins> if msg.timestamp.Before(w.opts.since) { // Skip the line because it's older than since return nil } line := msg.log <del> if w.opts.timestamp { </del> <ins> if w.opts.timestamp && addPrefix { </ins> prefix := append([]byte(msg.timestamp.Format(timeFormatOut)), delimiter[0]) line = append(prefix, line...) }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Docker has a hardcoded 16K limit on log line length, which splits lines longer than that into multiple messages in the log file. in , however, doesn't account for this when is set to and prints the timestamp in front of every entry in the log file instead of every logical line. This leads to random timestamps appearing in the middle of log lines, potentially breaking the applications that parse these logs. What you expected to happen: inserts timestamps once per logical line, so original log data is preserved. Alternatively, allow API to request the logs in a sane format like JSON, which would allow merging the lines back and combining them with the timestamps more intelligently on the application side. How to reproduce it (as minimally and precisely as possible): Create : apiVersion: v1 kind: Pod metadata: name: log-lines spec: containers: name: main image: busybox command: ['sh', '-c', 'while sleep 1; do echo -n ; for i in $(seq 1 ); do echo -n A; done; echo ; done'] Start the pod with Fetch the logs with , or using the API () Observe random timestamps inserted in the middle of the log lines. Environment: Kubernetes version (use ):
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I will work on it. /sig cli /remove-sig node
The logs are being corrupted by the API endpoint in kubelet, so shouldn't it be in scope of SIG node?
The log is not corrupted by k8s. The same log could be got using "docker container logs -t CONTAINERID" I think you should post your requirement to Docker.


Question: } // writeLogs writes logs into stdout, stderr. <del> func (w *logWriter) write(msg *logMessage) error { </del> <ins> func (w *logWriter) write(msg *logMessage, addPrefix bool) error { </ins> if msg.timestamp.Before(w.opts.since) { // Skip the line because it's older than since return nil } line := msg.log <del> if w.opts.timestamp { </del> <ins> if w.opts.timestamp && addPrefix { </ins> prefix := append([]byte(msg.timestamp.Format(timeFormatOut)), delimiter[0]) line = append(prefix, line...) }

Please refer following code Line 50 ~ 52 if config.Timestamps { logLine = append([]byte(msg.Timestamp.Format(jsonmessage.RFC3339NanoFixed)+" "), logLine...) }
kubelet doesn't rely on Docker when reading logs, it reads the files itself (see the code linked in the issue). I don't think Docker would change the forced line splitting (see for example), so this should be fixed in Kubernetes.
Your are right and kubelet read log file from host such as /var/log/pods/pods/NAMESPACEPOD-NAMExxx/....(JSON format) I will post a PR for it. /remove-sig cli /sig node
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with .


Question: } // writeLogs writes logs into stdout, stderr. <del> func (w *logWriter) write(msg *logMessage) error { </del> <ins> func (w *logWriter) write(msg *logMessage, addPrefix bool) error { </ins> if msg.timestamp.Before(w.opts.since) { // Skip the line because it's older than since return nil } line := msg.log <del> if w.opts.timestamp { </del> <ins> if w.opts.timestamp && addPrefix { </ins> prefix := append([]byte(msg.timestamp.Format(timeFormatOut)), delimiter[0]) line = append(prefix, line...) }

Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle rotten
I saw the attempt fixing this but it was scary (echo this comment: ). I suggested a way it may be improved. Also the proposed fix didn't introduce any e2e tests to validate that runtime behavior is not changed and runtimes keep the no-newline-on-splitted-log logic. are you or anybody on this thread are still interested to work on this? Please re-open if so. Meanwhile closing this ahead of bot doing it because the inactivity. /close
Closing this issue. <detailsIn response to : I saw the attempt fixing this but it was scary (echo this comment: ). I suggested a way it may be improved. Also the proposed fix didn't introduce any e2e tests to validate that runtime behavior is not changed and runtimes keep the no-newline-on-splitted-log logic. Instructions for interacting with me using PR comments are available .


Question: } // writeLogs writes logs into stdout, stderr. <del> func (w *logWriter) write(msg *logMessage) error { </del> <ins> func (w *logWriter) write(msg *logMessage, addPrefix bool) error { </ins> if msg.timestamp.Before(w.opts.since) { // Skip the line because it's older than since return nil } line := msg.log <del> if w.opts.timestamp { </del> <ins> if w.opts.timestamp && addPrefix { </ins> prefix := append([]byte(msg.timestamp.Format(timeFormatOut)), delimiter[0]) line = append(prefix, line...) }

If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
You can't reopen an issue/PR unless you authored it or you are a collaborator. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-lifecycle rotten /reopen will fix it.
Reopened this issue. <detailsIn response to : /remove-lifecycle rotten /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/triage accepted
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.


Question: } // writeLogs writes logs into stdout, stderr. <del> func (w *logWriter) write(msg *logMessage) error { </del> <ins> func (w *logWriter) write(msg *logMessage, addPrefix bool) error { </ins> if msg.timestamp.Before(w.opts.since) { // Skip the line because it's older than since return nil } line := msg.log <del> if w.opts.timestamp { </del> <ins> if w.opts.timestamp && addPrefix { </ins> prefix := append([]byte(msg.timestamp.Format(timeFormatOut)), delimiter[0]) line = append(prefix, line...) }

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available .


Question: } // writeLogs writes logs into stdout, stderr. <del> func (w *logWriter) write(msg *logMessage) error { </del> <ins> func (w *logWriter) write(msg *logMessage, addPrefix bool) error { </ins> if msg.timestamp.Before(w.opts.since) { // Skip the line because it's older than since return nil } line := msg.log <del> if w.opts.timestamp { </del> <ins> if w.opts.timestamp && addPrefix { </ins> prefix := append([]byte(msg.timestamp.Format(timeFormatOut)), delimiter[0]) line = append(prefix, line...) }

If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I received a bug report about this issue. The root cause is that the buffer reads up to a new line. If the line is greater than 4096 bytes and '--timestamps' is turrned on the kubelet will write the timestamp and the partial log line. We will need to refactor the ReadLogs function to allow for a partial line read.


Question: // in node status. By calling this function (GetVolumesToReportAttached), node status should be updated, and the volume // will not need to be updated until new changes are applied (detach is triggered again) verifyVolumeAttachedToNode(t, generatedVolumeName, nodeName1, cache.AttachStateAttached, asw) <del> verifyVolumeReportedAsAttachedToNode(t, logger, generatedVolumeName, nodeName1, true, asw) </del> <ins> verifyVolumeReportedAsAttachedToNode(t, logger, generatedVolumeName, nodeName1, true, asw, volumeAttachedCheckTimeout) </ins> }

TestRunOneVolumeDetachFailNodeWithReadWriteOnce 4 times since 03-19 23:00 PDT, intermittently ! https://k8s- Still investigating whether this is a tolerable flake or not with more history than testgrid view, or otherwise would appreciate input from anyone with expertise in this test, especially as regards to whether this should be a blocker for tomorrows beta cut. maybe: /sig storage /sig testing
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
is already open for this, looks like it is attempting to fix it
thank you!!! closing this in favor of that one, which I also to our CI signal project board for visibility. For completeness: since it's a longer standing issue, well identified as a flake and not a failure and also actively in progress, NOT considering it a blocker for the go/nogo signal for the beta tomorrow. /close
Closing this issue. <detailsIn response to : thank you!!! closing this in favor of that one, which I also to our CI signal project board for visibility. For completeness: since it's a longer standing issue, well identified as a flake and not a failure and also actively in progress, NOT considering it a blocker for the go/nogo signal for the beta tomorrow. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: // in node status. By calling this function (GetVolumesToReportAttached), node status should be updated, and the volume // will not need to be updated until new changes are applied (detach is triggered again) verifyVolumeAttachedToNode(t, generatedVolumeName, nodeName1, cache.AttachStateAttached, asw) <del> verifyVolumeReportedAsAttachedToNode(t, logger, generatedVolumeName, nodeName1, true, asw) </del> <ins> verifyVolumeReportedAsAttachedToNode(t, logger, generatedVolumeName, nodeName1, true, asw, volumeAttachedCheckTimeout) </ins> }

pull-kubernetes-unit TestRunOneVolumeDetachFailNodeWithReadWriteOnce Jan 7th, 2022 Check volume <fake-plugin/volume-nameis reported as attached to node <fail-detach-node, got false, expected true No response_ /sig storage
/cc
this is the top unit test flake according to
cc
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
I think this test is no longer flaky by checking the test grid, close it now.
Unfortunately this does still seem to be flaky: Here was the output from : /reopen
Reopened this issue. <detailsIn response to : Unfortunately this does still seem to be flaky: Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-lifecycle stale
still the case
How about adding some logging to help troubleshoot this? I looked at it a little, but it is hard to tell which part of the test is failing. PR
/
This is still happening
/triage accepted
From the logs, in the OneVolumeDetachFailNodeWithReadWriteOnce test, succeeded, but the next verification failed. In verifyVolumeAttachedToNode, the attachState is from calling and within asw, it calls . In verifyVolumeReportedAsAttachedToNode, the volumes are obtained from calling , it seems that the node is not in asw.nodesToUpdateStatusFor at this step. Cannot reproduce locally. I tried to run the test 100 times locally with debug.Stack()... 3 * 100 times all passed.
Tried to run stress test on my linux machine as well but still cannot reproduce. Command: Result: Also tried running n process in parallel. Command: Result:
It seems like it happens more frequently.


Question: // in node status. By calling this function (GetVolumesToReportAttached), node status should be updated, and the volume // will not need to be updated until new changes are applied (detach is triggered again) verifyVolumeAttachedToNode(t, generatedVolumeName, nodeName1, cache.AttachStateAttached, asw) <del> verifyVolumeReportedAsAttachedToNode(t, logger, generatedVolumeName, nodeName1, true, asw) </del> <ins> verifyVolumeReportedAsAttachedToNode(t, logger, generatedVolumeName, nodeName1, true, asw, volumeAttachedCheckTimeout) </ins> }

But I may have no time to dig into the cause. Let's see.
cc as the test author.
The problem is that this test has a lot of dependency on the sleeps that are hardcoded with a very narrow scope, but our CI seems to be able to stress it much more. As soon as you slightly change one of the sleep duration it starts to flake locally The test can not make an assumption that timing is perfect, even more in the sub second range can someone in sig-storage review this test and make it less dependant on those tight sleeps?
According to most failures, it failed at Line 660. Let me run some stress locally. with 250ms sleep with 100ms sleep
I agree this test case is too brittle due to its specific timing dependencies. To fix this, I suggest reworking the test case to: 1) Add retry with timeout to wait for final conditions 2) Remove the intermediate timing-based state transition checks. Instead, we can keep track of state transitions in the Fakes and validate those stats. wdyt?
Friendly ping from 1.27 CI Signal that this test is still a noisy flake on the master-blocking dashboard as CI Signal has flagged it in and


Question: {"name" : "BalancedResourceAllocation", "weight" : 1}, {"name" : "ServiceSpreadingPriority", "weight" : 1}, {"name" : "EqualPriority", "weight" : 1} <del> ] </del> <ins> ], "hardPodAffinitySymmetricWeight" : 10 </ins> }

( noticed this) HardPodAffinitySymmetricWeight should be in the , not in KubeSchedulerConfiguration If/when fixing this, we need to be careful to do it in a backward-compatilble way.
Do you mean ? It seems scheduler policy config's args can not passed to FitPredicates; it need to update related interface. For the : mark to be deprecated and set scheduler policy config to HardPodAffinitySymmetricWeight if policy config is empty.
Are you working on this? if no, I can take it.
, nop; please go ahead.
/ / / , When reviewing the PR, there's one question on : the default value of is 1, which means the weight (1) will always add to priorities if term matched (), does it our expectation? And the user has to set it to 0 to disable it. IMO, the default value of should be 0.
The value 1 (instead of 0) was chosen so that if you say A must run with B, there is an automatic preference that B should run with A. This seems like reasonable behavior.
Thanks very much; reasonable case to me. But it seems we can not disable it: if set to 0, the default funcs will set it to default (1); similar result to un-set. Anyway, no user complain this now :).


Question: numErrs: 0, }, { <ins> name: "invalid duplicate targetports (number with same protocol)", tweakSvc: func(s *api.Service) { s.Spec.Type = api.ServiceTypeClusterIP s.Spec.Ports = append(s.Spec.Ports, api.ServicePort{Name: "q", Port: 1, Protocol: "TCP", TargetPort: intstr.FromInt(8080)}) s.Spec.Ports = append(s.Spec.Ports, api.ServicePort{Name: "r", Port: 2, Protocol: "TCP", TargetPort: intstr.FromInt(8080)}) }, numErrs: 1, }, { name: "invalid duplicate targetports (name with same protocol)", tweakSvc: func(s *api.Service) { s.Spec.Type = api.ServiceTypeClusterIP s.Spec.Ports = append(s.Spec.Ports, api.ServicePort{Name: "q", Port: 1, Protocol: "TCP", TargetPort: intstr.FromString("http")}) s.Spec.Ports = append(s.Spec.Ports, api.ServicePort{Name: "r", Port: 2, Protocol: "TCP", TargetPort: intstr.FromString("http")}) }, numErrs: 1, }, { name: "valid duplicate targetports (number with different protocols)", tweakSvc: func(s *api.Service) { s.Spec.Type = api.ServiceTypeClusterIP s.Spec.Ports = append(s.Spec.Ports, api.ServicePort{Name: "q", Port: 1, Protocol: "TCP", TargetPort: intstr.FromInt(8080)}) s.Spec.Ports = append(s.Spec.Ports, api.ServicePort{Name: "r", Port: 2, Protocol: "UDP", TargetPort: intstr.FromInt(8080)}) }, numErrs: 0, }, { name: "valid duplicate targetports (name with different protocols)", tweakSvc: func(s *api.Service) { s.Spec.Type = api.ServiceTypeClusterIP s.Spec.Ports = append(s.Spec.Ports, api.ServicePort{Name: "q", Port: 1, Protocol: "TCP", TargetPort: intstr.FromString("http")}) s.Spec.Ports = append(s.Spec.Ports, api.ServicePort{Name: "r", Port: 2, Protocol: "UDP", TargetPort: intstr.FromString("http")}) }, numErrs: 0, }, { </ins> name: "valid type - cluster", tweakSvc: func(s *api.Service) { s.Spec.Type = api.ServiceTypeClusterIP

It is possible to express a Service where multiple front-side ports target the same back-side port. This is not very useful and may cause trouble for implementations of Services. `
just for the purpose of clarity, are you requesting that duplicate targetPorts should be rejected or the opposite? PS: I am new to Kubernetes so please pardon the dumb question :)
I think that should probably be rejected at validation unless we have a good reason WHY we need that.
Aye aye, thanks for the clarificiation!
/assign
GitHub didn't allow me to assign the following users: xiangpengzhao. Note that only can be assigned. <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
what problems did this cause with services? This change broke backwards compatibility for services that depended on this feature. For example, lets take the load balancer service The load balancer here terminates SSL so the backend is always port 80; however, you want to make sure you support both protocols so having the same targetPort here is a valid use case.
This is an interesting (ab)use of Service. Not really what I had in mind - the LB in a Service is generally assumed to be L4. I'll try to dredge up the issues that this was causing.
FYI, the does non layer4 things as well.
One issue is that this REQUIRES connection tracking when we otherwise might not need it. Consider two ServicePorts: my-service:80 mys-service:443 Let's say we map both services to a backend on port 8080, but we pass-through the VIP, so the pod sees packets as dest = my-service:8080. We MUST conntrack this VIP because we don't know whether to map the src port to 80 or 443. If service backends have unique TargetPorts, then we wouldn't have this issue. Port rewrites are at worst stateless, and at best can engage direct-return mode (no rewrites). It's worth noting that the PR which fixes this, which is proposed for revert, doesn't quite go far enough. It prevents a Service from using a given number twice and from using a given name twice, but not from using a number and a name that resolves to that number.


Question: numErrs: 0, }, { <ins> name: "invalid duplicate targetports (number with same protocol)", tweakSvc: func(s *api.Service) { s.Spec.Type = api.ServiceTypeClusterIP s.Spec.Ports = append(s.Spec.Ports, api.ServicePort{Name: "q", Port: 1, Protocol: "TCP", TargetPort: intstr.FromInt(8080)}) s.Spec.Ports = append(s.Spec.Ports, api.ServicePort{Name: "r", Port: 2, Protocol: "TCP", TargetPort: intstr.FromInt(8080)}) }, numErrs: 1, }, { name: "invalid duplicate targetports (name with same protocol)", tweakSvc: func(s *api.Service) { s.Spec.Type = api.ServiceTypeClusterIP s.Spec.Ports = append(s.Spec.Ports, api.ServicePort{Name: "q", Port: 1, Protocol: "TCP", TargetPort: intstr.FromString("http")}) s.Spec.Ports = append(s.Spec.Ports, api.ServicePort{Name: "r", Port: 2, Protocol: "TCP", TargetPort: intstr.FromString("http")}) }, numErrs: 1, }, { name: "valid duplicate targetports (number with different protocols)", tweakSvc: func(s *api.Service) { s.Spec.Type = api.ServiceTypeClusterIP s.Spec.Ports = append(s.Spec.Ports, api.ServicePort{Name: "q", Port: 1, Protocol: "TCP", TargetPort: intstr.FromInt(8080)}) s.Spec.Ports = append(s.Spec.Ports, api.ServicePort{Name: "r", Port: 2, Protocol: "UDP", TargetPort: intstr.FromInt(8080)}) }, numErrs: 0, }, { name: "valid duplicate targetports (name with different protocols)", tweakSvc: func(s *api.Service) { s.Spec.Type = api.ServiceTypeClusterIP s.Spec.Ports = append(s.Spec.Ports, api.ServicePort{Name: "q", Port: 1, Protocol: "TCP", TargetPort: intstr.FromString("http")}) s.Spec.Ports = append(s.Spec.Ports, api.ServicePort{Name: "r", Port: 2, Protocol: "UDP", TargetPort: intstr.FromString("http")}) }, numErrs: 0, }, { </ins> name: "valid type - cluster", tweakSvc: func(s *api.Service) { s.Spec.Type = api.ServiceTypeClusterIP

That is a much trickier (impossible to do statically) proposition. :( Kind of a mess for getting high-perf Service networking.
was this an issue that came up during performance testing or IRL?
Anyone has a workaround to deal with this? I think there is a lot of people using AWS ELBs on this way
if your use-case is Ingress, then you can set . This should avoid generating the invalid Service spec. Obviously, you will lose the redirect behavior.
This is not an abuse of Service, but a quite common and wide-adopted pattern, especially in the AWS world.
you can't re-open an issue/PR unless you authored it or you are assigned to it. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
you can't re-open an issue/PR unless you authored it or you are assigned to it. <detailsIn response to : /assign /reopen for further discussion Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen for further discussion.
I suspect we should probably revert the original PR. While I understand it might not have been the intention for Services, if I understand this correctly then it's a breaking change to a v1 API. Can we at least point to any documentation that indicates this was a code bug rather than a supported behavior?
Sentiment on today's sig call was that we should revert the original commit, since it broke lots of users, and then revisit to see if there's another solution.
FYI: the revert PR is
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with .


Question: numErrs: 0, }, { <ins> name: "invalid duplicate targetports (number with same protocol)", tweakSvc: func(s *api.Service) { s.Spec.Type = api.ServiceTypeClusterIP s.Spec.Ports = append(s.Spec.Ports, api.ServicePort{Name: "q", Port: 1, Protocol: "TCP", TargetPort: intstr.FromInt(8080)}) s.Spec.Ports = append(s.Spec.Ports, api.ServicePort{Name: "r", Port: 2, Protocol: "TCP", TargetPort: intstr.FromInt(8080)}) }, numErrs: 1, }, { name: "invalid duplicate targetports (name with same protocol)", tweakSvc: func(s *api.Service) { s.Spec.Type = api.ServiceTypeClusterIP s.Spec.Ports = append(s.Spec.Ports, api.ServicePort{Name: "q", Port: 1, Protocol: "TCP", TargetPort: intstr.FromString("http")}) s.Spec.Ports = append(s.Spec.Ports, api.ServicePort{Name: "r", Port: 2, Protocol: "TCP", TargetPort: intstr.FromString("http")}) }, numErrs: 1, }, { name: "valid duplicate targetports (number with different protocols)", tweakSvc: func(s *api.Service) { s.Spec.Type = api.ServiceTypeClusterIP s.Spec.Ports = append(s.Spec.Ports, api.ServicePort{Name: "q", Port: 1, Protocol: "TCP", TargetPort: intstr.FromInt(8080)}) s.Spec.Ports = append(s.Spec.Ports, api.ServicePort{Name: "r", Port: 2, Protocol: "UDP", TargetPort: intstr.FromInt(8080)}) }, numErrs: 0, }, { name: "valid duplicate targetports (name with different protocols)", tweakSvc: func(s *api.Service) { s.Spec.Type = api.ServiceTypeClusterIP s.Spec.Ports = append(s.Spec.Ports, api.ServicePort{Name: "q", Port: 1, Protocol: "TCP", TargetPort: intstr.FromString("http")}) s.Spec.Ports = append(s.Spec.Ports, api.ServicePort{Name: "r", Port: 2, Protocol: "UDP", TargetPort: intstr.FromString("http")}) }, numErrs: 0, }, { </ins> name: "valid type - cluster", tweakSvc: func(s *api.Service) { s.Spec.Type = api.ServiceTypeClusterIP

Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close


Question: MASTER_IP="" MINION_IPS="" <ins> KUBECTL_PATH=${KUBE_ROOT}/cluster/ubuntu/binaries/kubectl </ins> # Assumed Vars: #   KUBE_ROOT function test-build-release {

Though I have successfully installed kubernetes via :smiley: , still want to point out this doc missing the step to install . Thus, if user just follow the guide in the doc without having much knowledge about kubernetes, he/she would get error "It looks as if you don't have a compiled kubectl binary" during the step of . Manually install kubectl on at least master node should be necessary (suggest to let user also install on minion node for convenience), any way, the installation step is .
I forget to mention that.I will update the doc this tuned
I update the scripts in pr , you can try it now. I copied the kubectl binary from the kubernetes release and put it in , so there is no extra step for user to manually download . Actually, in the first version of ubuntu k8s deployment, I copied the to every node, but I think it is an overhead, so I removed it.I think this is just enough.
Doc update : Move the kubectl into the PATH, it may need sudo privilege so I decide not do for the user.
Have the same issue, please update the doc ASAP.
Great job :+1:
Also ran into this bug after recklessly running ( instead of ) :)


Question: pluginapi.RegisterDevicePluginServer(m.server, m) go m.server.Serve(sock) <del> // Wait till grpc server is ready. for i := 0; i < 10; i++ { services := m.server.GetServiceInfo() if len(services) > 0 { break } time.Sleep(1 * time.Second) </del> <ins> _, conn, err := dial(m.socket) if err != nil { return err </ins> } <ins> conn.Close() </ins> log.Println("Starting to serve on", m.socket) return nil

This is not 100% reproducable in my cluster. There is 20% chance of failing in my environment.
/area hw-accelerators mentioned a root cause that might be due to unit tests sharing the same Unix socket and interfering with each others. cc
Can you please share latest commit id of your code base using ?
Thanks very much, can you try the follow command please:
/assign
The latest commit id of my code base:
I found this is not 100% reproducable in my cluster. There is 20% chance of failing in my environment.
Result of go test command:
/sig scheduling
Thanks!
Hi, does these unit tests pass with PR
I run 100 times without failure.
Good to here that, Nice work! Thanks!


Question: return c.check(r) } <ins> // getExcludedChecks extracts the health check names to be excluded from the query param func getExcludedChecks(r *http.Request) sets.String { checks, found := r.URL.Query()["exclude"] if found { return sets.NewString(checks...) } return sets.NewString() } </ins> // handleRootHealthz returns an http.HandlerFunc that serves the provided checks. func handleRootHealthz(checks ...HealthzChecker) http.HandlerFunc { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { failed := false <ins> excluded := getExcludedChecks(r) </ins> var verboseOut bytes.Buffer for _, check := range checks { <ins> // no-op the check if we've specified we want to exclude the check if excluded.Has(check.Name()) { excluded.Delete(check.Name()) fmt.Fprintf(&verboseOut, "[+]%v excluded: okn", check.Name()) continue } </ins> if err := check.Check(r); err != nil { // don't include the error since this endpoint is public.  If someone wants more detail // they should have explicit permission to the detailed checks.

What would you like to be : Add support to the kube-apiserver so that we can disable health-checks via query params. Why is this needed: Currently, the healthz endpoint is inflexible in that any and all health checks are currently executed when the endpoint is hit. Since the kube-apiserver only currently supports a liveness check, this means all persistent health check failures will eventually result in a kubelet driven restart. However, this may not be desirable behavior if you have additional monitoring services layered on top of your control plane. For instance, take the etcd failure from the perspective of the kube-apiserver. Currently, persistent failures during the etcd health check from the api-server healthz endpoint will cause the kube-apiserver to be restarted. This sometimes helps (in the case of a faulty etcd-client connection) but also can be detrimental (the kube-apiserver restarts until the etcd cluster is functioning, then the kube-apiserver bombards the etcd cluster with requests to update local caches, saturating the etcd cluster with requests, causing it to timeout on its own liveness probes and trigger a kubelet-driven restart). Let's say that we could now disable etcd from our liveness health check. Then if we had an external monitoring service, we could ask independently ask etcd if it was healthy and we could also ask the kube-apiserver if it thought etcd was unhealthy. Based on the combination of those answers, we could enable more intelligent responses to certain situation, like restarting the kube-apiserver when it thought etcd was unhealthy but etcd was actually reporting as healthy but not doing so when etcd was actually unhealthy. This feature is backwards-compatible and opt-in. There would be no impact to existing behavior unless these query params are actually used. /sig api-machinery /kind feature
/cc
SGTM
/assign


Question: // Score is a list of plugins that should be invoked when ranking nodes that have passed the filtering phase. Score *PluginSet <del> // NormalizeScore is a list of plugins that should be invoked after the scoring phase to normalize scores. NormalizeScore *PluginSet </del> // Reserve is a list of plugins invoked when reserving a node to run the pod. Reserve *PluginSet

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Consolidate ) into Score plugin set in config API. Why is this needed: NormalizeScore extension point is implemented in . We decided to treat NormalizeScore as an "optional" step of Score plugin, see . However, we deferred the removal of due to API review delays. Now that PR was merged. We can start the actual removal from the API and start the API review process. /sig scheduling
/assign


Question: }) nc.leaseLister = leaseInformer.Lister() <del> nc.leaseInformerSynced = leaseInformer.Informer().HasSynced </del> <ins> if utilfeature.DefaultFeatureGate.Enabled(features.NodeLease) { nc.leaseInformerSynced = leaseInformer.Informer().HasSynced } else { // Always indicate that lease is synced to prevent syncing lease. nc.leaseInformerSynced = func() bool { return true } } </ins> nc.nodeLister = nodeInformer.Lister() nc.nodeInformerSynced = nodeInformer.Informer().HasSynced

Reboot tests have been failing since 10/12 Dashboard: https://k8s- Sample Failure: https://k8s- This test failure will block 1.13 beta, so please let's fix it before then. Thanks! /sig cluster-lifecycle /kind failing-test /priority important-soon
Note: this failure is similar to but is not the same failure, according to This is a consistent failure rather than a flake, and all nodes are failing to reboot. It's also been failing consistently since 10/11. /cc
Copying my notes here: Looking at one recent test run: The test failed because kubelet never became "not ready": When looking at one of the kubelet logs, I do see that kubelet had problems making Node updates to the api server: Errors continue until kubelet is restarted 8 minutes later (the test times out at 2 minutes):
So it seems like even though kubelet is unable to update its Node status, the Node is still showing up as Ready
Adding sig-node to this discussion: /sig node
I'm trying to repro on a dev 1.11/1.12 cluster so we can get master logs for the node controller
/cc adding from sig-node
/assign
was the only PR that went in during that time (if we don't count infra or gke changes). This PR touches the node controller and changes the node heartbeat, so it could be the culprit. I was able to grab the logs from the mater was it was still up. Snippet from the kube-controller- showed a lot of error: The PR author is out for a couple of days, so he cannot investigate. I can look at it later today maybe. If anyone from GKE wants to help, I can send you all the logs.
The node controller doesn't seem to be working at all as it wasn't marking the nodes not ready. suggested that the api group may not have been whitelisted in GKE, so I checked and that seemed to be true. We can whitelist it, but for an alpha feature, it should not be properly gated and not affect production. We can revert PR to unblock release, and wait for to re-submit a patched PR. /cc
Thanks for debugging this! Just got time to take a quick look. Does it mean should also be put behind the feature gate?


Question: }) nc.leaseLister = leaseInformer.Lister() <del> nc.leaseInformerSynced = leaseInformer.Informer().HasSynced </del> <ins> if utilfeature.DefaultFeatureGate.Enabled(features.NodeLease) { nc.leaseInformerSynced = leaseInformer.Informer().HasSynced } else { // Always indicate that lease is synced to prevent syncing lease. nc.leaseInformerSynced = func() bool { return true } } </ins> nc.nodeLister = nodeInformer.Lister() nc.nodeInformerSynced = nodeInformer.Informer().HasSynced

I think reverting is a good option to unblock stuff. - creating listed may not be necessary, but setting HasSynced is (so if we have to put it after feature gate, we should put both).
Is there any guideline on how to revert my original PR ? I haven't done it before. Just revert in my fork and submit a new PR? I also just put up a fix PR . I am fine with either submitting the fix PR or revert my original PR first.
/reopen /assign I will manually close this after verifying the bots.
Reopening this issue. <detailsIn response to : /reopen /assign I will manually close this after verifying the bots. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This is passing in the latest test run. Thanks for the quick turnaround
Nice! Closing this now. /close
Closing this issue. <detailsIn response to : Nice! Closing this now. /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: func TestIsCurrentInstance(t *testing.T) { cloud := &Cloud{ Config: Config{ <del> VMType: vmTypeVMSS, </del> <ins> VMType: vmTypeStandard, </ins> }, } testcases := []struct {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: If computer name prefix does not match VMSS name, then Azure cloud provider will use VMSS API instead of using metadata service data. This leads to throttling, because every affected kubelet is making API requests. What you expected to happen: Azure cloud provider should use metadata service data instead of calling VMSS API when instance is current. How to reproduce it (as minimally and precisely as possible): Set computer name prefix of the VMSS to something different than default. You should observe Azure provider making requests to the VMSS API rather than using metadata service, which leads to throttling if you have multiple kubelets with this bug. Anything else we need to know?: This is bug was introduced by - returns false even if instance is current, because it expects node name to match VMSS name + instance id, however node name is derived from the computer-name-prefix which can be different than VMSS name. Node name should be matched against , not against . Environment: Kubernetes version (use ): 1.18.x and higher Cloud provider or hardware configuration: azure /sig cloud-provider
/sig cloud-provider
/area provider/azure
CC
/remove-sig cloud-provider /sig cloud-provider (trying to remove the label)
/sig cloud-provider
Thanks for the reporting. Unfortunately, computer name prefix is not included in IMDS, so it can't be used without VMSS API for constructing the NodeName. Would check what could we do to workaround the issue.
Can you memorize the computer name prefix in the first request to the Azure API and then use it for the isCurrentInstance check?
opened to fix the issue. Would you like to have a look?
I can confirm that your change should fix the problem I reported. Do you know if your fix will be to 1.18.x?
is working on the cherry pick
Thank you for quick fix!


Question: parameters[k] = v.(string) } } <ins> if snapshotProvider, ok := snapshotClass.Object["driver"]; ok { snapshotter = snapshotProvider.(string) } </ins> } return testsuites.GetSnapshotClass(snapshotter, parameters, ns, suffix)

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: On running kubetest with external storage driver option with driver name "", encountered error of flag redefinition. The issue happens because: As part of this patch , a flag is defined for number of inline volumes. The driver name is used to construct the flag name. Thus, when running tests on an externally provided driver (in this instance the Persistent Disk CSI driver) with driver name "", a redefinition error happens since the ephemeral test suite code path is executed once for and once for External PD CSI driver. [example run command: kubetest --test --check-version-skew=false --testargs="-- --storage.testdriver=/path/to/gcp-compute-persistent-disk-csi-driver/test/k8s-integration/config/test-" Test config looks something like this: { StorageClass: FromFile: ... SnapshotClass: FromFile: ... DriverInfo: Name: ... } ] What you expected to happen: Test should run and succeed. How to reproduce it (as minimally and precisely as possible): Use an test-, with driver name as "", and run kubetest with focus as "External.Storage" kubetest --test --check-version-skew=false --testargs="-- --storage.testdriver=/path/to/gcp-compute-persistent-disk-csi-driver/test/k8s-integration/config/test-" Anything else we need to know?: Possible fixes options: , (), when using a FromFile or FromExistingClassName option to pass in SnapshotClass information, pickup the provisioner name from the class definition's "driver" field. the flag which has a dependency on driver names. (need to understand the impact of removing the command line flag) Environment: Kubernetes version (use ): latest master Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig storage
/assign Do you know why this parameter is set as a flag and not a part of the ?
To me, is something that defines what the driver can do. This flag is about what the test does and not related to driver capabilities.


Question: parameters[k] = v.(string) } } <ins> if snapshotProvider, ok := snapshotClass.Object["driver"]; ok { snapshotter = snapshotProvider.(string) } </ins> } return testsuites.GetSnapshotClass(snapshotter, parameters, ns, suffix)

Making it depend on the driver name was probably unnecessary. A single flag for all incarnations of that test probably would have been sufficient and thus better ("keep it simple, stupid!" - the stupid being me here...). It also shouldn't have used the global flag set. In other words:
/reopen I don't really want to get into the habit of defining per test case flags in There are thousands of test cases and it would be unwieldy to discover all the flags available and understand how to use them. What factors would influence how many volumes you want the test to create? Can we just hardcode a value in the test case?
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The defaults must be usable so that users do not need to understand them. The flags can be discovered with - not ideal, but at least they show up. That's effectively what the default value for the flag does. This is a way to to run a custom stress test: the default is so low, it just tests that the feature works. By increasing it as high as the underlying CSI driver and storage system supports, one can run this (and potentially other tests) as part of stress testing. Not sure whether it's worth supporting this. It seemed worthwhile at the time, and better than hard-coding just one value.
The bigger issue is: how do users of the test suite actually learn what tests they run and what those tests do? I suspect the answer to that is simply "read the source", in which case they'll also find the knobs that can be used to tweak testing.
Since increasing the parameter depends on the csi driver/storage system, then I would recommending splitting this up into 2 test cases: 1) test basic functionality 2) test stress scenario, and add the parameter to the driver test config similarly to other stress tests:
At least for the storage tests, because we are intending that they be run externally, having all the tunables in one place I think makes it easier to discover.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close.


Question: parameters[k] = v.(string) } } <ins> if snapshotProvider, ok := snapshotClass.Object["driver"]; ok { snapshotter = snapshotProvider.(string) } </ins> } return testsuites.GetSnapshotClass(snapshotter, parameters, ns, suffix)

If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale /help
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /remove-lifecycle stale /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale
I saw that the flag was removed in , I think we can close this one if doesn't apply anymore
/close The flag was removed in #diff-20517805986874f69e3254bed93d59996d5a6fd571a70ab8120736ded5aafa24
You can't close an active issue/PR unless you authored it or you are a collaborator. <detailsIn response to : /close The flag was removed in #diff-20517805986874f69e3254bed93d59996d5a6fd571a70ab8120736ded5aafa24 </details
/close
Closing this issue.


Question: parameters[k] = v.(string) } } <ins> if snapshotProvider, ok := snapshotClass.Object["driver"]; ok { snapshotter = snapshotProvider.(string) } </ins> } return testsuites.GetSnapshotClass(snapshotter, parameters, ns, suffix)

<detailsIn response to : /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details