Question: } func (f *FakeMounter) IsNotMountPoint(dir string) (bool, error) { <del> return IsNotMountPoint(f, dir) </del> <ins> return isNotMountPoint(f, dir) </ins> } func (f *FakeMounter) IsLikelyNotMountPoint(file string) (bool, error) {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: There's , and . The former is used by local and nfs volume plugins, while the latter is used by , which is also used by local and nfs volume plugins. What you expected to happen: Can we consolidate these? This is really confusing and could easily be the source of bugs. <!-- DO NOT EDIT BELOW THIS LINE --/kind cleanup
cc
/assign I'd like to clean up code.


Question: CLUSTER_IP_RANGE="${CONTAINER_SUBNET}" fi <ins> # If enabled kube-controller-manager will be started with the --enable-hostpath-provisioner flag ENABLE_HOSTPATH_PROVISIONER="${ENABLE_HOSTPATH_PROVISIONER:-true}" </ins> # OpenContrail networking plugin specific settings OPENCONTRAIL_TAG="${OPENCONTRAIL_TAG:-R2.20}" OPENCONTRAIL_KUBERNETES_TAG="${OPENCONTRAIL_KUBERNETES_TAG:-master}"

feature request Enables easier testing for pv/pvc, and all derivatives like pet sets. The only alternative is to use manually created pv of local type. I have time to work on it myself and created this feature to collect feedback/ideas on what provider should be enabled by default.
Since this was merged should we close this issue?


Question: }, }, }, <del> }) </del> <ins> }, true) </ins> }) // The following tests for remote command execution and port forwarding are

As far as I can tell, container liveness probes of type exec flat out don't work against clusters built at head. I had the kubelet always log the output of the exec'ed command, and it's always empty no matter what I exec, causing the probe to fail since it requires the output to be equal to the string "ok". If you want to test it, try adding a liveness probe of this form to any container: I also tried these lovely forms, to more closely match the liveness example under examples/: On the plus side, this may be an opportunistic time to switch over to
The reason that the e2e test for exec liveness probes hasn't caught this is that it only tests that containers are restarted when they fail liveness probes. It doesn't test that containers aren't restarted when their liveness checks should be succeeding. The e2e test's exec'ed commands also only ever produced empty output, rather than the expected "fail" output.
Is it possible to include this in a new pre-release, since it fixes a bug in health checking? I see it merged in master and I'm wondering what the policy is of releasing new code in a separate (0.16.*) release.


Question: importmap = "k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/healthz", importpath = "k8s.io/apiserver/pkg/server/healthz", deps = [ <ins> "//staging/src/k8s.io/apimachinery/pkg/util/sets:go_default_library", </ins> "//staging/src/k8s.io/apimachinery/pkg/util/wait:go_default_library", "//vendor/k8s.io/klog:go_default_library", ],

What would you like to be : Add support to the kube-apiserver so that we can disable health-checks via query params. Why is this needed: Currently, the healthz endpoint is inflexible in that any and all health checks are currently executed when the endpoint is hit. Since the kube-apiserver only currently supports a liveness check, this means all persistent health check failures will eventually result in a kubelet driven restart. However, this may not be desirable behavior if you have additional monitoring services layered on top of your control plane. For instance, take the etcd failure from the perspective of the kube-apiserver. Currently, persistent failures during the etcd health check from the api-server healthz endpoint will cause the kube-apiserver to be restarted. This sometimes helps (in the case of a faulty etcd-client connection) but also can be detrimental (the kube-apiserver restarts until the etcd cluster is functioning, then the kube-apiserver bombards the etcd cluster with requests to update local caches, saturating the etcd cluster with requests, causing it to timeout on its own liveness probes and trigger a kubelet-driven restart). Let's say that we could now disable etcd from our liveness health check. Then if we had an external monitoring service, we could ask independently ask etcd if it was healthy and we could also ask the kube-apiserver if it thought etcd was unhealthy. Based on the combination of those answers, we could enable more intelligent responses to certain situation, like restarting the kube-apiserver when it thought etcd was unhealthy but etcd was actually reporting as healthy but not doing so when etcd was actually unhealthy. This feature is backwards-compatible and opt-in. There would be no impact to existing behavior unless these query params are actually used. /sig api-machinery /kind feature
/cc
SGTM
/assign


Question: } } } <ins> func TestPredicatesRegistered(t *testing.T) { var functionNames []string // Files and directories which predicates may be referenced targetFiles := []string{ "./../../algorithmprovider/defaults/defaults.go", // Default algorithm "./../../factory/plugins.go",                     // Registered in init() "./../../../../../pkg/",                          // kubernetes/pkg, often used by kubelet or controller } // List all golang source files under ./predicates/, excluding test files and sub-directories. files, err := codeinspector.GetSourceCodeFiles(".") if err != nil { t.Errorf("unexpected error: %v when listing files in current directory", err) } // Get all public predicates in files. for _, filePath := range files { functions, err := codeinspector.GetPublicFunctions(filePath) if err == nil { functionNames = append(functionNames, functions...) } else { t.Errorf("unexpected error when parsing %s", filePath) } } // Check if all public predicates are referenced in target files. for _, functionName := range functionNames { args := []string{"-rl", functionName} args = append(args, targetFiles...) err := exec.Command("grep", args...).Run() if err != nil { switch err.Error() { case "exit status 2": t.Errorf("unexpected error when checking %s", functionName) case "exit status 1": t.Errorf("predicate %s is implemented as public but seems not registered or used in any other place", functionName) } } } } </ins>

I just noticed you forgot to register the image locality priority function in in I also missed it when I reviewed your PR. I have fixed this in . But it would be great if you could figure out a mechanism that would ensure that all predicate/priority functions that are tested (you can assume they will always live under ) are registered, and fail the test if not. That will allow us to prevent this mistake in the future.
Thanks! Will fix this soon!
Seems need to wait for in before I send PR? Otherwise this test will always fail ...


Question: if [ ! -w $(dirname `which gcloud`) ]; then sudo_prefix="sudo" fi <del> ${sudo_prefix} gcloud ${gcloud_prompt:-} components update alpha || true ${sudo_prefix} gcloud ${gcloud_prompt:-} components update beta || true ${sudo_prefix} gcloud ${gcloud_prompt:-} components update ${CMD_GROUP:-} || true ${sudo_prefix} gcloud ${gcloud_prompt:-} components update kubectl|| true </del> <ins> ${sudo_prefix} gcloud ${gcloud_prompt:-} components install alpha || true ${sudo_prefix} gcloud ${gcloud_prompt:-} components install beta || true ${sudo_prefix} gcloud ${gcloud_prompt:-} components install kubectl|| true </ins> ${sudo_prefix} gcloud ${gcloud_prompt:-} components update || true }

This is really annoying because it requires me to interact with my shell while running up / down whereas before the commands just ran to completion on their own.


Question: // Initialize passes a Kubernetes clientBuilder interface to the cloud provider func (vs *VSphere) Initialize(clientBuilder cloudprovider.ControllerClientBuilder, stop <-chan struct{}) { vs.kubeClient = clientBuilder.ClientOrDie("vsphere-legacy-cloud-provider") <ins> vs.nodeManager.SetNodeGetter(vs.kubeClient.CoreV1()) </ins> } // Initialize Node Informers

I am enabling CSI migration on a vSphere cluster. A node was fully drained, all its Pods are deleted, but just before KCM detaches in-tree vSphere volume volumes from it, KCM restarts. A new KCM starts, sees attached volumes in that are not needed by any Pod on the node and calls vSphere in-tree volume plugin . The in-tree vSphere volume plugin + cloud provider marks the volume as detached right away, without reaching to vCenter API. The volume remains attached forever. I don't think it's related to CSI migration, IMO a regular node drain + KCM restart at the same time can cause the same issue. KCM logs: The in-tree cloud provider has a and from it. It apparently has not processed all nodes at the time A/D controller tries to detach the node. The in-tree vSphere volume plugin + cloud provider detaches the volume in vCenter. That's really tricky, it's very hard / impossible to reproduce on a real cluster. I had to inject errors to Detach() in-tree volume call, so they fail until KCM is restarted. Newly started KCM would detach the volume just fine (no error would be injected there), but marks the volume as detached right away. For the record (and search-ability), the CSI driver that tries to attach the volume to another node complains either: Or: cc It could be somewhat related to , however, not all cases of can be related to KCM restart. As I mentioned above, it's very hard/impossible to synchronize KCM restart with draining a node in a real cluster. <details</details<detailsvSphere 7.0.3 </details<details</details<detailsOpenShift </details<detailsCRI-O </details<details</details
/sig storage
Tried just moving a Pod between nodes without CSI migration involved + restarting KCM when the volume is still attached to the old node. In-tree volume plugin on the new node says:
/assign
What CSI driver version and K8s version are you testing with?
Kubernetes is 1.25 and CSI driver is 2.7, still, I think it's not releated to the CSI driver, it's KCM who marks the in-tree volume as detached due to not populated node cache.


Question: // Initialize passes a Kubernetes clientBuilder interface to the cloud provider func (vs *VSphere) Initialize(clientBuilder cloudprovider.ControllerClientBuilder, stop <-chan struct{}) { vs.kubeClient = clientBuilder.ClientOrDie("vsphere-legacy-cloud-provider") <ins> vs.nodeManager.SetNodeGetter(vs.kubeClient.CoreV1()) </ins> } // Initialize Node Informers

/assign
/triage accepted
I found an easier way how to reproduce this without extra code in KCM: a StatefulSet with 1 Pod that uses 1 PVC. the node where the Pod runs. down kube-controller-manager to zero (not really possible for me) or kill all kube-controller-manager Pods. This will cause new KCM processes to wait for leader election. In my env. this gives me ~30 seconds for the next step. Delete the Pod. A new one will be scheduled on a different node (because the current node is cordoned) + there is no KCM running at this time that would detach the volume from the old node. KCM wins its leader election, sees a volume attached to a node where it is not needed -it calls of the volume plugin. VSphere cloud provider does not have its caches populated yet (= this bug) -the volume is not detached, but returns success.
dangling attach does not help here. The volume is still attached to the old node even after ~10 minutes. fails with and not with . I can see there is a lot of dangling attach code in the cloud provider, but I did not debug further.


Question: return allErrs } <ins> func ValidateNoNewFinalizers(newFinalizers []string, oldFinalizers []string, fldPath *field.Path) field.ErrorList { const newFinalizersErrorMsg string = `no new finalizers can be added if the object is being deleted` allErrs := field.ErrorList{} extra := sets.NewString(newFinalizers...).Difference(sets.NewString(oldFinalizers...)) if len(extra) != 0 { allErrs = append(allErrs, field.Forbidden(fldPath, fmt.Sprintf("no new finalizers can be added if the object is being deleted, found new finalizers %#v", extra.List()))) } return allErrs } </ins> func validateVolumes(volumes []api.Volume, fldPath *field.Path) (sets.String, field.ErrorList) { allErrs := field.ErrorList{}

says "If the deletionTimestamp of the object is non-nil, entries in this list can only be removed.". can you elucidate if I am assuming this does something different than reality
This is garbage collection finalizer field, not the namespace finalizer field that that new field was inspired from On Thursday, September 8, 2016, Tim Hockin wrote:
I don't parse that - is this not a general-purpose "refcount" thing? It seems to work that way, except not as documented
This is a general purpose field and it should work like Tim thinks it should work. can you take a look at this when you don't have anything pressing for 1.4?
The namespace finalizer should switch to using this field if it is currently doing something else.
I'm adding validation rule to prevent adding finalizers if deletionTimestamps is set. I'm still investigating why it can't be removed. This bug is strange.
Actually I guess it's a bug. It uses patch, and patch doesn't work on list of simple type, is a list of string.
Yep, it's a bug. I used to remove one finalizer and it worked as expected. I think this is a known bug of , we don't have a way to specify the mergekey of a list of simple type.
Is there a bug open for this? Frankly, if is going to fail for things like this, we should not be offering it as a feature, or it should use PUT rather than PATCH , Chao Xu wrote:
Created cannot use PUT, because it's doing a three way merge (current state, last configuration, current configuration).
These are intended for automated use, not human use, so kubectl edit not working is just unfortunate, not a blocker. Although it's really unfortunate that PATCH is so broken. , Chao Xu wrote:
Why does need to be 3-way? I don't think that is the semantic a user would expect from an edit operation AT ALL. It certainly surprised ME. WRT automated vs human - if PATCH doesn't work (worse - says it succeeded but didn't) for some class of fields, then PATCH doesn't work, and we should not support it. Full stop. It's going to end up causing someone an outage. These are not necessarily related to this bug, but I'd call them pretty serious flaws.


Question: return allErrs } <ins> func ValidateNoNewFinalizers(newFinalizers []string, oldFinalizers []string, fldPath *field.Path) field.ErrorList { const newFinalizersErrorMsg string = `no new finalizers can be added if the object is being deleted` allErrs := field.ErrorList{} extra := sets.NewString(newFinalizers...).Difference(sets.NewString(oldFinalizers...)) if len(extra) != 0 { allErrs = append(allErrs, field.Forbidden(fldPath, fmt.Sprintf("no new finalizers can be added if the object is being deleted, found new finalizers %#v", extra.List()))) } return allErrs } </ins> func validateVolumes(volumes []api.Volume, fldPath *field.Path) (sets.String, field.ErrorList) { allErrs := field.ErrorList{}

, Daniel Smith wrote:
Discussed in real life. PATCH works, it's just that there are certain operations you cannot express using the strategicmerge patch syntax. I was wrong. is building . So perhaps it uses PATCH over PUT due to the efficiency.
FWIW, I printed out the patch generated by when I removed one finalizer. The patch is empty, i.e., "{}".
, Chao Xu wrote: Well, no, if there are operations that can not be represented, it really DOESN'T work. I still think it should not use PUT, but I bet Brian can prove me wrong.


Question: Fail(fmt.Sprintf(format, a...), 1) } <ins> func providerIs(providers ...string) bool { if testContext.Provider == "" { Fail("testContext.Provider is not defined") } for _, provider := range providers { if strings.ToLower(provider) == strings.ToLower(testContext.Provider) { return true } } return false } </ins> type podCondition func(pod *api.Pod) (bool, error) func waitForPodCondition(c *client.Client, ns, podName, desc string, condition podCondition) error {

I noticed while running e2e tests locally that some tests were apparently being incorrectly skipped: I checked, and this is also true for our Jenkins runs. Currently it looks like and have this bug, though only the former actually skips the test, I think. The issue appears to be that the code inside the top-level in these files is being run before the testContext struct is filled out (and before is even called), and so testContext.Provider is empty. I'm not immediately sure what the proper fix is; I'm also worried that there might be other things broken by the fact that these blocks are called before initialization is complete. We also probably investigate if there is a better way to mark these tests as skipped; right now, there's no indication in the test report (, Jenkins test report, etc) that these tests were skipped.
I merged and which tried to disable these tests for non-{gke,gce} but apparently didn't follow-up and verify that they were still running on our platforms. /cc
Seems easy enough to fix. I'll have a PR out to fix this before Monday.


Question: } } <del> var reinvocationMarkerFixture = &corev1.Pod{ ObjectMeta: metav1.ObjectMeta{ Namespace: "default", Name:      "marker", }, Spec: corev1.PodSpec{ Containers: []v1.Container{{ Name:  "fake-name", Image: "fakeimage", }}, }, </del> <ins> func newReinvocationMarkerFixture(namespace string) *corev1.Pod { return &corev1.Pod{ ObjectMeta: metav1.ObjectMeta{ Namespace: namespace, Name:      "marker", Labels: map[string]string{ "marker": "true", }, }, Spec: corev1.PodSpec{ Containers: []v1.Container{{ Name:  "fake-name", Image: "fakeimage", }}, }, } </ins> }

Which jobs are failing: Which test(s) are failing: Since when has it been failing: Testgrid link: Reason for failure: Anything else we need to know: /sig api-machinery /assign
/cc
want to take a look at the audit event issue first? I can dig in after that if issues remain.
yes, I'm looking at the audit event issue now
actually the failure message suggests that the missing audit log is caused by unexpected admission invocation: I think it's unrelated to auditing Note that there are two tests named "no reinvocation of webhook B when in-tree or prior webhook mutations" :( and from the audit log (), it also suggests that the test is hitting a wrong endpoint
Thanks I'll dig in.


Question: InitContainers: []v1.Container{ { Image: image.GetE2EImage(), <del> Name:  podName, </del> <ins> Name:  "init-container", </ins> Command: []string{ "powershell.exe", "-Command",

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: aks-engine-azure-windows-master-containerd-serial-slow Which test(s) are failing: [sig-windows] [Feature:Windows] Kubelet-Stats [Serial] Kubelet stats collection for Windows nodes when running 10 pods should return within 10 seconds Since when has it been failing: Unknown. Testgrid link: Reason for failure: The test spawns 10 pods with the same pod name, which contains multiple containers with the same container name. Because of this, the test fails. Logs: Anything else we need to know:
/sig windows These SIGs are my best guesses for this issue. Please comment if I am incorrect about one. I am a bot run by vllry.


Question: MUST succeed. It MUST succeed when deleting the Deployment. release: v1.20 file: test/e2e/apps/deployment.go <ins> - testname: Deployment, status sub-resource codename: '[sig-apps] Deployment should validate Deployment Status endpoints [Conformance]' description: When a Deployment is created it MUST succeed. Attempt to read, update and patch its status sub-resource; all mutating sub-resource operations MUST be visible to subsequent reads. release: v1.22 file: test/e2e/apps/deployment.go </ins> - testname: 'PodDisruptionBudget: list and delete collection' codename: '[sig-apps] DisruptionController Listing PodDisruptionBudgets for all namespaces should list and delete a collection of PodDisruptionBudgets [Conformance]'

[X] Test PR: kuberenetes/kubernetes [x] Two weeks soak end date: 1 July 2021 [x] Test promotion PR: kubernetes/kubernetes According to this APIsnoop query, there are still some remaining StatefulSet endpoints which are untested. Create a watch to track deployment events. Create a deployment with a static label. Confirm that the pods are running. Get the deployment status. Parse the response and confirm that the deployment status conditions can be listed. Update the deployment status. Confirm via the watch that the status has been updated. Patch the deployment status. Confirm via the watch that the status has been patched. Using an existing as a template for a new for deployment lifecycle test. This query shows the endpoints hit within a short period of running the e2e test If a test with these calls gets merged, test coverage will go up by 1 point This test is also created with the goal of conformance promotion. /sig apps /sig testing /sig architecture /area conformance
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: rc=1 fi <ins> # ensure all generic features are added in alphabetic order lines=$(git grep 'genericfeatures[.].*:' -- pkg/features/kube_features.go) sorted_lines=$(echo "$lines" | sort -f) if [[ "$lines" != "$sorted_lines" ]]; then echo "Generic features in pkg/features/kube_features.go not sorted" >&2 echo >&2 echo "Expected:" >&2 echo "$sorted_lines" >&2 echo >&2 echo "Got:" >&2 echo "$lines" >&2 rc=1 fi </ins> exit $rc

This avoids drift between Kubernetes components and allows for cluster management tools that pass uniform feature gates to all components. xref:
/assign
Hey I f you stuck anywhere. Please free to involve me in this.


Question: mu.Unlock() } <ins> if !concreteOp.SkipWaitToCompletion { // SkipWaitToCompletion=false indicates this step has waited for the Pods to be scheduled. // So we reset the metrics in global registry; otherwise metrics gathered in this step // will be carried over to next step. legacyregistry.Reset() } </ins> case *churnOp: var namespace string if concreteOp.Namespace != nil {

In the new scheduler performance testing framework, (bool) is provided to benchmark performance of pods created in an "operator": (L11 below) However, the initialized Pods created in L5~L7 seem to be also benchmarked, although it doesn't specify . (please see the reproduce steps below). I think it's b/c once scheduler starts, the associated Prometheus artifacts are also up, and it's a single in-memory datastore. So no matter we start to collect the metric in which step, the prometheus data is already up-to-date. In the above workload config, the final data have benchmarked 1500 pods instead of 1000 pods. I guess it won't work either if running in the pod initialization step or not. Unless we manage a breakpoint at the end of each step, and calculate delta histogram for the step that specifies . Pods created in step without are not benchmarked. Apply the following diff: And then run: You will get an output sth. like: /sig scheduling
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Could you please help confirm if it's an issue?
cc
Unfortunately, does not apply for metrics gathered from . As you mentioned. The default gatherer collects all metrics produced by the scheduler. It's the scheduler itself who knows pod specs and would have to decide which pods will get ignored when updating metrics. The question is if computing metrics for a subset of pods processed by the scheduler is a valid use case.
This is probably what we want. Is something like that possible ?
I think being able to absolutely ignore certain pods for metrics is an overkill.
The use case is to ignore the pods created in steps that don't specify , so as to collect the Pod metrics create at particular step(s) (). Otherwise, the semantics of option is quite misleading. Nope, that's not the intention. We don't need to target certain pods, like using label selector or whatever. Just as I mentioned, measure pods created at particular steps.


Question: mu.Unlock() } <ins> if !concreteOp.SkipWaitToCompletion { // SkipWaitToCompletion=false indicates this step has waited for the Pods to be scheduled. // So we reset the metrics in global registry; otherwise metrics gathered in this step // will be carried over to next step. legacyregistry.Reset() } </ins> case *churnOp: var namespace string if concreteOp.Namespace != nil {

I would do something like the following: if a step has collectMetrics:false, we wait for them to finish scheduling, and then we take a snapshot of the histogram. Then for the next step that has collectMetrics:true, we take the diff with the latest snapshot to report the results. This might not be perfect, as we don't control if the metrics have finished syncing when we take the snapshot. But I think that's fine. Would doing this be possible? Now, if we have steps that run in parallel, one with collectMetrics:false and another with collectMetrics:true, then this doesn't work. Note that I don't know the details of how the gatherer works.
That adheres to what I have in my mind - we can only manage this on client side. I think it's fine. As we are already fetching the "snapshot" when pods get scheduled when equals true. True. An alternative is to design the case to be like: the step with to employ Pods that are all unschedulable, and then specify a custom metric "schedulere2eschedulingdurationseconds[result="unschedulable"]" to target a particular histogram instead of all histograms (default option) inside a histogramVec. Note this can only be possible when and some subsequent changes get in place.
We might just call Histogram.Clear() method which will clear what has been collected so far at the end of phase: Assuming individual phases have no pods in common and each phase can clear metrics without affecting other phases. Though, as said, it will work only for serial case.
WDYT
I'm afraid doesn't work. That was my initial thought as well, but no luck. I guess it's b/c, theoretically, we cannot reset a metric on the server side - Promoetheus' model doesn't support this. But let me know if I miss anything.
The performance tests assume all the metrics all gathered and accessed in the same binary. As long as that's the case, works.


Question: mu.Unlock() } <ins> if !concreteOp.SkipWaitToCompletion { // SkipWaitToCompletion=false indicates this step has waited for the Pods to be scheduled. // So we reset the metrics in global registry; otherwise metrics gathered in this step // will be carried over to next step. legacyregistry.Reset() } </ins> case *churnOp: var namespace string if concreteOp.Namespace != nil {

We already call in the code:
That was my understanding as well, but no luck :( Here is a diff to call in case when "concreteOp.CollectMetrics" is false: <details<summaryDiff:</summary<p</p</detailsAnd below is the output - it's still 1500 (500 init pods plus 1000 measured pods) instead of 1000 being benchmarked: <details<summaryOutput:</summary<p</p</details
Also, if you run the tests and together, the result proves doesn't work as expected:
But I do find a way to fix this: .


Question: } func (az *Cloud) getIPForMachine(nodeName types.NodeName) (string, error) { <ins> az.operationPollRateLimiter.Accept() </ins> machine, exists, err := az.getVirtualMachine(nodeName) if !exists { return "", cloudprovider.InstanceNotFound

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind feature What happened: In certain large cluster scenarios we would like to anticipate kubelet behavior hitting request limits, and backoff in response. E.g., What you expected to happen: This is a request for add'l backoff coverage, so nothing surprising. How to reproduce it (as minimally and precisely as possible): Build a cluster with as many nodes as your subscription will support on azure until kubelet behavior contributes to backend API request limits. Environment: Kubernetes version (use ): and Cloud provider or hardware configuration**: azure OS (e.g. from /etc/os-release): Ubuntu 16.04.2 LTS (Xenial Xerus) Kernel (e.g. ): Linux k8s-pool7--7 4.4.0-83-generic -Ubuntu SMP Mon Jun 26 17:54:43 UTC 2017 x8664 x8664 x86_64 GNU/Linux Install tools: acs-engine Others:
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig azure


Question: cmd.Flags().DurationVar(&o.drainer.Timeout, "timeout", o.drainer.Timeout, "The length of time to wait before giving up, zero means infinite") cmd.Flags().StringVarP(&o.drainer.Selector, "selector", "l", o.drainer.Selector, "Selector (label query) to filter on") cmd.Flags().StringVarP(&o.drainer.PodSelector, "pod-selector", "", o.drainer.PodSelector, "Label selector to filter pods on the node") <ins> cmd.Flags().BoolVar(&o.drainer.DisableEviction, "disable-eviction", o.drainer.DisableEviction, "Force drain to use delete, even if eviction is supported. This will bypass checking PodDisruptionBudgets, use with caution.") </ins> cmdutil.AddDryRunFlag(cmd) return cmd

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Allow to have an additional flag (e.g. or ) to allow a to drain pods off a node, even if they violate PodDisruptionBudgets Why is this needed: There are times where you want to drain pods off a node even if violating a PDB. For example, if you are running a one-worker node kubernetes cluster (not minikube), and you want to upgrade it. Or, like me, you are running a test cluster and want to scale down to zero nodes when you're not going to be using the cluster for a while in order to save costs. If that worker node has a deployment with a PDB, then will get stuck as it cannot get rid of the pod while respecting the PDB, which is expected; but in this case, it is a hindrance. Adding the switch will allow users to explicitly acknowledge that they want to ignore the PDBs (and accept the corresponding downtime for an application/service)
/sig node
+1
+1 Our company offers Kubernetes as PaaS. We manage Kubernetes, including IaaS OS updates and if a customer does not specify the PDB correctly, we are not able to drain the node for the OS updates. A flag would be werry helpfull for our use case.
+1
/active
I have created to address this. This is a pretty simple implementation, but I think it achieves the desired results. A suitable workflow would be: 1) Configure you component to use drain as normal 2) After some timeout, run drain with --disable-eviction option.
I'll wait until the PR is merged and released before testing, but if I understand right, it would require me do this, right? If so, I think it would be better if we did it a different way and have one call, something like this: Where implies that if the drain is still running after this time, switch to the mode without having to call kubectl again
I think that would require a sizeable refactor or add a great deal of complexity to the code, the same could be achieved with In practical terms, I'm focused on automatic node remediation via destroying the node and getting a new one. The timeout would be very large, the drain library is blocking, so I want it to fail quickly and requeue.


Question: cmd.Flags().DurationVar(&o.drainer.Timeout, "timeout", o.drainer.Timeout, "The length of time to wait before giving up, zero means infinite") cmd.Flags().StringVarP(&o.drainer.Selector, "selector", "l", o.drainer.Selector, "Selector (label query) to filter on") cmd.Flags().StringVarP(&o.drainer.PodSelector, "pod-selector", "", o.drainer.PodSelector, "Label selector to filter pods on the node") <ins> cmd.Flags().BoolVar(&o.drainer.DisableEviction, "disable-eviction", o.drainer.DisableEviction, "Force drain to use delete, even if eviction is supported. This will bypass checking PodDisruptionBudgets, use with caution.") </ins> cmdutil.AddDryRunFlag(cmd) return cmd

I'm going to be performing the calculation of when it's appropriate to evict/delete outside of the drain library, and I suspect most will be as well, so there's not really a great incentive to do it in the drain library itself IMO.
That's all good, I have no issue using the flag as long as does error out and return a non-zero code as expected (there's been a few times where it seems to error out, but still return 0)


Question: measurePods: 1000 - name: SchedulingMigratedInTreePVs <del> featureGates: CSIMigrationAWS: true </del> workloadTemplate: - opcode: createNodes countParam: $initNodes

https://perf- /kind failing-test <!-- If this is a flake, please add: /kind flake --/sig scheduling <!-- Please assign a SIG using: /sig SIG-NAME --
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc
The goleak check was in
cc
cc
At least didn't fail with a goroutine leak. It failed because of:
Good to see it's not a regression. Let me raise a PR to see whether is the only culprit.
Even if it's not the only culprit, I wouldn't think this is a regression, but rather a consequence of the fact that we didn't use goleak before.
it's just coincidental gets merged at the same timing of the goleak check PR. In other words, the scheduler perf test would also fail if the goleak PR didn't go in.
This can be solved by removing this line


Question: import ( goflag "flag" <del> "fmt" </del> "math/rand" "os" "time"

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: kubectl version --client=yes command is giving the following error message. Error message is repeated at the start and the end and also containing programming jargon Error: invalid argument "yes" for "--client" flag: strconv.ParseBool: parsing "yes": invalid syntax Examples: Print the client and server versions for the current context kubectl version Options: --client=false: Client version only (no server required). -o, --output='': One of 'yaml' or 'json'. --short=false: Print just the version number. Usage: kubectl version [flags] [options] Use "kubectl options" for a list of global command-line options (applies to all commands). invalid argument "yes" for "--client" flag: strconv.ParseBool: parsing "yes": invalid syntax What you expected to happen: The error message can be improved with one of the two variants invalid argument "yes" for "--client" flag Examples: Print the client and server versions for the current context kubectl version Options: --client=false: Client version only (no server required). -o, --output='': One of 'yaml' or 'json'. --short=false: Print just the version number. Usage: kubectl version [flags] [options] Use "kubectl options" for a list of global command-line options (applies to all commands). (or) Invalid argument "yes" for "--client" flag How to reproduce it (as minimally and precisely as possible): Run the command kubectl version --client=yes Anything else we need to know?: Environment: Kubernetes version (use ): 1.15 Cloud provider or hardware configuration: AWS OS (e.g: ): Amazon Linux Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig cli
Hi Thanks for posting this issue! It turns out that this duplicate output has been in kubectl, and all the other cli cmds, since . This should be fixed by , and hopefully we can cherry pick it back to at least .


Question: //CredentialsManager credentialManager *SecretCredentialManager <ins> nodeLister corelisters.NodeLister nodeGetter coreclients.NodesGetter </ins> // Mutexes registeredNodesLock   sync.RWMutex nodeInfoLock          sync.RWMutex

I am enabling CSI migration on a vSphere cluster. A node was fully drained, all its Pods are deleted, but just before KCM detaches in-tree vSphere volume volumes from it, KCM restarts. A new KCM starts, sees attached volumes in that are not needed by any Pod on the node and calls vSphere in-tree volume plugin . The in-tree vSphere volume plugin + cloud provider marks the volume as detached right away, without reaching to vCenter API. The volume remains attached forever. I don't think it's related to CSI migration, IMO a regular node drain + KCM restart at the same time can cause the same issue. KCM logs: The in-tree cloud provider has a and from it. It apparently has not processed all nodes at the time A/D controller tries to detach the node. The in-tree vSphere volume plugin + cloud provider detaches the volume in vCenter. That's really tricky, it's very hard / impossible to reproduce on a real cluster. I had to inject errors to Detach() in-tree volume call, so they fail until KCM is restarted. Newly started KCM would detach the volume just fine (no error would be injected there), but marks the volume as detached right away. For the record (and search-ability), the CSI driver that tries to attach the volume to another node complains either: Or: cc It could be somewhat related to , however, not all cases of can be related to KCM restart. As I mentioned above, it's very hard/impossible to synchronize KCM restart with draining a node in a real cluster. <details</details<detailsvSphere 7.0.3 </details<details</details<detailsOpenShift </details<detailsCRI-O </details<details</details
/sig storage
Tried just moving a Pod between nodes without CSI migration involved + restarting KCM when the volume is still attached to the old node. In-tree volume plugin on the new node says:
/assign
What CSI driver version and K8s version are you testing with?
Kubernetes is 1.25 and CSI driver is 2.7, still, I think it's not releated to the CSI driver, it's KCM who marks the in-tree volume as detached due to not populated node cache.


Question: //CredentialsManager credentialManager *SecretCredentialManager <ins> nodeLister corelisters.NodeLister nodeGetter coreclients.NodesGetter </ins> // Mutexes registeredNodesLock   sync.RWMutex nodeInfoLock          sync.RWMutex

/assign
/triage accepted
I found an easier way how to reproduce this without extra code in KCM: a StatefulSet with 1 Pod that uses 1 PVC. the node where the Pod runs. down kube-controller-manager to zero (not really possible for me) or kill all kube-controller-manager Pods. This will cause new KCM processes to wait for leader election. In my env. this gives me ~30 seconds for the next step. Delete the Pod. A new one will be scheduled on a different node (because the current node is cordoned) + there is no KCM running at this time that would detach the volume from the old node. KCM wins its leader election, sees a volume attached to a node where it is not needed -it calls of the volume plugin. VSphere cloud provider does not have its caches populated yet (= this bug) -the volume is not detached, but returns success.
dangling attach does not help here. The volume is still attached to the old node even after ~10 minutes. fails with and not with . I can see there is a lot of dangling attach code in the cloud provider, but I did not debug further.


Question: klog.Errorf("cannot convert to *v1.Pod: %v", t) return } <ins> klog.V(3).Infof("delete event for scheduled pod %s/%s ", pod.Namespace, pod.Name) </ins> // NOTE: Updates must be written to scheduler cache before invalidating // equivalence cache, because we could snapshot equivalence cache after the // invalidation and then snapshot the cache itself. If the cache is

This helps debugging scheduler's behavior, currently we don't have logs that indicate when the scheduler first saw a pod or a node. The log level should be v=3. /sig scheduling /cc
Do you mean add logging for callbacks in ?
yes, at least for add and delete events
Sure, I can also give this pr, before tomorrow
/assign
We actually have one v=2 log line for a Node being to the cache's tree: But it's probably good to keep this one.


Question: <ins> /* Copyright 2014 Google Inc. All rights reserved. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package cloudcfg import ( "bytes" "encoding/json" "reflect" "testing" "gopkg.in/v1/yaml" ) func TestYAMLPrinterPrint(t *testing.T) { type testStruct struct { Key        string         `yaml:"Key" json:"Key"` Map        map[string]int `yaml:"Map" json:"Map"` StringList []string       `yaml:"StringList" json:"StringList"` IntList    []int          `yaml:"IntList" json:"IntList"` } testData := testStruct{ "testValue", map[string]int{"TestSubkey": 1}, []string{"a", "b", "c"}, []int{1, 2, 3}, } printer := &YAMLPrinter{} buf := bytes.NewBuffer([]byte{}) err := printer.Print([]byte("invalidJSON"), buf) if err == nil { t.Error("Error: didn't fail on invalid JSON data") } jTestData, err := json.Marshal(&testData) if err != nil { t.Fatal("Unexpected error: couldn't marshal test data") } err = printer.Print(jTestData, buf) if err != nil { t.Fatal(err) } var poutput testStruct err = yaml.Unmarshal(buf.Bytes(), &poutput) if err != nil { t.Fatal(err) } if !reflect.DeepEqual(testData, poutput) { t.Error("Test data and unmarshaled data are not equal") } } </ins>

When trying to use against a containerized kubernetes cluster (i.e. deployed following ) I get the following errors: This is simply because the command is missing in the hyperkube container image. I will be creating a PR with a fix shortly.
mayukh istioctl manifest apply --set profile=demo Detected that your cluster does not support third party JWT authentication. Falling back to less secure first party JWT. See for details. Istio core installed Istiod installed Egress gateways installed Ingress gateways installed Addons installed Installation complete mayukh istio version istio: command not found mayukh kubectl -n istio-system get svc prometheus NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE prometheus ClusterIP 10.101.216.106 <none9090/TCP 20m mayukh kubectl -n istio-system get svc grafana NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE grafana ClusterIP 10.97.41.187 <none3000/TCP 21m mayukh -n istio-system get pod -l app=grafana -o jsonpath='{.items[0]}') 3000:3000 & [1] mayukh Forwarding from 127.0.0.1:3000 -3000 Forwarding from [::1]:3000 -3000 Handling connection for 3000 Handling connection for 3000 E0601 00:12:20. ] an error occurred forwarding 3000 -3000: error forwarding port 3000 to pod 6277818d8d58b0b6d52d07c776f1e930860a60466790efbaf9460a380ea497ad, uid : unable to do port forwarding: socat not found E0601 00:12:20. ] an error occurred forwarding 3000 -3000: error forwarding port 3000 to pod 6277818d8d58b0b6d52d07c776f1e930860a60466790efbaf9460a380ea497ad, uid : unable to do port forwarding: socat not found Handling connection for 3000 E0601 00:12:20. ]


Question: <ins> /* Copyright 2014 Google Inc. All rights reserved. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package cloudcfg import ( "bytes" "encoding/json" "reflect" "testing" "gopkg.in/v1/yaml" ) func TestYAMLPrinterPrint(t *testing.T) { type testStruct struct { Key        string         `yaml:"Key" json:"Key"` Map        map[string]int `yaml:"Map" json:"Map"` StringList []string       `yaml:"StringList" json:"StringList"` IntList    []int          `yaml:"IntList" json:"IntList"` } testData := testStruct{ "testValue", map[string]int{"TestSubkey": 1}, []string{"a", "b", "c"}, []int{1, 2, 3}, } printer := &YAMLPrinter{} buf := bytes.NewBuffer([]byte{}) err := printer.Print([]byte("invalidJSON"), buf) if err == nil { t.Error("Error: didn't fail on invalid JSON data") } jTestData, err := json.Marshal(&testData) if err != nil { t.Fatal("Unexpected error: couldn't marshal test data") } err = printer.Print(jTestData, buf) if err != nil { t.Fatal(err) } var poutput testStruct err = yaml.Unmarshal(buf.Bytes(), &poutput) if err != nil { t.Fatal(err) } if !reflect.DeepEqual(testData, poutput) { t.Error("Test data and unmarshaled data are not equal") } } </ins>

an error occurred forwarding 3000 -3000: error forwarding port 3000 to pod 6277818d8d58b0b6d52d07c776f1e930860a60466790efbaf9460a380ea497ad, uid : unable to do port forwarding: socat not found Handling connection for 3000 Handling connection for 3000


Question: // address is required. if len(spec.ServerAddressByClientCIDRs) == 0 { allErrs = append(allErrs, field.Required(fieldPath.Child("serverAddressByClientCIDRs"), "")) <ins> } else { for i, address := range spec.ServerAddressByClientCIDRs { idxPath := fieldPath.Child("serverAddressByClientCIDRs").Index(i) if len(address.ClientCIDR) > 0 { if _, _, err := net.ParseCIDR(address.ClientCIDR); err != nil { allErrs = append(allErrs, field.Invalid(idxPath.Child("clientCIDR"), address.ClientCIDR, fmt.Sprintf("must be a valid CIDR: %v", err))) } } } </ins> } return allErrs }

Right now it is possible to add a cluster for which it won't be possible to create a client. For example client-related code does: while validation checks only: cc:
switching this to 1.6 as it's too late for 1.5. ok? (please switch it right back if you disagree)
Not a release blocker for 1.7.
are you interested in fixing this?
of course, will do that.
[MILESTONENOTIFIER] Milestone Removed Important: This issue was missing labels required for the v1.8 milestone for more than 7 days: kind: Must specify exactly one of [, , ]. priority: Must specify exactly one of [, , ]. Removing it from the milestone. <detailsAdditional instructions available <a href=""here</aThe commands available for adding these labels are documented <a href=""here</a</details


Question: <del> // +build !cgo !linux </del> <ins> // +build !linux </ins> /* Copyright 2015 The Kubernetes Authors.

In cluster/saltbase/salt/kube-proxy/kube-proxy.manifest, we have this: But kube-proxy also has a flag. Is there a reason to use the approach instead of using the flag? cc as you wrote the code, in
/cc who's working on moving kube-proxy to a DaemonSet
In general, oomscoreadj is not expected to be directly updated by pods and kubelet is expected to manage it. If kube-proxy runs in the Guaranteed QoS class, then it's OOM score adj will be set to a low number. Until that happens, kube-proxy process may want to be shielded from OOM kills to an extent (this matters only when kube-proxy's memory usage memory request) Meanwhile, it does not make sense to have two means to configure the same thing. I'll let decide which option he prefers and sunset the other option.
Thanks for clarifying. At this point, I prefer to use the built-in flag instead of relying on composing bash commands. Sent for this.


Question: CPU utilization for the pod will not be defined and the autoscaler will not take any action. Further details of the autoscaling algorithm are given [here](../design/horizontal-pod-autoscaler.md#autoscaling-algorithm). <ins> Autoscaler uses heapster to collect CPU utilization. Therefore, it is required to deploy heapster monitoring in your cluster for autoscaling to work. </ins> Autoscaler accesses corresponding replication controller or deployment by scale sub-resource. Scale is an interface which allows to dynamically set the number of replicas and to learn the current state of them. More details on scale sub-resource can be found [here](../design/horizontal-pod-autoscaler.md#scale-subresource).

The heapster needs to be installed if end user want to use autoscaler feature, we should mention this in document.


Question: func prepareSubpathTarget(mounter Interface, subpath Subpath) (bool, string, error) { // Early check for already bind-mounted subpath. bindPathTarget := getSubpathBindTarget(subpath) <del> notMount, err := IsNotMountPoint(mounter, bindPathTarget) </del> <ins> notMount, err := mounter.IsNotMountPoint(bindPathTarget) </ins> if err != nil { if !os.IsNotExist(err) { return false, "", fmt.Errorf("error checking path %s for mount: %s", bindPathTarget, err)

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: There's , and . The former is used by local and nfs volume plugins, while the latter is used by , which is also used by local and nfs volume plugins. What you expected to happen: Can we consolidate these? This is really confusing and could easily be the source of bugs. <!-- DO NOT EDIT BELOW THIS LINE --/kind cleanup
cc
/assign I'd like to clean up code.


Question: [[constraint]] name = "github.com/modern-go/reflect2" <del> version = "1.0.0" </del> <ins> version = "1.0.1" </ins>

is a library yet is is instead of release tag. Please consider vendoring by tag whenever possible. Thanks.
We vendored it that way because the commit that fixed a bug (which we needed to be fixed and vendored back in) was This was merged on June 13. The release 1.1.4 () was released on July 1. If we want to update it to 1.1.4, I believe it should be fine. Between the commit we have vendored right now and the latest release, there are only three commits: adds tests merges (?) the case-sensitive fix - I'm not really sure what this commit is doing. fixes renaming when naming strategy is set (looks fine). Also, there are also no licensing changes. opinions?
/sig api-machinery
+1 to what said, it was a fire drill and 1.11.0 release was in trouble. Though in hind sight, we should have opened a issue to track that. something better to do next time. cc
I'm indifferent to updating this. I'm okay with moving up if it doesn't break anything, or I'm fine with leaving things as is. We don't require deps to match semver. We can version out any commit in a deps tree, as we rely on our vendored copy.
It is always a good idea to match semver and it should be a default policy. Even if it doesn't matter to you, it should, because it is a best practice. Moreover downstream it matters much more as dependencies may be un-vendored in favour of tested re-usable packages.
/cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Someone please take care of this or bot will eventually close this valid issue without action...
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or .


Question: [[constraint]] name = "github.com/modern-go/reflect2" <del> version = "1.0.0" </del> <ins> version = "1.0.1" </ins>

/lifecycle rotten
/lifecycle rotten
/remove-lifecycle rotten PR open (thanks,
oops! sorry, I meant to remove it. thanks!
Thanks.
please do feel free to open issues/prs of this sort early in the release cycle, so we can work through things needed well before the milestones/freeze etc. we are getting some heat about updating dependencies and we kind of settled on some guidelines here: if you think that needs updating, we could definitely debate that as well :) (on a PR against that doc)
it have nothing to do with release cycle or justification for upgrade if/when dependency has been vendored incorrectly. It is basically a vendoring bug. No debate is necessary although that kind of issues usually meet some resistance and even hostility here... Bugs are filed when they are discovered, regardless of release cycle. It would be wrong to sit on the bug without reporting it in order to synchronise with release cycle.


Question: "description": "ServicePort contains information on service's port.", "properties": { "appProtocol": { <del> "description": "The application protocol for this port. This field follows standard Kubernetes label syntax. Un-prefixed names are reserved for IANA standard service names (as per RFC-6335 and http://www.iana.org/assignments/service-names). Non-standard protocols should use prefixed names such as mycompany.com/my-custom-protocol.", </del> <ins> "description": "The application protocol for this port. This field follows standard Kubernetes label syntax. Un-prefixed names are reserved for IANA standard service names (as per RFC-6335 and https://www.iana.org/assignments/service-names). Non-standard protocols should use prefixed names such as mycompany.com/my-custom-protocol.", </ins> "type": "string" }, "name": {

We should make the change in PR API reference links to should use HTTPS, not HTTP Visit h and look at the IANA hyperlinks Once this fix is made, a PR to is needed to update the docs (by autogenerating them). This typically would be deferred until the next Kubernetes release. v1.23 not applicable not applicable not applicable not applicable not applicable
/sig docs
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I opened this issue so that someone else can open a new PR that is equivalent to .
/retitle Update links to IANA for Service Name and Transport Protocol Port Number Registry
I would like to work on this issue. May I assign it to myself?
Anyone - including you - is welcome to work on this issue! So long as you've you can open a pull request to propose a fix.
Thanks
/assign
I have made the PR. I hope this works. If not then will be happy to do the necessary changes.
Had to close the previous PR because of issues, here is the new one I made.


Question: CreateTCPLoadBalancer(name, region string, externalIP net.IP, ports []int, hosts []string, affinityType api.AffinityType) (string, error) // UpdateTCPLoadBalancer updates hosts under the specified load balancer. UpdateTCPLoadBalancer(name, region string, hosts []string) error <del> // DeleteTCPLoadBalancer deletes a specified load balancer. DeleteTCPLoadBalancer(name, region string) error </del> <ins> // EnsureTCPLoadBalancerDeleted deletes the specified load balancer if it // exists, returning nil if the load balancer specified either didn't exist or // was successfully deleted. // This construction is useful because many cloud providers' load balancers // have multiple underlying components, meaning a Get could say that the LB // doesn't exist even if some part of it is still laying around. EnsureTCPLoadBalancerDeleted(name, region string) error </ins> } // Instances is an abstract, pluggable interface for sets of instances.

All of our external load balancer tests failed for the better part of a day due to the GCE project that our tests run in hitting its quota for number of target pools. While the target pools have been deleted, the fact that we hit our quota indicates that there must be a leak somewhere, and we need to plug it.
For reference, the previous issue was
Could the leakage be related to ? The quota exhaustion occurred about 22 hrs after that.
I'd guess that this is due to GetTCPLoadBalancer returning that the load balancer doesn't exist if the forwarding rule is gone, ignoring the target pool, while the service controller considers its deleting done if the cloud provider says that the load balancer doesn't exist. I that logic to the cloud provider because the other implementations of DeleteTCPLoadBalancer weren't/won't be idempotent in the sense of returning ok if the LB doesn't exist. To fix it, we'll either need to make sure all Delete implementations are idempotent and remove the Get call in the controller, or add a method along the lines of IsTCPLoadBalancerDeleted to ask the cloud provider whether all pieces of it have been completely cleaned up. We can't make GetTCPLoadBalancer check that because it's relied on to tell us whether or not the balancer is fully created.
Copying over my commentary from on why the GKE e2e tests are leaking so many forwarding rules and target pools vs the GCE tests:
Is there a way for the services test to wait for the service controller to clean up the external resources before finishing (without just doing a sleep)?
Not without polling GCE directly for whether the resources still exist, although that may be worth it as a workaround. What would be perfect is that clean-up-all-cloud-provider-resources API method that we talked about a couple months ago.
Can we not perhaps use GCE namespaces or projects as a cheap way of getting this here?
I don't believe so. We can't use projects due to the fact that projects can't be programatically created (they're intended to be long-lived). I'm not aware of anything along the lines of a GCE namespace other than networks, which won't work because target pools and forwarding rules unfortunately don't seem to be scoped to networks.


Question: CreateTCPLoadBalancer(name, region string, externalIP net.IP, ports []int, hosts []string, affinityType api.AffinityType) (string, error) // UpdateTCPLoadBalancer updates hosts under the specified load balancer. UpdateTCPLoadBalancer(name, region string, hosts []string) error <del> // DeleteTCPLoadBalancer deletes a specified load balancer. DeleteTCPLoadBalancer(name, region string) error </del> <ins> // EnsureTCPLoadBalancerDeleted deletes the specified load balancer if it // exists, returning nil if the load balancer specified either didn't exist or // was successfully deleted. // This construction is useful because many cloud providers' load balancers // have multiple underlying components, meaning a Get could say that the LB // doesn't exist even if some part of it is still laying around. EnsureTCPLoadBalancerDeleted(name, region string) error </ins> } // Instances is an abstract, pluggable interface for sets of instances.

The target pool leak in the code might be fixed, but the forwarding rule + target pool leak from deleting GKE clusters too quickly is certainly still happening. Reopening for the sake of that half of the issue.
Can we just add a short sleep to jenkins after the last test has run before it deletes anything?
Yes, you could add a sleep to Ginkgo's AfterEachSuite(): But I'd be concerned about what the upper bound on the sleep period is. Can we get the tests and code in sync with a 1 minute latency here? (e.g. if the controller garbage collects every 60 seconds, then the test only needs to sleep for 60 sec?) And add a BEEG comment to the test's sleep config noting that increasing the sleep time will not improve things, otherwise I'm pretty sure that someone noting flakes in this test in future will think "maybe I should just increase the sleep period a little bit."
Actually, thinking about this a little more, it would be better to put the "sleep" logic in the k8s master teardown procedure (i.e. not in the e2e tests or in Jenkins), because otherwise users are going to run into the same resource leakage problems as we are. So master deletion blocks on cleanup of backend cloud resources.
There's no way to block master deletion from the master itself. The test framework (or user) just pulls the rug out from underneath the master. See
Robby's right -- is the real solution to this, but until we have that the best we can do is to have the wait happen elsewhere. The options seem to be: GKE itself Ginkgo's post-testing logic Jenkins' teardown logic I don't like putting it in GKE, because arbitrarily slowing down our API makes for a poor experience for the majority of people that don't still have resources being deleted. I'm not eager to put it in Gingko either, since won't that sleep also take effect whenever anyone runs an e2e test locally against a dev cluster? Putting it in the Jenkins logic makes the most sense to me. I'll send a PR and cc you.
We may also want to change the kube-down logic to bring down the master and then nodes in parallel to both speed it up and more closely resemble GKE's teardown logic.
I've deleted the old forwarding rules and target pools from the project.


Question: CreateTCPLoadBalancer(name, region string, externalIP net.IP, ports []int, hosts []string, affinityType api.AffinityType) (string, error) // UpdateTCPLoadBalancer updates hosts under the specified load balancer. UpdateTCPLoadBalancer(name, region string, hosts []string) error <del> // DeleteTCPLoadBalancer deletes a specified load balancer. DeleteTCPLoadBalancer(name, region string) error </del> <ins> // EnsureTCPLoadBalancerDeleted deletes the specified load balancer if it // exists, returning nil if the load balancer specified either didn't exist or // was successfully deleted. // This construction is useful because many cloud providers' load balancers // have multiple underlying components, meaning a Get could say that the LB // doesn't exist even if some part of it is still laying around. EnsureTCPLoadBalancerDeleted(name, region string) error </ins> } // Instances is an abstract, pluggable interface for sets of instances.

I'll periodically check to see whether anything is leaked after this point. I apologize if the deletions affected any running tests - everything looked to only be from GKE and parallel e2e runs.


Question: command: - /bin/sh - -c <del> - echo -998 > /proc/$$$/oom_score_adj && kube-proxy {{kubeconfig}} {{cluster_cidr}} --resource-container="" {{params}} 1>>/var/log/kube-proxy.log 2>&1 </del> <ins> - kube-proxy {{kubeconfig}} {{cluster_cidr}} --resource-container="" --oom-score-adj=-998 {{params}} 1>>/var/log/kube-proxy.log 2>&1 </ins> {{container_env}} {{kube_cache_mutation_detector_env_name}} {{kube_cache_mutation_detector_env_value}}


Question: }) /* <del> Testname: allowPrivilegeEscalation=false. Description: Configuring the allowPrivilegeEscalation to false, does not allow the privilege escalation operation. A container is configured with allowPrivilegeEscalation=false and a given uid (1000) which is not 0. When the container is run, the container is run using uid=1000. This test is marked LinuxOnly since Windows does not support running as UID / GID, or privilege escalation. </del> <ins> Release : v1.15 Testname: Security Context, allowPrivilegeEscalation=false. Description: Configuring the allowPrivilegeEscalation to false, does not allow the privilege escalation operation. A container is configured with allowPrivilegeEscalation=false and a given uid (1000) which is not 0. When the container is run, container's output MUST match with expected output verifying container ran with given uid i.e. uid=1000. [LinuxOnly]: This test is marked LinuxOnly since Windows does not support running as UID / GID, or privilege escalation. </ins> */ <del> It("should not allow privilege escalation when false [LinuxOnly] [NodeConformance]", func() { </del> <ins> framework.ConformanceIt("should not allow privilege escalation when false [LinuxOnly] [NodeConformance]", func() { </ins> podName := "alpine-nnp-false-" + string(uuid.NewUUID()) apeFalse := false if err := createAndMatchOutput(podName, "Effective uid: 1000", &apeFalse, 1000); err != nil {

/kind cleanup /area conformance /area test /sig node /sig testing /sig architecture /assign /assign We have a number of tests that only have the tag that may make sense as tests, we should identify whether these tests cases meet (or could meet) the . We should open PR's to promote these if so. I would advise starting with the test/common cases
GitHub didn't allow me to assign the following users: mgdevstack, mkurylec. Note that only and repo collaborators can be assigned. For more information please see <detailsIn response to : /kind cleanup /area conformance /area test /sig node /sig testing /sig architecture /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
The tests in test/e2enode don't actually get run as part of Conformance as near as I can tell. If we want to run them as part of conformance, they need to be migrated into test/e2e/common. Some of the tests in e2enode may not be able to be ported over because they assume a level access we don't have in a conformance environment.
regarding this testcase: [sig-storage] HostPath should support r/w [NodeConformance]: It uses hostPath and privileged=true, to read/write from /tmp folder. Which would you suggest is the best orientation to overcome the privilege mode setting and allow its promotion?
/assign
Regarding , the suite runs the test only if the kubelet configuration was required to have the QOS (cgroups-per-qos) enabled. This configuration for the kubelet is selected at the time the cluster is set up. Looking for where this might be configured found: where is set to true. For other settings it is also configured to true. where is set to true. BUT also found where it is mentioned that the flag is no longer available. Questions: would it be ok to promote this test to be conformance, even it can be skipped based on configuration of the cluster? And not supported in the containerized kubelet?
I would consider cgroups to be an implementation detail that Conformance shouldn't be verifying
It uses hostPath and privileged=true, to read/write from /tmp folder.


Question: }) /* <del> Testname: allowPrivilegeEscalation=false. Description: Configuring the allowPrivilegeEscalation to false, does not allow the privilege escalation operation. A container is configured with allowPrivilegeEscalation=false and a given uid (1000) which is not 0. When the container is run, the container is run using uid=1000. This test is marked LinuxOnly since Windows does not support running as UID / GID, or privilege escalation. </del> <ins> Release : v1.15 Testname: Security Context, allowPrivilegeEscalation=false. Description: Configuring the allowPrivilegeEscalation to false, does not allow the privilege escalation operation. A container is configured with allowPrivilegeEscalation=false and a given uid (1000) which is not 0. When the container is run, container's output MUST match with expected output verifying container ran with given uid i.e. uid=1000. [LinuxOnly]: This test is marked LinuxOnly since Windows does not support running as UID / GID, or privilege escalation. </ins> */ <del> It("should not allow privilege escalation when false [LinuxOnly] [NodeConformance]", func() { </del> <ins> framework.ConformanceIt("should not allow privilege escalation when false [LinuxOnly] [NodeConformance]", func() { </ins> podName := "alpine-nnp-false-" + string(uuid.NewUUID()) apeFalse := false if err := createAndMatchOutput(podName, "Effective uid: 1000", &apeFalse, 1000); err != nil {

Which would you suggest is the best orientation to overcome the privilege mode setting and allow its promotion? Michelle Au If my understanding is correct, on some platforms, writing to /tmp requires privileged. I don't see a way to to remove that requirement.
If my understanding is correct, on some platforms, writing to /tmp requires privileged. I don't see a way to to remove that requirement. Yeah writing to host doesn't seem like a thing we can expect of conformance tests. I wonder if it makes sense to tag this test with something to denote the requirement, ie: I see some tests in test/e2e_node use
The hostpath tests in already have a comment explaining this. Maybe the comment should be updated to be more clear that that is the reason why we can't add it to conformance.
That's /tmp on the host or in the container filesystem?
/tmp on the host
ping what else remains to be done before we close this out? I am guessing that we want to close , is there anything beyond that?
Once tests under (issue- ) and are promoted we are good to close this issue. Short update for reference- None of [NodeConformance] tests from "test/e2e_node" seems eligible for [Conformance] promotion at this moment due to node level dependencies. Remaining [NodeConformance] tests from "test/e2e" has following status - Waiting for to be closed and then below tests would be promoted to Conformance Needs to be Promoted Can not be promoted to Conformance as //Slow by design Can not be promoted due to Node level dependency
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
/lifecycle frozen
/assign


Question: t.Run(test.name, func(t *testing.T) { serviceName := types.NamespacedName{Namespace: "default", Name: "myservice"} <del> sgList, err := c.buildELBSecurityGroupList(serviceName, "aid", test.extraSGsAnnotation) </del> <ins> sgList, err := c.buildELBSecurityGroupList(serviceName, "aid", test.annotations) </ins> assert.NoError(t, err, "buildELBSecurityGroupList failed") extraSGs := sgList[1:] assert.True(t, sets.NewString(test.expectedSGs...).Equal(sets.NewString(extraSGs...)),

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind feature /sig aws What happened: In k8s 1.7, when creating a service of type LoadBalancer, K8S creates the ELB and a Security Group for it. For these resources, it adds a tag, constructed dynamically, containing the cluster id: "${cluster_id}" with the value "true" What you expected to happen: We'd like to have also a statically named tag , as it was in 1.5 ("KubernetesCluster") with the value of the cluster id. How to reproduce it (as minimally and precisely as possible): Running on AWS cloud provider, create a service of type LoadBalancer. Anything else we need to know?: The reason for what we expect is that we have tight security constraints that require us to fine tune the IAM policies for the Instance profiles that the controllers use. With a statically named tag we were able, in 1.5, to restrict the permissions in the IAM policy to act only on the k8s managed resources (ELBs, security groups). Here is the relevant part of the IAM policy: With a dynamically named tag we are not able to do this any more as AWS IAM doesn't support something like: We are sharing the Instance Profiles among different k8s clusters, hence the "" in the value of the tag. Environment: Kubernetes version (use ):Cloud provider or hardware configuration*: AWS OS (e.g. from /etc/os-release):Kernel (e.g. ):Install tools: kube-aws Others:
Reiterating the mentions to trigger a notification: <detailsIn response to : <!-- This form is for bug reports and feature requests ONLY! -- /sig aws In k8s 1.7, when creating a service of type LoadBalancer, K8S creates the ELB and a Security Group for it. For these resources, it adds a tag, constructed dynamically, containing the cluster id: "${cluster_id}" with the value "true" We'd like to have also a statically named tag , as it was in 1.5 ("KubernetesCluster") with the value of the cluster id. Running on AWS cloud provider, create a service of type LoadBalancer.


Question: t.Run(test.name, func(t *testing.T) { serviceName := types.NamespacedName{Namespace: "default", Name: "myservice"} <del> sgList, err := c.buildELBSecurityGroupList(serviceName, "aid", test.extraSGsAnnotation) </del> <ins> sgList, err := c.buildELBSecurityGroupList(serviceName, "aid", test.annotations) </ins> assert.NoError(t, err, "buildELBSecurityGroupList failed") extraSGs := sgList[1:] assert.True(t, sets.NewString(test.expectedSGs...).Equal(sets.NewString(extraSGs...)),

The reason for what we expect is that we have tight security constraints that require us to fine tune the IAM policies for the Instance profiles that the controllers use. With a statically named tag we were able, in 1.5, to restrict the permissions in the IAM policy to act only on the k8s managed resources (ELBs, security groups). Here is the relevant part of the IAM policy: Environment: Kubernetes version (use ):OS (e.g. from /etc/os-release):Others: </details
I believe they are moving away from the tag since there are many resources that can be shared by multiple clusters. could you use an annotation?
We removed the condition in roles in the end. Thank you for your suggestion.
That annotation affects the ELB but not any security groups that are created. can you please reopen this? You reported a real issue, and even if it no longer affects you, it needs to be tracked and fixed.
Indeed, this might be a feature that someone else would need.
In fact, this is a feature I need!
Food for thought: should we propagate the tags on the ELB to the SG, or should we introduce an annotation for the SG specifically?


Question: // are set to 0. func completeRollingUpdate(set *apps.StatefulSet, status *apps.StatefulSetStatus) { if set.Spec.UpdateStrategy.Type == apps.RollingUpdateStatefulSetStrategyType && <del> status.UpdatedReplicas == status.Replicas && status.ReadyReplicas == status.Replicas { </del> <ins> status.UpdatedReplicas == *set.Spec.Replicas && status.ReadyReplicas == *set.Spec.Replicas && status.Replicas == *set.Spec.Replicas { </ins> status.CurrentReplicas = status.UpdatedReplicas status.CurrentRevision = status.UpdateRevision }

My application is clustered and I have 2 pods running. I am using a rolling upgrade strategy and my partitioning number is 1. First I need to upgrade the pod with the ordinal number 1 so I triggered that upgrade with partitioning number 1 and my upgrade gets completed. After this, my pod having the ordinal number 0 gets restarted and it's automatically upgraded to a new image. This is not an expected behavior because still, my partition number is 1 and pod-0 should come up with an older image only. I am facing this issue for the v1.27 cluster. Prior to this, it was working fine. pod-0 should not upgrade to new image until I upgrade the partition number to 0.a Do helm install and upgrade pod-1 only keeping partition number to 1. No response <detailsconsole $ cat /etc/os-release $ uname -a C:wmic os get Caption, Version, BuildNumber, OSArchitecture </details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig/api-machinery
/sig apps /sig storage
Hi, Can someone suggest here? As this is introduced in my setup after the v1.27 upgrade? Is there any change in partition logic?
This can be reproduced by using the said STS. 1 - Deploy STS using attached that has a replica count is 2 and updatingstrategy.rolling.partition as 1. 2 - Do edit STS and update the image to 1.15. This will upgrade, pod-1 with image 1.15. 3 - Next delete the pod pod-0 and check that POD-0 came up with a new image which is not the expected behavior. POD 0 should not be updated to a new image until the partition is changed to 0. We have noticed this behavior change in v1.27.


Question: // are set to 0. func completeRollingUpdate(set *apps.StatefulSet, status *apps.StatefulSetStatus) { if set.Spec.UpdateStrategy.Type == apps.RollingUpdateStatefulSetStrategyType && <del> status.UpdatedReplicas == status.Replicas && status.ReadyReplicas == status.Replicas { </del> <ins> status.UpdatedReplicas == *set.Spec.Replicas && status.ReadyReplicas == *set.Spec.Replicas && status.Replicas == *set.Spec.Replicas { </ins> status.CurrentReplicas = status.UpdatedReplicas status.CurrentRevision = status.UpdateRevision }

Pla suggest : apiVersion: v1 kind: Service metadata: name: nginx labels: app: nginx spec: ports: port: 80 name: web clusterIP: None selector: app: nginx apiVersion: apps/v1 kind: StatefulSet metadata: name: web-am spec: updateStrategy: type: RollingUpdate rollingUpdate: partition: 1 serviceName: "nginx" replicas: 2 selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: name: nginx args: /bin/sh -c sleep 30;touch /tmp/healthy; sleep 3600 image: nginx:1.14.2 ports: containerPort: 80 name: web


Question: }) }) <del> Context("when pod using local volume with non-existant path", func() { ep := &eventPatterns{ reason:  "FailedMount", pattern: make([]string, 2)} ep.pattern = append(ep.pattern, "MountVolume.SetUp failed") ep.pattern = append(ep.pattern, "does not exist") It("should not be able to mount", func() { testVol := &localTestVolume{ node:            config.node0, hostDir:         "/non-existent/location/nowhere", localVolumeType: testVolType, } By("Creating local PVC and PV") createLocalPVCsPVs(config, []*localTestVolume{testVol}, testMode) pod, err := createLocalPod(config, testVol) Expect(err).To(HaveOccurred()) checkPodEvents(config, pod.Name, ep) verifyLocalVolume(config, testVol) cleanupLocalPVCsPVs(config, []*localTestVolume{testVol}) }) }) </del> <ins> }) } </ins> <del> Context("when pod's node is different from PV's NodeAffinity", func() { </del> <ins> Context("when local volume cannot be mounted [Slow]", func() { // TODO: // - make the pod create timeout shorter // - check for these errors in unit tests intead It("should fail mount due to non-existant path", func() { ep := &eventPatterns{ reason:  "FailedMount", pattern: make([]string, 2)} ep.pattern = append(ep.pattern, "MountVolume.SetUp failed") ep.pattern = append(ep.pattern, "does not exist") testVol := &localTestVolume{ node:            config.node0, hostDir:         "/non-existent/location/nowhere", localVolumeType: DirectoryLocalVolumeType, } By("Creating local PVC and PV") createLocalPVCsPVs(config, []*localTestVolume{testVol}, immediateMode) pod, err := createLocalPod(config, testVol) Expect(err).To(HaveOccurred()) checkPodEvents(config, pod.Name, ep) verifyLocalVolume(config, testVol) cleanupLocalPVCsPVs(config, []*localTestVolume{testVol}) }) </ins> <del> BeforeEach(func() { if len(config.nodes) < 2 { framework.Skipf("Runs only when number of nodes >= 2") } }) </del> <ins> It("should fail mount due to wrong node", func() { if len(config.nodes) < 2 { framework.Skipf("Runs only when number of nodes >= 2") } </ins> <del> ep := &eventPatterns{ reason:  "FailedScheduling", pattern: make([]string, 2)} ep.pattern = append(ep.pattern, "MatchNodeSelector") ep.pattern = append(ep.pattern, "VolumeNodeAffinityConflict") </del> <ins> ep := &eventPatterns{ reason:  "FailedMount", pattern: make([]string, 2)} ep.pattern = append(ep.pattern, "NodeSelectorTerm") ep.pattern = append(ep.pattern, "MountVolume.NodeAffinity check failed") </ins> <del> It("should not be able to mount due to different NodeAffinity", func() { testPodWithNodeName(config, testVolType, ep, config.nodes[1].Name, makeLocalPodWithNodeAffinity, testMode) }) </del> <ins> testVols := setupLocalVolumesPVCsPVs(config, DirectoryLocalVolumeType, config.node0, 1, immediateMode) testVol := testVols[0] </ins> <del> It("should not be able to mount due to different NodeSelector", func() { testPodWithNodeName(config, testVolType, ep, config.nodes[1].Name, makeLocalPodWithNodeSelector, testMode) }) </del> <ins> pod := makeLocalPodWithNodeName(config, testVol, config.nodes[1].Name) pod, err := config.client.CoreV1().Pods(config.ns).Create(pod) Expect(err).NotTo(HaveOccurred()) </ins> <del> }) </del> <ins> err = framework.WaitForPodNameRunningInNamespace(config.client, pod.Name, pod.Namespace) Expect(err).To(HaveOccurred()) checkPodEvents(config, pod.Name, ep) </ins> <del> Context("when pod's node is different from PV's NodeName", func() { </del> <ins> cleanupLocalVolumes(config, []*localTestVolume{testVol}) }) }) </ins> <del> BeforeEach(func() { if len(config.nodes) < 2 { framework.Skipf("Runs only when number of nodes >= 2") } }) </del> <ins> Context("when pod's node is different from PV's NodeAffinity", func() { var ( testVol    *localTestVolume volumeType localVolumeType ) BeforeEach(func() { if len(config.nodes) < 2 { framework.Skipf("Runs only when number of nodes >= 2") } </ins> <del> ep := &eventPatterns{ reason:  "FailedMount", pattern: make([]string, 2)} ep.pattern = append(ep.pattern, "NodeSelectorTerm") ep.pattern = append(ep.pattern, "MountVolume.NodeAffinity check failed") </del> <ins> volumeType = DirectoryLocalVolumeType setupStorageClass(config, &immediateMode) testVols := setupLocalVolumesPVCsPVs(config, volumeType, config.node0, 1, immediateMode) testVol = testVols[0] }) </ins> <del> It("should not be able to mount due to different NodeName", func() { testPodWithNodeName(config, testVolType, ep, config.nodes[1].Name, makeLocalPodWithNodeName, testMode) }) }) </del> <ins> AfterEach(func() { cleanupLocalVolumes(config, []*localTestVolume{testVol}) cleanupStorageClass(config) </ins> }) <del> } </del> <ins> It("should not be able to mount due to different NodeAffinity", func() { testPodWithNodeConflict(config, volumeType, config.nodes[1].Name, makeLocalPodWithNodeAffinity, immediateMode) }) It("should not be able to mount due to different NodeSelector", func() { testPodWithNodeConflict(config, volumeType, config.nodes[1].Name, makeLocalPodWithNodeSelector, immediateMode) }) }) </ins> Context("when using local volume provisioner", func() { var volumePath string

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /assign What happened: The tests are failing again because the scheduler messages have changed yet again. I'm going to remove the check for the exact messages because this is at least the 3rd time this has happened. Instead depend on unit/integration tests to check for specific failures. Also there's no need for these tests to run across all types of local volumes because it's just testing the scheduler part.


Question: scheme := runtime.NewScheme() scheme.AddKnownTypes(cmResource.GroupVersion(), &v1.ConfigMap{}) codecs := serializer.NewCodecFactory(scheme) <del> o := NewFieldManagedObjectTracker(scheme, codecs.UniversalDecoder(), fakeTypeConverter) </del> <ins> o := NewFieldManagedObjectTracker(scheme, codecs.UniversalDecoder(), configMapTypeConverter(scheme)) </ins> reaction := ObjectReaction(o) action := NewCreateAction(cmResource, "default", &v1.ConfigMap{

Smoke tests run by publishing-bot on client-go master are failing Ref for failure logs: Tests defined for client-go, after publishing, Since commit No response As part of , when module is used from within the staging directory, can be used by traversing to the parent directory. But once the module is made as a separate repository as part of publishing, then traversing to repository using path will not work. No response /sig-api-machinery
/sig api-machinery
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /sig api-machinery Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Looking
I'll remove the swagger file dependency, I don't think we strictly need it.
/triage accepted


Question: minRequestTimeout:       minRequestTimeout, } crdInformer.Informer().AddEventHandler(cache.ResourceEventHandlerFuncs{ <ins> AddFunc:    ret.createCustomResourceDefinition, </ins> UpdateFunc: ret.updateCustomResourceDefinition, DeleteFunc: func(obj interface{}) { ret.removeDeadStorage()

Which jobs are failing: pull-kubernetes-integration Which test(s) are failing: TestWebhookConverterWithDefaulting /assign /priority important-soon /area custom-resources /sig api-machinery /kind flake /milestone v1.16
/cc
several of the webhook converter tests have similar flakes. it looks like the wait for storage version to become effective times out occasionally Rare flakes seen in all of these tests: TestWebhookConverterWithWatchCache/nontrivial-converter TestWebhookConverterWithWatchCache/metadata-mutating-converter TestWebhookConverterWithDefaulting/metadata-mutating-converter TestWebhookConverterWithWatchCache/noop-converter TestWebhookConverterWithDefaulting/empty-response TestWebhookConverterWithoutWatchCache/nontrivial-converter/check-2 TestWebhookConverterWithoutWatchCache/noop-converter/check-0/v1beta2 TestWebhookConverterWithoutWatchCache/nontrivial-converter/check-0/v1beta2
cc
Yikes, I’ll sort it out.
iiuc sending an empty patch gets short-circuited and doesn't bump CR generation, even if the storage version gets changed, which means we do not re-write to etcd and go through the encoding path this would be racing and doing noop's if the first pass of didn't get the expected storage version I will send a fix to make the patch do actual mutation
It doesn't bump generation, but the bytes to persist in etcd would still be different if the storage version had changed, so they should persist. You can verify that by changing the storage version on a CRD yourself, do an empty patch, and see if the resourceVersion changes.
you're right. Generation is for non-metadata change only, and the RV changes. I queried etcd locally and the apiVersion did change for an empty patch (sidetracking: another observation is after changing the storage version (say v1 -v2; noop convertor), empty-patching v2 endpoint would bump generation, while empty-patching v1 endpoint wouldn't)
another theory: our cached storage strategy could be stale if the CRD informer merges multiple events together (e.g. the informer does a re-list).


Question: minRequestTimeout:       minRequestTimeout, } crdInformer.Informer().AddEventHandler(cache.ResourceEventHandlerFuncs{ <ins> AddFunc:    ret.createCustomResourceDefinition, </ins> UpdateFunc: ret.updateCustomResourceDefinition, DeleteFunc: func(obj interface{}) { ret.removeDeadStorage()

Two scenarios: -create-update events merged into an update event: strategy with UID:a was created on demand for CRD Foo Foo was deleted and recreated strategy with UID:b was created on demand for the new Foo new Foo got updated (e.g. switching storage version) the strategy with UID:b is supposed to be deleted and , to reflect the last update. But if the CRD informer merges the delete-create-update events into , the strategy with UID:a will be deleted, but the strategy with UID:b won't. -update events merged into a create event: strategy with UID:a was created on demand for CRD Foo Foo was updated (e.g. switching storage version) the strategy with UID:a is supposed to be deleted and , to reflect the last update. But if the CRD informer merges the create-update events into , the strategy with UID:a won't be deleted (as we don't react on create events). another race found in , which could happen in a single update.
I think I :


Question: "httpGet": { "host": "127.0.0.1", "port": 8080, <del> "path": "/healthz" </del> <ins> "path": "/healthz?exclude=etcd" </ins> }, "initialDelaySeconds": {{liveness_probe_initial_delay}}, "timeoutSeconds": 15 }, <ins> "readinessProbe": { "httpGet": { "host": "127.0.0.1", "port": 8080, "path": "/healthz" }, "timeoutSeconds": 15 }, </ins> "ports":[ { "name": "https", "containerPort": {{secure_port}},

What would you like to be : Add support to the kube-apiserver so that we can disable health-checks via query params. Why is this needed: Currently, the healthz endpoint is inflexible in that any and all health checks are currently executed when the endpoint is hit. Since the kube-apiserver only currently supports a liveness check, this means all persistent health check failures will eventually result in a kubelet driven restart. However, this may not be desirable behavior if you have additional monitoring services layered on top of your control plane. For instance, take the etcd failure from the perspective of the kube-apiserver. Currently, persistent failures during the etcd health check from the api-server healthz endpoint will cause the kube-apiserver to be restarted. This sometimes helps (in the case of a faulty etcd-client connection) but also can be detrimental (the kube-apiserver restarts until the etcd cluster is functioning, then the kube-apiserver bombards the etcd cluster with requests to update local caches, saturating the etcd cluster with requests, causing it to timeout on its own liveness probes and trigger a kubelet-driven restart). Let's say that we could now disable etcd from our liveness health check. Then if we had an external monitoring service, we could ask independently ask etcd if it was healthy and we could also ask the kube-apiserver if it thought etcd was unhealthy. Based on the combination of those answers, we could enable more intelligent responses to certain situation, like restarting the kube-apiserver when it thought etcd was unhealthy but etcd was actually reporting as healthy but not doing so when etcd was actually unhealthy. This feature is backwards-compatible and opt-in. There would be no impact to existing behavior unless these query params are actually used. /sig api-machinery /kind feature
/cc
SGTM
/assign


Question: func TestPodContainerDeviceAllocation(t *testing.T) { flag.Set("alsologtostderr", fmt.Sprintf("%t", true)) <del> var logLevel string flag.StringVar(&logLevel, "logLevel", "4", "test") flag.Lookup("v").Value.Set(logLevel) </del> res1 := TestResource{ resourceName:     "domain1.com/resource1", resourceQuantity: *resource.NewQuantity(int64(2), resource.DecimalSI),

This is not 100% reproducable in my cluster. There is 20% chance of failing in my environment.
/area hw-accelerators mentioned a root cause that might be due to unit tests sharing the same Unix socket and interfering with each others. cc
Can you please share latest commit id of your code base using ?
Thanks very much, can you try the follow command please:
/assign
The latest commit id of my code base:
I found this is not 100% reproducable in my cluster. There is 20% chance of failing in my environment.
Result of go test command:
/sig scheduling
Thanks!
Hi, does these unit tests pass with PR
I run 100 times without failure.
Good to here that, Nice work! Thanks!


Question: timeoutCtx, timeoutCancel := context.WithTimeout(ctx, le.config.RenewDeadline) defer timeoutCancel() err := wait.PollImmediateUntil(le.config.RetryPeriod, func() (bool, error) { <del> return le.tryAcquireOrRenew(), nil </del> <ins> done := make(chan bool, 1) go func() { defer close(done) done <- le.tryAcquireOrRenew() }() select { case <-timeoutCtx.Done(): return false, fmt.Errorf("failed to tryAcquireOrRenew %s", timeoutCtx.Err()) case result := <-done: return result, nil } </ins> }, timeoutCtx.Done()) le.maybeReportTransition() desc := le.config.Lock.Describe() if err == nil {

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: kube-cm kube-scheduler does not exit when kube-apiserver suspended What you expected to happen: kube-cm kube-scheduler should exit after LeaderElector stops leading How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig api-machinery /kind bug
/assign
/cc


Question: out.Filter = (*v1alpha1.PluginSet)(unsafe.Pointer(in.Filter)) out.PostFilter = (*v1alpha1.PluginSet)(unsafe.Pointer(in.PostFilter)) out.Score = (*v1alpha1.PluginSet)(unsafe.Pointer(in.Score)) <del> out.NormalizeScore = (*v1alpha1.PluginSet)(unsafe.Pointer(in.NormalizeScore)) </del> out.Reserve = (*v1alpha1.PluginSet)(unsafe.Pointer(in.Reserve)) out.Permit = (*v1alpha1.PluginSet)(unsafe.Pointer(in.Permit)) out.PreBind = (*v1alpha1.PluginSet)(unsafe.Pointer(in.PreBind))

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Consolidate ) into Score plugin set in config API. Why is this needed: NormalizeScore extension point is implemented in . We decided to treat NormalizeScore as an "optional" step of Score plugin, see . However, we deferred the removal of due to API review delays. Now that PR was merged. We can start the actual removal from the API and start the API review process. /sig scheduling
/assign


Question: // +k8s:deepcopy-gen=package <del> package testing </del> <ins> package testing // import "k8s.io/apiserver/pkg/endpoints/testing" </ins>

Look at one of the directories under (Please don't pick more than one directory for one PR as it will cause issues for reviewers) Find all the files named (example find -name ) See the following link it has a this tells tools how this package needs to be imported. So we should ensure that each has an import Add a description appropriate to the package (similar to the one below) Look through sub directories and see if there are any other important directories that are likely to be used by outside users and add a Why are we doing this? We perfer to use vanity urls instead of directly using to refer to code, so this helps ensure that tools honor our wish. See It would be good to have some package level documentation since these directories under staging/ are meant to be for use by folks outside of kubernetes github org. /good-first-issue
/kind documentation /sig architecture
currently working my through Not going to use / assign since there could be a number of PRs for this issue.
/lifecycle active /good-first-issue edit: hmm..
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /lifecycle active /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
May I try on
IIUC this means that there needs to be a even at the root of the repo? Then how does work? :thinking:
Hi , can I make a PR for this directory ?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or .


Question: // +k8s:deepcopy-gen=package <del> package testing </del> <ins> package testing // import "k8s.io/apiserver/pkg/endpoints/testing" </ins>

/lifecycle rotten
I guess this is still not done, was supposed to partially fix this issue. /reopen
Reopened this issue. <detailsIn response to : I guess this is still not done, was supposed to partially fix this issue. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-lifecycle rotten /area code-organization