Question: } ] }` ginkgo.By("Generate event list options") selector := fields.Set{ "involvedObject.kind":      "Node",

Which jobs are failing: Which test(s) are failing: Since when has it been failing: Last successful run (23 Oct 2020, 10:07 PDT): First unsuccessful run (23 Oct 2020, 12:08 PDT): Testgrid link: Reason for failure: Anything else we need to know: presubmit job in repository also started to fail in the same fashion. Attempted to debug it myself, found out the following stuff: last successful run was accessing the apiserver via HTTP: all unsuccessful runs attempted to access the apiserver via HTTPS but to no avail: perused the history of to find out that the PR is almost certainly the reason of the failures (or rather uncovered some problem with the NPD or its tests): first, it coincides with the start of the failures of , second, we changed the default hostname of the apiserver in node tests in that PR. Judging by the code and the changes in the aforementioned PR I think that a quick mitigation would be to add a flag to the of these jobs. /assign /sig node /cc
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Opened for a mitigation that would make those jobs stop failing.
This won't work, unfortunately. Copying response:
I have filed to address this issue, but it seems I could not run the test in advance to verify my patch, did you know how to manually trigger the presubmit job?
Unfortunately I don't :/. My PR was also a wild guess. could you take a look at the proposed fix? Thanks once again for your help - really appreciated :).
the submission of hasn't fixed the problems, unfortunately :/. Sample failing run: We still have some peculiar problem related to health checking: and now NPD panics, too:
Thanks for your report, I believe the reason is admission controller has been disabled in node e2e tests: We need to figure out a new approach now. /reopen
Reopened this issue.


Question: } ] }` ginkgo.By("Generate event list options") selector := fields.Set{ "involvedObject.kind":      "Node",

<detailsIn response to : Thanks for your report, I believe the reason is admission controller has been disabled in node e2e tests: We need to figure out a new approach now. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
filed . I hope we could completely address this issue this time.
/milestone v1.20


Question: TemplateArgument: &tc.templateArg, AllowMissingKeys: tc.allowMissingKeys, } <ins> if !sort.StringsAreSorted(printFlags.AllowedFormats()) { t.Fatalf("allowed formats are not sorted") } </ins> outputFormat := "jsonpath" p, err := printFlags.ToPrinter(outputFormat)

/kind bug What happened: When building kubernetes-1.7.0 packages for openSUSE, results differ for each build in and other man pages. coming from that uses and probably sets do not have any specific order, so it would need to be sorted on output. What you expected to happen: Every build should give identical results. See https://reproducible- for why this matters. How to reproduce it (as minimally and precisely as possible): See Anything else we need to know?: Environment: Kubernetes version: 1.7.0 OS (e.g. from /etc/os-release): openSUSE Tumbleweed
/sig release
somehow this bug seems to have been forgotten - can you help getting the right people to have a look?
- can you please confirm that PR works for you?
I tested it and it looks good.
/assign I enjoy reading my highlights to find the issue has already been addressed. :)
Sorry this wasn't addressed sooner. It does seem like it fell into a black hole, and pinging was the right way to resolve that. :)
/area build-release
related to /reopen
Reopening this issue. <detailsIn response to : related to Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: protocol:            protocol, socket:              sock, timeout:             timeout, <ins> activeClients:       newClientCache(), </ins> sessionAffinityType: api.ServiceAffinityNone, // default stickyMaxAgeMinutes: 180,                     // TODO: parameterize this in the API. }

Depends on UDP clients have a timeout (so we don't leak connections forever), but the test does not cover this. To do this, we probably need to track active connections, and plumb that all the way down to the goroutines spawned by the ProxyLoop routines. This is probably something we want to do anyway to be able to monitor.


Question: By("Before scale up finished setting 2nd pod to be not ready by breaking readiness probe") sst.BreakProbe(ss, testProbe) <ins> sst.WaitForStatus(ss, 0) </ins> sst.WaitForRunningAndNotReady(2, ss) By("Continue scale operation after the 2nd pod, and scaling down to 1 replica")

https://k8s- do you know the statefulset owners?
cc
I did not see this happening until recently.
The controller misses that the first pod transitioned to unready and proceeds to create the second pod. The test uses a client to hit the api server and will scale up the statefulset as soon as the broken pod is observed but the controller uses caches so it may observe the scaled up statefulset quicker than the updated pod.
I will update the test to ensure that we wait until the unreadiness is observed on StatefulSet.Status.Replicas field prior to the scaling operation
SGTM


Question: func MakeDefaultErrorFunc(client clientset.Interface, podQueue internalqueue.SchedulingQueue, schedulerCache internalcache.Cache, stopEverything <-chan struct{}) func(pod *v1.Pod, err error) { return func(pod *v1.Pod, err error) { if err == core.ErrNoNodesAvailable { <del> klog.V(4).Infof("Unable to schedule %v/%v: no nodes are registered to the cluster; waiting", pod.Namespace, pod.Name) </del> <ins> klog.V(2).Infof("Unable to schedule %v/%v: no nodes are registered to the cluster; waiting", pod.Namespace, pod.Name) </ins> } else { if _, ok := err.(*core.FitError); ok { <del> klog.V(4).Infof("Unable to schedule %v/%v: no fit: %v; waiting", pod.Namespace, pod.Name, err) </del> <ins> klog.V(2).Infof("Unable to schedule %v/%v: no fit: %v; waiting", pod.Namespace, pod.Name, err) </ins> } else if errors.IsNotFound(err) { <ins> klog.V(2).Infof("Unable to schedule %v/%v: possibly due to node not found: %v; waiting", pod.Namespace, pod.Name, err) </ins> if errStatus, ok := err.(errors.APIStatus); ok && errStatus.Status().Details.Kind == "node" { nodeName := errStatus.Status().Details.Name // when node is not found, We do not remove the node right away. Trying again to get

<!-- Please only use this template for submitting enhancement requests --What would you like to be : 1) Increase the log level of info logs to "v=2" when the scheduler or a node from its cache. 2) Increase the log level of info logs to "v=2" when the scheduler can't schedule a pod , , and add a log message for 2) report memory/cpu status of . Why is this needed: With cluster autoscaler now being used in production by some vendors, we want to have more visibility to events that could be related to the auto scaler to help debug customer issues. /sig scheduling /priority important-soon /assign
GitHub didn't allow me to assign the following users: congliu01. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : <!-- Please only use this template for submitting enhancement requests --1) Increase the log level of info logs to "v=2" when the scheduler or a node from its cache. 2) report memory/cpu status of . Why is this needed: With cluster autoscaler now being used in production by some vendors, we want to have more visibility to events that could be related to the auto scaler to help debug customer issues. /priority important-soon /assign </details
will consider this assigned to you.
/assign


Question: glog.Errorf("can't open %s, skipping this nodePort: %v", lp.String(), err) continue } <ins> if lp.protocol == "udp" { proxier.clearUdpConntrackForPort(lp.port) } </ins> replacementPortsMap[lp] = socket } // We're holding the port, so it's OK to install iptables rules.

Hi, We have a device (router in this case) that sends long living stream of udp packets to our kubernetes cluster. We then create a service and pods that expose a node port to receive the incoming udp packets. However, packets were not reach our containers. Further digging and found that because UDP packets already stream into our host (therefore conntrack entry already created) BEFORE our service is created (iptable rules ), the iptable rule didn't take effect. Since our UDP stream is long living, the conntrack entry never get timeout neither. To workaround the issue, we have to stop the UDP stream for sometime after the service is created so that the conntrack entry times out. If we use "conntrack -D" to clear all entries after service is deployed also workaround the issue. Though it is the behavior of the linux iptable that new rules doesn't affect already existing connection, we are trying to see if kubernetes proxy have any config that can clear the related conntrack entries when a new service is deployed?
I am interested in your problem. Can you give some more details how did you add iptables rules before service started. Did you use the PREROUTING chain to forward port to pod ?
OK I was able to reproduce the problem
I think the issue is more problematic and complex. It is explained here:
Yes I think it is the same conntrack issue as in the one reported in docker issue. But I also think that the kubernetes might have a better control over it, since it knows the service up/down. I think k8s already deleting the conntrack entries when removing services. Just wondering if the same can apply to adding services?
Yes you are right. Kubernetes as a Docker manager should take care of this. Deleting the conntrack entries is done in very simple way: Just call the application
But there is another problem because I removed conntrack entries and then I created service and still have problem. I had to clear conntrack entries after I started my service. I will try add this to source code and then try it.
In code it looks much better. I assume you used for your service to use some particular port. The fix is in place when node port is binded. Do you think there is any other scenario where conntrack entries should be cleared?


Question: glog.Errorf("can't open %s, skipping this nodePort: %v", lp.String(), err) continue } <ins> if lp.protocol == "udp" { proxier.clearUdpConntrackForPort(lp.port) } </ins> replacementPortsMap[lp] = socket } // We're holding the port, so it's OK to install iptables rules.

When I create PR I will let you know
The PR created: If you want verify it you can easily do it using local cluster Then clone kube-deploy project to create local k8s cluster with image you already created with fixes
Thanks for the fix. This is what we did manually to workaround so it should be good enough for us. It would be even better if you can clear the conntrack entry only for the affected entry, such as destination port, to minimizes the impact. Wondering if that possible? Will try out as well. Thanks.
I wanted do it like you mentioned but unfortunately conntract command doesn't give that possibilities. In parameters you can only pass the source and destination IP addresses.
Sorry didn't get back for a while. I tried the conntrack tool and seems like you can use --dport option: conntrack -D -s 172.27.209.165 -p udp --dport 53 udp 17 63 src=172.27.209.165 dst=171.70.168.183 sport= dport=53 src=171.70.168.183 dst=172.27.209.165 sport=53 dport= [ASSURED] mark=0 use=1 udp 17 33 src=172.27.209.165 dst=171.70.168.183 sport= dport=53 src=171.70.168.183 dst=172.27.209.165 sport=53 dport= [ASSURED] mark=0 use=1 conntrack v1.4.3 (conntrack-tools): 2 flow entries have been deleted.
Thanks for info I will try it
I've updated my PR for flag It works well
It's not clear to me why there would be a conntrack rule for your service port before we add any rules?
We have a router, which configured to export statistics using UDP to our host. That UDP stream is long live. If our host restarted, or we re-deploy our cluster (mainly during development), then udp packets received by host before K8S is provisioned.
But why do you have conntrack rules - simply receiving packets will not cause conntrack to create records.. , dachulin <:


Question: // TODO: This k8s-generic, well-known constant should be fetchable from another source, not be in this package KubeProxyClusterRoleName = "system:node-proxier" <ins> // KubeProxyClusterRoleBindingName sets the name for the kube-proxy CluterRoleBinding KubeProxyClusterRoleBindingName = "kubeadm:node-proxier" </ins> // KubeProxyServiceAccountName describes the name of the ServiceAccount for the kube-proxy addon KubeProxyServiceAccountName = "kube-proxy" <del> // KubeProxyClusterRoleBindingName sets the name for the kube-proxy CluterRoleBinding KubeProxyClusterRoleBindingName = "kubeam:node-proxier" </del> // KubeProxyConfigMapRoleName sets the name of ClusterRole for ConfigMap KubeProxyConfigMapRoleName = "kube-proxy" )

kubeadm-kinder-latest ci-kubernetes-e2e-kubeadm-kinder-latest.Overall E2EKubeadm suite.[sig-cluster-lifecycle] [area-kubeadm] proxy addon kube-proxy ServiceAccount should be bound to the system:node-proxier cluster role 2021-11-09 23:32 CET () E2EKubeadm suite: [sig-cluster-lifecycle] [area-kubeadm] proxy addon kube-proxy ServiceAccount should be bound to the system:node-proxier cluster role This issue seems to be linked to FYI: /cc /sig cluster-lifecycle /area kubeadm
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
fix PR is here, needs LGTM:
Also failing on Thanks !


Question: // TODO: This k8s-generic, well-known constant should be fetchable from another source, not be in this package KubeProxyClusterRoleName = "system:node-proxier" <ins> // KubeProxyClusterRoleBindingName sets the name for the kube-proxy CluterRoleBinding KubeProxyClusterRoleBindingName = "kubeadm:node-proxier" </ins> // KubeProxyServiceAccountName describes the name of the ServiceAccount for the kube-proxy addon KubeProxyServiceAccountName = "kube-proxy" <del> // KubeProxyClusterRoleBindingName sets the name for the kube-proxy CluterRoleBinding KubeProxyClusterRoleBindingName = "kubeam:node-proxier" </del> // KubeProxyConfigMapRoleName sets the name of ClusterRole for ConfigMap KubeProxyConfigMapRoleName = "kube-proxy" )

kubeadm-kinder-latest-on-1-22 E2EKubeadm suite.[sig-cluster-lifecycle] [area-kubeadm] proxy addon kube-proxy ServiceAccount should be bound to the system:node-proxier cluster role ci-kubernetes-e2e-kubeadm-kinder-latest-on-1-22.Overall task-06-e2e-kubeadm 2021-11-09 23:31 CET : E2EKubeadm suite: [sig-cluster-lifecycle] [area-kubeadm] proxy addon kube-proxy ServiceAccount should be bound to the system:node-proxier cluster role No response /sig cluster-lifecycle
/assign
/triage accepted will have a look and send a fix if it's kubeadm related.
/area kubeadm
fix PR is here, needs LGTM:


Question: if pv.Spec.CSI == nil || pv.Spec.CSI.NodeExpandSecretRef == nil { framework.Fail("creating pv without 'NodeExpandSecretRef'") } <ins> if pv.Spec.CSI.NodeExpandSecretRef.Namespace != f.Namespace.Name || pv.Spec.CSI.NodeExpandSecretRef.Name != secretName { framework.Failf("failed to set node expand secret ref, namespace: %s name: %s", pv.Spec.CSI.NodeExpandSecretRef.Namespace, pv.Spec.CSI.NodeExpandSecretRef.Name) } </ins> ginkgo.By("Expanding current pvc") newSize := resource.MustParse("6Gi")

Seen in pull-kubernetes-e2e-gce-cos-alpha-features Specific job failure
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
from the error message I don't know if it refers to the sig-storage error or to the sig-auth one
/sig storage
/kind failing-test
/assign From : ... I am not sure if it's related:
is the culprit. cannot be used inside a . This only gets caught when the test actually runs, otherwise Ginkgo never sees the incorrect .
I don’t understand why sometimes it can get the pv by namebut sometimes it failed
but then is a sig-auth test , not storage , sorry /sig auth this seems that was recently in
/remove sig-storage
/assign /priority important-soon
/unassign
is still failed. seems not fixing the whole problem. /reopen feel free to close if the ci is green.
Reopened this issue. <detailsIn response to : is still failed. /reopen feel free to close if the ci is green. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
the failure now looks like it is just a sig storage issue:
cc
/assign looks like the new test is still an issue
I think this line is the problem: When the test creates the object, it may take time for the PV to be provisioned and to be bound to the PVC. So you should do another before this call so you can get the latest PVC object with the bound PV.
I opened to get PVC again before using .
Only do PersistentVolumeClaims.Get() is not enough, whether we need to wait for sometimes to wait it bound? because sometimes the testcase is successful and sometime is failed.
If the Pod was able to run, then the PVC has to be bound.
so I can't understand why sometimes the testcase is successful and sometimes is failed.


Question: if pv.Spec.CSI == nil || pv.Spec.CSI.NodeExpandSecretRef == nil { framework.Fail("creating pv without 'NodeExpandSecretRef'") } <ins> if pv.Spec.CSI.NodeExpandSecretRef.Namespace != f.Namespace.Name || pv.Spec.CSI.NodeExpandSecretRef.Name != secretName { framework.Failf("failed to set node expand secret ref, namespace: %s name: %s", pv.Spec.CSI.NodeExpandSecretRef.Namespace, pv.Spec.CSI.NodeExpandSecretRef.Name) } </ins> ginkgo.By("Expanding current pvc") newSize := resource.MustParse("6Gi")

By default, the DelayBinding is false, it will wait for PVC bound during creation. This is weird.
It waits for the claim to be bound, but the test case still needs to refresh its pvc object.
Because of the refresh is async, so after the pvc was bound, we need to get the pvc again, isn't it?


Question: v1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/types" <ins> utilfeature "k8s.io/apiserver/pkg/util/feature" </ins> clientset "k8s.io/client-go/kubernetes" typedv1 "k8s.io/client-go/kubernetes/typed/core/v1" <ins> featuregatetesting "k8s.io/component-base/featuregate/testing" </ins> kubeapiservertesting "k8s.io/kubernetes/cmd/kube-apiserver/app/testing" <ins> "k8s.io/kubernetes/pkg/features" </ins> "k8s.io/kubernetes/test/integration" "k8s.io/kubernetes/test/integration/framework" )

See for context. This issue is creating to track work on the API validation changes. Validation relaxation of certain fields in the pod spec to allow for mutable scheduling directives.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/sig scheduling
Docs PR:


Question: gomega.Expect(warning.Warning).To(gomega.ContainSubstring("undefined field 'maxRetries'")) }) }) <ins> /* Release: v1.30 Testname: ValidatingAdmissionPolicy API Description: The admissionregistration.k8s.io API group MUST exist in the /apis discovery document. The admissionregistration.k8s.io/v1 API group/version MUST exist in the /apis/admissionregistration.k8s.io discovery document. The validatingadmisionpolicy and validatingadmissionpolicy/status resources MUST exist in the /apis/admissionregistration.k8s.io/v1 discovery document. The validatingadmisionpolicy resource must support create, get, list, watch, update, patch, delete, and deletecollection. */ ginkgo.It("should support ValidatingAdmissionPolicy API operations", func(ctx context.Context) { vapVersion := "v1" ginkgo.By("getting /apis") { discoveryGroups, err := f.ClientSet.Discovery().ServerGroups() framework.ExpectNoError(err) found := false for _, group := range discoveryGroups.Groups { if group.Name == admissionregistrationv1.GroupName { for _, version := range group.Versions { if version.Version == vapVersion { found = true break } } } } if !found { framework.Failf("expected ValidatingAdmissionPolicy API group/version, got %#v", discoveryGroups.Groups) } } ginkgo.By("getting /apis/admissionregistration.k8s.io") { group := &metav1.APIGroup{} err := f.ClientSet.Discovery().RESTClient().Get().AbsPath("/apis/admissionregistration.k8s.io").Do(ctx).Into(group) framework.ExpectNoError(err) found := false for _, version := range group.Versions { if version.Version == vapVersion { found = true break } } if !found { framework.Failf("expected ValidatingAdmissionPolicy API version, got %#v", group.Versions) } } ginkgo.By("getting /apis/admissionregistration.k8s.io/" + vapVersion) { resources, err := f.ClientSet.Discovery().ServerResourcesForGroupVersion(admissionregistrationv1.SchemeGroupVersion.String()) framework.ExpectNoError(err) foundVAP, foundVAPStatus := false, false for _, resource := range resources.APIResources { switch resource.Name { case "validatingadmissionpolicies": foundVAP = true case "validatingadmissionpolicies/status": foundVAPStatus = true } } if !foundVAP { framework.Failf("expected validatingadmissionpolicies, got %#v", resources.APIResources) } if !foundVAPStatus { framework.Failf("expected validatingadmissionpolicies/status, got %#v", resources.APIResources) } } client := f.ClientSet.AdmissionregistrationV1().ValidatingAdmissionPolicies() labelKey, labelValue := "example-e2e-vap-label", utilrand.String(8) label := fmt.Sprintf("%s=%s", labelKey, labelValue) template := &admissionregistrationv1.ValidatingAdmissionPolicy{ ObjectMeta: metav1.ObjectMeta{ GenerateName: "e2e-example-vap-", Labels: map[string]string{ labelKey: labelValue, }, }, Spec: admissionregistrationv1.ValidatingAdmissionPolicySpec{ Validations: []admissionregistrationv1.Validation{ { Expression: "object.spec.replicas <= 100", }, }, MatchConstraints: &admissionregistrationv1.MatchResources{ ResourceRules: []admissionregistrationv1.NamedRuleWithOperations{ { RuleWithOperations: admissionregistrationv1.RuleWithOperations{ Operations: []admissionregistrationv1.OperationType{"CREATE"}, Rule: admissionregistrationv1.Rule{ APIGroups:   []string{"apps"}, APIVersions: []string{"v1"}, Resources:   []string{"deployments"}, }, }, }, }, }, }, } ginkgo.DeferCleanup(func(ctx context.Context) { err := client.DeleteCollection(ctx, metav1.DeleteOptions{}, metav1.ListOptions{LabelSelector: label}) framework.ExpectNoError(err) }) ginkgo.By("creating") _, err := client.Create(ctx, template, metav1.CreateOptions{}) framework.ExpectNoError(err) _, err = client.Create(ctx, template, metav1.CreateOptions{}) framework.ExpectNoError(err) vapCreated, err := client.Create(ctx, template, metav1.CreateOptions{}) framework.ExpectNoError(err) ginkgo.By("getting") vapRead, err := client.Get(ctx, vapCreated.Name, metav1.GetOptions{}) framework.ExpectNoError(err) gomega.Expect(vapRead.UID).To(gomega.Equal(vapCreated.UID)) ginkgo.By("listing") list, err := client.List(ctx, metav1.ListOptions{LabelSelector: label}) framework.ExpectNoError(err) ginkgo.By("watching") framework.Logf("starting watch") vapWatch, err := client.Watch(ctx, metav1.ListOptions{ResourceVersion: list.ResourceVersion, LabelSelector: label}) framework.ExpectNoError(err) ginkgo.By("patching") patchBytes := []byte(`{"metadata":{"annotations":{"patched":"true"}},"spec":{"failurePolicy":"Ignore"}}`) vapPatched, err := client.Patch(ctx, vapCreated.Name, types.MergePatchType, patchBytes, metav1.PatchOptions{}) framework.ExpectNoError(err) gomega.Expect(vapPatched.Annotations).To(gomega.HaveKeyWithValue("patched", "true"), "patched object should have the applied annotation") gomega.Expect(vapPatched.Spec.FailurePolicy).To(gomega.HaveValue(gomega.Equal(admissionregistrationv1.Ignore)), "patched object should have the applied spec") ginkgo.By("updating") var vapUpdated *admissionregistrationv1.ValidatingAdmissionPolicy err = retry.RetryOnConflict(retry.DefaultRetry, func() error { vap, err := client.Get(ctx, vapCreated.Name, metav1.GetOptions{}) framework.ExpectNoError(err) vapToUpdate := vap.DeepCopy() vapToUpdate.Annotations["updated"] = "true" fail := admissionregistrationv1.Fail vapToUpdate.Spec.FailurePolicy = &fail vapUpdated, err = client.Update(ctx, vapToUpdate, metav1.UpdateOptions{}) return err }) framework.ExpectNoError(err, "failed to update validatingadmissionpolicy %q", vapCreated.Name) gomega.Expect(vapUpdated.Annotations).To(gomega.HaveKeyWithValue("updated", "true"), "updated object should have the applied annotation") gomega.Expect(vapUpdated.Spec.FailurePolicy).To(gomega.HaveValue(gomega.Equal(admissionregistrationv1.Fail)), "updated object should have the applied spec") framework.Logf("waiting for watch events with expected annotations") for sawAnnotation := false; !sawAnnotation; { select { case evt, ok := <-vapWatch.ResultChan(): if !ok { framework.Fail("watch channel should not close") } gomega.Expect(evt.Type).To(gomega.Equal(watch.Modified)) vapWatched, isFS := evt.Object.(*admissionregistrationv1.ValidatingAdmissionPolicy) if !isFS { framework.Failf("expected an object of type: %T, but got %T", &admissionregistrationv1.ValidatingAdmissionPolicy{}, evt.Object) } if vapWatched.Annotations["patched"] == "true" { sawAnnotation = true vapWatch.Stop() } else { framework.Logf("missing expected annotations, waiting: %#v", vapWatched.Annotations) } case <-time.After(wait.ForeverTestTimeout): framework.Fail("timed out waiting for watch event") } } ginkgo.By("getting /status") resource := admissionregistrationv1.SchemeGroupVersion.WithResource("validatingadmissionpolicies") vapStatusRead, err := f.DynamicClient.Resource(resource).Get(ctx, vapCreated.Name, metav1.GetOptions{}, "status") framework.ExpectNoError(err) gomega.Expect(vapStatusRead.GetObjectKind().GroupVersionKind()).To(gomega.Equal(admissionregistrationv1.SchemeGroupVersion.WithKind("ValidatingAdmissionPolicy"))) gomega.Expect(vapStatusRead.GetUID()).To(gomega.Equal(vapCreated.UID)) ginkgo.By("patching /status") patchBytes = []byte(`{"status":{"conditions":[{"type":"PatchStatusFailed","status":"False","reason":"e2e","message":"Set from an e2e test","lastTransitionTime":"2024-01-01T00:00:00Z"}]}}`) vapStatusPatched, err := client.Patch(ctx, vapCreated.Name, types.MergePatchType, patchBytes, metav1.PatchOptions{}, "status") framework.ExpectNoError(err) hasCondition := false for i := range vapStatusPatched.Status.Conditions { if vapStatusPatched.Status.Conditions[i].Type == "PatchStatusFailed" { hasCondition = true } } gomega.Expect(hasCondition).To(gomega.BeTrueBecause("expect the patched status exist")) ginkgo.By("updating /status") var vapStatusUpdated *admissionregistrationv1.ValidatingAdmissionPolicy err = retry.RetryOnConflict(retry.DefaultRetry, func() error { vap, err := client.Get(ctx, vapCreated.Name, metav1.GetOptions{}) framework.ExpectNoError(err) vapStatusToUpdate := vap.DeepCopy() vapStatusToUpdate.Status.Conditions = append(vapStatusToUpdate.Status.Conditions, metav1.Condition{ Type:               "StatusUpdateFailed", Status:             metav1.ConditionFalse, Reason:             "E2E", Message:            "Set from an e2e test", LastTransitionTime: metav1.NewTime(time.Now()), }) vapStatusUpdated, err = client.UpdateStatus(ctx, vapStatusToUpdate, metav1.UpdateOptions{}) return err }) framework.ExpectNoError(err, "failed to update status of validatingadmissionpolicy %q", vapCreated.Name) hasCondition = false for i := range vapStatusUpdated.Status.Conditions { if vapStatusUpdated.Status.Conditions[i].Type == "StatusUpdateFailed" { hasCondition = true } } gomega.Expect(hasCondition).To(gomega.BeTrueBecause("expect the updated status exist")) ginkgo.By("deleting") err = client.Delete(ctx, vapCreated.Name, metav1.DeleteOptions{}) framework.ExpectNoError(err) vapTmp, err := client.Get(ctx, vapCreated.Name, metav1.GetOptions{}) switch { case err == nil && vapTmp.GetDeletionTimestamp() != nil && len(vapTmp.GetFinalizers()) > 0: // deletion requested successfully, object is blocked by finalizers case err == nil: framework.Failf("expected deleted object, got %#v", vapTmp) case apierrors.IsNotFound(err): // deleted successfully default: framework.Failf("expected 404, got %#v", err) } list, err = client.List(ctx, metav1.ListOptions{LabelSelector: label}) var itemsWithoutFinalizer []admissionregistrationv1.ValidatingAdmissionPolicy for _, item := range list.Items { if len(item.GetFinalizers()) == 0 { itemsWithoutFinalizer = append(itemsWithoutFinalizer, item) } } framework.ExpectNoError(err) gomega.Expect(itemsWithoutFinalizer).To(gomega.HaveLen(2), "filtered list should have 2 items") ginkgo.By("deleting a collection") err = client.DeleteCollection(ctx, metav1.DeleteOptions{}, metav1.ListOptions{LabelSelector: label}) framework.ExpectNoError(err) list, err = client.List(ctx, metav1.ListOptions{LabelSelector: label}) var itemsColWithoutFinalizer []admissionregistrationv1.ValidatingAdmissionPolicy for _, item := range list.Items { if !(item.GetDeletionTimestamp() != nil && len(item.GetFinalizers()) > 0) { itemsColWithoutFinalizer = append(itemsColWithoutFinalizer, item) } } framework.ExpectNoError(err) gomega.Expect(itemsColWithoutFinalizer).To(gomega.BeEmpty(), "filtered list should have 0 items") }) /* Release: v1.30 Testname: ValidatingadmissionPolicyBinding API Description: The admissionregistration.k8s.io API group MUST exist in the /apis discovery document. The admissionregistration.k8s.io/v1 API group/version MUST exist in the /apis/admissionregistration.k8s.io discovery document. The ValidatingadmissionPolicyBinding resources MUST exist in the /apis/admissionregistration.k8s.io/v1 discovery document. The ValidatingadmissionPolicyBinding resource must support create, get, list, watch, update, patch, delete, and deletecollection. */ ginkgo.It("should support ValidatingAdmissionPolicyBinding API operations", func(ctx context.Context) { vapbVersion := "v1" ginkgo.By("getting /apis") { discoveryGroups, err := f.ClientSet.Discovery().ServerGroups() framework.ExpectNoError(err) found := false for _, group := range discoveryGroups.Groups { if group.Name == admissionregistrationv1.GroupName { for _, version := range group.Versions { if version.Version == vapbVersion { found = true break } } } } if !found { framework.Failf("expected ValidatingAdmissionPolicyBinding API group/version, got %#v", discoveryGroups.Groups) } } ginkgo.By("getting /apis/admissionregistration.k8s.io") { group := &metav1.APIGroup{} err := f.ClientSet.Discovery().RESTClient().Get().AbsPath("/apis/admissionregistration.k8s.io").Do(ctx).Into(group) framework.ExpectNoError(err) found := false for _, version := range group.Versions { if version.Version == vapbVersion { found = true break } } if !found { framework.Failf("expected ValidatingAdmissionPolicyBinding API version, got %#v", group.Versions) } } ginkgo.By("getting /apis/admissionregistration.k8s.io/" + vapbVersion) { resources, err := f.ClientSet.Discovery().ServerResourcesForGroupVersion(admissionregistrationv1.SchemeGroupVersion.String()) framework.ExpectNoError(err) foundVAPB := false for _, resource := range resources.APIResources { switch resource.Name { case "validatingadmissionpolicybindings": foundVAPB = true } } if !foundVAPB { framework.Failf("expected validatingadmissionpolicybindings, got %#v", resources.APIResources) } } client := f.ClientSet.AdmissionregistrationV1().ValidatingAdmissionPolicyBindings() labelKey, labelValue := "example-e2e-vapb-label", utilrand.String(8) label := fmt.Sprintf("%s=%s", labelKey, labelValue) template := &admissionregistrationv1.ValidatingAdmissionPolicyBinding{ ObjectMeta: metav1.ObjectMeta{ GenerateName: "e2e-example-vapb-", Labels: map[string]string{ labelKey: labelValue, }, }, Spec: admissionregistrationv1.ValidatingAdmissionPolicyBindingSpec{ PolicyName:        "replicalimit-policy.example.com", ValidationActions: []admissionregistrationv1.ValidationAction{admissionregistrationv1.Deny}, }, } ginkgo.DeferCleanup(func(ctx context.Context) { err := client.DeleteCollection(ctx, metav1.DeleteOptions{}, metav1.ListOptions{LabelSelector: label}) framework.ExpectNoError(err) }) ginkgo.By("creating") _, err := client.Create(ctx, template, metav1.CreateOptions{}) framework.ExpectNoError(err) _, err = client.Create(ctx, template, metav1.CreateOptions{}) framework.ExpectNoError(err) vapbCreated, err := client.Create(ctx, template, metav1.CreateOptions{}) framework.ExpectNoError(err) ginkgo.By("getting") vapbRead, err := client.Get(ctx, vapbCreated.Name, metav1.GetOptions{}) framework.ExpectNoError(err) gomega.Expect(vapbRead.UID).To(gomega.Equal(vapbCreated.UID)) ginkgo.By("listing") list, err := client.List(ctx, metav1.ListOptions{LabelSelector: label}) framework.ExpectNoError(err) ginkgo.By("watching") framework.Logf("starting watch") vapbWatch, err := client.Watch(ctx, metav1.ListOptions{ResourceVersion: list.ResourceVersion, LabelSelector: label}) framework.ExpectNoError(err) ginkgo.By("patching") patchBytes := []byte(`{"metadata":{"annotations":{"patched":"true"}},"spec":{"validationActions":["Warn"]}}`) vapbPatched, err := client.Patch(ctx, vapbCreated.Name, types.MergePatchType, patchBytes, metav1.PatchOptions{}) framework.ExpectNoError(err) gomega.Expect(vapbPatched.Annotations).To(gomega.HaveKeyWithValue("patched", "true"), "patched object should have the applied annotation") gomega.Expect(vapbPatched.Spec.ValidationActions).To(gomega.Equal([]admissionregistrationv1.ValidationAction{admissionregistrationv1.Warn}), "patched object should have the applied spec") ginkgo.By("updating") var vapbUpdated *admissionregistrationv1.ValidatingAdmissionPolicyBinding err = retry.RetryOnConflict(retry.DefaultRetry, func() error { vap, err := client.Get(ctx, vapbCreated.Name, metav1.GetOptions{}) framework.ExpectNoError(err) vapbToUpdate := vap.DeepCopy() vapbToUpdate.Annotations["updated"] = "true" vapbToUpdate.Spec.ValidationActions = []admissionregistrationv1.ValidationAction{admissionregistrationv1.Deny} vapbUpdated, err = client.Update(ctx, vapbToUpdate, metav1.UpdateOptions{}) return err }) framework.ExpectNoError(err, "failed to update validatingadmissionpolicybinding %q", vapbCreated.Name) gomega.Expect(vapbUpdated.Annotations).To(gomega.HaveKeyWithValue("updated", "true"), "updated object should have the applied annotation") gomega.Expect(vapbUpdated.Spec.ValidationActions).To(gomega.Equal([]admissionregistrationv1.ValidationAction{admissionregistrationv1.Deny}), "updated object should have the applied spec") framework.Logf("waiting for watch events with expected annotations") for sawAnnotation := false; !sawAnnotation; { select { case evt, ok := <-vapbWatch.ResultChan(): if !ok { framework.Fail("watch channel should not close") } gomega.Expect(evt.Type).To(gomega.Equal(watch.Modified)) vapbWatched, isFS := evt.Object.(*admissionregistrationv1.ValidatingAdmissionPolicyBinding) if !isFS { framework.Failf("expected an object of type: %T, but got %T", &admissionregistrationv1.ValidatingAdmissionPolicyBinding{}, evt.Object) } if vapbWatched.Annotations["patched"] == "true" { sawAnnotation = true vapbWatch.Stop() } else { framework.Logf("missing expected annotations, waiting: %#v", vapbWatched.Annotations) } case <-time.After(wait.ForeverTestTimeout): framework.Fail("timed out waiting for watch event") } } ginkgo.By("deleting") err = client.Delete(ctx, vapbCreated.Name, metav1.DeleteOptions{}) framework.ExpectNoError(err) vapbTmp, err := client.Get(ctx, vapbCreated.Name, metav1.GetOptions{}) switch { case err == nil && vapbTmp.GetDeletionTimestamp() != nil && len(vapbTmp.GetFinalizers()) > 0: // deletion requested successfully, object is blocked by finalizers case err == nil: framework.Failf("expected deleted object, got %#v", vapbTmp) case apierrors.IsNotFound(err): // deleted successfully default: framework.Failf("expected 404, got %#v", err) } list, err = client.List(ctx, metav1.ListOptions{LabelSelector: label}) var itemsWithoutFinalizer []admissionregistrationv1.ValidatingAdmissionPolicyBinding for _, item := range list.Items { if len(item.GetFinalizers()) == 0 { itemsWithoutFinalizer = append(itemsWithoutFinalizer, item) } } framework.ExpectNoError(err) gomega.Expect(itemsWithoutFinalizer).To(gomega.HaveLen(2), "filtered list should have 2 items") ginkgo.By("deleting a collection") err = client.DeleteCollection(ctx, metav1.DeleteOptions{}, metav1.ListOptions{LabelSelector: label}) framework.ExpectNoError(err) list, err = client.List(ctx, metav1.ListOptions{LabelSelector: label}) var itemsColWithoutFinalizer []admissionregistrationv1.ValidatingAdmissionPolicyBinding for _, item := range list.Items { if !(item.GetDeletionTimestamp() != nil && len(item.GetFinalizers()) > 0) { itemsColWithoutFinalizer = append(itemsColWithoutFinalizer, item) } } framework.ExpectNoError(err) gomega.Expect(itemsColWithoutFinalizer).To(gomega.BeEmpty(), "filtered list should have 0 items") }) </ins> }) func createBinding(bindingName string, uniqueLabel string, policyName string) *admissionregistrationv1.ValidatingAdmissionPolicyBinding {

from: <img width="902" alt="image" src=""from:
/kind failing-test /area conformance /sig api-machinery /assign cc
/triage accepted


Question: encoder := codecs.EncoderForVersion(info.Serializer, gv) return runtime.Encode(encoder, obj) } <del> // MarshalToYamlForCodecsWithShift adds spaces in front of each line so the indents line up // correctly in the manifest func MarshalToYamlForCodecsWithShift(obj runtime.Object, gv schema.GroupVersion, codecs serializer.CodecFactory) (string, error) { serial, err := MarshalToYamlForCodecs(obj, gv, codecs) if err != nil { return "", err } lines := strings.Split(string(serial), "n") var newSerial string for _, line := range lines { newSerial = newSerial + "    " + line + "n" } return newSerial, err } </del>

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: MarshalToYamlForCodecs adds an indent to the yaml output. This code can be combined with PrintBytesWithLinePrefix What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/assign rpothier
/sig cluster-lifecycle
/priority important-soon


Question: KUBE_ROOT=$(dirname "${BASH_SOURCE}")/.. LATEST=$(kube::release::semantic_version) <del> KUBE_GCS_NO_CACHING=n KUBE_GCS_MAKE_PUBLIC=y KUBE_GCS_UPLOAD_RELEASE=y KUBE_GCS_DELETE_EXISTING=y KUBE_GCS_RELEASE_BUCKET=kubernetes-release </del> <ins> KUBE_GCS_NO_CACHING='n' KUBE_GCS_MAKE_PUBLIC='y' KUBE_GCS_UPLOAD_RELEASE='y' KUBE_GCS_DELETE_EXISTING='y' KUBE_GCS_RELEASE_BUCKET='kubernetes-release' </ins> KUBE_GCS_RELEASE_PREFIX="ci/${LATEST}" <del> KUBE_GCS_LATEST_FILE="ci/latest.txt" KUBE_GCS_LATEST_CONTENTS=${LATEST} </del> <ins> KUBE_GCS_PUBLISH_VERSION="${LATEST}" </ins> source "$KUBE_ROOT/build/common.sh" MAX_ATTEMPTS=3 attempt=0 while [[ ${attempt} -lt ${MAX_ATTEMPTS} ]]; do <del> kube::release::gcs::release && kube::release::gcs::publish_latest && break || true </del> <ins> kube::release::gcs::release && kube::release::gcs::publish_ci && break || true </ins> attempt=$((attempt + 1)) sleep 5 done

(cc Despite doing very, very similar things, doesn't rely on . It should, or the logic should otherwise be consolidated. I'm going to maybe work on this to make easier.


Question: // Status code 409 StatusReasonConflict StatusReason = "Conflict" <ins> // StatusReasonGone means the item is no longer available at the server and no // forwarding address is known. // Status code 410 StatusReasonGone StatusReason = "Gone" </ins> // StatusReasonInvalid means the requested create or update operation cannot be // completed due to invalid data provided as part of the request. The client may // need to alter the request. When set, the client may use the StatusDetails

We sorta fixed this-- now it's a real status error, but we left the status code 500, so it still produces stack traces and potentially confuses clients. we should figure out the correct status code.
I think there is a lot more going on related to this in . Clients are re-listing after the error but not rewatching.
/sub
It turned out this was not causing job e2e flakes. Just sending 500s that were confusing. So lowering priority.
How about a NewBadRequest error?
that certainly sounds like an improvement.


Question: fakeClient = fakeclient.NewSimpleClientset() } fakeWatcher := watch.NewRaceFreeFake() <del> fakeClient.Fake.AddWatchReactor("*", core.DefaultWatchReactor(fakeWatcher, nil)) </del> <ins> fakeClient.Fake.PrependWatchReactor("volumeattachments", core.DefaultWatchReactor(fakeWatcher, nil)) </ins> host := volumetest.NewFakeVolumeHostWithCSINodeName( tmpDir, fakeClient,

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: /sig storage /kind flake
/assign The fake watch is not set up correctly.


Question: for mountDevice, mappingVolumeID := range deviceMappings { if volumeID == mappingVolumeID { if assign { <del> klog.Warningf("Got assignment call for already-assigned volume: %s@%s", mountDevice, mappingVolumeID) </del> <ins> // DescribeInstances shows the volume as attached / detaching, while Kubernetes // cloud provider thinks it's detached. // This can happened when the volume has just been detached from the same node // and AWS API returns stale data in this DescribeInstances ("eventual consistency"). // Fail the attachment and let A/D controller retry in a while, hoping that // AWS API returns consistent result next time (i.e. the volume is detached). status := volumeStatus[mappingVolumeID] klog.Warningf("Got assignment call for already-assigned volume: %s@%s, volume status: %s", mountDevice, mappingVolumeID, status) return mountDevice, false, fmt.Errorf("volume is still being detached from the node") </ins> } return mountDevice, true, nil }

E2e tests occasionally flake on AWS. It happens randomly to all tests that create a pod with a volume, write something to the volume, delete the pod and create a new one that reads from the volume. For example: /D controller detaches a volume from a node (= the old pod died) after that (under 1s) it tries to attach the same volume to the same node (=the new pod is scheduled and coincidentally it uses the same node). In most cases this works, however, few times per day it happens that the second attach (in Kubernetes terms, see below) does not succeed and the test times out. AWS cloud provider in Kubernetes heavily uses DescribeInstances and DescribeVolumes to check that a volume has been attached or detached: In step 1. above, Kubernetes issues AWS API call and polls until it reports that the volume has been detached. In step 2. above, Kubernetes issues AWS API call to check what devices are used on a node and then it tries to find a free device for the volume that should be attached. In this bug, in step 2 reports the volume as detaching, even that reported the volume as detached in step 1. There was no call in between! Kubernetes does not expect that this could ever happen and does not try attach the volume (it's not reported as detached, so why bother, right?). This is the code that finds out that the volume is not detached ("detaching" != "detached"): And few lines below we skip : We (OpenShift) run lot of Kubernetes e2e tests on AWS. We run them in a single AWS project and I guess we hit some timing issues / races on AWS side. We hit also AWS API quota pretty hard, but it does not seem to be related to this bug, I mention it only to illustrate the amount of things that run in parallel and that AWS may not like that load. Environment: Kubernetes v1.13.4 /kind bug /sig aws /sig storage
Flow from AWS API perspective: So far so good. = volume vol- is back detaching: All timing is related to a response received, processed and logged by Kubernetes, which may be different to time corresponding request was sent. Everything in this report happens in sequence, there is no pararellism for a single volume.


Question: for mountDevice, mappingVolumeID := range deviceMappings { if volumeID == mappingVolumeID { if assign { <del> klog.Warningf("Got assignment call for already-assigned volume: %s@%s", mountDevice, mappingVolumeID) </del> <ins> // DescribeInstances shows the volume as attached / detaching, while Kubernetes // cloud provider thinks it's detached. // This can happened when the volume has just been detached from the same node // and AWS API returns stale data in this DescribeInstances ("eventual consistency"). // Fail the attachment and let A/D controller retry in a while, hoping that // AWS API returns consistent result next time (i.e. the volume is detached). status := volumeStatus[mappingVolumeID] klog.Warningf("Got assignment call for already-assigned volume: %s@%s, volume status: %s", mountDevice, mappingVolumeID, status) return mountDevice, false, fmt.Errorf("volume is still being detached from the node") </ins> } return mountDevice, true, nil }

(Other volumes may be and often are attached / detached in parallel to this flow).
this looks like a bug in AWS itself, seems to return old data randomly. Can you please check AWS side of things? And we probably need fix / workaround on Kubernetes side. How can we check that a volume has been 100% detached if returns inconsistent data? We can check if a volume is in and not report is as attached here: This will cause Kubernetes to re-use the same device name and call . How bulletproof is it going to be? And most importantly, what other parts of Kubernetes depend on being consistent?
I confirmed internally. This is expected behavior for and since ec2 API follows a where resource mutation might be not reflected immediately. This could explain why returns a volume in detaching state even after the previous call returns without the volume (detached) I think this won't work in the case when the volume is actually stuck in detaching state, since attaching a stuck volume could simply make it worse. Given that, how about, in call, we return error when notice the volume is still being returned from and we rely on the AD controller retry to check the state next time, and hopefully the state converges.
If can return stale data, depending on it in (or anywhere else) probably won't help, because we may again get new or stale data "randomly". We call in today to get fresh list of used/free devices. Depending on timing, two cases happen: Sometimes it returns that the volume is detaching and I can see in logs. Handling as you suggest would help here. Sometimes it returns that the volume is detached. We find a new device name and call AWS and start polling for finished attach. This then returns that the volume is detaching from the node with the old device name. So we're actually attaching a volume that's being detached and you wrote that's a bad thing. In my logs, I can see that Kubernetes recovers from this error: Volume plugin AttachVolume() fails with A/D controller re-tries AttachVolume() calls AttachDisk() that calls to find a free device. returns that the volume is actually being attached to the node (I can see in logs) AttachDisk() polls until the volume is attached. -it works, but to me it feels like it's just an accident. Checking can't help in this case.


Question: for mountDevice, mappingVolumeID := range deviceMappings { if volumeID == mappingVolumeID { if assign { <del> klog.Warningf("Got assignment call for already-assigned volume: %s@%s", mountDevice, mappingVolumeID) </del> <ins> // DescribeInstances shows the volume as attached / detaching, while Kubernetes // cloud provider thinks it's detached. // This can happened when the volume has just been detached from the same node // and AWS API returns stale data in this DescribeInstances ("eventual consistency"). // Fail the attachment and let A/D controller retry in a while, hoping that // AWS API returns consistent result next time (i.e. the volume is detached). status := volumeStatus[mappingVolumeID] klog.Warningf("Got assignment call for already-assigned volume: %s@%s, volume status: %s", mountDevice, mappingVolumeID, status) return mountDevice, false, fmt.Errorf("volume is still being detached from the node") </ins> } return mountDevice, true, nil }

So, is there another way how to check that a volume is really detached? Or should we promote the error flow described above to official fix and add bunch of comments around so we won't remove / tweak the flow in future? What bad can happen when we call AWS on a volume that's detaching? AWS at least once reported that the volume is detached, so the volume probably is not be stuck.
I think it is only bad if the volume was previously stuck. But feels acceptable for me if it is being detached and the previous call in at least confirmed once. Confirming internally. But I suspect there isn't any. As long as the volume is not stuck (unmounted properly), we should be able to attach a detaching volume especially when it is at least confirmed being detached once from call in . Even if the first time it doesn't work, retry should help recover the situation. How do you think?
According pointed, AWS recommends two actions to manage the eventual consistency model (k8s already does 1, but AFAIK not 2, except when there's an exponential backoff among calls: 1) Run the appropriate call before modifying a resource 2) Add wait time between subsequent commands, even if returns a green response (this does infer that might return stale data) Currently, k8s assumes a state based on a single green response, e.g., is called repeatedly with an exponential backoff until it returns a given volume state. What if it required at least two positive responses instead? What are the chances that would return stale data twice? I'd assume that two calls returning a detached volume (with exponential backoff in between, which enforces point 2 above) would be reliable enough. The downside is that this would increase the API usage, which is already high.
do you know if the native AWS calls may be more stale-data-proof?
Looking at implementation, it tries with constant backoff of 15 seconds (which I consider pretty large) and it returns when the first returns that the volume is in use, so basically the same as what Kubernetes does.
From what I saw, sometimes it happens that reports a volume as detached, subsequent reports that the volume is not attached, succeeds and only after that shows that the volume is (from the old device name) instead of . So Kubernetes got at least two confirmations that the volume is detached.


Question: for mountDevice, mappingVolumeID := range deviceMappings { if volumeID == mappingVolumeID { if assign { <del> klog.Warningf("Got assignment call for already-assigned volume: %s@%s", mountDevice, mappingVolumeID) </del> <ins> // DescribeInstances shows the volume as attached / detaching, while Kubernetes // cloud provider thinks it's detached. // This can happened when the volume has just been detached from the same node // and AWS API returns stale data in this DescribeInstances ("eventual consistency"). // Fail the attachment and let A/D controller retry in a while, hoping that // AWS API returns consistent result next time (i.e. the volume is detached). status := volumeStatus[mappingVolumeID] klog.Warningf("Got assignment call for already-assigned volume: %s@%s, volume status: %s", mountDevice, mappingVolumeID, status) return mountDevice, false, fmt.Errorf("volume is still being detached from the node") </ins> } return mountDevice, true, nil }

What would be the correct delay to wait for second (third, ...) confirmation that the volume is really detached? IMO it would hurt our performance a lot.
The Wait function simply calls DescribeXX under the hood and uses some hard coded timeout. So it's similar to what kubernetes does like said.


Question: b.plugin.volumeLocks.LockKey(b.pdName) defer b.plugin.volumeLocks.UnlockKey(b.pdName) <del> // TODO: handle failed mounts here. </del> notmnt, err := b.mounter.IsLikelyNotMountPoint(dir) if err != nil && !os.IsNotExist(err) { glog.Errorf("Cannot validate mount point: %s %v", dir, err)

while reading some cinder code, i came accross , we try to detach disk even if we didn't try to mount. Is this call mandatory ? (from what i recall AWS EBS do not have such mecanism) Also ping
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig openstack
cc
that's was before attach/detach controller, a revisit is necessary
Sure i'll take care of it /assign


Question: ctx, cancel := context.WithTimeout(context.Background(), 3600*time.Second) defer cancel() <ins> klog.V(2).Infof("Waiting for client certificate to be issued") </ins> return csr.WaitForCertificate(ctx, client, req) }

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Currently, during kubelet bootstrapping, while waiting for the node certificate being signed, there is no logs showing what is happening. I would like to add some logging to for easier troubleshooting. Why is this needed: When troubleshooting kubelet bootstrapping process on a node, if there is some issue with certificate signing mechanism, kubelet will wait for its certificate being signed for without a single line of log. Often, the last lines of kubelet log would look like the following, and kubelet will be in an confusing "infinite" wait. And then, some healthchecking monitor for kubelet may kick in and restart kubelet, resulting in a confusing loop. Adding some logging here will greatly help troubleshooting bootstrapping process on the node side. /sig node
The logging would be in the following func: I think we should control the frequency of logging (by remembering the timestamp of last log).


Question: // Group Pods by Node name. nodeToDaemonPods := make(map[string][]*v1.Pod) for _, pod := range claimedPods { <del> // Skip terminating pods if pod.DeletionTimestamp != nil { continue } </del> nodeName := pod.Spec.NodeName nodeToDaemonPods[nodeName] = append(nodeToDaemonPods[nodeName], pod) }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: The DaemonSet controller will schedule a pod on a node when the old pod is still in Terminating state. This breaks the expectation that only one pod will be running at a time per node. What you expected to happen: The DaemonSet controller should not let two pods running at the same time on the same node. How to reproduce it (as minimally and precisely as possible): Create a DaemonSet. Set terminationGracePeriodSeconds to be relatively long. Kill a pod. Watch a new pod start on the same node when the old pod is still terminating. Anything else we need to know?: Seems this behavior was introduced in Environment: Kubernetes version (use ): 1.7.2 Cloud provider or hardware configuration**: GKE OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig apps
/cc
cc
cc: I think the expected behavior is correct, the terminating pod could be performing an important step such as 'removing a lock' or 'cleanup' etc., we should allow that to gracefully die before creating a new one. IMHO we should revoke to old behavior, but why did we need this new behavior in the first place? That is not clear to me from the PR and Issue.
We don't currently guarantee at least or at most one scheduling for DaemonSet. Both behaviors are under consideration via issue and in the future work section of the . The reason for is to "Make sure that controllers are not adopting children that are marked for deletion" (). This is necessary for GC to work properly.
Well the behavior may be technically undefined, but can I request that in the future changes like this be mentioned in the release notes?
Yes, certainly.


Question: // Group Pods by Node name. nodeToDaemonPods := make(map[string][]*v1.Pod) for _, pod := range claimedPods { <del> // Skip terminating pods if pod.DeletionTimestamp != nil { continue } </del> nodeName := pod.Spec.NodeName nodeToDaemonPods[nodeName] = append(nodeToDaemonPods[nodeName], pod) }

Also "The DaemonSet controller should not let two pods running at the same time on the same node." is not necessarily the same thing as respecting termination prior to launching a new Pod. The former implies a more strict requirement than the latter. I think we should consider at least attempting to wait for a terminating Pod to be deleted prior to creating another Pod on the Node for same DaemonSet as default behavior.
will something like work then?


Question: cmd.Flags().Var(&options.password, clientcmd.FlagPassword, clientcmd.FlagPassword+" for the user entry in kubeconfig") cmd.Flags().Var(&options.authProvider, flagAuthProvider, "Auth provider for the user entry in kubeconfig") cmd.Flags().StringSlice(flagAuthProviderArg, nil, "'key=value' arguments for the auth provider") <ins> cmd.Flags().Var(&options.execCommand, flagExecCommand, "Command for the exec credential plugin for the user entry in kubeconfig") cmd.Flags().Var(&options.execAPIVersion, flagExecAPIVersion, "API version of the exec credential plugin for the user entry in kubeconfig") cmd.Flags().StringSlice(flagExecArg, nil, "New arguments for the exec credential plugin command for the user entry in kubeconfig") cmd.Flags().StringArray(flagExecEnv, nil, "'key=value' environment values for the exec credential plugin") </ins> f := cmd.Flags().VarPF(&options.embedCertData, clientcmd.FlagEmbedCerts, "", "Embed client cert/key for the user entry in kubeconfig") f.NoOptDefVal = "true"

Now it's possible to configure third-party auth providers using command with option. It's nice to have the same possibility for the recently . Example: To specify the executable path we can run This will generate a new config for the user: Various auxiliary configuration options are also could be . For instance: to define an argument for the executable; to set an environment variable; to specify a desired API version. /kind feature
I'm going to start working on this when the API is promoted to beta.
/sig auth
/assign
Can someone share why you can't do : Is there a way to set map values (e.g. something that is -name: value ; versus something that is -context: )
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
I would like to work on it, if this is still persist.
cc
This still seems useful and is outstanding. any objections to taking it over?
No, it's okay, I'm working on it. I'll post the code this week, sorry for the delay
I think this would help out with generating valid kubectl configurations for our EKS developers. I'm using Pulumi to build EKS environments and if all the flags are available, I think I can generate the necessary entries to assume a role. Thanks! Ex: name: k8s.developer user: exec: apiVersion: args: token -i cluster-eksCluster- -r arn:aws:iam:::role/k8sDevAdminRole- command: aws-iam-authenticator env: null
done :)
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
/priority important-longterm


Question: FilterFunc: func(obj interface{}) bool { switch t := obj.(type) { case *v1.Pod: <del> return unassignedNonTerminatedPod(t) </del> <ins> return unassignedNonTerminatedPod(t) && responsibleForPod(t, schedulerName) </ins> case cache.DeletedFinalStateUnknown: if pod, ok := t.Obj.(*v1.Pod); ok { <del> return unassignedNonTerminatedPod(pod) </del> <ins> return unassignedNonTerminatedPod(pod) && responsibleForPod(pod, schedulerName) </ins> } runtime.HandleError(fmt.Errorf("unable to convert object %T to *v1.Pod in %T", obj, c)) return false

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Scheduler's pod informer has a selector that . This is incorrect. Scheduler must watch all the pods of the cluster, including those scheduled by other schedulers in order to update its cache of the state of the cluster. Otherwise, it will miss important events and the amount of available resources, pods on nodes, ... which are needed for scheduler to work correctly, will not be updated properly. /sig scheduling /assign
cc
For updating it cache, shouldn't the default scheduler rely on node info/status? Also its not very clear, what information the default scheduler needs from other pods (not to be scheduled by it) that it makes decision on? Just trying to understand the issue more clearly.
Scheduler watches informer events to update its cache. Node status is not enough to update all the information that scheduler caches. Scheduler must know what pods are placed on what nodes. For example, in order to check inter-pod affinity/anti-affinity it needs to know the placement of all the pods, including those scheduled by other schedulers.
But this information is present here: So it seems that your PR will now make sure to get the scheduled pods' information directly from the pod informer.
So when I check the history, I see it was here:
have we seen any specific issues due to this change (like between default scheduler and DS controller)?
Yes, but scheduler relies on its pod informers to keep this location up to date. That's exactly why it is important for the scheduler to receive updates for all the pods in the cluster. Yes. We missed the problem at the time. This is problematic only in clusters that use multiple schedulers. I initially thought that DS controller could cause issues as well, but DS controller does not set , so it must not cause an issue. We have seen instances of scheduler's missing certain events in cluster and as a result scheduler's cache getting stale, but I doubt if it is related to this particular issue. This only affects clusters with more than one scheduler. If you run the test that I have written in with the head, it will fail. That tests simulate a multi-scheduler scenario.


Question: return nil, fmt.Errorf("conversion for %v returned %d objects, expected %d", in.GetObjectKind().GroupVersionKind(), len(convertedObjects.Items), len(objectsToConvert)) } <del> // start a deepcopy of the input and fill in the converted objects from the response at the right spots. </del> <ins> // Fill in the converted objects from the response at the right spots. </ins> // The response list might be sparse because objects had the right version already. <del> convertedList := list.DeepCopy() convertedList.SetAPIVersion(desiredAPIVersion) </del> convertedIndex := 0 for i := range convertedList.Items { original := &convertedList.Items[i]

If a CR converter is wired in and it modifies the input list, the delegating CR converter does not protect against this. A converter cannot effect the input list Not really possible externally - it's a coding issue that the current converters don't hit. No response <details</details<detailsN/A </details<details</details<details</details<details</details<details</details
/sig api-machinery /assign
/triage accepted


Question: } } <ins> func TestGetZoneByProviderID(t *testing.T) { tests := []struct { providerID string expectedZone cloudprovider.Zone fail        bool description string }{ { providerID:   ProviderName + "://project-example-164317/us-central1-f/kubernetes-node-fhx1", expectedZone: cloudprovider.Zone{FailureDomain: "us-central1-f", Region: "us-central1"}, fail:         false, description:  "standard gce providerID", }, { providerID:   ProviderName + "://project-example-164317/us-central1-f/kubernetes-node-fhx1/", expectedZone: cloudprovider.Zone{}, fail:         true, description:  "too many slashes('/') trailing", }, { providerID:   ProviderName + "://project-example.164317//kubernetes-node-fhx1", expectedZone: cloudprovider.Zone{}, fail:         true, description:  "too many slashes('/') embedded", }, { providerID:   ProviderName + "://project-example-164317/uscentral1f/kubernetes-node-fhx1", expectedZone: cloudprovider.Zone{}, fail:         true, description:  "invalid name of the GCE zone", }, } gce := &GCECloud{ localZone: "us-central1-f", region:    "us-central1", } for _, test := range tests { zone, err := gce.GetZoneByProviderID(test.providerID) if (err != nil) != test.fail { t.Errorf("Expected to fail=%t, provider ID %v, tests %s", test.fail, test, test.description) } if test.fail { continue } if zone != test.expectedZone { t.Errorf("Expected %v, but got %v", test.expectedZone, zone) } } } </ins> func TestGenerateCloudConfigs(t *testing.T) { configBoilerplate := ConfigGlobal{ TokenURL:           "",

Is this a BUG REPORT or FEATURE REQUEST?: As part of the initiative for external cloud providers (), we GetZoneByProviderID and GetZoneByNodeName. See for more details. The previous PR did not actually implement those methods for existing cloud providers in . These methods should be implemented by all cloud providers in the future. cc
Could someone add the label and the label for this issue please?
/sig cluster-lifecycle
/area cloudprovider
/assign wlan0
cc We need the following label - help-wanted and add it to the 1.8(?) milestone cc to implement it for their respective clouds.
I'll try to have a look by the end of the week.
Ok, I will take care of the Rackspace and OpenStack.
Thank you !
- Is there any listing where to see the already implemented cloud provider ?
No providers have this yet. Current PRs: Openstack: Rackspace: Please contribute if you have something ready :)
Sorry for the delay guys :)
Have you implemented them for gce?
Intending to work on it tonight, but if you have something I can work on azure
Yes, I have prepared for GCE.
Azure .
Cool, than you all! Any other providers we should implement this for in time for v1.8?
I'll take a look tomorrow to see if there's something left
Let's see which providers are left before closing this...
- I guess this one should be closed ?
If you say that all providers have this now, thanks, great!


Question: st.MakePod().Name("p1").UID("p1").Node("node1").Priority(midPriority).Obj(), st.MakePod().Name("p2").UID("p2").Node("node2").Priority(midPriority).Obj(), }, <del> expected: [][]Candidate{ { &candidate{victims: &extenderv1.Victims{}, name: "node1"}, &candidate{victims: &extenderv1.Victims{}, name: "node2"}, }, }, expectedNumFilterCalled: []int32{4}, </del> <ins> expected:                [][]Candidate{{}}, fakeFilterRC:            framework.Unschedulable, expectedNumFilterCalled: []int32{2}, </ins> }, { name: "a pod that fits on one node with no preemption",

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --Kubernetes 1.19 (confirmed with 1.19.7 and 1.19.10) kube-scheduler crashes and restarts with the following errors in . panic: runtime error: index out of range [0] with length 0 The problem code is line 389. When is nil or is empty, the error happens. If we skip when it's nil or empty, the error is gone. panic: runtime error: invalid memory address or nil poster dereference The problem is does not exist and returns nil sometimes. Simply skipping it won't solve the problem. The scheduler will reach either line 456 or 341. The scheduler works without failures. When there are preemptions in a cluster. Kubernetes version (use ): v1.19.10 or v1.19.7 Cloud provider or hardware configuration: bare metal OS (e.g: ): centos 7 Kernel (e.g. ): Linux 5.4.77-7.el7pie SMP Sat Nov 21 01:16:27 UTC 2020 x8664 x8664 x8664 GNU/Linux Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scheduling
/cc
/assign
Are you running the vanilla scheduler? or a customized one that leverages the utilities in
It's a custom scheduler with some internal out-of-tree plugins (just like ), but it doesn't have any custom postFilter plugins. Which utility functions are you referring to? Thanks.
are those custom plugins filter plugins? do they maintain state?


Question: st.MakePod().Name("p1").UID("p1").Node("node1").Priority(midPriority).Obj(), st.MakePod().Name("p2").UID("p2").Node("node2").Priority(midPriority).Obj(), }, <del> expected: [][]Candidate{ { &candidate{victims: &extenderv1.Victims{}, name: "node1"}, &candidate{victims: &extenderv1.Victims{}, name: "node2"}, }, }, expectedNumFilterCalled: []int32{4}, </del> <ins> expected:                [][]Candidate{{}}, fakeFilterRC:            framework.Unschedulable, expectedNumFilterCalled: []int32{2}, </ins> }, { name: "a pod that fits on one node with no preemption",

If yes, then one hypothesis is that those custom filters filter the node in the filter phase, but not when executed in the preemption phase, and so this will result in a candidate node with no victims?
Yes, it includes filter plugins and uses cycle state.
you need to make sure that those filter plugins play nicely when executed again in the preemption phase in the same cycle: i.e., produce the same result.
Thanks What do you mean by performing filter in the preemption phase? would you mind elaborating it a little?
What additional logic is needed to handle the preemption case?
in the preemption phase we run the filters again to check if removing lower priority pods make the node schedulable: we later add them one by one to reduce the set of victim pods to the absolute minimum: My hypothesis is that somehow your filter returns success when adding all the pods back, and so you end up with zero victims and a candidate node. This shouldn't happen because if the pod fits the node without removing any victims, then we shouldn't be running preemption in the first place. So the theory is that the custom filter is returning false when run in the filter phase, but returns true when executed in the preemption phase perhaps because it makes some assumptions about cyclestate that makes it behave this way (e.g. that a the filter will be executed once per node in a scheduling cycle).
Disabled the filter plugin, but still see the same issues.
are you able to change the scheduler code and run the test again? I can send a patch tomorrow to add some debugging messages to help us root cause the issue.
yes, thanks a lot! really appreciate it!
You are right. We still use a deprecated Predicate extender, which causes the problem. After disabling it, it's working fine. It used to work with 1.18 though. We are retiring it. Thank you so much!!!
If the test cases can help debug scheduler extender Webhook (Predicate) too, I'd like to try it. Thanks again!
How can we improve the preemption logic to prevent bugs and issues in custom Filters or Predicates crashing the scheduler?
for starter, we should check the list len, the scheduler should not panic in all cases.


Question: st.MakePod().Name("p1").UID("p1").Node("node1").Priority(midPriority).Obj(), st.MakePod().Name("p2").UID("p2").Node("node2").Priority(midPriority).Obj(), }, <del> expected: [][]Candidate{ { &candidate{victims: &extenderv1.Victims{}, name: "node1"}, &candidate{victims: &extenderv1.Victims{}, name: "node2"}, }, }, expectedNumFilterCalled: []int32{4}, </del> <ins> expected:                [][]Candidate{{}}, fakeFilterRC:            framework.Unschedulable, expectedNumFilterCalled: []int32{2}, </ins> }, { name: "a pod that fits on one node with no preemption",

Other than that, I think we need to clearly document what the preemption logic does (that it executes the prefilter extensions points and the filter plugins multiple times) and the expectations from filter plugins (that they may get executed more than once in the same cycle)
Is it a predicate extender or preemption extender? If it's a predicate extender, I don't quite think we found the root cause - b/c predicate extenders are not invoked during preemption (). It makes more sense if it's a preemption extender - if the preemption extender mutates the to make it invalid. If it's the case, could you verify it by adding debug info before and after L157 to check the validness of ?
That's always true :)
filed a PR to mitigate it.
It's a predicate extender. I'll debug it more.
Added some debug info. after and . Here's the result. returned 4 candidates, but one of candidates () has zero pod. The candidate happened to be the best candidate and hence caused the "" error. It should have nothing to do with since our extender doesn't support Preemption. Disabling it may change pod placements/candidates selection and bypassed the error. It's like a coincidence. To remove the panic, a simple fix is to check if is empty. If it is, do not add the node to candidates. Alternatively, a change to may work too. Here is the log with debug info.
This looks more promising. Back to digging into the root cause, it seems returns an empty but with . So that implies firstly we have a non-nil , and then in , the is separated into two list: violatingVictims and nonViolatingVictims. Finally the two victims list are tried one by one to run . It looks like returns {fits=true, nil error} for every victim, which is quite abnormal. I think it's problematic that every pod is retrievable and in which we can debug into.
Yes, should not return fit=true with empty pods in the first place, but it's hard to locate the problematic code in this function.
The following code in looks questionable to me. If is empty, should the function return nil, nil? Also, if is true for all extenders, the loop won't do anything and candidates will remain the same.


Question: st.MakePod().Name("p1").UID("p1").Node("node1").Priority(midPriority).Obj(), st.MakePod().Name("p2").UID("p2").Node("node2").Priority(midPriority).Obj(), }, <del> expected: [][]Candidate{ { &candidate{victims: &extenderv1.Victims{}, name: "node1"}, &candidate{victims: &extenderv1.Victims{}, name: "node2"}, }, }, expectedNumFilterCalled: []int32{4}, </del> <ins> expected:                [][]Candidate{{}}, fakeFilterRC:            framework.Unschedulable, expectedNumFilterCalled: []int32{2}, </ins> }, { name: "a pod that fits on one node with no preemption",

Why do we bother creating a new candidates later? Can it just return candidates?
It can prevent from returning candidates with empty pods, but what about ? It's possible to add nodes without victim pods to candidates if returned by contain such nodes. How about adding the following check to CallExtender?
Updated the PR based on the finding and discussions.
Yes, we can use a flag to mark if all extenders don't support preemption. If yes, simply return the candidates.
Technically this can exist due to faulty extender implementation. If we really want to guard it, instead of logging the error and continue, I'm more inclined to return the error immediately as this is a fatal error - the victimsMap cannot be used either for latter extender or further preemptor nominating.
Is my understanding correct? In , if victimsMap is empty, should it return instead of ?
The only case victimsMap can be empty is empty candidates, and so return either doesn't quite matter, right?
You are right.
As long as one extender returns an invalid victimsMap (with empty pods), s returns an error immediately? What about ? Should it return an error immediately if it returnswith empty pods?
I think so. Because the result it returned is faulty, and we don't want to continue based on it The same. We may don't want to continue the scheduling cycle based on faulty result.


Question: By("Confirming that stateful set scale up will halt with unhealthy stateful pod") sst.BreakProbe(ss, testProbe) sst.WaitForRunningAndNotReady(*ss.Spec.Replicas, ss) <ins> sst.WaitForStatus(ss, 0) </ins> sst.UpdateReplicas(ss, 3) sst.ConfirmStatefulPodCount(1, ss, 10*time.Second)

https://k8s- do you know the statefulset owners?
cc
I did not see this happening until recently.
The controller misses that the first pod transitioned to unready and proceeds to create the second pod. The test uses a client to hit the api server and will scale up the statefulset as soon as the broken pod is observed but the controller uses caches so it may observe the scaled up statefulset quicker than the updated pod.
I will update the test to ensure that we wait until the unreadiness is observed on StatefulSet.Status.Replicas field prior to the scaling operation
SGTM


Question: expected := `# HELP kube_pod_resource_limit [ALPHA] Resources limit for workloads on the cluster, broken down by pod. This shows the resource usage the scheduler and kubelet expect per pod for resources along with the unit for the resource if any. # TYPE kube_pod_resource_limit gauge kube_pod_resource_limit{namespace="test",node="node-one",pod="foo",priority="",resource="custom",scheduler="",unit=""} 6 <del> kube_pod_resource_limit{namespace="test",node="node-one",pod="foo",priority="",resource="memory",scheduler="",unit="bytes"} 2e+09 </del> <ins> kube_pod_resource_limit{namespace="test",node="node-one",pod="foo",priority="",resource="memory",scheduler="",unit="bytes"} 2.68435456e+09 </ins> # HELP kube_pod_resource_request [ALPHA] Resources requested by workloads on the cluster, broken down by pod. This shows the resource usage the scheduler and kubelet expect per pod for resources along with the unit for the resource if any. # TYPE kube_pod_resource_request gauge kube_pod_resource_request{namespace="test",node="node-one",pod="foo",priority="",resource="cpu",scheduler="",unit="cores"} 2

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --I found that for with returns a very small value. The value should be almost the same as the one of . My minimal : Result: While the top three lines () are good, the float values in the bottom three lines () might be broken. Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig api-machinery
/assign
/cc /triage accepted


Question: "client_config.go", "config.go", "doc.go", <ins> "flag.go", </ins> "helpers.go", "loader.go", "merged_client_builder.go",

Because always return names with a prefix like , it cannot be directly used to iterate over all namespaces to execute other commands. I've compiled from source: Reviewing some scripts, I came across a usage of like this: That line fetches namespace names to later run in a loop: That seemed to be a perfect use case for the flag, but it can't be used without extra hacks to get rid of the prefix (FWIW, is invalid). Changing the current output of might not be possible or desirable, for this could impact existing users that rely on the output format. If I'm wrong about the above limitation, than it would make sense to output just the name when and there is a single resource type, while using the ideas introduced in to show the resource type only when there is more than one type in the output, or when is explicitly set. Before: After:
could you help me ping the appropriate team to triage this? Thanks!
Please re-post your question to . We are trying to consolidate the channels to which questions for help/support are posted so that we can improve our efficiency in responding to your requests, and to make it easier for you to find answers to frequently asked questions and how to address common use cases. We regularly see messages posted in multiple forums, with the full response thread only in one place or, worse, spread across multiple forums. Also, the large volume of support issues on github is making it difficult for us to use issues to identify real bugs. The Kubernetes team scans stackoverflow on a regular basis, and will try to ensure your questions don't go unanswered. Before posting a new question, please search stackoverflow for answers to similar questions, and also familiarize yourself with: Again, thanks for using Kubernetes. The Kubernetes Team
perhaps I didn't do a good job at explaining the problem, but this is not a question of how to do something, but a report on a problem with the CLI design.
Sorry about that :-/. I'll try to route to the proper team.
update: console examples to the description.
the -o format is designed for scripting, the use case is just different from yours. You should be able to paste the output of into . Does do what you want? If this is a common use case we could create another -o flag value.


Question: "client_config.go", "config.go", "doc.go", <ins> "flag.go", </ins> "helpers.go", "loader.go", "merged_client_builder.go",

the use cases I've observed doing code reviews: over all namespaces to get certain resources from that namespace: over all namespaces, then iterate of over resource types that produce logs and fetch logs one by one: Above we can use for listing pods because later we'll pass it to that accept the syntax. We cannot do the same for namespaces, because doesn't like the format.
Alternatively we could be kubectl recognize as valid
I'll take a look on it if it's not taken by anyone yet. CC
Have you started this?
Only investigated due to being out of office since Thursday last week. I can work on it in upcoming week, but I'll understand if it's piority issue and you'll reassign it.
Ready :-)
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list .
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: // Add entries to the map for both the current and persistent portals // pointing to the SCSI host for those connections <del> portal := strings.TrimSpace(string(addr)) + ":" + strings.TrimSpace(string(port)) </del> <ins> // JoinHostPort will add `[]` around IPv6 addresses. portal := net.JoinHostPort(strings.TrimSpace(string(addr)), strings.TrimSpace(string(port))) </ins> portalHostMap[portal] = hostNumber <del> persistentPortal := strings.TrimSpace(string(persistentAddr)) + ":" + strings.TrimSpace(string(persistentPort)) </del> <ins> persistentPortal := net.JoinHostPort(strings.TrimSpace(string(persistentAddr)), strings.TrimSpace(string(persistentPort))) </ins> portalHostMap[persistentPortal] = hostNumber } }

Kubelet was not able to attach + mount an iSCSI PV when it has IPv6 server address. Volume is mounted + attached. Set up iSCSI target that's reachable over IPv6 and create a PV for it. For example: Kubelet logs: Notice around the IP address. Various iSCSI files in /sys and /dev/disk/by-path do not have these square brackets and kubelet is then not able to find the iSCSI host in /sys. That's where comes from. No response <detailsA recent master: console NAME="Fedora Linux" VERSION="36 (Workstation Edition)" </details<detailshack/local-up- </details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage /priority important-soon
/assign


Question: var err error if useRealProxier { <ins> nodeIP := utilnode.GetNodeIP(client, nodeName) if nodeIP == nil { klog.V(0).Infof("can't determine this node's IP, assuming 127.0.0.1") nodeIP = net.ParseIP("127.0.0.1") } </ins> // Real proxier with fake iptables, sysctl, etc underneath it. //var err error proxier, err = iptables.NewProxier(

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Using Kubernetes 1.16.1, I started kube-proxy in iptables mode as a static pod on a master node. The master node was intentionally (and permanently) not available in the API as a Node object. kube-proxy started up and terminated soon after because it failed to (which is supposed to function as the bound address) and eventually returned an error: What you expected to happen: I expected the same behavior as in Kubernetes 1.15 where failure to look up the Node in the API would result in . How to reproduce it (as minimally and precisely as possible): Configure kube-proxy with iptables mode on Kubernetes 1.16 without the hosting Node being available in the Kubernetes API (under the OS hostname). Anything else we need to know?: PR (Fix for kube-proxy to wait for some duration for the node to be defined) introduced an additional . I believe this is the part that is causing the change in behavior: in Kubernetes 1.15, the check would be omitted and cause to set the default bind address to 127.0.0.1 (when iptables mode is set; I haven't checked other modes). The PR's primary purpose seems to be a different one (retrying node IP retrieval from the API), so my understanding is that the check was not really needed; unless I'm misunderstanding things which I don't want to rule out at all. The quick workaround was to explicitly configure kube-proxy with . However, I'm not entirely sure if that's a good solution long-term compared to letting Kubernetes do The Right Thing, which is what I'd like to clarify through this issue. Environment: Kubernetes version (use ): v1.16.1 Cloud provider or hardware configuration: Kubernetes running on DigitalOcean VMs (droplets) OS (e.g: ): Debian GNU/Linux 9 (stretch) Kernel (e.g.


Question: var err error if useRealProxier { <ins> nodeIP := utilnode.GetNodeIP(client, nodeName) if nodeIP == nil { klog.V(0).Infof("can't determine this node's IP, assuming 127.0.0.1") nodeIP = net.ParseIP("127.0.0.1") } </ins> // Real proxier with fake iptables, sysctl, etc underneath it. //var err error proxier, err = iptables.NewProxier(

): Linux master-k8s-1-16-test1 4.19.0-0.bpo.6-amd64 SMP Debian 4.19.67-2+deb10u1~bpo9+1 (2019-09-30) x86_64 GNU/Linux /sig network /cc who were involved in the referenced PR.
I made a similar comment in , wondering if this is a breaking change in v1.16 that we should revert? Previously if is nil, kube-proxy defaults to but now kube-proxy always exits with error instead of defaulting to local host. I think a lot of external cloud providers relied on this behavior because they needed to talk to API server prior to node addresses being set by the provider. Thoughts /assign
Opened to preserve behavior before v1.16 where node IP defaults to 127.0.01 is none is found. PTAL


Question: containerDetails map[string]*docker.Container expectedRemoved  []string }{ <ins> // Remove oldest containers. </ins> { containers: []docker.APIContainers{ {

Successfully ran containers with "onFailure" restartPolicy are restarting after purging oldest docker containers. Kubelet purges/removes oldest containers periodically from docker. After purging, we loose the exit code and history of previously ran containers and restart containers as if they were never run.
Thanks & I can help fix this. Let me know. Thanks
Thanks for reporting Just to make sure I understand the issue: You ran some containers with an "onFailure" restartPolicy. When the Kubelet purged old containers the container was restarted (unexpectedly).
Ah, after reading again I think the issue is that after the containers are purged, when the container exits successfully it get restarted although it shouldn't. Is that correct?
After reading through the code I found a bug in the implementation that is most likely causing this error. Should have a PR soon.
Yes. Correct. After initial container run, it should never be run again, since it already ran to completion. Will wait for your PR.
PR is out. This should fix the issue. Feel free to test if you can :)


Question: job := &batchv1.Job{ ObjectMeta: metav1.ObjectMeta{ <del> Labels:          labels, Annotations:     annotations, Name:            name, OwnerReferences: []metav1.OwnerReference{*metav1.NewControllerRef(cj, controllerKind)}, </del> <ins> Labels:            labels, Annotations:       annotations, Name:              name, CreationTimestamp: metav1.Time{Time: scheduledTime}, OwnerReferences:   []metav1.OwnerReference{*metav1.NewControllerRef(cj, controllerKind)}, </ins> }, } cj.Spec.JobTemplate.Spec.DeepCopyInto(&job.Spec)

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Cronjob should include the last time when a job was successfully completed. Why is this needed: Currently, a Cronjob's only includes the last time when a job was scheduled. This makes monitoring cronjobs much more complicated, because instead of simply comparing to the current timestamp, the monitoring system now has to hunt for the last job related to the cronjob and persist the state somewhere.
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I'd be interesting in picking this up if the apps team thinks this is a good ticket - it seems like it's fairly beginner friendly.
Hi how is this going on? I could take it over if you don't have the time for this issue. :)
Yes please do! I meant to update that I actually wasn't able to pick this up, but it slipped my mind
/assign
Hi do you have any thoughts on this feature?
/unassign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Any updates?
Reiterating the mentions to trigger a notification: <detailsIn response to : Any updates? Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Trying to understand the usage of lastSuccessfulTime. In a normally working case, lastSuccessfulTime lastScheduledTime =last job completed lastSuccessfulTime < lastScheduledTime =last job completed, but a newer one is running Is that the expectation ?
We wanted to use it in monitoring (e.g. fire an alert if is more than X ago). could also mean that the job has been failing for some time, so I don't know what would you get out of that. I guess you could also use as a proxy for "job has failed at least once"?


Question: job := &batchv1.Job{ ObjectMeta: metav1.ObjectMeta{ <del> Labels:          labels, Annotations:     annotations, Name:            name, OwnerReferences: []metav1.OwnerReference{*metav1.NewControllerRef(cj, controllerKind)}, </del> <ins> Labels:            labels, Annotations:       annotations, Name:              name, CreationTimestamp: metav1.Time{Time: scheduledTime}, OwnerReferences:   []metav1.OwnerReference{*metav1.NewControllerRef(cj, controllerKind)}, </ins> }, } cj.Spec.JobTemplate.Spec.DeepCopyInto(&job.Spec)

Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten
that's exactly what is being asked here. /close
Closing this issue. <detailsIn response to : /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
"successfully scheduled" != "successfully completed", and most of the users care more about the latter.
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign /triage accepted
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at .


Question: job := &batchv1.Job{ ObjectMeta: metav1.ObjectMeta{ <del> Labels:          labels, Annotations:     annotations, Name:            name, OwnerReferences: []metav1.OwnerReference{*metav1.NewControllerRef(cj, controllerKind)}, </del> <ins> Labels:            labels, Annotations:       annotations, Name:              name, CreationTimestamp: metav1.Time{Time: scheduledTime}, OwnerReferences:   []metav1.OwnerReference{*metav1.NewControllerRef(cj, controllerKind)}, </ins> }, } cj.Spec.JobTemplate.Spec.DeepCopyInto(&job.Spec)

/lifecycle rotten
This is described in and being actively implemented in /close
Closing this issue. <detailsIn response to : This is described in and being actively implemented in /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
It doesn't look like it was actually implemented, though. Searching for only shows API changes, there don't seem to be any changes to the CronJob controller logic. Or am I missing something?
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign /remove-lifecycle rotten