Question: import ( "context" <ins> "encoding/json" </ins> "fmt" "math/rand" "time"

[x] Original approval issue: [x] Test pr: [x] Two weeks soak end date: 19 November 2020 This issue is created to allow edit by /wip /hold According to this APIsnoop query, there are still some remaining Deployment endpoints which are untested. a Deployment with a static label the Deployment with a new Label and updated data the Deployment to ensure it's patched all Deployments in all Namespaces find the Deployment(1) ensure that the Deployment is found and is patched Namespaced Deployment(1) via a Collection with a LabelSelector Discover useragents: List endpoints hit by the test: Display endpoint coverage change: If a test with these calls gets merged, test coverage will go up by 5 points* This test is also created with the goal of conformance promotion.
/sig testing /sig architecture /area conformance
BobyMcbobs commented in original Issue: Unsure if the endpoint replaceAppsV1NamespacedDeploymentStatus is able to be hit. When this endpoint is attempted to be updated the data after retrieving the update resource doesn't reflect the values in the update
/assign
When using the appv1.Deployment structure instead of maps of interfaces to build the structure in a patch of the Deployment, this error comes up. Still in progress yment-static":"true"}: does not match template , spec.selector: Invalid value: "null": field is immutable]
replaceAppsV1NamespacedDeploymentStatus to test. Increase cover by 1 Endpoint
/reopen PR reverted by due to flaky test
Reopened this issue. <detailsIn response to : /reopen PR reverted by due to flaky test Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/reopen Soaking, ready for promotion 19 November 2020
Reopened this issue. <detailsIn response to : /reopen Soaking, ready for promotion 19 November 2020 Instructions for interacting with me using PR comments are available .


Question: import ( "context" <ins> "encoding/json" </ins> "fmt" "math/rand" "time"

If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-lifecycle stale
/triage accepted
/merged


Question: WithStdinData("abcd1234n"). ExecOrDie() gomega.Expect(runOutput).ToNot(gomega.ContainSubstring("stdin closed")) <del> g = func(pods []*v1.Pod) sort.Interface { return sort.Reverse(controller.ActivePods(pods)) } runTestPod, _, err = polymorphichelpers.GetFirstPod(f.ClientSet.CoreV1(), ns, "run=run-test-3", 1*time.Minute, g) </del> <ins> g := func(pods []*v1.Pod) sort.Interface { return sort.Reverse(controller.ActivePods(pods)) } runTestPod, _, err := polymorphichelpers.GetFirstPod(f.ClientSet.CoreV1(), ns, "run=run-test-3", 1*time.Minute, g) </ins> gomega.Expect(err).To(gomega.BeNil()) if !e2epod.CheckPodsRunningReady(c, ns, []string{runTestPod.Name}, time.Minute) { e2elog.Failf("Pod %q of Job %q should still be running", runTestPod.Name, "run-test-3")

The test case "executing a command with run and attach without stdin" is inherently racy. The test runs, and expected to see in the output. The case when the test passes: case 1: Kubernetes creates and starts the container without stdin; The container exits before attach runs; kubectl falls back to get container log, which contains . case 2: Kubernetes creates and starts the container without stdin; kubectl attach before "stdin closed" is printed; kubectl attach contains . The case when the test fails: Kubernetes creates and starts the container without stdin; The container prints "stdin closed", but hasn't exited yet; kubectl attach contains nothing. One example failure: Related issue: Can we modify this test not to check the output? Or use to check output instead? /cc
Uploaded to try to address
/reopen This test is still inherently racy, even after
Fix in (I hope!)
Moving into the current milestone as it was not fixed in 1.14
Hello I'm a bug triage shadow for the 1.16 release cycle and considering this issue is tagged for 1.16, I'd like to check its status. Code freeze is starting on August 29th (about 3 days from now). Do we still target this issue to be fixed for 1.16?
Thanks . PR is available and awaiting review!
What is the PR in question? Is it targeting 1.16 or 1.17?
/reopen This test is having a lot of failures these days
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
is right, it seems to be present in kind jobs now only
/assign let's see if we start to recover a bit of stability :smile:
The rate of failures decreased a lot since last changes Checking the logs of the failures it seems that the problem is that the pod has not finished before checking the output, as Lantao described The test fails Collecting events from namespace The framework logs the pods the kubelet thinks is on the node
note that it's still racy, but it's hopefully somewhat mitigated.
/reopen
Reopened this issue.


Question: WithStdinData("abcd1234n"). ExecOrDie() gomega.Expect(runOutput).ToNot(gomega.ContainSubstring("stdin closed")) <del> g = func(pods []*v1.Pod) sort.Interface { return sort.Reverse(controller.ActivePods(pods)) } runTestPod, _, err = polymorphichelpers.GetFirstPod(f.ClientSet.CoreV1(), ns, "run=run-test-3", 1*time.Minute, g) </del> <ins> g := func(pods []*v1.Pod) sort.Interface { return sort.Reverse(controller.ActivePods(pods)) } runTestPod, _, err := polymorphichelpers.GetFirstPod(f.ClientSet.CoreV1(), ns, "run=run-test-3", 1*time.Minute, g) </ins> gomega.Expect(err).To(gomega.BeNil()) if !e2epod.CheckPodsRunningReady(c, ns, []string{runTestPod.Name}, time.Minute) { e2elog.Failf("Pod %q of Job %q should still be running", runTestPod.Name, "run-test-3")

<detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
any updates here?
nope, totally agree with the description, is racy by design, couldn't find any way or trick to avoid the randomness
EDIT: It's late and I'm actually not sure I'm remembering this correctly, take this comment with a heap of salt: Specific sub-tests were fixed previously I think, but this one sub-case: That one we can't use the "Wait for stdin input to signal we're attached" hack, so it's just racy and broken. We could remove that case maybe. I'm not sure who I'd want to weigh in on that, probably someone more active in node / CLI.
not super common atm but still happening. discussing
/unassign has a fix for this


Question: valueFrom: fieldRef: fieldPath: metadata.name <del> command: - /bin/sh - -c - /kubemark --morph=kubelet --name=$(NODE_NAME) {{hollow_kubelet_params}} --kubeconfig=/kubeconfig/kubelet.kubeconfig $(CONTENT_TYPE) --alsologtostderr 1>>/var/log/kubelet-$(NODE_NAME).log 2>&1 </del> <ins> command: [ "/kubemark", "--morph=kubelet", "--name=$(NODE_NAME)", "--kubeconfig=/kubeconfig/kubelet.kubeconfig", "$(CONTENT_TYPE)", "--log-file=/var/log/kubelet-$(NODE_NAME).log", "--logtostderr=false", {{hollow_kubelet_params}} ] </ins> volumeMounts: - name: kubeconfig-volume mountPath: /kubeconfig

There are CVEs happening in different images - we already migrated Kubmeark from busybox in , but we should go one step further and migrate the base image to distroless. There are some things that would need to be changed mentioned here: /sig scalability
Can I work on this? /assign
/remove-help
- sure that would be great.
Hi based on the discussion in , the kubemark image has dependency on and maybe more other utilities. So it seems impossible to migrate the base image to distroless? Or do we need something like debian-hyperkube-base?
The mount one we may get rid of - basically we should be able to switch to FakeMounter:


Question: "//vendor/github.com/onsi/gomega:go_default_library", "//vendor/k8s.io/api/core/v1:go_default_library", "//vendor/k8s.io/api/settings/v1alpha1:go_default_library", <ins> "//vendor/k8s.io/apimachinery/pkg/api/errors:go_default_library", </ins> "//vendor/k8s.io/apimachinery/pkg/apis/meta/v1:go_default_library", "//vendor/k8s.io/apimachinery/pkg/labels:go_default_library", "//vendor/k8s.io/apimachinery/pkg/watch:go_default_library",

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: is consistently failing because of PodPreset feature tests, which require settings/v1alpha1, which is no longer enabled by default What you expected to happen: . How to reproduce it (as minimally and precisely as possible): . Anything else we need to know?: This was caused by and kubernetes/test-infra Environment: . Kubernetes version (use ): . Cloud provider or hardware configuration**: . OS (e.g. from /etc/os-release): . Kernel (e.g. ): . Install tools: . Others: .
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig service-catalog
does the alpha CI job enable alpha API versions?
file has "AllAlpha=true" Maybe would be better?


Question: } // apply volume ownership <del> if !c.readOnly && fsGroup != nil { err := volume.SetVolumeOwnership(c, fsGroup) if err != nil { // attempt to rollback mount. glog.Error(log("mounter.SetupAt failed to set fsgroup volume ownership for [%s]: %v", c.volumeID, err)) glog.V(4).Info(log("mounter.SetupAt attempting to unpublish volume %s due to previous error", c.volumeID)) if unpubErr := csi.NodeUnpublishVolume(ctx, c.volumeID, dir); unpubErr != nil { glog.Error(log( "mounter.SetupAt failed to unpublish volume [%s]: %v (caused by previous NodePublish error: %v)", c.volumeID, unpubErr, err, )) return fmt.Errorf("%v (caused by %v)", unpubErr, err) } </del> <ins> // The following logic is derived from https://github.com/kubernetes/kubernetes/issues/66323 // if fstype is "", then skip fsgroup (could be indication of non-block filesystem) // if fstype is provided and pv.AccessMode == ReadWriteOnly, then apply fsgroup </ins> <del> if unmountErr := removeMountDir(c.plugin, dir); unmountErr != nil { glog.Error(log( "mounter.SetupAt failed to clean mount dir [%s]: %v (caused by previous NodePublish error: %v)", dir, unmountErr, err, )) return fmt.Errorf("%v (caused by %v)", unmountErr, err) } </del> <ins> err = c.applyFSGroup(fsType, fsGroup) if err != nil { // attempt to rollback mount. fsGrpErr := fmt.Errorf("applyFSGroup failed for vol %s: %v", c.volumeID, err) if unpubErr := csi.NodeUnpublishVolume(ctx, c.volumeID, dir); unpubErr != nil { glog.Error(log("NodeUnpublishVolume failed for [%s]: %v", c.volumeID, unpubErr)) return fsGrpErr } </ins> <del> return err </del> <ins> if unmountErr := removeMountDir(c.plugin, dir); unmountErr != nil { glog.Error(log("removeMountDir failed for [%s]: %v", dir, unmountErr)) return fsGrpErr </ins> } <del> glog.V(4).Info(log("mounter.SetupAt sets fsGroup to [%d] for %s", *fsGroup, c.volumeID)) </del> <ins> return fsGrpErr </ins> } glog.V(4).Infof(log("mounter.SetUp successfully requested NodePublish [%s]", dir))

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: fsgroup is only intended for RWO volumes because it recursively changes the permissions of all the directories in the volume. This can cause problems for nfs-type volumes because multiple pods could access the same volume and have the permissions changed out from underneath. What you expected to happen: Determine whether or not the plugin supports fsgroup either via: access mode. This is still a bit ambiguous for nfs volume types, that can theoretically support rwo, especially since Kubernetes does not do any sort of access mode enforcement. a new csi capability
CC
Are you saying that CSI internal driver should check driver capability for RWO, if supported, only then apply value ? What would the be ?
Yes one possible way is to infer fsgroup is supported for all rwo volumes, which may not be true? Like a nfs volume technically supports rwo but not fsgroup. So the other option is to make it a new CSI capability. I'm not sure exactly sure how best to describe it though, maybe something about volume ownership management.
Will it work if is not set then don't chown the ownership? I would think that for shared file system types(nfs, glusterfs) - won't be set. For raw block storage too - I think we shouldn't have to do this.
what do you mean by ? Shouldnt have to solve this issue ?
I meant - which is obvious. I just meant that the entire mechanism of changing ownership of files should apply only to volume types with block storage filesystems. I am just wondering if we can get away with detecting fsType and not have to introduce new capability in CSI.
agreed, do not map well to current csi.Capabilities enums which makes it hard to determine RWO when mounting. Perhaps, instead of a new capability, we revisit the existing capabilities to see if there are any way RWO can be reliably inferred from returned capabilities.


Question: } // apply volume ownership <del> if !c.readOnly && fsGroup != nil { err := volume.SetVolumeOwnership(c, fsGroup) if err != nil { // attempt to rollback mount. glog.Error(log("mounter.SetupAt failed to set fsgroup volume ownership for [%s]: %v", c.volumeID, err)) glog.V(4).Info(log("mounter.SetupAt attempting to unpublish volume %s due to previous error", c.volumeID)) if unpubErr := csi.NodeUnpublishVolume(ctx, c.volumeID, dir); unpubErr != nil { glog.Error(log( "mounter.SetupAt failed to unpublish volume [%s]: %v (caused by previous NodePublish error: %v)", c.volumeID, unpubErr, err, )) return fmt.Errorf("%v (caused by %v)", unpubErr, err) } </del> <ins> // The following logic is derived from https://github.com/kubernetes/kubernetes/issues/66323 // if fstype is "", then skip fsgroup (could be indication of non-block filesystem) // if fstype is provided and pv.AccessMode == ReadWriteOnly, then apply fsgroup </ins> <del> if unmountErr := removeMountDir(c.plugin, dir); unmountErr != nil { glog.Error(log( "mounter.SetupAt failed to clean mount dir [%s]: %v (caused by previous NodePublish error: %v)", dir, unmountErr, err, )) return fmt.Errorf("%v (caused by %v)", unmountErr, err) } </del> <ins> err = c.applyFSGroup(fsType, fsGroup) if err != nil { // attempt to rollback mount. fsGrpErr := fmt.Errorf("applyFSGroup failed for vol %s: %v", c.volumeID, err) if unpubErr := csi.NodeUnpublishVolume(ctx, c.volumeID, dir); unpubErr != nil { glog.Error(log("NodeUnpublishVolume failed for [%s]: %v", c.volumeID, unpubErr)) return fsGrpErr } </ins> <del> return err </del> <ins> if unmountErr := removeMountDir(c.plugin, dir); unmountErr != nil { glog.Error(log("removeMountDir failed for [%s]: %v", dir, unmountErr)) return fsGrpErr </ins> } <del> glog.V(4).Info(log("mounter.SetupAt sets fsGroup to [%d] for %s", *fsGroup, c.volumeID)) </del> <ins> return fsGrpErr </ins> } glog.V(4).Infof(log("mounter.SetUp successfully requested NodePublish [%s]", dir))

Currently, this is what the code is doing which we knew needed to be revisited:
Users may not fully understand what fsgroup is and set it anyway, or a default psp with some fsgroup may be defined in which it will automatically be set on all pods. The first case we can attribute to user error, but the second case may not have a good workaround and could cause permission issues on rwx volumes. So I think we do need to solve this.
Yes, agreed this has to be solved. As you pointed out user error or improperly configured PSP can set value that does not match the correct attribute, causing permission issues. Here is what I think can be done without new CSI capabilities: , then do not apply fsgroup then: , apply fsgroup , do not apply fsgroup to void stated issue then: , apply fsgroup I think that may help deduce when to apply fsgroup.
If mount reference count <= 1, apply fsgroup If mount reference count 1, do not apply fsgroup to void stated issue A RWM volume (say NFS) can be mounted on multiple nodes. Do you have reliable count of mounts across all nodes?
In addition, that would be a change in semantics to how we support rwx volumes today. If we don't have a new CSI capability, then I think the best course of action is to only apply fsgroup for RWO volumes. Some questions remain: Do all RWO volumes support fsgroup? RWX is a superset of RWO. So if a user requests a RWO volume, it could still be satisfied by a RWX volume type. Can we detect this and still not apply the fsgroup even though the volume capability will say RWO?
If there's not a reliable way to enumerate node mounts (thinking about it, for CSI external drivers, the answer is NO). Theres is no reliable way to have k8s autocorrect a bad fsgroup/PV.AccessMode combo. So we would have to rely on the config provided by user/admin. I have come to similar conclusion as -- mainly, apply fsgroup only to volumes with access modes. The other k8s specified access modes, , cannot be reliably applied.


Question: } // apply volume ownership <del> if !c.readOnly && fsGroup != nil { err := volume.SetVolumeOwnership(c, fsGroup) if err != nil { // attempt to rollback mount. glog.Error(log("mounter.SetupAt failed to set fsgroup volume ownership for [%s]: %v", c.volumeID, err)) glog.V(4).Info(log("mounter.SetupAt attempting to unpublish volume %s due to previous error", c.volumeID)) if unpubErr := csi.NodeUnpublishVolume(ctx, c.volumeID, dir); unpubErr != nil { glog.Error(log( "mounter.SetupAt failed to unpublish volume [%s]: %v (caused by previous NodePublish error: %v)", c.volumeID, unpubErr, err, )) return fmt.Errorf("%v (caused by %v)", unpubErr, err) } </del> <ins> // The following logic is derived from https://github.com/kubernetes/kubernetes/issues/66323 // if fstype is "", then skip fsgroup (could be indication of non-block filesystem) // if fstype is provided and pv.AccessMode == ReadWriteOnly, then apply fsgroup </ins> <del> if unmountErr := removeMountDir(c.plugin, dir); unmountErr != nil { glog.Error(log( "mounter.SetupAt failed to clean mount dir [%s]: %v (caused by previous NodePublish error: %v)", dir, unmountErr, err, )) return fmt.Errorf("%v (caused by %v)", unmountErr, err) } </del> <ins> err = c.applyFSGroup(fsType, fsGroup) if err != nil { // attempt to rollback mount. fsGrpErr := fmt.Errorf("applyFSGroup failed for vol %s: %v", c.volumeID, err) if unpubErr := csi.NodeUnpublishVolume(ctx, c.volumeID, dir); unpubErr != nil { glog.Error(log("NodeUnpublishVolume failed for [%s]: %v", c.volumeID, unpubErr)) return fsGrpErr } </ins> <del> return err </del> <ins> if unmountErr := removeMountDir(c.plugin, dir); unmountErr != nil { glog.Error(log("removeMountDir failed for [%s]: %v", dir, unmountErr)) return fsGrpErr </ins> } <del> glog.V(4).Info(log("mounter.SetupAt sets fsGroup to [%d] for %s", *fsGroup, c.volumeID)) </del> <ins> return fsGrpErr </ins> } glog.V(4).Infof(log("mounter.SetUp successfully requested NodePublish [%s]", dir))

Even if we were to introduce additional plugin capabilities that the CO can query (from driver) to find out what modes can be applied, there would still be the possibility of user/admin mis-configuration. Probably not, Is this where you think an additional capability would help ( ) That is a good point. Can't think of a way to detect/guess RWX when RWO is specified, this without the driver providing that info. Can we adopt the rule until we decide what capabilities would look like ?
I will just try to summarize the conversation from Slack. We can safely assume that only applies to block storage type with file system on top of it. So for all - volumes that need to have called MUST have a file system() on them. Is that assumption correct? If it is - can we not use presence of field or we can query the local volume for fs type (like how we query selinux relabelling capability) and call only when - volume is RW and has a valid block storage file system type. Will this not solve our use case? EDIT: renamed to .
We'll need to make it clear that is only for block-based storage systems, and the field cannot be reused for some multi-protocol file server (ie, nfsv3, nfsv4, smb, etc). For that case, then the plugin will need to expose its own new storageclass parameter + volume attribute to specify protocol.
Can we not use to get fsType - and use that information for calling ? It may be problematic to rely entirely on present inside csi source.
the only problem with is that in CSI volume info is intentionally opaque to the CSI code inside k8s. All mount logic is delegated to the driver.
Instead of a new capability or trying to deduce how best to apply fsgroup, why not pass the to the driver and let it decide how to apply it (either as a attribute for the mount or top level request param)? Right now, all mount logic has been delegated to the external CSI driver except for applying permission. Thoughts ?
that is true but we can still inspect file system on the mounted path because by the time gets called volume is already mounted. This should be no different than how selinux support is determined. what you think?
is this approved for 1.12 milestone? Thank you for working on this,
still discussing how best to approach this.


Question: } // apply volume ownership <del> if !c.readOnly && fsGroup != nil { err := volume.SetVolumeOwnership(c, fsGroup) if err != nil { // attempt to rollback mount. glog.Error(log("mounter.SetupAt failed to set fsgroup volume ownership for [%s]: %v", c.volumeID, err)) glog.V(4).Info(log("mounter.SetupAt attempting to unpublish volume %s due to previous error", c.volumeID)) if unpubErr := csi.NodeUnpublishVolume(ctx, c.volumeID, dir); unpubErr != nil { glog.Error(log( "mounter.SetupAt failed to unpublish volume [%s]: %v (caused by previous NodePublish error: %v)", c.volumeID, unpubErr, err, )) return fmt.Errorf("%v (caused by %v)", unpubErr, err) } </del> <ins> // The following logic is derived from https://github.com/kubernetes/kubernetes/issues/66323 // if fstype is "", then skip fsgroup (could be indication of non-block filesystem) // if fstype is provided and pv.AccessMode == ReadWriteOnly, then apply fsgroup </ins> <del> if unmountErr := removeMountDir(c.plugin, dir); unmountErr != nil { glog.Error(log( "mounter.SetupAt failed to clean mount dir [%s]: %v (caused by previous NodePublish error: %v)", dir, unmountErr, err, )) return fmt.Errorf("%v (caused by %v)", unmountErr, err) } </del> <ins> err = c.applyFSGroup(fsType, fsGroup) if err != nil { // attempt to rollback mount. fsGrpErr := fmt.Errorf("applyFSGroup failed for vol %s: %v", c.volumeID, err) if unpubErr := csi.NodeUnpublishVolume(ctx, c.volumeID, dir); unpubErr != nil { glog.Error(log("NodeUnpublishVolume failed for [%s]: %v", c.volumeID, unpubErr)) return fsGrpErr } </ins> <del> return err </del> <ins> if unmountErr := removeMountDir(c.plugin, dir); unmountErr != nil { glog.Error(log("removeMountDir failed for [%s]: %v", dir, unmountErr)) return fsGrpErr </ins> } <del> glog.V(4).Info(log("mounter.SetupAt sets fsGroup to [%d] for %s", *fsGroup, c.volumeID)) </del> <ins> return fsGrpErr </ins> } glog.V(4).Infof(log("mounter.SetUp successfully requested NodePublish [%s]", dir))

It would be a 1.12.
[MILESTONENOTIFIER] Milestone Issue: Up-to-date for process <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
Approved for 1.12
Based on previous post, this is a summary: If then is not applied (because it could be an indication of a nonblock fs, or error) Done is provided if fstype is invalid Done if fstype is valid and then apply fsgroup logic else ignore fsGroup Done
- any updates on ?
Waiting for a review and approval.
you have a outstanding comment on the PR. Can you please answer that?
How does this approach relate to inline CSI volumes? There's no PersistentVolume involved, and hence no way to specify an access mode. , but my driver ignores it (always using tmpfs). I considered having my CSI driver read the mounting pod's spec to see if it has a security context with "fsGroup" set, since that detail is not supplied to the driver, as . My goal is to have the volume mounted with the files on it readable by the container's user, but there's no way for my driver to know who that user will be. Am I supposed to just use 0444 for the ownership bits, to allow everyone to read the files?


Question: "//cmd/kube-proxy/app:go_default_library", "//pkg/client/metrics/prometheus:go_default_library", "//pkg/version/prometheus:go_default_library", <ins> "//vendor/github.com/spf13/pflag:go_default_library", </ins> "//vendor/k8s.io/apiserver/pkg/util/flag:go_default_library", "//vendor/k8s.io/apiserver/pkg/util/logs:go_default_library", ],

There are no sig labels on this issue. Please add a sig label. <detailsA sig label can be by either: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. See the . The in method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
probably caused by
Could be a clash between these two?
cc
I still see the issue with local-cluster on the latest master: F0225 17:06:49. ] error: v1alpha1.KubeProxyConfiguration: FeatureGates: ReadMapCB: expect { or n, but found ", error found in byte of ......
looks like that's another issue, can you please file a new bug? with more logs? this one was about none of the hyperkube based processes starting up.
will do as it is also broke container kube-proxy as well. Same type of error.


Question: } ip, err := az.getIPForMachine(name) if err != nil { <del> glog.Errorf("error: az.NodeAddresses, az.getIPForMachine(%s), err=%v", name, err) return nil, err </del> <ins> if az.CloudProviderBackoff { glog.V(2).Infof("NodeAddresses(%s) backing off", name) ip, err = az.GetIPForMachineWithRetry(name) if err != nil { glog.V(2).Infof("NodeAddresses(%s) abort backoff", name) return nil, err } } else { glog.Errorf("error: az.NodeAddresses, az.getIPForMachine(%s), err=%v", name, err) return nil, err } </ins> } return []v1.NodeAddress{

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind feature What happened: In certain large cluster scenarios we would like to anticipate kubelet behavior hitting request limits, and backoff in response. E.g., What you expected to happen: This is a request for add'l backoff coverage, so nothing surprising. How to reproduce it (as minimally and precisely as possible): Build a cluster with as many nodes as your subscription will support on azure until kubelet behavior contributes to backend API request limits. Environment: Kubernetes version (use ): and Cloud provider or hardware configuration**: azure OS (e.g. from /etc/os-release): Ubuntu 16.04.2 LTS (Xenial Xerus) Kernel (e.g. ): Linux k8s-pool7--7 4.4.0-83-generic -Ubuntu SMP Mon Jun 26 17:54:43 UTC 2017 x8664 x8664 x86_64 GNU/Linux Install tools: acs-engine Others:
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig azure


Question: '${MASTER_IP}' create-flanneld-opts '127.0.0.1' '${MASTER_IP}' <del> sudo -E -p '[sudo] password to start master: ' -- /bin/bash -ce ' </del> <ins> FLANNEL_OTHER_NET_CONFIG='${FLANNEL_OTHER_NET_CONFIG}' sudo -E -p '[sudo] password to start master: ' -- /bin/bash -ce ' </ins> ${BASH_DEBUG_FLAGS} cp ~/kube/default/* /etc/default/ cp ~/kube/init_conf/* /etc/init/

and provide for configuring Flannel's parameter but none of the others listed in


Question: }{ { code:     http.StatusBadRequest, <del> expected: true, </del> <ins> expected: false, </ins> }, { code:     http.StatusInternalServerError,

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: If you create a service that reference a Basic IP in a standard load balancer cluster, the service will stuck at "Pending" state for days without detailed error from Azure response. When you describe the service, it only has following: What you expected to happen: In this case Azure responded 400 with detailed error message that the referenced IP is a basic IP and cannot be used on standard LB. I would expect the detailed error message show up as events for easier diagnostic. How to reproduce it (as minimally and precisely as possible): Create a service that reference a Basic IP in a standard load balancer cluster Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/assign /area provider/azure /sig cloud-provider
/area cloudprovider


Question: func subnet(service *v1.Service) *string { if requiresInternalLoadBalancer(service) { <del> if l, found := service.Annotations[ServiceAnnotationLoadBalancerInternalSubnet]; found { </del> <ins> if l, found := service.Annotations[ServiceAnnotationLoadBalancerInternalSubnet]; found && strings.TrimSpace(l) != "" { </ins> return &l } }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: When service is set with ILB annotation and empty string subnet, e.g. The service would be pending forever with following error: What you expected to happen: ILB should be provisioned successfully. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Others: /sig azure /kind bug /assign
/milestone v1.14 We're entering and this looks relevant to the 1.14 release. Please if I am incorrect
Thanks


Question: }, }, }, <ins> Required: []string{"versions"}, </ins> }, }, Dependencies: []string{

FYI. Currently the field of TPR is marked as which is wrong, actually if you try to create a TPR with empty versions, it will cause api server panic. Versions should be necessary field of TPR, we should validate this somehow.


Question: statefulSetTimeout = 10 * time.Minute // statefulPodTimeout is a timeout for stateful pods to change state statefulPodTimeout = 5 * time.Minute <ins> testFinalizer = "example.com/test-finalizer" </ins> ) var httpProbe = &v1.Probe{

My application is clustered and I have 2 pods running. I am using a rolling upgrade strategy and my partitioning number is 1. First I need to upgrade the pod with the ordinal number 1 so I triggered that upgrade with partitioning number 1 and my upgrade gets completed. After this, my pod having the ordinal number 0 gets restarted and it's automatically upgraded to a new image. This is not an expected behavior because still, my partition number is 1 and pod-0 should come up with an older image only. I am facing this issue for the v1.27 cluster. Prior to this, it was working fine. pod-0 should not upgrade to new image until I upgrade the partition number to 0.a Do helm install and upgrade pod-1 only keeping partition number to 1. No response <detailsconsole $ cat /etc/os-release $ uname -a C:wmic os get Caption, Version, BuildNumber, OSArchitecture </details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig/api-machinery
/sig apps /sig storage
Hi, Can someone suggest here? As this is introduced in my setup after the v1.27 upgrade? Is there any change in partition logic?
This can be reproduced by using the said STS. 1 - Deploy STS using attached that has a replica count is 2 and updatingstrategy.rolling.partition as 1. 2 - Do edit STS and update the image to 1.15. This will upgrade, pod-1 with image 1.15. 3 - Next delete the pod pod-0 and check that POD-0 came up with a new image which is not the expected behavior. POD 0 should not be updated to a new image until the partition is changed to 0. We have noticed this behavior change in v1.27.


Question: statefulSetTimeout = 10 * time.Minute // statefulPodTimeout is a timeout for stateful pods to change state statefulPodTimeout = 5 * time.Minute <ins> testFinalizer = "example.com/test-finalizer" </ins> ) var httpProbe = &v1.Probe{

Pla suggest : apiVersion: v1 kind: Service metadata: name: nginx labels: app: nginx spec: ports: port: 80 name: web clusterIP: None selector: app: nginx apiVersion: apps/v1 kind: StatefulSet metadata: name: web-am spec: updateStrategy: type: RollingUpdate rollingUpdate: partition: 1 serviceName: "nginx" replicas: 2 selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: name: nginx args: /bin/sh -c sleep 30;touch /tmp/healthy; sleep 3600 image: nginx:1.14.2 ports: containerPort: 80 name: web


Question: expectedErrMsg:      fmt.Errorf("getError"), }, { <del> name:                "InstanceMetadataByProviderID should report error if VMSS instanceID is invalid", metadataName:        "vmss_$123", providerID:          providerID, vmType:              vmTypeVMSS, useInstanceMetadata: true, expectedErrMsg:      fmt.Errorf("failed to parse VMSS instanceID %q: strconv.ParseInt: parsing %q: invalid syntax", "$123", "$123"), }, { </del> name:         "InstanceMetadataByProviderID should get metadata from Azure API if cloud.UseInstanceMetadata is false", metadataName: "vm1", providerID:   providerID,

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: If computer name prefix does not match VMSS name, then Azure cloud provider will use VMSS API instead of using metadata service data. This leads to throttling, because every affected kubelet is making API requests. What you expected to happen: Azure cloud provider should use metadata service data instead of calling VMSS API when instance is current. How to reproduce it (as minimally and precisely as possible): Set computer name prefix of the VMSS to something different than default. You should observe Azure provider making requests to the VMSS API rather than using metadata service, which leads to throttling if you have multiple kubelets with this bug. Anything else we need to know?: This is bug was introduced by - returns false even if instance is current, because it expects node name to match VMSS name + instance id, however node name is derived from the computer-name-prefix which can be different than VMSS name. Node name should be matched against , not against . Environment: Kubernetes version (use ): 1.18.x and higher Cloud provider or hardware configuration: azure /sig cloud-provider
/sig cloud-provider
/area provider/azure
CC
/remove-sig cloud-provider /sig cloud-provider (trying to remove the label)
/sig cloud-provider
Thanks for the reporting. Unfortunately, computer name prefix is not included in IMDS, so it can't be used without VMSS API for constructing the NodeName. Would check what could we do to workaround the issue.
Can you memorize the computer name prefix in the first request to the Azure API and then use it for the isCurrentInstance check?
opened to fix the issue. Would you like to have a look?
I can confirm that your change should fix the problem I reported. Do you know if your fix will be to 1.18.x?
is working on the cherry pick
Thank you for quick fix!


Question: <ins> /* Copyright 2022 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package plugin import ( "context" "errors" "fmt" "time" v1 "k8s.io/api/core/v1" resourceapi "k8s.io/api/resource/v1alpha3" apierrors "k8s.io/apimachinery/pkg/api/errors" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/fields" utilversion "k8s.io/apimachinery/pkg/util/version" "k8s.io/apimachinery/pkg/util/wait" "k8s.io/client-go/kubernetes" "k8s.io/klog/v2" "k8s.io/kubernetes/pkg/kubelet/pluginmanager/cache" ) // defaultClientCallTimeout is the default amount of time that a DRA driver has // to respond to any of the gRPC calls. kubelet uses this value by passing nil // to RegisterPlugin. Some tests use a different, usually shorter timeout to // speed up testing. // // This is half of the kubelet retry period (according to // https://github.com/kubernetes/kubernetes/commit/0449cef8fd5217d394c5cd331d852bd50983e6b3). const defaultClientCallTimeout = 45 * time.Second // RegistrationHandler is the handler which is fed to the pluginwatcher API. type RegistrationHandler struct { // backgroundCtx is used for all future activities of the handler. // This is necessary because it implements APIs which don't // provide a context. backgroundCtx context.Context kubeClient    kubernetes.Interface getNode       func() (*v1.Node, error) } var _ cache.PluginHandler = &RegistrationHandler{} // NewPluginHandler returns new registration handler. // // Must only be called once per process because it manages global state. // If a kubeClient is provided, then it synchronizes ResourceSlices // with the resource information provided by plugins. func NewRegistrationHandler(kubeClient kubernetes.Interface, getNode func() (*v1.Node, error)) *RegistrationHandler { handler := &RegistrationHandler{ // The context and thus logger should come from the caller. backgroundCtx: klog.NewContext(context.TODO(), klog.LoggerWithName(klog.TODO(), "DRA registration handler")), kubeClient:    kubeClient, getNode:       getNode, } // When kubelet starts up, no DRA driver has registered yet. None of // the drivers are usable until they come back, which might not happen // at all. Therefore it is better to not advertise any local resources // because pods could get stuck on the node waiting for the driver // to start up. // // This has to run in the background. go handler.wipeResourceSlices("") return handler } // wipeResourceSlices deletes ResourceSlices of the node, optionally just for a specific driver. func (h *RegistrationHandler) wipeResourceSlices(driver string) { if h.kubeClient == nil { return } ctx := h.backgroundCtx logger := klog.FromContext(ctx) backoff := wait.Backoff{ Duration: time.Second, Factor:   2, Jitter:   0.2, Cap:      5 * time.Minute, Steps:    100, } // Error logging is done inside the loop. Context cancellation doesn't get logged. _ = wait.ExponentialBackoffWithContext(ctx, backoff, func(ctx context.Context) (bool, error) { node, err := h.getNode() if apierrors.IsNotFound(err) { return false, nil } if err != nil { logger.Error(err, "Unexpected error checking for node") return false, nil } fieldSelector := fields.Set{resourceapi.ResourceSliceSelectorNodeName: node.Name} if driver != "" { fieldSelector[resourceapi.ResourceSliceSelectorDriver] = driver } err = h.kubeClient.ResourceV1alpha3().ResourceSlices().DeleteCollection(ctx, metav1.DeleteOptions{}, metav1.ListOptions{FieldSelector: fieldSelector.String()}) switch { case err == nil: logger.V(3).Info("Deleted ResourceSlices", "fieldSelector", fieldSelector) return true, nil case apierrors.IsUnauthorized(err): // This can happen while kubelet is still figuring out // its credentials. logger.V(5).Info("Deleting ResourceSlice failed, retrying", "fieldSelector", fieldSelector, "err", err) return false, nil default: // Log and retry for other errors. logger.V(3).Info("Deleting ResourceSlice failed, retrying", "fieldSelector", fieldSelector, "err", err) return false, nil } }) } // RegisterPlugin is called when a plugin can be registered. func (h *RegistrationHandler) RegisterPlugin(pluginName string, endpoint string, versions []string, pluginClientTimeout *time.Duration) error { // Prepare a context with its own logger for the plugin. // // The lifecycle of the plugin's background activities is tied to our // root context, so canceling that will also cancel the plugin. // // The logger injects the plugin name as additional value // into all log output related to the plugin. ctx := h.backgroundCtx logger := klog.FromContext(ctx) logger = klog.LoggerWithValues(logger, "pluginName", pluginName) ctx = klog.NewContext(ctx, logger) logger.V(3).Info("Register new DRA plugin", "endpoint", endpoint) highestSupportedVersion, err := h.validateVersions(pluginName, versions) if err != nil { return fmt.Errorf("version check of plugin %s failed: %w", pluginName, err) } var timeout time.Duration if pluginClientTimeout == nil { timeout = defaultClientCallTimeout } else { timeout = *pluginClientTimeout } ctx, cancel := context.WithCancelCause(ctx) pluginInstance := &Plugin{ backgroundCtx:           ctx, cancel:                  cancel, conn:                    nil, endpoint:                endpoint, highestSupportedVersion: highestSupportedVersion, clientCallTimeout:       timeout, } // Storing endpoint of newly registered DRA Plugin into the map, where plugin name will be the key // all other DRA components will be able to get the actual socket of DRA plugins by its name. if draPlugins.add(pluginName, pluginInstance) { logger.V(1).Info("Already registered, previous plugin was replaced") } return nil } func (h *RegistrationHandler) validateVersions( pluginName string, versions []string, ) (*utilversion.Version, error) { if len(versions) == 0 { return nil, errors.New("empty list for supported versions") } // Validate version newPluginHighestVersion, err := utilversion.HighestSupportedVersion(versions) if err != nil { // HighestSupportedVersion includes the list of versions in its error // if relevant, no need to repeat it here. return nil, fmt.Errorf("none of the versions are supported: %w", err) } existingPlugin := draPlugins.get(pluginName) if existingPlugin == nil { return newPluginHighestVersion, nil } if existingPlugin.highestSupportedVersion.LessThan(newPluginHighestVersion) { return newPluginHighestVersion, nil } return nil, fmt.Errorf("another plugin instance is already registered with a higher supported version: %q < %q", newPluginHighestVersion, existingPlugin.highestSupportedVersion) } // DeRegisterPlugin is called when a plugin has removed its socket, // signaling it is no longer available. func (h *RegistrationHandler) DeRegisterPlugin(pluginName string) { if p := draPlugins.delete(pluginName); p != nil { logger := klog.FromContext(p.backgroundCtx) logger.V(3).Info("Deregister DRA plugin", "endpoint", p.endpoint) // Clean up the ResourceSlices for the deleted Plugin since it // may have died without doing so itself and might never come // back. go h.wipeResourceSlices(pluginName) return } logger := klog.FromContext(h.backgroundCtx) logger.V(3).Info("Deregister DRA plugin not necessary, was already removed") } // ValidatePlugin is called by kubelet's plugin watcher upon detection // of a new registration socket opened by DRA plugin. func (h *RegistrationHandler) ValidatePlugin(pluginName string, endpoint string, versions []string) error { _, err := h.validateVersions(pluginName, versions) if err != nil { return fmt.Errorf("invalid versions of plugin %s: %w", pluginName, err) } return err } </ins>

The PR above moved everything related to into because that was where most of the code already was, but didn't renamed the files to avoid a larger diff and merge conflicts. Once is merged, we can do the renaming. Code clarity. /sig node
/triage accepted /priority important-soon
/assign
a PR can be created for this now.
will do


Question: const ( pvDeletionTimeout       = 3 * time.Minute statefulSetReadyTimeout = 3 * time.Minute <ins> taintKeyPrefix          = "zoneTaint_" </ins> ) var _ = utils.SIGDescribe("Regional PD", func() {

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: Testgrid: https://k8s-
/assign
The test's assumption about test parameter is wrong. Previously it assumed the group name is the same across all zones, but they are different in regional clusters. Looking into using node taints as the permanent fix, as suggested by the TODO comment in the test.
Hm I think taint based evictions is still an alpha feature.
OK. I propose the following solution: Get the zone of Schedule the StatefulSet pod to that zone using node affinity; wait for the pod to run Remove the node affinity from StatefulSet Delete
Can it be simplified? Schedule the pod Figure out which zone that pod got scheduled in Delete the instance group of that zone
Unfortunately I don't know of a good way to look up the instance group given the zone, since there could be instance groups from multiple clusters in the same project
How about: Schedule the pod Figure out the zone Taint all the nodes in the zone Kill the pod
Taint with NoSchedule? yeah that sounds good


Question: return multiGroupVersioner{target: gv, acceptedGroupKinds: groupKinds} } <ins> // NewCoercingMultiGroupVersioner returns the provided group version for any incoming kind. // Incoming kinds that match the provided groupKinds are preferred. // Kind may be empty in the provided group kind, in which case any kind will match. // Examples: //   gv=mygroup/__internal, groupKinds=mygroup/Foo, anothergroup/Bar //   KindForGroupVersionKinds(yetanother/v1/Baz, anothergroup/v1/Bar) -> mygroup/__internal/Bar (matched preferred group/kind) // //   gv=mygroup/__internal, groupKinds=mygroup, anothergroup //   KindForGroupVersionKinds(yetanother/v1/Baz, anothergroup/v1/Bar) -> mygroup/__internal/Bar (matched preferred group) // //   gv=mygroup/__internal, groupKinds=mygroup, anothergroup //   KindForGroupVersionKinds(yetanother/v1/Baz, yetanother/v1/Bar) -> mygroup/__internal/Baz (no preferred group/kind match, uses first kind in list) func NewCoercingMultiGroupVersioner(gv schema.GroupVersion, groupKinds ...schema.GroupKind) GroupVersioner { return multiGroupVersioner{target: gv, acceptedGroupKinds: groupKinds, coerce: true} } </ins> // KindForGroupVersionKinds returns the target group version if any kind matches any of the original group kinds. It will // use the originating kind where possible. func (v multiGroupVersioner) KindForGroupVersionKinds(kinds []schema.GroupVersionKind) (schema.GroupVersionKind, bool) {

What happened: Set up a HA cluster with 1.9 and 1.10 API servers (occurs on any apiserver skew boundary where a cohabitating resource's storage version changed groups) Observed errors when modifying workload objects: when modified via the 1.10 server, saw this error in the 1.9 server: when modified via the 1.9 server, saw this error in the 1.10 server: When this error was encountered, all watchers for the resource in question would be dropped and have to re-establish, and the internal watchers feeding the watch cache would drop and re-establish What you expected to happen: Watch events from all supported storage versions are processed correctly How to reproduce it (as minimally and precisely as possible): Start a server from master (for inspectability, store in json). With Create a deployment (e.g. kube-dns in kube-system) Establish a watch via extensions (kubectl get deployments.extensions --all-namespaces -w) Establish a watch via apps (kubectl get --all-namespaces -w) Tail the kube-apiserver log Change the apps/v1 deployment in etcd to extensions/v1beta1: Observer client watches drop and server log errors: Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: /sig api-machinery /cc /priority critical-urgent
error is triggered by this returning false: constructed here: the info about expected cohabitating groups is not included in the list of acceptable inputs


Question: // updated. The updated Endpoints object or an error will be returned. func (adapter *EndpointsAdapter) Update(namespace string, endpoints *corev1.Endpoints) (*corev1.Endpoints, error) { endpoints, err := adapter.endpointClient.Endpoints(namespace).Update(endpoints) <del> if err == nil && adapter.endpointSliceClient != nil { _, err = adapter.ensureEndpointSliceFromEndpoints(namespace, endpoints) </del> <ins> if err == nil { err = adapter.EnsureEndpointSliceFromEndpoints(namespace, endpoints) </ins> } return endpoints, err } <del> // ensureEndpointSliceFromEndpoints accepts a namespace and Endpoints resource // and creates or updates a corresponding EndpointSlice. The EndpointSlice // and/or an error will be returned. func (adapter *EndpointsAdapter) ensureEndpointSliceFromEndpoints(namespace string, endpoints *corev1.Endpoints) (*discovery.EndpointSlice, error) { </del> <ins> // EnsureEndpointSliceFromEndpoints accepts a namespace and Endpoints resource // and creates or updates a corresponding EndpointSlice if an endpointSliceClient // exists. An error will be returned if it fails to sync the EndpointSlice. func (adapter *EndpointsAdapter) EnsureEndpointSliceFromEndpoints(namespace string, endpoints *corev1.Endpoints) error { if adapter.endpointSliceClient == nil { return nil } </ins> endpointSlice := endpointSliceFromEndpoints(endpoints) <del> _, err := adapter.endpointSliceClient.EndpointSlices(namespace).Get(endpointSlice.Name, metav1.GetOptions{}) </del> <ins> currentEndpointSlice, err := adapter.endpointSliceClient.EndpointSlices(namespace).Get(endpointSlice.Name, metav1.GetOptions{}) </ins> if err != nil { if errors.IsNotFound(err) { <del> return adapter.endpointSliceClient.EndpointSlices(namespace).Create(endpointSlice) </del> <ins> if _, err = adapter.endpointSliceClient.EndpointSlices(namespace).Create(endpointSlice); errors.IsAlreadyExists(err) { err = nil } </ins> } <del> return nil, err </del> <ins> return err } if apiequality.Semantic.DeepEqual(currentEndpointSlice.Endpoints, endpointSlice.Endpoints) && apiequality.Semantic.DeepEqual(currentEndpointSlice.Ports, endpointSlice.Ports) && apiequality.Semantic.DeepEqual(currentEndpointSlice.Labels, endpointSlice.Labels) { return nil </ins> } <del> return adapter.endpointSliceClient.EndpointSlices(namespace).Update(endpointSlice) </del> <ins> _, err = adapter.endpointSliceClient.EndpointSlices(namespace).Update(endpointSlice) return err </ins> } // endpointSliceFromEndpoints generates an EndpointSlice from an Endpoints

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: EndpointSlice for masters was not created after enabling EndpointSlice on a pre-existing cluster. Only those EndpointSlices managed by kube-controller-manager were created. What you expected to happen: EndpointSlice for masters should be created after enabling EndpointSlice on a pre-existing cluster, the same as the EndpointSlices managed by kube-controller-manager. How to reproduce it (as minimally and precisely as possible): kubeadm to create a cluster: the EndpointSlice feature by editing the kube-apiserver, kube-controller-manager, kube-proxy manifest kube-apiserver and kube-controller-manager is restarted, the EndpointSlice in namespace will be missing. Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools:Network plugin and version (if this is a network-related bug): Others:
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/remove-triage unresolved /assign


Question: import ( "fmt" <del> "strings" </del> "time" "k8s.io/kubernetes/pkg/api" <del> client "k8s.io/kubernetes/pkg/client/unversioned" </del> "k8s.io/kubernetes/pkg/util" . "github.com/onsi/ginkgo"

We are running heapster, dns server, kube-proxy etc. in kube-system namespace. Some e2e tests try to make sure all those kube-system pods are at ready state before firing the real tests, others just simply ingore them. But both cases might cause e2e flakiness when some of those kube-system containers are in crashloop, for example, for heapster crashloop, for influxdb crashloop, etc. When such cases happen, the developer doesn't have much information on why since the nodes might be teared down. We should watch those kube-system pods, and onContainerFailure, we should collect the related logs. One simple way is using kubectl logs <Pod<Container--previous cc/
is this related enough to and that you can bundle it in?
The logs from the pods should be retrieved via the api server. This is different from the kubelet logs. I'll upload a PR for this.
Great job It will be very helpful for debugging.


Question: t.Fatalf("failed to mark all StatefulSet pods to ready: %v", err) } } <ins> // add for issue: https://github.com/kubernetes/kubernetes/issues/108837 func TestStatefulSetStatusWithPodFail(t *testing.T) { limitedPodNumber := 2 controlPlaneConfig := framework.NewIntegrationTestControlPlaneConfig() controlPlaneConfig.GenericConfig.AdmissionControl = &fakePodFailAdmission{ limitedPodNumber: limitedPodNumber, } _, s, closeFn := framework.RunAnAPIServer(controlPlaneConfig) defer closeFn() config := restclient.Config{Host: s.URL} c, err := clientset.NewForConfig(&config) if err != nil { t.Fatalf("Could not create clientset: %v", err) } resyncPeriod := 12 * time.Hour informers := informers.NewSharedInformerFactory(clientset.NewForConfigOrDie(restclient.AddUserAgent(&config, "statefulset-informers")), resyncPeriod) ssc := statefulset.NewStatefulSetController( informers.Core().V1().Pods(), informers.Apps().V1().StatefulSets(), informers.Core().V1().PersistentVolumeClaims(), informers.Apps().V1().ControllerRevisions(), clientset.NewForConfigOrDie(restclient.AddUserAgent(&config, "statefulset-controller")), ) ns := framework.CreateTestingNamespace("test-pod-fail", s, t) defer framework.DeleteTestingNamespace(ns, s, t) ctx, cancel := context.WithCancel(context.Background()) defer cancel() informers.Start(ctx.Done()) go ssc.Run(ctx, 5) sts := newSTS("sts", ns.Name, 4) _, err = c.AppsV1().StatefulSets(sts.Namespace).Create(context.TODO(), sts, metav1.CreateOptions{}) if err != nil { t.Fatalf("Could not create statefuleSet %s: %v", sts.Name, err) } wantReplicas := limitedPodNumber var gotReplicas int32 if err := wait.PollImmediate(pollInterval, pollTimeout, func() (bool, error) { newSTS, err := c.AppsV1().StatefulSets(sts.Namespace).Get(context.TODO(), sts.Name, metav1.GetOptions{}) if err != nil { return false, err } gotReplicas = newSTS.Status.Replicas return gotReplicas == int32(wantReplicas), nil }); err != nil { t.Fatalf("StatefulSet %s status has %d replicas, want replicas %d: %v", sts.Name, gotReplicas, wantReplicas, err) } } </ins>

I create a ResourceQuota with two pods limit in my namespace. Then, I create a statefuleSet with 3 replicas. As expected, two pods are created successfully, and the third pod failed to create. But, the ReadyReplicas is not 2 ,but 1. The ReadyReplicas is correct 1Create a ResourceQuota with two pods limit in a new namespace. 2Create a statefulSet with 3 replicas 3Check if the ReadyReplicas of statefulSet matchs the number of stateful running pods. 1According to the logic of the statefulSet controller, new pod created in current sync is not be immediately counted into the ReadyReplicas. 2Failure to create pod will cause to return error, then skip updating status. <details</details<details</details<detailskind v1.23.3 </details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig apps
/assign


Question: Name: "mypsp", }, Spec: policy.PodSecurityPolicySpec{ <ins> AllowedUnsafeSysctls: []string{"kernel.*", "net.ipv4.ip_local_port_range"}, ForbiddenSysctls:     []string{"net.ipv4.ip_default_ttl"}, </ins> SELinux: policy.SELinuxStrategyOptions{ Rule: policy.SELinuxStrategyRunAsAny, },

/kind bug What happened: does not show the description for the new field in What you expected to happen: should describe the new field. How to reproduce it (as minimally and precisely as possible): Environment: Kubernetes version (use ): master
/sig auth
GitHub didn't allow me to assign the following users: fanzhangio. Note that only and repo collaborators can be assigned. <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
If nobody has started yet, I wanna to pick it up.
was working on it AFAIR.
Indeed. if you don't mind, I'd take over.


Question: defer testKubelet.Cleanup() kubelet := testKubelet.kubelet pods := newTestPods(5) <ins> now := metav1.NewTime(time.Now()) </ins> pods[0].Status.Phase = v1.PodFailed pods[1].Status.Phase = v1.PodSucceeded <ins> // The pod is terminating, should not filter out. </ins> pods[2].Status.Phase = v1.PodRunning <ins> pods[2].DeletionTimestamp = &now pods[2].Status.ContainerStatuses = []v1.ContainerStatus{ {State: v1.ContainerState{ Running: &v1.ContainerStateRunning{ StartedAt: now, }, }}, } </ins> pods[3].Status.Phase = v1.PodPending <ins> pods[4].Status.Phase = v1.PodRunning </ins> expected := []*v1.Pod{pods[2], pods[3], pods[4]} kubelet.podManager.SetPods(pods)

pointed out that for scheduling and kubelet admission, we should consider resources of terminated-but-not-yet-deleted pods to still be in use, because it may take an arbitrary amount of time for the system to deallocate resources. We didn't check to see if this is currently how it is implemented or not. cc/ /sig scheduling
cc/
For terminated-but-not-yet-deleted & phase is case, the Pod is still in scheduler's cache, so it's handled well in scheduler; the Pod will be removed from cache when it's deleted. But in kubelet, it filtered terminated-but-not-yet-deleted out () when using scheduler.predicates. The option in my mind is to ~pass all Pods (including terminated-but-not-yet-deleted) to each admin, and let each admin decide whether filter terminated-but-not-yet-deleted out~ only filter out finished (Failed or Succeed) pods for Admits.
In -, it already consider by . So both scheduler & kubelet handles terminated-but-not-yet-deleted pods well.
Great, thanks for looking into it!
should this issue be closed then?
I think so.
Just to clarify: Kubelet correctly accounts the resources used by terminated-but-not-yet-deleted pods, i.e, kubelet does not count the resource usage of those pods towards total usage on the node. Scheduler, however, does not do the same. Scheduler waits for a pod to be deleted, and then it deducts its resource request from the associated node total usage. Upon deletion of a pod, scheduler is notified and acts: So, scheduler and kubelet behaviors are not exactly the same. I am not sure if scheduler can receive an event when a pod's container status changes. If it could, we could make the logic the same as that of Kubelet.
I guess the current behavior is OK because it is conservative (scheduler considers the resources to be in use even though kubelet considers them free). And actually it seems that scheduler is more "correct" than kubelet, since the resources really aren't usable until the pod is deleted, right? (IIUC that was Vish's original point)
I agree. The current behavior is correct, but a bit conservative. Basically, the scheduler side is more conservative and waits until a pod is deleted from the cache before considering its resources released.


Question: defer testKubelet.Cleanup() kubelet := testKubelet.kubelet pods := newTestPods(5) <ins> now := metav1.NewTime(time.Now()) </ins> pods[0].Status.Phase = v1.PodFailed pods[1].Status.Phase = v1.PodSucceeded <ins> // The pod is terminating, should not filter out. </ins> pods[2].Status.Phase = v1.PodRunning <ins> pods[2].DeletionTimestamp = &now pods[2].Status.ContainerStatuses = []v1.ContainerStatus{ {State: v1.ContainerState{ Running: &v1.ContainerStateRunning{ StartedAt: now, }, }}, } </ins> pods[3].Status.Phase = v1.PodPending <ins> pods[4].Status.Phase = v1.PodRunning </ins> expected := []*v1.Pod{pods[2], pods[3], pods[4]} kubelet.podManager.SetPods(pods)

That said, I believe resources of terminated-but-not-deleted pods are usable on nodes (should be verified). So, the current behavior is a bit less than ideal in terms of efficiency, but AFAIK the duration between termination and deletion is only a few seconds. So, it may not be worth optimizing.
I thought was saying the opposite (see my first entry in this issue). Maybe he can clarify.


Question: podClient = f.PodClient() }) <del> ginkgo.Context("when creating a pod in the host PID namespace", func() { makeHostPidPod := func(podName, image string, command []string, hostPID bool) *v1.Pod { return &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name: podName, }, Spec: v1.PodSpec{ RestartPolicy: v1.RestartPolicyNever, HostPID:       hostPID, Containers: []v1.Container{ { Image:   image, Name:    podName, Command: command, }, }, }, } } createAndWaitHostPidPod := func(podName string, hostPID bool) { podClient.Create(makeHostPidPod(podName, framework.BusyBoxImage, []string{"sh", "-c", "pidof nginx || true"}, hostPID, )) podClient.WaitForSuccess(podName, framework.PodStartTimeout) } nginxPid := "" ginkgo.BeforeEach(func() { nginxPodName := "nginx-hostpid-" + string(uuid.NewUUID()) podClient.CreateSync(makeHostPidPod(nginxPodName, imageutils.GetE2EImage(imageutils.Nginx), nil, true, )) output := f.ExecShellInContainer(nginxPodName, nginxPodName, "cat /var/run/nginx.pid") nginxPid = strings.TrimSpace(output) }) ginkgo.It("should show its pid in the host PID namespace [LinuxOnly] [NodeFeature:HostAccess]", func() { busyboxPodName := "busybox-hostpid-" + string(uuid.NewUUID()) createAndWaitHostPidPod(busyboxPodName, true) logs, err := e2epod.GetPodLogs(f.ClientSet, f.Namespace.Name, busyboxPodName, busyboxPodName) if err != nil { framework.Failf("GetPodLogs for pod %q failed: %v", busyboxPodName, err) } pids := strings.TrimSpace(logs) framework.Logf("Got nginx's pid %q from pod %q", pids, busyboxPodName) if pids == "" { framework.Failf("nginx's pid should be seen by hostpid containers") } pidSets := sets.NewString(strings.Split(pids, " ")...) if !pidSets.Has(nginxPid) { framework.Failf("nginx's pid should be seen by hostpid containers") } }) ginkgo.It("should not show its pid in the non-hostpid containers [LinuxOnly] [NodeFeature:HostAccess]", func() { busyboxPodName := "busybox-non-hostpid-" + string(uuid.NewUUID()) createAndWaitHostPidPod(busyboxPodName, false) logs, err := e2epod.GetPodLogs(f.ClientSet, f.Namespace.Name, busyboxPodName, busyboxPodName) if err != nil { framework.Failf("GetPodLogs for pod %q failed: %v", busyboxPodName, err) } pids := strings.TrimSpace(logs) framework.Logf("Got nginx's pid %q from pod %q", pids, busyboxPodName) pidSets := sets.NewString(strings.Split(pids, " ")...) if pidSets.Has(nginxPid) { framework.Failf("nginx's pid should not be seen by non-hostpid containers") } }) }) </del> ginkgo.Context("When creating a container with runAsUser", func() { makeUserPod := func(podName, image string, command []string, userid int64) *v1.Pod { return &v1.Pod{

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: , Which test(s) are failing: Since when has it been failing: since this PR introduced them Testgrid link: Note that this is the flakiest test in both dashboards, and failing quite frequently. Reason for failure: unknown Anything else we need to know: cc
/sig node
see: , both getting empty PID and multiple PIDs.
opened revert at
for discoverability for the release team /priority critical-urgent /milestone v1.17


Question: return err } <ins> // ignoreStatusForbiddenFromError returns nil if the status code is StatusForbidden. // This happens when AuthorizationFailed is reported from Azure API. func ignoreStatusForbiddenFromError(err error) error { if err == nil { return nil } v, ok := err.(autorest.DetailedError) if ok && v.StatusCode == http.StatusForbidden { return nil } return err } </ins> /// getVirtualMachine calls 'VirtualMachinesClient.Get' with a timed cache /// The service side has throttling control that delays responses if there're multiple requests onto certain vm /// resource request in short period.

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: controller-manager is stuck in deleting an incorrectly configured LB type service. it cannot be recovered until restarted. What you expected to happen: controller-manager should recover. At least it should allow us to create the service without restarting it. How to reproduce it (as minimally and precisely as possible): put a random IP in above yaml and make sure the resource group doen't exist. Anything else we need to know?: Environment: Kubernetes version (use ): 1.12.4 Cloud provider or hardware configuration: AKS OS (e.g: ): Kernel (e.g. ): Install tools: Others:
/sig azure
I also want to add that the above repro is stuck in 403. It's possible for it to be stuck in 404 as well when the SP has the right permission.
Can you share the log for 404?
Thanks for reporting the issue. /assign Do you mean other services couldn't be configured when this happens?
correct.
FYI, we had the same issue when the Idle Timeout annotation was set to "3" which isn't a supported case. Will this PR also cover this edge case ?
Unfortunately , this is still not fixed. The idle timeout is not 403 or 404. We need another fix for this (and possibly more for other annotations)


Question: new.Spec.RequiresRepublish = &requiresRepublish }, }, <ins> { name: "StorageCapacity changed", modify: func(new *storage.CSIDriver) { new.Spec.StorageCapacity = &notStorageCapacity }, }, </ins> } for _, test := range successCases { t.Run(test.name, func(t *testing.T) {

Up to 1.21, that field is immutable. That means in practice, a CSIDriver object has to be deleted and then re-created with a different value instead. Mutation via delete/created is unnecessarily complex and as the only consumer of that field (the kube-scheduler) can handle mutation just fine (it'll always ask the informer cache for the current value), it's better to allow mutation.
/sig storage /assign
This is independent of allowing other fields to become mutable, but that might also make sense.
/triage accepted


Question: if in.XEmbeddedResource { out.VendorExtensible.AddExtension("x-kubernetes-embedded-resource", true) } <del> if in.XIntOrString { out.VendorExtensible.AddExtension("x-kubernetes-int-or-string", true) } </del> return nil }

The server-side validation does not check the type of values with . Create a CRD like: Then create a CR: It is not rejected, but should because must not be a boolean.


Question: started = !exists } podStatus.ContainerStatuses[i].Started = &started <ins> if started { var ready bool if c.State.Running == nil { ready = false } else if result, ok := m.readinessManager.Get(kubecontainer.ParseContainerID(c.ContainerID)); ok { ready = result == results.Success } else { // The check whether there is a probe which hasn't run yet. _, exists := m.getWorker(podUID, c.Name, readiness) ready = !exists } podStatus.ContainerStatuses[i].Ready = ready } </ins> } // init containers are ready if they have exited with success or if a readiness probe has // succeeded.

If I have probe and then all good. Previous version of the pod is terminated after startup probe success, but if wasn't specified the pod becomes ready immediately. It would be nice to add new state (for example starting) which would have pods until a is successful. Also, it would mean that container not ready for connection. Thank you for attention!
/sig cluster-lifecycle
/remove-sig cluster-lifecycle /sig apps try asking in #kubernetes-users on k8s slack.
they are silent in the chat. Who is involved in the development of startup probe?
i think this topic better fits SIG Node. try asking in their channel on slack , but be patient ;) /sig node /remove-sig apps
Those labels are not set on the issue: <detailsIn response to : i think this topic better fits SIG Node. /sig node /remove-sig apps </details
I think this is a feature request, not support, so I changed labels. On its face it SEEMS reasonable, but I am not close enough to the details to say. This is definitely sig-node.
Adding a new pod lifecycle state has been determined to be intractable in the past. Is the proposed change keeping the pod in !Ready until the first success from ?
Where do you usually discuss this? Just wondering. Yes, I want that pod considered not ready until will not be successful. If readiness probe doesn't present then a pod can become ready. How can a pod be ready if it is not running?
how long wait?
Looking at this issue, I can confirm it, at least superficially - a pod with a perma-failing is "ready" if there is no readinessGate defined and "not ready" if it is defined. The question I have is whether we think that is "correct" or not? Initially I thought it was correct, but the more I think about it the less convinced I am. If I have defined a startupProbe and it is not passing, my app should not be ready. Does anyone disagree?
Also, the app should not be ready in status. It's funny, but without a readiness probe, pods in the final state have the status ready and accept requests. This is unexpected.
Whether a pod is ready while terminating is an entirely different question, IMO.


Question: started = !exists } podStatus.ContainerStatuses[i].Started = &started <ins> if started { var ready bool if c.State.Running == nil { ready = false } else if result, ok := m.readinessManager.Get(kubecontainer.ParseContainerID(c.ContainerID)); ok { ready = result == results.Success } else { // The check whether there is a probe which hasn't run yet. _, exists := m.getWorker(podUID, c.Name, readiness) ready = !exists } podStatus.ContainerStatuses[i].Ready = ready } </ins> } // init containers are ready if they have exited with success or if a readiness probe has // succeeded.

Apps that server network traffic probably need to "drain" and to do that they need to stay "ready" for a while as upstream LBs de-program. Let's not conflate issues, please.
The fix seems simple - lets see what CI says.
Sorry, I don't spend much time on slack... you could have checked the release notes in 1.18 ;-) Tim's fix seems good, thanks!
/reopen Until we merge all related PRs...
Reopened this issue. <detailsIn response to : /reopen Until we merge all related PRs... Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
All merged. /close
Closing this issue. <detailsIn response to : All merged. /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: %.png: %.seqdiag $(FONT) seqdiag --no-transparency -a -f '$(FONT)' $< <ins> # Build the stuff via a docker image .PHONY: docker docker: docker build -t clustering-seqdiag . docker run --rm clustering-seqdiag | tar xvf - docker-clean: docker rmi clustering-seqdiag || true docker images -q --filter "dangling=true" | xargs docker rmi fix-clock-skew: boot2docker ssh sudo date -u -D "%Y%m%d%H%M.%S" --set "$(shell date -u +%Y%m%d%H%M.%S)" </ins>

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug /priority failing-test /area test-infra What happened: job is failing. See https://k8s- https://k8s- https://k8s- https://k8s- What you expected to happen: test should pass
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/assign
Reiterating the mentions to trigger a notification: <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I'm guessing that something got messed up in the bazel cache when reverting from 0.5.2 to 0.4.5. I'm not sure how best to fix.
How about updating everything to bazel 0.5.3 at once? I know you're trying to do a step-by-step approach to avoid breakages, but I suspect it will be cheaper to just deal with a brief outage while we update? (aka update the bazelbuild image to 0.5.3 and then merge your PR with updated and working BUILD rules?)
there's at least one other issue I've discovered with the latest go rules - the binary no longer works, due to changes introduced in that might be arguably better (at least everything still builds?), though it's frustrating that it makes things more broken in a different way.


Question: "log" "os" "os/exec" <ins> "os/signal" </ins> "path/filepath" "strings" "time"

Once e2e tests are running, the tests keep running even if I press and exit the process that controls the TTY. But the tests actually keep running in the background and they keep printing stuff on a shell prompt which has already returned and thereby clobbering the STDOUT
/assign
GitHub didn't allow me to assign the following users: oomichi. Note that only can be assigned. <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: }) It("should have all expected certs on the master", func() { <del> if testContext.Provider != "gce" && testContext.Provider != "gke" { </del> <ins> if !providerIs("gce", "gke") { </ins> By(fmt.Sprintf("Skipping MasterCerts test for cloud provider %s (only supported for gce and gke)", testContext.Provider)) return }

I noticed while running e2e tests locally that some tests were apparently being incorrectly skipped: I checked, and this is also true for our Jenkins runs. Currently it looks like and have this bug, though only the former actually skips the test, I think. The issue appears to be that the code inside the top-level in these files is being run before the testContext struct is filled out (and before is even called), and so testContext.Provider is empty. I'm not immediately sure what the proper fix is; I'm also worried that there might be other things broken by the fact that these blocks are called before initialization is complete. We also probably investigate if there is a better way to mark these tests as skipped; right now, there's no indication in the test report (, Jenkins test report, etc) that these tests were skipped.
I merged and which tried to disable these tests for non-{gke,gce} but apparently didn't follow-up and verify that they were still running on our platforms. /cc
Seems easy enough to fix. I'll have a PR out to fix this before Monday.


Question: if ds.Status.DesiredNumberScheduled == desiredNumberScheduled && ds.Status.CurrentNumberScheduled == currentNumberScheduled && ds.Status.NumberMisscheduled == numberMisscheduled { return nil } var updateErr, getErr error for i := 0; i <= StatusUpdateRetries; i++ { ds.Status.DesiredNumberScheduled = desiredNumberScheduled

The test is in flaky suite and no one seems to care. We need an owner for this test. cc Last failure log:
do you think you can take a look at this?
This is happening because of a bad assumption. The assumption is: if the reflector's store is consistent with etcd, the store given back to clients of the framework.Controller is too. I'd really like a general fix applicable to all controllers for this problem, but fixing it just for the RCs is pretty easy. In more detail: This function reports when the reflector has listed from apiserver. But the store given to the reflector is the fifo of the framework.Controller. The store given back to the caller is still not consistent, and will not be till the framework.Controller puts all the objects from the fifo into the client state store (). The rc specific fix: Before starting the rc manager, seed the expectations store with an expectation like {rc:foo, adds: foo.Status.Replicas, deletes:0}. This will wait for status.Replicas number of ADD events till a timeout. The general fix: Somehow expose this through framework.Controller.HasSynced (clients don't care about the fifo anyway, shouldn't it just be an implementation detail?) ideas? cc'ing people who have written controllers that might be interested I don't think this (and generally anything flaky for 1 release) is a p0, I can fix it if you don't mind dealing with the flakes till I've cleared things off my plate. I'd be more than happy to review a fix.
Well, it's a P0 because test is flaky. E.g last run:
- did you have chance to look into it?
The "DaemonRestart" tests are still labeled at . I will send a PR to promote them.
are you still planning to send aPR to promote the DaemonRestart tests out of flaky? I think this is the right thing to do.
Ack; sorry, I lost this in the muck.


Question: ObjectMeta: metav1.ObjectMeta{ Name: string(kl.nodeName), Labels: map[string]string{ <del> metav1.LabelHostname:       kl.hostname, metav1.LabelOS:             goruntime.GOOS, metav1.LabelArch:           goruntime.GOARCH, metav1.LabelFluentdDsReady: "true", </del> <ins> metav1.LabelHostname: kl.hostname, metav1.LabelOS:       goruntime.GOOS, metav1.LabelArch:     goruntime.GOARCH, </ins> }, }, Spec: v1.NodeSpec{

PR introduced an annotation for fluentd migration from static pod to the daemonset to avoid two instances of fluentd on a single node during the transition stage. I didn't read the pr carefully, but based on the code at , every node object has one label be default: "". I think this is wrong. 1) Not everyone use fluentd as the logging agent. 2) It is hard-coded in Kubelet code as part of kubeletnodestatus. No matter running fluentd as a static pod or a daemonset on the node, Kubelet shouldn't treat it different from other pods. The right approach is introducing an external controller to apply this cloud-provider / kubernetes vendor (i.e. GKE) specific label to the node. cc/
The string "fluentd" shouldn't appear anywhere in our code. Please remove it ASAP.
Absolutely agree with you, thanks for bringing this up! Is it possible to apply labels to the node as part of the node startup script/salt?
To clarify, I'm thinking about possible ways to provide a fix for that ASAP. Writing external controller is a best solution, but here we have a release coming and this label is intended to be only a temporary measure to perform upgrade from using fluentd as static pods to using daemon set.
There are many ways to achieve this outside the core system. But I believe GKE already has an existing mechanism to add this kind of labels to the node, and the consistency is important for GKE management. cc/
OK, thanks! However, I believe we should be sure not to break everyone's upgrade process, i.e. after master upgrade to 1.6 on any platform, there should be only one fluentd agent on each node. I propose changing the configure scripts and salt to set up needed labels in kubelet's flag (). WDYT?
It seems to me like upgrades could survive without the explicit dependency on fluentd: I'm not sure if upgrades call initialNode or not, my own ignorance ? If they do, and this is removed, can't an external controller just do the work of re--adding that label if its dropped?
thanks for raising this. Let's sync f2f
cc
WDYT about exposing kubelet version as a label and have a selector in fluentd ds config targeting version 1.6+?


Question: ObjectMeta: metav1.ObjectMeta{ Name: string(kl.nodeName), Labels: map[string]string{ <del> metav1.LabelHostname:       kl.hostname, metav1.LabelOS:             goruntime.GOOS, metav1.LabelArch:           goruntime.GOARCH, metav1.LabelFluentdDsReady: "true", </del> <ins> metav1.LabelHostname: kl.hostname, metav1.LabelOS:       goruntime.GOOS, metav1.LabelArch:     goruntime.GOARCH, </ins> }, }, Spec: v1.NodeSpec{

and I talked about this issue a bit for migrating kube-proxy. One idea we came up with was that of running a controller that will identify kubelet versions and apply taints to nodes running static pods. The daemonset version would not tolerate that taint and that way we keep daemon set pods off of nodes running static pods. Have you considered that approach?
It doesn't differ from a controller applying labels. I'd like to avoid necessity of introducing such controller, since it's not clear what is it's responsibility (generic controller for applying labels on nodes based on some rules? a custom controller just for fluentd purposes). In GKE we can do it on GKE control plane side, but we need a story for open source.
Move applying labels to startup scripts/salt seems to be the least intrusive and easiest to implement way.
Actually what I proposed in is not an option since nodeSelector in PodSpec is old style selector.
I'm not a big fan of approach with salt, but it seems there is no better option.
Re: Instead of exposing Kubelet version as a label, why not using field selector. Kubelet version is part of NodeStatus already. I just simply don't want to have a duplicate info for the same object.
is it possible to schedule based on fieldselector? I'm only aware of using field selector while listing objects.
Also node-labels option is in alpha
But it's present since 1.2 What is the status of this flag? Can we use it for fluentd purposes or better avoid this?
Maybe it does make sense to implement a temporary ad-hoc controller? It's the easiest to deploy solution, just another in fluentd addon.
I know. It totally doesn't make sense to introduce anything custom. I'll use just the flag. It's rather hacky but unfortunately I can't see anything better.
Cool thing about controller is that you can deploy it into any cluster and it will work, with CA also. You don't have to set up your node labels manually then or restart the cluster.
There is a number of open question with having a new controller: it run on the master or a node? we guarantee that it will be always running? it have any resource requirements? I'd like to avoid discussing it right before the release.
Having a controller will help with kube-proxy transition too in the future.


Question: ObjectMeta: metav1.ObjectMeta{ Name: string(kl.nodeName), Labels: map[string]string{ <del> metav1.LabelHostname:       kl.hostname, metav1.LabelOS:             goruntime.GOOS, metav1.LabelArch:           goruntime.GOARCH, metav1.LabelFluentdDsReady: "true", </del> <ins> metav1.LabelHostname: kl.hostname, metav1.LabelOS:       goruntime.GOOS, metav1.LabelArch:     goruntime.GOARCH, </ins> }, }, Spec: v1.NodeSpec{

, Piotr Szczesniak <:
is this a 1.6 release blocker?
I think yes. The PR is already sent and approved. I need to fix e2e tests because it doesn't work well with the test infra (e2e tests passes locally). EDIT: I'll do it hopefully today.
The PR is ready to be merged and lgtmed. Waiting for approvals.


Question: expectedPatchDataPattern: `{"status":{"$setElementOrder/conditions":[{"type":"currentType"}],"conditions":[{"lastProbeTime":"2020-05-13T01:01:01Z","message":"newMessage","reason":"newReason","type":"currentType"}]}}`, }, { <del> name: "Should make patch request if pod condition already exists and is identical but nominated node name is different", </del> <ins> name: "Should not make patch request if pod condition already exists and is identical and nominated node name is not set", </ins> currentPodConditions: []v1.PodCondition{ { Type:               "currentType",

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: Conformance - GCE - master (ci-kubernetes-gce-conformance-latest) Which test(s) are failing: [sig-scheduling] SchedulerPreemption [Serial] PreemptionExecutionPath runs ReplicaSets to verify preemption running path [Conformance]Changes Since when has it been failing: 2020-06-08 22:27 PDT Testgrid link: Reason for failure: Regression by Anything else we need to know: /sig scheduling /priority critical-urgent
cc/ /assign


Question: MUST match to expected used and total allowed resource quota count within namespace. release: v1.16 file: test/e2e/apimachinery/resource_quota.go <ins> - testname: ResourceQuota, manage lifecycle of a ResourceQuota codename: '[sig-api-machinery] ResourceQuota should manage the lifecycle of a ResourceQuota [Conformance]' description: Attempt to create a ResourceQuota for CPU and Memory quota limits. Creation MUST be successful. Attempt to list all namespaces with a label selector which MUST succeed. One list MUST be found. The ResourceQuota when patched MUST succeed. Given the patching of the ResourceQuota, the fields MUST equal the new values. It MUST succeed at deleting a collection of ResourceQuota via a label selector. release: v1.25 file: test/e2e/apimachinery/resource_quota.go </ins> - testname: ResourceQuota, quota scope, BestEffort and NotBestEffort scope codename: '[sig-api-machinery] ResourceQuota should verify ResourceQuota with best effort scope. [Conformance]'

[x] test approval issue : [x] test pr : 7 July 2022 [X] two weeks soak end date : 21 July 2022 [X] test promotion pr : According to this APIsnoop query, there is still some remaining ResourceQuota endpoints which are untested. The Status endpoints will be addressed in another test. listCoreV1ResourceQuotaForAllNamespacespatchCoreV1NamespacedResourceQuotadeleteCoreV1CollectionNamespacedResourceQuota Using a number of existing e2e test practices a new has been created for three ResourceQuota endpoints. The e2e logs for this test are listed below. This query shows the endpoints hit within a short period of running the e2e test If a test with these calls gets merged, test coverage will go up by 3 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/triage accepted
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: expectedErr:     true, }, { <del> desc:        "correct LUN and no error shall be returned if everything is good", vmList:      map[string]string{"vm1": "PowerState/Running"}, nodeName:    "vm1", existedDisk: compute.Disk{Name: to.StringPtr("disk-name"), DiskProperties: &compute.DiskProperties{Encryption: &compute.Encryption{DiskEncryptionSetID: &diskEncryptionSetID, Type: compute.EncryptionAtRestWithCustomerKey}}, Tags: testTags}, </del> <ins> desc:     "correct LUN and no error shall be returned if everything is good", vmList:   map[string]string{"vm1": "PowerState/Running"}, nodeName: "vm1", existedDisk: compute.Disk{Name: to.StringPtr("disk-name"), DiskProperties: &compute.DiskProperties{ Encryption: &compute.Encryption{DiskEncryptionSetID: &diskEncryptionSetID, Type: compute.EncryptionAtRestWithCustomerKey}, DiskSizeGB: to.Int32Ptr(4096), }, Tags: testTags}, </ins> expectedLun: 1, expectedErr: false, },

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: there is a disk resize scenario: In the beginning, it’s 3TiB with cachingMode=ReadOnly, and then disk detached, resized to 4TiB, then user find they could not attach that disk back to any agent node, I think that’s a bug we need to fix. What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /kind bug /assign /priority important-soon /sig cloud-provider /area provider/azure
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: sanitizedSpecVolID := kstrings.EscapeQualifiedNameForDisk(specVolID) return path.Join(host.GetVolumeDevicePluginDir(csiPluginName), sanitizedSpecVolID, "data") } <ins> // hasReadWriteOnce returns true if modes contains v1.ReadWriteOnce func hasReadWriteOnce(modes []api.PersistentVolumeAccessMode) bool { if modes == nil { return false } for _, mode := range modes { if mode == api.ReadWriteOnce { return true } } return false } </ins>

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: fsgroup is only intended for RWO volumes because it recursively changes the permissions of all the directories in the volume. This can cause problems for nfs-type volumes because multiple pods could access the same volume and have the permissions changed out from underneath. What you expected to happen: Determine whether or not the plugin supports fsgroup either via: access mode. This is still a bit ambiguous for nfs volume types, that can theoretically support rwo, especially since Kubernetes does not do any sort of access mode enforcement. a new csi capability
CC
Are you saying that CSI internal driver should check driver capability for RWO, if supported, only then apply value ? What would the be ?
Yes one possible way is to infer fsgroup is supported for all rwo volumes, which may not be true? Like a nfs volume technically supports rwo but not fsgroup. So the other option is to make it a new CSI capability. I'm not sure exactly sure how best to describe it though, maybe something about volume ownership management.
Will it work if is not set then don't chown the ownership? I would think that for shared file system types(nfs, glusterfs) - won't be set. For raw block storage too - I think we shouldn't have to do this.
what do you mean by ? Shouldnt have to solve this issue ?
I meant - which is obvious. I just meant that the entire mechanism of changing ownership of files should apply only to volume types with block storage filesystems. I am just wondering if we can get away with detecting fsType and not have to introduce new capability in CSI.
agreed, do not map well to current csi.Capabilities enums which makes it hard to determine RWO when mounting. Perhaps, instead of a new capability, we revisit the existing capabilities to see if there are any way RWO can be reliably inferred from returned capabilities.


Question: sanitizedSpecVolID := kstrings.EscapeQualifiedNameForDisk(specVolID) return path.Join(host.GetVolumeDevicePluginDir(csiPluginName), sanitizedSpecVolID, "data") } <ins> // hasReadWriteOnce returns true if modes contains v1.ReadWriteOnce func hasReadWriteOnce(modes []api.PersistentVolumeAccessMode) bool { if modes == nil { return false } for _, mode := range modes { if mode == api.ReadWriteOnce { return true } } return false } </ins>

Currently, this is what the code is doing which we knew needed to be revisited:
Users may not fully understand what fsgroup is and set it anyway, or a default psp with some fsgroup may be defined in which it will automatically be set on all pods. The first case we can attribute to user error, but the second case may not have a good workaround and could cause permission issues on rwx volumes. So I think we do need to solve this.
Yes, agreed this has to be solved. As you pointed out user error or improperly configured PSP can set value that does not match the correct attribute, causing permission issues. Here is what I think can be done without new CSI capabilities: , then do not apply fsgroup then: , apply fsgroup , do not apply fsgroup to void stated issue then: , apply fsgroup I think that may help deduce when to apply fsgroup.
If mount reference count <= 1, apply fsgroup If mount reference count 1, do not apply fsgroup to void stated issue A RWM volume (say NFS) can be mounted on multiple nodes. Do you have reliable count of mounts across all nodes?
In addition, that would be a change in semantics to how we support rwx volumes today. If we don't have a new CSI capability, then I think the best course of action is to only apply fsgroup for RWO volumes. Some questions remain: Do all RWO volumes support fsgroup? RWX is a superset of RWO. So if a user requests a RWO volume, it could still be satisfied by a RWX volume type. Can we detect this and still not apply the fsgroup even though the volume capability will say RWO?
If there's not a reliable way to enumerate node mounts (thinking about it, for CSI external drivers, the answer is NO). Theres is no reliable way to have k8s autocorrect a bad fsgroup/PV.AccessMode combo. So we would have to rely on the config provided by user/admin. I have come to similar conclusion as -- mainly, apply fsgroup only to volumes with access modes. The other k8s specified access modes, , cannot be reliably applied.


Question: sanitizedSpecVolID := kstrings.EscapeQualifiedNameForDisk(specVolID) return path.Join(host.GetVolumeDevicePluginDir(csiPluginName), sanitizedSpecVolID, "data") } <ins> // hasReadWriteOnce returns true if modes contains v1.ReadWriteOnce func hasReadWriteOnce(modes []api.PersistentVolumeAccessMode) bool { if modes == nil { return false } for _, mode := range modes { if mode == api.ReadWriteOnce { return true } } return false } </ins>

Even if we were to introduce additional plugin capabilities that the CO can query (from driver) to find out what modes can be applied, there would still be the possibility of user/admin mis-configuration. Probably not, Is this where you think an additional capability would help ( ) That is a good point. Can't think of a way to detect/guess RWX when RWO is specified, this without the driver providing that info. Can we adopt the rule until we decide what capabilities would look like ?
I will just try to summarize the conversation from Slack. We can safely assume that only applies to block storage type with file system on top of it. So for all - volumes that need to have called MUST have a file system() on them. Is that assumption correct? If it is - can we not use presence of field or we can query the local volume for fs type (like how we query selinux relabelling capability) and call only when - volume is RW and has a valid block storage file system type. Will this not solve our use case? EDIT: renamed to .
We'll need to make it clear that is only for block-based storage systems, and the field cannot be reused for some multi-protocol file server (ie, nfsv3, nfsv4, smb, etc). For that case, then the plugin will need to expose its own new storageclass parameter + volume attribute to specify protocol.
Can we not use to get fsType - and use that information for calling ? It may be problematic to rely entirely on present inside csi source.
the only problem with is that in CSI volume info is intentionally opaque to the CSI code inside k8s. All mount logic is delegated to the driver.
Instead of a new capability or trying to deduce how best to apply fsgroup, why not pass the to the driver and let it decide how to apply it (either as a attribute for the mount or top level request param)? Right now, all mount logic has been delegated to the external CSI driver except for applying permission. Thoughts ?
that is true but we can still inspect file system on the mounted path because by the time gets called volume is already mounted. This should be no different than how selinux support is determined. what you think?
is this approved for 1.12 milestone? Thank you for working on this,
still discussing how best to approach this.