Question: verifyAttachedVolume(t, attachedVolumes, generatedVolumeName2, string(volumeName), node2Name, devicePath2, true /* expectedMountedByNode */, false /* expectNonZeroDetachRequestedTime */) } <ins> // Test_SetNodeStatusUpdateNeededError expects the map nodesToUpdateStatusFor // to be empty if the SetNodeStatusUpdateNeeded is called on a node that // does not exist in the actual state of the world func Test_SetNodeStatusUpdateNeededError(t *testing.T) { // Arrange volumePluginMgr, _ := volumetesting.GetTestVolumePluginMgr(t) asw := NewActualStateOfWorld(volumePluginMgr) nodeName := types.NodeName("node-1") // Act asw.SetNodeStatusUpdateNeeded(nodeName) // Assert nodesToUpdateStatusFor := asw.GetNodesToUpdateStatusFor() if len(nodesToUpdateStatusFor) != 0 { t.Fatalf("nodesToUpdateStatusFor should be empty as nodeName does not exist") } } </ins> func verifyAttachedVolume( t *testing.T, attachedVolumes []AttachedVolume,

This is most likely due to lack of PodCIDR] 6m 14s 4 {kubelet kubernetes-minion-group-yxra} Warning FailedMount Unable to mount volumes for pod "test-ipj93_default(-b5bb-11e6-bab0-)": timeout expired waiting for volumes to attach/mount for pod "test-ipj93"/"default". list of unattached/unmounted volumes=[data] 6m 14s 4 {kubelet kubernetes-minion-group-yxra} Warning FailedSync Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod "test-ipj93"/"default". list of unattached/unmounted volumes=[data] What you expected to happen: GCE PD to attach and mount successfully post the upgrade How to reproduce it (as minimally and precisely as possible): Running the upgrade steps in the doc for upgrade of manual test from 1.4 to 1.5 Anything else do we need to know:
Ran the upgrade again. This time the .
kube-controller-
Current theory is because kubelet deletes and recreates the node API object during upgrade (because externalID changed), see the new object does not contain the correct status (it is written by the Attach/Detach controller to the old object but not subseqently updated). If this proves to be the issue, the proposed fix is to force the attach/detach controller to update the node status when the node's external ID changes. is working to verify this theory by executing this code path (without a full upgrade). Once he has validated the theory, he will work on the fix, verify it, merge it to master, and get it cherry-picked to 1.5.
: is expected to fix this problem?
: Yes


Question: import ( "fmt" "io/ioutil" <ins> "sort" </ins> "strings" "github.com/spf13/cobra"

/kind bug What happened: When building kubernetes-1.7.0 packages for openSUSE, results differ for each build in and other man pages. coming from that uses and probably sets do not have any specific order, so it would need to be sorted on output. What you expected to happen: Every build should give identical results. See https://reproducible- for why this matters. How to reproduce it (as minimally and precisely as possible): See Anything else we need to know?: Environment: Kubernetes version: 1.7.0 OS (e.g. from /etc/os-release): openSUSE Tumbleweed
/sig release
somehow this bug seems to have been forgotten - can you help getting the right people to have a look?
- can you please confirm that PR works for you?
I tested it and it looks good.
/assign I enjoy reading my highlights to find the issue has already been addressed. :)
Sorry this wasn't addressed sooner. It does seem like it fell into a black hole, and pinging was the right way to resolve that. :)
/area build-release
related to /reopen
Reopening this issue. <detailsIn response to : related to Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: if plugin.getDefaultNetwork() == nil { return errors.New("cni config uninitialized") } <ins> // If the CNI configuration has the ipRanges capability, we need a PodCIDR assigned for _, p := range plugin.getDefaultNetwork().NetworkConfig.Plugins { if p.Network.Capabilities["ipRanges"] { if plugin.podCidr == "" { return errors.New("no PodCIDR set") } break } } </ins> return nil } <ins> // Event handles any change events. The only event ever sent is the PodCIDR change. // No network plugins support changing an already-set PodCIDR func (plugin *cniNetworkPlugin) Event(name string, details map[string]interface{}) { if name != network.NET_PLUGIN_EVENT_POD_CIDR_CHANGE { return } plugin.Lock() defer plugin.Unlock() podCIDR, ok := details[network.NET_PLUGIN_EVENT_POD_CIDR_CHANGE_DETAIL_CIDR].(string) if !ok { glog.Warningf("%s event didn't contain pod CIDR", network.NET_PLUGIN_EVENT_POD_CIDR_CHANGE) return } if plugin.podCidr != "" { glog.Warningf("Ignoring subsequent pod CIDR update to %s", podCIDR) return } plugin.podCidr = podCIDR } </ins> func (plugin *cniNetworkPlugin) Name() string { return CNIPluginName }

/kind feature The CNI "capability arguments" parameter - currently used to pass port mappings - has a few new capabilities: bandwidth limiting and network address management. The dockershim's networking driver should use them. The network address is just passing the PodCIDR dynamically to the plugin, so it no longer has to retrieve it from the APIServer. Bandwidth limiting is already implemented in Kubenet. It is configured via annotations. There is no upgrade risk in providing these values; non-compliant plugins will simply ignore them. Administrators need to explicitly enable each capability individually in their CNI configuration file. To avoid skew between runtimes, the other runtimes should also be upgraded to pass these. We may also wish to look in to a way to add this more directly to the CRI, so we don't have this skew problem, but that's a different issue.
/sig network
/assign
fyi.
Looks like this was never done for CRI?
cc


Question: other platforms like Windows. release: v1.9 file: test/e2e/common/networking.go <ins> - testname: Proxy, validate ProxyWithPath responses codename: '[sig-network] Proxy version v1 A set of valid responses are returned for both pod and service ProxyWithPath [Conformance]' description: Attempt to create a pod and a service. A set of pod and service endpoints MUST be accessed via ProxyWithPath using a list of http methods. A valid response MUST be returned for each endpoint. release: v1.21 file: test/e2e/network/proxy.go </ins> - testname: Proxy, logs service endpoint codename: '[sig-network] Proxy version v1 should proxy through a service and a pod  [Conformance]' description: Select any node in the cluster to invoke  /logs endpoint  using the

[x] two weeks soak start date : testgrid-link [x] two weeks soak end date : [ ] test promotion pr : kubernetes/kubernetes#? According to this APIsnoop query, there are still some remaining ProxyWithPath endpoints which are untested. with this query you can filter untested endpoints by their category and eligiblity for conformance. e.g below shows a query to find all conformance eligible untested,stable,core endpoints Create a pod with a static label using the agnhost image Confirm that the pod is in the running phase Iterate through a list of http methods and check the response from the porter app Confirm that each response code is 200 OK If a test with these calls gets merged, test coverage will go up by 4 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance Where/How does PATCH/HEAD/OPTIONS requests for the following endpoints line up with the API reference? connectCoreV1PatchNamespacedPodProxyWithPath connectCoreV1HeadNamespacedPodProxyWithPath connectCoreV1OptionsNamespacedPodProxyWithPath Pod v1 core: Proxy Operations the following endpoints are listed as valid endpoints; connectCoreV1HeadNamespacedPodProxyWithPath connectCoreV1OptionsNamespacedPodProxyWithPath These requests are getting logged as a verb. Line : api/openapi-spec Line : api/openapi-spec Looking at there doesn't look to be a mapping for the above methods. According to this APIsnoop query, there are still some remaining ProxyWithPath endpoints which are untested. with this query you can filter untested endpoints by their category and eligiblity for conformance. e.g below shows a query to find all conformance eligible untested,stable,core endpoints Create a pod with a static label using the agnhost image Confirm that the pod is in the running phase Create a service and link it to the pod Iterate through a list of http methods and check the response from the porter app Confirm that each response code is 200 OK If a test with these calls gets merged, test coverage will go up by 4 points This test is also created with the goal of conformance promotion.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance.


Question: other platforms like Windows. release: v1.9 file: test/e2e/common/networking.go <ins> - testname: Proxy, validate ProxyWithPath responses codename: '[sig-network] Proxy version v1 A set of valid responses are returned for both pod and service ProxyWithPath [Conformance]' description: Attempt to create a pod and a service. A set of pod and service endpoints MUST be accessed via ProxyWithPath using a list of http methods. A valid response MUST be returned for each endpoint. release: v1.21 file: test/e2e/network/proxy.go </ins> - testname: Proxy, logs service endpoint codename: '[sig-network] Proxy version v1 should proxy through a service and a pod  [Conformance]' description: Select any node in the cluster to invoke  /logs endpoint  using the

The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/re-open
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: import ( "fmt" <ins> "net" </ins> "time" v1 "k8s.io/api/core/v1"

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Using Kubernetes 1.16.1, I started kube-proxy in iptables mode as a static pod on a master node. The master node was intentionally (and permanently) not available in the API as a Node object. kube-proxy started up and terminated soon after because it failed to (which is supposed to function as the bound address) and eventually returned an error: What you expected to happen: I expected the same behavior as in Kubernetes 1.15 where failure to look up the Node in the API would result in . How to reproduce it (as minimally and precisely as possible): Configure kube-proxy with iptables mode on Kubernetes 1.16 without the hosting Node being available in the Kubernetes API (under the OS hostname). Anything else we need to know?: PR (Fix for kube-proxy to wait for some duration for the node to be defined) introduced an additional . I believe this is the part that is causing the change in behavior: in Kubernetes 1.15, the check would be omitted and cause to set the default bind address to 127.0.0.1 (when iptables mode is set; I haven't checked other modes). The PR's primary purpose seems to be a different one (retrying node IP retrieval from the API), so my understanding is that the check was not really needed; unless I'm misunderstanding things which I don't want to rule out at all. The quick workaround was to explicitly configure kube-proxy with . However, I'm not entirely sure if that's a good solution long-term compared to letting Kubernetes do The Right Thing, which is what I'd like to clarify through this issue. Environment: Kubernetes version (use ): v1.16.1 Cloud provider or hardware configuration: Kubernetes running on DigitalOcean VMs (droplets) OS (e.g: ): Debian GNU/Linux 9 (stretch) Kernel (e.g.


Question: import ( "fmt" <ins> "net" </ins> "time" v1 "k8s.io/api/core/v1"

): Linux master-k8s-1-16-test1 4.19.0-0.bpo.6-amd64 SMP Debian 4.19.67-2+deb10u1~bpo9+1 (2019-09-30) x86_64 GNU/Linux /sig network /cc who were involved in the referenced PR.
I made a similar comment in , wondering if this is a breaking change in v1.16 that we should revert? Previously if is nil, kube-proxy defaults to but now kube-proxy always exits with error instead of defaulting to local host. I think a lot of external cloud providers relied on this behavior because they needed to talk to API server prior to node addresses being set by the provider. Thoughts /assign
Opened to preserve behavior before v1.16 where node IP defaults to 127.0.01 is none is found. PTAL


Question: fs.StringVar(&c.TLSMinVersion, "tls-min-version", c.TLSMinVersion, "Minimum TLS version supported. "+ "Value must match version names from https://golang.org/pkg/crypto/tls/#pkg-constants.") <ins> fs.BoolVar(&c.RotateCertificates, "rotate-certificates", c.RotateCertificates, "<Warning: Beta feature> Auto rotate the kubelet client certificates by requesting new certificates from the kube-apiserver when the certificate expiration approaches.") </ins> fs.Int32Var(&c.RegistryPullQPS, "registry-qps", c.RegistryPullQPS, "If > 0, limit registry pull QPS to this value.  If 0, unlimited.") fs.Int32Var(&c.RegistryBurst, "registry-burst", c.RegistryBurst, "Maximum size of a bursty pulls, temporarily allows pulls to burst to this number, while still not exceeding registry-qps. Only used if --registry-qps > 0")

/kind cleanup What happened: and both require opt-in configuration knobs to the kubelet. They are kinda global settings, you probably either opt into dynamic cert rotation or you don't, so they qualify for being in the struct. The knob for enabling dynamic serving certs was (I fully support that :+1:). What you expected to happen: I'd like the same to happen to  to be consistent. (Also, we're planning to use the config file for the kubelet by default in kubeadm soon, and it'd be nice to be able to set this value then in a structured manner) /assign


Question: fs.StringVar(&c.TLSMinVersion, "tls-min-version", c.TLSMinVersion, "Minimum TLS version supported. "+ "Value must match version names from https://golang.org/pkg/crypto/tls/#pkg-constants.") <ins> fs.BoolVar(&c.RotateCertificates, "rotate-certificates", c.RotateCertificates, "<Warning: Beta feature> Auto rotate the kubelet client certificates by requesting new certificates from the kube-apiserver when the certificate expiration approaches.") </ins> fs.Int32Var(&c.RegistryPullQPS, "registry-qps", c.RegistryPullQPS, "If > 0, limit registry pull QPS to this value.  If 0, unlimited.") fs.Int32Var(&c.RegistryBurst, "registry-burst", c.RegistryBurst, "Maximum size of a bursty pulls, temporarily allows pulls to burst to this number, while still not exceeding registry-qps. Only used if --registry-qps > 0")

Flag name: Help text: <Warning: Beta featureAuto rotate the kubelet client certificates by requesting new certificates from the kube-apiserver when the certificate expiration approaches. This is part of migrating the Kubelet command-line to a Kubernetes-style API. The flag should either be migrated to the Kubelet's API group, or simply removed from the Kubelet. If this could be considered an instance-specific flag, or a descriptor of local topology managed by the Kubelet, see: If this flag is only registered in os-specific builds, see: /assign /sig node /kind feature /priority important-soon /milestone v1.11 /status approved-for-milestone
[MILESTONENOTIFIER] Milestone Issue: Up-to-date for process <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. : New functionality. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details


Question: Format:      "", }, }, <del> "listMeta": { </del> <ins> "metadata": { </ins> SchemaProps: spec.SchemaProps{ Description: "Standard list metadata", Default:     map[string]interface{}{},

Listing s on v1.31.0 returns an object with a field instead of the standard field for (list) objects: Listing s on v1.30.3 behaves correctly: This is also propagated to the Swagger spec (which is how I noticed it, since I maintain the Rust bindings for the client API). v1.31.0: v1.30.3: As explained above. v1.31.0 with feature gate enabled and runtime config . With kind, this is: the API server request as in the above curl command in any way you like as long as it lets you see the raw JSON response. No response <details</details<detailskind 0.23.0 </details<details</details<details</details<details</details<details</details
/sig api-machinery
It seems there may be a discrepancy in the API response format between v1.30.3 and v1.31.0. The object is expected to follow standard Kubernetes API conventions, which typically use the field for list objects.
This deviation might cause compatibility issues for API clients that rely on this structure.
If this deviation from the expected API structure wasn't intentional, it should be aligned with the standard conventions.
The presence of instead of could disrupt client integrations, as you've encountered with the Swagger spec and Rust bindings.
cc
/triage accepted
/assign
this must be fixed to json:"metadata" protobuf:"bytes,1,opt,name=metadata"`` the serialization will not work with standard clients
/priority critical-urgent
Shoot, that's clearly a mistake.


Question: "flexvolume_mounted_volume_resize.go", "flexvolume_online_resize.go", "generic_persistent_volume-disruptive.go", <ins> "host_path_type.go", </ins> "in_tree_volumes.go", "mounted_volume_resize.go", "nfs_persistent_volume-disruptive.go",

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: This feature is lacking coverage, especially for containerized kubelet environments.
/assign since I'm the owner of this feature.
And just fixes a bug for containerized kubelet when the mount file is an empty file.
Are we going to enable these tests in v1.11? We're getting code freeze soon.
There is a local containerized cluster pull and CI job that we can probably leverage. Right now it only runs conformance tests, we may be able to add new suites to enable other tests. Cc who can point us to the owner for the containerized testing effort
yes, seems to be holding up pretty well - https://k8s-,%20master%20(dev)
In I made pull-kubernetes-local-e2e-containerized to run also all test with string. I'm enabling some tests (subpath!) in , feel free to suggest more tests.
Ref
IIUC, the default host path type is . And when host path type is , we expect container runtime to create the host path if it doesn't exist (docker is doing that today). Is that true? If that is, we should revert part of , because container runtime should still create host path if it does not exist. /cc I tried to enforce , and got a lot of errors , e.g.:
Correct. This is mainly for backwards compatibility.
I see. If so, I'll remove the CRI definition "If the hostPath doesn't exist (e.g. hostPath volume), runtimes should report errors". Thanks!
SGTM. Refer to .
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale will help fix this.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close.


Question: "flexvolume_mounted_volume_resize.go", "flexvolume_online_resize.go", "generic_persistent_volume-disruptive.go", <ins> "host_path_type.go", </ins> "in_tree_volumes.go", "mounted_volume_resize.go", "nfs_persistent_volume-disruptive.go",

If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with .


Question: "flexvolume_mounted_volume_resize.go", "flexvolume_online_resize.go", "generic_persistent_volume-disruptive.go", <ins> "host_path_type.go", </ins> "in_tree_volumes.go", "mounted_volume_resize.go", "nfs_persistent_volume-disruptive.go",

Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: # Run boilerplate check if [[ ${#files_need_boilerplate[@]} -gt 0 ]]; then for file in "${files_need_boilerplate[@]}"; do <del> echo "Boilerplate header is wrong for: ${file}" </del> <ins> echo "Boilerplate header is wrong for: ${file}" >&2 </ins> done exit 1

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: When running on my commits locally I got the following error: Affected PRs:
/sig testing
/cc
There's likely something up with your local environment. These jobs are running correctly in CI, although you need to look at the build log for output


Question: serviceAccountName: metrics-server containers: - name: metrics-server <del> image: k8s.gcr.io/metrics-server-amd64:v0.3.0 </del> <ins> image: k8s.gcr.io/metrics-server-amd64:v0.3.1 </ins> command: - /metrics-server - --metric-resolution=30s

This started after was merged. The HPA test started being very flaky since then: https://k8s- The HPA test in containerd e2e become even more flaky since then with the same failure pattern: https://k8s-
I think this is probably /priority critical-urgent Release team hasn't looked close enough at this issue as we've been trying to clear other issues in front of it. If we crack those and this is unrelated root cause, then this one's going to be a release blocker.
It's going to be supremely hard to debug this unless we have logs from metrics-server and/or snapshots of the HPA over time. I'm trying to reproduce on a local cluster, but so far, the test passes consistently. did you manage to reproduce locally?
is there any way to grab logs of pods from a testing cluster?
you probably want to save it as part of the artifacts?
yeah, what's the process for that?
somewhere along
See for an attempt at debugging this.
can we provide the data is looking for? we're seeing this quite often.
ack, more logs would be helpful. Currently trying to get usable logs off the CI with the aforementioned PR, but I keep making minor mistakes in the log collection script :-/. It seems to repro every few runs though, so with any luck I'll see it again soon.
Got a WIP fix up: Need to do one more tweak before merge.
Re-running tests with the above PR, if they succeed I'll merge and get someone to do a build when the people capable of running builds are on next.
status check? I'm worried this is a 1.12 blocker and need to make a call by end of day today regarding delaying the release.
According to our testing, metrics-server 0.3.1 seems to have fixed it - thank you


Question: if cpuList[1] != 1 { cpu1, cpu2 = cpuList[1], 1 } <ins> if isMultiNUMA() { cpuList = cpuset.MustParse(getCoreSiblingList(0)).ToSlice() cpu2 = cpuList[1] } } else if isMultiNUMA() { cpuList = cpuset.MustParse(getCoreSiblingList(0)).ToSlice() cpu1, cpu2 = cpuList[1], cpuList[2] </ins> } expAllowedCPUsListRegex = fmt.Sprintf("^%dn$", cpu1) err = f.PodClient().MatchContainerOutput(pod1.Name, pod1.Spec.Containers[0].Name, expAllowedCPUsListRegex) framework.ExpectNoError(err, "expected log not found in container [%s] of pod [%s]",

Which jobs are failing: N/A - AFAICT CI jobs works fine Which test(s) are failing: Since when has it been failing: no historical record on CI Testgrid link: N/A Reason for failure: I believe the tests make too many assumptions around the hardware, and the code is actually OK. to be confirmed. Anything else we need to know:
/sig node /sig testing
also affects the recently topology manager e2e tests. My PR uses a different approach to check resource alignment so is not affected. /cc
I'll work on a PR to fix this issue ASAP
Thanks for opening this issue. We knew about this, just had not opened an issue to track it, so thanks for doing that! Deepthi has been looing at CPU Manager tests, not sure if she has some PR in progress or not. I'll make sure to forward this to her (can find her user id here ....) in case she has something in progress


Question: // Implements storage.Interface. func (c *Cacher) List(ctx context.Context, key string, resourceVersion string, filter FilterFunc, listObj runtime.Object) error { <del> if !c.ListFromCache { </del> <ins> if resourceVersion == "" { // If resourceVersion is not specified, serve it from underlying // storage (for backward compatibility). </ins> return c.storage.List(ctx, key, resourceVersion, filter, listObj) } <ins> // If resourceVersion is specified, serve it from cache. // It's guaranteed that the returned value is at least that // fresh as the given resourceVersion. </ins> listRV, err := ParseListResourceVersion(resourceVersion) if err != nil { return err

The label(s) cannot be applied. These labels are supported: <detailsIn response to : This one is trying to use with a generator to create an rc. This was deprecated and now is removed, so the test is failing now. /label sig/undefined Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/label failing-test
The label(s) cannot be applied. These labels are supported: <detailsIn response to : /label failing-test Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/kind failing-test
At first I thought this was because generators were removed from . That still might be the cause of this, but I'm not 100% sure off-hand.
/sig cli
/assign
Funny thing about this is if you look in the build logs it is not immediately obvious that it is caused by generators being removed from because actually has sort of an unexpected behavior, where it tells you it is deprecated and gives you a pod instead... For example: Output:
/reopen should fix this is one of the affected jobs
Reopened this issue. <detailsIn response to : /reopen should fix this Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Since it's been a few weeks, I wanted to provide an update on this. I'm still waiting to confirm this is resolved by It looks like it is still running the old tests not marked as . Since 1.17.9 was released 14 days ago, and k8s-ci-robot only merged the PR 13 days ago, I'm guessing we need to wait until there is a 1.17.10 to see it skip the deprecated tests.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity.


Question: // Implements storage.Interface. func (c *Cacher) List(ctx context.Context, key string, resourceVersion string, filter FilterFunc, listObj runtime.Object) error { <del> if !c.ListFromCache { </del> <ins> if resourceVersion == "" { // If resourceVersion is not specified, serve it from underlying // storage (for backward compatibility). </ins> return c.storage.List(ctx, key, resourceVersion, filter, listObj) } <ins> // If resourceVersion is specified, serve it from cache. // It's guaranteed that the returned value is at least that // fresh as the given resourceVersion. </ins> listRV, err := ParseListResourceVersion(resourceVersion) if err != nil { return err

Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: apierrors "k8s.io/apimachinery/pkg/api/errors" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/labels" <ins> "k8s.io/apimachinery/pkg/types" </ins> "k8s.io/apimachinery/pkg/util/sets" <ins> "k8s.io/apimachinery/pkg/util/strategicpatch" </ins> "k8s.io/apimachinery/pkg/util/wait" "k8s.io/client-go/kubernetes/scheme" v1core "k8s.io/client-go/kubernetes/typed/core/v1"

This issue is to discuss an e2e testing plan for Ephemeral Containers. /sig node
As a first step towards this issue, I am planning to create node-e2e tests (under ) to verify the following: With a running pod client adds a valid EphemeralContainer resource, and the ephemeral container starts from the ephemeral container, list processes of itself and other containers, and then signal other processes (if is ) from the ephemeral container, ping & curl other containers try to read/write to stdin/stdout of the ephemeral container verify resource request/limit values against calculated expectations check oom scores of processes in the ephemeral container verify volume mounts restarts according to policy, when process exits client tries to add EphemeralContainer resource with ports (should be rejected) client tries to add EphemeralContainer resource with probes (should be rejected) client tries to add EphemeralContainer resource with resources (should be rejected) client tries to add EphemeralContainer resource with securityContext (should be rejected) removes the ephemeral container, and the ephemeral container is stopped and deleted adds multiple ephemeral containers, should all startWith pod of non-running status (TBD)
Awesome, thanks! This is a great start, here are some things to look out for: Visibility should work, but I expect signaling to fail without which is not currently allowed. iirc was interested in how this would be tested for ephemeral containers. Can you point us towards any examples that would help us get started? Ephemeral containers will never restart (and the policy should always be never) These will be denied by the api server without ever making it to the kubelet, so they are better tested in the apiserver integration tests (and I think they are already). Removing ephemeral containers is not currently allowed. /assign
Validation of ephemeralContainers is well tested in Testing of removing ephemeral containers is in ,
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with .


Question: apierrors "k8s.io/apimachinery/pkg/api/errors" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/labels" <ins> "k8s.io/apimachinery/pkg/types" </ins> "k8s.io/apimachinery/pkg/util/sets" <ins> "k8s.io/apimachinery/pkg/util/strategicpatch" </ins> "k8s.io/apimachinery/pkg/util/wait" "k8s.io/client-go/kubernetes/scheme" v1core "k8s.io/client-go/kubernetes/typed/core/v1"

Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen /assign /priority important-soon
Reopened this issue. <detailsIn response to : /reopen /assign /priority important-soon Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-contributor-experience at . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Rotten issues close after 30d of inactivity.


Question: apierrors "k8s.io/apimachinery/pkg/api/errors" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/labels" <ins> "k8s.io/apimachinery/pkg/types" </ins> "k8s.io/apimachinery/pkg/util/sets" <ins> "k8s.io/apimachinery/pkg/util/strategicpatch" </ins> "k8s.io/apimachinery/pkg/util/wait" "k8s.io/client-go/kubernetes/scheme" v1core "k8s.io/client-go/kubernetes/typed/core/v1"

Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-contributor-experience at . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen /remove-lifecycle rotten
Reopened this issue. <detailsIn response to : /reopen /remove-lifecycle rotten Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: } } <ins> func ipsEqual(lhs, rhs []string) bool { if len(lhs) != len(rhs) { return false } for i := range lhs { if lhs[i] != rhs[i] { return false } } return true } </ins> func (proxier *Proxier) openPortal(service string, info *serviceInfo) error { args := iptablesPortalArgs(info.portalIP, info.portalPort, info.protocol, proxier.listenAddress, info.proxyPort, service) existed, err := proxier.iptables.EnsureRule(iptables.TableNAT, iptablesProxyChain, args...)

I'm not sure if it's the intended behaviour, but when I specify publicIPs for a service, external portals are only opened if createExternalLoadBalancer is set, too (this is checked in the OnUpdate function in ). After reading the docs, I would have expected that IPTables rules for external IPs are independent of the createExternalLoadBalancer flag. Also, we're running kubernetes locally, so setting createExternalLoadBalancer doesn't work since there's no cloud provider specified.
I was about to say it should work, but then I went and re-read the code and found at least one bug. Let me dig a bit more
You're right - the IPs actually propagate to the proxy, but it ignores them if createExternal... is not set. I think it is kosher to just do it. I'm prepping a PR
PR is up - if you feel bold you could test it in your environment...
Thanks, it's working fine with your patch.


Question: err                                              error }{ { <ins> predicateName: "ShouldRunDaemonPod", </ins> ds: &extensions.DaemonSet{ Spec: extensions.DaemonSetSpec{ Selector: &metav1.LabelSelector{MatchLabels: simpleDaemonSetLabel},

Is this a BUG REPORT or FEATURE REQUEST?: /kind enhancement /sig apps Description: is a key function of DaemonSet; it's better to have more unit test case for it, e.g. node status, pod on the node. There's several test cases to the DaemonSet's finial decision, which are kind of integration test to me; prefer to provide more unit test case for this key function.
GitHub didn't allow me to assign the following users: mathspanda. Note that only can be assigned. <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
do you have any specific cases or shall I figure out something? I can take a look at this issue if you don't mind.
, , thanks very much for our effort on this enhancement; as you known, DaemonSet will use scheduler's predicates to decide which node can place pods; there's several predicates errors , the purpose of this issue is to trigger every error in unit test for code coverage.
Sure, thanks. I'll cover these predicates.
Also I've found interesting issue when I wanted to add test for predicate. It looks like is not handled or maybe I did something wrong, I couldn't find check. According following comments it should fail because AWS EBS forbids any two pods mounting the same volume ID : Test case: Any thoughts? is it correct behavior?


Question: if pInfo.Attempts != wantAttempts { t.Errorf("[%s] Pod schedule attempt unexpected, got %v, want %v", name, pInfo.Attempts, wantAttempts) } <del> if pInfo.InitialAttemptTimestamp != wantInitialAttemptTs { t.Errorf("[%s] Pod initial schedule attempt timestamp unexpected, got %v, want %v", name, pInfo.InitialAttemptTimestamp, wantInitialAttemptTs) </del> <ins> if *pInfo.InitialAttemptTimestamp != wantInitialAttemptTs { t.Errorf("[%s] Pod initial schedule attempt timestamp unexpected, got %v, want %v", name, *pInfo.InitialAttemptTimestamp, wantInitialAttemptTs) </ins> } }

is recording the time that a Pod is gated. We use the timestamp when the scheduler inserts the pod into the queue: The period of time when a Pod fails PreEnque (like being gated) shouldn't be accounted in the podschedulingdurationseconds. Create a Pod with scheduling gates. Wait some time before removing the gate. Observe the metric No response <details</details<detailsAny </details<details</details<details</details<details</details<details</details
/sig scheduling
/help /triage accepted
This request has been marked as needing help from a contributor. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help /triage accepted Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
would you want to take a stab on this issue?
Yup, I can give it a shot. /assign helayoty
, as we discussed, we can address this issue with different solutions: the pod timestamp to the current time , so the gated time is substituted. timestamp . Use a pointer to the InitialAttemptTimestamp and interpret as the newest timestamp so the pod will go to the back of the queue. I'll go ahead and implement solution no. 3, wdyt?
sgtm. Any concerns ?
I think it should be option 1 + 3, and pInfo.Timestamp should be pInfo.InitialAttemptTimestamp, right? as records the time based on :
I think we should revisit this issue, in terms of its assumption: So is it reasonable to always exclude time spent in PreEnqueue from ? Note that PreEnqueue != SchedulingGates, is one of the PreEnqueue's usecases.


Question: if pInfo.Attempts != wantAttempts { t.Errorf("[%s] Pod schedule attempt unexpected, got %v, want %v", name, pInfo.Attempts, wantAttempts) } <del> if pInfo.InitialAttemptTimestamp != wantInitialAttemptTs { t.Errorf("[%s] Pod initial schedule attempt timestamp unexpected, got %v, want %v", name, pInfo.InitialAttemptTimestamp, wantInitialAttemptTs) </del> <ins> if *pInfo.InitialAttemptTimestamp != wantInitialAttemptTs { t.Errorf("[%s] Pod initial schedule attempt timestamp unexpected, got %v, want %v", name, *pInfo.InitialAttemptTimestamp, wantInitialAttemptTs) </ins> } }

In other words, some user of PreEnqueue may want to include the time spent in PreEnqueue.
is a signal that cluster providers can use to build SLOs. The time spent before passing PreEnqueue is not under control of kube-scheduler, so considering it would not allow to build an SLO. Are you aware of other users of PreEnqueue?
True, but comparing to state that metric X can or cannot be used to build SLO (which is hard IMO), I'm in favor of giving suggestion on SLI/SLO's semantic definition, for example: the official SLI definition of has a footnote clearly explaining its definition: So if a pod is gated, it's not "immediately schedulable", and hence doesn't fall into the definition of podstartup_latency.
A viable solution might be providing an extra label like to explicitly exclude the time spent on preenqueue.
That is true, but there is no way today to exclude those pods from the metric, or separate the time. Maybe we need an extra metric, but I think the new metric could be the one that includes the time guarded in PreEnqueue. It should probably be part of the KEP. The old metric is already stable, so we can't add labels. And it might already be in use for SLOs.
cc
I think we should be working on that. And treat it as part of GA requirement for PodSchedulingReadiness.
An edge(maybe not edge) case is a pod's scheduling attempts is 0 but the is a long time, so prefer not to account for the waiting time when gated. For users not using PreEnqueue, they will not be impacted, but for users using PreEnqueue, it's a break change for it increases the duration time. And we can add a new metric like to measure how long a pod stays in a scheduler.
I thought of "attempts", it's just I'm not sure it's a good idea to rely on "attempts" to define a SLI/SLO, b/c "attempts" is obtained from the in-memory object and in a HA config env, it's only updated by the primary scheduler instance. In other words, the value is inaccurate when the lease owner switches.
attempt=0 is good enough for most scenarios
do we have a conclusion?


Question: if pInfo.Attempts != wantAttempts { t.Errorf("[%s] Pod schedule attempt unexpected, got %v, want %v", name, pInfo.Attempts, wantAttempts) } <del> if pInfo.InitialAttemptTimestamp != wantInitialAttemptTs { t.Errorf("[%s] Pod initial schedule attempt timestamp unexpected, got %v, want %v", name, pInfo.InitialAttemptTimestamp, wantInitialAttemptTs) </del> <ins> if *pInfo.InitialAttemptTimestamp != wantInitialAttemptTs { t.Errorf("[%s] Pod initial schedule attempt timestamp unexpected, got %v, want %v", name, *pInfo.InitialAttemptTimestamp, wantInitialAttemptTs) </ins> } }

Should we move on with the solution or remove this issue and the linked PR from the current release?
I would like to move forward with this proposal. ?
To move it forward, I think we should add new metric(s) to explicitly tell it's including or excluding the time spent on PreEnqueue. Given is in stable level: We will need a new metric like "podschedulingexcludingpreenqueuedurationseconds".
28 Bug Triage Shadow here. Just checking-in, if this is still on track for the 1.28 release?
We don't have a conclusion yet, but as a bug, it should qualify for changes after the freezes.


Question: return slicesToCreate, slicesToUpdate, events } <del> hintsEnabled := t.hintsPopulatedByService.Has(si.ServiceKey) t.SetHints(si.ServiceKey, si.AddressType, allocatedHintsByZone) </del> <ins> t.lock.Lock() defer t.lock.Unlock() hintsEnabled := t.hasPopulatedHintsLocked(si.ServiceKey) t.setHintsLocked(si.ServiceKey, si.AddressType, allocatedHintsByZone) </ins> // if hints were not enabled before, we publish an event to indicate we enabled them. if !hintsEnabled {

I noticed that kube-controller-manager terminated with a concurrent map access: No concurrent map access, and no fatal error. No reproducer. I noticed this in an internal CI run. This appears to be an obvious (in retrospect) logic error with a straightforward solution, so I assume that it isn't necessary to provide a reproducer. This is from an OpenShift cluster based on Kubernetes 1.27. AWS. Red Hat Enterprise Linux CoreOS 414.92.-0. No response No response No response
/sig network
/assign
/triage accepted


Question: framework.Logf("Error listing EndpointSlices: %v", err) return false, nil } <del> if len(esList.Items) != 1 { framework.Logf("Waiting for 1 EndpointSlice to exist, got %d", len(esList.Items)) </del> <ins> if len(esList.Items) == 0 { framework.Logf("Waiting for at least 1 EndpointSlice to exist, got %d", len(esList.Items)) </ins> return false, nil } <del> epSlice := esList.Items[0] if len(epSlice.Ports) != 1 { return false, fmt.Errorf("Expected EndpointSlice to have 1 Port, got %d", len(epSlice.Ports)) } port := epSlice.Ports[0] if *port.Port != int32(80) { return false, fmt.Errorf("Expected port to be 80, got %d", *port.Port) } if len(epSlice.Endpoints) != 1 { return false, fmt.Errorf("Expected EndpointSlice to have 1 endpoints, got %d", len(epSlice.Endpoints)) } endpoint := epSlice.Endpoints[0] if len(endpoint.Addresses) != 1 { return false, fmt.Errorf("Expected EndpointSlice endpoint to have 1 address, got %d", len(endpoint.Addresses)) } address := endpoint.Addresses[0] if address != "10.1.2.3" { return false, fmt.Errorf("Expected EndpointSlice to have 10.1.2.3 as address, got %s", address) </del> <ins> // Due to informer caching, it's possible for the controller // to create a second EndpointSlice if it does not see the // first EndpointSlice that was created during a sync. All // EndpointSlices created should be valid though. for _, epSlice := range esList.Items { if len(epSlice.Ports) != 1 { return false, fmt.Errorf("Expected EndpointSlice to have 1 Port, got %d", len(epSlice.Ports)) } port := epSlice.Ports[0] if *port.Port != int32(80) { return false, fmt.Errorf("Expected port to be 80, got %d", *port.Port) } if len(epSlice.Endpoints) != 1 { return false, fmt.Errorf("Expected EndpointSlice to have 1 endpoints, got %d", len(epSlice.Endpoints)) } endpoint := epSlice.Endpoints[0] if len(endpoint.Addresses) != 1 { return false, fmt.Errorf("Expected EndpointSlice endpoint to have 1 address, got %d", len(endpoint.Addresses)) } address := endpoint.Addresses[0] if address != "10.1.2.3" { return false, fmt.Errorf("Expected EndpointSlice to have 10.1.2.3 as address, got %s", address) } </ins> } return true, nil

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: Which test(s) are flaking: Testgrid link: Reason for failure: Anything else we need to know: /sig network <!-- Please see the deflaking doc () for more guidance! --
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/assign
Reviewing this as I code the CI Signal report No longer flaking on However, the test is flaking on other jobs as can be seen here Jobs affected ci-kubernetes-e2e-gce-cos-k8sbeta-default, ci-kubernetes-e2e-gce-multizone ci-kubernetes-e2e-gci-gce-containerd links to the flakes can be found in Triage link above Tentative Observation : No flaking results on jobs using kind but some flaking on gce jobs?? what would be the next best step to carry out for this? cc
Sorry I've been underwater for a bit. It looks like the test is expecting only one EndpointSlice to be created but sometimes an extra one is created due to informer cache being out of date. Although not ideal, this would still be a valid outcome. I need to update the test to consider this a valid outcome.
any updates on this?
Thanks for the reminder, filed to fix this.
/triage accepted


Question: manager.clock = clock.NewFakeClock(time.Now()) err := manager.Start() <ins> lock.Unlock() </ins> if tc.expectedError != nil { if !strings.Contains(err.Error(), tc.expectedError.Error()) { t.Errorf("unexpected error message. Got: %s want %s", err.Error(), tc.expectedError.Error())

/kind failing-test <!-- If this is a flake, please add: /kind flake --<!-- Please assign a SIG using: /sig SIG-NAME --
/sig node
I find the reason for it, but I don't know the solution is suitable, correct me if not.
/triage accepted /assign


Question: case "PUT": // Update a resource. route := ws.PUT(action.Path).To(UpdateResource(updater, ctxFn, action.Namer, codec, resource, admit)). Filter(m). <del> Doc("update the specified " + kind). Operation("update" + kind). </del> <ins> Doc("replace the specified " + kind). Operation("replace" + kind). </ins> Reads(versionedObject) addParams(route, action.Params) ws.Route(route)

sending the following json to update a service: "{"apiVersion":"v1beta1","port":6380,"containerPort":6379}" fails with 422 and message: "service "" is invalid: name: required value ''" It's not clear from the message what is exactly the problem. is the "name" field missing? isn't it called "id" at the moment? if yes, another issue () points on the fact that including the id fails the request with another error.
similar issue happens when creating a service: input (based on the guestbook example) : POST fails with :
Internal names are not converted to external during rendering of the status message. Name is the newer api version field name for what used to be called id
but the input json contains "id", so why does it fail in the first place?
I'm referring to the first comment, where update failed because you omitted the ID of the service you were updating
Update requires the full resource to be specified. GET the resource, modify the fields you want to change, and do a PUT.
Using for the field name conversion problem.
As mentioned, PUT should not accept partial updates - PATCH should, and the issue for that is . This is documented in : "Otherwise, PUT expects the whole object to be specified." Where else does it need to be documented?


Question: } func (a *AdmissionOptions) Validate() []error { <ins> if a == nil { return nil } </ins> errs := []error{} <ins> registeredPlugins := sets.NewString(a.Plugins.Registered()...) for _, name := range a.PluginNames { if !registeredPlugins.Has(name) { errs = append(errs, fmt.Errorf("admission-control plugin %q is invalid", name)) } } </ins> return errs }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: param is not validated in func Validate What you expected to happen: validate admission plugins, if not registered, return err to prevent going on until call . How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig auth
/kind bug
from the user's perspective, a reasonable error is returned


Question: want: []bool{true, false, false, true, false, true, false, true, false}, }, { <ins> name: "no filtering when QHint is enabled", nodeFn: func() *v1.Node { return st.MakeNode().Name("fake-node").Label("hostname", "fake-node").Capacity(cpu8).Obj() }, existingPods: []*v1.Pod{ st.MakePod().Name("p").HostPort(80).Obj(), }, pods: []*v1.Pod{ st.MakePod().Name("p1").Req(cpu4).Obj(), st.MakePod().Name("p2").Req(cpu16).Obj(), st.MakePod().Name("p3").Req(cpu4).Req(cpu8).Obj(), st.MakePod().Name("p4").NodeAffinityIn("hostname", []string{"fake-node"}).Obj(), st.MakePod().Name("p5").NodeAffinityNotIn("hostname", []string{"fake-node"}).Obj(), st.MakePod().Name("p6").Obj(), st.MakePod().Name("p7").Node("invalid-node").Obj(), st.MakePod().Name("p8").HostPort(8080).Obj(), st.MakePod().Name("p9").HostPort(80).Obj(), }, qHintEnabled: true, want:         []bool{true, true, true, true, true, true, true, true, true}, }, { </ins> name: "tainted node, pods with a single constraint", nodeFn: func() *v1.Node { node := st.MakeNode().Name("fake-node").Obj()

In our custom plugin, we want to move pods that is unscheduled by the plugin to activeQ/backoffQ when a new Pod is created. So, we define like this: But, even when a new Pod is created, the Pod that was unscheduled by our custom plugin isn't moved to backoffQ/activeQ. when a new Pod is created, the Pod unscheduled by our custom plugin is moved to backoffQ/activeQ. a custom plugin that has pod events in EventsToRegister. a new Pod that the custom plugin will reject. another new Pod that makes the Pod created in (1) schedulable. can see the Pod created in (1) isn't scheduled immediately (it should've been scheduled immediately after (3) because (3) should move the Pod created in (1) to activeQ/backoffQ.) If I understand correctly and if it's not expected behavior, this bug has a non-small impact, especially in v1.24 because we change to 5 min. v1.23.4 but I think latest master has the same issue. <details</details<details</details<details</details<details</details<details</details
/assign /sig scheduling
Maybe it's only on a newly created unscheduled Pod. We've moved correctly for events related to assigned Pod.
And deleting Node events has the same issue...? I guess we need to run when some Nodes are deleted. In in-tree plugins, only pod topology spread uses the deleting Node event and I guess this bug affects its performance. (Even when some Nodes are deleted, the Pods, that were unscheduled by Pod topology spread, aren't moved to activeQ/backoffQ by that deleting events.)
Hm, it may be a problem that events related to Node and Pod are not always called when an event occurs. (Even if it's intended behavior, at least we need to write this on the comment of .) /retitle scheduler: events in related to Pod and Node don't always work
To summarize: The problem is "not all events related to Pod or Node don't always make move from unschedulableQ to active/backoffQ and it may not be intended for custom-plugin developers because this behavior isn't even documented." I think we should move all unschedulable Pods with all Pod and Node events.


Question: want: []bool{true, false, false, true, false, true, false, true, false}, }, { <ins> name: "no filtering when QHint is enabled", nodeFn: func() *v1.Node { return st.MakeNode().Name("fake-node").Label("hostname", "fake-node").Capacity(cpu8).Obj() }, existingPods: []*v1.Pod{ st.MakePod().Name("p").HostPort(80).Obj(), }, pods: []*v1.Pod{ st.MakePod().Name("p1").Req(cpu4).Obj(), st.MakePod().Name("p2").Req(cpu16).Obj(), st.MakePod().Name("p3").Req(cpu4).Req(cpu8).Obj(), st.MakePod().Name("p4").NodeAffinityIn("hostname", []string{"fake-node"}).Obj(), st.MakePod().Name("p5").NodeAffinityNotIn("hostname", []string{"fake-node"}).Obj(), st.MakePod().Name("p6").Obj(), st.MakePod().Name("p7").Node("invalid-node").Obj(), st.MakePod().Name("p8").HostPort(8080).Obj(), st.MakePod().Name("p9").HostPort(80).Obj(), }, qHintEnabled: true, want:         []bool{true, true, true, true, true, true, true, true, true}, }, { </ins> name: "tainted node, pods with a single constraint", nodeFn: func() *v1.Node { node := st.MakeNode().Name("fake-node").Obj()

If concerned about performance degradation, we can extend or , for example, to have the function like ? This problem has a non-small impact, especially in v1.24+ because we change PodMaxUnschedulableQDuration to 5 min. Let's see the details... ↓ We move the unscheduled Pods (rejected by the plugin that has Pod add/update events in ) in or But, the target Pods against this moving are only unschedulable pods which have any affinity term that matches "pod". So,,,, users cannot move from unschedulableQ to activeQ/backoffQ with all assignedPod events? For example, for Pod Topology Spread (it uses Pod All event in ), the new assigned Pod creation may make some Pods, that are rejected by Pod Topology Spread, schedulable. Regardless of whether that unschedulable Pods have any affinity term that matches a newly created assigned pod or not. Of course this is not just a problem for Pod Topology Spread, it could be a problem for user's custom-plugins. I guess we should pass all unscheduled Pods to . Or at least we should document this behavior. No issue on this. We don't move the unschedulable Pods with those events at all. That's a problem... Even when a plugin uses Pod add/update/delete event in and wants to move the rejected Pods with non-scheduled Pod's add/update/delete events, scheduling queue doesn't move the unschedulable Pods with those events. (In our company, we have the custom plugin for and the plugin wants the scheduler queye to move the Pods rejected by the plugin from unschedulableQ to active/backoffQ when a new non-scheduled Pod is created.) We move only unscheduled Pods that pass . checks the Node can run that unscheduled Pod from the points of view of some in-tree filter plugins like noderesources/nodeport/nodeAffinity/nodename. The problem that arises here is similar to "Assigned Pod's add/update events". For example, for Pod Topology Spread (it uses Node Add event in ), the new Node creation may make some Pods, that are rejected by Pod Topology Spread, schedulable. Regardless of the results of against that Node. (In , we change the skew calculation of Pod Topology Spread to exclude Nodes that don't match the node affinity.


Question: want: []bool{true, false, false, true, false, true, false, true, false}, }, { <ins> name: "no filtering when QHint is enabled", nodeFn: func() *v1.Node { return st.MakeNode().Name("fake-node").Label("hostname", "fake-node").Capacity(cpu8).Obj() }, existingPods: []*v1.Pod{ st.MakePod().Name("p").HostPort(80).Obj(), }, pods: []*v1.Pod{ st.MakePod().Name("p1").Req(cpu4).Obj(), st.MakePod().Name("p2").Req(cpu16).Obj(), st.MakePod().Name("p3").Req(cpu4).Req(cpu8).Obj(), st.MakePod().Name("p4").NodeAffinityIn("hostname", []string{"fake-node"}).Obj(), st.MakePod().Name("p5").NodeAffinityNotIn("hostname", []string{"fake-node"}).Obj(), st.MakePod().Name("p6").Obj(), st.MakePod().Name("p7").Node("invalid-node").Obj(), st.MakePod().Name("p8").HostPort(8080).Obj(), st.MakePod().Name("p9").HostPort(80).Obj(), }, qHintEnabled: true, want:         []bool{true, true, true, true, true, true, true, true, true}, }, { </ins> name: "tainted node, pods with a single constraint", nodeFn: func() *v1.Node { node := st.MakeNode().Name("fake-node").Obj()

But, after is implemented, the calculation can be configured to include those Nodes.) We don't move the unschedulable Pods with Node delete events... That's a problem. Even when the plugin use Node delete event in , scheduling queue doesn't move the unschedulable Pods with any Node delete events.)
/cc Could you please take a look
These are all intended optimizations. These basic checks prevent the scheduler from retrying unschedulable pods over and and over. So far we haven't had the need to make these checks configurable as they check well established scheduling features. +1 on documenting the behavior, but I'm not sure about changing it or making it configurable (that adds complexity that might not be justified). What is it that you are trying to do?
What is it that you are trying to do? We are trying to remove flushing in queue in , right? Without improving like I suggest, some Pods in unschedulableQ cannot move back to activeQ forever in some cases if we remove flushing. (Actually, our company's custom plugin needs to wait flush because non-assigned Pod's events don't let scheduling queue move unschedulable Pods rejected by that plugin.) So, I think we should simply inform all events to queue and let queue move unschedulable Pods with that events. This can happen in some custom plugins like our company's one and also even in in-tree plugins. For example, Pod Topology Spread (after is implemented). When a user uses and , the creation of Node that doesn't match nodeAffinity/nodeSelector may change the scheduling result.
cc who implemented NodeInclusionPolicy. Can you provide a concrete example? The policy just refers to the topology spread calculations. It doesn't mean that the node doesn't have to match the nodeaffinity to be able to accept the pod. I'm asking what is your custom plugin trying to do. Depending on whether it makes sense, we need to evaluate if it's worth the complexity.
For example, for Pod Topology Spread (it uses Pod All event in EventsToRegister), the new assigned Pod creation may make some Pods, that are rejected by Pod Topology Spread, schedulable This is actually a problem here, 5 mins would be too long for background flush. But it doesn't mean that we should simply shorten the interval time.


Question: want: []bool{true, false, false, true, false, true, false, true, false}, }, { <ins> name: "no filtering when QHint is enabled", nodeFn: func() *v1.Node { return st.MakeNode().Name("fake-node").Label("hostname", "fake-node").Capacity(cpu8).Obj() }, existingPods: []*v1.Pod{ st.MakePod().Name("p").HostPort(80).Obj(), }, pods: []*v1.Pod{ st.MakePod().Name("p1").Req(cpu4).Obj(), st.MakePod().Name("p2").Req(cpu16).Obj(), st.MakePod().Name("p3").Req(cpu4).Req(cpu8).Obj(), st.MakePod().Name("p4").NodeAffinityIn("hostname", []string{"fake-node"}).Obj(), st.MakePod().Name("p5").NodeAffinityNotIn("hostname", []string{"fake-node"}).Obj(), st.MakePod().Name("p6").Obj(), st.MakePod().Name("p7").Node("invalid-node").Obj(), st.MakePod().Name("p8").HostPort(8080).Obj(), st.MakePod().Name("p9").HostPort(80).Obj(), }, qHintEnabled: true, want:         []bool{true, true, true, true, true, true, true, true, true}, }, { </ins> name: "tainted node, pods with a single constraint", nodeFn: func() *v1.Node { node := st.MakeNode().Name("fake-node").Obj()

For example, Pod Topology Spread (after is implemented). When a user uses NodeAffinityPolicy: Ignore and WhenUnsatisfiable: DoNotSchedule, the creation of Node that doesn't match nodeAffinity/nodeSelector may change the scheduling result. The skew of a new created node is always 0, so it's still unfit I think.
Our company's custom plugin is like plugin. And like coscheduling's prefilter, our plugin rejects Pods in PreFilter when the number of Pods is less than the defined number. Yes, so, maybe coscheduling plugin has the same issue as our plugin. The coscheduling plugin also uses Pod Add event and I guess it's for non-scheduled Pod's creation. (I don't know much about the coscheduling plugin though.) Oh, that's true Please forget about that example. But, how about like this scenario: user uses and Pod is rejected by Pod Topology Spread because of the skew. e.g. a user uses and TopologyKey is hostname, NodeA/NodeB has 10 Pods each, and NodeC has 8 Pods. adds un-tolerated taint to NodeC skew calculation result should be changed and the Pod rejected in (2) will be allowed by Pod Topology Spread. But the Pod isn't moved from unschedulableQ without flushing because of this issue.
Maybe we can find some pointcuts by the in , e.g. if pod was rejected by PodTopologySpread, we'll when Assigned Pod's add/update events arrived. For customize plugins, we may have to do some refactoring to register the filter func together with the event.
So now NodeC is excluded from calculations. Hence the pod fits in either NodeA or NodeB, which were unchanged. Tricky... I'll have to think about it. We really need those quick checks to pass for performance in most scenarios. /triage accepted
Thanks for bringing this up. Some comments below: You're right. Pods failed by PodTopologySpread may not get re-queued immediately. However, an assigned Pod's add/update event would only impact unschedulable Pods with cross-node scheduling directives. As of today, it's InterPodAffinity and PodTopologySpread.


Question: want: []bool{true, false, false, true, false, true, false, true, false}, }, { <ins> name: "no filtering when QHint is enabled", nodeFn: func() *v1.Node { return st.MakeNode().Name("fake-node").Label("hostname", "fake-node").Capacity(cpu8).Obj() }, existingPods: []*v1.Pod{ st.MakePod().Name("p").HostPort(80).Obj(), }, pods: []*v1.Pod{ st.MakePod().Name("p1").Req(cpu4).Obj(), st.MakePod().Name("p2").Req(cpu16).Obj(), st.MakePod().Name("p3").Req(cpu4).Req(cpu8).Obj(), st.MakePod().Name("p4").NodeAffinityIn("hostname", []string{"fake-node"}).Obj(), st.MakePod().Name("p5").NodeAffinityNotIn("hostname", []string{"fake-node"}).Obj(), st.MakePod().Name("p6").Obj(), st.MakePod().Name("p7").Node("invalid-node").Obj(), st.MakePod().Name("p8").HostPort(8080).Obj(), st.MakePod().Name("p9").HostPort(80).Obj(), }, qHintEnabled: true, want:         []bool{true, true, true, true, true, true, true, true, true}, }, { </ins> name: "tainted node, pods with a single constraint", nodeFn: func() *v1.Node { node := st.MakeNode().Name("fake-node").Obj()

So instead of "pass all unscheduled Pods to movePodsToActiveOrBackoffQueue", I'd like to come up with a solution like: It's pretty rare to move pods depending on Unschedulable Pods' events. In your particular case, maybe a proactive call would behave more efficiently. It doesn't sound correlated, is it? As the check will just pre-verify the basic checks which are orthogonal with PodTopologySpread. In other words, it's impossible that a Pod failed can become schedulable eventually. Updated: The example is a valid point... Can you share some use-cases to move Pods upon node's deletion? In the before, I don't think we have any, until the last release the introduction of feature (as in that case deleting a node would impact the number of domains).
to come up with a solution like: If we do so, also, we need to provide the way to pass some functions to decide which Pods should pass to like for out-of-tree plugins? call would behave more efficiently. Ah~, that's a good idea :) Maybe like this (similar to ) user uses WhenUnsatisfiable: DoNotSchedule. Pod is rejected by Pod Topology Spread because of the skew. e.g. a user uses maxSkew: 1 and TopologyKey is hostname, NodeA/NodeB has 10 Pods each, and NodeC has 8 Pods. deletes NodeC skew calculation result should be changed and the Pod rejected in (2) will be allowed by Pod Topology Spread. But the Pod isn't moved from unschedulableQ without flushing because of this issue. So,,, ignoring some events for performance in scheduler side (without providing a way to change this) could be inconvenient for some plugins. What I thought in the above previous comment () is to define "which kind of resource's events should be ignored" in plugin side. (similar to 's idea? ) We can add pre filter functions in like: Then we can set any pre-filter functions for each plugin. And, we can define some default behavior for PreFilterFn and use them when is empty. So, for example, if most of the plugins are considered to prefer to ignore the event for Node that doesn't match nodeAffinity/nodeSelector, we can define this behavior as the default behavior.


Question: want: []bool{true, false, false, true, false, true, false, true, false}, }, { <ins> name: "no filtering when QHint is enabled", nodeFn: func() *v1.Node { return st.MakeNode().Name("fake-node").Label("hostname", "fake-node").Capacity(cpu8).Obj() }, existingPods: []*v1.Pod{ st.MakePod().Name("p").HostPort(80).Obj(), }, pods: []*v1.Pod{ st.MakePod().Name("p1").Req(cpu4).Obj(), st.MakePod().Name("p2").Req(cpu16).Obj(), st.MakePod().Name("p3").Req(cpu4).Req(cpu8).Obj(), st.MakePod().Name("p4").NodeAffinityIn("hostname", []string{"fake-node"}).Obj(), st.MakePod().Name("p5").NodeAffinityNotIn("hostname", []string{"fake-node"}).Obj(), st.MakePod().Name("p6").Obj(), st.MakePod().Name("p7").Node("invalid-node").Obj(), st.MakePod().Name("p8").HostPort(8080).Obj(), st.MakePod().Name("p9").HostPort(80).Obj(), }, qHintEnabled: true, want:         []bool{true, true, true, true, true, true, true, true, true}, }, { </ins> name: "tainted node, pods with a single constraint", nodeFn: func() *v1.Node { node := st.MakeNode().Name("fake-node").Obj()

What do you think about this idea?
Thanks. It makes sense to me. To summarize, the following cases doesn't properly re-queue relevant pods: Pod's add/update events -scheduled Pod's add/update/delete events 's add/update/delete events For 1 & 2, I've commented above with solutions. (1 needs core changes similar like podAffinity, 2 may leverage the Activate hook) For 3, with 's examples, I'm convinced this is some area we need to improve. One simple and stratightforward solution is to plumb a hook to let the plugin claim "alright, upon Node events, move the pod failed by me anyways". Then once this knob is registered, we move pods failed by that plugin back unconditionally; instead of proceeding with . This hook should be available to in-tree and out-of-tree plugins. In terms of in-tree plugins, it's just PodTopologySpread (we discussed this in today's sig meeting, even InterPodAffinity doesn't fall into this case) So, as a first step, let's ensure we're on the same page and agree on the ^^ summary. Next, we can better evaluate trade-offs and come up with an optimal solution.
Thanks for discussing this topic in the sig meeting. But, I am still believing that all logic related to a plugin should be controllable from the plugin side and we should not create something that can be achieved in in-tree plugins but not in out-of-tree plugins. So... the benefit of the "customizable PreCheck approach" () isn't only to solve this issue. We can use this feature for other optimization: like improving the whole performance of scheduler by moving only Pods that are more likely to be able to be scheduled by defining more detailed events on with customizable PreCheck. Many plugins have a particular scenario that makes rejected Pods schedulable and the current doesn't have an abundant enough ability of expression for this. (So, today many plugins need to say "register this hoge event on foo resource. This event may make rejected Pods schedulable" on comments in . ) Yes, I watched the sig-meeting recording as well and the opinion that the "customizable PreCheck approach" increases some implementation complexity surely makes sense. But,,, isn't this complexity increasing acceptable even if the above is taken into account?


Question: want: []bool{true, false, false, true, false, true, false, true, false}, }, { <ins> name: "no filtering when QHint is enabled", nodeFn: func() *v1.Node { return st.MakeNode().Name("fake-node").Label("hostname", "fake-node").Capacity(cpu8).Obj() }, existingPods: []*v1.Pod{ st.MakePod().Name("p").HostPort(80).Obj(), }, pods: []*v1.Pod{ st.MakePod().Name("p1").Req(cpu4).Obj(), st.MakePod().Name("p2").Req(cpu16).Obj(), st.MakePod().Name("p3").Req(cpu4).Req(cpu8).Obj(), st.MakePod().Name("p4").NodeAffinityIn("hostname", []string{"fake-node"}).Obj(), st.MakePod().Name("p5").NodeAffinityNotIn("hostname", []string{"fake-node"}).Obj(), st.MakePod().Name("p6").Obj(), st.MakePod().Name("p7").Node("invalid-node").Obj(), st.MakePod().Name("p8").HostPort(8080).Obj(), st.MakePod().Name("p9").HostPort(80).Obj(), }, qHintEnabled: true, want:         []bool{true, true, true, true, true, true, true, true, true}, }, { </ins> name: "tainted node, pods with a single constraint", nodeFn: func() *v1.Node { node := st.MakeNode().Name("fake-node").Obj()

EDIT: It is a bad habit of mine to always write long sentences even though I am not very good at English. Summarizing the above: I think it's not a good idea to create something that can be achieved in in-tree plugins but not in out-of-tree plugins. If we need a solid advantage to extend like this, then it has the advantage of optimizing the queue per plugin like described above.
I didn't jump in the thread since it's actively on the discussion. But, the issue is the similar one. (This time, we can solve that issue by changing scheduling queue logic since It's the k/k feature though.)
I'm not sure I fully understand your intention. The slack thread was discussing the AssignedPod update case, so it fallls into case 1 in , which we already agreed to improve in upstream.
Hi Are you working on this? I think this is quite important and if you're not working on this, I can follow.
Sorry, leave here for a while. I wanted to wait for (and then forgot to come back here. ) That proposal is quite similar to what I want to see in the conclusion of this issue; every plugin can judge if each Pod in unschedQ can be moved to schedQ/backoffQ. Do you think it makes sense to add "By which event the pod will be moved back to schedQ/backoffQ" on 's argument? I know and isn't needed in DefaultEnqueue plugin for API, but that's the last missing puzzle piece : If has , enqueue plugin can judge if the event makes it possible to schedule the Pod. For example, I believe we can move ↓ this logic to Pod Affinity plugin by implementing in Pod Affinity plugin. Pod Affinity plugin will remember Pods rejected by itself in . And in , if the given Pod was the one rejected by Pod Affinity, is AssignedPod/Created|Updated and is Pod affinity with the rejected Pod, then move the Pod back to activeQ/backoffQ. Otherwise reject the Pod moving. We can resolve (3) by implementing as well. (by exactly the way you said.↓) My original concern in this issue is that having logic for specific plugins in the scheduler core may prevent out-of-tree plugins' extendability. The scheduler itself should be pure, and shouldn't do anything special for any plugins implementation ideally.


Question: want: []bool{true, false, false, true, false, true, false, true, false}, }, { <ins> name: "no filtering when QHint is enabled", nodeFn: func() *v1.Node { return st.MakeNode().Name("fake-node").Label("hostname", "fake-node").Capacity(cpu8).Obj() }, existingPods: []*v1.Pod{ st.MakePod().Name("p").HostPort(80).Obj(), }, pods: []*v1.Pod{ st.MakePod().Name("p1").Req(cpu4).Obj(), st.MakePod().Name("p2").Req(cpu16).Obj(), st.MakePod().Name("p3").Req(cpu4).Req(cpu8).Obj(), st.MakePod().Name("p4").NodeAffinityIn("hostname", []string{"fake-node"}).Obj(), st.MakePod().Name("p5").NodeAffinityNotIn("hostname", []string{"fake-node"}).Obj(), st.MakePod().Name("p6").Obj(), st.MakePod().Name("p7").Node("invalid-node").Obj(), st.MakePod().Name("p8").HostPort(8080).Obj(), st.MakePod().Name("p9").HostPort(80).Obj(), }, qHintEnabled: true, want:         []bool{true, true, true, true, true, true, true, true, true}, }, { </ins> name: "tainted node, pods with a single constraint", nodeFn: func() *v1.Node { node := st.MakeNode().Name("fake-node").Obj()

If we can move all such logic to each plugin, that'd be perfect.
^ is more like a new feature than just a solution to this problem. I created another issue and describe my proposal in detail. Let's have a discussion there.
It can be resolved by moving all preCheck into each plugin's QueueingHint. Let me retitle this issue so that the current status is easy to understand. /retitle scheduler: move all preCheck to QueueingHint
This issue has not been updated in over 1 year, and should be re-triaged. You can: Confirm that this issue is still relevant with (org members only) Close this issue with For more details on the triage process, see /remove-triage accepted
/triage accepted


Question: return &loadbalancerList[0], nil } <del> func waitLoadbalancerActiveProvisioningStatus(client *gophercloud.ServiceClient, loadbalancerID string) error { </del> <ins> func waitLoadbalancerActiveProvisioningStatus(client *gophercloud.ServiceClient, loadbalancerID string) (string, error) { </ins> start := time.Now().Second() for { loadbalancer, err := loadbalancers.Get(client, loadbalancerID).Extract() if err != nil { <del> return err </del> <ins> return "", err </ins> } if loadbalancer.ProvisioningStatus == "ACTIVE" { <del> return nil </del> <ins> return "ACTIVE", nil } else if loadbalancer.ProvisioningStatus == "ERROR" { return "ERROR", fmt.Errorf("Loadbalancer has gone into ERROR state") </ins> } time.Sleep(1 * time.Second) if time.Now().Second()-start >= loadbalancerActiveTimeoutSeconds { <del> return fmt.Errorf("Loadbalancer failed to go into ACTIVE provisioning status within alloted time") </del> <ins> return loadbalancer.ProvisioningStatus, fmt.Errorf("Loadbalancer failed to go into ACTIVE provisioning status within alloted time") </ins> } } }

LBaaS implementation (v1 or v2) will not support services that require to load balancer more than one port.
Fixed for LBaaS v2 by I suspect LBaaS v1 can't support multiple ports (not sure if reusing the VIP across multiple loadbalancers is a universally supported thing). I also believe LBaaS v1 is also mostly gone in favour of v2 in deployments so trying to address this in v1 is extremely low priority unless someone steps forward to ask for it. So: I think we can close this bug now, but I lack the powers to do that directly.
thank you for the update.


Question: if err != nil { return nil, fmt.Errorf("failed to get container info from cadvisor: %v", err) } <del> // removeTerminatedContainerInfo will also remove pod level cgroups, so save the infos into allInfos first allInfos := infos infos = removeTerminatedContainerInfo(infos) </del> <ins> filteredInfos, allInfos := filterTerminatedContainerInfoAndAssembleByPodCgroupKey(infos) </ins> // Map each container to a pod and update the PodStats with container data. podToStats := map[statsapi.PodReference]*statsapi.PodStats{} <del> for key, cinfo := range infos { </del> <ins> for key, cinfo := range filteredInfos { </ins> // On systemd using devicemapper each mount into the container has an // associated cgroup. We ignore them to ensure we do not get duplicate // entries in our summary. For details on .mount units:

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --For single node with hundreds and thousands of pods and containers, the cpu usage of kubelet becomes high, one bottleneck is the cadvisor stats provider as below: the method iterate all containerInfos for every podStats cause high cpu the method should not iterate allcontainerInfos for every podStats, use map instead Kubernetes version (use ): v1.19.4 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig node
/triage accepted
/assign
I’ve done an analysis on the code and this is what I have found, I believe we have two issues to address. There are multiple range loops being performed over v2.ContainerInfo / PodStat maps within the ListPodStatus method which can be contributing to the high cpu utilisation. The allInfos argument which is supplied to the getCadvisorContainerInfo function is a map which uses a cgroup path as the key, in order to find matching container info to podUID, processing of the keys in the map needs to be performed until a match is found. Examples of keys are as follows: /kubepods/burstable/podf9e3a173-1470-451a-9ca2-/d8426b82582fbcfae16085bc794c49769d3a9648bd5463976aca4071f9342754 /docker/2dfa17fc81e781212bcc26e189ebb724721fe097394b4de3cbeb7e069bf23290 My recommendation would be to apply the following updates: Refactor the getCadvisorContainerInfo function to allow for a filter and key type parameter to be supplied by the caller, this enables reducing the number of range loops being performed over the container info map by removing elements that are not required and the ability to set the key being used in the returned container infos map to enable fast lookup of container info.


Question: if err != nil { return nil, fmt.Errorf("failed to get container info from cadvisor: %v", err) } <del> // removeTerminatedContainerInfo will also remove pod level cgroups, so save the infos into allInfos first allInfos := infos infos = removeTerminatedContainerInfo(infos) </del> <ins> filteredInfos, allInfos := filterTerminatedContainerInfoAndAssembleByPodCgroupKey(infos) </ins> // Map each container to a pod and update the PodStats with container data. podToStats := map[statsapi.PodReference]*statsapi.PodStats{} <del> for key, cinfo := range infos { </del> <ins> for key, cinfo := range filteredInfos { </ins> // On systemd using devicemapper each mount into the container has an // associated cgroup. We ignore them to ensure we do not get duplicate // entries in our summary. For details on .mount units:

My initial thoughts are to use an enumeration which has two values KEYCGROUPPATH to provide backwards compatibility and KEYPODUID to enable fast lookups via podUID Refactor ListPodStats method to be more efficient in constructing the PodStats slice and taking into account the fast lookup via podUID I think this is the best approach instead of pushing the data structure implementation for the container info upstream to the cadviser clients, this will at least initially provide a better optimisation scoped within the kubelet/stats package and reduce other possible impacts to changing the cadvisor clients Please advise.
Just realized this looks like a duplicate issue and looking as fixed by PR Both this issue and were raised by Zac
/unassign
/close
You can't close an active issue/PR unless you authored it or you are a collaborator. <detailsIn response to : /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Didn't think I could close this, leaving this with
this will be automatically closed once the PR merges


Question: if tc.expectCreate { expectedCreates = 1 } <ins> if tc.ranPreviously && !tc.stillActive { completionTime := tc.jobCreationTime.Add(10 * time.Second) if cjCopy.Status.LastSuccessfulTime == nil || !cjCopy.Status.LastSuccessfulTime.Time.Equal(completionTime) { t.Errorf("cj.status.lastSuccessfulTime: %s expected, got %#v", completionTime, cj.Status.LastSuccessfulTime) } } </ins> if len(jc.Jobs) != expectedCreates { t.Errorf("%s: expected %d job started, actually %v", name, expectedCreates, len(jc.Jobs)) }

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Cronjob should include the last time when a job was successfully completed. Why is this needed: Currently, a Cronjob's only includes the last time when a job was scheduled. This makes monitoring cronjobs much more complicated, because instead of simply comparing to the current timestamp, the monitoring system now has to hunt for the last job related to the cronjob and persist the state somewhere.
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I'd be interesting in picking this up if the apps team thinks this is a good ticket - it seems like it's fairly beginner friendly.
Hi how is this going on? I could take it over if you don't have the time for this issue. :)
Yes please do! I meant to update that I actually wasn't able to pick this up, but it slipped my mind
/assign
Hi do you have any thoughts on this feature?
/unassign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Any updates?
Reiterating the mentions to trigger a notification: <detailsIn response to : Any updates? Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Trying to understand the usage of lastSuccessfulTime. In a normally working case, lastSuccessfulTime lastScheduledTime =last job completed lastSuccessfulTime < lastScheduledTime =last job completed, but a newer one is running Is that the expectation ?
We wanted to use it in monitoring (e.g. fire an alert if is more than X ago). could also mean that the job has been failing for some time, so I don't know what would you get out of that. I guess you could also use as a proxy for "job has failed at least once"?


Question: if tc.expectCreate { expectedCreates = 1 } <ins> if tc.ranPreviously && !tc.stillActive { completionTime := tc.jobCreationTime.Add(10 * time.Second) if cjCopy.Status.LastSuccessfulTime == nil || !cjCopy.Status.LastSuccessfulTime.Time.Equal(completionTime) { t.Errorf("cj.status.lastSuccessfulTime: %s expected, got %#v", completionTime, cj.Status.LastSuccessfulTime) } } </ins> if len(jc.Jobs) != expectedCreates { t.Errorf("%s: expected %d job started, actually %v", name, expectedCreates, len(jc.Jobs)) }

Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten
that's exactly what is being asked here. /close
Closing this issue. <detailsIn response to : /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
"successfully scheduled" != "successfully completed", and most of the users care more about the latter.
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign /triage accepted
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at .


Question: if tc.expectCreate { expectedCreates = 1 } <ins> if tc.ranPreviously && !tc.stillActive { completionTime := tc.jobCreationTime.Add(10 * time.Second) if cjCopy.Status.LastSuccessfulTime == nil || !cjCopy.Status.LastSuccessfulTime.Time.Equal(completionTime) { t.Errorf("cj.status.lastSuccessfulTime: %s expected, got %#v", completionTime, cj.Status.LastSuccessfulTime) } } </ins> if len(jc.Jobs) != expectedCreates { t.Errorf("%s: expected %d job started, actually %v", name, expectedCreates, len(jc.Jobs)) }

/lifecycle rotten
This is described in and being actively implemented in /close
Closing this issue. <detailsIn response to : This is described in and being actively implemented in /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
It doesn't look like it was actually implemented, though. Searching for only shows API changes, there don't seem to be any changes to the CronJob controller logic. Or am I missing something?
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign /remove-lifecycle rotten


Question: return &framework.QueuedPodInfo{ PodInfo:                 podInfo, Timestamp:               now, <del> InitialAttemptTimestamp: now, </del> <ins> InitialAttemptTimestamp: nil, </ins> UnschedulablePlugins:    sets.New(plugins...), } }

is recording the time that a Pod is gated. We use the timestamp when the scheduler inserts the pod into the queue: The period of time when a Pod fails PreEnque (like being gated) shouldn't be accounted in the podschedulingdurationseconds. Create a Pod with scheduling gates. Wait some time before removing the gate. Observe the metric No response <details</details<detailsAny </details<details</details<details</details<details</details<details</details
/sig scheduling
/help /triage accepted
This request has been marked as needing help from a contributor. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help /triage accepted Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
would you want to take a stab on this issue?
Yup, I can give it a shot. /assign helayoty
, as we discussed, we can address this issue with different solutions: the pod timestamp to the current time , so the gated time is substituted. timestamp . Use a pointer to the InitialAttemptTimestamp and interpret as the newest timestamp so the pod will go to the back of the queue. I'll go ahead and implement solution no. 3, wdyt?
sgtm. Any concerns ?
I think it should be option 1 + 3, and pInfo.Timestamp should be pInfo.InitialAttemptTimestamp, right? as records the time based on :
I think we should revisit this issue, in terms of its assumption: So is it reasonable to always exclude time spent in PreEnqueue from ? Note that PreEnqueue != SchedulingGates, is one of the PreEnqueue's usecases.


Question: return &framework.QueuedPodInfo{ PodInfo:                 podInfo, Timestamp:               now, <del> InitialAttemptTimestamp: now, </del> <ins> InitialAttemptTimestamp: nil, </ins> UnschedulablePlugins:    sets.New(plugins...), } }

In other words, some user of PreEnqueue may want to include the time spent in PreEnqueue.
is a signal that cluster providers can use to build SLOs. The time spent before passing PreEnqueue is not under control of kube-scheduler, so considering it would not allow to build an SLO. Are you aware of other users of PreEnqueue?
True, but comparing to state that metric X can or cannot be used to build SLO (which is hard IMO), I'm in favor of giving suggestion on SLI/SLO's semantic definition, for example: the official SLI definition of has a footnote clearly explaining its definition: So if a pod is gated, it's not "immediately schedulable", and hence doesn't fall into the definition of podstartup_latency.
A viable solution might be providing an extra label like to explicitly exclude the time spent on preenqueue.
That is true, but there is no way today to exclude those pods from the metric, or separate the time. Maybe we need an extra metric, but I think the new metric could be the one that includes the time guarded in PreEnqueue. It should probably be part of the KEP. The old metric is already stable, so we can't add labels. And it might already be in use for SLOs.
cc
I think we should be working on that. And treat it as part of GA requirement for PodSchedulingReadiness.
An edge(maybe not edge) case is a pod's scheduling attempts is 0 but the is a long time, so prefer not to account for the waiting time when gated. For users not using PreEnqueue, they will not be impacted, but for users using PreEnqueue, it's a break change for it increases the duration time. And we can add a new metric like to measure how long a pod stays in a scheduler.
I thought of "attempts", it's just I'm not sure it's a good idea to rely on "attempts" to define a SLI/SLO, b/c "attempts" is obtained from the in-memory object and in a HA config env, it's only updated by the primary scheduler instance. In other words, the value is inaccurate when the lease owner switches.
attempt=0 is good enough for most scenarios
do we have a conclusion?


Question: return &framework.QueuedPodInfo{ PodInfo:                 podInfo, Timestamp:               now, <del> InitialAttemptTimestamp: now, </del> <ins> InitialAttemptTimestamp: nil, </ins> UnschedulablePlugins:    sets.New(plugins...), } }

Should we move on with the solution or remove this issue and the linked PR from the current release?
I would like to move forward with this proposal. ?
To move it forward, I think we should add new metric(s) to explicitly tell it's including or excluding the time spent on PreEnqueue. Given is in stable level: We will need a new metric like "podschedulingexcludingpreenqueuedurationseconds".
28 Bug Triage Shadow here. Just checking-in, if this is still on track for the 1.28 release?
We don't have a conclusion yet, but as a bug, it should qualify for changes after the freezes.


Question: } } <ins> func updateDS(t *testing.T, dsClient appstyped.DaemonSetInterface, dsName string, updateFunc func(*apps.DaemonSet)) *apps.DaemonSet { var ds *apps.DaemonSet if err := retry.RetryOnConflict(retry.DefaultBackoff, func() error { newDS, err := dsClient.Get(dsName, metav1.GetOptions{}) if err != nil { return err } updateFunc(newDS) ds, err = dsClient.Update(newDS) return err }); err != nil { t.Fatalf("Failed to update DaemonSet: %v", err) } return ds } </ins> func forEachFeatureGate(t *testing.T, tf func(t *testing.T)) { for _, fg := range featureGates() { func() {

https://k8s- /sig apps /priority critical-urgent /assign PR that the test: /cc
GitHub didn't allow me to assign the following users: mortent. Note that only and repo collaborators can be assigned. For more information please see <detailsIn response to : https://k8s- /priority critical-urgent /assign /cc Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/kind flake
I'm looking into this.
The test is still flaky, but with different symptom: An example flake: https://k8s- It's likely because the collision count was updated in a previous sync loop, but the in the cache store hasn't been updated and we used a stale collision count while calculating the new hash: /assign