Question: dockernat "github.com/docker/go-connections/nat" "github.com/golang/glog" <ins> utilerrors "k8s.io/apimachinery/pkg/util/errors" </ins> "k8s.io/kubernetes/pkg/credentialprovider" runtimeapi "k8s.io/kubernetes/pkg/kubelet/apis/cri/v1alpha1/runtime" <ins> "k8s.io/kubernetes/pkg/kubelet/dockershim/libdocker" </ins> "k8s.io/kubernetes/pkg/kubelet/types" "k8s.io/kubernetes/pkg/security/apparmor" <del> "k8s.io/kubernetes/pkg/kubelet/dockershim/libdocker" </del> <ins> "k8s.io/kubernetes/pkg/util/parsers" </ins> ) const (

Could you also verify whether fixed your problem?
tested against your commits cherry-picked on top of 1.8 beta, the fix seems to work
thanks.
[MILESTONENOTIFIER] Milestone Labels Complete Issue label settings: sig/node: Issue will be escalated to these SIGs if needed. priority/important-soon: Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. kind/bug: Fixes a bug discovered during the current release. <detailsAdditional instructions available <a href=""here</aThe commands available for adding these labels are documented <a href=""here</a</details


Question: MemoryManager:                                  {Default: false, PreRelease: featuregate.Alpha}, CPUCFSQuotaPeriod:                              {Default: false, PreRelease: featuregate.Alpha}, TopologyManager:                                {Default: true, PreRelease: featuregate.Beta}, <del> ServiceNodeExclusion:                           {Default: true, PreRelease: featuregate.GA, LockToDefault: true}, // remove in 1.23 NodeDisruptionExclusion:                        {Default: true, PreRelease: featuregate.GA, LockToDefault: true}, // remove in 1.22 </del> StorageObjectInUseProtection:                   {Default: true, PreRelease: featuregate.GA}, SupportPodPidsLimit:                            {Default: true, PreRelease: featuregate.GA, LockToDefault: true}, // remove in 1.23 SupportNodePidsLimit:                           {Default: true, PreRelease: featuregate.GA, LockToDefault: true}, // remove in 1.23

What is missing? Based on SIG call it's probably "tests" but I am not sure. Could use some investigation and followup - how do we get it to GA? /sig network /kind cleanup
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/assign
This one seems not a good first one, I'll take this to myself and start bothering people in Slack to check what's missing, mostly as I've seen this is also related to cloud providers :)
I think this one originally. According to the KEP in This should have went GA in v.19 and be removed in v1.20. Given it's still marked beta, at the very least we should mark it GA in v1.20 and remove it in v1.21. anything I'm missing here?
yeap. Honest question, changing a Feature Gate to GA process is only changing the related line in and also putting a comment that this goes to GA? If so, probably this one can also be a good first issue and we can open a future issue for the removal in v1.21 once v1.20 ships. I think this will also need to be changed in the in v1.21 right? If is ok with that, I would like then to mark this as a good first issue to turn the feature GA in v1.20 :)
Well, taking a look into how secure is to move to GA: In the KEP there are some considerations about removing the IsMasterNode function, which seems to be already dealt in commit E2E tests needs to be also checked about usage of IsMasterNode(), which I could not find anymore and also seems to be removed from the same commit above. About the same Kep, it's directly related to other Feature Gate: ` So what I think that needs to be done: Promote LegacyNodeRoleBehavior to GA and Default it to false as proposed in the Kep -I'll open another issue just for this According to the LegacyNodeRoleBehavior should be removed in v1.19 but according to the Kep in v1.21, so let's stick with v1.21? ServiceNodeExclusion - Should be moved to GA, already defaulted to true. This issue is going to deal ONLY with this.


Question: MemoryManager:                                  {Default: false, PreRelease: featuregate.Alpha}, CPUCFSQuotaPeriod:                              {Default: false, PreRelease: featuregate.Alpha}, TopologyManager:                                {Default: true, PreRelease: featuregate.Beta}, <del> ServiceNodeExclusion:                           {Default: true, PreRelease: featuregate.GA, LockToDefault: true}, // remove in 1.23 NodeDisruptionExclusion:                        {Default: true, PreRelease: featuregate.GA, LockToDefault: true}, // remove in 1.22 </del> StorageObjectInUseProtection:                   {Default: true, PreRelease: featuregate.GA}, SupportPodPidsLimit:                            {Default: true, PreRelease: featuregate.GA, LockToDefault: true}, // remove in 1.23 SupportNodePidsLimit:                           {Default: true, PreRelease: featuregate.GA, LockToDefault: true}, // remove in 1.23

NodeDisruptionExclusion - Should be moved to GA, already defaulted to true. I'll open another issue for this I'm afraid of this having some sort of direct impact in cloud providers, do you want me to take this to the sig-cloudproviders meet? Otherwise can we move the above Feature Gates to GA, and schedule their removal for v1.21, wdyt?
Per deprecation lifecycle it seems that a feature needs 6 months or 2 releases (the longer one) in Beta before going to GA. So this actually seems to be the missing part before moving to GA. As the 3 FeatureGates are directed related, probably this issue should be: Move all of them to GA in v1.21 Default LegacyNodeRoleBehavior to false Sounds a good approach? I can keep this opened until we reach v1.21
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
/remove-triage unresolved
/retitle Move Feature gate ServiceNodeExclusion to GA and LockToDefault
/triage accepted as this FG is scheduled to be moved to GA in v1.21, do you want to tackle on this?
fixed in /close
Closing this issue. <detailsIn response to : fixed in /close </details


Question: node.Status.NodeInfo.BootID = info.BootID } <del> rootfs, err := kl.GetCachedRootFsInfo() if err != nil { node.Status.Capacity[v1.ResourceStorage] = resource.MustParse("0Gi") } else { for rName, rCap := range cadvisor.StorageScratchCapacityFromFsInfo(rootfs) { node.Status.Capacity[rName] = rCap } } if hasDedicatedImageFs, _ := kl.HasDedicatedImageFs(); hasDedicatedImageFs { imagesfs, err := kl.ImagesFsInfo() </del> <ins> if utilfeature.DefaultFeatureGate.Enabled(features.LocalStorageCapacityIsolation) { rootfs, err := kl.GetCachedRootFsInfo() </ins> if err != nil { <del> node.Status.Capacity[v1.ResourceStorageOverlay] = resource.MustParse("0Gi") </del> <ins> node.Status.Capacity[v1.ResourceStorageScratch] = resource.MustParse("0Gi") </ins> } else { <del> for rName, rCap := range cadvisor.StorageOverlayCapacityFromFsInfo(imagesfs) { </del> <ins> for rName, rCap := range cadvisor.StorageScratchCapacityFromFsInfo(rootfs) { </ins> node.Status.Capacity[rName] = rCap } } <ins> if hasDedicatedImageFs, _ := kl.HasDedicatedImageFs(); hasDedicatedImageFs { imagesfs, err := kl.ImagesFsInfo() if err != nil { node.Status.Capacity[v1.ResourceStorageOverlay] = resource.MustParse("0Gi") } else { for rName, rCap := range cadvisor.StorageOverlayCapacityFromFsInfo(imagesfs) { node.Status.Capacity[rName] = rCap } } } </ins> } // Set Allocatable.

Local storage is not expected to be available as a scheduling resource in v1.7. Kubelet is incorrectly exposing a resource named . This is incorrect because the feature is alpha and so it requires opt-in via feature flags and, the resource names are invalid. The expected resource names are listed - and . cc This is v1.7 blocker because we cannot expose a new resource in the API and rename it in a subsequent release.
CC I'll find someone on the storage team to address this.
/assign
GitHub didn't allow me to assign the following users: verult. Note that only can be assigned. <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: docker_pull( name = "debian-hyperkube-base-amd64", <del> digest = "sha256:fc1b461367730660ac5a40c1eb2d1b23221829acf8a892981c12361383b3742b", </del> <ins> digest = "sha256:d83594ecd85345144584523e7fa5388467edf5d2dfa30d0a1bcbf184cddf4a7b", </ins> registry = "k8s.gcr.io", repository = "debian-hyperkube-base-amd64", <del> tag = "0.8",  # ignored, but kept here for documentation </del> <ins> tag = "0.9",  # ignored, but kept here for documentation </ins> ) docker_pull(

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Upgraded to Kuberenetes 1.9 and now IPVS mode in kube-proxy stopped working due to What you expected to happen: Normal startup as it's been working flawless in 1.8.5 How to reproduce it (as minimally and precisely as possible): Use kube-proxy with IPVS service mode on 1.9.0 hyperkube image from GCR Anything else we need to know?: If i exec into the running kube-proxy pod and do then restart the container kube-proxy starts in ipvs mode fine Environment: Kubernetes version (use ):Cloud provider or hardware configuration: On-prem OS (e.g. from /etc/os-release): CoreOS Container Linux by CoreOS stable (1576.4.0) Kernel (e.g. ): 4.13.16-coreos-r2 Install tools: Own Others:
/sig network
According to jell in #sig-network these images are to be considered experimental and not used in production and might be missing binarys? , could someone shed some light on this
To be more precise: hyperkube binary can hide some options of underlying applications (e.g. of fix for that: ) and whole docker image of hyperkube - as this issue shows, can miss some useful binaries. That's why I named it as a tool to do experiments, while for real deployments probably a better choice would be to use underlying applications directly using own docker image, carefully tuned.
Is this connected to and by chance the changes hasn't propagated down to the public images yet?
image is based on (which is based on ) (and ) has a separate image based on But as has included support, IMO also should be extended by command, after this line.
Seems this issue already got fixed? Can close now?
Where do you get the info it's fixed, all my kube-proxy instances is still failbacking to iptables due to missing ipset that was present in 1.8.5
It was fixed by merging So, as it's in master, the fix will land in next release.
Ok, so it's fixed in the next version. I'll just switch away from Hyperkube then. Thanks
FYI: I've rebuilt hyperkube image for CoreOS with installed. Just for testing purposes. Everything works for me with it.


Question: docker_pull( name = "debian-hyperkube-base-amd64", <del> digest = "sha256:fc1b461367730660ac5a40c1eb2d1b23221829acf8a892981c12361383b3742b", </del> <ins> digest = "sha256:d83594ecd85345144584523e7fa5388467edf5d2dfa30d0a1bcbf184cddf4a7b", </ins> registry = "k8s.gcr.io", repository = "debian-hyperkube-base-amd64", <del> tag = "0.8",  # ignored, but kept here for documentation </del> <ins> tag = "0.9",  # ignored, but kept here for documentation </ins> ) docker_pull(

Waiting for the next release. :relaxed:
So does have ipset. So please use that. However does not have ipset (see below). The commit above was merged well before we shipped v1.9.0, But hyperkube image does not use the debian-iptables, it uses debian-hyperkube-base image if you see Anyone want to log a PR for hyperkube image to have ipset?
Can we re-open this, only fixes this on debian, not amd64!
/reopen
you can't re-open an issue/PR unless you authored it or you are assigned to it. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
will need to reopen, then :)


Question: // on port 443 // If the Service is nil, that means the handling for the API groupversion is handled locally on this server. // The call will simply delegate to the normal handler chain to be fulfilled. <ins> // +optional </ins> optional ServiceReference service = 1; // Group is the API group name this server hosts

/sig api-machinery /kind bug /cc this makes any updates on a "Local" apiservice failing by kubectl (unless you disable the client-side validation). a quick repro: (edit whatever a local apiservice) a label "foo: bar" will fail on the schema validation # "" was not valid: # * : Invalid value: "The edited file failed validation": ValidationError(): missing required field "service" in client-side v1.15.0 server-side v1.15.2 referencing the published openapi v2 for apiservice#spec below:
/label help-wanted
The label(s) cannot be applied. These labels are supported: <detailsIn response to : /label help-wanted Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/help
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I can on this if no one else is currently working on it.
sure, feel free to tag me to request reviewing..
awesome, I will /assign
are you still working on this one?


Question: } } <ins> func SkipIfCSIMigrationEnabled() { if utilfeature.DefaultFeatureGate.Enabled(csiMigration) { skipInternalf(1, "Only supported when %v feature is disabled", csiMigration) } } </ins> // SkipIfMissingResource skips if the gvr resource is missing. func SkipIfMissingResource(dynamicClient dynamic.Interface, gvr schema.GroupVersionResource, namespace string) { resourceClient := dynamicClient.Resource(gvr).Namespace(namespace)

Right now, we have e2e test for that executes only CSI mock driver with artificial limit of 2 volumes per node. We need a test with real CSI driver(s). Proposal: Allow individual test storage drivers to specify if they support CSI limits. Add a test that picks a random node and creates as many pods with a PV that the volume can handle (128 on GCE!). Check that the node can't handle any other volume. Make this test part of . Add similar test(s) for in-tree volume plugins. Make sure the tests work with in-tree volume plugins migrated to CSI.
/sig storage
/assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
This sounds important /remove-lifecycle rotten /lifecycle frozen
/priority backlog
To add to this I believe we should test that an in-tree plugin and its "replacement" CSI Driver share the same attach limits (both max and current usage)
/cc
I see this test has been here: But I want to add a more basic/faster version of the test that simply checks if a driver has then has been set in CSINode, similar to wdyt, would it be useful?
This is a really old issue :-D. I think the required test has been implemented, however, we do not run it in CI, because it takes too much time to provision 128 PDs that a GCP node can handle. /unassign That sounds useful!
/assign opened to close this for good :)


Question: resource *storageframework.VolumeResource } var ( <ins> dInfo   = driver.GetDriverInfo() </ins> eDriver storageframework.EphemeralTestDriver l       local )

Let's assume this scenario: linux node, zone A, tainted windows node, zone B, untainted In a multizone cluster, while trying to run the generic ephemeral volume tests in Windows nodes it might be possible that a volume that's supposed to be for a Windows node is created in the zone for the Linux node instead. The generic ephemeral storage tests use : [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral [Testpattern: Generic Ephemeral-volume (default fs) (immediate)] ephemeral [Testpattern: Generic Ephemeral-volume (block volmode) (late-binding)] ephemeral This issue is happening because we're using the immediate testpattern which is not compatible with a multizone cluster. /kind failing-test /sig storage
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I think "immediate" binding in general won't work in a multi-zone cluster. What do we do for other tests that explicitly specify immediate mode?
I think nothing special in any of the tests besides , it's just another test pattern that's tested
Do you see failures of other "immediate" mode tests? I'm wondering if there is some logic to detect and automatically skip those tests
that was the only failure, after I the skip I didn't see any other testpatterns that failed with "immediate" mode in other test groups.
I checked tests that passed with VolumeBindingMode immediate which are found in the topology tests, the topology tests detect if the testdriver has the field set to work and modify the clone of the StorageClass adding the field to it, I'm going to do the same for the Generic Ephemeral Volume tests.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.


Question: resource *storageframework.VolumeResource } var ( <ins> dInfo   = driver.GetDriverInfo() </ins> eDriver storageframework.EphemeralTestDriver l       local )

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue with Mark this issue as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close not-planned [1]:
Closing this issue, marking it as "Not Planned". <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue with Mark this issue as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: return false, err } <del> type ContainerState uint32 </del> <ins> type ContainerState int </ins> const ( <del> ContainerStateWaiting ContainerState = 1 << iota </del> <ins> ContainerStateWaiting ContainerState = iota </ins> ContainerStateRunning ContainerStateTerminated ContainerStateUnknown

Slow conformance test has caused issues like We should make runtime/image conformance test run faster without breaking the test coverage, possibly by: Shorten consistently check interval . Remove redundant test cases such as ... /cc


Question: "k8s.io/client-go/pkg/version" "k8s.io/client-go/tools/cache" "k8s.io/client-go/tools/record" <del> "k8s.io/cloud-provider" </del> <ins> cloudprovider "k8s.io/cloud-provider" </ins> nodehelpers "k8s.io/cloud-provider/node/helpers" servicehelpers "k8s.io/cloud-provider/service/helpers" cloudvolume "k8s.io/cloud-provider/volume"

E2e tests occasionally flake on AWS. It happens randomly to all tests that create a pod with a volume, write something to the volume, delete the pod and create a new one that reads from the volume. For example: /D controller detaches a volume from a node (= the old pod died) after that (under 1s) it tries to attach the same volume to the same node (=the new pod is scheduled and coincidentally it uses the same node). In most cases this works, however, few times per day it happens that the second attach (in Kubernetes terms, see below) does not succeed and the test times out. AWS cloud provider in Kubernetes heavily uses DescribeInstances and DescribeVolumes to check that a volume has been attached or detached: In step 1. above, Kubernetes issues AWS API call and polls until it reports that the volume has been detached. In step 2. above, Kubernetes issues AWS API call to check what devices are used on a node and then it tries to find a free device for the volume that should be attached. In this bug, in step 2 reports the volume as detaching, even that reported the volume as detached in step 1. There was no call in between! Kubernetes does not expect that this could ever happen and does not try attach the volume (it's not reported as detached, so why bother, right?). This is the code that finds out that the volume is not detached ("detaching" != "detached"): And few lines below we skip : We (OpenShift) run lot of Kubernetes e2e tests on AWS. We run them in a single AWS project and I guess we hit some timing issues / races on AWS side. We hit also AWS API quota pretty hard, but it does not seem to be related to this bug, I mention it only to illustrate the amount of things that run in parallel and that AWS may not like that load. Environment: Kubernetes v1.13.4 /kind bug /sig aws /sig storage
Flow from AWS API perspective: So far so good. = volume vol- is back detaching: All timing is related to a response received, processed and logged by Kubernetes, which may be different to time corresponding request was sent. Everything in this report happens in sequence, there is no pararellism for a single volume.


Question: "k8s.io/client-go/pkg/version" "k8s.io/client-go/tools/cache" "k8s.io/client-go/tools/record" <del> "k8s.io/cloud-provider" </del> <ins> cloudprovider "k8s.io/cloud-provider" </ins> nodehelpers "k8s.io/cloud-provider/node/helpers" servicehelpers "k8s.io/cloud-provider/service/helpers" cloudvolume "k8s.io/cloud-provider/volume"

(Other volumes may be and often are attached / detached in parallel to this flow).
this looks like a bug in AWS itself, seems to return old data randomly. Can you please check AWS side of things? And we probably need fix / workaround on Kubernetes side. How can we check that a volume has been 100% detached if returns inconsistent data? We can check if a volume is in and not report is as attached here: This will cause Kubernetes to re-use the same device name and call . How bulletproof is it going to be? And most importantly, what other parts of Kubernetes depend on being consistent?
I confirmed internally. This is expected behavior for and since ec2 API follows a where resource mutation might be not reflected immediately. This could explain why returns a volume in detaching state even after the previous call returns without the volume (detached) I think this won't work in the case when the volume is actually stuck in detaching state, since attaching a stuck volume could simply make it worse. Given that, how about, in call, we return error when notice the volume is still being returned from and we rely on the AD controller retry to check the state next time, and hopefully the state converges.
If can return stale data, depending on it in (or anywhere else) probably won't help, because we may again get new or stale data "randomly". We call in today to get fresh list of used/free devices. Depending on timing, two cases happen: Sometimes it returns that the volume is detaching and I can see in logs. Handling as you suggest would help here. Sometimes it returns that the volume is detached. We find a new device name and call AWS and start polling for finished attach. This then returns that the volume is detaching from the node with the old device name. So we're actually attaching a volume that's being detached and you wrote that's a bad thing. In my logs, I can see that Kubernetes recovers from this error: Volume plugin AttachVolume() fails with A/D controller re-tries AttachVolume() calls AttachDisk() that calls to find a free device. returns that the volume is actually being attached to the node (I can see in logs) AttachDisk() polls until the volume is attached. -it works, but to me it feels like it's just an accident. Checking can't help in this case.


Question: "k8s.io/client-go/pkg/version" "k8s.io/client-go/tools/cache" "k8s.io/client-go/tools/record" <del> "k8s.io/cloud-provider" </del> <ins> cloudprovider "k8s.io/cloud-provider" </ins> nodehelpers "k8s.io/cloud-provider/node/helpers" servicehelpers "k8s.io/cloud-provider/service/helpers" cloudvolume "k8s.io/cloud-provider/volume"

So, is there another way how to check that a volume is really detached? Or should we promote the error flow described above to official fix and add bunch of comments around so we won't remove / tweak the flow in future? What bad can happen when we call AWS on a volume that's detaching? AWS at least once reported that the volume is detached, so the volume probably is not be stuck.
I think it is only bad if the volume was previously stuck. But feels acceptable for me if it is being detached and the previous call in at least confirmed once. Confirming internally. But I suspect there isn't any. As long as the volume is not stuck (unmounted properly), we should be able to attach a detaching volume especially when it is at least confirmed being detached once from call in . Even if the first time it doesn't work, retry should help recover the situation. How do you think?
According pointed, AWS recommends two actions to manage the eventual consistency model (k8s already does 1, but AFAIK not 2, except when there's an exponential backoff among calls: 1) Run the appropriate call before modifying a resource 2) Add wait time between subsequent commands, even if returns a green response (this does infer that might return stale data) Currently, k8s assumes a state based on a single green response, e.g., is called repeatedly with an exponential backoff until it returns a given volume state. What if it required at least two positive responses instead? What are the chances that would return stale data twice? I'd assume that two calls returning a detached volume (with exponential backoff in between, which enforces point 2 above) would be reliable enough. The downside is that this would increase the API usage, which is already high.
do you know if the native AWS calls may be more stale-data-proof?
Looking at implementation, it tries with constant backoff of 15 seconds (which I consider pretty large) and it returns when the first returns that the volume is in use, so basically the same as what Kubernetes does.
From what I saw, sometimes it happens that reports a volume as detached, subsequent reports that the volume is not attached, succeeds and only after that shows that the volume is (from the old device name) instead of . So Kubernetes got at least two confirmations that the volume is detached.


Question: "k8s.io/client-go/pkg/version" "k8s.io/client-go/tools/cache" "k8s.io/client-go/tools/record" <del> "k8s.io/cloud-provider" </del> <ins> cloudprovider "k8s.io/cloud-provider" </ins> nodehelpers "k8s.io/cloud-provider/node/helpers" servicehelpers "k8s.io/cloud-provider/service/helpers" cloudvolume "k8s.io/cloud-provider/volume"

What would be the correct delay to wait for second (third, ...) confirmation that the volume is really detached? IMO it would hurt our performance a lot.
The Wait function simply calls DescribeXX under the hood and uses some hard coded timeout. So it's similar to what kubernetes does like said.


Question: // deleteSecret deletes the secret with the given name from the host // cluster. func deleteSecret(clientset internalclientset.Interface, name, namespace string) error { <del> return clientset.Core().Secrets(namespace).Delete(name, &metav1.DeleteOptions{}) </del> <ins> orphanDependents := false return clientset.Core().Secrets(namespace).Delete(name, &metav1.DeleteOptions{OrphanDependents: &orphanDependents}) </ins> } // isNotFound checks if the given error is a NotFound status error.

Problem: When users sends a cascading deletion request to federation i.e they set orphanDependents=false, then federation should also set orphanDependents=false, when it sends the request to underlying clusters.
Fixed by


Question: "//pkg/scheduler/internal/queue:go_default_library", "//pkg/scheduler/util:go_default_library", "//staging/src/k8s.io/api/core/v1:go_default_library", <ins> "//staging/src/k8s.io/apimachinery/pkg/api/resource:go_default_library", </ins> "//staging/src/k8s.io/apimachinery/pkg/apis/meta/v1:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/runtime:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/util/sets:go_default_library",

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Today, the scheduler re-processes unschedulable nodes . Node updates are sent by Kubelet frequently (every 10 seconds) and often times they don't change any of the node properties that could potentially make them feasible. We should be smarter about these updates and move pods to the schedulable queue only when certain properties of nodes change that could make them feasible. These properties include node allocatable, node conditions, taints, labels, .... Why is this needed: Improves performance of the scheduler and avoids doing unnecessary attempts to schedule pods. This is particularly helpful when a cluster has a lot of unschedulable pods. <!-- DO NOT EDIT BELOW THIS LINE --/kind feature This is not a feature really. It is performance optimization, but in the absence of such a label, I need to use "feature" here. /sig scheduling
I can help to fix it. /assign mlmhl
The main reason for this problem is / update their status too frequently and most of these updates are heartbeats without any meaningful change. Recently sig-node introduces a new to alleviate this problem. With this mechanism, kubelet will split heartbeat and node status updates into two independent operations, and reduce the frequency of status updates to once per 1 minute(or more). In the long run this mechanism can indeed alleviate our problems, but at present it does not help us much as this is still an alpha feature and the node status updates frequency is still a bit high(per 1 minute right now).
Maybe we can try comparing old and new nodes after stripping their heartbeat info, like this:
I think that had sent earlier is more efficient. However, that PR does not filter out . In Wei's code, the heartbeat time is removed correctly. I think we should fix this issue and resubmit that PR.
could you resubmit a PR to: disregard in conditions comparison: and (optionally) disregard timestamp in taints comparison:
Thanks for all your advice, a new PR is submitted: This PR does not include your second suggestion(disregard timestamp in taints comparison) as I'm not sure if can be changed when node updating.


Question: "//pkg/scheduler/internal/queue:go_default_library", "//pkg/scheduler/util:go_default_library", "//staging/src/k8s.io/api/core/v1:go_default_library", <ins> "//staging/src/k8s.io/apimachinery/pkg/api/resource:go_default_library", </ins> "//staging/src/k8s.io/apimachinery/pkg/apis/meta/v1:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/runtime:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/util/sets:go_default_library",

If it can be changed, I will add this filter later, otherwise I think we can keep the current implementation to make the compare operation simpler and faster. What's more, just as I said in the PR's description, I'm not sure how much side effects this change has on scheduler's resource consumption, as it always copy node's conditions before comparison.
I am thinking whether we could move this logic(whether node became more schedulable) to kubelet, kubelet reports it through heartbeats.


Question: "time" "k8s.io/kubernetes/pkg/api" <ins> "k8s.io/kubernetes/pkg/api/errors" </ins> "k8s.io/kubernetes/pkg/api/meta" "k8s.io/kubernetes/pkg/api/testapi" apitesting "k8s.io/kubernetes/pkg/api/testing"

When attempting to fetch a resource version that is too old, the behavior of watch (for both HTTP and web sockets) changed when watch cache was enabled. The watch cache returns a 410 - without the watch cache, we return 200 and then write an "error" entry to the watch stream. This broke a client that dependent on the watch behavior, and moreover is inconsistent with other errors we set in the registry. Also, for watch over web sockets, most javascript clients will be unable to get at a 410 error from the connection, because browsers don't get access to the data. We should restore the watch cache behavior to be consistent with the previous behavior as this is a regression in API semantics. Resource without watch cache, watching too old: Resource with watch cache, watching too old:
I think this is candidate for back port to 1.2 to restore the previous behavior.
since this broke your javascript client, FYI because of watch cache
+1 for backport
hmm - we enabled watch cache in 1.1 - so this should also be broken in 1.1
It's possible that it was always nondeterministic and just depends on how fast the call to the storage layer executes. I think all of our in-tree clients will handle it either way.
I'm fairly certain this never used to behave this way, because OpenShift hasn't enabled the watch catch and we have never had a report of this until we enabled the watch cache for Kube resources just recently. , Daniel Smith wrote:
I see what you mean about ordering - etcd remote call would almost never be fast enough to beat the watch event. In practice, I suspect no one ever saw this without the watch error. Either way, for web sockets if we don't fix this clients can't get this error (the browser eats the 410 during the websocket negotiation and client code can't get access). So it's at least broken for web sockets. , Clayton Coleman wrote:
Well, it should be easy to just not wait at all, maybe? I'm OK with changing this behavior.
Although, are web clients XSS hardened? I'm not sure that we're optimizing for them currently.
If you mean CSRF, it isn't an issue with bearer token auth.


Question: "time" "k8s.io/kubernetes/pkg/api" <ins> "k8s.io/kubernetes/pkg/api/errors" </ins> "k8s.io/kubernetes/pkg/api/meta" "k8s.io/kubernetes/pkg/api/testapi" apitesting "k8s.io/kubernetes/pkg/api/testing"

Basic auth should already have a big warning around it saying "not safe with browsers"
Maybe I'm missing something, but it seems deterministic to me. This is the uncached watch impl: once the resourceVersion parses, there is no path that returns a direct error, so all errors would have to be returned as an event via the ResultChan(). I don't see us reading from that until after we've already written a 200 and flushed:
Having read the docs, I was very surprised that "type": "ERROR" notices are sent as 200 (even over plain http), with the semantic code e.g. 410 only represented under inside. I see there was good motivation here to stick to 200, but it should be called out in the documentation. doesn't show at all how watch returns errors, and it even says "clients must handle the case by recognizing the status code 410 Gone" which really makes one expect an actual HTTP 410.
I agree this should be called out in the docs. Can you file a bug there and/or send a PR? I don't think this ~4 year old issue is the best place to track that. (Good detective work finding this, though!)


Question: // BusyBoxImage is the image URI of BusyBox. BusyBoxImage = imageutils.GetE2EImage(imageutils.BusyBox) <ins> // AgnHostImage is the image URI of AgnHost AgnHostImage = imageutils.GetE2EImage(imageutils.Agnhost) </ins> // For parsing Kubectl version for version-skewed testing. gitVersionRegexp = regexp.MustCompile("GitVersion:"(v.+?)"")

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: N/A Which test(s) are failing: Since when has it been failing: Ever Reason for failure: Vm on Azure with no public IP assigned aren't able to perform a command to the internet. Anything else we need to know: The test (see ) tries to test the internet connection for a pod by pinging 8.8.8.8. But azure does not provide ICMP support for VM without a public IP therefore this test cannot succeed.(see ) A fix would be to change the test method to a tcp request instead of a ping request (the test already performs a http get to in its BeforeSuite (see ))
/sig network /sig testing /sig azure
you may refer to (Pods don't have connectivity to external networks )
cc WDYT about the proposal to change this from an ICMP call? We probably need input from sig-architecture as well since this as well /sig architecture
/triage unresolved
An alternative is to issue a TCP/UDP request to 8.8.8.8 on port 53.
ICMP is dropped by Azure for security reasons. This is intended. Can use curl as a replacement. I will fix the corresponding test.
SGTM
Can you please review the related PR? I'll in the PR again. Thanks!


Question: "//test/e2e/framework:go_default_library", "//vendor/github.com/onsi/ginkgo:go_default_library", "//vendor/github.com/onsi/gomega:go_default_library", <ins> "//vendor/k8s.io/utils/exec:go_default_library", </ins> ], )

It would better to have e2e for iSCSI persistent volume. /sig storage
/assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close


Question: glog.V(4).Infof("UpdateLoadBalancer(%v, %v, %v)", clusterName, loadBalancerName, hosts) ports := service.Spec.Ports <del> if len(ports) > 1 { return fmt.Errorf("multiple ports are not yet supported in openstack load balancers") } else if len(ports) == 0 { </del> <ins> if len(ports) == 0 { </ins> return fmt.Errorf("no ports provided to openstack load balancer") }

LBaaS implementation (v1 or v2) will not support services that require to load balancer more than one port.
Fixed for LBaaS v2 by I suspect LBaaS v1 can't support multiple ports (not sure if reusing the VIP across multiple loadbalancers is a universally supported thing). I also believe LBaaS v1 is also mostly gone in favour of v2 in deployments so trying to address this in v1 is extremely low priority unless someone steps forward to ask for it. So: I think we can close this bug now, but I lack the powers to do that directly.
thank you for the update.


Question: apiService := &v1.APIService{ Spec: v1.APIServiceSpec{ <ins> Group:   "apiservicegroup", Version: "v1", </ins> Service: &v1.ServiceReference{Name: "dummy"}, }, }

When installing the metrics-server to implement the API and the prometheus-adapter to implement the API, it is possible to make the OpenAPI handler fail to initialize with this error log in apiserver: This will result in kubectl discovery calls to /openapi/v2 to fail with a 503: The OpenAPI handler should not fail to initialize if both and are serving the same path for different API groups/versions. I can provide exact installation steps if needed, however, a high-level repro is: install metrics-server to serve install prometheus-adapter to serve ensure resourceRules is populated in the prometheus-adapter config so that resource metrics are also served by restarting apiserver will result in the OpenAPI failure and 503 responses to /openapi/v2 path No response <details</details<detailsAWS </details<details</details<details</details<details</details<details</details
Please let me know if I should reopen the issue in
/sig api-machinery
/sig instrumentation [metrics apis]
Would you mind taking a look when have time? Thank you^^ /triage accepted
I have a question. should be serving under . Why does an apiservice serving another gvk register paths under ? cc we've made the assumption that an APIService (such as metrics-server or prometheus-adapter) serves only gvks under its own registered group version. I'll do some more investigation but just wondering if prometheus doesn't follow that assumption?
the prometheus-adapter can also be configured to serve and replace the metrics-server by enabling these metrics, However, if the prometheus-adapter fronts and metrics-server fronts , it does seem like the correct behavior would be for prometheus-adapter to serve until an operator updates the APIService to reference prometheus-adapter rather than metrics-server. We're seeing this issue occur for upgrades between 1.27 and 1.28 so it does look this path duplication did not cause the OpenAPI handler to fail on 1.27.
I just upgraded from 1.27 to 1.28 and running into this problem. Some kubectl apply operations fail, and ArgoCD cannot load its custom resources. Any suggested workaround? Would removing the from the prometheus configuration resolve it? Or perhaps disable metrics-server?
Simply looking at the code diff between 1.27 and 1.28 I can't pinpoint the cause.


Question: apiService := &v1.APIService{ Spec: v1.APIServiceSpec{ <ins> Group:   "apiservicegroup", Version: "v1", </ins> Service: &v1.ServiceReference{Name: "dummy"}, }, }

Can I get your exact repro steps? suggests that the conflicting paths should be removed before the merge. We always call this function with
One other question is if this is reproducible by just applying a set of yaml/installations or does it require a cluster upgrade to trigger? Thanks!
hit same problem and we cannot install any other new helm plugins. maybe we need to move forward
Here are verbose repro instructions. An apiserver restart on 1.28 will induce the issue.
Hello. I'm still having some trouble reproducing although I'm not 100% sure if the configuration is correct. I'm trying to repro via kind. afaik installing via helm has prometheus adapter serving . seems to suggest that prometheus adapter is serving under . Applying the update to the configmap per the instructions didn't change the apiservice on my end. Not sure if this is my misconfiguration or intended. I've also tested via integration test that having two openapis serving the same path doesn't cause errors and one will get filtered out. A simple workaround is probably to delete the metrics-server if it is already superseded by the prometheus-adapter.
The is not set up in an resource for us, we only created an APIService with . When I go fetch the openAPI schema served up by the endpoint, it has schema information for in it if that is enabled using . If I remove then the schema fetched from the server doesn't have stuff in it, and the error is resolved. We have been keeping around because I think something uses it directly, or used to. Maybe ? That could have changed But in any case, this problem is quite severe when it happens so it would be nice to fix it as there may be others who do the same update and have to do some deep diving into the logs and all kinds of crazy diagnostics and guesswork to figure it out.
How does the get reached without an APIService? The resources would be unavailable for the apiserver to query and it'll be through something else? I see, adding resources for me doesn't populate the OpenAPI. I wonder if this could be cloud provider specific. I'm totally supportive of fixing, it's just quite difficult to reproduce on my part .
Hmm good question.


Question: apiService := &v1.APIService{ Spec: v1.APIServiceSpec{ <ins> Group:   "apiservicegroup", Version: "v1", </ins> Service: &v1.ServiceReference{Name: "dummy"}, }, }

In our staging cluster I tried bringing back the in the custom metrics server configuration and restarting all the masters, and the problem did not come back, everything seems OK now. So it maybe isn't that easy to reproduce
After updating the configmap, restarting the prometheus-adapter pod to pickup the changes and restarting the apiserver are both required to induce the issue. I have been able to deterministically reproduce this following those steps, please follow up here if you are still having issues.
That's good to hear. I'll need some more help with reproducing it. I've been able to verify that restarting the prometheus adapter pod picks up the configmap changes but restarting the apiserver hasn't caused any crashes for me. I've tested this both on the latest master branch and 1.28.
While I'm yet to fully understand these changes, the 2 PRs below (both of which went into 1.28) seem likely candidates for a regression here: - introduced lazy loading in openAPI v2 handler to load/compute the aggregator spec when requested by client - switched the spec merging function from to any clue if one of the above changes could be causing mischief?
Also from example request at the top, the impact seems to be to the v2 API: Nathan - when you get a chance, could you also confirm if v1 API doesn't run into this?
OpenAPI only supports v2 and v3.
hi, any clue about this issue?
It's possible but I haven't been able to reproduce it. Can I ask if this is happening on cloud providers outside AWS?
Here are the apiservices and relevant pod objects with a working repro from these steps If it's useful, I can provide yamls for these objects in addition to the settings applied in the prometheus-adapter configmap. Can you confirm that the pods backing your prometheus-adapter and metrics-server are running?
I meet this problem
We the problem in GCP as well.
ACK in Alibaba Cloud also meets this problem
We were able to fix the v2 availability by and re-installing the kube-prometheus helm chart. Unfortunately, after restarting the k8s api server, v2 availability is broken again.
yes, I'm running on kind. I'll do some more digging, assuming it's related to those PRs linked by but still haven't pinpointed anything.


Question: apiService := &v1.APIService{ Spec: v1.APIServiceSpec{ <ins> Group:   "apiservicegroup", Version: "v1", </ins> Service: &v1.ServiceReference{Name: "dummy"}, }, }

I tried to reproduce on my local setup with based on steps + , but I could still get a successful response when I executed . Then I reverted one of the step mentioned in to and waited until the new pod came alive and then did a apiserver restart and I was able to reproduce the same thing on as well i.e; I get the 503 response when I execute following command after the fact. After this point, even if i change it back to and restart the apiserver , the error is persistent. Let me know if this works for you to reproduce ? PS: Also, i want to call out that there is no issue with , its .
Thanks, alright I found a way to reproduce it consistently without restarting the apiserver. Still investigating the root cause. Thanks for the detailed reproduction and
Quick update: This issue was a combination of bugs both in k/k and prometheus adapter. I opened to mitigate on k/k side which should fix the issue. The issue was difficult to reproduce because it requires a very specific ordering of OpenAPI merging to trigger, which is why I suspect many people were able to reproduce after some restarts to kube-apiserver. Prometheus and metrics server serve differing OpenAPIs for the endpoint. Prometheus also doesn't seem to be registering discovery's OpenAPI properly (missing OpenAPI for but having the OpenAPI for whereas it is all present in metrics server. This inconsistency in how prometheus and metrics server handle the OpenAPI surfaced the merging issue.
a little wired question: why do we need to restart kube-apiserver to reproduce this problem?
Tried to reproduce the issue after cherry picking the fix and didn't run into below error. Fix is working as expected.
It can look like restarting prometheus-adapter "fixes" the issue? I am also curious about suggestion of removing the section from the config map works as a temporary workaround until released upstream (we use and cannot delete the apiregistration)? As I understand the fix for this issue is currently in


Question: fi source "${KUBE_HOME}/kube-env" <ins> if [[ -n "${KUBE_USER:-}" ]]; then if ! [[ "${KUBE_USER}" =~ ^[-._@a-zA-Z0-9]+$ ]]; then echo "Bad KUBE_USER format." exit 1 fi fi </ins> config-ip-firewall create-dirs ensure-local-ssds

under kubetest.Up Nov 22 2023, last 7 runs failed No response /sig
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
APIServer failed for
Yes it is failing for me too!
/sig testing
the generator config source must include the env to handle the "external" provider deprecation: /sig cloud-provider /area provider/gcp
/remove-sig testing /sig release
all other jobs seem green in the 1.29 blocking/informing, post that change. i will try sending a PR for this, but you need to find a non-USA approver if you want it merged before Monday. /assign
never mind, the generator is giving me errors: /unassign
/cc


Question: // from the registered device plugins. func (m *ManagerImpl) Allocate(node *schedulercache.NodeInfo, attrs *lifecycle.PodAdmitAttributes) error { pod := attrs.Pod <ins> devicesToReuse := make(map[string]sets.String) </ins> // TODO: Reuse devices between init containers and regular containers. for _, container := range pod.Spec.InitContainers { <del> if err := m.allocateContainerResources(pod, &container); err != nil { </del> <ins> if err := m.allocateContainerResources(pod, &container, devicesToReuse); err != nil { </ins> return err } <ins> m.podDevices.addContainerAllocatedResources(string(pod.UID), container.Name, devicesToReuse) </ins> } for _, container := range pod.Spec.Containers { <del> if err := m.allocateContainerResources(pod, &container); err != nil { </del> <ins> if err := m.allocateContainerResources(pod, &container, devicesToReuse); err != nil { </ins> return err } <ins> m.podDevices.removeContainerAllocatedResources(string(pod.UID), container.Name, devicesToReuse) </ins> } m.mutex.Lock()

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug /area hw-accelerators What happened: In the context of the device plugin, when using init containers in your pod spec, the current behavior will not reuse devices between init containers and regular containers. This is problematic as Pod scheduling is currently wrong: , the current predicate algorithm rightly assumes the maximum number of devices in use at one time is . However, the pod predicate ran on the node is the sum of devices used by regular and init containers. Meaning that your pod can bounce back between the scheduler and a node infinitely. Also given that many of the extended resources are typically more expensive that the built-in ones like CPU, Memory and Storage this bug has very real financial drawbacks in terms of resource usage (e.g: using 4 GPUs on a pod that only uses 2 GPUs at a time). Furthermore, this unexpected behavior was caught during implementation (but we did not catch the scheduling implications) and decision was taken to ignore it for the first implementation. We need to document it for 1.9 What you expected to happen: Devices can be reused by Init Containers Correct resource accounting In terms of the current behavior of the Allocate call to the device plugin, I'd expect the allocate call to be done before every Init containers start and once at the end of the last Init container (for regular containers). /assign
Yep, I realized the inconsistency between the predicate behavior of scheduler and that of node too. Having said that, considering we treat a pod as a whole when predicating currently, it's kind of difficult to reuse the runtime informations between init containers and regular containers. The real crux of the matter might be whether the scenario of requiring extended resources in init containers is common, if so, the issue should be addressed ASAP.
If I remembered correctly, the current design to improve DP includes a reconciliation in admit phase of Pod, it may be able to fix Allocate call for
can you point me to said design ? I'm not sure that the Allocate call for init containers is planned to be fixed as described.
Can you guys please point to the code where is happening. Looks to me that same logic is executed on both the scheduler and the node to calculate a pod's resource request and which is  On the node, function call tree looks like this: Is it possible to add few more words in explanation?


Question: // from the registered device plugins. func (m *ManagerImpl) Allocate(node *schedulercache.NodeInfo, attrs *lifecycle.PodAdmitAttributes) error { pod := attrs.Pod <ins> devicesToReuse := make(map[string]sets.String) </ins> // TODO: Reuse devices between init containers and regular containers. for _, container := range pod.Spec.InitContainers { <del> if err := m.allocateContainerResources(pod, &container); err != nil { </del> <ins> if err := m.allocateContainerResources(pod, &container, devicesToReuse); err != nil { </ins> return err } <ins> m.podDevices.addContainerAllocatedResources(string(pod.UID), container.Name, devicesToReuse) </ins> } for _, container := range pod.Spec.Containers { <del> if err := m.allocateContainerResources(pod, &container); err != nil { </del> <ins> if err := m.allocateContainerResources(pod, &container, devicesToReuse); err != nil { </ins> return err } <ins> m.podDevices.removeContainerAllocatedResources(string(pod.UID), container.Name, devicesToReuse) </ins> } m.mutex.Lock()

PR that the mentioned change: Code in question is here:
Above looks incorrect description. Predicate that run on the scheduler and on the node is same i.e max(init, regular). Pod admission will pass at the node. Problem might happen at allocation where allocated resources is not the max but the sum of (init, regular). And in that case, pod would hang there in fail state rather 'bounce back infinitely'.
Problem might happen at allocation where allocated resources is not the max but the sum of (init, regular). And in that case, pod would hang there in fail state rather 'bounce back infinitely'. Absolutely correct, sorry to make you confused, I think Renaud name it mainly bacause the allocate func is supposed to be called during the predicate phase, that is, passed in as a part of PredicateAdmitHandler.
The original idea was onced listed in as a sub-task: Now it's removed (I don't know why ...), while I believe the idea was: device resources allocation during pod admit phase. This comment is also related if needed during reconcile in 1 and cache these devices info cached devices later during container create phase
Handle device resources allocation during pod admit phase. This comment (comment) is related Call Allocate() if needed during reconcile in 1 and cache these devices info Use cached devices later during container create phase Current code behaviors exactly as you described, but to address the issue, the key point might be to decide which devices of init containers could be reused by which regular containers, and generate cache accordingly in step2; when creating containers, we simply fetch runtime information from cache.
Thanks a lot for filing this issue, I think for 1.9, we probably want to list this as a known issue and may advise people to not request gpu resources in init containers. We can target to fix this problem in 1.10.
Agreed We don't have a clear solution for this tough. It's pretty clear to me that device resources allocation during pod admit phase is not the right model when init containers comes into the picture. GPUs are pretty simple example. On of the things we plan to do is scrub the GPU memory during the Allocate phase. If GPUs are re-used between initContainers and regular containers the current model doesn't fit.


Question: // from the registered device plugins. func (m *ManagerImpl) Allocate(node *schedulercache.NodeInfo, attrs *lifecycle.PodAdmitAttributes) error { pod := attrs.Pod <ins> devicesToReuse := make(map[string]sets.String) </ins> // TODO: Reuse devices between init containers and regular containers. for _, container := range pod.Spec.InitContainers { <del> if err := m.allocateContainerResources(pod, &container); err != nil { </del> <ins> if err := m.allocateContainerResources(pod, &container, devicesToReuse); err != nil { </ins> return err } <ins> m.podDevices.addContainerAllocatedResources(string(pod.UID), container.Name, devicesToReuse) </ins> } for _, container := range pod.Spec.Containers { <del> if err := m.allocateContainerResources(pod, &container); err != nil { </del> <ins> if err := m.allocateContainerResources(pod, &container, devicesToReuse); err != nil { </ins> return err } <ins> m.podDevices.removeContainerAllocatedResources(string(pod.UID), container.Name, devicesToReuse) </ins> } m.mutex.Lock()

do you think if you have time to implement the solution you mentioned in the above comment We can try to get it into a 1.9 sub release. I think for now, lets just support device reuse among initContainers and normal containers. We can revisit this part when we need to guarantee an Allocate call per container.
Right what I am mentioning here is that it is not possible to do that. The point of the device plugin is so that we can execute vendor specific code before making the device available in the container (e.g: Scrub the GPU memory). I don't think the proposed solution addresses this problem.
Sure. But to clarify, I think if we dont't expect big shock on current implementation, we could only provide limited support on the point, that is, without re-generate runtime information for regular containers, only those whose requested resource is equal to or is a subset of that in init containers could make a resuse(e.g. if we have two init containers, each requests 2 devices, then a regular container that request 3 devices still could not reuse the devices.) Besides, if a initcontainer requests 2 devices, and we have two regular containers, each requests 1 device, then due to the lack of runtime info re-generation, the devices could only be reused by one of the containers, not both of them. In sum, since we don't re-generate runtime information, we could only reuse devices per container, not per device, like we do in scheduler. Is that what you mean?
Hmm, that is a good question. I forgot that we now return the combined runtime information in AllocResponse. Then for now, perhaps we should just return an error in device plugin manager Allocate() in case both init containers and containers request the same device plugin resource?
You mean we give up reuse and prohibit users to request same kind of device in init containers and regular containers? that seems a bit weird. And if we decide to do so, we might need to do the check at the first beginning, i.e. in API validation, not wait until it goes into DevicePlugin.
I think for 1.9, we just return an error when both init containers and normal containers of a pod request the same device resource, and document this in the device plugin user guide.


Question: // from the registered device plugins. func (m *ManagerImpl) Allocate(node *schedulercache.NodeInfo, attrs *lifecycle.PodAdmitAttributes) error { pod := attrs.Pod <ins> devicesToReuse := make(map[string]sets.String) </ins> // TODO: Reuse devices between init containers and regular containers. for _, container := range pod.Spec.InitContainers { <del> if err := m.allocateContainerResources(pod, &container); err != nil { </del> <ins> if err := m.allocateContainerResources(pod, &container, devicesToReuse); err != nil { </ins> return err } <ins> m.podDevices.addContainerAllocatedResources(string(pod.UID), container.Name, devicesToReuse) </ins> } for _, container := range pod.Spec.Containers { <del> if err := m.allocateContainerResources(pod, &container); err != nil { </del> <ins> if err := m.allocateContainerResources(pod, &container, devicesToReuse); err != nil { </ins> return err } <ins> m.podDevices.removeContainerAllocatedResources(string(pod.UID), container.Name, devicesToReuse) </ins> } m.mutex.Lock()

The reason is because in the current device plugin implementation, we don't have a good way to free devices used by init containers of an active pod. It is hard to do this kind of validation at API level or even scheduler because we can't know whether an extended resource is device resource or not. So for now, I think we should probably just return a clear error message during admission handler. I am not sure yet whether there will be strong use cases that need to share devices between init containers and normal containers. We can revisit this based on user feedback. If there is strong user demand to support this, something we could do is to differentiate init containers and normal containers in podDevices, and free up devices used by init containers when any normal container is about to run. That would request some code structure change, and we may wait to see whether there is strong demand to support this.
It is hard to do this at API level or even scheduler because we can't know whether an extended resource is device resource or not. Oh, you are right, I ignored the resource awareness issue. I also agree that it's hard to address the issue without big change to current implementation. But I still feel the inconsistency of behavior between API and kubelet might confusing users. However, its effect should be sustainable, since we haven't received feedback about init containers up to now.
Right. So for now, perhaps just return an error when this happens, to give clear error signals.
Oh, I'm afraid we miss something, the check may not work as expected. As I commented, because init-containers run sequentially, we did reuse between init containers in , that is, if we have two init containers, each request 1 device, then we expect 1 device in total in scheduler, but expect 2 in deviceplugin. The check could not cover the scenario.
Thinking about this problem a bit more, I wonder whether we may tackle this issue from the following approaches, ordered by their complexities: 1) Just return an error whenever an init container requests a device plugin resource, and document in the device plugin user guide that requiring device plugin resource in init containers are not supported. This may be the short-term change we can try to get into 1.9. 2) Maintain a list of devices allocated to init containers in ManagerImpl.Allocate().


Question: // from the registered device plugins. func (m *ManagerImpl) Allocate(node *schedulercache.NodeInfo, attrs *lifecycle.PodAdmitAttributes) error { pod := attrs.Pod <ins> devicesToReuse := make(map[string]sets.String) </ins> // TODO: Reuse devices between init containers and regular containers. for _, container := range pod.Spec.InitContainers { <del> if err := m.allocateContainerResources(pod, &container); err != nil { </del> <ins> if err := m.allocateContainerResources(pod, &container, devicesToReuse); err != nil { </ins> return err } <ins> m.podDevices.addContainerAllocatedResources(string(pod.UID), container.Name, devicesToReuse) </ins> } for _, container := range pod.Spec.Containers { <del> if err := m.allocateContainerResources(pod, &container); err != nil { </del> <ins> if err := m.allocateContainerResources(pod, &container, devicesToReuse); err != nil { </ins> return err } <ins> m.podDevices.removeContainerAllocatedResources(string(pod.UID), container.Name, devicesToReuse) </ins> } m.mutex.Lock()

Try to allocate from this list first for later init containers and normal containers. The required no. of devices we pass to ManagerImpl.devicesToAllocate() should subtract the no. of devices we have allocated to init containers. This may require some code structure change around ManagerImpl.Allocate(), ManagerImpl.allocateContainerResources(), and ManagerImpl.devicesToAllocate(). However, Allocate gRPC call is still issued during Admission all together, and it is possible that we issue an Allocate gRPC for [dev1] (say dev1 allocated to initContianer1), followed by another Allocate gRPC call for [dev1, dev2] (say dev1 and dev2 allocated to initContainer2), with none of containers started or finished. This assumes containers running in the same pod share the same security context, and device plugin doesn't need to perform per-container allocation operations. We just use Allocate gRPC call to learn container runtime settings. 3) Take a further step from 2), during Admission, we just determine what devices to allocate to containers without issuing the actual Allocate gRPC calls. I.e., we only fill deviceIds field in podDevices.deviceAllocateInfo but leaves allocResp field empty. During ManagerImpl.GetDeviceRunContainerOptions(), we check for this case and issue the actual Allocate gRPC call for the container. This ensures we can issue a gRPC Allocate call per container. However, if a pod is restarted multiple times, we don't guarantee Allocate gRPC call will be issued during subsequent pod runs. 4) Take a further step from 3), we can check for overlapping device allocations in ManagerImpl.GetDeviceRunContainerOptions() and invalidate their allocResp so that an Allocate gRPC call will be issued next time for containers using re-allocated devices. E.g., for containerA, if we find it is allocated with [dev1, dev2] and dev1 is also allocated to containerB, we set containerB's allocResp to nil so that next time before containerB is running, we will issue an Allocate gRPC call for dev1. Note that this approach does have the drawback that device plugin failure would cause previously running pods not be able to restart, so we need to carefully evaluate our use cases to see whether we need to implement this. My suggestion is to implement 1) now and gather some user feedbacks to determine whether we need to implement the following approaches.


Question: // from the registered device plugins. func (m *ManagerImpl) Allocate(node *schedulercache.NodeInfo, attrs *lifecycle.PodAdmitAttributes) error { pod := attrs.Pod <ins> devicesToReuse := make(map[string]sets.String) </ins> // TODO: Reuse devices between init containers and regular containers. for _, container := range pod.Spec.InitContainers { <del> if err := m.allocateContainerResources(pod, &container); err != nil { </del> <ins> if err := m.allocateContainerResources(pod, &container, devicesToReuse); err != nil { </ins> return err } <ins> m.podDevices.addContainerAllocatedResources(string(pod.UID), container.Name, devicesToReuse) </ins> } for _, container := range pod.Spec.Containers { <del> if err := m.allocateContainerResources(pod, &container); err != nil { </del> <ins> if err := m.allocateContainerResources(pod, &container, devicesToReuse); err != nil { </ins> return err } <ins> m.podDevices.removeContainerAllocatedResources(string(pod.UID), container.Name, devicesToReuse) </ins> } m.mutex.Lock()

I see this a very big limitation. Special devices/resources in init containers is a very common use case IMO. Therefore we should try to fix this. This approach perfectly makes sense to me. What is your concern in implementing this? Even today, IIUC, Allocate call is made per container. Is there any other value that this approach is adding over 1) ? I am not able to see any gains with 2) or 3) over 1). Comments please.
I feel we may not be able to implement 2) for 1.9 timeframe. Option 2) also doesn't address the case mentioned above: Option 3) covers that case for pods without restarts. Option 4) fully covers the requirement we we will scrub old container data before giving it to a new container, even if they both run in the same pod, bug it does have the mentioned drawback during device plugin failure.
Take a further step from 3), we can check for overlapping device allocations in ManagerImpl.GetDeviceRunContainerOptions() and invalidate their allocResp so that an Allocate gRPC call will be issued next time for containers using re-allocated devices. E.g., for containerA, if we find it is allocated with [dev1, dev2] and dev1 is also allocated to containerB, we set containerB's allocResp to nil so that next time before containerB is running, we will issue an Allocate gRPC call for dev1. Note that this approach does have the drawback that device plugin failure would cause previously running pods not be able to restart, so we need to carefully evaluate our use cases to see whether we need to implement this. How about making the allocation call optional? That is , if we find a contaienr using re-allocated devices, we'd try to issue an Allocate gRPC call first, and still use the cached setting if allocation failed. And, if we decide not to address the issue in v1.9, which means we have enough time to work on it later, I think approach 2) might not be a good choice. Because it has a premise that devices are stateless, recycle actions before reuse(e.g: Scrub the GPU memory mentioned above) could not be supported. And this way, as I commented above, we could only reuse devices per container, not per device, like we do in scheduler.


Question: // from the registered device plugins. func (m *ManagerImpl) Allocate(node *schedulercache.NodeInfo, attrs *lifecycle.PodAdmitAttributes) error { pod := attrs.Pod <ins> devicesToReuse := make(map[string]sets.String) </ins> // TODO: Reuse devices between init containers and regular containers. for _, container := range pod.Spec.InitContainers { <del> if err := m.allocateContainerResources(pod, &container); err != nil { </del> <ins> if err := m.allocateContainerResources(pod, &container, devicesToReuse); err != nil { </ins> return err } <ins> m.podDevices.addContainerAllocatedResources(string(pod.UID), container.Name, devicesToReuse) </ins> } for _, container := range pod.Spec.Containers { <del> if err := m.allocateContainerResources(pod, &container); err != nil { </del> <ins> if err := m.allocateContainerResources(pod, &container, devicesToReuse); err != nil { </ins> return err } <ins> m.podDevices.removeContainerAllocatedResources(string(pod.UID), container.Name, devicesToReuse) </ins> } m.mutex.Lock()

I noticed that you have already add the limitation into the , so I think it's already the decision that we add the limitation as a short-time solution in v1.9 So I posted a PR to add the check in code,, PTAL :)
Use Case: Let say there is a which node has two Devices of a resource type X and the device IDs are X1 and X2. User pod has one init container, lets call it cont-init and one regular container, lets call it cont-reg. Both cont-init and cont-reg requests 2 unit of resource X. cont-init will run some vendor specific code on allocated Devices and then cont-reg will use these 'initiated' devices. I think this is the use case that mentioned. And i feel 2) can handle this. 2) approach will allocate both the devices, X1 and X2, to both the containers. When the containers start, init container, cont-init, will be run first and will run to completion and here devices will get initialized. Strictly after this cont-reg will be started and now this regular container has initialized devices. Where i am wrong?
I guess in RenaudWasTaken 's case, the initialization task is considered to be managed in the plugin, not in init-container, and in init-container we only use the device, so then we might need to call plugin to do recycle action before the device reused by reguler containers. In practical use, both of the two scenarios may exist, it depends on the request of user. Besides, if we don't issue an Allocate gRPC call, we could not ensure a ”complete reuse“: say an init-container requests 2 devices, and we have two regular containers, each requests 1 device, then due to the lack of runtime info re-generation, the devices could only be reused by one of the containers, not both of them.
It is the init containers' job to initialize things and if in some special case device plugin is doing the initialization part, in that case, init containers will not be needed and serve no purpose in pod spec. Specific to example of memory scrubbing, i think this also makes more sense to be done in init container. Once Device's memory is scrubbed, it can be used by regular container. Let try to run your use case with approach 2).


Question: // from the registered device plugins. func (m *ManagerImpl) Allocate(node *schedulercache.NodeInfo, attrs *lifecycle.PodAdmitAttributes) error { pod := attrs.Pod <ins> devicesToReuse := make(map[string]sets.String) </ins> // TODO: Reuse devices between init containers and regular containers. for _, container := range pod.Spec.InitContainers { <del> if err := m.allocateContainerResources(pod, &container); err != nil { </del> <ins> if err := m.allocateContainerResources(pod, &container, devicesToReuse); err != nil { </ins> return err } <ins> m.podDevices.addContainerAllocatedResources(string(pod.UID), container.Name, devicesToReuse) </ins> } for _, container := range pod.Spec.Containers { <del> if err := m.allocateContainerResources(pod, &container); err != nil { </del> <ins> if err := m.allocateContainerResources(pod, &container, devicesToReuse); err != nil { </ins> return err } <ins> m.podDevices.removeContainerAllocatedResources(string(pod.UID), container.Name, devicesToReuse) </ins> } m.mutex.Lock()

There are three containers in pod, cont-init(requested 2 unit or resource X), cont-reg1(requested 1 unit of resource X) and cont-reg2(requested 1 unit of resource X). Now in approach 2), gRPC Allocate call is made for cont-init requesting X1 and X2. Next, gRPC Allocate call is made for cont-reg1. This time resource id will be send in Allocate from the previously allocated [X1, X2]. Lets say X1 is requested for cont-reg1 Next, gRPC Allocate for cont-reg2 and resource with ID X2. All this allocation info gets stored in internal cache. Now when container runs, runtimeOptions will be collected from internal cache. Where do you see a problem in this flow?
/cc
Yep, it could work if the plugin behavior the way you described, but I'm not sure if all kinds of devices could support "pre-allocation", that is, doing allocation on a same device multiple times at the same time, to prepare runtime informations for several containers that reuse the device later. Also I have no idea if all plugin developers consider their plugins "stateless", that is, managing initialization in init-containers. As I don't have enough knowledge on hardware, would like to hear opinions from other guys :)
This would defeat the purpose of a container. In the case of GPUs, when you run a container you expect the GPU to always be in the same state. Furthermore GPUs can't be shared (exclusive usage) between containers meaning that Init containers should behave in the same way as regular containers. With the only difference being that they are executed sequentially and before regular containers. Additionally, operations to setup a device before making them available in a container, whether functionally (load the driver, insert libraries, ....) or security wise (scrub the memory), should be handled by the infrastructure not the users pod spec.
I agree that it's a sane behavior to forbid the use of Device Plugin Resources in Init Containers for the 1.9 release
moving forward,
/cc I think what long-term solution we decide to implement will largely depend on use-case requirements from different device plugins. Would like to hear some thoughts from different device plugin developers on this. Can people cc-ed in this issue take a look at the proposed options mentioned in and provide some feedbacks on which one may better fit your use cases?


Question: // from the registered device plugins. func (m *ManagerImpl) Allocate(node *schedulercache.NodeInfo, attrs *lifecycle.PodAdmitAttributes) error { pod := attrs.Pod <ins> devicesToReuse := make(map[string]sets.String) </ins> // TODO: Reuse devices between init containers and regular containers. for _, container := range pod.Spec.InitContainers { <del> if err := m.allocateContainerResources(pod, &container); err != nil { </del> <ins> if err := m.allocateContainerResources(pod, &container, devicesToReuse); err != nil { </ins> return err } <ins> m.podDevices.addContainerAllocatedResources(string(pod.UID), container.Name, devicesToReuse) </ins> } for _, container := range pod.Spec.Containers { <del> if err := m.allocateContainerResources(pod, &container); err != nil { </del> <ins> if err := m.allocateContainerResources(pod, &container, devicesToReuse); err != nil { </ins> return err } <ins> m.podDevices.removeContainerAllocatedResources(string(pod.UID), container.Name, devicesToReuse) </ins> } m.mutex.Lock()

Answering on behalf of and Solution 1 makes sense for the moment Solution 2 defeats the purpose of the device plugin, which is setting up a device before making it available to a container. solution 4 seems to be a sane solution. As mentioned in my previous comment, as long as devices are exclusive, I expect the devices exposed in my container to always be in the same state at the start of my container. When / If we implement a mechanism for sharing a GPU / devices across containers then the scenario described by is something that we can have. It doesn't however fit the current situation.
Solution number 1 seems totally reasonable short-term. Long term, simplifying and documenting the relationship between plugin callbacks and container lifecycle would help plugin authors.
For reference, I implemented option 2 solution in PR The change itself is small. This is an alternative short-term fix we may consider to get into a 1.9 patch release. I think the main benefit of taking this fix compared to Option 1 solution is that we can change the underlying implementation to support init containers later with less user-facing impact.
No mater we take PR or PR for 1.9, I think we need to have this issue approved for 1.9 milestone. or or can any of you approve this issue for 1.9? Thanks!


Question: importsBelowBase := strings.HasPrefix(imp, i.BaseDir) importsAllowed := false for _, allowed := range i.AllowedImports { <del> importsAllowed = importsAllowed || strings.HasPrefix(imp, allowed) </del> <ins> exactlyImportsAllowed := imp == allowed importsBelowAllowed := strings.HasPrefix(imp, fmt.Sprintf("%s/", allowed)) importsAllowed = importsAllowed || (importsBelowAllowed || exactlyImportsAllowed) </ins> } return importsBelowRoot && !importsBelowBase && !importsAllowed

uses everywhere, directly and via client-go. On the other hand, we have in :
any idea why the import checker does not catch this?
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals


Question: if w.skipIfBlocked { select { case w.destChan <- t: <del> w.fired = true </del> default: } } else { w.destChan <- t <del> w.fired = true </del> } if w.stepInterval > 0 {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: I was writing a variant of clock.FakeClock, and noticed a couple of bugs in . Look at the pointer comparison in --- the two pointers will never be equal, but the author clearly expected equality when is the index of the fakeClockWaiter corresponding to the fakeTimer. The two pointers will never be equal because one is a pointer into the fakeClock's slice while the other pointer is to a field of the . The consequence of this bug is that the Stop function is ineffective. Also, determines the returned value based on the original copy of the waiter, which --- being a copy --- did not get updated if/when the waiter fired. Note also that does not exercise FakeClock::NewTimer at all. What you expected to happen: I expected FakeClock Timers to work properly. How to reproduce it (as minimally and precisely as possible): Use the TestFakeTimer func introduced in Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
The fake timer's Reset() method also does not work correctly.


Question: EmptyDir: &v1.EmptyDirVolumeSource{SizeLimit: &sizeLimit}, }, v1.ResourceRequirements{}), }, <ins> { evictionPriority: 0, // This pod should not be evicted because it uses less than its limit pod:              diskConsumingPod("emptydir-disk-below-sizelimit", useUnderLimit, nil, v1.ResourceRequirements{Limits: containerLimit}), }, </ins> }) }) })

Is this a BUG REPORT or FEATURE REQUEST?: Bug Report /kind bug What happened: improved disk eviction testing to make it more precise. For example, it expects a pod which uses 299 Mb of disk not to be evicted if it sets its limit to 300 Mi. As shown on , this test passes for all OS distributions except CoreOS. I have ruled out an issue with the storage driver and backing filesystem since both COS and CoreOS use overlay backed by extfs. What you expected to happen: I expected the pod using 299Mb not to be evicted when its limit is 300 Mb. I expected not to see this event: How to reproduce it (as minimally and precisely as possible): The container runs this command to use 299Mb: Each iteration writes 1Mb to disk, producing a total of 299 Mb of disk usage. The pod has set to 300Mi. The test has the LocalStorageCapacityIsolation feature gate enabled. Anything else we need to know?: This may be a discrepancy in the (which we use to measure container disk usage) or , which is used by the test to write exact quantities of disk.
/sig node
cc can you take a look at this after thanksgiving?
I bumped the coreos image... going to see if it clears up the flake
It appears the root filesystem is being included in the disk space calculation . Do you think the coreos image might be slightly larger than the other images under test?
One more code snippet that is interesting:
thats the container's root filesystem
correct. the feature appears to add and compare to the quota: The reproducible steps do not take into account the rootfs space. If the quota is set to 300Mi, then the under loop count would have to take into account the rootfs space to not exceed the limit. I do not see anywhere in the tests or the reproduce steps this part of the calculation.
The refers only to the writable layer of the container, and excludes base (image) layers. With the provided command, the only files that are in the container's rootfs are the ones written by the command, and should amount to 299Mb. Additionally, the container does not write any logs (through stdout), so that value should be 0.


Question: EmptyDir: &v1.EmptyDirVolumeSource{SizeLimit: &sizeLimit}, }, v1.ResourceRequirements{}), }, <ins> { evictionPriority: 0, // This pod should not be evicted because it uses less than its limit pod:              diskConsumingPod("emptydir-disk-below-sizelimit", useUnderLimit, nil, v1.ResourceRequirements{Limits: containerLimit}), }, </ins> }) }) })

I ran a quick test on a vm with the current image, and found that both and correctly wrote and measured disk space correctly. I also tested as it inside a container, and verified that we are measuring the container's writable layer correctly. My guess now is that it has something to do with how we calculate disk usage by logging with journald. I am looking into it
Actually, the previous comment was incorrect. When I ran the test, I found that each file was 1028 kb instead of 1024 kb, which resulted in the command writing 300.3 Mb instead of 299 Mb. I will change the test to require slightly less precision. This appears to be because an empty file in coreos takes up 4 kb. Test this by running . On other OS we test on, empty files use 0 kb. This is what accounts for the difference.
I tracked down an issue to the dd call. The stdout and stderr need a redirect to /dev/null to suppress the following messages: Problem Line: Fix:
I PRed that fix and changed it to stderr, since the other tests seem to use stderr.
leaving this open until we get a green run
I'll continue to test locally to see if I find anything else.
opened PR with the help of and CoreOS defaults to using the actual size instead of the apprent size.


Question: } } <ins> func TestPIDPressure(t *testing.T) { testCases := []struct { name                               string podsToMake                         []podToMake evictPodIndex                      int noPressurePIDUsage                 string pressurePIDUsageWithGracePeriod    string pressurePIDUsageWithoutGracePeriod string totalPID                           string }{ { name: "eviction due to pid pressure", podsToMake: []podToMake{ {name: "high-priority-high-usage", priority: highPriority, pidUsage: 900}, {name: "default-priority-low-usage", priority: defaultPriority, pidUsage: 100}, {name: "default-priority-medium-usage", priority: defaultPriority, pidUsage: 400}, {name: "low-priority-high-usage", priority: lowPriority, pidUsage: 600}, {name: "low-priority-low-usage", priority: lowPriority, pidUsage: 50}, }, evictPodIndex:                      3, // we expect the low-priority-high-usage pod to be evicted noPressurePIDUsage:                 "300", pressurePIDUsageWithGracePeriod:    "700", pressurePIDUsageWithoutGracePeriod: "1200", totalPID:                           "2000", }, } for _, tc := range testCases { t.Run(tc.name, func(t *testing.T) { podMaker := makePodWithPIDStats summaryStatsMaker := makePIDStats pods := []*v1.Pod{} podStats := map[*v1.Pod]statsapi.PodStats{} for _, podToMake := range tc.podsToMake { pod, podStat := podMaker(podToMake.name, podToMake.priority, podToMake.pidUsage) pods = append(pods, pod) podStats[pod] = podStat } podToEvict := pods[tc.evictPodIndex] activePodsFunc := func() []*v1.Pod { return pods } fakeClock := testingclock.NewFakeClock(time.Now()) podKiller := &mockPodKiller{} diskInfoProvider := &mockDiskInfoProvider{dedicatedImageFs: ptr.To(false)} diskGC := &mockDiskGC{err: nil} nodeRef := &v1.ObjectReference{Kind: "Node", Name: "test", UID: types.UID("test"), Namespace: ""} config := Config{ MaxPodGracePeriodSeconds: 5, PressureTransitionPeriod: time.Minute * 5, Thresholds: []evictionapi.Threshold{ { Signal:   evictionapi.SignalPIDAvailable, Operator: evictionapi.OpLessThan, Value: evictionapi.ThresholdValue{ Quantity: quantityMustParse("1200"), }, }, { Signal:   evictionapi.SignalPIDAvailable, Operator: evictionapi.OpLessThan, Value: evictionapi.ThresholdValue{ Quantity: quantityMustParse("1500"), }, GracePeriod: time.Minute * 2, }, }, } summaryProvider := &fakeSummaryProvider{result: summaryStatsMaker(tc.totalPID, tc.noPressurePIDUsage, podStats)} manager := &managerImpl{ clock:                        fakeClock, killPodFunc:                  podKiller.killPodNow, imageGC:                      diskGC, containerGC:                  diskGC, config:                       config, recorder:                     &record.FakeRecorder{}, summaryProvider:              summaryProvider, nodeRef:                      nodeRef, nodeConditionsLastObservedAt: nodeConditionsObservedAt{}, thresholdsFirstObservedAt:    thresholdsObservedAt{}, } // create a pod to test admission podToAdmit, _ := podMaker("pod-to-admit", defaultPriority, 50) // synchronize _, err := manager.synchronize(diskInfoProvider, activePodsFunc) if err != nil { t.Fatalf("Manager expects no error but got %v", err) } // we should not have PID pressure if manager.IsUnderPIDPressure() { t.Fatalf("Manager should not report PID pressure") } // try to admit our pod (should succeed) if result := manager.Admit(&lifecycle.PodAdmitAttributes{Pod: podToAdmit}); !result.Admit { t.Fatalf("Admit pod: %v, expected: %v, actual: %v", podToAdmit, true, result.Admit) } // induce soft threshold for PID pressure fakeClock.Step(1 * time.Minute) summaryProvider.result = summaryStatsMaker(tc.totalPID, tc.pressurePIDUsageWithGracePeriod, podStats) _, err = manager.synchronize(diskInfoProvider, activePodsFunc) if err != nil { t.Fatalf("Manager expects no error but got %v", err) } // now, we should have PID pressure if !manager.IsUnderPIDPressure() { t.Errorf("Manager should report PID pressure since soft threshold was met") } // verify no pod was yet killed because there has not yet been enough time passed if podKiller.pod != nil { t.Errorf("Manager should not have killed a pod yet, but killed: %v", podKiller.pod.Name) } // step forward in time past the grace period fakeClock.Step(3 * time.Minute) // no change in PID stats to simulate continued pressure _, err = manager.synchronize(diskInfoProvider, activePodsFunc) if err != nil { t.Fatalf("Manager expects no error but got %v", err) } // verify PID pressure is still reported if !manager.IsUnderPIDPressure() { t.Errorf("Manager should still report PID pressure") } // verify the right pod was killed with the right grace period. if podKiller.pod != podToEvict { t.Errorf("Manager chose to kill pod: %v, but should have chosen %v", podKiller.pod.Name, podToEvict.Name) } if podKiller.gracePeriodOverride == nil { t.Errorf("Manager chose to kill pod but should have had a grace period override.") } observedGracePeriod := *podKiller.gracePeriodOverride if observedGracePeriod != manager.config.MaxPodGracePeriodSeconds { t.Errorf("Manager chose to kill pod with incorrect grace period.  Expected: %d, actual: %d", manager.config.MaxPodGracePeriodSeconds, observedGracePeriod) } // reset state podKiller.pod = nil podKiller.gracePeriodOverride = nil // remove PID pressure by simulating increased PID availability fakeClock.Step(20 * time.Minute) summaryProvider.result = summaryStatsMaker(tc.totalPID, tc.noPressurePIDUsage, podStats) // Simulate increased PID availability _, err = manager.synchronize(diskInfoProvider, activePodsFunc) if err != nil { t.Fatalf("Manager expects no error but got %v", err) } // verify PID pressure is resolved if manager.IsUnderPIDPressure() { t.Errorf("Manager should not report PID pressure") } // re-induce PID pressure fakeClock.Step(1 * time.Minute) summaryProvider.result = summaryStatsMaker(tc.totalPID, tc.pressurePIDUsageWithoutGracePeriod, podStats) _, err = manager.synchronize(diskInfoProvider, activePodsFunc) if err != nil { t.Fatalf("Manager expects no error but got %v", err) } // verify PID pressure is reported again if !manager.IsUnderPIDPressure() { t.Errorf("Manager should report PID pressure") } // verify the right pod was killed with the right grace period. if podKiller.pod != podToEvict { t.Errorf("Manager chose to kill pod: %v, but should have chosen %v", podKiller.pod.Name, podToEvict.Name) } if podKiller.gracePeriodOverride == nil { t.Errorf("Manager chose to kill pod but should have had a grace period override.") } observedGracePeriod = *podKiller.gracePeriodOverride if observedGracePeriod != int64(0) { t.Errorf("Manager chose to kill pod with incorrect grace period.  Expected: %d, actual: %d", 0, observedGracePeriod) } // try to admit our pod (should fail) if result := manager.Admit(&lifecycle.PodAdmitAttributes{Pod: podToAdmit}); result.Admit { t.Fatalf("Admit pod: %v, expected: %v, actual: %v", podToAdmit, false, result.Admit) } // reduce PID pressure fakeClock.Step(1 * time.Minute) summaryProvider.result = summaryStatsMaker(tc.totalPID, tc.noPressurePIDUsage, podStats) podKiller.pod = nil // reset state _, err = manager.synchronize(diskInfoProvider, activePodsFunc) if err != nil { t.Fatalf("Manager expects no error but got %v", err) } // we should have PID pressure (because transition period not yet met) if !manager.IsUnderPIDPressure() { t.Errorf("Manager should report PID pressure") } // no pod should have been killed if podKiller.pod != nil { t.Errorf("Manager chose to kill pod: %v when no pod should have been killed", podKiller.pod.Name) } // try to admit our pod (should fail) if result := manager.Admit(&lifecycle.PodAdmitAttributes{Pod: podToAdmit}); result.Admit { t.Fatalf("Admit pod: %v, expected: %v, actual: %v", podToAdmit, false, result.Admit) } // move the clock past the transition period fakeClock.Step(5 * time.Minute) summaryProvider.result = summaryStatsMaker(tc.totalPID, tc.noPressurePIDUsage, podStats) _, err = manager.synchronize(diskInfoProvider, activePodsFunc) if err != nil { t.Fatalf("Manager expects no error but got %v", err) } // we should not have PID pressure (because transition period met) if manager.IsUnderPIDPressure() { t.Errorf("Manager should not report PID pressure") } // no pod should have been killed if podKiller.pod != nil { t.Errorf("Manager chose to kill pod: %v when no pod should have been killed", podKiller.pod.Name) } // try to admit our pod (should succeed) if result := manager.Admit(&lifecycle.PodAdmitAttributes{Pod: podToAdmit}); !result.Admit { t.Fatalf("Admit pod: %v, expected: %v, actual: %v", podToAdmit, true, result.Admit) } }) } } </ins> func TestAdmitUnderNodeConditions(t *testing.T) { manager := &managerImpl{} pods := []*v1.Pod{

We should add a unit test to Eviction manager that accounts for PID Pressure. The test should cover similar cases as DiskPressure or MemoryPressure. 1) PIDPressure doesn't exist on node. 2) Force PidPressure on node We should include test cases for reclaming resources in terms of pressure and verifying differences between soft and hard eviction.
/sig node /kind feature /help
This request has been marked as needing help from a contributor. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /sig node /kind feature /help </details
/triage accepted /priority important-longterm
/good-first-issue
This request has been marked as suitable for new contributors. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
it looks like has requested to work on this.
/assign
Hi What do you mean by, "verifying differences between soft and hard eviction."?
It’s the same idea as what we do for disk, memory or inode pressure.