Question: // Known values for the tag. const ( <del> tagValueTrue  = "true" tagValueFalse = "false" </del> <ins> tagValueTrue       = "true" tagValueFalse      = "false" tagExtensionPrefix = "x-kubernetes-" </ins> ) <ins> func getOpenAPITagValue(comments []string) []string { return types.ExtractCommentTags("+", comments)[tagName] } </ins> func hasOpenAPITagValue(comments []string, value string) bool { <del> tagValues := types.ExtractCommentTags("+", comments)[tagName] if tagValues == nil { return false } </del> <ins> tagValues := getOpenAPITagValue(comments) </ins> for _, val := range tagValues { if val == value { return true

My intend here was avoiding repeated code for adding each of these tags somewhere (in openapi-gen code or some root level place). Let me distinguish these tags in two categories with respect to OpenAPI code generator: that need some sort of processing that affects OpenAPI spec (e.g. +optional removes the field from OpenAPI field) that need no more processing and will be directly put into OpenAPI spec (e.g. +gvk that we don't have but we can define as a way to define Group/Version/Kind of an object vs register method) My argument is about the second one. I think patch strategy here falls into this category. The question is only about naming it. Naming Option 1: +patch-strategy=... This looks cleaner but we need to define an alias somewhere to tell generator this is the same as x-kubernetes-patch-strategy. The alias definition and the tags are in two places. Putting them in the generator code make the code non-generic and we don't want to do that. Putting these aliases somewhere as an argument to the generator begs for bugs as people that look at the type and add this tag to their types need to know (maybe by reading a doc, but who do that) that it needs to pass the alias to the generator. I, in 6 month, will look at a type and see a tag and copy the tag to my type and expect it to work that will not with this option. Naming Option 2: use something generic for these type of tags. after all generator does not need to do any processing. I suggest +openapi-tag:tag=value that will result in "x-kubernetes-tag: value" in OpenAPI spec. There is no alias defined anywhere and without any code change or alias copying or any knowledge about passing any argument, a tag can be to OpenAPI spec. The value of these tags and their use cases are outside of the scope of what I am arguing about. Anyway a solution is better than no solution, I would suggest we continue with Option 2 as Alpha and change it later if we see fit. Adding an aliasing mechanism to this option later should be easy.
On patch strategy, I had assumed we'd read the value from what's in the struct tag, rather than requiring a net new tag?


Question: // Known values for the tag. const ( <del> tagValueTrue  = "true" tagValueFalse = "false" </del> <ins> tagValueTrue       = "true" tagValueFalse      = "false" tagExtensionPrefix = "x-kubernetes-" </ins> ) <ins> func getOpenAPITagValue(comments []string) []string { return types.ExtractCommentTags("+", comments)[tagName] } </ins> func hasOpenAPITagValue(comments []string, value string) bool { <del> tagValues := types.ExtractCommentTags("+", comments)[tagName] if tagValues == nil { return false } </del> <ins> tagValues := getOpenAPITagValue(comments) </ins> for _, val := range tagValues { if val == value { return true

, Mehdy Bohlool <:
The get stuff is less urgent for me as long as we aren't doing even more hacky things to work around the current limitations, and can make a commitment to delivering it in a specific release (sounds like 1.7). For service catalog there was a plan to fork kubectl, compile the service catalog types into the binary, and provide the fork as a separate binary. I have some concerns with this approach. I would rather provide a temporary hack in the code that can provide a consistent user experience from day one and re-implement later vs having users get used to downloading separate binaries for each aggregated apiserver. I have no plans to do any work for on this for the core Kubernetes types, and can punt on it until the scope of the work is understood to be relatively small and easily upgraded to the server-side get.


Question: if !g.registeredMaster { return ControllerManagerMetrics{}, fmt.Errorf("Master's Kubelet is not registered. Skipping ControllerManager's metrics gathering") } <del> output, err := g.getMetricsFromPod(g.client, fmt.Sprintf("%v-%v", "kube-controller-manager", g.masterName), metav1.NamespaceSystem, ports.InsecureKubeControllerManagerPort) </del> <ins> podName := fmt.Sprintf("%v-%v", "kube-controller-manager", g.masterName) g.waitForControllerManagerReadyOnce.Do(func() { e2epod.WaitForPodNameRunningInNamespace(g.client, podName, metav1.NamespaceSystem) }) output, err := g.getMetricsFromPod(g.client, podName, metav1.NamespaceSystem, ports.InsecureKubeControllerManagerPort) </ins> if err != nil { return ControllerManagerMetrics{}, err }

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: pull-kubernetes-e2e-gce ci-kubernetes-e2e-gci-gce Which test(s) are flaking: [sig-storage] In-tree Volumes [Driver: hostPath] [Testpattern: Inline-volume (default fs)] subPath should support readOnly file specified in the volumeMount [LinuxOnly] [sig-storage] In-tree Volumes [Driver: emptydir] [Testpattern: Inline-volume (default fs)] subPath should support file as subpath [LinuxOnly] [sig-storage] In-tree Volumes [Driver: local][LocalVolumeType: dir-link] [Testpattern: Pre-provisioned PV (default fs)] volumes should allow exec of files on the volume [sig-storage] In-tree Volumes [Driver: local][LocalVolumeType: blockfs] [Testpattern: Pre-provisioned PV (default fs)] subPath should support existing single file [LinuxOnly] [sig-storage] In-tree Volumes [Driver: local][LocalVolumeType: dir-bindmounted] [Testpattern: Pre-provisioned PV (default fs)] volumes should store data [sig-storage] In-tree Volumes [Driver: gcepd] [Testpattern: Inline-volume (default fs)] volumes should store data [sig-storage] In-tree Volumes [Driver: gluster] [Testpattern: Inline-volume (default fs)] subPath should be able to unmount after the subpath directory is deleted [sig-storage] In-tree Volumes [Driver: local][LocalVolumeType: tmpfs] [Testpattern: Pre-provisioned PV (default fs)] subPath should be able to unmount after the subpath directory is deleted [sig-storage] In-tree Volumes [Driver: gcepd] [Testpattern: Dynamic PV (default fs)] subPath should support existing directory [sig-storage] In-tree Volumes [Driver: gluster] [Testpattern: Inline-volume (default fs)] subPath should support non-existent path [sig-storage] In-tree Volumes [Driver: hostPathSymlink] [Testpattern: Inline-volume (default fs)] subPath should support existing directories when readOnly specified in the volumeSource [sig-storage] In-tree Volumes [Driver: hostPath] [Testpattern: Inline-volume (default fs)]


Question: if !g.registeredMaster { return ControllerManagerMetrics{}, fmt.Errorf("Master's Kubelet is not registered. Skipping ControllerManager's metrics gathering") } <del> output, err := g.getMetricsFromPod(g.client, fmt.Sprintf("%v-%v", "kube-controller-manager", g.masterName), metav1.NamespaceSystem, ports.InsecureKubeControllerManagerPort) </del> <ins> podName := fmt.Sprintf("%v-%v", "kube-controller-manager", g.masterName) g.waitForControllerManagerReadyOnce.Do(func() { e2epod.WaitForPodNameRunningInNamespace(g.client, podName, metav1.NamespaceSystem) }) output, err := g.getMetricsFromPod(g.client, podName, metav1.NamespaceSystem, ports.InsecureKubeControllerManagerPort) </ins> if err != nil { return ControllerManagerMetrics{}, err }

subPath should be able to unmount after the subpath directory is deleted [sig-storage] In-tree Volumes [Driver: local][LocalVolumeType: tmpfs] [Testpattern: Pre-provisioned PV (default fs)] subPath should support existing directories when readOnly specified in the volumeSource [sig-storage] In-tree Volumes [Driver: local][LocalVolumeType: dir-link-bindmounted] [Testpattern: Pre-provisioned PV (default fs)] subPath should support existing single file [LinuxOnly] [sig-storage] In-tree Volumes [Driver: local][LocalVolumeType: block] [Testpattern: Pre-provisioned PV (ext3)] volumes should allow exec of files on the volume [sig-storage] In-tree Volumes [Driver: gcepd] [Testpattern: Dynamic PV (default fs)(allowExpansion)] volume-expand Verify if offline PVC expansion works [sig-storage] In-tree Volumes [Driver: gcepd] [Testpattern: Dynamic PV (ext3)] volumes should store data Testgrid link: Reason for failure: Failure fetching metrics: Anything else we need to know: This appears to have restarted abruptly around 2/11-2/12 /sig storage /priority important-soon
xref I think we have a general failure to start containers reliably. Almost every instance of these test failures contained the pod status error of "OCI runtime start failed: container process is already dead: unknown"
after sweeping a bunch of the in-tree test failures, almost all seemed to have as a root cause, triggered by very short-lived containers used to check file content or permissions. The only test I saw failing that didn't seem to fit that was: [sig-storage] In-tree Volumes [Driver: nfs] [Testpattern: Dynamic PV (default fs)] subPath should support non-existent path [sig-storage] In-tree Volumes [Driver: nfs] [Testpattern: Dynamic PV (default fs)] subPath should support existing directory examples: I'd recommend repurposing this issue to address that specifically and leave the OCI issue for the short-lived container issue
Here's a PR for context: So, I think this PR is another example of flakes. Here are some links to previous runs: and same message: This prow: has the above message and a few more, all A lot of in-tree volume issues, but there were some persistent volume ones too.


Question: if !g.registeredMaster { return ControllerManagerMetrics{}, fmt.Errorf("Master's Kubelet is not registered. Skipping ControllerManager's metrics gathering") } <del> output, err := g.getMetricsFromPod(g.client, fmt.Sprintf("%v-%v", "kube-controller-manager", g.masterName), metav1.NamespaceSystem, ports.InsecureKubeControllerManagerPort) </del> <ins> podName := fmt.Sprintf("%v-%v", "kube-controller-manager", g.masterName) g.waitForControllerManagerReadyOnce.Do(func() { e2epod.WaitForPodNameRunningInNamespace(g.client, podName, metav1.NamespaceSystem) }) output, err := g.getMetricsFromPod(g.client, podName, metav1.NamespaceSystem, ports.InsecureKubeControllerManagerPort) </ins> if err != nil { return ControllerManagerMetrics{}, err }

But, that showed up once so far from what I've seen.
The "error getting c-m metrics" failures appear to have just restarted:
/priority critical-urgent bumping priority since we're seeing significant numbers of failures that just started in the last two days
this is failing many test runs per day, do you have updated status on this?
I haven't had chance to fully debug this yet. looked into this little bit and he suspected that these tests started to flake same time around was merged. I should be able to take a closer look tomorrow.
This occurrence fails because it's trying to get the metrics from the kube-controller-manager pod but it is not ready until one minute later if we grep in the kube-apiserver for the controller pod name we can observe that after the pod is up it starts to get 200 instead of 404
keeping open until shows this is resolved
seen again in
this time the container is running but still it can't find it
the errors comes from the request to the apiserver on How can we check that the pod is found and that the apiserver found it?
the first successful get of /metrics logged by the controller manager in that run is at: We're also not checking the error in the once.Do to make sure the pod was running, and once running, we should probably then also check the metrics path on the pod. Maybe something like this:
still happening: message logging missed using the correct errors, so the messages aren't helpful:
what's this stack trace in the apiserver? is this related? a 503 shouldn't cause that, right?
again it seems is not really waiting because it fails just 200 ms after the INFO msg, I think is because it checks if the pod is running, but the message is that the pod doesn't really exist. The fact that there are 9 tests failing with the same error and we are using has to be related to the way Ginkgo use parallel.
It turns out that the method used to wait for the controller manager pod fails if the pod doesn't exist
Ah, so first we should wait for the pod to exist, then be ready, then return metrics
This start to look promising
yes, looks good. thanks for all the work /close
Closing this issue. <detailsIn response to : yes, looks good.


Question: if !g.registeredMaster { return ControllerManagerMetrics{}, fmt.Errorf("Master's Kubelet is not registered. Skipping ControllerManager's metrics gathering") } <del> output, err := g.getMetricsFromPod(g.client, fmt.Sprintf("%v-%v", "kube-controller-manager", g.masterName), metav1.NamespaceSystem, ports.InsecureKubeControllerManagerPort) </del> <ins> podName := fmt.Sprintf("%v-%v", "kube-controller-manager", g.masterName) g.waitForControllerManagerReadyOnce.Do(func() { e2epod.WaitForPodNameRunningInNamespace(g.client, podName, metav1.NamespaceSystem) }) output, err := g.getMetricsFromPod(g.client, podName, metav1.NamespaceSystem, ports.InsecureKubeControllerManagerPort) </ins> if err != nil { return ControllerManagerMetrics{}, err }

thanks for all the work Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Fixed by


Question: if err := sst.Scale(&ss, 0); err != nil { errList = append(errList, fmt.Sprintf("%v", err)) } <del> sst.waitForStatus(&ss, 0) </del> <ins> sst.WaitForStatus(&ss, 0) </ins> Logf("Deleting statefulset %v", ss.Name) if err := c.Apps().StatefulSets(ss.Namespace).Delete(ss.Name, nil); err != nil { errList = append(errList, fmt.Sprintf("%v", err))

https://k8s- do you know the statefulset owners?
cc
I did not see this happening until recently.
The controller misses that the first pod transitioned to unready and proceeds to create the second pod. The test uses a client to hit the api server and will scale up the statefulset as soon as the broken pod is observed but the controller uses caches so it may observe the scaled up statefulset quicker than the updated pod.
I will update the test to ensure that we wait until the unreadiness is observed on StatefulSet.Status.Replicas field prior to the scaling operation
SGTM


Question: func TestHexCIDR(t *testing.T) { tests := []struct { <ins> name      string </ins> input     string output    string expectErr bool }{ { <del> input:  "1.2.0.0/16", </del> <ins> name:   "IPv4 masked", input:  "1.2.3.4/16", </ins> output: "01020000/ffff0000", }, { <ins> name:   "IPv4 host", </ins> input:  "172.17.0.2/32", output: "ac110002/ffffffff", }, { <ins> name:   "IPv6 masked", input:  "2001:dead:beef::cafe/64", output: "2001deadbeef00000000000000000000/ffffffffffffffff0000000000000000", }, { name:   "IPv6 host", input:  "2001::5/128", output: "20010000000000000000000000000005/ffffffffffffffffffffffffffffffff", }, { name:      "invalid CIDR", </ins> input:     "foo", expectErr: true, },

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): No What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): hexCIDR and CIDR Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): latest on master Environment: Cloud provider or hardware configuration: Bare-metal, 3 node config using KubeAdm OS (e.g. from /etc/os-release): CentOS 7 Kernel (e.g. ): Linux devstack-77 3.10.0-514.10.2.el7.x8664 SMP Fri Mar 3 00:04:05 UTC 2017 x8664 x8664 x8664 GNU/Linux Install tools: KubeAdm Others: What happened: Investigating the code, hexCIDR() expects the CIDR passed to be for an IPv4 address and doesn't support an IPv6 CIDR. What you expected to happen: Support either IPv4 or IPv6 CIDRs. How to reproduce it (as minimally and precisely as possible): Anything else we need to know: Will be working on a solution for this. Note: In some of the call paths, a PodIP string is used and passed in as a CIDR with a "/32" appended. At some point, this would need to be updated as well, to use "/128" if an IPv6 PodIP was used.
I see there is an asciiCIDR() method too that needs addressing, but only in the UTs.


Question: func TestHexCIDR(t *testing.T) { tests := []struct { <ins> name      string </ins> input     string output    string expectErr bool }{ { <del> input:  "1.2.0.0/16", </del> <ins> name:   "IPv4 masked", input:  "1.2.3.4/16", </ins> output: "01020000/ffff0000", }, { <ins> name:   "IPv4 host", </ins> input:  "172.17.0.2/32", output: "ac110002/ffffffff", }, { <ins> name:   "IPv6 masked", input:  "2001:dead:beef::cafe/64", output: "2001deadbeef00000000000000000000/ffffffffffffffff0000000000000000", }, { name:   "IPv6 host", input:  "2001::5/128", output: "20010000000000000000000000000005/ffffffffffffffffffffffffffffffff", }, { name:      "invalid CIDR", </ins> input:     "foo", expectErr: true, },

Appears that it is expecting to reverse hex back to string, but test is not showing case of mask being properly applied. For example, "2001::5/64" when converted to hex and then back to string will be "2001::/64". Will update test to be more accurate.
/area networking
These labels do not exist in this repository: . <detailsIn response to : /area networking Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. I understand the commands that are listed . </details
/sig network
One question I have... Can the callers (bandwidth?) of the function work with an IPv6 based hex CIDR? In the future, if we support pod IPs that are IPv6 addresses, and we modify the code to add /128 vs /32, as is done currently, this would create an IPv6 based hex CIDR.
The only caller is currently the ingress/egress shaping function in kubelet. That code also needs to be updated for IPv6. For example, assumes IPv4 when adding CIDRs to the shaper list.
Agree.I wanted to separate this out, partly to make the change smaller, but also, when I looked at the callers (this shaping functionality) I wasn't sure if it would support IPv6. I looked a bit at the "tc" command and saw some notes that indicate that it doesn't support IPv6 yet. Is that still accurate? Would it be ok to tackle that separately?


Question: Spec:       v1.NodeSpec{}, Status: v1.NodeStatus{ Capacity: v1.ResourceList{ <del> v1.ResourceCPU:     *resource.NewMilliQuantity(2000, resource.DecimalSI), v1.ResourceMemory:  *resource.NewQuantity(10E9, resource.BinarySI), v1.ResourceStorage: *resource.NewQuantity(500*mb, resource.BinarySI), v1.ResourcePods:    *resource.NewQuantity(0, resource.DecimalSI), </del> <ins> v1.ResourceCPU:    *resource.NewMilliQuantity(2000, resource.DecimalSI), v1.ResourceMemory: *resource.NewQuantity(10E9, resource.BinarySI), v1.ResourcePods:   *resource.NewQuantity(0, resource.DecimalSI), </ins> }, Allocatable: v1.ResourceList{ <del> v1.ResourceCPU:     *resource.NewMilliQuantity(0, resource.DecimalSI), v1.ResourceMemory:  *resource.NewQuantity(10E9, resource.BinarySI), v1.ResourceStorage: *resource.NewQuantity(500*mb, resource.BinarySI), v1.ResourcePods:    *resource.NewQuantity(0, resource.DecimalSI), </del> <ins> v1.ResourceCPU:    *resource.NewMilliQuantity(0, resource.DecimalSI), v1.ResourceMemory: *resource.NewQuantity(10E9, resource.BinarySI), v1.ResourcePods:   *resource.NewQuantity(0, resource.DecimalSI), </ins> }, }, }

Local storage is not expected to be available as a scheduling resource in v1.7. Kubelet is incorrectly exposing a resource named . This is incorrect because the feature is alpha and so it requires opt-in via feature flags and, the resource names are invalid. The expected resource names are listed - and . cc This is v1.7 blocker because we cannot expose a new resource in the API and rename it in a subsequent release.
CC I'll find someone on the storage team to address this.
/assign
GitHub didn't allow me to assign the following users: verult. Note that only can be assigned. <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: fsInfo, err := p.cadvisor.GetDirFsInfo(mountpoint) if err != nil { msg := "Failed to get the info of the filesystem with mountpoint" <del> if err == cadvisorfs.ErrNoSuchDevice { </del> <ins> if errors.Is(err, cadvisorfs.ErrNoSuchDevice) || errors.Is(err, cadvisorfs.ErrDeviceNotInPartitionsMap) || errors.Is(err, cadvisormemory.ErrDataNotFound) { </ins> klog.V(2).InfoS(msg, "mountpoint", mountpoint, "err", err) } else { klog.ErrorS(err, msg, "mountpoint", mountpoint)

What happened: New overlay filesystems created for Pods after kubelet start trigger repeated error messages. For example: Error messages are logged every 10 seconds for every filesystem created after kubelet start. For large kubelet nodes, this issue creates an excessive amount of error logs. What you expected to happen: New overlay filesystems created for Pods after kubelet start should not log error messages. How to reproduce it (as minimally and precisely as possible): Use cristatsprovider with cri-o and configure overlay storage. UsingLegacyCadvisorStats() needs to return false per for cristatsprovider to be used. Start with an empty /var/lib/containers/storage and schedule Pods on the node. Anything else we need to know?: As part of , cadvisor.GetFsInfoByFsUUID() was changed to cadvisor.GetDirFsInfo() in getFsInfo(); however, the test for cadvisorfs.ErrNoSuchDevice was not modified, changing how getFsInfo() handles a cache miss from cadvisor. cadvisor caches UUIDs/partitions/mounts on kubelet start and the cache is never refreshed. Given that it's expected that getFsInfo() will fail to retrieve data from cadvisor (see comments in ImageFsStats() and ), this condition should not be logged as an error. Alternatively, cadvisor fs could be modified to refresh on cache miss enabling getFsInfo() to return data on these filesystems. Environment: Kubernetes version (use ): 1.18.6 CRI: CRI-O
/sig node cc
ptal
thanks for reporting the issue. I will take a look at in upcoming week.
/assign
Sorry this is taking slightly longer. Currently held up in something little urgent, but as soon as that's over I will look into this one.
Do you have the time to look at this issue ? I can help with testing / debugging if required. Environment Kubernetes 1.19.3 Cri-o 1.19.0
Sorry I haven't gotten a chance to look into it yet, I would certainly appreciate any help in debugging. Thanks.
Right now I am thinking of either handling the cache miss better, or returning the even if the fs type is not in cadvisor, or both.
Issues go stale after 90d of inactivity. Mark the issue as fresh with .


Question: fsInfo, err := p.cadvisor.GetDirFsInfo(mountpoint) if err != nil { msg := "Failed to get the info of the filesystem with mountpoint" <del> if err == cadvisorfs.ErrNoSuchDevice { </del> <ins> if errors.Is(err, cadvisorfs.ErrNoSuchDevice) || errors.Is(err, cadvisorfs.ErrDeviceNotInPartitionsMap) || errors.Is(err, cadvisormemory.ErrDataNotFound) { </ins> klog.V(2).InfoS(msg, "mountpoint", mountpoint, "err", err) } else { klog.ErrorS(err, msg, "mountpoint", mountpoint)

Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle rotten
kindly ping - has this already been implemented ?
/assign
Fix for k/k in and cadvisor in
/remove-lifecycle rotten
hi do you know when the fixes will be released ? Thanks!
I don't, unfortunately.
the fix has made it into , so I imagine the change will come in by 1.22
/triage accepted /priority important-soon
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
So v1.22 is out for a while now. Is there someone who has already upgraded to v1.22 and can confirm that the getFsInfo errors are fixed?
haircommander mentioned that the fix made it into cadvisor 0.40.0-rc.1. But cadvisor version bump did not happen in k8s yet, probably due to issues mentioned here: iirc kind did not work with image kindest/node v1.22.1, I did not look into logs however. Will update post once I have access to the affected device. Edit:
I still have these logs with version 1.22.2
I have got the same error log about kubelet kubernetes version 1.22.2 containerd: 1.4.11 Is this problem have been resolve?


Question: fsInfo, err := p.cadvisor.GetDirFsInfo(mountpoint) if err != nil { msg := "Failed to get the info of the filesystem with mountpoint" <del> if err == cadvisorfs.ErrNoSuchDevice { </del> <ins> if errors.Is(err, cadvisorfs.ErrNoSuchDevice) || errors.Is(err, cadvisorfs.ErrDeviceNotInPartitionsMap) || errors.Is(err, cadvisormemory.ErrDataNotFound) { </ins> klog.V(2).InfoS(msg, "mountpoint", mountpoint, "err", err) } else { klog.ErrorS(err, msg, "mountpoint", mountpoint)

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
I still have these logs with version 1.23.1
turns out we ended up being blocked on , so the fix has yet to be merged. let's push for it in 1.24
I'll wait for 1.24. With 1.22.5 My cluster has some of these messages but is stable. But with 1.23.x I get disk pressure because my sys logs get spammed quickly. I hoped going away fomr cadvisor to CRI stats woudl help, but instead of stat errors I now have these FS errors.
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository.


Question: fsInfo, err := p.cadvisor.GetDirFsInfo(mountpoint) if err != nil { msg := "Failed to get the info of the filesystem with mountpoint" <del> if err == cadvisorfs.ErrNoSuchDevice { </del> <ins> if errors.Is(err, cadvisorfs.ErrNoSuchDevice) || errors.Is(err, cadvisorfs.ErrDeviceNotInPartitionsMap) || errors.Is(err, cadvisormemory.ErrDataNotFound) { </ins> klog.V(2).InfoS(msg, "mountpoint", mountpoint, "err", err) } else { klog.ErrorS(err, msg, "mountpoint", mountpoint)

</details
This is still relevant and requires a fix with
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale Related PR still not merged.
SnoozeThis is useful if we don't want to keep polling until that PR is closed.
() I will wait until is merged and then add a comment.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
Hi folks, I still have this issue in most recent Kubernetes 1.24.6. cAdvisor 0.44.0 that fix went into that version, is now a long time ago already in 1.24.0. Dep bump to runc 1.1.0, cadvisor 0.44.0 Automated cherry pick of : Bump cAdvisor to v0.44.1 28.4.2022 So is there possibly any chance to progress with a review of this PR please?
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.


Question: fsInfo, err := p.cadvisor.GetDirFsInfo(mountpoint) if err != nil { msg := "Failed to get the info of the filesystem with mountpoint" <del> if err == cadvisorfs.ErrNoSuchDevice { </del> <ins> if errors.Is(err, cadvisorfs.ErrNoSuchDevice) || errors.Is(err, cadvisorfs.ErrDeviceNotInPartitionsMap) || errors.Is(err, cadvisormemory.ErrDataNotFound) { </ins> klog.V(2).InfoS(msg, "mountpoint", mountpoint, "err", err) } else { klog.ErrorS(err, msg, "mountpoint", mountpoint)

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
/remove-lifecycle stale
Can we have any news on that topic please?
/remove-lifecycle rotten
This issue is labeled with but has not been updated in over 90 days, and should be re-triaged. Important-soon issues must be staffed and worked on either currently, or very soon, ideally in time for the next release. You can: Confirm that this issue is still relevant with (org members only) Deprioritize it with or Close this issue with For more details on the triage process, see /remove-triage accepted
Resolved.


Question: return err } <del> proxyBytes, err := kubeadmutil.MarshalToYamlForCodecsWithShift(cfg.KubeProxy.Config, kubeproxyconfigv1alpha1.SchemeGroupVersion, </del> <ins> proxyBytes, err := kubeadmutil.MarshalToYamlForCodecs(cfg.KubeProxy.Config, kubeproxyconfigv1alpha1.SchemeGroupVersion, </ins> kubeproxyconfigscheme.Codecs) if err != nil { return fmt.Errorf("error when marshaling: %v", err) } <ins> var prefixBytes bytes.Buffer apiclient.PrintBytesWithLinePrefix(&prefixBytes, proxyBytes, "    ") </ins> var proxyConfigMapBytes, proxyDaemonSetBytes []byte proxyConfigMapBytes, err = kubeadmutil.ParseTemplate(KubeProxyConfigMap19, struct {

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: MarshalToYamlForCodecs adds an indent to the yaml output. This code can be combined with PrintBytesWithLinePrefix What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/assign rpothier
/sig cluster-lifecycle
/priority important-soon


Question: "k8s.io/utils/pointer" ) <ins> func init() { utilruntime.Must(metricsfeatures.AddFeatureGates(utilfeature.DefaultMutableFeatureGate)) } </ins> // proxyRun defines the interface to run a specified ProxyServer type proxyRun interface { Run() error

As per, , wire up metrics/slis for kube-proxy
/assign
/sig network /sig instrumentation
This sounds OK, but we need to translate it into real work-to-do - volunteers? /triage accepted
is tackling this one :)


Question: "", }, { <del> "a test case with valid tags [LinuxOnly] [NodeConformance] [Serial]", </del> <ins> "a test case with valid tags [LinuxOnly] [NodeConformance] [Serial] [Disruptive]", </ins> "", }, {

What would you like to be : Our tooling that makes the list of conformance tests needs to allow conformance tests to be marked as disruptive. Why is this needed: To test default, conformance behavior which may be disruptive (e.g. taint eviction) ref /sig testing /area conformance


Question: "", }, { <del> "a test case with valid tags [LinuxOnly] [NodeConformance] [Serial]", </del> <ins> "a test case with valid tags [LinuxOnly] [NodeConformance] [Serial] [Disruptive]", </ins> "", }, {

What happened: Ran a conformance test run (using Sonobuoy) on One of the taint based tests a NoExecute taint to a node which evicted all workloads including the Sonobuoy workload itself. If this had been a prod workload, numerous prod workloads could have been impacted. What you expected to happen: Test run as normal with reasonable results. Able to run conformance tests with minimal impact on a running cluster. How to reproduce it (as minimally and precisely as possible): on a v1.16.0 cluster. Anything else we need to know?: Environment: Kubernetes version (use ): /area conformance
/sig testing /sig architecture /sig scheduling
Summarizing the many conversations on the PR and in Slack regarding this: We wish we had identified the lack of a tag sooner and could have taken better action. Since the release is 2 days away and our own tooling currently prohibits us from marking tests as both and , we've decided to let the release proceed with this "bug". Risk: Someone running the conformance tests may unexpectedly have their workloads evicted Mitigation: Sonobuoy v0.16.0 will provide the capabilities to the user to run a "conformance-lite" (name TBD) run if they'd like to run the conformance tests against a cluster with prod workloads. This way end-users can still be reasonably sure of their cluster's continued well-being. This mode will manually skip the tests in question until they can be more appropriately marked as Sonobuoy will also provide, consistent with past behavior, the means to run the full conformance suite including these disruptive tests. Eventual solution: Sonobuoy will, most likely, need to continue offering a "non-disruptive-conformance" mode to aid in end user testing but the regexp will change and should be able to just skip tests K8s tooling needs to be tweaked to allow for tests to be marked as The tag should be to these tests.


Question: api "k8s.io/kubernetes/pkg/apis/core" "k8s.io/kubernetes/pkg/features" "k8s.io/kubernetes/pkg/kubelet/client" <ins> utilpointer "k8s.io/utils/pointer" </ins> // ensure types are installed _ "k8s.io/kubernetes/pkg/apis/core/install"

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Some k8s client tried to evict a pod by setting negative value for deletionGracePeriodSeconds. But all eventual delete calls (and force-delete calls) made by kubelet, pod-garbage-collector, etc were not able to delete that pod, because apiserver doesnt allow deletionGracePeriodSeconds to be increased once it is set. So force-delete (which set it to 0) doesn't take effect when it's already negative. Not even kubectl is able to call delete with negative grace-period. But curl with an even more negative grace-period value worked. What you expected to happen: force-delete works. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: I think the issue is in the code below As the comment suggest, in phase 1, the was set to a negative value (meaning was in the past), and in phase 2, has to be set even lower, otherwise, it will be considered as pending graceful deletion and not performing the immediate delete. I think there're two possible fixes. not allowing setting in the past ( should be non-negative). In that case, we could convert to 0 if negative. that could be set in the past. In that case, in phase 2, we treat negative value of the same as 0, and allow immediate delete. Thoughts? Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig api-machinery
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig api-machinery Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/cc /triage accepted
No component in the system should ever treat the timestamp of DeletionTimestamp as authoritative.


Question: api "k8s.io/kubernetes/pkg/apis/core" "k8s.io/kubernetes/pkg/features" "k8s.io/kubernetes/pkg/kubelet/client" <ins> utilpointer "k8s.io/utils/pointer" </ins> // ensure types are installed _ "k8s.io/kubernetes/pkg/apis/core/install"

In Kube, components are required to calculate the time period from when they observe the event in their local clock (i.e. 30s from now), not use the deletion timestamp. A negative value of deletion grace period should be rejected by the apiserver as an invalid value (bad request / invalid field).
attempts to take a missing or positive deletion grace period negative should be rejected, I agree. if a negative value is already persisted, I'd be inclined to allow a negative value in an update and clip to 0
As I mentioned in the PR, eviction is passing negatives through to the store: In kubectl, we specify the default value for GracePeriod is -1. In the case of delete, we simply don't set the value in the deleteOptions we send to the API. In the case of eviction, we send the -1 value, and that gets passed along to the store.
I don't see the -1 getting sent to the eviction API. It looks like values < 0 result in a nil value being sent to the API: I would not expect the API to accept a negative value as-is.
Looks like you are correct. Do the deletionOptions go through their own validation or through some kind of validation for the evict API? The original bug report mentions 'evict' though it's not clear if the eviction API was used or not.
ValidateDeleteOptions is called inside pod deletion, but did not check that the field was non-negative. I don't see any eviction-specific validation.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle rotten
/lifecycle rotten
Im investigating the and . Surprisingly see you here...


Question: // Add claim Name as a part of Portworx Volume Labels locator.VolumeLabels[pvcClaimLabel] = p.options.PVC.Name <ins> locator.VolumeLabels[pvcNamespaceLabel] = p.options.PVC.Namespace for k, v := range p.options.PVC.Annotations { if _, present := spec.VolumeLabels[k]; present { glog.Warningf("not saving annotation: %s=%s in spec labels due to an existing key", k, v) continue } spec.VolumeLabels[k] = v } </ins> volumeID, err := driver.Create(locator, source, spec) if err != nil { glog.Errorf("Error creating Portworx Volume : %v", err) <ins> return "", 0, nil, err </ins> } glog.Infof("Successfully created Portworx volume for PVC: %v", p.options.PVC.Name)

BUG REPORT: During the create workflow, Portworx native driver has the PVC namespace information. But this isn't getting passed to the Portworx Create API. As a result, the Portworx service side handler doesn't get the namespace information. It will also be helpful to send PVC annotations in the Create API call. /kind bug What happened: Portworx native driver didn't send PVC namespace to the Portworx Create API What you expected to happen: Portworx native driver should send the PVC namespace to the Portworx Create API
There are no sig labels on this issue. Please add a sig label. <detailsA sig label can be by either: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. See the . The in method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I'll be fixing this issue in an upcoming PR.


Question: if equals { t.Errorf("Should have not been considered equal since first is not in the second array of groups") } <ins> // The first pair matches, but the second does not newIpPermission2 := ec2.IpPermission{ UserIdGroupPairs: []*ec2.UserIdGroupPair{ {GroupId: aws.String("firstGroupId")}, {GroupId: aws.String("fourthGroupId")}, }, } equals = ipPermissionExists(&newIpPermission2, &oldIpPermission, false) if equals { t.Errorf("Should have not been considered equal since first is not in the second array of groups") } </ins> } func TestIpPermissionExistsHandlesRangeSubsets(t *testing.T) {

Per : the surrounding loop is unconditionally terminated (SA4004) It does look very suspicious.
Pertinent patches: AWS: Set up security groups, to mirror GCE firewalling Fix AWS ip permission comparison on security group
This needs to be triaged as a release-blocker or not for 1.5
all issues must be labeled either release blocker or non release blocking by end of day 18 November 2016 PST. (or please move it to 1.6) cc
Moving to 1.7 as late to happen in 1.6. Feel free to switch back if this is incorrect.
This was a bug where we had multiple userid/group pairs, but we only ever called it with 0 or 1 elements. Fix in but kicking out of the 1.7 milestone.


Question: e2eskipper.Skipf("Cluster doesn't support %q volumes -- skipping", pattern.VolType) } } <ins> // A driver might support the Topology capability which is incompatible with the VolumeBindingMode immediate because // volumes might be provisioned immediately in a different zone to where the workload is located. if pattern.BindingMode == storagev1.VolumeBindingImmediate && len(dInfo.TopologyKeys) > 0 { e2eskipper.Skipf("VolumeBindingMode immediate is not compatible with a multi-topology environment.") } </ins> l = local{}

Let's assume this scenario: linux node, zone A, tainted windows node, zone B, untainted In a multizone cluster, while trying to run the generic ephemeral volume tests in Windows nodes it might be possible that a volume that's supposed to be for a Windows node is created in the zone for the Linux node instead. The generic ephemeral storage tests use : [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral [Testpattern: Generic Ephemeral-volume (default fs) (immediate)] ephemeral [Testpattern: Generic Ephemeral-volume (block volmode) (late-binding)] ephemeral This issue is happening because we're using the immediate testpattern which is not compatible with a multizone cluster. /kind failing-test /sig storage
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I think "immediate" binding in general won't work in a multi-zone cluster. What do we do for other tests that explicitly specify immediate mode?
I think nothing special in any of the tests besides , it's just another test pattern that's tested
Do you see failures of other "immediate" mode tests? I'm wondering if there is some logic to detect and automatically skip those tests
that was the only failure, after I the skip I didn't see any other testpatterns that failed with "immediate" mode in other test groups.
I checked tests that passed with VolumeBindingMode immediate which are found in the topology tests, the topology tests detect if the testdriver has the field set to work and modify the clone of the StorageClass adding the field to it, I'm going to do the same for the Generic Ephemeral Volume tests.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.


Question: e2eskipper.Skipf("Cluster doesn't support %q volumes -- skipping", pattern.VolType) } } <ins> // A driver might support the Topology capability which is incompatible with the VolumeBindingMode immediate because // volumes might be provisioned immediately in a different zone to where the workload is located. if pattern.BindingMode == storagev1.VolumeBindingImmediate && len(dInfo.TopologyKeys) > 0 { e2eskipper.Skipf("VolumeBindingMode immediate is not compatible with a multi-topology environment.") } </ins> l = local{}

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue with Mark this issue as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close not-planned [1]:
Closing this issue, marking it as "Not Planned". <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue with Mark this issue as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: "bytes" "io" "io/ioutil" <ins> "net/http" </ins> "os" "os/exec" "path" "path/filepath" "strings" "testing" <ins> "k8s.io/api/core/v1" "k8s.io/apimachinery/pkg/api/errors" "k8s.io/apimachinery/pkg/runtime" "k8s.io/apimachinery/pkg/runtime/schema" "k8s.io/client-go/rest/fake" "k8s.io/kubernetes/pkg/api/legacyscheme" cmdtesting "k8s.io/kubernetes/pkg/kubectl/cmd/testing" "k8s.io/kubernetes/pkg/kubectl/scheme" </ins> ) type FileType int

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: panics What you expected to happen: Syntax error, as it is not legit cp syntax. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Occurs in every version 1.9.x and 1.10.x, but 1.8.x is safe. is similar, but it has a fix candidate already going in, that is not fixing this, so opening a separate issue. Environment: Kubernetes version (use ):Cloud provider or hardware configuration: VM OS (e.g. from /etc/os-release): Ubuntu 18.04 Kernel (e.g. ): /sig cli
It seems to be fixed in 1.11 by
It was merged three months ago, any idea why it isn't in 1.10 too?
It was merged on March 31 and 1.10 was released on March 27: Can confirm it is not present in 1.10: Maybe we can cherry-pick it?
I've opened
Fixed by /close
Closing this issue. <detailsIn response to : Fixed by Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: testRSAdoptMatchingAndReleaseNotMatching(f) }) <del> ginkgo.It("Replicaset should have a working scale subresource", func() { </del> <ins> /* Release: v1.21 Testname: ReplicaSet, completes the scaling of a ReplicaSet subresource Description: Create a ReplicaSet (RS) with a single Pod. The Pod MUST be verified that it is running. The RS MUST get and verify the scale subresource count. The RS MUST update and verify the scale subresource. The RS MUST patch and verify a scale subresource. */ framework.ConformanceIt("Replicaset should have a working scale subresource", func() { </ins> testRSScaleSubresources(f) }) })

[x] two weeks soak end date : 26 Feb 2021 According to this APIsnoop query, there are still some remaining RESOURCENAME endpoints which are untested. with this query you can filter untested endpoints by their category and eligiblity for conformance. e.g below shows a query to find all conformance eligible untested,stable,core endpoints Create a ReplicaSet with a static label Read the ReplicaSetScale Update the ReplicaSetScale Get the ReplicaSetScale to ensure it is updated Patch the ReplicaSetScale Read the ReplicaSetScale to ensure it is Patch Delete the Namespace and ReplicaSet Due to the complexity of setting up the resources for APIService we have used the current e2e test as template. It has been extended in a for review. example endpoint | useragent ----------------------------------------|
patchAppsV1NamespacedReplicaSetScale | (linux/amd64) kubernetes/$Format -- [sig-apps] ReplicaSet Replicaset should have a working scale subresource readAppsV1NamespacedReplicaSetScale | (linux/amd64) kubernetes/$Format -- [sig-apps] ReplicaSet Replicaset should have a working scale subresource replaceAppsV1NamespacedReplicaSetScale | (linux/amd64) kubernetes/$Format -- [sig-apps] ReplicaSet Replicaset should have a working scale subresource (3 rows) If a test with these calls gets merged, test coverage will go up by 3 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture
/triage accepted
Presented in Conformance meeting of 9 Feb 2021 and it was agreed that that method see to achieve the purpose and a PR can be created. A similar approach would be taken for the DepoymentScale endpoints.
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: e2essh "k8s.io/kubernetes/test/e2e/framework/ssh" ) <del> const etcdImage = "3.5.0-rc.0-0" </del> <ins> const etcdImage = "3.5.0-0" </ins> // EtcdUpgrade upgrades etcd on GCE. func EtcdUpgrade(targetStorage, targetVersion string) error {

/sig scalability This issue is tracking process of upgrading etcd to 3.5.0 planned for 1.22 K8s release. Etcd has recently released Beta versions and is starting the process of correctness and performance testing. To reduce change of finding performance regression late, we will start qualifying Beta Etcd releases on GKE master branch and will upgrade to 3.5.0 as soon as it is available. Steps: [x] Do preliminary scale testing of etcd v3.5.0-beta.3 release [x] Release etcd v3.5.0-beta.3 image (, ) [x] Release etcd v3.5.0-rc.0 image () [x] Upgrade etcd server to v3.5.0-rc.0 () [x] Wait for 3.5.0 release and upgrade etcd client () [x] Release etcd v3.5.0 image (, ) [x] Upgrade etcd server to v3.5.0 /milestone v1.22
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/area etcd
Does this meant that the 3.5.0 should be promoted like
Yes, created
/area release-eng release-team /sig release For tracking: cc:
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /area release-eng release-team /sig release cc: Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I do think this can be pointed in the Major Themes for the 1.22 release. I would like your opinions on this. If you do think it has a place in it, I will add it to the .
How can I be a contributor to this issue? what are the technical checklist that needs to be achieved to close out on the issue?


Question: e2essh "k8s.io/kubernetes/test/e2e/framework/ssh" ) <del> const etcdImage = "3.5.0-rc.0-0" </del> <ins> const etcdImage = "3.5.0-0" </ins> // EtcdUpgrade upgrades etcd on GCE. func EtcdUpgrade(targetStorage, targetVersion string) error {

PR to update Etcd is ready, but we found a blocker for merging . There is one test failing I will take a look into it on Monday, still more eyes on the problem could speed up resolution.
We'll also need a new kube-cross image w/ etcd v3.5.0. I'm working on that here:
We just need to upgrade the server to use new image like in
ok, looks like is busy. Let me take a look /assign
Thanks.


Question: operationPollRateLimiter := flowcontrol.NewTokenBucketRateLimiter(10, 100) // 10 qps, 100 bucket size. <del> return &GCECloud{ </del> <ins> gce := &GCECloud{ </ins> service:                  service, serviceBeta:              serviceBeta, containerService:         containerService,

Unit tests are needed for the GCE cloud provider disk interface.
Assigned to
/assign
GitHub didn't allow me to assign the following users: verult. Note that only can be assigned.. <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. I understand the commands that are listed . </details


Question: // reorganizeTaints returns the updated set of taints, taking into account old taints that were not updated, // old taints that were updated, old taints that were deleted, and new taints. <del> func reorganizeTaints(obj runtime.Object, overwrite bool, taintsToAdd []v1.Taint, taintsToRemove []v1.Taint) ([]v1.Taint, error) { node, ok := obj.(*v1.Node) if !ok { return nil, fmt.Errorf("unexpected type %T, expected Node", obj) } </del> <ins> func reorganizeTaints(node *v1.Node, overwrite bool, taintsToAdd []v1.Taint, taintsToRemove []v1.Taint) (string, []v1.Taint, error) { </ins> newTaints := append([]v1.Taint{}, taintsToAdd...) oldTaints := node.Spec.Taints // add taints that already existing but not updated to newTaints <del> for _, oldTaint := range oldTaints { existsInNew := false for _, taint := range newTaints { if taint.MatchTaint(&oldTaint) { existsInNew = true break } } if !existsInNew { newTaints = append(newTaints, oldTaint) } </del> <ins> added := addTaints(oldTaints, &newTaints) allErrs, deleted := deleteTaints(taintsToRemove, &newTaints) if (added && deleted) || overwrite { return MODIFIED, newTaints, utilerrors.NewAggregate(allErrs) } else if added { return TAINTED, newTaints, utilerrors.NewAggregate(allErrs) </ins> } <ins> return UNTAINTED, newTaints, utilerrors.NewAggregate(allErrs) } // deleteTaints deletes the given taints from the node's taintlist. func deleteTaints(taintsToRemove []v1.Taint, newTaints *[]v1.Taint) ([]error, bool) { </ins> allErrs := []error{} <ins> var removed bool </ins> for _, taintToRemove := range taintsToRemove { <del> removed := false </del> <ins> removed = false </ins> if len(taintToRemove.Effect) > 0 { <del> newTaints, removed = v1helper.DeleteTaint(newTaints, &taintToRemove) </del> <ins> *newTaints, removed = v1helper.DeleteTaint(*newTaints, &taintToRemove) </ins> } else { <del> newTaints, removed = v1helper.DeleteTaintsByKey(newTaints, taintToRemove.Key) </del> <ins> *newTaints, removed = v1helper.DeleteTaintsByKey(*newTaints, taintToRemove.Key) </ins> } if !removed { allErrs = append(allErrs, fmt.Errorf("taint %q not found", taintToRemove.ToString())) } } <del> return newTaints, utilerrors.NewAggregate(allErrs) </del> <ins> return allErrs, removed } // addTaints adds the newTaints list to existing ones and updates the newTaints List. // TODO: This needs a rewrite to take only the new values instead of appended newTaints list to be consistent. func addTaints(oldTaints []v1.Taint, newTaints *[]v1.Taint) bool { for _, oldTaint := range oldTaints { existsInNew := false for _, taint := range *newTaints { if taint.MatchTaint(&oldTaint) { existsInNew = true break } } if !existsInNew { *newTaints = append(*newTaints, oldTaint) } } return len(oldTaints) != len(*newTaints) </ins> } func parseTaints(spec []string) ([]v1.Taint, []v1.Taint, error) {

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"6+", GitVersion:"v1.6.0-alpha.3.1185+-dirty Server Version: version.Info{Major:"1", Minor:"6+", GitVersion:"v1.6.0-alpha.3.1185+-dirty" What happened: Kubectl is always reporting as node <XYZis tainted irrespective of operation when used with local cluster. taint nodes XYZ dedicated- node "XYZ" tainted What you expected to happen: It should have shown exact operation like tainted, untainted modified and overwritten but because of it is always returning tainted. How to reproduce it (as minimally and precisely as possible): taint nodes XYZ dedicated-
cc/
Regarding Yes I agree we should provide more clearer output/feedback. But providing something that makes output more confusing than existing is something would be even more concerning. So as I stated , it would be good to discuss/decide how the output/feedback should like for various operations for one and multi nodes scenarios.
I think having 1 return values is an exponential improvement on what we have now, even if they aren't perfect for all cases. Rather then a disproportionately long debate over a relatively simple improvement.... Could you guys suggest something simple that you're confident it will be easy to agree on, and we can implement it as a second pass ? we are happy to implement anything with n 1 return values :).
I think we'd be happy to consider any reasonable PR. I'm not sure why was closed; I didn't look at it though so I don't know if it solves the problem.
(hit close accidentally)
, I am not sure why was closed. But I raised a PR around 15 days ago() to address this.


Question: return nil, fmt.Errorf("can't allocate unregistered device %v", resource) } devices = sets.NewString() <ins> // Allocates from reusableDevices list first. for device := range reusableDevices { devices.Insert(device) needed-- if needed == 0 { return devices, nil } } </ins> // Needs to allocate additional devices. if m.allocatedDevices[resource] == nil { m.allocatedDevices[resource] = sets.NewString()

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug /area hw-accelerators What happened: In the context of the device plugin, when using init containers in your pod spec, the current behavior will not reuse devices between init containers and regular containers. This is problematic as Pod scheduling is currently wrong: , the current predicate algorithm rightly assumes the maximum number of devices in use at one time is . However, the pod predicate ran on the node is the sum of devices used by regular and init containers. Meaning that your pod can bounce back between the scheduler and a node infinitely. Also given that many of the extended resources are typically more expensive that the built-in ones like CPU, Memory and Storage this bug has very real financial drawbacks in terms of resource usage (e.g: using 4 GPUs on a pod that only uses 2 GPUs at a time). Furthermore, this unexpected behavior was caught during implementation (but we did not catch the scheduling implications) and decision was taken to ignore it for the first implementation. We need to document it for 1.9 What you expected to happen: Devices can be reused by Init Containers Correct resource accounting In terms of the current behavior of the Allocate call to the device plugin, I'd expect the allocate call to be done before every Init containers start and once at the end of the last Init container (for regular containers). /assign
Yep, I realized the inconsistency between the predicate behavior of scheduler and that of node too. Having said that, considering we treat a pod as a whole when predicating currently, it's kind of difficult to reuse the runtime informations between init containers and regular containers. The real crux of the matter might be whether the scenario of requiring extended resources in init containers is common, if so, the issue should be addressed ASAP.
If I remembered correctly, the current design to improve DP includes a reconciliation in admit phase of Pod, it may be able to fix Allocate call for
can you point me to said design ? I'm not sure that the Allocate call for init containers is planned to be fixed as described.
Can you guys please point to the code where is happening. Looks to me that same logic is executed on both the scheduler and the node to calculate a pod's resource request and which is  On the node, function call tree looks like this: Is it possible to add few more words in explanation?


Question: return nil, fmt.Errorf("can't allocate unregistered device %v", resource) } devices = sets.NewString() <ins> // Allocates from reusableDevices list first. for device := range reusableDevices { devices.Insert(device) needed-- if needed == 0 { return devices, nil } } </ins> // Needs to allocate additional devices. if m.allocatedDevices[resource] == nil { m.allocatedDevices[resource] = sets.NewString()

PR that the mentioned change: Code in question is here:
Above looks incorrect description. Predicate that run on the scheduler and on the node is same i.e max(init, regular). Pod admission will pass at the node. Problem might happen at allocation where allocated resources is not the max but the sum of (init, regular). And in that case, pod would hang there in fail state rather 'bounce back infinitely'.
Problem might happen at allocation where allocated resources is not the max but the sum of (init, regular). And in that case, pod would hang there in fail state rather 'bounce back infinitely'. Absolutely correct, sorry to make you confused, I think Renaud name it mainly bacause the allocate func is supposed to be called during the predicate phase, that is, passed in as a part of PredicateAdmitHandler.
The original idea was onced listed in as a sub-task: Now it's removed (I don't know why ...), while I believe the idea was: device resources allocation during pod admit phase. This comment is also related if needed during reconcile in 1 and cache these devices info cached devices later during container create phase
Handle device resources allocation during pod admit phase. This comment (comment) is related Call Allocate() if needed during reconcile in 1 and cache these devices info Use cached devices later during container create phase Current code behaviors exactly as you described, but to address the issue, the key point might be to decide which devices of init containers could be reused by which regular containers, and generate cache accordingly in step2; when creating containers, we simply fetch runtime information from cache.
Thanks a lot for filing this issue, I think for 1.9, we probably want to list this as a known issue and may advise people to not request gpu resources in init containers. We can target to fix this problem in 1.10.
Agreed We don't have a clear solution for this tough. It's pretty clear to me that device resources allocation during pod admit phase is not the right model when init containers comes into the picture. GPUs are pretty simple example. On of the things we plan to do is scrub the GPU memory during the Allocate phase. If GPUs are re-used between initContainers and regular containers the current model doesn't fit.


Question: return nil, fmt.Errorf("can't allocate unregistered device %v", resource) } devices = sets.NewString() <ins> // Allocates from reusableDevices list first. for device := range reusableDevices { devices.Insert(device) needed-- if needed == 0 { return devices, nil } } </ins> // Needs to allocate additional devices. if m.allocatedDevices[resource] == nil { m.allocatedDevices[resource] = sets.NewString()

do you think if you have time to implement the solution you mentioned in the above comment We can try to get it into a 1.9 sub release. I think for now, lets just support device reuse among initContainers and normal containers. We can revisit this part when we need to guarantee an Allocate call per container.
Right what I am mentioning here is that it is not possible to do that. The point of the device plugin is so that we can execute vendor specific code before making the device available in the container (e.g: Scrub the GPU memory). I don't think the proposed solution addresses this problem.
Sure. But to clarify, I think if we dont't expect big shock on current implementation, we could only provide limited support on the point, that is, without re-generate runtime information for regular containers, only those whose requested resource is equal to or is a subset of that in init containers could make a resuse(e.g. if we have two init containers, each requests 2 devices, then a regular container that request 3 devices still could not reuse the devices.) Besides, if a initcontainer requests 2 devices, and we have two regular containers, each requests 1 device, then due to the lack of runtime info re-generation, the devices could only be reused by one of the containers, not both of them. In sum, since we don't re-generate runtime information, we could only reuse devices per container, not per device, like we do in scheduler. Is that what you mean?
Hmm, that is a good question. I forgot that we now return the combined runtime information in AllocResponse. Then for now, perhaps we should just return an error in device plugin manager Allocate() in case both init containers and containers request the same device plugin resource?
You mean we give up reuse and prohibit users to request same kind of device in init containers and regular containers? that seems a bit weird. And if we decide to do so, we might need to do the check at the first beginning, i.e. in API validation, not wait until it goes into DevicePlugin.
I think for 1.9, we just return an error when both init containers and normal containers of a pod request the same device resource, and document this in the device plugin user guide.


Question: return nil, fmt.Errorf("can't allocate unregistered device %v", resource) } devices = sets.NewString() <ins> // Allocates from reusableDevices list first. for device := range reusableDevices { devices.Insert(device) needed-- if needed == 0 { return devices, nil } } </ins> // Needs to allocate additional devices. if m.allocatedDevices[resource] == nil { m.allocatedDevices[resource] = sets.NewString()

The reason is because in the current device plugin implementation, we don't have a good way to free devices used by init containers of an active pod. It is hard to do this kind of validation at API level or even scheduler because we can't know whether an extended resource is device resource or not. So for now, I think we should probably just return a clear error message during admission handler. I am not sure yet whether there will be strong use cases that need to share devices between init containers and normal containers. We can revisit this based on user feedback. If there is strong user demand to support this, something we could do is to differentiate init containers and normal containers in podDevices, and free up devices used by init containers when any normal container is about to run. That would request some code structure change, and we may wait to see whether there is strong demand to support this.
It is hard to do this at API level or even scheduler because we can't know whether an extended resource is device resource or not. Oh, you are right, I ignored the resource awareness issue. I also agree that it's hard to address the issue without big change to current implementation. But I still feel the inconsistency of behavior between API and kubelet might confusing users. However, its effect should be sustainable, since we haven't received feedback about init containers up to now.
Right. So for now, perhaps just return an error when this happens, to give clear error signals.
Oh, I'm afraid we miss something, the check may not work as expected. As I commented, because init-containers run sequentially, we did reuse between init containers in , that is, if we have two init containers, each request 1 device, then we expect 1 device in total in scheduler, but expect 2 in deviceplugin. The check could not cover the scenario.
Thinking about this problem a bit more, I wonder whether we may tackle this issue from the following approaches, ordered by their complexities: 1) Just return an error whenever an init container requests a device plugin resource, and document in the device plugin user guide that requiring device plugin resource in init containers are not supported. This may be the short-term change we can try to get into 1.9. 2) Maintain a list of devices allocated to init containers in ManagerImpl.Allocate().


Question: return nil, fmt.Errorf("can't allocate unregistered device %v", resource) } devices = sets.NewString() <ins> // Allocates from reusableDevices list first. for device := range reusableDevices { devices.Insert(device) needed-- if needed == 0 { return devices, nil } } </ins> // Needs to allocate additional devices. if m.allocatedDevices[resource] == nil { m.allocatedDevices[resource] = sets.NewString()

Try to allocate from this list first for later init containers and normal containers. The required no. of devices we pass to ManagerImpl.devicesToAllocate() should subtract the no. of devices we have allocated to init containers. This may require some code structure change around ManagerImpl.Allocate(), ManagerImpl.allocateContainerResources(), and ManagerImpl.devicesToAllocate(). However, Allocate gRPC call is still issued during Admission all together, and it is possible that we issue an Allocate gRPC for [dev1] (say dev1 allocated to initContianer1), followed by another Allocate gRPC call for [dev1, dev2] (say dev1 and dev2 allocated to initContainer2), with none of containers started or finished. This assumes containers running in the same pod share the same security context, and device plugin doesn't need to perform per-container allocation operations. We just use Allocate gRPC call to learn container runtime settings. 3) Take a further step from 2), during Admission, we just determine what devices to allocate to containers without issuing the actual Allocate gRPC calls. I.e., we only fill deviceIds field in podDevices.deviceAllocateInfo but leaves allocResp field empty. During ManagerImpl.GetDeviceRunContainerOptions(), we check for this case and issue the actual Allocate gRPC call for the container. This ensures we can issue a gRPC Allocate call per container. However, if a pod is restarted multiple times, we don't guarantee Allocate gRPC call will be issued during subsequent pod runs. 4) Take a further step from 3), we can check for overlapping device allocations in ManagerImpl.GetDeviceRunContainerOptions() and invalidate their allocResp so that an Allocate gRPC call will be issued next time for containers using re-allocated devices. E.g., for containerA, if we find it is allocated with [dev1, dev2] and dev1 is also allocated to containerB, we set containerB's allocResp to nil so that next time before containerB is running, we will issue an Allocate gRPC call for dev1. Note that this approach does have the drawback that device plugin failure would cause previously running pods not be able to restart, so we need to carefully evaluate our use cases to see whether we need to implement this. My suggestion is to implement 1) now and gather some user feedbacks to determine whether we need to implement the following approaches.


Question: return nil, fmt.Errorf("can't allocate unregistered device %v", resource) } devices = sets.NewString() <ins> // Allocates from reusableDevices list first. for device := range reusableDevices { devices.Insert(device) needed-- if needed == 0 { return devices, nil } } </ins> // Needs to allocate additional devices. if m.allocatedDevices[resource] == nil { m.allocatedDevices[resource] = sets.NewString()

I see this a very big limitation. Special devices/resources in init containers is a very common use case IMO. Therefore we should try to fix this. This approach perfectly makes sense to me. What is your concern in implementing this? Even today, IIUC, Allocate call is made per container. Is there any other value that this approach is adding over 1) ? I am not able to see any gains with 2) or 3) over 1). Comments please.
I feel we may not be able to implement 2) for 1.9 timeframe. Option 2) also doesn't address the case mentioned above: Option 3) covers that case for pods without restarts. Option 4) fully covers the requirement we we will scrub old container data before giving it to a new container, even if they both run in the same pod, bug it does have the mentioned drawback during device plugin failure.
Take a further step from 3), we can check for overlapping device allocations in ManagerImpl.GetDeviceRunContainerOptions() and invalidate their allocResp so that an Allocate gRPC call will be issued next time for containers using re-allocated devices. E.g., for containerA, if we find it is allocated with [dev1, dev2] and dev1 is also allocated to containerB, we set containerB's allocResp to nil so that next time before containerB is running, we will issue an Allocate gRPC call for dev1. Note that this approach does have the drawback that device plugin failure would cause previously running pods not be able to restart, so we need to carefully evaluate our use cases to see whether we need to implement this. How about making the allocation call optional? That is , if we find a contaienr using re-allocated devices, we'd try to issue an Allocate gRPC call first, and still use the cached setting if allocation failed. And, if we decide not to address the issue in v1.9, which means we have enough time to work on it later, I think approach 2) might not be a good choice. Because it has a premise that devices are stateless, recycle actions before reuse(e.g: Scrub the GPU memory mentioned above) could not be supported. And this way, as I commented above, we could only reuse devices per container, not per device, like we do in scheduler.


Question: return nil, fmt.Errorf("can't allocate unregistered device %v", resource) } devices = sets.NewString() <ins> // Allocates from reusableDevices list first. for device := range reusableDevices { devices.Insert(device) needed-- if needed == 0 { return devices, nil } } </ins> // Needs to allocate additional devices. if m.allocatedDevices[resource] == nil { m.allocatedDevices[resource] = sets.NewString()

I noticed that you have already add the limitation into the , so I think it's already the decision that we add the limitation as a short-time solution in v1.9 So I posted a PR to add the check in code,, PTAL :)
Use Case: Let say there is a which node has two Devices of a resource type X and the device IDs are X1 and X2. User pod has one init container, lets call it cont-init and one regular container, lets call it cont-reg. Both cont-init and cont-reg requests 2 unit of resource X. cont-init will run some vendor specific code on allocated Devices and then cont-reg will use these 'initiated' devices. I think this is the use case that mentioned. And i feel 2) can handle this. 2) approach will allocate both the devices, X1 and X2, to both the containers. When the containers start, init container, cont-init, will be run first and will run to completion and here devices will get initialized. Strictly after this cont-reg will be started and now this regular container has initialized devices. Where i am wrong?
I guess in RenaudWasTaken 's case, the initialization task is considered to be managed in the plugin, not in init-container, and in init-container we only use the device, so then we might need to call plugin to do recycle action before the device reused by reguler containers. In practical use, both of the two scenarios may exist, it depends on the request of user. Besides, if we don't issue an Allocate gRPC call, we could not ensure a ”complete reuse“: say an init-container requests 2 devices, and we have two regular containers, each requests 1 device, then due to the lack of runtime info re-generation, the devices could only be reused by one of the containers, not both of them.
It is the init containers' job to initialize things and if in some special case device plugin is doing the initialization part, in that case, init containers will not be needed and serve no purpose in pod spec. Specific to example of memory scrubbing, i think this also makes more sense to be done in init container. Once Device's memory is scrubbed, it can be used by regular container. Let try to run your use case with approach 2).


Question: return nil, fmt.Errorf("can't allocate unregistered device %v", resource) } devices = sets.NewString() <ins> // Allocates from reusableDevices list first. for device := range reusableDevices { devices.Insert(device) needed-- if needed == 0 { return devices, nil } } </ins> // Needs to allocate additional devices. if m.allocatedDevices[resource] == nil { m.allocatedDevices[resource] = sets.NewString()

There are three containers in pod, cont-init(requested 2 unit or resource X), cont-reg1(requested 1 unit of resource X) and cont-reg2(requested 1 unit of resource X). Now in approach 2), gRPC Allocate call is made for cont-init requesting X1 and X2. Next, gRPC Allocate call is made for cont-reg1. This time resource id will be send in Allocate from the previously allocated [X1, X2]. Lets say X1 is requested for cont-reg1 Next, gRPC Allocate for cont-reg2 and resource with ID X2. All this allocation info gets stored in internal cache. Now when container runs, runtimeOptions will be collected from internal cache. Where do you see a problem in this flow?
/cc
Yep, it could work if the plugin behavior the way you described, but I'm not sure if all kinds of devices could support "pre-allocation", that is, doing allocation on a same device multiple times at the same time, to prepare runtime informations for several containers that reuse the device later. Also I have no idea if all plugin developers consider their plugins "stateless", that is, managing initialization in init-containers. As I don't have enough knowledge on hardware, would like to hear opinions from other guys :)
This would defeat the purpose of a container. In the case of GPUs, when you run a container you expect the GPU to always be in the same state. Furthermore GPUs can't be shared (exclusive usage) between containers meaning that Init containers should behave in the same way as regular containers. With the only difference being that they are executed sequentially and before regular containers. Additionally, operations to setup a device before making them available in a container, whether functionally (load the driver, insert libraries, ....) or security wise (scrub the memory), should be handled by the infrastructure not the users pod spec.
I agree that it's a sane behavior to forbid the use of Device Plugin Resources in Init Containers for the 1.9 release
moving forward,
/cc I think what long-term solution we decide to implement will largely depend on use-case requirements from different device plugins. Would like to hear some thoughts from different device plugin developers on this. Can people cc-ed in this issue take a look at the proposed options mentioned in and provide some feedbacks on which one may better fit your use cases?


Question: return nil, fmt.Errorf("can't allocate unregistered device %v", resource) } devices = sets.NewString() <ins> // Allocates from reusableDevices list first. for device := range reusableDevices { devices.Insert(device) needed-- if needed == 0 { return devices, nil } } </ins> // Needs to allocate additional devices. if m.allocatedDevices[resource] == nil { m.allocatedDevices[resource] = sets.NewString()

Answering on behalf of and Solution 1 makes sense for the moment Solution 2 defeats the purpose of the device plugin, which is setting up a device before making it available to a container. solution 4 seems to be a sane solution. As mentioned in my previous comment, as long as devices are exclusive, I expect the devices exposed in my container to always be in the same state at the start of my container. When / If we implement a mechanism for sharing a GPU / devices across containers then the scenario described by is something that we can have. It doesn't however fit the current situation.
Solution number 1 seems totally reasonable short-term. Long term, simplifying and documenting the relationship between plugin callbacks and container lifecycle would help plugin authors.
For reference, I implemented option 2 solution in PR The change itself is small. This is an alternative short-term fix we may consider to get into a 1.9 patch release. I think the main benefit of taking this fix compared to Option 1 solution is that we can change the underlying implementation to support init containers later with less user-facing impact.
No mater we take PR or PR for 1.9, I think we need to have this issue approved for 1.9 milestone. or or can any of you approve this issue for 1.9? Thanks!


Question: <ins> /* Copyright 2018 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package create import ( "fmt" "github.com/spf13/cobra" batchv1 "k8s.io/api/batch/v1" batchv1beta1 "k8s.io/api/batch/v1beta1" corev1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/runtime" "k8s.io/cli-runtime/pkg/genericclioptions" "k8s.io/cli-runtime/pkg/genericclioptions/resource" batchv1beta1client "k8s.io/client-go/kubernetes/typed/batch/v1beta1" cmdutil "k8s.io/kubernetes/pkg/kubectl/cmd/util" "k8s.io/kubernetes/pkg/kubectl/scheme" "k8s.io/kubernetes/pkg/kubectl/util/templates" ) var ( cronjobLong = templates.LongDesc(` Create a cronjob with the specified name.`) cronjobExample = templates.Examples(` # Create a cronjob kubectl create cronjob my-job --image=busybox # Create a cronjob with command kubectl create cronjob my-job --image=busybox -- date # Create a cronjob with schedule kubectl create cronjob test-job --image=busybox --schedule="*/1 * * * *"`) ) type CreateCronJobOptions struct { PrintFlags *genericclioptions.PrintFlags PrintObj func(obj runtime.Object) error Name     string Image    string Schedule string Command  []string Restart  string Namespace string Client    batchv1beta1client.BatchV1beta1Interface DryRun    bool Builder   *resource.Builder Cmd       *cobra.Command genericclioptions.IOStreams } func NewCreateCronJobOptions(ioStreams genericclioptions.IOStreams) *CreateCronJobOptions { return &CreateCronJobOptions{ PrintFlags: genericclioptions.NewPrintFlags("created").WithTypeSetter(scheme.Scheme), IOStreams:  ioStreams, } } // NewCmdCreateCronJob is a command to to create CronJobs. func NewCmdCreateCronJob(f cmdutil.Factory, ioStreams genericclioptions.IOStreams) *cobra.Command { o := NewCreateCronJobOptions(ioStreams) cmd := &cobra.Command{ Use:     "cronjob NAME --image=image --schedule='0/5 * * * ?' -- [COMMAND] [args...]", Aliases: []string{"cj"}, Short:   cronjobLong, Long:    cronjobLong, Example: cronjobExample, Run: func(cmd *cobra.Command, args []string) { cmdutil.CheckErr(o.Complete(f, cmd, args)) cmdutil.CheckErr(o.Validate()) cmdutil.CheckErr(o.Run()) }, } o.PrintFlags.AddFlags(cmd) cmdutil.AddApplyAnnotationFlags(cmd) cmdutil.AddValidateFlags(cmd) cmdutil.AddDryRunFlag(cmd) cmd.Flags().StringVar(&o.Image, "image", o.Image, "Image name to run.") cmd.Flags().StringVar(&o.Schedule, "schedule", o.Schedule, "A schedule in the Cron format the job should be run with.") cmd.Flags().StringVar(&o.Restart, "restart", o.Restart, "job's restart policy. supported values: OnFailure, Never") return cmd } func (o *CreateCronJobOptions) Complete(f cmdutil.Factory, cmd *cobra.Command, args []string) error { name, err := NameFromCommandArgs(cmd, args) if err != nil { return err } o.Name = name if len(args) > 1 { o.Command = args[1:] } if len(o.Restart) == 0 { o.Restart = "OnFailure" } clientConfig, err := f.ToRESTConfig() if err != nil { return err } o.Client, err = batchv1beta1client.NewForConfig(clientConfig) if err != nil { return err } o.Namespace, _, err = f.ToRawKubeConfigLoader().Namespace() if err != nil { return err } o.Builder = f.NewBuilder() o.Cmd = cmd o.DryRun = cmdutil.GetDryRunFlag(cmd) if o.DryRun { o.PrintFlags.Complete("%s (dry run)") } printer, err := o.PrintFlags.ToPrinter() if err != nil { return err } o.PrintObj = func(obj runtime.Object) error { return printer.PrintObj(obj, o.Out) } return nil } func (o *CreateCronJobOptions) Validate() error { if len(o.Image) == 0 { return fmt.Errorf("--image must be specified") } if len(o.Schedule) == 0 { return fmt.Errorf("--schedule must be specified") } return nil } func (o *CreateCronJobOptions) Run() error { var cronjob *batchv1beta1.CronJob cronjob = o.createCronJob() if !o.DryRun { var err error cronjob, err = o.Client.CronJobs(o.Namespace).Create(cronjob) if err != nil { return fmt.Errorf("failed to create cronjob: %v", err) } } return o.PrintObj(cronjob) } func (o *CreateCronJobOptions) createCronJob() *batchv1beta1.CronJob { return &batchv1beta1.CronJob{ TypeMeta: metav1.TypeMeta{APIVersion: batchv1beta1.SchemeGroupVersion.String(), Kind: "CronJob"}, ObjectMeta: metav1.ObjectMeta{ Name: o.Name, }, Spec: batchv1beta1.CronJobSpec{ Schedule: o.Schedule, JobTemplate: batchv1beta1.JobTemplateSpec{ ObjectMeta: metav1.ObjectMeta{ Name: o.Name, }, Spec: batchv1.JobSpec{ Template: corev1.PodTemplateSpec{ Spec: corev1.PodSpec{ Containers: []corev1.Container{ { Name:    o.Name, Image:   o.Image, Command: o.Command, }, }, RestartPolicy: corev1.RestartPolicy(o.Restart), }, }, }, }, }, } } </ins>

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Why is this needed: Because of is not recommended to create cronjob (with flags) anymore. But there is no imperative command available for to create cronjob. /sig CLI <!-- DO NOT EDIT BELOW THIS LINE --/kind feature
/assign
GitHub didn't allow me to assign the following users: Pingan2017. Note that only and repo collaborators can be assigned. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
i wanna work on this. give me several days
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale


Question: // set the DeleteGracePeriods to 0 if the object has pendingFinalizers but not supporting graceful deletion if pendingFinalizers { klog.V(6).Infof("update the DeletionTimestamp to "now" and GracePeriodSeconds to 0 for object %s, because it has pending finalizers", name) <del> err = markAsDeleting(existing) </del> <ins> err = markAsDeleting(existing, time.Now()) </ins> if err != nil { return nil, err }

What happened: a custom resource defintion a namespace a custom resource in the namespace with a finalizer attached the namespace custom resource in the namespace will get its deletionTimestamp field updated about every 8 seconds. What you expected to happen: The resource would have the deletionTimestamp set once then not be updated again. This becomes a hassle if a controller needs to perform an operation that takes longer than the update period as it will need to perform a read-modify-write with retry loop when removing the finalizer. How to reproduce it (as minimally and precisely as possible): Environment: Kubernetes version (use ):Cloud provider or hardware configuration: GKE 1.11
/sig api-machinery
/area custom-resources deletionTimestamps should not update even if subsequent deletion requests are made
/assign
as this description: if object has pending finalizers, the deletionTimestamps should be updated. Do we need to change this logic ?
deletionTimestamp is supposed to be immutable: storage modifying that timestamp is unexpected, at least to me
we should at least ensure that this documentation is honored:
opened to do that


Question: "k8s.io/apimachinery/pkg/runtime" "k8s.io/apimachinery/pkg/runtime/schema" "k8s.io/apiserver/pkg/admission" <ins> utilpointer "k8s.io/utils/pointer" </ins> ) // RESTDeleteStrategy defines deletion behavior on an object that follows Kubernetes

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Some k8s client tried to evict a pod by setting negative value for deletionGracePeriodSeconds. But all eventual delete calls (and force-delete calls) made by kubelet, pod-garbage-collector, etc were not able to delete that pod, because apiserver doesnt allow deletionGracePeriodSeconds to be increased once it is set. So force-delete (which set it to 0) doesn't take effect when it's already negative. Not even kubectl is able to call delete with negative grace-period. But curl with an even more negative grace-period value worked. What you expected to happen: force-delete works. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: I think the issue is in the code below As the comment suggest, in phase 1, the was set to a negative value (meaning was in the past), and in phase 2, has to be set even lower, otherwise, it will be considered as pending graceful deletion and not performing the immediate delete. I think there're two possible fixes. not allowing setting in the past ( should be non-negative). In that case, we could convert to 0 if negative. that could be set in the past. In that case, in phase 2, we treat negative value of the same as 0, and allow immediate delete. Thoughts? Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig api-machinery
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig api-machinery Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/cc /triage accepted
No component in the system should ever treat the timestamp of DeletionTimestamp as authoritative.


Question: "k8s.io/apimachinery/pkg/runtime" "k8s.io/apimachinery/pkg/runtime/schema" "k8s.io/apiserver/pkg/admission" <ins> utilpointer "k8s.io/utils/pointer" </ins> ) // RESTDeleteStrategy defines deletion behavior on an object that follows Kubernetes

In Kube, components are required to calculate the time period from when they observe the event in their local clock (i.e. 30s from now), not use the deletion timestamp. A negative value of deletion grace period should be rejected by the apiserver as an invalid value (bad request / invalid field).
attempts to take a missing or positive deletion grace period negative should be rejected, I agree. if a negative value is already persisted, I'd be inclined to allow a negative value in an update and clip to 0
As I mentioned in the PR, eviction is passing negatives through to the store: In kubectl, we specify the default value for GracePeriod is -1. In the case of delete, we simply don't set the value in the deleteOptions we send to the API. In the case of eviction, we send the -1 value, and that gets passed along to the store.
I don't see the -1 getting sent to the eviction API. It looks like values < 0 result in a nil value being sent to the API: I would not expect the API to accept a negative value as-is.
Looks like you are correct. Do the deletionOptions go through their own validation or through some kind of validation for the evict API? The original bug report mentions 'evict' though it's not clear if the eviction API was used or not.
ValidateDeleteOptions is called inside pod deletion, but did not check that the field was non-negative. I don't see any eviction-specific validation.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle rotten
/lifecycle rotten
Im investigating the and . Surprisingly see you here...