Question:
While investigating some slowness in mounting/unmounting operations of PVs (in this case EFS volumes) for pods, we observed errors like these: fortunately it does recover quickly on Kubelet’s next retry: Looking through the Kubelet code I found out that the issue is not related to the umount calls themselves, but rather a race condition inside where we call umount. Because of the way golang’s function (called inside ) is implemented, it for the child process to finish after it is . Since there isn't any locking on the child process across those function calls, it's possible that the child process exits event before the Wait() call starts. In such case the wait call returns with a error (). This error is more likely to happen especially if the gap between the Start() and Wait() calls is more (which could for e,g happen if the goroutine had CPU cycles snatched away from it for some time in between those calls). A similar issue also exists in the . While this is a self-recovering issue (because of Kubelet retries), it does seem to increase the pod startup/cleanup latency upto a 1s. Open to suggestions for a fix. The simplest one occurring to me is to treat the error as a special one on which we don't actually return an error for the mount/unmount calls. /sig storage /sig scalability /assign
Another thing is when analyzing a distribution of such errors across different nodes, we observed a skew. The top 10% of the nodes were seeing more than 90% of these errors. And these are nodes that are running hotter (w.r.t the pod churn) compared to others: ! Given this, I'm wondering if those errors are higher on those nodes just because of statistical significance or other causes like more GC pauses happening on those nodes.
/triage accepted
^ I've opened this PR to fix the Mount/Unmount functions in the file. In general we might want to make a similar fix to other invocations of exec.Run() and exec.CombinedOutput() within our codebase. But that'll be a bigger change. Happy to open another issue for it.
Sorry for opening the PR against wrong repo, here's the right one -

fsckErrorsCorrected = 1 // 'fsck' found errors but exited without correcting them fsckErrorsUncorrected = 4 <ins> // Error thrown by exec cmd.Run() when process spawned by cmd.Start() completes before cmd.Wait() is called (see - k/k issue #103753) errNoChildProcesses = "wait: no child processes" </ins> ) // Mounter provides the default implementation of mount.Interface


Question:
While investigating some slowness in mounting/unmounting operations of PVs (in this case EFS volumes) for pods, we observed errors like these: fortunately it does recover quickly on Kubelet’s next retry: Looking through the Kubelet code I found out that the issue is not related to the umount calls themselves, but rather a race condition inside where we call umount. Because of the way golang’s function (called inside ) is implemented, it for the child process to finish after it is . Since there isn't any locking on the child process across those function calls, it's possible that the child process exits event before the Wait() call starts. In such case the wait call returns with a error (). This error is more likely to happen especially if the gap between the Start() and Wait() calls is more (which could for e,g happen if the goroutine had CPU cycles snatched away from it for some time in between those calls). A similar issue also exists in the . While this is a self-recovering issue (because of Kubelet retries), it does seem to increase the pod startup/cleanup latency upto a 1s. Open to suggestions for a fix. The simplest one occurring to me is to treat the error as a special one on which we don't actually return an error for the mount/unmount calls. /sig storage /sig scalability /assign
Another thing is when analyzing a distribution of such errors across different nodes, we observed a skew. The top 10% of the nodes were seeing more than 90% of these errors. And these are nodes that are running hotter (w.r.t the pod churn) compared to others: ! Given this, I'm wondering if those errors are higher on those nodes just because of statistical significance or other causes like more GC pauses happening on those nodes.
/triage accepted
^ I've opened this PR to fix the Mount/Unmount functions in the file. In general we might want to make a similar fix to other invocations of exec.Run() and exec.CombinedOutput() within our codebase. But that'll be a bigger change. Happy to open another issue for it.
Sorry for opening the PR against wrong repo, here's the right one -

command := exec.Command(mountCmd, mountArgs...) output, err := command.CombinedOutput() if err != nil { <ins> if err.Error() == errNoChildProcesses { if command.ProcessState.Success() { // We don't consider errNoChildProcesses an error if the process itself succeeded (see - k/k issue #103753). return nil } // Rewrite err with the actual exit error of the process. err = &exec.ExitError{ProcessState: command.ProcessState} } </ins> klog.Errorf("Mount failed: %vnMounting command: %snMounting arguments: %snOutput: %sn", err, mountCmd, mountArgsLogStr, string(output)) return fmt.Errorf("mount failed: %vnMounting command: %snMounting arguments: %snOutput: %s", err, mountCmd, mountArgsLogStr, string(output))


Question:
While investigating some slowness in mounting/unmounting operations of PVs (in this case EFS volumes) for pods, we observed errors like these: fortunately it does recover quickly on Kubelet’s next retry: Looking through the Kubelet code I found out that the issue is not related to the umount calls themselves, but rather a race condition inside where we call umount. Because of the way golang’s function (called inside ) is implemented, it for the child process to finish after it is . Since there isn't any locking on the child process across those function calls, it's possible that the child process exits event before the Wait() call starts. In such case the wait call returns with a error (). This error is more likely to happen especially if the gap between the Start() and Wait() calls is more (which could for e,g happen if the goroutine had CPU cycles snatched away from it for some time in between those calls). A similar issue also exists in the . While this is a self-recovering issue (because of Kubelet retries), it does seem to increase the pod startup/cleanup latency upto a 1s. Open to suggestions for a fix. The simplest one occurring to me is to treat the error as a special one on which we don't actually return an error for the mount/unmount calls. /sig storage /sig scalability /assign
Another thing is when analyzing a distribution of such errors across different nodes, we observed a skew. The top 10% of the nodes were seeing more than 90% of these errors. And these are nodes that are running hotter (w.r.t the pod churn) compared to others: ! Given this, I'm wondering if those errors are higher on those nodes just because of statistical significance or other causes like more GC pauses happening on those nodes.
/triage accepted
^ I've opened this PR to fix the Mount/Unmount functions in the file. In general we might want to make a similar fix to other invocations of exec.Run() and exec.CombinedOutput() within our codebase. But that'll be a bigger change. Happy to open another issue for it.
Sorry for opening the PR against wrong repo, here's the right one -

command := exec.Command("umount", target) output, err := command.CombinedOutput() if err != nil { <ins> if err.Error() == errNoChildProcesses { if command.ProcessState.Success() { // We don't consider errNoChildProcesses an error if the process itself succeeded (see - k/k issue #103753). return nil } // Rewrite err with the actual exit error of the process. err = &exec.ExitError{ProcessState: command.ProcessState} } </ins> return fmt.Errorf("unmount failed: %vnUnmounting arguments: %snOutput: %s", err, target, string(output)) } return nil


Question:
This shipped in 1.22 as deprecated in 1.22. As a hacky fix to unbreak tests on 1.23 dev, I bumped the deprecation policy one release: We will need to revert that before 1.23 ships and figure out a better solution. /kind cleanup /milestone v1.23 /sig node
/triage accepted
I want to try it. Can I take this issue?
I have explained how to fix this on PR:
/reopen reverted in as fix did not work
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
Hi, Im the bug triage shadow. Since we're in 1.23 milestone code freeze, could we move this issue to milestone 1.24?
this is critical for 1.23 and I think it just got forgotten... we need to get this fixed before test freeze
/priority important-soon

Subsystem:         KubeletSubsystem, Name:              AssignedConfigKey, Help:              "The node's understanding of intended config. The count is always 1.", <del> DeprecatedVersion: "1.23.0", </del> <ins> DeprecatedVersion: "1.22.0", </ins> StabilityLevel:    metrics.ALPHA, }, []string{ConfigSourceLabelKey, ConfigUIDLabelKey, ConfigResourceVersionLabelKey, KubeletConfigKeyLabelKey},


Question:
This shipped in 1.22 as deprecated in 1.22. As a hacky fix to unbreak tests on 1.23 dev, I bumped the deprecation policy one release: We will need to revert that before 1.23 ships and figure out a better solution. /kind cleanup /milestone v1.23 /sig node
/triage accepted
I want to try it. Can I take this issue?
I have explained how to fix this on PR:
/reopen reverted in as fix did not work
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
Hi, Im the bug triage shadow. Since we're in 1.23 milestone code freeze, could we move this issue to milestone 1.24?
this is critical for 1.23 and I think it just got forgotten... we need to get this fixed before test freeze
/priority important-soon

Subsystem:         KubeletSubsystem, Name:              ActiveConfigKey, Help:              "The config source the node is actively using. The count is always 1.", <del> DeprecatedVersion: "1.23.0", </del> <ins> DeprecatedVersion: "1.22.0", </ins> StabilityLevel:    metrics.ALPHA, }, []string{ConfigSourceLabelKey, ConfigUIDLabelKey, ConfigResourceVersionLabelKey, KubeletConfigKeyLabelKey},


Question:
This shipped in 1.22 as deprecated in 1.22. As a hacky fix to unbreak tests on 1.23 dev, I bumped the deprecation policy one release: We will need to revert that before 1.23 ships and figure out a better solution. /kind cleanup /milestone v1.23 /sig node
/triage accepted
I want to try it. Can I take this issue?
I have explained how to fix this on PR:
/reopen reverted in as fix did not work
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
Hi, Im the bug triage shadow. Since we're in 1.23 milestone code freeze, could we move this issue to milestone 1.24?
this is critical for 1.23 and I think it just got forgotten... we need to get this fixed before test freeze
/priority important-soon

Subsystem:         KubeletSubsystem, Name:              LastKnownGoodConfigKey, Help:              "The config source the node will fall back to when it encounters certain errors. The count is always 1.", <del> DeprecatedVersion: "1.23.0", </del> <ins> DeprecatedVersion: "1.22.0", </ins> StabilityLevel:    metrics.ALPHA, }, []string{ConfigSourceLabelKey, ConfigUIDLabelKey, ConfigResourceVersionLabelKey, KubeletConfigKeyLabelKey},


Question:
This shipped in 1.22 as deprecated in 1.22. As a hacky fix to unbreak tests on 1.23 dev, I bumped the deprecation policy one release: We will need to revert that before 1.23 ships and figure out a better solution. /kind cleanup /milestone v1.23 /sig node
/triage accepted
I want to try it. Can I take this issue?
I have explained how to fix this on PR:
/reopen reverted in as fix did not work
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
Hi, Im the bug triage shadow. Since we're in 1.23 milestone code freeze, could we move this issue to milestone 1.24?
this is critical for 1.23 and I think it just got forgotten... we need to get this fixed before test freeze
/priority important-soon

Subsystem:         KubeletSubsystem, Name:              ConfigErrorKey, Help:              "This metric is true (1) if the node is experiencing a configuration-related error, false (0) otherwise.", <del> DeprecatedVersion: "1.23.0", </del> <ins> DeprecatedVersion: "1.22.0", </ins> StabilityLevel:    metrics.ALPHA, }, )


Question:
This shipped in 1.22 as deprecated in 1.22. As a hacky fix to unbreak tests on 1.23 dev, I bumped the deprecation policy one release: We will need to revert that before 1.23 ships and figure out a better solution. /kind cleanup /milestone v1.23 /sig node
/triage accepted
I want to try it. Can I take this issue?
I have explained how to fix this on PR:
/reopen reverted in as fix did not work
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
Hi, Im the bug triage shadow. Since we're in 1.23 milestone code freeze, could we move this issue to milestone 1.24?
this is critical for 1.23 and I think it just got forgotten... we need to get this fixed before test freeze
/priority important-soon

framework.ExpectNoError(err) beforeKC = kc } <ins> // show hidden metrics for release 1.22 beforeKC = updateShowHiddenMetricsForVersion(beforeKC, "1.22") </ins> // reset the node's assigned/active/last-known-good config by setting the source to nil, // so each test starts from a clean-slate (&nodeConfigTestCase{


Question:
This shipped in 1.22 as deprecated in 1.22. As a hacky fix to unbreak tests on 1.23 dev, I bumped the deprecation policy one release: We will need to revert that before 1.23 ships and figure out a better solution. /kind cleanup /milestone v1.23 /sig node
/triage accepted
I want to try it. Can I take this issue?
I have explained how to fix this on PR:
/reopen reverted in as fix did not work
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
Hi, Im the bug triage shadow. Since we're in 1.23 milestone code freeze, could we move this issue to milestone 1.24?
this is critical for 1.23 and I think it just got forgotten... we need to get this fixed before test freeze
/priority important-soon

framework.ExpectNoError(err) localKC = kc } <ins> // show hidden metrics for release 1.22 localKC = updateShowHiddenMetricsForVersion(localKC, "1.22") </ins> }) ginkgo.AfterEach(func() {


Question:
This shipped in 1.22 as deprecated in 1.22. As a hacky fix to unbreak tests on 1.23 dev, I bumped the deprecation policy one release: We will need to revert that before 1.23 ships and figure out a better solution. /kind cleanup /milestone v1.23 /sig node
/triage accepted
I want to try it. Can I take this issue?
I have explained how to fix this on PR:
/reopen reverted in as fix did not work
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
Hi, Im the bug triage shadow. Since we're in 1.23 milestone code freeze, could we move this issue to milestone 1.24?
this is critical for 1.23 and I think it just got forgotten... we need to get this fixed before test freeze
/priority important-soon

return nil }, timeout, interval).Should(gomega.BeNil()) } <ins> func updateShowHiddenMetricsForVersion(cfg *kubeletconfig.KubeletConfiguration, version string) *kubeletconfig.KubeletConfiguration { if cfg == nil { return &kubeletconfig.KubeletConfiguration{ ShowHiddenMetricsForVersion: version, } } cfg.ShowHiddenMetricsForVersion = version return cfg } </ins>


Question:
This shipped in 1.22 as deprecated in 1.22. As a hacky fix to unbreak tests on 1.23 dev, I bumped the deprecation policy one release: We will need to revert that before 1.23 ships and figure out a better solution. /kind cleanup /milestone v1.23 /sig node
/triage accepted
I want to try it. Can I take this issue?
I have explained how to fix this on PR:
/reopen reverted in as fix did not work
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
Hi, Im the bug triage shadow. Since we're in 1.23 milestone code freeze, could we move this issue to milestone 1.24?
this is critical for 1.23 and I think it just got forgotten... we need to get this fixed before test freeze
/priority important-soon


Question:
This shipped in 1.22 as deprecated in 1.22. As a hacky fix to unbreak tests on 1.23 dev, I bumped the deprecation policy one release: We will need to revert that before 1.23 ships and figure out a better solution. /kind cleanup /milestone v1.23 /sig node
/triage accepted
I want to try it. Can I take this issue?
I have explained how to fix this on PR:
/reopen reverted in as fix did not work
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
Hi, Im the bug triage shadow. Since we're in 1.23 milestone code freeze, could we move this issue to milestone 1.24?
this is critical for 1.23 and I think it just got forgotten... we need to get this fixed before test freeze
/priority important-soon


Question:
This shipped in 1.22 as deprecated in 1.22. As a hacky fix to unbreak tests on 1.23 dev, I bumped the deprecation policy one release: We will need to revert that before 1.23 ships and figure out a better solution. /kind cleanup /milestone v1.23 /sig node
/triage accepted
I want to try it. Can I take this issue?
I have explained how to fix this on PR:
/reopen reverted in as fix did not work
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
Hi, Im the bug triage shadow. Since we're in 1.23 milestone code freeze, could we move this issue to milestone 1.24?
this is critical for 1.23 and I think it just got forgotten... we need to get this fixed before test freeze
/priority important-soon


Question:
This shipped in 1.22 as deprecated in 1.22. As a hacky fix to unbreak tests on 1.23 dev, I bumped the deprecation policy one release: We will need to revert that before 1.23 ships and figure out a better solution. /kind cleanup /milestone v1.23 /sig node
/triage accepted
I want to try it. Can I take this issue?
I have explained how to fix this on PR:
/reopen reverted in as fix did not work
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
Hi, Im the bug triage shadow. Since we're in 1.23 milestone code freeze, could we move this issue to milestone 1.24?
this is critical for 1.23 and I think it just got forgotten... we need to get this fixed before test freeze
/priority important-soon


Question:
When calling GetAllocatableDevices, the API returns all the registered devices, regardless of their health status. Thing is, the - and in general doesn't make much sense to report as allocatable Unhealthy devices, as is also documented in the (see last paragraph) The API should not report unhealthy device. This means the reported device set can potentially change across multiple invocations of the API, as . a kube = 1.21 environment, enable the feature gate any device plugin - including sample device plugin sure 1+ devices are reported unhealthy the podresources GetAllocatableResources API, observe the unhealthy device is reported in the return value. N/A Kubernetes version (use ): 1.21 and beyond Cloud provider or hardware configuration: not relevant OS (e.g: ): any Kernel (e.g. ): not relevant Install tools: not relevant Network plugin and version (if this is a network-related bug): not relevant Others: not relevant
/sig node
/triage accepted

return false } <del> // GetAllocatableDevices returns information about all the devices known to the manager </del> <ins> // GetAllocatableDevices returns information about all the healthy devices known to the manager </ins> func (m *ManagerImpl) GetAllocatableDevices() ResourceDeviceInstances { m.mutex.Lock() <del> resp := m.allDevices.Clone() m.mutex.Unlock() klog.V(4).InfoS("Known devices", "numDevices", len(resp)) </del> <ins> defer m.mutex.Unlock() resp := m.allDevices.Filter(m.healthyDevices) klog.V(4).InfoS("GetAllocatableDevices", "known", len(m.allDevices), "allocatable", len(resp)) </ins> return resp }


Question:
When calling GetAllocatableDevices, the API returns all the registered devices, regardless of their health status. Thing is, the - and in general doesn't make much sense to report as allocatable Unhealthy devices, as is also documented in the (see last paragraph) The API should not report unhealthy device. This means the reported device set can potentially change across multiple invocations of the API, as . a kube = 1.21 environment, enable the feature gate any device plugin - including sample device plugin sure 1+ devices are reported unhealthy the podresources GetAllocatableResources API, observe the unhealthy device is reported in the return value. N/A Kubernetes version (use ): 1.21 and beyond Cloud provider or hardware configuration: not relevant OS (e.g: ): any Kernel (e.g. ): not relevant Install tools: not relevant Network plugin and version (if this is a network-related bug): not relevant Others: not relevant
/sig node
/triage accepted

as.True(testManager.isDevicePluginResource(resourceName2)) } <ins> func TestGetAllocatableDevicesMultipleResources(t *testing.T) { socketDir, socketName, _, err := tmpSocketDir() topologyStore := topologymanager.NewFakeManager() require.NoError(t, err) defer os.RemoveAll(socketDir) testManager, err := newManagerImpl(socketName, nil, topologyStore) as := assert.New(t) as.NotNil(testManager) as.Nil(err) resource1Devs := []pluginapi.Device{ {ID: "R1Device1", Health: pluginapi.Healthy}, {ID: "R1Device2", Health: pluginapi.Healthy}, {ID: "R1Device3", Health: pluginapi.Unhealthy}, } resourceName1 := "domain1.com/resource1" e1 := &endpointImpl{} testManager.endpoints[resourceName1] = endpointInfo{e: e1, opts: nil} testManager.genericDeviceUpdateCallback(resourceName1, resource1Devs) resource2Devs := []pluginapi.Device{ {ID: "R2Device1", Health: pluginapi.Healthy}, } resourceName2 := "other.domain2.org/resource2" e2 := &endpointImpl{} testManager.endpoints[resourceName2] = endpointInfo{e: e2, opts: nil} testManager.genericDeviceUpdateCallback(resourceName2, resource2Devs) allocatableDevs := testManager.GetAllocatableDevices() as.Equal(2, len(allocatableDevs)) devInstances1, ok := allocatableDevs[resourceName1] as.True(ok) checkAllocatableDevicesConsistsOf(as, devInstances1, []string{"R1Device1", "R1Device2"}) devInstances2, ok := allocatableDevs[resourceName2] as.True(ok) checkAllocatableDevicesConsistsOf(as, devInstances2, []string{"R2Device1"}) } func TestGetAllocatableDevicesHealthTransition(t *testing.T) { socketDir, socketName, _, err := tmpSocketDir() topologyStore := topologymanager.NewFakeManager() require.NoError(t, err) defer os.RemoveAll(socketDir) testManager, err := newManagerImpl(socketName, nil, topologyStore) as := assert.New(t) as.NotNil(testManager) as.Nil(err) resource1Devs := []pluginapi.Device{ {ID: "R1Device1", Health: pluginapi.Healthy}, {ID: "R1Device2", Health: pluginapi.Healthy}, {ID: "R1Device3", Health: pluginapi.Unhealthy}, } // Adds three devices for resource1, two healthy and one unhealthy. // Expects allocatable devices for resource1 to be 2. resourceName1 := "domain1.com/resource1" e1 := &endpointImpl{} testManager.endpoints[resourceName1] = endpointInfo{e: e1, opts: nil} testManager.genericDeviceUpdateCallback(resourceName1, resource1Devs) allocatableDevs := testManager.GetAllocatableDevices() as.Equal(1, len(allocatableDevs)) devInstances, ok := allocatableDevs[resourceName1] as.True(ok) checkAllocatableDevicesConsistsOf(as, devInstances, []string{"R1Device1", "R1Device2"}) // Unhealthy device becomes healthy resource1Devs = []pluginapi.Device{ {ID: "R1Device1", Health: pluginapi.Healthy}, {ID: "R1Device2", Health: pluginapi.Healthy}, {ID: "R1Device3", Health: pluginapi.Healthy}, } testManager.genericDeviceUpdateCallback(resourceName1, resource1Devs) allocatableDevs = testManager.GetAllocatableDevices() as.Equal(1, len(allocatableDevs)) devInstances, ok = allocatableDevs[resourceName1] as.True(ok) checkAllocatableDevicesConsistsOf(as, devInstances, []string{"R1Device1", "R1Device2", "R1Device3"}) } func checkAllocatableDevicesConsistsOf(as *assert.Assertions, devInstances DeviceInstances, expectedDevs []string) { as.Equal(len(expectedDevs), len(devInstances)) for _, deviceID := range expectedDevs { _, ok := devInstances[deviceID] as.True(ok) } } </ins> func constructDevices(devices []string) checkpoint.DevicesPerNUMA { ret := checkpoint.DevicesPerNUMA{} for _, dev := range devices {


Question:
When calling GetAllocatableDevices, the API returns all the registered devices, regardless of their health status. Thing is, the - and in general doesn't make much sense to report as allocatable Unhealthy devices, as is also documented in the (see last paragraph) The API should not report unhealthy device. This means the reported device set can potentially change across multiple invocations of the API, as . a kube = 1.21 environment, enable the feature gate any device plugin - including sample device plugin sure 1+ devices are reported unhealthy the podresources GetAllocatableResources API, observe the unhealthy device is reported in the return value. N/A Kubernetes version (use ): 1.21 and beyond Cloud provider or hardware configuration: not relevant OS (e.g: ): any Kernel (e.g. ): not relevant Install tools: not relevant Network plugin and version (if this is a network-related bug): not relevant Others: not relevant
/sig node
/triage accepted

} return clone } <ins> // Filter takes a condition set expressed as map[string]sets.String and returns a new // ResourceDeviceInstances with only the devices matching the condition set. func (rdev ResourceDeviceInstances) Filter(cond map[string]sets.String) ResourceDeviceInstances { filtered := NewResourceDeviceInstances() for resourceName, filterIDs := range cond { if _, exists := rdev[resourceName]; !exists { continue } filtered[resourceName] = DeviceInstances{} for instanceID, instance := range rdev[resourceName] { if filterIDs.Has(instanceID) { filtered[resourceName][instanceID] = instance } } } return filtered } </ins>


Question:
When calling GetAllocatableDevices, the API returns all the registered devices, regardless of their health status. Thing is, the - and in general doesn't make much sense to report as allocatable Unhealthy devices, as is also documented in the (see last paragraph) The API should not report unhealthy device. This means the reported device set can potentially change across multiple invocations of the API, as . a kube = 1.21 environment, enable the feature gate any device plugin - including sample device plugin sure 1+ devices are reported unhealthy the podresources GetAllocatableResources API, observe the unhealthy device is reported in the return value. N/A Kubernetes version (use ): 1.21 and beyond Cloud provider or hardware configuration: not relevant OS (e.g: ): any Kernel (e.g. ): not relevant Install tools: not relevant Network plugin and version (if this is a network-related bug): not relevant Others: not relevant
/sig node
/triage accepted

package devicemanager import ( <ins> "encoding/json" </ins> "testing" "github.com/stretchr/testify/require" <ins> "k8s.io/apimachinery/pkg/util/sets" pluginapi "k8s.io/kubelet/pkg/apis/deviceplugin/v1beta1" </ins> "k8s.io/kubernetes/pkg/kubelet/cm/devicemanager/checkpoint" )


Question:
When calling GetAllocatableDevices, the API returns all the registered devices, regardless of their health status. Thing is, the - and in general doesn't make much sense to report as allocatable Unhealthy devices, as is also documented in the (see last paragraph) The API should not report unhealthy device. This means the reported device set can potentially change across multiple invocations of the API, as . a kube = 1.21 environment, enable the feature gate any device plugin - including sample device plugin sure 1+ devices are reported unhealthy the podresources GetAllocatableResources API, observe the unhealthy device is reported in the return value. N/A Kubernetes version (use ): 1.21 and beyond Cloud provider or hardware configuration: not relevant OS (e.g: ): any Kernel (e.g. ): not relevant Install tools: not relevant Network plugin and version (if this is a network-related bug): not relevant Others: not relevant
/sig node
/triage accepted

} } } <ins> func TestResourceDeviceInstanceFilter(t *testing.T) { var expected string var cond map[string]sets.String var resp ResourceDeviceInstances devs := ResourceDeviceInstances{ "foo": DeviceInstances{ "dev-foo1": pluginapi.Device{ ID: "foo1", }, "dev-foo2": pluginapi.Device{ ID: "foo2", }, "dev-foo3": pluginapi.Device{ ID: "foo3", }, }, "bar": DeviceInstances{ "dev-bar1": pluginapi.Device{ ID: "bar1", }, "dev-bar2": pluginapi.Device{ ID: "bar2", }, "dev-bar3": pluginapi.Device{ ID: "bar3", }, }, "baz": DeviceInstances{ "dev-baz1": pluginapi.Device{ ID: "baz1", }, "dev-baz2": pluginapi.Device{ ID: "baz2", }, "dev-baz3": pluginapi.Device{ ID: "baz3", }, }, } resp = devs.Filter(map[string]sets.String{}) expected = `{}` expectResourceDeviceInstances(t, resp, expected) cond = map[string]sets.String{ "foo": sets.NewString("dev-foo1", "dev-foo2"), "bar": sets.NewString("dev-bar1"), } resp = devs.Filter(cond) expected = `{"bar":{"dev-bar1":{"ID":"bar1"}},"foo":{"dev-foo1":{"ID":"foo1"},"dev-foo2":{"ID":"foo2"}}}` expectResourceDeviceInstances(t, resp, expected) cond = map[string]sets.String{ "foo": sets.NewString("dev-foo1", "dev-foo2", "dev-foo3"), "bar": sets.NewString("dev-bar1", "dev-bar2", "dev-bar3"), "baz": sets.NewString("dev-baz1", "dev-baz2", "dev-baz3"), } resp = devs.Filter(cond) expected = `{"bar":{"dev-bar1":{"ID":"bar1"},"dev-bar2":{"ID":"bar2"},"dev-bar3":{"ID":"bar3"}},"baz":{"dev-baz1":{"ID":"baz1"},"dev-baz2":{"ID":"baz2"},"dev-baz3":{"ID":"baz3"}},"foo":{"dev-foo1":{"ID":"foo1"},"dev-foo2":{"ID":"foo2"},"dev-foo3":{"ID":"foo3"}}}` expectResourceDeviceInstances(t, resp, expected) cond = map[string]sets.String{ "foo": sets.NewString("dev-foo1", "dev-foo2", "dev-foo3", "dev-foo4"), "bar": sets.NewString("dev-bar1", "dev-bar2", "dev-bar3", "dev-bar4"), "baz": sets.NewString("dev-baz1", "dev-baz2", "dev-baz3", "dev-bar4"), } resp = devs.Filter(cond) expected = `{"bar":{"dev-bar1":{"ID":"bar1"},"dev-bar2":{"ID":"bar2"},"dev-bar3":{"ID":"bar3"}},"baz":{"dev-baz1":{"ID":"baz1"},"dev-baz2":{"ID":"baz2"},"dev-baz3":{"ID":"baz3"}},"foo":{"dev-foo1":{"ID":"foo1"},"dev-foo2":{"ID":"foo2"},"dev-foo3":{"ID":"foo3"}}}` expectResourceDeviceInstances(t, resp, expected) cond = map[string]sets.String{ "foo": sets.NewString("dev-foo1", "dev-foo4", "dev-foo7"), "bar": sets.NewString("dev-bar1", "dev-bar4", "dev-bar7"), "baz": sets.NewString("dev-baz1", "dev-baz4", "dev-baz7"), } resp = devs.Filter(cond) expected = `{"bar":{"dev-bar1":{"ID":"bar1"}},"baz":{"dev-baz1":{"ID":"baz1"}},"foo":{"dev-foo1":{"ID":"foo1"}}}` expectResourceDeviceInstances(t, resp, expected) } func expectResourceDeviceInstances(t *testing.T, resp ResourceDeviceInstances, expected string) { // per docs in https://pkg.go.dev/encoding/json#Marshal // "Map values encode as JSON objects. The map's key type must either be a string, an integer type, or // implement encoding.TextMarshaler. The map keys are sorted [...]" // so this check is expected to be stable and not flaky data, err := json.Marshal(resp) if err != nil { t.Fatalf("unexpected JSON marshalling error: %v", err) } got := string(data) if got != expected { t.Errorf("expected %q got %q", expected, got) } } </ins>


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --Following link on sig-network page () is broken Link should be working Kubernetes version (use ):1.21 Cloud provider or hardware configuration:None OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig-network
/sig network
/triage accepted

}, { "ImportPath": "github.com/google/cadvisor/api", <del> "Comment": "0.10.1-92-g41a0c30", "Rev": "41a0c30fbf4df4d5d711b752785febb6ed5330a4" </del> <ins> "Comment": "0.10.1-103-gbfaf70b", "Rev": "bfaf70b2555fcaba212130da04a21302344e38f5" </ins> }, { "ImportPath": "github.com/google/cadvisor/container", <del> "Comment": "0.10.1-92-g41a0c30", "Rev": "41a0c30fbf4df4d5d711b752785febb6ed5330a4" </del> <ins> "Comment": "0.10.1-103-gbfaf70b", "Rev": "bfaf70b2555fcaba212130da04a21302344e38f5" </ins> }, { "ImportPath": "github.com/google/cadvisor/events", <del> "Comment": "0.10.1-92-g41a0c30", "Rev": "41a0c30fbf4df4d5d711b752785febb6ed5330a4" </del> <ins> "Comment": "0.10.1-103-gbfaf70b", "Rev": "bfaf70b2555fcaba212130da04a21302344e38f5" </ins> }, { "ImportPath": "github.com/google/cadvisor/fs", <del> "Comment": "0.10.1-92-g41a0c30", "Rev": "41a0c30fbf4df4d5d711b752785febb6ed5330a4" </del> <ins> "Comment": "0.10.1-103-gbfaf70b", "Rev": "bfaf70b2555fcaba212130da04a21302344e38f5" </ins> }, { "ImportPath": "github.com/google/cadvisor/healthz", <del> "Comment": "0.10.1-92-g41a0c30", "Rev": "41a0c30fbf4df4d5d711b752785febb6ed5330a4" </del> <ins> "Comment": "0.10.1-103-gbfaf70b", "Rev": "bfaf70b2555fcaba212130da04a21302344e38f5" </ins> }, { "ImportPath": "github.com/google/cadvisor/http", <del> "Comment": "0.10.1-92-g41a0c30", "Rev": "41a0c30fbf4df4d5d711b752785febb6ed5330a4" </del> <ins> "Comment": "0.10.1-103-gbfaf70b", "Rev": "bfaf70b2555fcaba212130da04a21302344e38f5" </ins> }, { "ImportPath": "github.com/google/cadvisor/info/v1", <del> "Comment": "0.10.1-92-g41a0c30", "Rev": "41a0c30fbf4df4d5d711b752785febb6ed5330a4" </del> <ins> "Comment": "0.10.1-103-gbfaf70b", "Rev": "bfaf70b2555fcaba212130da04a21302344e38f5" </ins> }, { "ImportPath": "github.com/google/cadvisor/info/v2", <del> "Comment": "0.10.1-92-g41a0c30", "Rev": "41a0c30fbf4df4d5d711b752785febb6ed5330a4" </del> <ins> "Comment": "0.10.1-103-gbfaf70b", "Rev": "bfaf70b2555fcaba212130da04a21302344e38f5" </ins> }, { "ImportPath": "github.com/google/cadvisor/manager", <del> "Comment": "0.10.1-92-g41a0c30", "Rev": "41a0c30fbf4df4d5d711b752785febb6ed5330a4" </del> <ins> "Comment": "0.10.1-103-gbfaf70b", "Rev": "bfaf70b2555fcaba212130da04a21302344e38f5" </ins> }, { "ImportPath": "github.com/google/cadvisor/metrics", <del> "Comment": "0.10.1-92-g41a0c30", "Rev": "41a0c30fbf4df4d5d711b752785febb6ed5330a4" </del> <ins> "Comment": "0.10.1-103-gbfaf70b", "Rev": "bfaf70b2555fcaba212130da04a21302344e38f5" </ins> }, { "ImportPath": "github.com/google/cadvisor/pages", <del> "Comment": "0.10.1-92-g41a0c30", "Rev": "41a0c30fbf4df4d5d711b752785febb6ed5330a4" </del> <ins> "Comment": "0.10.1-103-gbfaf70b", "Rev": "bfaf70b2555fcaba212130da04a21302344e38f5" </ins> }, { "ImportPath": "github.com/google/cadvisor/storage", <del> "Comment": "0.10.1-92-g41a0c30", "Rev": "41a0c30fbf4df4d5d711b752785febb6ed5330a4" </del> <ins> "Comment": "0.10.1-103-gbfaf70b", "Rev": "bfaf70b2555fcaba212130da04a21302344e38f5" </ins> }, { "ImportPath": "github.com/google/cadvisor/summary", <del> "Comment": "0.10.1-92-g41a0c30", "Rev": "41a0c30fbf4df4d5d711b752785febb6ed5330a4" </del> <ins> "Comment": "0.10.1-103-gbfaf70b", "Rev": "bfaf70b2555fcaba212130da04a21302344e38f5" </ins> }, { "ImportPath": "github.com/google/cadvisor/utils", <del> "Comment": "0.10.1-92-g41a0c30", "Rev": "41a0c30fbf4df4d5d711b752785febb6ed5330a4" </del> <ins> "Comment": "0.10.1-103-gbfaf70b", "Rev": "bfaf70b2555fcaba212130da04a21302344e38f5" </ins> }, { "ImportPath": "github.com/google/cadvisor/validate", <del> "Comment": "0.10.1-92-g41a0c30", "Rev": "41a0c30fbf4df4d5d711b752785febb6ed5330a4" </del> <ins> "Comment": "0.10.1-103-gbfaf70b", "Rev": "bfaf70b2555fcaba212130da04a21302344e38f5" </ins> }, { "ImportPath": "github.com/google/cadvisor/version", <del> "Comment": "0.10.1-92-g41a0c30", "Rev": "41a0c30fbf4df4d5d711b752785febb6ed5330a4" </del> <ins> "Comment": "0.10.1-103-gbfaf70b", "Rev": "bfaf70b2555fcaba212130da04a21302344e38f5" </ins> }, { "ImportPath": "github.com/google/gofuzz",


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --Following link on sig-network page () is broken Link should be working Kubernetes version (use ):1.21 Cloud provider or hardware configuration:None OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig-network
/sig network
/triage accepted

// TODO(jnagal): Infer schema through reflection. (See bigquery/client/example) func (self *bigqueryStorage) GetSchema() *bigquery.TableSchema { <del> fields := make([]*bigquery.TableFieldSchema, 18) </del> <ins> fields := make([]*bigquery.TableFieldSchema, 19) </ins> i := 0 fields[i] = &bigquery.TableFieldSchema{ Type: typeTimestamp,


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --Following link on sig-network page () is broken Link should be working Kubernetes version (use ):1.21 Cloud provider or hardware configuration:None OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig-network
/sig network
/triage accepted

"flag" "fmt" "io/ioutil" <del> "net/http" </del> "strings" <del> "code.google.com/p/goauth2/oauth" "code.google.com/p/goauth2/oauth/jwt" </del> bigquery "code.google.com/p/google-api-go-client/bigquery/v2" <ins> "golang.org/x/oauth2" "golang.org/x/oauth2/jwt" </ins> ) var (


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --Following link on sig-network page () is broken Link should be working Kubernetes version (use ):1.21 Cloud provider or hardware configuration:None OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig-network
/sig network
/triage accepted

type Client struct { service   *bigquery.Service <del> token     *oauth.Token </del> <ins> token     *oauth2.Token </ins> datasetId string tableId   string } // Helper method to create an authenticated connection. <del> func connect() (*oauth.Token, *bigquery.Service, error) { </del> <ins> func connect() (*oauth2.Token, *bigquery.Service, error) { </ins> if *clientId == "" { return nil, nil, fmt.Errorf("no client id specified") }


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --Following link on sig-network page () is broken Link should be working Kubernetes version (use ):1.21 Cloud provider or hardware configuration:None OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig-network
/sig network
/triage accepted

return nil, nil, fmt.Errorf("could not access credential file %v - %v", pemFile, err) } <del> t := jwt.NewToken(*serviceAccount, authScope, pemBytes) token, err := t.Assert(&http.Client{}) </del> <ins> jwtConfig := &jwt.Config{ Email:      *serviceAccount, Scopes:     []string{authScope}, PrivateKey: pemBytes, TokenURL:   "https://accounts.google.com/o/oauth2/token", } token, err := jwtConfig.TokenSource(oauth2.NoContext).Token() </ins> if err != nil { <del> fmt.Printf("Invalid token: %vn", err) </del> return nil, nil, err } <del> config := &oauth.Config{ ClientId:     *clientId, ClientSecret: *clientSecret, Scope:        authScope, AuthURL:      "https://accounts.google.com/o/oauth2/auth", TokenURL:     "https://accounts.google.com/o/oauth2/token", </del> <ins> if !token.Valid() { return nil, nil, fmt.Errorf("invalid token for BigQuery oauth") </ins> } <del> transport := &oauth.Transport{ Token:  token, Config: config, </del> <ins> config := &oauth2.Config{ ClientID:     *clientId, ClientSecret: *clientSecret, Scopes:       []string{authScope}, Endpoint: oauth2.Endpoint{ AuthURL:  "https://accounts.google.com/o/oauth2/auth", TokenURL: "https://accounts.google.com/o/oauth2/token", }, </ins> } <del> client := transport.Client() </del> <ins> client := config.Client(oauth2.NoContext, token) </ins> service, err := bigquery.New(client) if err != nil {


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --Following link on sig-network page () is broken Link should be working Kubernetes version (use ):1.21 Cloud provider or hardware configuration:None OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig-network
/sig network
/triage accepted

} // Refresh expired token. <del> if c.token.Expired() { </del> <ins> if !c.token.Valid() { </ins> token, service, err := connect() if err != nil { return nil, err


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --As described in issue , we found kubernetes have been running normally, but the control-plane nodes's metrics value of startTime is similar to the timestamp. Get the correct metrics of node As described in issue anything else we need to provide? As described in issue Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
/remove-sig node /sig instrumentation /assign
I think the issue is about Summary API having start time for node metric, but this timestamp is incorrect. This is node issue, not instrumentation. /sig node /remove-sig instrumentation /unassign
When discussing adding node start time metric to /metrics/resource endpoint I remember that SIG Node response to this was that there is no good way to define node start time. I don't think we will be able to provide correct value here. As summary API is technically deprecated, it makes no sense to fix it. We should warn users against using it and wait for removal of Summary API.
Okay, let's wait for the release of metrics-server v0.6.0
Do you know if this is fixed in metrics server?
/triage needs-information Can you please add info on name of timeserias and other steps to repro ti understand exactly what is needed here. Thank you!
in metrics server v0.6.0, we will use /metrics/resource Kubelet endpoint instead summary API
as issue described , the result is: for example : the startup time of the node test-us-west-1-control-plane-577lp is "startTime": "2021-08-03T19:33:50Z",, and timestamp reported by metrics is "2021-08-03T19:34:36Z", but as described, output from control plane node

apiVersion: apps/v1 kind: Deployment metadata: <del> name: metrics-server-v0.5.0 </del> <ins> name: metrics-server-v0.5.1 </ins> namespace: kube-system labels: k8s-app: metrics-server addonmanager.kubernetes.io/mode: Reconcile <del> version: v0.5.0 </del> <ins> version: v0.5.1 </ins> spec: selector: matchLabels: k8s-app: metrics-server <del> version: v0.5.0 </del> <ins> version: v0.5.1 </ins> template: metadata: name: metrics-server labels: k8s-app: metrics-server <del> version: v0.5.0 </del> <ins> version: v0.5.1 </ins> spec: securityContext: seccompProfile:


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --As described in issue , we found kubernetes have been running normally, but the control-plane nodes's metrics value of startTime is similar to the timestamp. Get the correct metrics of node As described in issue anything else we need to provide? As described in issue Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
/remove-sig node /sig instrumentation /assign
I think the issue is about Summary API having start time for node metric, but this timestamp is incorrect. This is node issue, not instrumentation. /sig node /remove-sig instrumentation /unassign
When discussing adding node start time metric to /metrics/resource endpoint I remember that SIG Node response to this was that there is no good way to define node start time. I don't think we will be able to provide correct value here. As summary API is technically deprecated, it makes no sense to fix it. We should warn users against using it and wait for removal of Summary API.
Okay, let's wait for the release of metrics-server v0.6.0
Do you know if this is fixed in metrics server?
/triage needs-information Can you please add info on name of timeserias and other steps to repro ti understand exactly what is needed here. Thank you!
in metrics server v0.6.0, we will use /metrics/resource Kubelet endpoint instead summary API
as issue described , the result is: for example : the startup time of the node test-us-west-1-control-plane-577lp is "startTime": "2021-08-03T19:33:50Z",, and timestamp reported by metrics is "2021-08-03T19:34:36Z", but as described, output from control plane node

kubernetes.io/os: linux containers: - name: metrics-server <del> image: k8s.gcr.io/metrics-server/metrics-server:v0.5.0 </del> <ins> image: k8s.gcr.io/metrics-server/metrics-server:v0.5.1 </ins> command: - /metrics-server - --metric-resolution=30s


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --As described in issue , we found kubernetes have been running normally, but the control-plane nodes's metrics value of startTime is similar to the timestamp. Get the correct metrics of node As described in issue anything else we need to provide? As described in issue Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
/remove-sig node /sig instrumentation /assign
I think the issue is about Summary API having start time for node metric, but this timestamp is incorrect. This is node issue, not instrumentation. /sig node /remove-sig instrumentation /unassign
When discussing adding node start time metric to /metrics/resource endpoint I remember that SIG Node response to this was that there is no good way to define node start time. I don't think we will be able to provide correct value here. As summary API is technically deprecated, it makes no sense to fix it. We should warn users against using it and wait for removal of Summary API.
Okay, let's wait for the release of metrics-server v0.6.0
Do you know if this is fixed in metrics server?
/triage needs-information Can you please add info on name of timeserias and other steps to repro ti understand exactly what is needed here. Thank you!
in metrics server v0.6.0, we will use /metrics/resource Kubelet endpoint instead summary API
as issue described , the result is: for example : the startup time of the node test-us-west-1-control-plane-577lp is "startTime": "2021-08-03T19:33:50Z",, and timestamp reported by metrics is "2021-08-03T19:34:36Z", but as described, output from control plane node

- --memory={{ base_metrics_server_memory }} - --extra-memory={{ metrics_server_memory_per_node }}Mi - --threshold=5 <del> - --deployment=metrics-server-v0.5.0 </del> <ins> - --deployment=metrics-server-v0.5.1 </ins> - --container=metrics-server - --poll-period=30000 - --estimator=exponential


Question:
I'm seeing a panic in kube apiserver (see stack trace below). It appears to be linked to the migration away from Hashicorp's LRU implementation to in The Hashicorp implementation is thread-safe, however the forked copy in utils is not. See the comment here Looking at the utils PR, I see there's actually a wrapper around the thread unsafe LRU cache, and that's being used here: The LRU isn't a lot of code and I think I've convinced myself the locking works fine, so not sure why the element Value is coming back as nil instead of *entry. Stack trace: Not to panic Not sure, but I see it in our CI fairly often Kubernetes version (use ): 1.22Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/cc
/sig api-machinery
if Add(key, nil) was called, that could set the value to nil in an existing object... does that happen?
The nil value we're hitting is on the linked list element, which is managed by the LRU code, not the user-facing value given in -- that gets saved The panic we hit is , where it tries to re-cast the LL element value back to the lru entry type. I don't see how that can happen.
The actual cache is perfectly happy with nil values:
From is mutating since it moves an element to the front when it's accessed. Thanks !
would you consider sending a PR to fix it? /assign /triage accepted
is the PR to fix

log.Printf("Error parsing response (%#v): %s", response, err) return err } <del> log.Printf("Got initial state from etcd: %+v", manifests) </del> <ins> log.Printf("Got state from etcd: %+v", manifests) </ins> updateChannel <- manifestUpdate{etcdSource, manifests} return nil }


Question:
I'm seeing a panic in kube apiserver (see stack trace below). It appears to be linked to the migration away from Hashicorp's LRU implementation to in The Hashicorp implementation is thread-safe, however the forked copy in utils is not. See the comment here Looking at the utils PR, I see there's actually a wrapper around the thread unsafe LRU cache, and that's being used here: The LRU isn't a lot of code and I think I've convinced myself the locking works fine, so not sure why the element Value is coming back as nil instead of *entry. Stack trace: Not to panic Not sure, but I see it in our CI fairly often Kubernetes version (use ): 1.22Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/cc
/sig api-machinery
if Add(key, nil) was called, that could set the value to nil in an existing object... does that happen?
The nil value we're hitting is on the linked list element, which is managed by the LRU code, not the user-facing value given in -- that gets saved The panic we hit is , where it tries to re-cast the LL element value back to the lru entry type. I don't see how that can happen.
The actual cache is perfectly happy with nil values:
From is mutating since it moves an element to the front when it's accessed. Thanks !
would you consider sending a PR to fix it? /assign /triage accepted
is the PR to fix

defer util.HandleCrash() for { watchResponse := <-watchChannel <del> log.Printf("Got change: %#v", watchResponse) </del> // This means the channel has been closed. if watchResponse == nil { return } <ins> log.Printf("Got etcd change: %#v", watchResponse) </ins> manifests, err := kl.extractFromEtcd(watchResponse) if err != nil { log.Printf("Error handling response from etcd: %#v", err)


Question:
I'm seeing a panic in kube apiserver (see stack trace below). It appears to be linked to the migration away from Hashicorp's LRU implementation to in The Hashicorp implementation is thread-safe, however the forked copy in utils is not. See the comment here Looking at the utils PR, I see there's actually a wrapper around the thread unsafe LRU cache, and that's being used here: The LRU isn't a lot of code and I think I've convinced myself the locking works fine, so not sure why the element Value is coming back as nil instead of *entry. Stack trace: Not to panic Not sure, but I see it in our CI fairly often Kubernetes version (use ): 1.22Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/cc
/sig api-machinery
if Add(key, nil) was called, that could set the value to nil in an existing object... does that happen?
The nil value we're hitting is on the linked list element, which is managed by the LRU code, not the user-facing value given in -- that gets saved The panic we hit is , where it tries to re-cast the LL element value back to the lru entry type. I don't see how that can happen.
The actual cache is perfectly happy with nil values:
From is mutating since it moves an element to the front when it's accessed. Thanks !
would you consider sending a PR to fix it? /assign /triage accepted
is the PR to fix

// Sync the configured list of containers (desired state) with the host current state func (kl *Kubelet) SyncManifests(config []api.ContainerManifest) error { <del> log.Printf("Desired:%#v", config) </del> <ins> log.Printf("Desired: %#v", config) </ins> var err error desired := map[string]bool{} for _, manifest := range config {


Question:
I'm seeing a panic in kube apiserver (see stack trace below). It appears to be linked to the migration away from Hashicorp's LRU implementation to in The Hashicorp implementation is thread-safe, however the forked copy in utils is not. See the comment here Looking at the utils PR, I see there's actually a wrapper around the thread unsafe LRU cache, and that's being used here: The LRU isn't a lot of code and I think I've convinced myself the locking works fine, so not sure why the element Value is coming back as nil instead of *entry. Stack trace: Not to panic Not sure, but I see it in our CI fairly often Kubernetes version (use ): 1.22Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/cc
/sig api-machinery
if Add(key, nil) was called, that could set the value to nil in an existing object... does that happen?
The nil value we're hitting is on the linked list element, which is managed by the LRU code, not the user-facing value given in -- that gets saved The panic we hit is , where it tries to re-cast the LL element value back to the lru entry type. I don't see how that can happen.
The actual cache is perfectly happy with nil values:
From is mutating since it moves an element to the front when it's accessed. Thanks !
would you consider sending a PR to fix it? /assign /triage accepted
is the PR to fix

continue } if !exists { <del> log.Printf("Network container doesn't exit, creating") </del> <ins> log.Printf("Network container doesn't exist, creating") </ins> netName, err = kl.createNetworkContainer(&manifest) if err != nil { log.Printf("Failed to create network container: %#v", err)


Question:
I'm seeing a panic in kube apiserver (see stack trace below). It appears to be linked to the migration away from Hashicorp's LRU implementation to in The Hashicorp implementation is thread-safe, however the forked copy in utils is not. See the comment here Looking at the utils PR, I see there's actually a wrapper around the thread unsafe LRU cache, and that's being used here: The LRU isn't a lot of code and I think I've convinced myself the locking works fine, so not sure why the element Value is coming back as nil instead of *entry. Stack trace: Not to panic Not sure, but I see it in our CI fairly often Kubernetes version (use ): 1.22Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/cc
/sig api-machinery
if Add(key, nil) was called, that could set the value to nil in an existing object... does that happen?
The nil value we're hitting is on the linked list element, which is managed by the LRU code, not the user-facing value given in -- that gets saved The panic we hit is , where it tries to re-cast the LL element value back to the lru entry type. I don't see how that can happen.
The actual cache is perfectly happy with nil values:
From is mutating since it moves an element to the front when it's accessed. Thanks !
would you consider sending a PR to fix it? /assign /triage accepted
is the PR to fix

} } existingContainers, _ := kl.ListContainers() <del> log.Printf("Existing:n%#v Desired: %#v", existingContainers, desired) </del> <ins> log.Printf("Existing: %#v Desired: %#v", existingContainers, desired) </ins> for _, container := range existingContainers { // Skip containers that we didn't create to allow users to manually // spin up their own containers if they want.


Question:
SSH'ed into a GKE node running on GCP and was welcomed by following MOTD message: a) Noticed download link is not working, I suspect this storage bucket is no longer used (?) b) Link to get Kuberntes source version does not work: This should be following (without ) Working links :) This is defined in the "configure-" file bellow: As this seems GCP specific (IIUC), I think filtering "-gke.501" will fix part a) For b) I am not sure what the correct download link should be Kubernetes version: Kubernetes v1.18.20-gke.501 Cloud provider: GCP OS: COS
Pull request to remove "-gke." from link to Github to make it valid. Regarding the wrong download link to GKE source code release, I noticed that: When running on GKE the link has (GCS storage bucket name) while in OOS Kubernetes it is . (So GKE is patched here) I think is depreciated and replaced by but in either case link don' work. It's unclear how one can download the source code.
/sig cloud-provider release for triagingnot sure if the link is maintained by release team /area provider/gcp
Was able to confirm the right download location ( GCS bucket) to download GKE source release. Submitted separate PR to fix the URL.
/assign /triage accepted
This is an related to the GKE product. I've filed this to the public :
/close
Closing this issue. <detailsIn response to : /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

https://storage.googleapis.com/kubernetes-release/release/${version}/kubernetes-src.tar.gz It is based on the Kubernetes source at: <del> https://github.com/kubernetes/kubernetes/tree/${gitref} </del> <ins> https://github.com/kubernetes/kubernetes/tree/${gitref/-gke.*/} </ins> ${devel} For Kubernetes copyright and licensing information, see: /home/kubernetes/LICENSES


Question:
N/A [sig-node] NodeProblemDetector: N/A https://prow- Not cgroupv2 compatible when access cpu and memory data
/sig node
Hi, can i work on this ?
Sure. Thanks
/assign
/triage accepted
/priority important-soon
/cc
are you still working on this? Were you able to setup environment to repro the test failure? Let us know if you need any help. Thanks!
Sorry for the delay, I followed the wrong guide and ended up getting stuck. But now I have managed to set it up. Please give me few more days to work on it.
Thanks Feel free to reach out here or on slack if you need any help, thanks for working on this!
/retitle Make nodeproblemdetector test cgroupv2 compatible
Failing job:
Testgrid:

} func getMemoryStat(f *framework.Framework, host string) (rss, workingSet float64) { <del> memCmd := "cat /sys/fs/cgroup/memory/system.slice/node-problem-detector.service/memory.usage_in_bytes && cat /sys/fs/cgroup/memory/system.slice/node-problem-detector.service/memory.stat" </del> <ins> var memCmd string isCgroupV2 := isHostRunningCgroupV2(f, host) if isCgroupV2 { memCmd = "cat /sys/fs/cgroup/system.slice/node-problem-detector.service/memory.current && cat /sys/fs/cgroup/system.slice/node-problem-detector.service/memory.stat" } else { memCmd = "cat /sys/fs/cgroup/memory/system.slice/node-problem-detector.service/memory.usage_in_bytes && cat /sys/fs/cgroup/memory/system.slice/node-problem-detector.service/memory.stat" } </ins> result, err := e2essh.SSH(memCmd, host, framework.TestContext.Provider) framework.ExpectNoError(err) framework.ExpectEqual(result.Code, 0)


Question:
N/A [sig-node] NodeProblemDetector: N/A https://prow- Not cgroupv2 compatible when access cpu and memory data
/sig node
Hi, can i work on this ?
Sure. Thanks
/assign
/triage accepted
/priority important-soon
/cc
are you still working on this? Were you able to setup environment to repro the test failure? Let us know if you need any help. Thanks!
Sorry for the delay, I followed the wrong guide and ended up getting stuck. But now I have managed to set it up. Please give me few more days to work on it.
Thanks Feel free to reach out here or on slack if you need any help, thanks for working on this!
/retitle Make nodeproblemdetector test cgroupv2 compatible
Failing job:
Testgrid:

memoryUsage, err := strconv.ParseFloat(lines[0], 64) framework.ExpectNoError(err) <ins> var rssToken, inactiveFileToken string if isCgroupV2 { // Use Anon memory for RSS as cAdvisor on cgroupv2 // see https://github.com/google/cadvisor/blob/a9858972e75642c2b1914c8d5428e33e6392c08a/container/libcontainer/handler.go#L799 rssToken = "anon" inactiveFileToken = "inactive_file" } else { rssToken = "total_rss" inactiveFileToken = "total_inactive_file" } </ins> var totalInactiveFile float64 for _, line := range lines[1:] { tokens := strings.Split(line, " ") <del> if tokens[0] == "total_rss" { </del> <ins> if tokens[0] == rssToken { </ins> rss, err = strconv.ParseFloat(tokens[1], 64) framework.ExpectNoError(err) } <del> if tokens[0] == "total_inactive_file" { </del> <ins> if tokens[0] == inactiveFileToken { </ins> totalInactiveFile, err = strconv.ParseFloat(tokens[1], 64) framework.ExpectNoError(err) }


Question:
N/A [sig-node] NodeProblemDetector: N/A https://prow- Not cgroupv2 compatible when access cpu and memory data
/sig node
Hi, can i work on this ?
Sure. Thanks
/assign
/triage accepted
/priority important-soon
/cc
are you still working on this? Were you able to setup environment to repro the test failure? Let us know if you need any help. Thanks!
Sorry for the delay, I followed the wrong guide and ended up getting stuck. But now I have managed to set it up. Please give me few more days to work on it.
Thanks Feel free to reach out here or on slack if you need any help, thanks for working on this!
/retitle Make nodeproblemdetector test cgroupv2 compatible
Failing job:
Testgrid:

} func getCPUStat(f *framework.Framework, host string) (usage, uptime float64) { <del> cpuCmd := "cat /sys/fs/cgroup/cpu/system.slice/node-problem-detector.service/cpuacct.usage && cat /proc/uptime | awk '{print $1}'" </del> <ins> var cpuCmd string if isHostRunningCgroupV2(f, host) { cpuCmd = " cat /sys/fs/cgroup/cpu.stat | grep 'usage_usec' | sed 's/[^0-9]*//g' && cat /proc/uptime | awk '{print $1}'" } else { cpuCmd = "cat /sys/fs/cgroup/cpu/system.slice/node-problem-detector.service/cpuacct.usage && cat /proc/uptime | awk '{print $1}'" } </ins> result, err := e2essh.SSH(cpuCmd, host, framework.TestContext.Provider) framework.ExpectNoError(err) framework.ExpectEqual(result.Code, 0)


Question:
N/A [sig-node] NodeProblemDetector: N/A https://prow- Not cgroupv2 compatible when access cpu and memory data
/sig node
Hi, can i work on this ?
Sure. Thanks
/assign
/triage accepted
/priority important-soon
/cc
are you still working on this? Were you able to setup environment to repro the test failure? Let us know if you need any help. Thanks!
Sorry for the delay, I followed the wrong guide and ended up getting stuck. But now I have managed to set it up. Please give me few more days to work on it.
Thanks Feel free to reach out here or on slack if you need any help, thanks for working on this!
/retitle Make nodeproblemdetector test cgroupv2 compatible
Failing job:
Testgrid:

return } <ins> func isHostRunningCgroupV2(f *framework.Framework, host string) bool { result, err := e2essh.SSH("stat -fc %T /sys/fs/cgroup/", host, framework.TestContext.Provider) framework.ExpectNoError(err) framework.ExpectEqual(result.Code, 0) // 0x63677270 == CGROUP2_SUPER_MAGIC // https://www.kernel.org/doc/html/latest/admin-guide/cgroup-v2.html return strings.Contains(result.Stdout, "cgroup2") || strings.Contains(result.Stdout, "0x63677270") } </ins> func getNpdPodStat(f *framework.Framework, nodeName string) (cpuUsage, rss, workingSet float64) { summary, err := e2ekubelet.GetStatsSummary(f.ClientSet, nodeName) framework.ExpectNoError(err)


Question:
<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --ci-cos-cgroupv2-containerd-node-e2e <!-- You can identify the SIG from the "prowjobconfigurl" on the testgrid dashboard for a test --<!-- Please see the deflaking doc () for more guidance! --/sig node
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : #### Which jobs are flaking: links to appreciated links to specific failures in spyglass appreciated </details
On cgroupv2, it looks like like the number of for cgroup is sometimes higher than is currently expected in the test. We currently expect this however, in a few runs of the test we've seen We may need to adjust this value similar as was done in for
/cc
yes, good idea. I think we need to adjust the PageFaults test as well. I've opened a PR:
/assign /triage accepted /area test /priority important-soon

"UserDefinedMetrics": gomega.BeEmpty(), }) } <ins> expectedPageFaultsUpperBound := 1000000 </ins> expectedMajorPageFaultsUpperBound := 10 if IsCgroup2UnifiedMode() { <del> expectedMajorPageFaultsUpperBound = 1000 </del> <ins> // On cgroupv2 these stats are recursive, so make sure they are at least like the value set // above for the container. expectedPageFaultsUpperBound = 1e9 expectedMajorPageFaultsUpperBound = 100000 </ins> } podsContExpectations := sysContExpectations().(*gstruct.FieldsMatcher)


Question:
<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --ci-cos-cgroupv2-containerd-node-e2e <!-- You can identify the SIG from the "prowjobconfigurl" on the testgrid dashboard for a test --<!-- Please see the deflaking doc () for more guidance! --/sig node
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : #### Which jobs are flaking: links to appreciated links to specific failures in spyglass appreciated </details
On cgroupv2, it looks like like the number of for cgroup is sometimes higher than is currently expected in the test. We currently expect this however, in a few runs of the test we've seen We may need to adjust this value similar as was done in for
/cc
yes, good idea. I think we need to adjust the PageFaults test as well. I've opened a PR:
/assign /triage accepted /area test /priority important-soon

"UsageBytes":      bounded(10*e2evolume.Kb, memoryLimit), "WorkingSetBytes": bounded(10*e2evolume.Kb, memoryLimit), "RSSBytes":        bounded(1*e2evolume.Kb, memoryLimit), <del> "PageFaults":      bounded(0, 1000000), </del> <ins> "PageFaults":      bounded(0, expectedPageFaultsUpperBound), </ins> "MajorPageFaults": bounded(0, expectedMajorPageFaultsUpperBound), }) runtimeContExpectations := sysContExpectations().(*gstruct.FieldsMatcher)


Question:
pull-kubernetes-unit TestUpdatePodWithTerminatedPod 10/8 (when it was introduced in ) No response No response Running with shows immediate failures: /sig node
/cc
/assign I will try to fix this. At present, I found that adding before the status check will definitely trigger the error
Looks like it stopped flaking? Was this fixed? /triage accepted
If the test runs fast enough, it won't failed but it's not guaranteed. fixed in
It's still flaking
/assign
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
this started flaking more on 3/21 - adding to the milestone for investigation to make sure something didn't regress here
cc
/unassign /assign

func TestUpdatePodWithTerminatedPod(t *testing.T) { podWorkers, _ := createPodWorkers() terminatedPod := newPodWithPhase("0000-0000-0000", "done-pod", v1.PodSucceeded) <del> runningPod := &kubecontainer.Pod{ID: "0000-0000-0001", Name: "done-pod"} </del> <ins> orphanedPod := &kubecontainer.Pod{ID: "0000-0000-0001", Name: "orphaned-pod"} </ins> pod := newPod("0000-0000-0002", "running-pod") podWorkers.UpdatePod(UpdatePodOptions{


Question:
pull-kubernetes-unit TestUpdatePodWithTerminatedPod 10/8 (when it was introduced in ) No response No response Running with shows immediate failures: /sig node
/cc
/assign I will try to fix this. At present, I found that adding before the status check will definitely trigger the error
Looks like it stopped flaking? Was this fixed? /triage accepted
If the test runs fast enough, it won't failed but it's not guaranteed. fixed in
It's still flaking
/assign
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
this started flaking more on 3/21 - adding to the milestone for investigation to make sure something didn't regress here
cc
/unassign /assign

}) podWorkers.UpdatePod(UpdatePodOptions{ UpdateType: kubetypes.SyncPodKill, <del> RunningPod: runningPod, </del> <ins> RunningPod: orphanedPod, </ins> }) <ins> drainAllWorkers(podWorkers) </ins> if podWorkers.IsPodKnownTerminated(pod.UID) == true { t.Errorf("podWorker state should not be terminated") } if podWorkers.IsPodKnownTerminated(terminatedPod.UID) == false { t.Errorf("podWorker state should be terminated") } <del> if podWorkers.IsPodKnownTerminated(runningPod.ID) == true { t.Errorf("podWorker state should not be marked terminated for a running pod") </del> <ins> if podWorkers.IsPodKnownTerminated(orphanedPod.ID) == false { t.Errorf("podWorker state should be terminated for orphaned pod") </ins> } }


Question:
As per the PRs kubernetes/kubernetes kubernetes/kubernetes, the defaulting behaviour of the pod exec parameters had been changed however in the code there is a comment where it is mentioned it defaults to True The comment about the default value must be removed and the openapi spec must be fixed Present in the file as comment and also in the OpenApiSpec swagger If this issue seems appropriate can I take it up please <details</details<details</details<details</details<details</details<details</details<details</details
/sig docs
/sig node
/assign
/assign
/help /triage accepted
This request has been marked as needing help from a contributor. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help /triage accepted </details
/kind documentation

"uniqueItems": true }, { <del> "description": "Redirect the standard error stream of the pod for this call. Defaults to true.", </del> <ins> "description": "Redirect the standard error stream of the pod for this call.", </ins> "in": "query", "name": "stderr", "type": "boolean",


Question:
As per the PRs kubernetes/kubernetes kubernetes/kubernetes, the defaulting behaviour of the pod exec parameters had been changed however in the code there is a comment where it is mentioned it defaults to True The comment about the default value must be removed and the openapi spec must be fixed Present in the file as comment and also in the OpenApiSpec swagger If this issue seems appropriate can I take it up please <details</details<details</details<details</details<details</details<details</details<details</details
/sig docs
/sig node
/assign
/assign
/help /triage accepted
This request has been marked as needing help from a contributor. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help /triage accepted </details
/kind documentation

"uniqueItems": true }, { <del> "description": "Redirect the standard output stream of the pod for this call. Defaults to true.", </del> <ins> "description": "Redirect the standard output stream of the pod for this call.", </ins> "in": "query", "name": "stdout", "type": "boolean",


Question:
As per the PRs kubernetes/kubernetes kubernetes/kubernetes, the defaulting behaviour of the pod exec parameters had been changed however in the code there is a comment where it is mentioned it defaults to True The comment about the default value must be removed and the openapi spec must be fixed Present in the file as comment and also in the OpenApiSpec swagger If this issue seems appropriate can I take it up please <details</details<details</details<details</details<details</details<details</details<details</details
/sig docs
/sig node
/assign
/assign
/help /triage accepted
This request has been marked as needing help from a contributor. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help /triage accepted </details
/kind documentation

optional bool stdin = 1; // Redirect the standard output stream of the pod for this call. <del> // Defaults to true. </del> // +optional optional bool stdout = 2; // Redirect the standard error stream of the pod for this call. <del> // Defaults to true. </del> // +optional optional bool stderr = 3;


Question:
As per the PRs kubernetes/kubernetes kubernetes/kubernetes, the defaulting behaviour of the pod exec parameters had been changed however in the code there is a comment where it is mentioned it defaults to True The comment about the default value must be removed and the openapi spec must be fixed Present in the file as comment and also in the OpenApiSpec swagger If this issue seems appropriate can I take it up please <details</details<details</details<details</details<details</details<details</details<details</details
/sig docs
/sig node
/assign
/assign
/help /triage accepted
This request has been marked as needing help from a contributor. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help /triage accepted </details
/kind documentation

Stdin bool `json:"stdin,omitempty" protobuf:"varint,1,opt,name=stdin"` // Redirect the standard output stream of the pod for this call. <del> // Defaults to true. </del> // +optional Stdout bool `json:"stdout,omitempty" protobuf:"varint,2,opt,name=stdout"` // Redirect the standard error stream of the pod for this call. <del> // Defaults to true. </del> // +optional Stderr bool `json:"stderr,omitempty" protobuf:"varint,3,opt,name=stderr"`


Question:
As per the PRs kubernetes/kubernetes kubernetes/kubernetes, the defaulting behaviour of the pod exec parameters had been changed however in the code there is a comment where it is mentioned it defaults to True The comment about the default value must be removed and the openapi spec must be fixed Present in the file as comment and also in the OpenApiSpec swagger If this issue seems appropriate can I take it up please <details</details<details</details<details</details<details</details<details</details<details</details
/sig docs
/sig node
/assign
/assign
/help /triage accepted
This request has been marked as needing help from a contributor. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help /triage accepted </details
/kind documentation

var map_PodExecOptions = map[string]string{ "":          "PodExecOptions is the query options to a Pod's remote exec call.", "stdin":     "Redirect the standard input stream of the pod for this call. Defaults to false.", <del> "stdout":    "Redirect the standard output stream of the pod for this call. Defaults to true.", "stderr":    "Redirect the standard error stream of the pod for this call. Defaults to true.", </del> <ins> "stdout":    "Redirect the standard output stream of the pod for this call.", "stderr":    "Redirect the standard error stream of the pod for this call.", </ins> "tty":       "TTY if true indicates that a tty will be allocated for the exec call. Defaults to false.", "container": "Container in which to execute the command. Defaults to only container if there is only one container in the pod.", "command":   "Command is the remote command to execute. argv array. Not executed within a shell.",


Question:
runs in a container. Talos sets OOM score to -450 for the kubelet container and passes in the kubelet config. End result is that OOM score adj is -999 though. should have configured OOM score . Source code analysis. First, OOM score is set correctly from the config (flags): But then, in container manager, OOM score is reset to -999 via the following: This function should skip action if the PID is not running on the host: But the function looks to be confusing: When kubelet runs in a container with PID namespace shared with the host, kubelet's PID namespace will be equal to PID 1 namespace. If kubelet runs in a container with unshared PID namespace, kubelet's PID is 1 and its PID namespace is identical to PID 1 namespace. So how this function can return ever? Moreover, function is always setting OOM score to a hardcoded constant -999 ignoring configuation (). No response <details, reproduced with 1.22. </details<details</details<details</details<details</details<details</details<details</details
/sig node
/triage accepted
/assign
I don't think this method needs to change... I have make a draft PR for the default value to use the flag if present, but I would like more time to add some testing. Maybe you can have a look: ?
Thanks, this PR looks good to me, as it's going to set the OOM score to the configured value. And in our case kubelet runs in a container (not on the host, with proper OOM score via OCI spec), so this code should be even skipped, not sure how this can be addressed.
neither do I...
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale

RuntimeCgroupsName:    s.RuntimeCgroups, SystemCgroupsName:     s.SystemCgroups, KubeletCgroupsName:    s.KubeletCgroups, <ins> KubeletOOMScoreAdj:    s.OOMScoreAdj, </ins> CgroupsPerQOS:         s.CgroupsPerQOS, CgroupRoot:            s.CgroupRoot, CgroupDriver:          s.CgroupDriver,


Question:
runs in a container. Talos sets OOM score to -450 for the kubelet container and passes in the kubelet config. End result is that OOM score adj is -999 though. should have configured OOM score . Source code analysis. First, OOM score is set correctly from the config (flags): But then, in container manager, OOM score is reset to -999 via the following: This function should skip action if the PID is not running on the host: But the function looks to be confusing: When kubelet runs in a container with PID namespace shared with the host, kubelet's PID namespace will be equal to PID 1 namespace. If kubelet runs in a container with unshared PID namespace, kubelet's PID is 1 and its PID namespace is identical to PID 1 namespace. So how this function can return ever? Moreover, function is always setting OOM score to a hardcoded constant -999 ignoring configuation (). No response <details, reproduced with 1.22. </details<details</details<details</details<details</details<details</details<details</details
/sig node
/triage accepted
/assign
I don't think this method needs to change... I have make a draft PR for the default value to use the flag if present, but I would like more time to add some testing. Maybe you can have a look: ?
Thanks, this PR looks good to me, as it's going to set the OOM score to the configured value. And in our case kubelet runs in a container (not on the host, with proper OOM score via OCI spec), so this code should be even skipped, not sure how this can be addressed.
neither do I...
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale

RuntimeCgroupsName    string SystemCgroupsName     string KubeletCgroupsName    string <ins> KubeletOOMScoreAdj    int32 </ins> ContainerRuntime      string CgroupsPerQOS         bool CgroupRoot            string


Question:
runs in a container. Talos sets OOM score to -450 for the kubelet container and passes in the kubelet config. End result is that OOM score adj is -999 though. should have configured OOM score . Source code analysis. First, OOM score is set correctly from the config (flags): But then, in container manager, OOM score is reset to -999 via the following: This function should skip action if the PID is not running on the host: But the function looks to be confusing: When kubelet runs in a container with PID namespace shared with the host, kubelet's PID namespace will be equal to PID 1 namespace. If kubelet runs in a container with unshared PID namespace, kubelet's PID is 1 and its PID namespace is identical to PID 1 namespace. So how this function can return ever? Moreover, function is always setting OOM score to a hardcoded constant -999 ignoring configuation (). No response <details, reproduced with 1.22. </details<details</details<details</details<details</details<details</details<details</details
/sig node
/triage accepted
/assign
I don't think this method needs to change... I have make a draft PR for the default value to use the flag if present, but I would like more time to add some testing. Maybe you can have a look: ?
Thanks, this PR looks good to me, as it's going to set the OOM score to the configured value. And in our case kubelet runs in a container (not on the host, with proper OOM score via OCI spec), so this code should be even skipped, not sure how this can be addressed.
neither do I...
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale

kubecontainer "k8s.io/kubernetes/pkg/kubelet/container" "k8s.io/kubernetes/pkg/kubelet/lifecycle" "k8s.io/kubernetes/pkg/kubelet/pluginmanager/cache" <del> "k8s.io/kubernetes/pkg/kubelet/qos" </del> "k8s.io/kubernetes/pkg/kubelet/stats/pidlimit" "k8s.io/kubernetes/pkg/kubelet/status" schedulerframework "k8s.io/kubernetes/pkg/scheduler/framework"


Question:
runs in a container. Talos sets OOM score to -450 for the kubelet container and passes in the kubelet config. End result is that OOM score adj is -999 though. should have configured OOM score . Source code analysis. First, OOM score is set correctly from the config (flags): But then, in container manager, OOM score is reset to -999 via the following: This function should skip action if the PID is not running on the host: But the function looks to be confusing: When kubelet runs in a container with PID namespace shared with the host, kubelet's PID namespace will be equal to PID 1 namespace. If kubelet runs in a container with unshared PID namespace, kubelet's PID is 1 and its PID namespace is identical to PID 1 namespace. So how this function can return ever? Moreover, function is always setting OOM score to a hardcoded constant -999 ignoring configuation (). No response <details, reproduced with 1.22. </details<details</details<details</details<details</details<details</details<details</details
/sig node
/triage accepted
/assign
I don't think this method needs to change... I have make a draft PR for the default value to use the flag if present, but I would like more time to add some testing. Maybe you can have a look: ?
Thanks, this PR looks good to me, as it's going to set the OOM score to the configured value. And in our case kubelet runs in a container (not on the host, with proper OOM score via OCI spec), so this code should be even skipped, not sure how this can be addressed.
neither do I...
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale

} cont.ensureStateFunc = func(_ cgroups.Manager) error { <del> return ensureProcessInContainerWithOOMScore(os.Getpid(), qos.KubeletOOMScoreAdj, cont.manager) </del> <ins> return ensureProcessInContainerWithOOMScore(os.Getpid(), int(cm.KubeletOOMScoreAdj), cont.manager) </ins> } systemContainers = append(systemContainers, cont) } else { cm.periodicTasks = append(cm.periodicTasks, func() { <del> if err := ensureProcessInContainerWithOOMScore(os.Getpid(), qos.KubeletOOMScoreAdj, nil); err != nil { </del> <ins> if err := ensureProcessInContainerWithOOMScore(os.Getpid(), int(cm.KubeletOOMScoreAdj), nil); err != nil { </ins> klog.ErrorS(err, "Failed to ensure process in container with oom score") return }