Question: } func preCheckForNode(nodeInfo *framework.NodeInfo) queue.PreEnqueueCheck { <ins> if utilfeature.DefaultFeatureGate.Enabled(features.SchedulerQueueingHints) { // QHint is initially created from the motivation of replacing this preCheck. // It assumes that the scheduler only has in-tree plugins, which is problematic for our extensibility. // Here, we skip preCheck if QHint is enabled, and we eventually remove it after QHint is graduated. return nil } </ins> // Note: the following checks doesn't take preemption into considerations, in very rare // cases (e.g., node resizing), "pod" may still fail a check but preemption helps. We deliberately // chose to ignore those cases as unschedulable pods will be re-queued eventually.

So instead of "pass all unscheduled Pods to movePodsToActiveOrBackoffQueue", I'd like to come up with a solution like: It's pretty rare to move pods depending on Unschedulable Pods' events. In your particular case, maybe a proactive call would behave more efficiently. It doesn't sound correlated, is it? As the check will just pre-verify the basic checks which are orthogonal with PodTopologySpread. In other words, it's impossible that a Pod failed can become schedulable eventually. Updated: The example is a valid point... Can you share some use-cases to move Pods upon node's deletion? In the before, I don't think we have any, until the last release the introduction of feature (as in that case deleting a node would impact the number of domains).
to come up with a solution like: If we do so, also, we need to provide the way to pass some functions to decide which Pods should pass to like for out-of-tree plugins? call would behave more efficiently. Ah~, that's a good idea :) Maybe like this (similar to ) user uses WhenUnsatisfiable: DoNotSchedule. Pod is rejected by Pod Topology Spread because of the skew. e.g. a user uses maxSkew: 1 and TopologyKey is hostname, NodeA/NodeB has 10 Pods each, and NodeC has 8 Pods. deletes NodeC skew calculation result should be changed and the Pod rejected in (2) will be allowed by Pod Topology Spread. But the Pod isn't moved from unschedulableQ without flushing because of this issue. So,,, ignoring some events for performance in scheduler side (without providing a way to change this) could be inconvenient for some plugins. What I thought in the above previous comment () is to define "which kind of resource's events should be ignored" in plugin side. (similar to 's idea? ) We can add pre filter functions in like: Then we can set any pre-filter functions for each plugin. And, we can define some default behavior for PreFilterFn and use them when is empty. So, for example, if most of the plugins are considered to prefer to ignore the event for Node that doesn't match nodeAffinity/nodeSelector, we can define this behavior as the default behavior.


Question: } func preCheckForNode(nodeInfo *framework.NodeInfo) queue.PreEnqueueCheck { <ins> if utilfeature.DefaultFeatureGate.Enabled(features.SchedulerQueueingHints) { // QHint is initially created from the motivation of replacing this preCheck. // It assumes that the scheduler only has in-tree plugins, which is problematic for our extensibility. // Here, we skip preCheck if QHint is enabled, and we eventually remove it after QHint is graduated. return nil } </ins> // Note: the following checks doesn't take preemption into considerations, in very rare // cases (e.g., node resizing), "pod" may still fail a check but preemption helps. We deliberately // chose to ignore those cases as unschedulable pods will be re-queued eventually.

What do you think about this idea?
Thanks. It makes sense to me. To summarize, the following cases doesn't properly re-queue relevant pods: Pod's add/update events -scheduled Pod's add/update/delete events 's add/update/delete events For 1 & 2, I've commented above with solutions. (1 needs core changes similar like podAffinity, 2 may leverage the Activate hook) For 3, with 's examples, I'm convinced this is some area we need to improve. One simple and stratightforward solution is to plumb a hook to let the plugin claim "alright, upon Node events, move the pod failed by me anyways". Then once this knob is registered, we move pods failed by that plugin back unconditionally; instead of proceeding with . This hook should be available to in-tree and out-of-tree plugins. In terms of in-tree plugins, it's just PodTopologySpread (we discussed this in today's sig meeting, even InterPodAffinity doesn't fall into this case) So, as a first step, let's ensure we're on the same page and agree on the ^^ summary. Next, we can better evaluate trade-offs and come up with an optimal solution.
Thanks for discussing this topic in the sig meeting. But, I am still believing that all logic related to a plugin should be controllable from the plugin side and we should not create something that can be achieved in in-tree plugins but not in out-of-tree plugins. So... the benefit of the "customizable PreCheck approach" () isn't only to solve this issue. We can use this feature for other optimization: like improving the whole performance of scheduler by moving only Pods that are more likely to be able to be scheduled by defining more detailed events on with customizable PreCheck. Many plugins have a particular scenario that makes rejected Pods schedulable and the current doesn't have an abundant enough ability of expression for this. (So, today many plugins need to say "register this hoge event on foo resource. This event may make rejected Pods schedulable" on comments in . ) Yes, I watched the sig-meeting recording as well and the opinion that the "customizable PreCheck approach" increases some implementation complexity surely makes sense. But,,, isn't this complexity increasing acceptable even if the above is taken into account?


Question: } func preCheckForNode(nodeInfo *framework.NodeInfo) queue.PreEnqueueCheck { <ins> if utilfeature.DefaultFeatureGate.Enabled(features.SchedulerQueueingHints) { // QHint is initially created from the motivation of replacing this preCheck. // It assumes that the scheduler only has in-tree plugins, which is problematic for our extensibility. // Here, we skip preCheck if QHint is enabled, and we eventually remove it after QHint is graduated. return nil } </ins> // Note: the following checks doesn't take preemption into considerations, in very rare // cases (e.g., node resizing), "pod" may still fail a check but preemption helps. We deliberately // chose to ignore those cases as unschedulable pods will be re-queued eventually.

EDIT: It is a bad habit of mine to always write long sentences even though I am not very good at English. Summarizing the above: I think it's not a good idea to create something that can be achieved in in-tree plugins but not in out-of-tree plugins. If we need a solid advantage to extend like this, then it has the advantage of optimizing the queue per plugin like described above.
I didn't jump in the thread since it's actively on the discussion. But, the issue is the similar one. (This time, we can solve that issue by changing scheduling queue logic since It's the k/k feature though.)
I'm not sure I fully understand your intention. The slack thread was discussing the AssignedPod update case, so it fallls into case 1 in , which we already agreed to improve in upstream.
Hi Are you working on this? I think this is quite important and if you're not working on this, I can follow.
Sorry, leave here for a while. I wanted to wait for (and then forgot to come back here. ) That proposal is quite similar to what I want to see in the conclusion of this issue; every plugin can judge if each Pod in unschedQ can be moved to schedQ/backoffQ. Do you think it makes sense to add "By which event the pod will be moved back to schedQ/backoffQ" on 's argument? I know and isn't needed in DefaultEnqueue plugin for API, but that's the last missing puzzle piece : If has , enqueue plugin can judge if the event makes it possible to schedule the Pod. For example, I believe we can move ↓ this logic to Pod Affinity plugin by implementing in Pod Affinity plugin. Pod Affinity plugin will remember Pods rejected by itself in . And in , if the given Pod was the one rejected by Pod Affinity, is AssignedPod/Created|Updated and is Pod affinity with the rejected Pod, then move the Pod back to activeQ/backoffQ. Otherwise reject the Pod moving. We can resolve (3) by implementing as well. (by exactly the way you said.↓) My original concern in this issue is that having logic for specific plugins in the scheduler core may prevent out-of-tree plugins' extendability. The scheduler itself should be pure, and shouldn't do anything special for any plugins implementation ideally.


Question: } func preCheckForNode(nodeInfo *framework.NodeInfo) queue.PreEnqueueCheck { <ins> if utilfeature.DefaultFeatureGate.Enabled(features.SchedulerQueueingHints) { // QHint is initially created from the motivation of replacing this preCheck. // It assumes that the scheduler only has in-tree plugins, which is problematic for our extensibility. // Here, we skip preCheck if QHint is enabled, and we eventually remove it after QHint is graduated. return nil } </ins> // Note: the following checks doesn't take preemption into considerations, in very rare // cases (e.g., node resizing), "pod" may still fail a check but preemption helps. We deliberately // chose to ignore those cases as unschedulable pods will be re-queued eventually.

If we can move all such logic to each plugin, that'd be perfect.
^ is more like a new feature than just a solution to this problem. I created another issue and describe my proposal in detail. Let's have a discussion there.
It can be resolved by moving all preCheck into each plugin's QueueingHint. Let me retitle this issue so that the current status is easy to understand. /retitle scheduler: move all preCheck to QueueingHint
This issue has not been updated in over 1 year, and should be re-triaged. You can: Confirm that this issue is still relevant with (org members only) Close this issue with For more details on the triage process, see /remove-triage accepted
/triage accepted


Question: if !tc.jobStillNotFoundInLister { js = append(js, job) } <ins> } else { job.Status.CompletionTime = &metav1.Time{Time: job.ObjectMeta.CreationTimestamp.Add(time.Second * 10)} job.Status.Conditions = append(job.Status.Conditions, batchv1.JobCondition{ Type:   batchv1.JobComplete, Status: v1.ConditionTrue, }) if !tc.jobStillNotFoundInLister { js = append(js, job) } </ins> } } else { cj.ObjectMeta.CreationTimestamp = metav1.Time{Time: justBeforeTheHour()}

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Cronjob should include the last time when a job was successfully completed. Why is this needed: Currently, a Cronjob's only includes the last time when a job was scheduled. This makes monitoring cronjobs much more complicated, because instead of simply comparing to the current timestamp, the monitoring system now has to hunt for the last job related to the cronjob and persist the state somewhere.
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I'd be interesting in picking this up if the apps team thinks this is a good ticket - it seems like it's fairly beginner friendly.
Hi how is this going on? I could take it over if you don't have the time for this issue. :)
Yes please do! I meant to update that I actually wasn't able to pick this up, but it slipped my mind
/assign
Hi do you have any thoughts on this feature?
/unassign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Any updates?
Reiterating the mentions to trigger a notification: <detailsIn response to : Any updates? Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Trying to understand the usage of lastSuccessfulTime. In a normally working case, lastSuccessfulTime lastScheduledTime =last job completed lastSuccessfulTime < lastScheduledTime =last job completed, but a newer one is running Is that the expectation ?
We wanted to use it in monitoring (e.g. fire an alert if is more than X ago). could also mean that the job has been failing for some time, so I don't know what would you get out of that. I guess you could also use as a proxy for "job has failed at least once"?


Question: if !tc.jobStillNotFoundInLister { js = append(js, job) } <ins> } else { job.Status.CompletionTime = &metav1.Time{Time: job.ObjectMeta.CreationTimestamp.Add(time.Second * 10)} job.Status.Conditions = append(job.Status.Conditions, batchv1.JobCondition{ Type:   batchv1.JobComplete, Status: v1.ConditionTrue, }) if !tc.jobStillNotFoundInLister { js = append(js, job) } </ins> } } else { cj.ObjectMeta.CreationTimestamp = metav1.Time{Time: justBeforeTheHour()}

Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten
that's exactly what is being asked here. /close
Closing this issue. <detailsIn response to : /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
"successfully scheduled" != "successfully completed", and most of the users care more about the latter.
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign /triage accepted
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at .


Question: if !tc.jobStillNotFoundInLister { js = append(js, job) } <ins> } else { job.Status.CompletionTime = &metav1.Time{Time: job.ObjectMeta.CreationTimestamp.Add(time.Second * 10)} job.Status.Conditions = append(job.Status.Conditions, batchv1.JobCondition{ Type:   batchv1.JobComplete, Status: v1.ConditionTrue, }) if !tc.jobStillNotFoundInLister { js = append(js, job) } </ins> } } else { cj.ObjectMeta.CreationTimestamp = metav1.Time{Time: justBeforeTheHour()}

/lifecycle rotten
This is described in and being actively implemented in /close
Closing this issue. <detailsIn response to : This is described in and being actively implemented in /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
It doesn't look like it was actually implemented, though. Searching for only shows API changes, there don't seem to be any changes to the CronJob controller logic. Or am I missing something?
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign /remove-lifecycle rotten


Question: ginkgo.By("scheduling a pod with a container that verifies %COMPUTERNAME% matches selected node name") image := imageutils.GetConfig(imageutils.BusyBox) podName := "host-process-test-pod" <ins> // We're passing this to powershell.exe -Command. Inside, we must use apostrophes for strings. command := fmt.Sprintf(`& {if ('%s' -ine "$(hostname)") { exit -1 }}`, targetNode.Name) </ins> pod := &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name: podName,

Windows GCE jobs: [sig-windows] [Feature:WindowsHostProcessContainers] [MinimumKubeletVersion:1.22] HostProcess containers should run as a process on the host/node [sig-windows] [Feature:WindowsHostProcessContainers] [MinimumKubeletVersion:1.22] HostProcess containers should support various volume mount types Since the jobs were reintroduced. Windows COMPUTERNAMEs are typically limited to 15 characters [1], while the kubelet nodes spawned in the GCE test jobs are much longer (e.g.: `). The names might be truncated, resulting in the test failure. [1] No response /sig windows /sig testing
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
dupe of /close
Closing this issue. <detailsIn response to : dupe of /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: ginkgo.By("scheduling a pod with a container that verifies %COMPUTERNAME% matches selected node name") image := imageutils.GetConfig(imageutils.BusyBox) podName := "host-process-test-pod" <ins> // We're passing this to powershell.exe -Command. Inside, we must use apostrophes for strings. command := fmt.Sprintf(`& {if ('%s' -ine "$(hostname)") { exit -1 }}`, targetNode.Name) </ins> pod := &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name: podName,

I want to run the e2e test but it failed. Reason: Node name and hostname is longer than 15 characters. But the ComputerName only used the first 15 characters. So they don't equal and the test failed. The e2etest should pass regardless of the hostname length. Deploy a Windows with long nodename (longer than 15 characters). Then, run e2e test. No response <details</details<details</details<details</details<details</details<details</details<details</details
/sig testing
Can you show the error message?
STEP: Destroying namespace "host-process-test-windows-2574" for this suite. • Failure [6.334 seconds] [sig-windows] [Feature:WindowsHostProcessContainers] [Excluded:WindowsDocker] [MinimumKubeletVersion:1.22] HostProcess containers should run as a process on the host/node [It] Nov 21 22:12:49.003: Expected <v1.PodPhase: Failed to equal <v1.PodPhase: Succeeded {"msg":"FAILED [sig-windows] [Feature:WindowsHostProcessContainers] [Excluded:WindowsDocker] [MinimumKubeletVersion:1.22] HostProcess containers should run as a process on the host/node","total":1,"completed":0,"skipped":1147,"failed":1,"failures":["[sig-windows] [Feature:WindowsHostProcessContainers] [Excluded:WindowsDocker] [MinimumKubeletVersion:1.22] HostProcess containers should run as a process on the host/node"]}
/sig windows does this hostname error ring a bell?
We discussed this a bit on slack, the test is using %COMPUTERNAME% to validate we are running on the host. This value is truncated to 15 chars on windows due to netbios. We could use the hostname, which isn't truncated, with a fallback to computer name for this validation to make the test validate more scenarios. Important to note that some scenarios might not work if in older environments that use netbios but for our test purposes this will work. Long discussion on netbios here:
We'll be working on this as part of a broader windows operational readiness work. Thanks wenli for digging into it.


Question: ginkgo.By("scheduling a pod with a container that verifies %COMPUTERNAME% matches selected node name") image := imageutils.GetConfig(imageutils.BusyBox) podName := "host-process-test-pod" <ins> // We're passing this to powershell.exe -Command. Inside, we must use apostrophes for strings. command := fmt.Sprintf(`& {if ('%s' -ine "$(hostname)") { exit -1 }}`, targetNode.Name) </ins> pod := &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name: podName,

/assign
Just for broader context... the reason wenli discovered this is, bc we heavily test cluster API over here :), where the dynamically generated host name appends machine-set + cluster characters, so the machines have names like this: to the machines initial name, and it's hardcoded. We want to try to improve this upstream , and have a proposal to do this in flight also:
/sig windows /remove-sig testing
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.


Question: ginkgo.By("scheduling a pod with a container that verifies %COMPUTERNAME% matches selected node name") image := imageutils.GetConfig(imageutils.BusyBox) podName := "host-process-test-pod" <ins> // We're passing this to powershell.exe -Command. Inside, we must use apostrophes for strings. command := fmt.Sprintf(`& {if ('%s' -ine "$(hostname)") { exit -1 }}`, targetNode.Name) </ins> pod := &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name: podName,

After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen being fixed by
Reopened this issue. <detailsIn response to : /reopen being fixed by Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.


Question: ginkgo.By("scheduling a pod with a container that verifies %COMPUTERNAME% matches selected node name") image := imageutils.GetConfig(imageutils.BusyBox) podName := "host-process-test-pod" <ins> // We're passing this to powershell.exe -Command. Inside, we must use apostrophes for strings. command := fmt.Sprintf(`& {if ('%s' -ine "$(hostname)") { exit -1 }}`, targetNode.Name) </ins> pod := &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name: podName,

This bot triages issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue with Mark this issue as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close not-planned [1]:
Closing this issue, marking it as "Not Planned". <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue with Mark this issue as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: proper information. release: v1.9 file: test/e2e/kubectl/kubectl.go <ins> - testname: Kubectl, diff Deployment codename: '[sig-cli] Kubectl client Kubectl diff should check if kubectl diff finds a difference for Deployments [Conformance]' description: Create a Deployment with httpd image. Declare the same Deployment with a different image, busybox. Diff of live Deployment with declared Deployment MUST include the difference between live and declared image. release: v1.19 file: test/e2e/kubectl/kubectl.go </ins> - testname: Kubectl, create service, replication controller codename: '[sig-cli] Kubectl client Kubectl expose should create services for rc  [Conformance]' description: Create a Pod running agnhost listening to port 6379. Using kubectl

and server-side dry-run are being promoted to stable features in 1.18. Conformance should ensure that cloud providers support these features. As a user, I expect to work, which requires that cloud providers for cloud default webhooks. Conformance would generally avoid cloud-specific issues like this one: This suggestion is from /area conformance
/sig testing /sig architecture
/wg apply
There are some good references for conformance tests in the conformance meeting notes:
once merges, we need to promote the e2e test to conformance in a follow up PR, in accordance to this doc: It seems like we meet the conformance test requirements. Here's the process specifically, which includes sig-arch and a review board group:


Question: return kl.PodResourcesAreReclaimed(pod, status) } <del> // filterOutTerminatedPods returns the pods that could still have running // containers </del> <ins> // filterOutTerminatedPods returns pods that are not in a terminal phase // or are known to be fully terminated. This method should only be used // when the set of pods being filtered is upstream of the pod worker, i.e. // the pods the pod manager is aware of. </ins> func (kl *Kubelet) filterOutTerminatedPods(pods []*v1.Pod) []*v1.Pod { <del> var filteredPods []*v1.Pod </del> <ins> filteredPods := make([]*v1.Pod, 0, len(pods)) </ins> for _, p := range pods { <del> if !kl.podWorkers.CouldHaveRunningContainers(p.UID) { </del> <ins> if kl.podWorkers.IsPodKnownTerminated(p.UID) { continue } if p.Status.Phase == v1.PodSucceeded || p.Status.Phase == v1.PodFailed { </ins> continue } filteredPods = append(filteredPods, p)

There appears to be a change of behavior / possible regression in k8s 1.22 with respect to pod lifecycle behavior. We've started to see some cases when multiple pods are being created (often as part of a k8s job) the following will happen: pods get created by e.g. job controller (e.g. say 5). Each pod in job requests majority of node memory so only one such pod can fit on node at a time. schedules [first] pod to node, rest (4 pods) stay in pending accepts first pod, runs it to completion pod get scheduled to the node rejects the pod during pod admission, e.g. with , despite the fact that the first pod has already completed successfully This results in many of pods backing the job getting into Failed phase and having to be recreated by job controller. The job can often get into BackofLimit exceeded to hitting backoffLimit. The root issue seems to be a mismatch of when kubelet actually considers the pod to complete and what it reports to API server (which causes the followup pods to be scheduled to it). I'm still investigating further, but this may be related to the pod lifecycle changes in 1.22 -- I've created a e2e test as minimal repro of this behavior -- (Test fails) 22 build log -- 22 kubelet log -- (Test passes) 21 build log -- 21 kubelet log -- <!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --- Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository.


Question: return kl.PodResourcesAreReclaimed(pod, status) } <del> // filterOutTerminatedPods returns the pods that could still have running // containers </del> <ins> // filterOutTerminatedPods returns pods that are not in a terminal phase // or are known to be fully terminated. This method should only be used // when the set of pods being filtered is upstream of the pod worker, i.e. // the pods the pod manager is aware of. </ins> func (kl *Kubelet) filterOutTerminatedPods(pods []*v1.Pod) []*v1.Pod { <del> var filteredPods []*v1.Pod </del> <ins> filteredPods := make([]*v1.Pod, 0, len(pods)) </ins> for _, p := range pods { <del> if !kl.podWorkers.CouldHaveRunningContainers(p.UID) { </del> <ins> if kl.podWorkers.IsPodKnownTerminated(p.UID) { continue } if p.Status.Phase == v1.PodSucceeded || p.Status.Phase == v1.PodFailed { </ins> continue } filteredPods = append(filteredPods, p)

</details
/sig node
/cc
Looking at the kubelet log in 1.22 () here's what I see -- First pod starts up... starts dies after ~10 seconds since the job is completed (the container simply runs a ) Pod completes, status is updated to API server Next pod comes in but is rejected with despite the earlier pod already being completed/terminated
did the kubelet pod lifecycle rework result in a reordering of kubelet resource bookeeping w.r.t. pod status? it seems reasonable for the scheduler to send another pod to a kubelet once it reports the pod complete and the use of the resources is done from an API perspective
I am also seeing this same behavior, but with "OutOfPods" rather than "OutOfMemory", when running a job with parallelism=100, completions=400 on GKE (which has a 110 pod per node limit).
this is easily reproducible on a local cluster as well start with this: then create this job: result on 1.22 is the job hangs with 2 completions and a bunch of rejections by the kubelet: on 1.21, the same job completes all pods successfully in a couple minutes with no OutOfpods errors
verified introduced the OutOfPods issue
Shame we didn’t have an e2e which caught this given how easy the reproducer is. This looks to be the opposite of the previous cpu admission bug (ignoring things that are terminating but not terminated). We are weak on admission testing in general. This potentially implies admission is looking at the wrong set of pods (admitted + terminating). Note there is already a bug in 1.21 and before where force deleted pods are incorrectly ignored from the set of allocated resources while they are terminating, but this doesn’t immediately seem to be that issue (since i doubt those jobs are being force deleted)
yeah, there was no force deletion involved in (or any deletion at all, actually... the completed pods just sat there, completed)
The set of admitted pods considered for pod count is everything in the pod manager filtered by filterOutTerminatingPods, which calls CouldHaveRunningContainers which returns true once a pod is seen by the workers (shortly after admission, and until we’ve confirmed via syncTerminatingPod that no containers are running). Whatever the problem is it’s likely in the interaction of that.


Question: return kl.PodResourcesAreReclaimed(pod, status) } <del> // filterOutTerminatedPods returns the pods that could still have running // containers </del> <ins> // filterOutTerminatedPods returns pods that are not in a terminal phase // or are known to be fully terminated. This method should only be used // when the set of pods being filtered is upstream of the pod worker, i.e. // the pods the pod manager is aware of. </ins> func (kl *Kubelet) filterOutTerminatedPods(pods []*v1.Pod) []*v1.Pod { <del> var filteredPods []*v1.Pod </del> <ins> filteredPods := make([]*v1.Pod, 0, len(pods)) </ins> for _, p := range pods { <del> if !kl.podWorkers.CouldHaveRunningContainers(p.UID) { </del> <ins> if kl.podWorkers.IsPodKnownTerminated(p.UID) { continue } if p.Status.Phase == v1.PodSucceeded || p.Status.Phase == v1.PodFailed { </ins> continue } filteredPods = append(filteredPods, p)

A pod the pod worker hasn’t seen yet is also filtered out (that means bursts of pods can get in, which seems the opposite of this problem, and is the bug i mentioned before).
Edit: one wrong thing spotted while looking - I'm not sure it fixes the issue you're seeing, but it's definitely wrong for admission (admission is filtering the known pod set down to remove pods that are terminated, but shouldn't filter pods that aren't admitted yet). This needs a node e2e reproducer that can run in serial (resource consumption is really hard to test from parallel, is slow in serial, and node e2e is back in a good state now) - / is that something either of you could help me with? I believe the reason the existing tests don't catch this is the job controller is creating pods fast enough that we see races in admission (multiple pods are admitted in one call to HandlePodAdditions) that are not caught in the current serial node e2e admission tests which rarely create more than one at a time.
Job pods still go through the scheduler to get assigned to a node, so even though the job controller can create pods in parallel, I'd still expect them to arrive at the node in an orderly fashion. It's still possible multiple arrive in a single HandlePodAdditions call, of course. It seems like kubelet admission is using laggy information that doesn't reflect pod completion the kubelet itself wrote to the pod API.
The bug turned out to be much simpler. Pods that are completed never go to the pod worker, so once a pod has reached terminal state it was being back to the list of active pods. The set of e2es that would have prevented this regression: node e2e that creates multiple pods at the same time and verifies that admission works correctly when HandlePodAdditions takes two pods at once node e2e that verifies that completed pods don't consume resources for admission purposes in the kubelet job controller e2e that attempts to verify some part of this reliably (ideally in a way that can be run in parallel e2e)
another related issue:
Linking here for completeness - a similar symptom might get exposed after this fix as described in The kubelet considers resources for terminating pods to be in use (they are!), but the scheduler ignores terminating pods and schedules new pods.


Question: return kl.PodResourcesAreReclaimed(pod, status) } <del> // filterOutTerminatedPods returns the pods that could still have running // containers </del> <ins> // filterOutTerminatedPods returns pods that are not in a terminal phase // or are known to be fully terminated. This method should only be used // when the set of pods being filtered is upstream of the pod worker, i.e. // the pods the pod manager is aware of. </ins> func (kl *Kubelet) filterOutTerminatedPods(pods []*v1.Pod) []*v1.Pod { <del> var filteredPods []*v1.Pod </del> <ins> filteredPods := make([]*v1.Pod, 0, len(pods)) </ins> for _, p := range pods { <del> if !kl.podWorkers.CouldHaveRunningContainers(p.UID) { </del> <ins> if kl.podWorkers.IsPodKnownTerminated(p.UID) { continue } if p.Status.Phase == v1.PodSucceeded || p.Status.Phase == v1.PodFailed { </ins> continue } filteredPods = append(filteredPods, p)

Because the kubelet now considers terminating pods, it rejects those rapidly rescheduled pods. The follow up is more involved but requires the kubelet to allow pods to be potentially admitted that can't be run currently due to those terminating pods. See the linked issue.


Question: }, }, } <ins> if runtime.GOOS != "darwin" { // Skip this error case on darwin, see https://github.com/golang/go/issues/21512 errorCases["spec.timeZone: Invalid value: "AMERICA/new_york": unknown time zone AMERICA/new_york"] = batch.CronJob{ ObjectMeta: metav1.ObjectMeta{ Name:      "mycronjob", Namespace: metav1.NamespaceDefault, UID:       types.UID("1a2b3c"), }, Spec: batch.CronJobSpec{ Schedule:          "0 * * * *", TimeZone:          &timeZoneBadCasing, ConcurrencyPolicy: batch.AllowConcurrent, JobTemplate: batch.JobTemplateSpec{ Spec: batch.JobSpec{ Template: validPodTemplateSpec, }, }, }, } } </ins> for k, v := range errorCases { errs := ValidateCronJobCreate(&v, corevalidation.PodValidationOptions{})

test withnot FAIL In a macos. do or golang upstrem disscusion. We use time.LoadLocation to validate timezone setting in object spec like cronjob.(It is ok) We also add a test try to validate is not a crroct value. This test is not true sometimes, for example, in default macos this is a true time_zone value.(maybe also true in other case insensitive os, but I have no device to try) At least we should get the test to pass on macos, I hope. <details</details<details</details<details</details<details</details<details</details<details</details
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /sig test (I guess) Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig testing (I guess)
see
/remove-sig testing /sig apps
I was very surprised to find that this error only occurs on darwin, and does not occur on windows, although windows is also case-sensitive by default. When using , windows finally use (see ), but darwin and linux have more choices (see ) they will first try , using the method, which will call the system's open method to open , because darwin is case-insensitive by default, and this operation will succeed.
/triage accepted


Question: "os" "path/filepath" "runtime" <ins> "strings" </ins> "testing" "time"

Since upgrading from Kubernetes to we no longer get memory metrics from the endpoint on . Example: In our previous version, we would get a response like the following that included utilisation metrics: Hit the endpoint of on a cluster running version . Kubernetes version (use ): Cloud provider or hardware configuration: Provisioned on AWS with OS (e.g: ): Flatcar Container Linux by Kinvolk 2765.2.6 (Oklo) Kernel (e.g. ): 5.10.43-flatcar
/sig node
Hi AlexLastwhich container runtimes are you using? containerd?
Hi AlexLast, I have test on v1.22 with containerd, I did not reproduce this issue. If memory limit is set in the container spec of the pod, /stats/summary will report availableBytes, if memory limit is not set, /stats/summary will not be reported. just like this: ! ! code is here:
/cc We have reports of missing memory metrics on 1.20.8 and 1.21.2 now... was there a cadvisor or similar change that happened?
also /cc as an FYI
If you are using cri-o, it looks like a similar issue to and (possibly an issue with cadvisor/the way cri-o integrates with cadvisor). This would also apply to the dockershim
I'm not able to repro on GKE v1.20.8-gke.700 on containerd
Can you please provide more info about your environment, container runtime etc and possible repro steps? Thanks!
/triage needs-information
Hi, First of all, I am sorry to cut in on you. I have the same issue with Kubelet, in my case with Kubeadm. Here you are the issue: Here you are my installation: Thanks in advance
^ quick summary of the above, appears to be single-node ubuntu (not sure what kernel?) containerd 1.4.6 k8s 1.21.2 kubeadm installer
It is one node. It is only for making easier where is the problem .


Question: "os" "path/filepath" "runtime" <ins> "strings" </ins> "testing" "time"

Kubernetes version: 1.21.2 Cloud provider or hardware configuration: Bare metal (VMware) OS: Ubuntu 18.04.5 LTS Kernel: 4.15.0-147-generic Install tools: Kubeadm Network plugin and version (if this is a network-related bug): Weave 2.8.1 CRI: Containerd 1.4.6
Sure, we're setting the following flags for : Version: We deploy our clusters using , these missing metrics are for all pods in the cluster, including pods with memory requests and limits set.
Hitting the same issue on such kind of environments Then I tested and found no issue on environment
Besides memory metrics, we noticed a loss of network in pods. I commented at the block of container and the block of pod The loss of pods occurs at the same time as loss of pods
I created cluster as follows on 1.21.2 with kubetest containerd 1.4.6 on ubuntu. This is on older version of ubuntu 18.04 LTS on kernel I don't see memory metrics being lost here.
Can we have more details on exact kernel/OS image being used here? I can't seem to repro this on 1.21.2 with same containerd/ ubuntu version as mentioned in
Hi, may I know your output of network metrics in pods?
Hi, I have tested with two kernel versions: P.S: I have just updated my in order to make easier the test. You can use it (it is necessary to install and ). Thanks in advance
We tested with two kernel versions too.
I was able to repro finally -- it looks like missing key here is that you must be using the systemd cgroup driver. With this patch and starting cluster as follows:
Hi do you mean that if using systemd cgroup driver, this issue would happen. In my environment, both containerd and kubelet are using cgroupfs driver, but the issue still exists containerd config dump
Hi, Here you are my Containerd & Kubelet config: ` As far as I know, systemd cgroup driver should be configured in order to have a proper perfomance in Kubernetes. Is it a bug with Kubelet?


Question: "os" "path/filepath" "runtime" <ins> "strings" </ins> "testing" "time"

Thanks in advance
Hi thanks for the response, I've found the following in the , so will have a go at setting that parameter
Thanks for all the info! I was only able to repro this on systemd cgroup driver as mentioned in It did not repro for me when using cgroupfs. This will need some additional investigation to figure out the root cause and why memory metrics are not being reported correctly and see if this is a kubelet or cAdvisor issue. /cc
Hi, Thanks . By the way, there is a new issue reporting the same behaviour:
After setting kubelet's log verbosity level to 10, I found logs like Is this related? And I checked that according the log, all the pod sandboxID are missing
Thanks I was able to confirm that switching to fixed the missing metrics. From a bit of research, it's my understanding that using the driver on a node with is not recommended - can you confirm?
By stepping into the code, I found that the container info returned by this func is nil, so that the will always be false. Thus will always be . That explains the last log has expected fields. But because of 1), this func cannot be called. There's really a label-related issue on cadvisor: (Thanks With the workaround, I can recover the missed metrics
/remove-triage needs-information /triage accepted
Thanks for digging deeper into this I'm still a bit confused on the relationship between this and systemd cgroup driver. Do the labels being missing have something to do with systemd cgroup driver being used? Also are you using a different socket path for containerd i.e. not the default as mentioned in ?
We didn't use the default but used and not specified
Interesting! In the test I made in the container-runtime-endpoint is set to default (i.e. ).
With debugging, I found that on kubernetes 1.21.2 when using different cgroup driver, the cgroupPath format of containers are different. : systemd + containerd:cgroupfs : systemd + containerd: systemd : cgroupfs + containerd: cgroupfs The would return a map using the path.Base of cgroupPath Later, we use containerID to check to add metrics. So only in the third case, the key of are . That's how the cgroup driver impacts the metrics.


Question: "os" "path/filepath" "runtime" <ins> "strings" </ins> "testing" "time"

To sum up, and are two different cases that caused this issue. For the case, I think it's a bug we need fix: we should make sure the key of always be BTW do you know why the cgroupPath format are different when using different cgroup driver?
I believe it's expected that the cgroup naming will differ when using systemd cgroup driver vs cgroupfs. Kubelet internally has some logic for parsing and converting between the two different formats, e.g. and I left some comments on your proposed PR: Thanks for helping look into this issue!
/priority important-soon


Question: return lw.storage.WatchList(context.TODO(), lw.resourcePrefix, options.ResourceVersion, Everything) } <ins> // cacherWatch implements watch.Interface to return a single error type errWatcher struct { result chan watch.Event } func newErrWatcher(err error) *errWatcher { // Create an error event errEvent := watch.Event{Type: watch.Error} switch err := err.(type) { case runtime.Object: errEvent.Object = err case *errors.StatusError: errEvent.Object = &err.ErrStatus default: errEvent.Object = &unversioned.Status{ Status:  unversioned.StatusFailure, Message: err.Error(), Reason:  unversioned.StatusReasonInternalError, Code:    http.StatusInternalServerError, } } // Create a watcher with room for a single event, populate it, and close the channel watcher := &errWatcher{result: make(chan watch.Event, 1)} watcher.result <- errEvent close(watcher.result) return watcher } // Implements watch.Interface. func (c *errWatcher) ResultChan() <-chan watch.Event { return c.result } // Implements watch.Interface. func (c *errWatcher) Stop() { // no-op } </ins> // cacherWatch implements watch.Interface type cacheWatcher struct { sync.Mutex

When attempting to fetch a resource version that is too old, the behavior of watch (for both HTTP and web sockets) changed when watch cache was enabled. The watch cache returns a 410 - without the watch cache, we return 200 and then write an "error" entry to the watch stream. This broke a client that dependent on the watch behavior, and moreover is inconsistent with other errors we set in the registry. Also, for watch over web sockets, most javascript clients will be unable to get at a 410 error from the connection, because browsers don't get access to the data. We should restore the watch cache behavior to be consistent with the previous behavior as this is a regression in API semantics. Resource without watch cache, watching too old: Resource with watch cache, watching too old:
I think this is candidate for back port to 1.2 to restore the previous behavior.
since this broke your javascript client, FYI because of watch cache
+1 for backport
hmm - we enabled watch cache in 1.1 - so this should also be broken in 1.1
It's possible that it was always nondeterministic and just depends on how fast the call to the storage layer executes. I think all of our in-tree clients will handle it either way.
I'm fairly certain this never used to behave this way, because OpenShift hasn't enabled the watch catch and we have never had a report of this until we enabled the watch cache for Kube resources just recently. , Daniel Smith wrote:
I see what you mean about ordering - etcd remote call would almost never be fast enough to beat the watch event. In practice, I suspect no one ever saw this without the watch error. Either way, for web sockets if we don't fix this clients can't get this error (the browser eats the 410 during the websocket negotiation and client code can't get access). So it's at least broken for web sockets. , Clayton Coleman wrote:
Well, it should be easy to just not wait at all, maybe? I'm OK with changing this behavior.
Although, are web clients XSS hardened? I'm not sure that we're optimizing for them currently.
If you mean CSRF, it isn't an issue with bearer token auth.


Question: return lw.storage.WatchList(context.TODO(), lw.resourcePrefix, options.ResourceVersion, Everything) } <ins> // cacherWatch implements watch.Interface to return a single error type errWatcher struct { result chan watch.Event } func newErrWatcher(err error) *errWatcher { // Create an error event errEvent := watch.Event{Type: watch.Error} switch err := err.(type) { case runtime.Object: errEvent.Object = err case *errors.StatusError: errEvent.Object = &err.ErrStatus default: errEvent.Object = &unversioned.Status{ Status:  unversioned.StatusFailure, Message: err.Error(), Reason:  unversioned.StatusReasonInternalError, Code:    http.StatusInternalServerError, } } // Create a watcher with room for a single event, populate it, and close the channel watcher := &errWatcher{result: make(chan watch.Event, 1)} watcher.result <- errEvent close(watcher.result) return watcher } // Implements watch.Interface. func (c *errWatcher) ResultChan() <-chan watch.Event { return c.result } // Implements watch.Interface. func (c *errWatcher) Stop() { // no-op } </ins> // cacherWatch implements watch.Interface type cacheWatcher struct { sync.Mutex

Basic auth should already have a big warning around it saying "not safe with browsers"
Maybe I'm missing something, but it seems deterministic to me. This is the uncached watch impl: once the resourceVersion parses, there is no path that returns a direct error, so all errors would have to be returned as an event via the ResultChan(). I don't see us reading from that until after we've already written a 200 and flushed:
Having read the docs, I was very surprised that "type": "ERROR" notices are sent as 200 (even over plain http), with the semantic code e.g. 410 only represented under inside. I see there was good motivation here to stick to 200, but it should be called out in the documentation. doesn't show at all how watch returns errors, and it even says "clients must handle the case by recognizing the status code 410 Gone" which really makes one expect an actual HTTP 410.
I agree this should be called out in the docs. Can you file a bug there and/or send a PR? I don't think this ~4 year old issue is the best place to track that. (Good detective work finding this, though!)


Question: } info := infos[0] mapping := info.ResourceMapping() <del> printer, err := f.PrinterForMapping(cmd, false, nil, mapping, allNamespaces) </del> <ins> // no need to print namespace for root-scoped resources printWithNamespace := allNamespaces if mapping != nil && mapping.Scope.Name() == meta.RESTScopeNameRoot { printWithNamespace = false } printer, err := f.PrinterForMapping(cmd, false, nil, mapping, printWithNamespace) </ins> if err != nil { return err }

This is wrong, cluster scoped resources should never have a NAMESPACE column. We need to fix and add a test to ensure this doesn't regress.
The presence of the flag is what triggers this.
I would help fix it.
I mean to add a test after modification, but found that current test framework could not support the case: To cover the scene, we need a real NewHumanReadablePrinter in test case, to ensure the output format, and modify the related funcs in FakeFactory to support pass in param withNamespace. I feel this not like behavior of unit test to some extent. The loss may outweighs the gain to have such a big impact on the whole. What's your opinion
[MILESTONENOTIFIER] Milestone Removed Important: This issue was missing the label for more than 7 days. <details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details


Question: } } <del> err := utilfeature.DefaultFeatureGate.SetFromMap(o.config.FeatureGates) if err != nil { </del> <ins> if err := o.processHostnameOverrideFlag(); err != nil { return err } if err := utilfeature.DefaultFeatureGate.SetFromMap(o.config.FeatureGates); err != nil { </ins> return err } return nil } <ins> // processHostnameOverrideFlag processes hostname-override flag func (o *Options) processHostnameOverrideFlag() error { // Check if hostname-override flag is set and use value since configFile always overrides if len(o.hostnameOverride) > 0 { hostName := strings.TrimSpace(o.hostnameOverride) if len(hostName) == 0 { return fmt.Errorf("empty hostname-override is invalid") } o.config.HostnameOverride = strings.ToLower(hostName) } return nil } </ins> // Validate validates all the required options. func (o *Options) Validate(args []string) error { if len(args) != 0 {

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: kube-proxy came up cleanly but mis-configured iptable rules due to a mis-match in the hostname and node name within the cluster triggering a default behavior. In this case the hostname returned the fqdn of the host and the cluster only had the hostname. This resulted in the following warning messages when kube-proxy started. Which then presents itself as pods not being able to access clusterip ip addresses with no route errors. What you expected to happen: Would it be better to fail to in this scenario rather than try to fall back to some default? I spent a good amount of time trying to track down the Error messages that were a result of this before finally working back to taking a closer at the Warning message. How to reproduce it (as minimally and precisely as possible): Start the kube-proxy with set to an invalid hostname should recreate this issue. Anything else we need to know?: While I think we can chalk up this issue to mis-configuration the purpose of opening this issue is really to have a conversation about the default behavior. I've seen several other issues filed that appear to be related to this behavior: , , . Would throwing an error and exiting non-zero be a better default action when the nodeIP is nil? What use case(s) does a default of support? If there must be a default behavior here maybe the IP address on the interface associated with the default gateway is a better default than ? Maybe the issue isn't in the proxier code but the kube-proxy itself should error when it fails to get node info rather than a warning? Environment: I'm not sure how much of this is important to this but here's what my environment was when I ran into this. Kubernetes version: 1.8.5 Cloud provider or hardware configuration: Bare metal OS (e.g. from /etc/os-release): Centos 7 and Ubuntu 16.04
This is similar (though not the same symptom) to for which was supposed to fix it but was closed. One of the objections was up til now, kube-proxy has had no requirement that the hostname = nodename or even that the hostname be defined.


Question: } } <del> err := utilfeature.DefaultFeatureGate.SetFromMap(o.config.FeatureGates) if err != nil { </del> <ins> if err := o.processHostnameOverrideFlag(); err != nil { return err } if err := utilfeature.DefaultFeatureGate.SetFromMap(o.config.FeatureGates); err != nil { </ins> return err } return nil } <ins> // processHostnameOverrideFlag processes hostname-override flag func (o *Options) processHostnameOverrideFlag() error { // Check if hostname-override flag is set and use value since configFile always overrides if len(o.hostnameOverride) > 0 { hostName := strings.TrimSpace(o.hostnameOverride) if len(hostName) == 0 { return fmt.Errorf("empty hostname-override is invalid") } o.config.HostnameOverride = strings.ToLower(hostName) } return nil } </ins> // Validate validates all the required options. func (o *Options) Validate(args []string) error { if len(args) != 0 {

Making kube-proxy exit on failure to retrieve the node object might cause regressions in some legitimate use-cases. But I have no idea what those are. ?
I have a hard time imagining what the legit cases might be. I am confused, though - I only see this field being used in one very specific place, that would not have the effect you are claiming.
I can't see the error message from log? If kube-proxy failed to create kube-ipvs0 dummy interface, it will log something like "Failed to create dummy interface: kube-ipvs0, error:...".
This breaks Services of type and with . This is problematic for AWS environments where the kubelet is configured to override the hostname (AWS instances default to the short name and for the cloud provider integrations to work this needs to be overridden to the private fqdn of the host).
Any chance we could get this prioritized as a bug fix for v1.11?
cc
After putting a bit more thought into this, would it be possible to standardize on an environment variable that could be set through the downward api to override the default behavior of treating the hostname as the node name in the proxy? That would simplify the configuration on the kubeadm side (no need to override the command with the downward api, only setting the appropriate env variable), and would also provide a more accurate way to retrieve the actual node name for kube-proxy when deployed as a daemonset without the need to interact with the kubelet directly. I would suggest that we could use for this purpose. I'd also be happy to take a stab at implementing the changes needed for the v1.12 cycle.


Question: } } <del> err := utilfeature.DefaultFeatureGate.SetFromMap(o.config.FeatureGates) if err != nil { </del> <ins> if err := o.processHostnameOverrideFlag(); err != nil { return err } if err := utilfeature.DefaultFeatureGate.SetFromMap(o.config.FeatureGates); err != nil { </ins> return err } return nil } <ins> // processHostnameOverrideFlag processes hostname-override flag func (o *Options) processHostnameOverrideFlag() error { // Check if hostname-override flag is set and use value since configFile always overrides if len(o.hostnameOverride) > 0 { hostName := strings.TrimSpace(o.hostnameOverride) if len(hostName) == 0 { return fmt.Errorf("empty hostname-override is invalid") } o.config.HostnameOverride = strings.ToLower(hostName) } return nil } </ins> // Validate validates all the required options. func (o *Options) Validate(args []string) error { if len(args) != 0 {

Pods not getting scheduled on the new node joined to the cluster using kubespray The node joined to the cluster but new pods schedueld on the nodes are failing with error "get https://10.233.0.1 : dial tcp 10.233.0.1: 443 connection refused" Network plugin - calico kube-proxy running in ipvs mode new worker node, can connect to master server ip address over 6443 new worker node, cannot access cluster ip 10.233.0.1 over port 443 , nodelocaldns and kube-proxy pods are in running state on new node -proxy logs show error "invalid nodeIP, initializing kube-proxy with 127.0.0.1 as nodeIP" Unable to write event: 'Post https://127.0.0.1:6443/api/v1/namespaces/default/events dial tcp 127.0.0.1:6443 connection refused" -ln output do not show any services addr show output do not show kube-ipvs0 and calico interfaces Can someone please help. Thanks


Question: <ins> /* Copyright 2021 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package rest import ( "context" "testing" v1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/runtime" "k8s.io/apimachinery/pkg/runtime/schema" utilpointer "k8s.io/utils/pointer" ) type mockStrategy struct { RESTDeleteStrategy RESTGracefulDeleteStrategy } func (m mockStrategy) ObjectKinds(obj runtime.Object) ([]schema.GroupVersionKind, bool, error) { gvk := obj.GetObjectKind().GroupVersionKind() if len(gvk.Kind) == 0 { return nil, false, runtime.NewMissingKindErr("object has no kind field ") } if len(gvk.Version) == 0 { return nil, false, runtime.NewMissingVersionErr("object has no apiVersion field") } return []schema.GroupVersionKind{obj.GetObjectKind().GroupVersionKind()}, false, nil } func TestBeforeDelete(t *testing.T) { type args struct { strategy RESTDeleteStrategy ctx      context.Context pod      *v1.Pod options  *metav1.DeleteOptions } makePod := func(deletionGracePeriodSeconds int64) *v1.Pod { return &v1.Pod{ TypeMeta: metav1.TypeMeta{APIVersion: "v1", Kind: "Pod"}, ObjectMeta: metav1.ObjectMeta{ DeletionTimestamp:          &metav1.Time{}, DeletionGracePeriodSeconds: &deletionGracePeriodSeconds, }, } } makeOption := func(gracePeriodSeconds int64) *metav1.DeleteOptions { return &metav1.DeleteOptions{ GracePeriodSeconds: &gracePeriodSeconds, } } tests := []struct { name                           string args                           args wantGraceful                   bool wantGracefulPending            bool wantGracePeriodSeconds         *int64 wantDeletionGracePeriodSeconds *int64 wantErr                        bool }{ { name: "when DeletionGracePeriodSeconds=-1, GracePeriodSeconds=-1", args: args{ pod:     makePod(-1), options: makeOption(-1), }, // want 1 wantDeletionGracePeriodSeconds: utilpointer.Int64(1), wantGracePeriodSeconds:         utilpointer.Int64(1), wantGraceful:                   false, wantGracefulPending:            true, }, { name: "when DeletionGracePeriodSeconds=-1, GracePeriodSeconds=0", args: args{ pod:     makePod(-1), options: makeOption(0), }, // want 0 wantDeletionGracePeriodSeconds: utilpointer.Int64(0), wantGracePeriodSeconds:         utilpointer.Int64(0), wantGraceful:                   true, wantGracefulPending:            false, }, { name: "when DeletionGracePeriodSeconds=-1, GracePeriodSeconds=1", args: args{ pod:     makePod(-1), options: makeOption(1), }, // want 1 wantDeletionGracePeriodSeconds: utilpointer.Int64(1), wantGracePeriodSeconds:         utilpointer.Int64(1), wantGraceful:                   false, wantGracefulPending:            true, }, { name: "when DeletionGracePeriodSeconds=-1, GracePeriodSeconds=2", args: args{ pod:     makePod(-1), options: makeOption(2), }, // want 1 wantDeletionGracePeriodSeconds: utilpointer.Int64(1), wantGracePeriodSeconds:         utilpointer.Int64(2), wantGraceful:                   false, wantGracefulPending:            true, }, { name: "when DeletionGracePeriodSeconds=0, GracePeriodSeconds=-1", args: args{ pod:     makePod(0), options: makeOption(-1), }, // want 0 wantDeletionGracePeriodSeconds: utilpointer.Int64(0), wantGracePeriodSeconds:         utilpointer.Int64(1), wantGraceful:                   false, wantGracefulPending:            false, }, { name: "when DeletionGracePeriodSeconds=0, GracePeriodSeconds=0", args: args{ pod:     makePod(0), options: makeOption(0), }, // want 0 wantDeletionGracePeriodSeconds: utilpointer.Int64(0), wantGracePeriodSeconds:         utilpointer.Int64(0), wantGraceful:                   false, wantGracefulPending:            false, }, { name: "when DeletionGracePeriodSeconds=0, GracePeriodSeconds=1", args: args{ pod:     makePod(0), options: makeOption(1), }, // want 0 wantDeletionGracePeriodSeconds: utilpointer.Int64(0), wantGracePeriodSeconds:         utilpointer.Int64(1), wantGraceful:                   false, wantGracefulPending:            false, }, { name: "when DeletionGracePeriodSeconds=0, GracePeriodSeconds=2", args: args{ pod:     makePod(0), options: makeOption(2), }, // want 0 wantDeletionGracePeriodSeconds: utilpointer.Int64(0), wantGracePeriodSeconds:         utilpointer.Int64(2), wantGraceful:                   false, wantGracefulPending:            false, }, { name: "when DeletionGracePeriodSeconds=1, GracePeriodSeconds=-1", args: args{ pod:     makePod(1), options: makeOption(-1), }, // want 1 wantDeletionGracePeriodSeconds: utilpointer.Int64(1), wantGracePeriodSeconds:         utilpointer.Int64(1), wantGraceful:                   false, wantGracefulPending:            true, }, { name: "when DeletionGracePeriodSeconds=1, GracePeriodSeconds=0", args: args{ pod:     makePod(1), options: makeOption(0), }, // want 0 wantDeletionGracePeriodSeconds: utilpointer.Int64(0), wantGracePeriodSeconds:         utilpointer.Int64(0), wantGraceful:                   true, wantGracefulPending:            false, }, { name: "when DeletionGracePeriodSeconds=1, GracePeriodSeconds=1", args: args{ pod:     makePod(1), options: makeOption(1), }, // want 1 wantDeletionGracePeriodSeconds: utilpointer.Int64(1), wantGracePeriodSeconds:         utilpointer.Int64(1), wantGraceful:                   false, wantGracefulPending:            true, }, { name: "when DeletionGracePeriodSeconds=1, GracePeriodSeconds=2", args: args{ pod:     makePod(1), options: makeOption(2), }, // want 1 wantDeletionGracePeriodSeconds: utilpointer.Int64(1), wantGracePeriodSeconds:         utilpointer.Int64(2), wantGraceful:                   false, wantGracefulPending:            true, }, { name: "when DeletionGracePeriodSeconds=2, GracePeriodSeconds=-1", args: args{ pod:     makePod(2), options: makeOption(-1), }, // want 1 wantDeletionGracePeriodSeconds: utilpointer.Int64(1), wantGracePeriodSeconds:         utilpointer.Int64(1), wantGraceful:                   true, wantGracefulPending:            false, }, { name: "when DeletionGracePeriodSeconds=2, GracePeriodSeconds=0", args: args{ pod:     makePod(2), options: makeOption(0), }, // want 0 wantDeletionGracePeriodSeconds: utilpointer.Int64(0), wantGracePeriodSeconds:         utilpointer.Int64(0), wantGraceful:                   true, wantGracefulPending:            false, }, { name: "when DeletionGracePeriodSeconds=2, GracePeriodSeconds=1", args: args{ pod:     makePod(2), options: makeOption(1), }, // want 1 wantDeletionGracePeriodSeconds: utilpointer.Int64(1), wantGracePeriodSeconds:         utilpointer.Int64(1), wantGraceful:                   true, wantGracefulPending:            false, }, { name: "when DeletionGracePeriodSeconds=2, GracePeriodSeconds=2", args: args{ pod:     makePod(2), options: makeOption(2), }, // want 2 wantDeletionGracePeriodSeconds: utilpointer.Int64(2), wantGracePeriodSeconds:         utilpointer.Int64(2), wantGraceful:                   false, wantGracefulPending:            true, }, } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { if tt.args.strategy == nil { tt.args.strategy = mockStrategy{} } if tt.args.ctx == nil { tt.args.ctx = context.Background() } gotGraceful, gotGracefulPending, err := BeforeDelete(tt.args.strategy, tt.args.ctx, tt.args.pod, tt.args.options) if (err != nil) != tt.wantErr { t.Errorf("BeforeDelete() error = %v, wantErr %v", err, tt.wantErr) return } if gotGraceful != tt.wantGraceful { t.Errorf("BeforeDelete() gotGraceful = %v, want %v", gotGraceful, tt.wantGraceful) } if gotGracefulPending != tt.wantGracefulPending { t.Errorf("BeforeDelete() gotGracefulPending = %v, want %v", gotGracefulPending, tt.wantGracefulPending) } if gotGracefulPending != tt.wantGracefulPending { t.Errorf("BeforeDelete() gotGracefulPending = %v, want %v", gotGracefulPending, tt.wantGracefulPending) } if !utilpointer.Int64Equal(tt.args.pod.DeletionGracePeriodSeconds, tt.wantDeletionGracePeriodSeconds) { t.Errorf("metadata.DeletionGracePeriodSeconds = %v, want %v", utilpointer.Int64Deref(tt.args.pod.DeletionGracePeriodSeconds, 0), utilpointer.Int64Deref(tt.wantDeletionGracePeriodSeconds, 0)) } if !utilpointer.Int64Equal(tt.args.options.GracePeriodSeconds, tt.wantGracePeriodSeconds) { t.Errorf("options.GracePeriodSeconds = %v, want %v", utilpointer.Int64Deref(tt.args.options.GracePeriodSeconds, 0), utilpointer.Int64Deref(tt.wantGracePeriodSeconds, 0)) } }) } } </ins>

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Some k8s client tried to evict a pod by setting negative value for deletionGracePeriodSeconds. But all eventual delete calls (and force-delete calls) made by kubelet, pod-garbage-collector, etc were not able to delete that pod, because apiserver doesnt allow deletionGracePeriodSeconds to be increased once it is set. So force-delete (which set it to 0) doesn't take effect when it's already negative. Not even kubectl is able to call delete with negative grace-period. But curl with an even more negative grace-period value worked. What you expected to happen: force-delete works. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: I think the issue is in the code below As the comment suggest, in phase 1, the was set to a negative value (meaning was in the past), and in phase 2, has to be set even lower, otherwise, it will be considered as pending graceful deletion and not performing the immediate delete. I think there're two possible fixes. not allowing setting in the past ( should be non-negative). In that case, we could convert to 0 if negative. that could be set in the past. In that case, in phase 2, we treat negative value of the same as 0, and allow immediate delete. Thoughts? Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig api-machinery
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig api-machinery Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/cc /triage accepted
No component in the system should ever treat the timestamp of DeletionTimestamp as authoritative.


Question: <ins> /* Copyright 2021 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package rest import ( "context" "testing" v1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/runtime" "k8s.io/apimachinery/pkg/runtime/schema" utilpointer "k8s.io/utils/pointer" ) type mockStrategy struct { RESTDeleteStrategy RESTGracefulDeleteStrategy } func (m mockStrategy) ObjectKinds(obj runtime.Object) ([]schema.GroupVersionKind, bool, error) { gvk := obj.GetObjectKind().GroupVersionKind() if len(gvk.Kind) == 0 { return nil, false, runtime.NewMissingKindErr("object has no kind field ") } if len(gvk.Version) == 0 { return nil, false, runtime.NewMissingVersionErr("object has no apiVersion field") } return []schema.GroupVersionKind{obj.GetObjectKind().GroupVersionKind()}, false, nil } func TestBeforeDelete(t *testing.T) { type args struct { strategy RESTDeleteStrategy ctx      context.Context pod      *v1.Pod options  *metav1.DeleteOptions } makePod := func(deletionGracePeriodSeconds int64) *v1.Pod { return &v1.Pod{ TypeMeta: metav1.TypeMeta{APIVersion: "v1", Kind: "Pod"}, ObjectMeta: metav1.ObjectMeta{ DeletionTimestamp:          &metav1.Time{}, DeletionGracePeriodSeconds: &deletionGracePeriodSeconds, }, } } makeOption := func(gracePeriodSeconds int64) *metav1.DeleteOptions { return &metav1.DeleteOptions{ GracePeriodSeconds: &gracePeriodSeconds, } } tests := []struct { name                           string args                           args wantGraceful                   bool wantGracefulPending            bool wantGracePeriodSeconds         *int64 wantDeletionGracePeriodSeconds *int64 wantErr                        bool }{ { name: "when DeletionGracePeriodSeconds=-1, GracePeriodSeconds=-1", args: args{ pod:     makePod(-1), options: makeOption(-1), }, // want 1 wantDeletionGracePeriodSeconds: utilpointer.Int64(1), wantGracePeriodSeconds:         utilpointer.Int64(1), wantGraceful:                   false, wantGracefulPending:            true, }, { name: "when DeletionGracePeriodSeconds=-1, GracePeriodSeconds=0", args: args{ pod:     makePod(-1), options: makeOption(0), }, // want 0 wantDeletionGracePeriodSeconds: utilpointer.Int64(0), wantGracePeriodSeconds:         utilpointer.Int64(0), wantGraceful:                   true, wantGracefulPending:            false, }, { name: "when DeletionGracePeriodSeconds=-1, GracePeriodSeconds=1", args: args{ pod:     makePod(-1), options: makeOption(1), }, // want 1 wantDeletionGracePeriodSeconds: utilpointer.Int64(1), wantGracePeriodSeconds:         utilpointer.Int64(1), wantGraceful:                   false, wantGracefulPending:            true, }, { name: "when DeletionGracePeriodSeconds=-1, GracePeriodSeconds=2", args: args{ pod:     makePod(-1), options: makeOption(2), }, // want 1 wantDeletionGracePeriodSeconds: utilpointer.Int64(1), wantGracePeriodSeconds:         utilpointer.Int64(2), wantGraceful:                   false, wantGracefulPending:            true, }, { name: "when DeletionGracePeriodSeconds=0, GracePeriodSeconds=-1", args: args{ pod:     makePod(0), options: makeOption(-1), }, // want 0 wantDeletionGracePeriodSeconds: utilpointer.Int64(0), wantGracePeriodSeconds:         utilpointer.Int64(1), wantGraceful:                   false, wantGracefulPending:            false, }, { name: "when DeletionGracePeriodSeconds=0, GracePeriodSeconds=0", args: args{ pod:     makePod(0), options: makeOption(0), }, // want 0 wantDeletionGracePeriodSeconds: utilpointer.Int64(0), wantGracePeriodSeconds:         utilpointer.Int64(0), wantGraceful:                   false, wantGracefulPending:            false, }, { name: "when DeletionGracePeriodSeconds=0, GracePeriodSeconds=1", args: args{ pod:     makePod(0), options: makeOption(1), }, // want 0 wantDeletionGracePeriodSeconds: utilpointer.Int64(0), wantGracePeriodSeconds:         utilpointer.Int64(1), wantGraceful:                   false, wantGracefulPending:            false, }, { name: "when DeletionGracePeriodSeconds=0, GracePeriodSeconds=2", args: args{ pod:     makePod(0), options: makeOption(2), }, // want 0 wantDeletionGracePeriodSeconds: utilpointer.Int64(0), wantGracePeriodSeconds:         utilpointer.Int64(2), wantGraceful:                   false, wantGracefulPending:            false, }, { name: "when DeletionGracePeriodSeconds=1, GracePeriodSeconds=-1", args: args{ pod:     makePod(1), options: makeOption(-1), }, // want 1 wantDeletionGracePeriodSeconds: utilpointer.Int64(1), wantGracePeriodSeconds:         utilpointer.Int64(1), wantGraceful:                   false, wantGracefulPending:            true, }, { name: "when DeletionGracePeriodSeconds=1, GracePeriodSeconds=0", args: args{ pod:     makePod(1), options: makeOption(0), }, // want 0 wantDeletionGracePeriodSeconds: utilpointer.Int64(0), wantGracePeriodSeconds:         utilpointer.Int64(0), wantGraceful:                   true, wantGracefulPending:            false, }, { name: "when DeletionGracePeriodSeconds=1, GracePeriodSeconds=1", args: args{ pod:     makePod(1), options: makeOption(1), }, // want 1 wantDeletionGracePeriodSeconds: utilpointer.Int64(1), wantGracePeriodSeconds:         utilpointer.Int64(1), wantGraceful:                   false, wantGracefulPending:            true, }, { name: "when DeletionGracePeriodSeconds=1, GracePeriodSeconds=2", args: args{ pod:     makePod(1), options: makeOption(2), }, // want 1 wantDeletionGracePeriodSeconds: utilpointer.Int64(1), wantGracePeriodSeconds:         utilpointer.Int64(2), wantGraceful:                   false, wantGracefulPending:            true, }, { name: "when DeletionGracePeriodSeconds=2, GracePeriodSeconds=-1", args: args{ pod:     makePod(2), options: makeOption(-1), }, // want 1 wantDeletionGracePeriodSeconds: utilpointer.Int64(1), wantGracePeriodSeconds:         utilpointer.Int64(1), wantGraceful:                   true, wantGracefulPending:            false, }, { name: "when DeletionGracePeriodSeconds=2, GracePeriodSeconds=0", args: args{ pod:     makePod(2), options: makeOption(0), }, // want 0 wantDeletionGracePeriodSeconds: utilpointer.Int64(0), wantGracePeriodSeconds:         utilpointer.Int64(0), wantGraceful:                   true, wantGracefulPending:            false, }, { name: "when DeletionGracePeriodSeconds=2, GracePeriodSeconds=1", args: args{ pod:     makePod(2), options: makeOption(1), }, // want 1 wantDeletionGracePeriodSeconds: utilpointer.Int64(1), wantGracePeriodSeconds:         utilpointer.Int64(1), wantGraceful:                   true, wantGracefulPending:            false, }, { name: "when DeletionGracePeriodSeconds=2, GracePeriodSeconds=2", args: args{ pod:     makePod(2), options: makeOption(2), }, // want 2 wantDeletionGracePeriodSeconds: utilpointer.Int64(2), wantGracePeriodSeconds:         utilpointer.Int64(2), wantGraceful:                   false, wantGracefulPending:            true, }, } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { if tt.args.strategy == nil { tt.args.strategy = mockStrategy{} } if tt.args.ctx == nil { tt.args.ctx = context.Background() } gotGraceful, gotGracefulPending, err := BeforeDelete(tt.args.strategy, tt.args.ctx, tt.args.pod, tt.args.options) if (err != nil) != tt.wantErr { t.Errorf("BeforeDelete() error = %v, wantErr %v", err, tt.wantErr) return } if gotGraceful != tt.wantGraceful { t.Errorf("BeforeDelete() gotGraceful = %v, want %v", gotGraceful, tt.wantGraceful) } if gotGracefulPending != tt.wantGracefulPending { t.Errorf("BeforeDelete() gotGracefulPending = %v, want %v", gotGracefulPending, tt.wantGracefulPending) } if gotGracefulPending != tt.wantGracefulPending { t.Errorf("BeforeDelete() gotGracefulPending = %v, want %v", gotGracefulPending, tt.wantGracefulPending) } if !utilpointer.Int64Equal(tt.args.pod.DeletionGracePeriodSeconds, tt.wantDeletionGracePeriodSeconds) { t.Errorf("metadata.DeletionGracePeriodSeconds = %v, want %v", utilpointer.Int64Deref(tt.args.pod.DeletionGracePeriodSeconds, 0), utilpointer.Int64Deref(tt.wantDeletionGracePeriodSeconds, 0)) } if !utilpointer.Int64Equal(tt.args.options.GracePeriodSeconds, tt.wantGracePeriodSeconds) { t.Errorf("options.GracePeriodSeconds = %v, want %v", utilpointer.Int64Deref(tt.args.options.GracePeriodSeconds, 0), utilpointer.Int64Deref(tt.wantGracePeriodSeconds, 0)) } }) } } </ins>

In Kube, components are required to calculate the time period from when they observe the event in their local clock (i.e. 30s from now), not use the deletion timestamp. A negative value of deletion grace period should be rejected by the apiserver as an invalid value (bad request / invalid field).
attempts to take a missing or positive deletion grace period negative should be rejected, I agree. if a negative value is already persisted, I'd be inclined to allow a negative value in an update and clip to 0
As I mentioned in the PR, eviction is passing negatives through to the store: In kubectl, we specify the default value for GracePeriod is -1. In the case of delete, we simply don't set the value in the deleteOptions we send to the API. In the case of eviction, we send the -1 value, and that gets passed along to the store.
I don't see the -1 getting sent to the eviction API. It looks like values < 0 result in a nil value being sent to the API: I would not expect the API to accept a negative value as-is.
Looks like you are correct. Do the deletionOptions go through their own validation or through some kind of validation for the evict API? The original bug report mentions 'evict' though it's not clear if the eviction API was used or not.
ValidateDeleteOptions is called inside pod deletion, but did not check that the field was non-negative. I don't see any eviction-specific validation.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle rotten
/lifecycle rotten
Im investigating the and . Surprisingly see you here...


Question: } // Even if PodCIDR is assigned, but NetworkUnavailable condition is // set to true, we need to process the node to set the condition. <ins> networkUnavailableTaint := &v1.Taint{Key: algorithm.TaintNodeNetworkUnavailable, Effect: v1.TaintEffectNoSchedule} </ins> _, cond := v1node.GetNodeCondition(&newNode.Status, v1.NodeNetworkUnavailable) <del> if cond == nil || cond.Status != v1.ConditionFalse { </del> <ins> if cond == nil || cond.Status != v1.ConditionFalse || !utiltaints.TaintExists(newNode.Spec.Taints, networkUnavailableTaint) { </ins> return ca.AllocateOrOccupyCIDR(newNode) } return nil

forked from As from the discussion with and we have to start checking taints wherever we check for node.Spec.Unschedulable. Also overlaps on some parts with /sig node /assign
cc
No sure the discussion history :); personally, prefer to deprecate and replace it with taint.
- same here, the goal would be to deprecate its usage, start checking taints everywhere unschedulable is used. Also, as long as we as we have v1, we might have to keep checking for both.
great, there's same subtask in my mind: deprecated description to the taint for command with taint others ??
- yup, but for 4 it is more something like check taints wherever we checn for unschedulable, since we cannot just replace until v1 is gone.
, my major point is to handle it task by task to make sure we did not miss anything :).
yes we definitely should
- should we define a specific taint for unschedulable (key and effect) ? (i.e what happens when a user uses cordon/uncordon)
I'm not sure about having specific Taint for Unschedulable is a good idea. Pretty much any Taint with NoSchedule effect will do (and this is what we should be checking).
- I agree, but in case of we can add a taint with key or something like that with a effect. This might help us especially if the node has no taints.
, not sure whether we should handle specially; AFAIK, set this flag for node maintenance, this taint should not be tolerated.
The whole point of moving from to Taints is to make things more configurable. as they're decision makers here.
If we are discussing about deprecate/re-implement by Unschedulable taint, +1 as it make sense. Current cordon is causing confusion in community since you can always taint it instead, and tainting is what most Kubernetes setup tool is doing today.
I agree with that we should use NoSchedule taint. I also agree that generally no pod should tolerate NoSchedule taints, but they may want to do so if the desire, knowing that anything could happen to the node they are scheduled on. For example, today DS pods tolerate "Unschedulable" bit.


Question: } // Even if PodCIDR is assigned, but NetworkUnavailable condition is // set to true, we need to process the node to set the condition. <ins> networkUnavailableTaint := &v1.Taint{Key: algorithm.TaintNodeNetworkUnavailable, Effect: v1.TaintEffectNoSchedule} </ins> _, cond := v1node.GetNodeCondition(&newNode.Status, v1.NodeNetworkUnavailable) <del> if cond == nil || cond.Status != v1.ConditionFalse { </del> <ins> if cond == nil || cond.Status != v1.ConditionFalse || !utiltaints.TaintExists(newNode.Spec.Taints, networkUnavailableTaint) { </ins> return ca.AllocateOrOccupyCIDR(newNode) } return nil

They may as well want to tolerate "NoSchedule" taints.
/sig scheduling
Agree; it's up to 'user' to decide whether tolerates the taint, we just need to taint it and doc what's happened in node for taints :)
We should have a mechanism that allows a cluster admin to limit types of tolerations that users can set. This is needed to prevent abuse in multi-tenant clusters. For example, cluster admin may want to taint nodes with rare resources to prevent pods from certain group of users to land on them. Users shouldn't be able to tolerate such taints. That said, this is a separate topic of its own which is related to policy and should be discussed separately.
- we’re already facing this in production for our spare nodes. I’ll bring this up in our next sig-meeting, I’ll also file an issue for this


Question: }, &api.Namespace{}, store, <del> 0, </del> <ins> 5*time.Minute, </ins> ) reflector.Run() return &lifecycle{ <del> Handler:            admission.NewHandler(admission.Create, admission.Delete), </del> <ins> Handler:            admission.NewHandler(admission.Create, admission.Update, admission.Delete), </ins> client:             c, store:              store, immortalNamespaces: util.NewStringSet(api.NamespaceDefault),

We should have a single NamespaceLifecycle plugin that enforces the Namespace rules now that all providers are off NamespaceAutoProvision. We may also want to make this no longer a user choice to configure and hard-wire the server to always run this check first to simplify configuration errors. For an example


Question: subnet: netip.MustParsePrefix("fd00:1:2:3::/64"), want:   netip.MustParseAddr("fd00:1:2:3:FFFF:FFFF:FFFF:FFFF"), }, <ins> { name:   "ipv6 00fc::/112", subnet: netip.MustParsePrefix("00fc::/112"), want:   netip.MustParseAddr("fc::ffff"), }, { name:   "ipv6 fc00::/112", subnet: netip.MustParsePrefix("fc00::/112"), want:   netip.MustParseAddr("fc00::ffff"), }, </ins> } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) {

We ran into an issue with an old version of k8s 1.18 that we were running where subnet for cluster IP is given as . The control plane comes up just fine but then when you try to create a k8s service, it fails due to a broken behavior with the IP Allocator Since this is not really a supported version of the k8s, I went around digging up a bit more to see what is the regular behavior and found this to be very inconsistent across a few release branches. bash go test -timeout 30m -run ^TestAllocateNextFC$ -v === RUN TestAllocateNextFC ? --- PASS: TestAllocateNextFC (0.00s) PASS ok 0.566s go func TestAllocateNextFC(t *testing.T) { , cidr, err := netutils.ParseCIDRSloppy("fc::/112") if err != nil { t.Fatal(err) } // Need to be replaced with the suitable handler for that version of the branch r, err := newTestAllocator(cidr) if err != nil { t.Fatal(err) } defer r.Destroy() ip, err := r.AllocateNext() if err != nil { t.Fatalf("%s", err.Error()) } fmt.Println(ip.String()) } Add the above Test case under the IP allocator test case No response_ <detailsNA </details<detailsNA </details<detailsNA </details<detailsNA </details<detailsNA </details<detailsNA </details
/sig network /assign /kind bug
Thanks for helping me with this issue.
/triage accepted /assign


Question: // 1.  spec.containers[*].image // 2.  spec.initContainers[*].image // 3.  spec.activeDeadlineSeconds <ins> // 4.  spec.terminationGracePeriodSeconds </ins> containerErrs, stop := ValidateContainerUpdates(newPod.Spec.Containers, oldPod.Spec.Containers, specPath.Child("containers")) allErrs = append(allErrs, containerErrs...)


Question: // 1.  spec.containers[*].image // 2.  spec.initContainers[*].image // 3.  spec.activeDeadlineSeconds <ins> // 4.  spec.terminationGracePeriodSeconds </ins> containerErrs, stop := ValidateContainerUpdates(newPod.Spec.Containers, oldPod.Spec.Containers, specPath.Child("containers")) allErrs = append(allErrs, containerErrs...)


Question: } } <del> // LocalEndpoint returns the full path to a windows named pipe func LocalEndpoint(path, file string) string { u := url.URL{ Scheme: npipeProtocol, Path:   path, } return u.String() + "//./pipe/" + file </del> <ins> // LocalEndpoint empty implementation func LocalEndpoint(path, file string) (string, error) { return "", fmt.Errorf("LocalEndpoints are unsupported in this build") </ins> } var tickCount = syscall.NewLazyDLL("kernel32.dll").NewProc("GetTickCount64")

What happened: I replaced the kubelet on one of my 1.14 nodes with a build from master, and now I'm getting this error starting up. Test passes since 5/31/2019 commit have been failing on testgrid. If you look at the logs, the Windows nodes are not coming up. What you expected to happen: No errors starting up kubelet How to reproduce it (as minimally and precisely as possible): Build master, then try to run the kubelet on Windows Anything else we need to know?: /sig windows Environment: Kubernetes version (use ): Cloud provider or hardware configuration: Azure OS (e.g: ): Windows Server 2019 Kernel (e.g. ): 10.0..529 Install tools: aks-engine
cc /priority critical-urgent
/milestone v1.15 Release team - it looks like a recent PR broke windows nodes. We will need a fix here for 1.15
Ok, so the problem here is a little more complicated and we're actually seeing a bug that is actually in the code for quite some time, just that we never used that particular code path. But first: TL;DR: Quick fix is to start the kubelet on Windows with: --feature-gates="KubeletPodResources=false" Now, as to why this is happening: PR [1] was merged 3 days ago and it graduates KubeletPodResources feature as Beta. That means kubelet will enable it by default.With the introduction of the above feature, this branch of code is entered when starting kubelet [2]. We can see a call to the function: defined here [3].While innocent enough, gets the necessary npipe by calling: . is defined here [4] for Windows . The problem is actually that url.URL can't handle Windows style paths [5] and will replace "" with "%5C" as it's to be expected. The paths are transformed along the way from the default "/var/lib/kubelet" to "varlibkubelet" by defined here [6]. More specifically in the use of to create a path. It's inputs are unix style but returns the path Windows style as it's to be expected.The solution is to replace the usage of url.URL in with some alternative package or just implement it.


Question: } } <del> // LocalEndpoint returns the full path to a windows named pipe func LocalEndpoint(path, file string) string { u := url.URL{ Scheme: npipeProtocol, Path:   path, } return u.String() + "//./pipe/" + file </del> <ins> // LocalEndpoint empty implementation func LocalEndpoint(path, file string) (string, error) { return "", fmt.Errorf("LocalEndpoints are unsupported in this build") </ins> } var tickCount = syscall.NewLazyDLL("kernel32.dll").NewProc("GetTickCount64")

[1] [2] [3] [4] [5] [6]
Thanks for the analysis! I was a bit concerned around the fact that this started showing up around my commit: but the root cause here is indeed, as you said, the graduation of feature to Beta and the way the invalid escape sequences are being introduced.
GCE tests failing too /cc
I'm trying a quick to see if anything else is broken. There are other functions in that same file doing that substitution already so it may make sense to use that as a workaround until a better solution is available.
Still working on it - looks like the path to create the directories is also busted. With still getting


Question: It is planned to improve accounting for resources which are shared by all containers in a pod, such as [EmptyDir volumes](volumes.md#emptydir). <del> The current system only supports container limits for CPU and Memory. </del> <ins> The current system only supports container requests and limits for CPU and Memory. </ins> It is planned to add new resource types, including a node disk space resource, and a framework for adding custom [resource types](../design/resources.md#resource-types). <del> The current system does not facilitate overcommitment of resources because resources reserved with container limits are assured.  It is planned to support multiple levels of [Quality of Service](http://issue.k8s.io/168). </del> <ins> Kubernetes supports overcommitment of resources by supporting multiple levels of [Quality of Service](http://issue.k8s.io/168). </ins> Currently, one unit of CPU means different things on different cloud providers, and on different machine types within the same cloud providers.  For example, on AWS, the capacity of a node

did which addresses some of them, but turns up a lot more.
Alright, I will do that.


Question: "k8s.io/kubernetes/pkg/watch" "github.com/golang/glog" <ins> "net/http" </ins> ) const maxTriesPerEvent = 12

I keep seeing these server rejected event errors every now and then. My guess is it is trying to update an event that was expired in etcd. However I don't expect an error, since there is specific logic in to handle that use case. Any ideas why this may happening? I am using kubernetes 1.2. Here is an example, from a host getting to a cluster:
Found the problem - the recordEvent logic in was expecting the wrong status code. I've submitted a PR for it. This could be reproduced by: 1 - Update load balancers by remove a node, then adding it back 2 - Remove all events manually from etcd for that namespace 3 - Remove node and add it back again - this results in server rejected event error


Question: "k8s.io/kubernetes/pkg/scheduler/apis/config/scheme" ) <del> // Default creates a default configuration of the latset versioned type. </del> <ins> // Default creates a default configuration of the latest versioned type. </ins> // This function needs to be updated whenever we bump the scheduler's component config version. func Default() (*config.KubeSchedulerConfiguration, error) { versionedCfg := v1beta2.KubeSchedulerConfiguration{}

ci-kubernetes-e2e-gci-gce-alpha-features kubetest.Up Jul 02 2021, 01:26:42 UTC+0200 /sig release
based on testgrid probably cc <img width="1203" alt="Screen Shot 2021-07-02 at 4 24 25 PM" src="https://user-"
/cc working on it
/assign
the scheduler fails to start with this error: it seems the feature gates from the command line does not take effect in the defaulting phrase, do you know how to fix it?
ah, found there is a cycle here: The scheduler options , but the default config's default values depend on the feature gates options. I think this disables other feature gate controlled logic in the defaulting too:
I tried to fix it by updating the component config after the CLI flags (including the feature gates) are parsed: PTAL
fixed the flake. is back green now.
thanks!