Question:
getNodeConditionPredicate() in makes our code hard to understand because it hides the node-condition-based filtering in the node lister, which is totally non-obvious. We should get rid of this function and have NodeController add NoSchedule taints for these situations instead. (Alas, we might not actually be able to get rid of getNodeConditionPredicate() completely until we get rid of the Unschedulable field of PodSpec, but at least we can get rid of all the other code in this function.) If there are pods that should still be able to schedule in any of these situations (e.g. DaemonSet pods?) we should add tolerations in admission control for them (e.g. see ). cc/ cc/ Sub-tasks according to the design doc: [x] Add node taints label and feature flag () [x] In the node controller, taint Nodes according to the Node Condition () [x] In DaemonSet, update DaemonSetController to Tolerant new Taints () [x] In admissionController, add MemoryPressure/DiskPressure toleration for no BestEffort pod () [x] In scheduler, disable the current behaviour of filtering out Nodes. Instead, pods will not be scheduled to tainted nodes if a toleration does not exist in its PodSpec () [x] Add e2e test for this feature () [x] Update doc for this feature ()
I think you are referring Unschedulable field of NodeSpec?
Yes sorry, that was a typo.
- please CC me to all PRs
See also
Note that memory and disk pressure are checked in separate predicates, not in the NodeConditionPredicate. These should be represented as NoSchedule taints too.
Setting NoSchedule taints corresponding to the node conditions that today cause pods not to schedule onto a node, should also allow us to use tolerations rather than special-case code in the DaemonSet controller (and thereby make it easier to have DaemonSet pods be scheduled by the default scheduler.)
ref/
The task list was moved to the issue description .
/cc
/reopen
you can't re-open an issue/PR unless you authored it or you are assigned to it. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/reopen
/reopen
Currently we already have toleration on dsc for taint related to node problems (e.g NodeNotReady and NodeUnreachable). But to me ds should respect the new taints (outOfDisk, NodeMemoryPressure etc..), any insights ?
/reopen
Yes, it's one of sub-task of this project. This issue is closed by 'partially fixes' :(, I had updated PRs to 'part of #'.
- I can handle the dsc tolerations if you want. cc to keep you on the loop.
, sure, thanks very much :).
- just saw that you moved node conditions into predicate func, once adding node taiting and the according tolerations to dsc, do we'll still need to have node conditions checking ? (since the we're tainting by node condition). Also, correct me if I'm wrong, but from what I understood: dsc tolerates new taints with effect noExec since we don't wanna evict any already-running pods ?
We won't, but it'll take time for us to get there. We'll roll out Tainting as alpha (obviously), later move it to beta and GA in 1.10 at the earliest. We can't remove Condition checking logic before that. dsc is a daemon set right? New taints are NoSchedule only, so running Pods won't be affected (at least on the control plane level). Kubelet can still decide to evict them based on problem it observes.
- seems logic for the timeline. From sub-tasks Since new Taints are NoSchedule Only, Daemon set shouldn't have toleration for these ? It should instead respect the new taints when applied and not schedule on these nodes, or I'm I missing something ?
The task for DS in my mind: For all DS pods, it should tolerant MemoryPressure & DiskPressure For critical DS pods, it should tolerant OutOfDisk And if the taint by condition feature is enabled, we should check OutOfDisk by taints instead of predicates; the refactor is trying to make this simpler :).
- You're right, I'll send a PR after yours get merged :)
Do we have a to do list so folks can help out with I'd like to help when I get some time. I have a question about timeline. If alpha makes it into v1.8, will I be able to schedule to NotReady nodes (no CNI) if I reference the alpha tolerations? Or do you mean it won't be functional at all until v1.10?
It means you'll need to flip feature gate flag for it to work.
, here is the task list: I'm working on related PRs for them; please ping me when you got time :).
- i anticipate we will also add a CPUPressure condition which will also mean that no BestEffort pod can be scheduled to that node if the kubelet is running with a static cpu assignment policy. /cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
I think it's done (as alpha). - do we want to keep this issue open?
We can close this.

return false, false, false, nil } <del> // TODO: Move it to the predicates for _, c := range node.Status.Conditions { if critical { break } // TODO: There are other node status that the DaemonSet should ideally respect too, //       e.g. MemoryPressure, and DiskPressure if c.Type == v1.NodeOutOfDisk && c.Status == v1.ConditionTrue { // the kubelet will evict this pod if it needs to. Let kubelet // decide whether to continue running this pod so leave shouldContinueRunning // set to true shouldSchedule = false } } </del> reasons, nodeInfo, err := dsc.simulate(newPod, node, ds) if err != nil { glog.Warningf("DaemonSet Predicates failed on node %s for ds '%s/%s' due to unexpected error: %v", node.Name, ds.ObjectMeta.Namespace, ds.ObjectMeta.Name, err)


Question:
getNodeConditionPredicate() in makes our code hard to understand because it hides the node-condition-based filtering in the node lister, which is totally non-obvious. We should get rid of this function and have NodeController add NoSchedule taints for these situations instead. (Alas, we might not actually be able to get rid of getNodeConditionPredicate() completely until we get rid of the Unschedulable field of PodSpec, but at least we can get rid of all the other code in this function.) If there are pods that should still be able to schedule in any of these situations (e.g. DaemonSet pods?) we should add tolerations in admission control for them (e.g. see ). cc/ cc/ Sub-tasks according to the design doc: [x] Add node taints label and feature flag () [x] In the node controller, taint Nodes according to the Node Condition () [x] In DaemonSet, update DaemonSetController to Tolerant new Taints () [x] In admissionController, add MemoryPressure/DiskPressure toleration for no BestEffort pod () [x] In scheduler, disable the current behaviour of filtering out Nodes. Instead, pods will not be scheduled to tainted nodes if a toleration does not exist in its PodSpec () [x] Add e2e test for this feature () [x] Update doc for this feature ()
I think you are referring Unschedulable field of NodeSpec?
Yes sorry, that was a typo.
- please CC me to all PRs
See also
Note that memory and disk pressure are checked in separate predicates, not in the NodeConditionPredicate. These should be represented as NoSchedule taints too.
Setting NoSchedule taints corresponding to the node conditions that today cause pods not to schedule onto a node, should also allow us to use tolerations rather than special-case code in the DaemonSet controller (and thereby make it easier to have DaemonSet pods be scheduled by the default scheduler.)
ref/
The task list was moved to the issue description .
/cc
/reopen
you can't re-open an issue/PR unless you authored it or you are assigned to it. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/reopen
/reopen
Currently we already have toleration on dsc for taint related to node problems (e.g NodeNotReady and NodeUnreachable). But to me ds should respect the new taints (outOfDisk, NodeMemoryPressure etc..), any insights ?
/reopen
Yes, it's one of sub-task of this project. This issue is closed by 'partially fixes' :(, I had updated PRs to 'part of #'.
- I can handle the dsc tolerations if you want. cc to keep you on the loop.
, sure, thanks very much :).
- just saw that you moved node conditions into predicate func, once adding node taiting and the according tolerations to dsc, do we'll still need to have node conditions checking ? (since the we're tainting by node condition). Also, correct me if I'm wrong, but from what I understood: dsc tolerates new taints with effect noExec since we don't wanna evict any already-running pods ?
We won't, but it'll take time for us to get there. We'll roll out Tainting as alpha (obviously), later move it to beta and GA in 1.10 at the earliest. We can't remove Condition checking logic before that. dsc is a daemon set right? New taints are NoSchedule only, so running Pods won't be affected (at least on the control plane level). Kubelet can still decide to evict them based on problem it observes.
- seems logic for the timeline. From sub-tasks Since new Taints are NoSchedule Only, Daemon set shouldn't have toleration for these ? It should instead respect the new taints when applied and not schedule on these nodes, or I'm I missing something ?
The task for DS in my mind: For all DS pods, it should tolerant MemoryPressure & DiskPressure For critical DS pods, it should tolerant OutOfDisk And if the taint by condition feature is enabled, we should check OutOfDisk by taints instead of predicates; the refactor is trying to make this simpler :).
- You're right, I'll send a PR after yours get merged :)
Do we have a to do list so folks can help out with I'd like to help when I get some time. I have a question about timeline. If alpha makes it into v1.8, will I be able to schedule to NotReady nodes (no CNI) if I reference the alpha tolerations? Or do you mean it won't be functional at all until v1.10?
It means you'll need to flip feature gate flag for it to work.
, here is the task list: I'm working on related PRs for them; please ping me when you got time :).
- i anticipate we will also add a CPUPressure condition which will also mean that no BestEffort pod can be scheduled to that node if the kubelet is running with a static cpu assignment policy. /cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
I think it's done (as alpha). - do we want to keep this issue open?
We can close this.

} var insufficientResourceErr error for _, r := range reasons { glog.V(4).Infof("DaemonSet Predicates failed on node %s for ds '%s/%s' for reason: %v", node.Name, ds.ObjectMeta.Namespace, ds.ObjectMeta.Name, r.GetReason()) switch reason := r.(type) {


Question:
getNodeConditionPredicate() in makes our code hard to understand because it hides the node-condition-based filtering in the node lister, which is totally non-obvious. We should get rid of this function and have NodeController add NoSchedule taints for these situations instead. (Alas, we might not actually be able to get rid of getNodeConditionPredicate() completely until we get rid of the Unschedulable field of PodSpec, but at least we can get rid of all the other code in this function.) If there are pods that should still be able to schedule in any of these situations (e.g. DaemonSet pods?) we should add tolerations in admission control for them (e.g. see ). cc/ cc/ Sub-tasks according to the design doc: [x] Add node taints label and feature flag () [x] In the node controller, taint Nodes according to the Node Condition () [x] In DaemonSet, update DaemonSetController to Tolerant new Taints () [x] In admissionController, add MemoryPressure/DiskPressure toleration for no BestEffort pod () [x] In scheduler, disable the current behaviour of filtering out Nodes. Instead, pods will not be scheduled to tainted nodes if a toleration does not exist in its PodSpec () [x] Add e2e test for this feature () [x] Update doc for this feature ()
I think you are referring Unschedulable field of NodeSpec?
Yes sorry, that was a typo.
- please CC me to all PRs
See also
Note that memory and disk pressure are checked in separate predicates, not in the NodeConditionPredicate. These should be represented as NoSchedule taints too.
Setting NoSchedule taints corresponding to the node conditions that today cause pods not to schedule onto a node, should also allow us to use tolerations rather than special-case code in the DaemonSet controller (and thereby make it easier to have DaemonSet pods be scheduled by the default scheduler.)
ref/
The task list was moved to the issue description .
/cc
/reopen
you can't re-open an issue/PR unless you authored it or you are assigned to it. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/reopen
/reopen
Currently we already have toleration on dsc for taint related to node problems (e.g NodeNotReady and NodeUnreachable). But to me ds should respect the new taints (outOfDisk, NodeMemoryPressure etc..), any insights ?
/reopen
Yes, it's one of sub-task of this project. This issue is closed by 'partially fixes' :(, I had updated PRs to 'part of #'.
- I can handle the dsc tolerations if you want. cc to keep you on the loop.
, sure, thanks very much :).
- just saw that you moved node conditions into predicate func, once adding node taiting and the according tolerations to dsc, do we'll still need to have node conditions checking ? (since the we're tainting by node condition). Also, correct me if I'm wrong, but from what I understood: dsc tolerates new taints with effect noExec since we don't wanna evict any already-running pods ?
We won't, but it'll take time for us to get there. We'll roll out Tainting as alpha (obviously), later move it to beta and GA in 1.10 at the earliest. We can't remove Condition checking logic before that. dsc is a daemon set right? New taints are NoSchedule only, so running Pods won't be affected (at least on the control plane level). Kubelet can still decide to evict them based on problem it observes.
- seems logic for the timeline. From sub-tasks Since new Taints are NoSchedule Only, Daemon set shouldn't have toleration for these ? It should instead respect the new taints when applied and not schedule on these nodes, or I'm I missing something ?
The task for DS in my mind: For all DS pods, it should tolerant MemoryPressure & DiskPressure For critical DS pods, it should tolerant OutOfDisk And if the taint by condition feature is enabled, we should check OutOfDisk by taints instead of predicates; the refactor is trying to make this simpler :).
- You're right, I'll send a PR after yours get merged :)
Do we have a to do list so folks can help out with I'd like to help when I get some time. I have a question about timeline. If alpha makes it into v1.8, will I be able to schedule to NotReady nodes (no CNI) if I reference the alpha tolerations? Or do you mean it won't be functional at all until v1.10?
It means you'll need to flip feature gate flag for it to work.
, here is the task list: I'm working on related PRs for them; please ping me when you got time :).
- i anticipate we will also add a CPUPressure condition which will also mean that no BestEffort pod can be scheduled to that node if the kubelet is running with a static cpu assignment policy. /cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
I think it's done (as alpha). - do we want to keep this issue open?
We can close this.

var emitEvent bool // we try to partition predicates into two partitions here: intentional on the part of the operator and not. switch reason { <ins> case predicates.ErrNodeOutOfDisk: // the kubelet will evict this pod if it needs to. Let kubelet // decide whether to continue running this pod so leave shouldContinueRunning // set to true shouldSchedule = false </ins> // intentional case predicates.ErrNodeSelectorNotMatch,


Question:
getNodeConditionPredicate() in makes our code hard to understand because it hides the node-condition-based filtering in the node lister, which is totally non-obvious. We should get rid of this function and have NodeController add NoSchedule taints for these situations instead. (Alas, we might not actually be able to get rid of getNodeConditionPredicate() completely until we get rid of the Unschedulable field of PodSpec, but at least we can get rid of all the other code in this function.) If there are pods that should still be able to schedule in any of these situations (e.g. DaemonSet pods?) we should add tolerations in admission control for them (e.g. see ). cc/ cc/ Sub-tasks according to the design doc: [x] Add node taints label and feature flag () [x] In the node controller, taint Nodes according to the Node Condition () [x] In DaemonSet, update DaemonSetController to Tolerant new Taints () [x] In admissionController, add MemoryPressure/DiskPressure toleration for no BestEffort pod () [x] In scheduler, disable the current behaviour of filtering out Nodes. Instead, pods will not be scheduled to tainted nodes if a toleration does not exist in its PodSpec () [x] Add e2e test for this feature () [x] Update doc for this feature ()
I think you are referring Unschedulable field of NodeSpec?
Yes sorry, that was a typo.
- please CC me to all PRs
See also
Note that memory and disk pressure are checked in separate predicates, not in the NodeConditionPredicate. These should be represented as NoSchedule taints too.
Setting NoSchedule taints corresponding to the node conditions that today cause pods not to schedule onto a node, should also allow us to use tolerations rather than special-case code in the DaemonSet controller (and thereby make it easier to have DaemonSet pods be scheduled by the default scheduler.)
ref/
The task list was moved to the issue description .
/cc
/reopen
you can't re-open an issue/PR unless you authored it or you are assigned to it. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/reopen
/reopen
Currently we already have toleration on dsc for taint related to node problems (e.g NodeNotReady and NodeUnreachable). But to me ds should respect the new taints (outOfDisk, NodeMemoryPressure etc..), any insights ?
/reopen
Yes, it's one of sub-task of this project. This issue is closed by 'partially fixes' :(, I had updated PRs to 'part of #'.
- I can handle the dsc tolerations if you want. cc to keep you on the loop.
, sure, thanks very much :).
- just saw that you moved node conditions into predicate func, once adding node taiting and the according tolerations to dsc, do we'll still need to have node conditions checking ? (since the we're tainting by node condition). Also, correct me if I'm wrong, but from what I understood: dsc tolerates new taints with effect noExec since we don't wanna evict any already-running pods ?
We won't, but it'll take time for us to get there. We'll roll out Tainting as alpha (obviously), later move it to beta and GA in 1.10 at the earliest. We can't remove Condition checking logic before that. dsc is a daemon set right? New taints are NoSchedule only, so running Pods won't be affected (at least on the control plane level). Kubelet can still decide to evict them based on problem it observes.
- seems logic for the timeline. From sub-tasks Since new Taints are NoSchedule Only, Daemon set shouldn't have toleration for these ? It should instead respect the new taints when applied and not schedule on these nodes, or I'm I missing something ?
The task for DS in my mind: For all DS pods, it should tolerant MemoryPressure & DiskPressure For critical DS pods, it should tolerant OutOfDisk And if the taint by condition feature is enabled, we should check OutOfDisk by taints instead of predicates; the refactor is trying to make this simpler :).
- You're right, I'll send a PR after yours get merged :)
Do we have a to do list so folks can help out with I'd like to help when I get some time. I have a question about timeline. If alpha makes it into v1.8, will I be able to schedule to NotReady nodes (no CNI) if I reference the alpha tolerations? Or do you mean it won't be functional at all until v1.10?
It means you'll need to flip feature gate flag for it to work.
, here is the task list: I'm working on related PRs for them; please ping me when you got time :).
- i anticipate we will also add a CPUPressure condition which will also mean that no BestEffort pod can be scheduled to that node if the kubelet is running with a static cpu assignment policy. /cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
I think it's done (as alpha). - do we want to keep this issue open?
We can close this.

if !fit { predicateFails = append(predicateFails, reasons...) } if critical { // If the pod is marked as critical and support for critical pod annotations is enabled, // check predicates for critical pods only. fit, reasons, err = predicates.EssentialPredicates(pod, nil, nodeInfo) } else { fit, reasons, err = predicates.GeneralPredicates(pod, nil, nodeInfo) <ins> ncFit, ncReasons := NodeConditionPredicates(nodeInfo) fit = ncFit && fit reasons = append(reasons, ncReasons...) </ins> } if err != nil { return false, predicateFails, err


Question:
getNodeConditionPredicate() in makes our code hard to understand because it hides the node-condition-based filtering in the node lister, which is totally non-obvious. We should get rid of this function and have NodeController add NoSchedule taints for these situations instead. (Alas, we might not actually be able to get rid of getNodeConditionPredicate() completely until we get rid of the Unschedulable field of PodSpec, but at least we can get rid of all the other code in this function.) If there are pods that should still be able to schedule in any of these situations (e.g. DaemonSet pods?) we should add tolerations in admission control for them (e.g. see ). cc/ cc/ Sub-tasks according to the design doc: [x] Add node taints label and feature flag () [x] In the node controller, taint Nodes according to the Node Condition () [x] In DaemonSet, update DaemonSetController to Tolerant new Taints () [x] In admissionController, add MemoryPressure/DiskPressure toleration for no BestEffort pod () [x] In scheduler, disable the current behaviour of filtering out Nodes. Instead, pods will not be scheduled to tainted nodes if a toleration does not exist in its PodSpec () [x] Add e2e test for this feature () [x] Update doc for this feature ()
I think you are referring Unschedulable field of NodeSpec?
Yes sorry, that was a typo.
- please CC me to all PRs
See also
Note that memory and disk pressure are checked in separate predicates, not in the NodeConditionPredicate. These should be represented as NoSchedule taints too.
Setting NoSchedule taints corresponding to the node conditions that today cause pods not to schedule onto a node, should also allow us to use tolerations rather than special-case code in the DaemonSet controller (and thereby make it easier to have DaemonSet pods be scheduled by the default scheduler.)
ref/
The task list was moved to the issue description .
/cc
/reopen
you can't re-open an issue/PR unless you authored it or you are assigned to it. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/reopen
/reopen
Currently we already have toleration on dsc for taint related to node problems (e.g NodeNotReady and NodeUnreachable). But to me ds should respect the new taints (outOfDisk, NodeMemoryPressure etc..), any insights ?
/reopen
Yes, it's one of sub-task of this project. This issue is closed by 'partially fixes' :(, I had updated PRs to 'part of #'.
- I can handle the dsc tolerations if you want. cc to keep you on the loop.
, sure, thanks very much :).
- just saw that you moved node conditions into predicate func, once adding node taiting and the according tolerations to dsc, do we'll still need to have node conditions checking ? (since the we're tainting by node condition). Also, correct me if I'm wrong, but from what I understood: dsc tolerates new taints with effect noExec since we don't wanna evict any already-running pods ?
We won't, but it'll take time for us to get there. We'll roll out Tainting as alpha (obviously), later move it to beta and GA in 1.10 at the earliest. We can't remove Condition checking logic before that. dsc is a daemon set right? New taints are NoSchedule only, so running Pods won't be affected (at least on the control plane level). Kubelet can still decide to evict them based on problem it observes.
- seems logic for the timeline. From sub-tasks Since new Taints are NoSchedule Only, Daemon set shouldn't have toleration for these ? It should instead respect the new taints when applied and not schedule on these nodes, or I'm I missing something ?
The task for DS in my mind: For all DS pods, it should tolerant MemoryPressure & DiskPressure For critical DS pods, it should tolerant OutOfDisk And if the taint by condition feature is enabled, we should check OutOfDisk by taints instead of predicates; the refactor is trying to make this simpler :).
- You're right, I'll send a PR after yours get merged :)
Do we have a to do list so folks can help out with I'd like to help when I get some time. I have a question about timeline. If alpha makes it into v1.8, will I be able to schedule to NotReady nodes (no CNI) if I reference the alpha tolerations? Or do you mean it won't be functional at all until v1.10?
It means you'll need to flip feature gate flag for it to work.
, here is the task list: I'm working on related PRs for them; please ping me when you got time :).
- i anticipate we will also add a CPUPressure condition which will also mean that no BestEffort pod can be scheduled to that node if the kubelet is running with a static cpu assignment policy. /cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
I think it's done (as alpha). - do we want to keep this issue open?
We can close this.

return len(predicateFails) == 0, predicateFails, nil } <ins> func NodeConditionPredicates(nodeInfo *schedulercache.NodeInfo) (bool, []algorithm.PredicateFailureReason) { reasons := []algorithm.PredicateFailureReason{} for _, c := range nodeInfo.Node().Status.Conditions { // TODO: There are other node status that the DaemonSet should ideally respect too, //       e.g. MemoryPressure, and DiskPressure if c.Type == v1.NodeOutOfDisk && c.Status == v1.ConditionTrue { reasons = append(reasons, predicates.ErrNodeSelectorNotMatch) break } } return len(reasons) == 0, reasons } </ins> // newControllerRef creates a ControllerRef pointing to the given DaemonSet. func newControllerRef(ds *extensions.DaemonSet) *metav1.OwnerReference { blockOwnerDeletion := true


Question:
getNodeConditionPredicate() in makes our code hard to understand because it hides the node-condition-based filtering in the node lister, which is totally non-obvious. We should get rid of this function and have NodeController add NoSchedule taints for these situations instead. (Alas, we might not actually be able to get rid of getNodeConditionPredicate() completely until we get rid of the Unschedulable field of PodSpec, but at least we can get rid of all the other code in this function.) If there are pods that should still be able to schedule in any of these situations (e.g. DaemonSet pods?) we should add tolerations in admission control for them (e.g. see ). cc/ cc/ Sub-tasks according to the design doc: [x] Add node taints label and feature flag () [x] In the node controller, taint Nodes according to the Node Condition () [x] In DaemonSet, update DaemonSetController to Tolerant new Taints () [x] In admissionController, add MemoryPressure/DiskPressure toleration for no BestEffort pod () [x] In scheduler, disable the current behaviour of filtering out Nodes. Instead, pods will not be scheduled to tainted nodes if a toleration does not exist in its PodSpec () [x] Add e2e test for this feature () [x] Update doc for this feature ()
I think you are referring Unschedulable field of NodeSpec?
Yes sorry, that was a typo.
- please CC me to all PRs
See also
Note that memory and disk pressure are checked in separate predicates, not in the NodeConditionPredicate. These should be represented as NoSchedule taints too.
Setting NoSchedule taints corresponding to the node conditions that today cause pods not to schedule onto a node, should also allow us to use tolerations rather than special-case code in the DaemonSet controller (and thereby make it easier to have DaemonSet pods be scheduled by the default scheduler.)
ref/
The task list was moved to the issue description .
/cc
/reopen
you can't re-open an issue/PR unless you authored it or you are assigned to it. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/reopen
/reopen
Currently we already have toleration on dsc for taint related to node problems (e.g NodeNotReady and NodeUnreachable). But to me ds should respect the new taints (outOfDisk, NodeMemoryPressure etc..), any insights ?
/reopen
Yes, it's one of sub-task of this project. This issue is closed by 'partially fixes' :(, I had updated PRs to 'part of #'.
- I can handle the dsc tolerations if you want. cc to keep you on the loop.
, sure, thanks very much :).
- just saw that you moved node conditions into predicate func, once adding node taiting and the according tolerations to dsc, do we'll still need to have node conditions checking ? (since the we're tainting by node condition). Also, correct me if I'm wrong, but from what I understood: dsc tolerates new taints with effect noExec since we don't wanna evict any already-running pods ?
We won't, but it'll take time for us to get there. We'll roll out Tainting as alpha (obviously), later move it to beta and GA in 1.10 at the earliest. We can't remove Condition checking logic before that. dsc is a daemon set right? New taints are NoSchedule only, so running Pods won't be affected (at least on the control plane level). Kubelet can still decide to evict them based on problem it observes.
- seems logic for the timeline. From sub-tasks Since new Taints are NoSchedule Only, Daemon set shouldn't have toleration for these ? It should instead respect the new taints when applied and not schedule on these nodes, or I'm I missing something ?
The task for DS in my mind: For all DS pods, it should tolerant MemoryPressure & DiskPressure For critical DS pods, it should tolerant OutOfDisk And if the taint by condition feature is enabled, we should check OutOfDisk by taints instead of predicates; the refactor is trying to make this simpler :).
- You're right, I'll send a PR after yours get merged :)
Do we have a to do list so folks can help out with I'd like to help when I get some time. I have a question about timeline. If alpha makes it into v1.8, will I be able to schedule to NotReady nodes (no CNI) if I reference the alpha tolerations? Or do you mean it won't be functional at all until v1.10?
It means you'll need to flip feature gate flag for it to work.
, here is the task list: I'm working on related PRs for them; please ping me when you got time :).
- i anticipate we will also add a CPUPressure condition which will also mean that no BestEffort pod can be scheduled to that node if the kubelet is running with a static cpu assignment policy. /cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
I think it's done (as alpha). - do we want to keep this issue open?
We can close this.

ErrMaxVolumeCountExceeded    = newPredicateFailureError("MaxVolumeCount") ErrNodeUnderMemoryPressure   = newPredicateFailureError("NodeUnderMemoryPressure") ErrNodeUnderDiskPressure     = newPredicateFailureError("NodeUnderDiskPressure") <ins> ErrNodeOutOfDisk             = newPredicateFailureError("NodeOutOfDisk") </ins> ErrVolumeNodeConflict        = newPredicateFailureError("NoVolumeNodeConflict") // ErrFakePredicate is used for test only. The fake predicates returning false also returns error // as ErrFakePredicate.


Question:
https://k8s- Pod has a restart count 1 FYI, all relevant messages about this pod in the From the sync loop timeline, some containers died. Check the log again: The sandbox was GC'd right after it was created. This led to kubelet killing all the containers in the pod since the sandbox needed to be recreated. Need to check whether this was caused by the container state emulated by fakedockerclient, or by a real bug. /cc
/cc FYI.
kuberuntime GC routine has a minAge protection such that it can't GC any pod sandbox that was created no more than 30 seconds ago). This doesn't seem to have be respected from the log. I suspect that the fakedockerclient was not returning a proper timestamp.
GC logic might be faulty. I see a similar failure in https://k8s- EDIT: The node name is The sandbox was created and removed within 2 seconds
- by looking at the logs, can you tell if there's an easy repro?
So given that this happens also in non-kubemark suites, it seems like a release blocker to me. Anyway - - do you think it's releated to number of pod on a node, or it's just that it happens from time to time, and in scalability suites (no matter if it's kubemark or not) we have more of those tries (due to scale)?
I think the timestamp comparison logic might be wrong in the sandbox garbage collector. This would only happen (in my theory) when the the snapshot of all containers/sandboxes on the node are taken when: sandbox is created but not running (started) yet. other containers in the pod haven't not been created. In this case, the fail-safe (minAge) kicks in, and apparently failed in the tests discussed above. This would more likely to happen when during batch creation of more pods. kubelet would usually heal itself by restarting all container in the pod (which has the undesirable side-effect of killing the user containers of course). I think other tests might not catch this because 1) they don't perform large batch creation, and 2) they don't check the restart count.
Oh the above comment was completely based on my theory. Reproducing this deterministically on a working node may need some hacking in the kubelet code (e.g., inject a long enough sleep after sandbox creation). I will try writing a unit test first to verify the behavior.
Can't reproduce this in my cluster with artificially injected latency... Adding more debug information in For people who are interested in checking the logs and identify the victim pod, here are the instructions: Download the artifacts: E.g., Find the (density) pod with restart count 0:
Didn't reproduce the problem either. I think so. GC can't distinguish just-created sandboxes from failed sandboxes. Especially when the time from sandbox creation to ready is longer than GC's minAge. I'm wondering should runtime (e.g. dockershim) filter out those created-but-not-ready sandboxes from sandbox list? or introduce another sandbox state in CRI?
has been merged 11 hours ago and scalability suite hasn't flaked since. Will monitor and see if there is any need to reopen.
OK. But because , there are still potential problems under big pressure.
First, this issue is no longer a release blocker after the fix. Let's open a separate issue to discuss potential improvement, instead of commenting on a closed issue :) Also, FWIW, we used the same logic in the old docker manager, so this is also not a regression.
Agreed. Opened for this.

// TestListContainers creates several containers and then list them to check // whether the correct metadatas, states, and labels are returned. func TestListContainers(t *testing.T) { <del> ds, _, _ := newTestDockerService() </del> <ins> ds, _, fakeClock := newTestDockerService() </ins> podName, namespace := "foo", "bar" containerName, image := "sidecar", "logger"


Question:
https://k8s- Pod has a restart count 1 FYI, all relevant messages about this pod in the From the sync loop timeline, some containers died. Check the log again: The sandbox was GC'd right after it was created. This led to kubelet killing all the containers in the pod since the sandbox needed to be recreated. Need to check whether this was caused by the container state emulated by fakedockerclient, or by a real bug. /cc
/cc FYI.
kuberuntime GC routine has a minAge protection such that it can't GC any pod sandbox that was created no more than 30 seconds ago). This doesn't seem to have be respected from the log. I suspect that the fakedockerclient was not returning a proper timestamp.
GC logic might be faulty. I see a similar failure in https://k8s- EDIT: The node name is The sandbox was created and removed within 2 seconds
- by looking at the logs, can you tell if there's an easy repro?
So given that this happens also in non-kubemark suites, it seems like a release blocker to me. Anyway - - do you think it's releated to number of pod on a node, or it's just that it happens from time to time, and in scalability suites (no matter if it's kubemark or not) we have more of those tries (due to scale)?
I think the timestamp comparison logic might be wrong in the sandbox garbage collector. This would only happen (in my theory) when the the snapshot of all containers/sandboxes on the node are taken when: sandbox is created but not running (started) yet. other containers in the pod haven't not been created. In this case, the fail-safe (minAge) kicks in, and apparently failed in the tests discussed above. This would more likely to happen when during batch creation of more pods. kubelet would usually heal itself by restarting all container in the pod (which has the undesirable side-effect of killing the user containers of course). I think other tests might not catch this because 1) they don't perform large batch creation, and 2) they don't check the restart count.
Oh the above comment was completely based on my theory. Reproducing this deterministically on a working node may need some hacking in the kubelet code (e.g., inject a long enough sleep after sandbox creation). I will try writing a unit test first to verify the behavior.
Can't reproduce this in my cluster with artificially injected latency... Adding more debug information in For people who are interested in checking the logs and identify the victim pod, here are the instructions: Download the artifacts: E.g., Find the (density) pod with restart count 0:
Didn't reproduce the problem either. I think so. GC can't distinguish just-created sandboxes from failed sandboxes. Especially when the time from sandbox creation to ready is longer than GC's minAge. I'm wondering should runtime (e.g. dockershim) filter out those created-but-not-ready sandboxes from sandbox list? or introduce another sandbox state in CRI?
has been merged 11 hours ago and scalability suite hasn't flaked since. Will monitor and see if there is any need to reopen.
OK. But because , there are still potential problems under big pressure.
First, this issue is no longer a release blocker after the fix. Let's open a separate issue to discuss potential improvement, instead of commenting on a closed issue :) Also, FWIW, we used the same logic in the old docker manager, so this is also not a regression.
Agreed. Opened for this.

expected := []*runtimeapi.Container{} state := runtimeapi.ContainerState_CONTAINER_RUNNING <del> var createdAt int64 = 0 </del> <ins> var createdAt int64 = fakeClock.Now().UnixNano() </ins> for i := range configs { // We don't care about the sandbox id; pass a bogus one. sandboxID := fmt.Sprintf("sandboxid%d", i)


Question:
https://k8s- Pod has a restart count 1 FYI, all relevant messages about this pod in the From the sync loop timeline, some containers died. Check the log again: The sandbox was GC'd right after it was created. This led to kubelet killing all the containers in the pod since the sandbox needed to be recreated. Need to check whether this was caused by the container state emulated by fakedockerclient, or by a real bug. /cc
/cc FYI.
kuberuntime GC routine has a minAge protection such that it can't GC any pod sandbox that was created no more than 30 seconds ago). This doesn't seem to have be respected from the log. I suspect that the fakedockerclient was not returning a proper timestamp.
GC logic might be faulty. I see a similar failure in https://k8s- EDIT: The node name is The sandbox was created and removed within 2 seconds
- by looking at the logs, can you tell if there's an easy repro?
So given that this happens also in non-kubemark suites, it seems like a release blocker to me. Anyway - - do you think it's releated to number of pod on a node, or it's just that it happens from time to time, and in scalability suites (no matter if it's kubemark or not) we have more of those tries (due to scale)?
I think the timestamp comparison logic might be wrong in the sandbox garbage collector. This would only happen (in my theory) when the the snapshot of all containers/sandboxes on the node are taken when: sandbox is created but not running (started) yet. other containers in the pod haven't not been created. In this case, the fail-safe (minAge) kicks in, and apparently failed in the tests discussed above. This would more likely to happen when during batch creation of more pods. kubelet would usually heal itself by restarting all container in the pod (which has the undesirable side-effect of killing the user containers of course). I think other tests might not catch this because 1) they don't perform large batch creation, and 2) they don't check the restart count.
Oh the above comment was completely based on my theory. Reproducing this deterministically on a working node may need some hacking in the kubelet code (e.g., inject a long enough sleep after sandbox creation). I will try writing a unit test first to verify the behavior.
Can't reproduce this in my cluster with artificially injected latency... Adding more debug information in For people who are interested in checking the logs and identify the victim pod, here are the instructions: Download the artifacts: E.g., Find the (density) pod with restart count 0:
Didn't reproduce the problem either. I think so. GC can't distinguish just-created sandboxes from failed sandboxes. Especially when the time from sandbox creation to ready is longer than GC's minAge. I'm wondering should runtime (e.g. dockershim) filter out those created-but-not-ready sandboxes from sandbox list? or introduce another sandbox state in CRI?
has been merged 11 hours ago and scalability suite hasn't flaked since. Will monitor and see if there is any need to reopen.
OK. But because , there are still potential problems under big pressure.
First, this issue is no longer a release blocker after the fix. Let's open a separate issue to discuss potential improvement, instead of commenting on a closed issue :) Also, FWIW, we used the same logic in the old docker manager, so this is also not a regression.
Agreed. Opened for this.

// TestListSandboxes creates several sandboxes and then list them to check // whether the correct metadatas, states, and labels are returned. func TestListSandboxes(t *testing.T) { <del> ds, _, _ := newTestDockerService() </del> <ins> ds, _, fakeClock := newTestDockerService() </ins> name, namespace := "foo", "bar" configs := []*runtimeapi.PodSandboxConfig{} for i := 0; i < 3; i++ {


Question:
https://k8s- Pod has a restart count 1 FYI, all relevant messages about this pod in the From the sync loop timeline, some containers died. Check the log again: The sandbox was GC'd right after it was created. This led to kubelet killing all the containers in the pod since the sandbox needed to be recreated. Need to check whether this was caused by the container state emulated by fakedockerclient, or by a real bug. /cc
/cc FYI.
kuberuntime GC routine has a minAge protection such that it can't GC any pod sandbox that was created no more than 30 seconds ago). This doesn't seem to have be respected from the log. I suspect that the fakedockerclient was not returning a proper timestamp.
GC logic might be faulty. I see a similar failure in https://k8s- EDIT: The node name is The sandbox was created and removed within 2 seconds
- by looking at the logs, can you tell if there's an easy repro?
So given that this happens also in non-kubemark suites, it seems like a release blocker to me. Anyway - - do you think it's releated to number of pod on a node, or it's just that it happens from time to time, and in scalability suites (no matter if it's kubemark or not) we have more of those tries (due to scale)?
I think the timestamp comparison logic might be wrong in the sandbox garbage collector. This would only happen (in my theory) when the the snapshot of all containers/sandboxes on the node are taken when: sandbox is created but not running (started) yet. other containers in the pod haven't not been created. In this case, the fail-safe (minAge) kicks in, and apparently failed in the tests discussed above. This would more likely to happen when during batch creation of more pods. kubelet would usually heal itself by restarting all container in the pod (which has the undesirable side-effect of killing the user containers of course). I think other tests might not catch this because 1) they don't perform large batch creation, and 2) they don't check the restart count.
Oh the above comment was completely based on my theory. Reproducing this deterministically on a working node may need some hacking in the kubelet code (e.g., inject a long enough sleep after sandbox creation). I will try writing a unit test first to verify the behavior.
Can't reproduce this in my cluster with artificially injected latency... Adding more debug information in For people who are interested in checking the logs and identify the victim pod, here are the instructions: Download the artifacts: E.g., Find the (density) pod with restart count 0:
Didn't reproduce the problem either. I think so. GC can't distinguish just-created sandboxes from failed sandboxes. Especially when the time from sandbox creation to ready is longer than GC's minAge. I'm wondering should runtime (e.g. dockershim) filter out those created-but-not-ready sandboxes from sandbox list? or introduce another sandbox state in CRI?
has been merged 11 hours ago and scalability suite hasn't flaked since. Will monitor and see if there is any need to reopen.
OK. But because , there are still potential problems under big pressure.
First, this issue is no longer a release blocker after the fix. Let's open a separate issue to discuss potential improvement, instead of commenting on a closed issue :) Also, FWIW, we used the same logic in the old docker manager, so this is also not a regression.
Agreed. Opened for this.

expected := []*runtimeapi.PodSandbox{} state := runtimeapi.PodSandboxState_SANDBOX_READY <del> var createdAt int64 = 0 </del> <ins> var createdAt int64 = fakeClock.Now().UnixNano() </ins> for i := range configs { id, err := ds.RunPodSandbox(configs[i]) assert.NoError(t, err)


Question:
https://k8s- Pod has a restart count 1 FYI, all relevant messages about this pod in the From the sync loop timeline, some containers died. Check the log again: The sandbox was GC'd right after it was created. This led to kubelet killing all the containers in the pod since the sandbox needed to be recreated. Need to check whether this was caused by the container state emulated by fakedockerclient, or by a real bug. /cc
/cc FYI.
kuberuntime GC routine has a minAge protection such that it can't GC any pod sandbox that was created no more than 30 seconds ago). This doesn't seem to have be respected from the log. I suspect that the fakedockerclient was not returning a proper timestamp.
GC logic might be faulty. I see a similar failure in https://k8s- EDIT: The node name is The sandbox was created and removed within 2 seconds
- by looking at the logs, can you tell if there's an easy repro?
So given that this happens also in non-kubemark suites, it seems like a release blocker to me. Anyway - - do you think it's releated to number of pod on a node, or it's just that it happens from time to time, and in scalability suites (no matter if it's kubemark or not) we have more of those tries (due to scale)?
I think the timestamp comparison logic might be wrong in the sandbox garbage collector. This would only happen (in my theory) when the the snapshot of all containers/sandboxes on the node are taken when: sandbox is created but not running (started) yet. other containers in the pod haven't not been created. In this case, the fail-safe (minAge) kicks in, and apparently failed in the tests discussed above. This would more likely to happen when during batch creation of more pods. kubelet would usually heal itself by restarting all container in the pod (which has the undesirable side-effect of killing the user containers of course). I think other tests might not catch this because 1) they don't perform large batch creation, and 2) they don't check the restart count.
Oh the above comment was completely based on my theory. Reproducing this deterministically on a working node may need some hacking in the kubelet code (e.g., inject a long enough sleep after sandbox creation). I will try writing a unit test first to verify the behavior.
Can't reproduce this in my cluster with artificially injected latency... Adding more debug information in For people who are interested in checking the logs and identify the victim pod, here are the instructions: Download the artifacts: E.g., Find the (density) pod with restart count 0:
Didn't reproduce the problem either. I think so. GC can't distinguish just-created sandboxes from failed sandboxes. Especially when the time from sandbox creation to ready is longer than GC's minAge. I'm wondering should runtime (e.g. dockershim) filter out those created-but-not-ready sandboxes from sandbox list? or introduce another sandbox state in CRI?
has been merged 11 hours ago and scalability suite hasn't flaked since. Will monitor and see if there is any need to reopen.
OK. But because , there are still potential problems under big pressure.
First, this issue is no longer a release blocker after the fix. Let's open a separate issue to discuss potential improvement, instead of commenting on a closed issue :) Also, FWIW, we used the same logic in the old docker manager, so this is also not a regression.
Agreed. Opened for this.

// (kubecontainer) types. const ( statusRunningPrefix = "Up" <ins> statusCreatedPrefix = "Created" </ins> statusExitedPrefix  = "Exited" )


Question:
https://k8s- Pod has a restart count 1 FYI, all relevant messages about this pod in the From the sync loop timeline, some containers died. Check the log again: The sandbox was GC'd right after it was created. This led to kubelet killing all the containers in the pod since the sandbox needed to be recreated. Need to check whether this was caused by the container state emulated by fakedockerclient, or by a real bug. /cc
/cc FYI.
kuberuntime GC routine has a minAge protection such that it can't GC any pod sandbox that was created no more than 30 seconds ago). This doesn't seem to have be respected from the log. I suspect that the fakedockerclient was not returning a proper timestamp.
GC logic might be faulty. I see a similar failure in https://k8s- EDIT: The node name is The sandbox was created and removed within 2 seconds
- by looking at the logs, can you tell if there's an easy repro?
So given that this happens also in non-kubemark suites, it seems like a release blocker to me. Anyway - - do you think it's releated to number of pod on a node, or it's just that it happens from time to time, and in scalability suites (no matter if it's kubemark or not) we have more of those tries (due to scale)?
I think the timestamp comparison logic might be wrong in the sandbox garbage collector. This would only happen (in my theory) when the the snapshot of all containers/sandboxes on the node are taken when: sandbox is created but not running (started) yet. other containers in the pod haven't not been created. In this case, the fail-safe (minAge) kicks in, and apparently failed in the tests discussed above. This would more likely to happen when during batch creation of more pods. kubelet would usually heal itself by restarting all container in the pod (which has the undesirable side-effect of killing the user containers of course). I think other tests might not catch this because 1) they don't perform large batch creation, and 2) they don't check the restart count.
Oh the above comment was completely based on my theory. Reproducing this deterministically on a working node may need some hacking in the kubelet code (e.g., inject a long enough sleep after sandbox creation). I will try writing a unit test first to verify the behavior.
Can't reproduce this in my cluster with artificially injected latency... Adding more debug information in For people who are interested in checking the logs and identify the victim pod, here are the instructions: Download the artifacts: E.g., Find the (density) pod with restart count 0:
Didn't reproduce the problem either. I think so. GC can't distinguish just-created sandboxes from failed sandboxes. Especially when the time from sandbox creation to ready is longer than GC's minAge. I'm wondering should runtime (e.g. dockershim) filter out those created-but-not-ready sandboxes from sandbox list? or introduce another sandbox state in CRI?
has been merged 11 hours ago and scalability suite hasn't flaked since. Will monitor and see if there is any need to reopen.
OK. But because , there are still potential problems under big pressure.
First, this issue is no longer a release blocker after the fix. Let's open a separate issue to discuss potential improvement, instead of commenting on a closed issue :) Also, FWIW, we used the same logic in the old docker manager, so this is also not a regression.
Agreed. Opened for this.

name := "/" + c.Name id := GetFakeContainerID(name) f.appendContainerTrace("Created", id) <ins> timestamp := f.Clock.Now() </ins> // The newest container should be in front, because we assume so in GetPodStatus() f.RunningContainerList = append([]dockertypes.Container{ <del> {ID: id, Names: []string{name}, Image: c.Config.Image, Labels: c.Config.Labels}, </del> <ins> {ID: id, Names: []string{name}, Image: c.Config.Image, Created: timestamp.Unix(), State: statusCreatedPrefix, Labels: c.Config.Labels}, </ins> }, f.RunningContainerList...) f.ContainerMap[id] = convertFakeContainer(&FakeContainer{ <del> ID: id, Name: name, Config: c.Config, HostConfig: c.HostConfig, CreatedAt: f.Clock.Now()}) </del> <ins> ID: id, Name: name, Config: c.Config, HostConfig: c.HostConfig, CreatedAt: timestamp}) </ins> f.normalSleep(100, 25, 25)


Question:
https://k8s- Pod has a restart count 1 FYI, all relevant messages about this pod in the From the sync loop timeline, some containers died. Check the log again: The sandbox was GC'd right after it was created. This led to kubelet killing all the containers in the pod since the sandbox needed to be recreated. Need to check whether this was caused by the container state emulated by fakedockerclient, or by a real bug. /cc
/cc FYI.
kuberuntime GC routine has a minAge protection such that it can't GC any pod sandbox that was created no more than 30 seconds ago). This doesn't seem to have be respected from the log. I suspect that the fakedockerclient was not returning a proper timestamp.
GC logic might be faulty. I see a similar failure in https://k8s- EDIT: The node name is The sandbox was created and removed within 2 seconds
- by looking at the logs, can you tell if there's an easy repro?
So given that this happens also in non-kubemark suites, it seems like a release blocker to me. Anyway - - do you think it's releated to number of pod on a node, or it's just that it happens from time to time, and in scalability suites (no matter if it's kubemark or not) we have more of those tries (due to scale)?
I think the timestamp comparison logic might be wrong in the sandbox garbage collector. This would only happen (in my theory) when the the snapshot of all containers/sandboxes on the node are taken when: sandbox is created but not running (started) yet. other containers in the pod haven't not been created. In this case, the fail-safe (minAge) kicks in, and apparently failed in the tests discussed above. This would more likely to happen when during batch creation of more pods. kubelet would usually heal itself by restarting all container in the pod (which has the undesirable side-effect of killing the user containers of course). I think other tests might not catch this because 1) they don't perform large batch creation, and 2) they don't check the restart count.
Oh the above comment was completely based on my theory. Reproducing this deterministically on a working node may need some hacking in the kubelet code (e.g., inject a long enough sleep after sandbox creation). I will try writing a unit test first to verify the behavior.
Can't reproduce this in my cluster with artificially injected latency... Adding more debug information in For people who are interested in checking the logs and identify the victim pod, here are the instructions: Download the artifacts: E.g., Find the (density) pod with restart count 0:
Didn't reproduce the problem either. I think so. GC can't distinguish just-created sandboxes from failed sandboxes. Especially when the time from sandbox creation to ready is longer than GC's minAge. I'm wondering should runtime (e.g. dockershim) filter out those created-but-not-ready sandboxes from sandbox list? or introduce another sandbox state in CRI?
has been merged 11 hours ago and scalability suite hasn't flaked since. Will monitor and see if there is any need to reopen.
OK. But because , there are still potential problems under big pressure.
First, this issue is no longer a release blocker after the fix. Let's open a separate issue to discuss potential improvement, instead of commenting on a closed issue :) Also, FWIW, we used the same logic in the old docker manager, so this is also not a regression.
Agreed. Opened for this.

} f.appendContainerTrace("Started", id) container, ok := f.ContainerMap[id] <ins> timestamp := f.Clock.Now() </ins> if !ok { <del> container = convertFakeContainer(&FakeContainer{ID: id, Name: id}) </del> <ins> container = convertFakeContainer(&FakeContainer{ID: id, Name: id, CreatedAt: timestamp}) </ins> } container.State.Running = true container.State.Pid = os.Getpid() <del> container.State.StartedAt = dockerTimestampToString(f.Clock.Now()) </del> <ins> container.State.StartedAt = dockerTimestampToString(timestamp) </ins> container.NetworkSettings.IPAddress = "2.3.4.5" f.ContainerMap[id] = container f.updateContainerStatus(id, statusRunningPrefix)


Question:
<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): Is this a BUG REPORT or FEATURE REQUEST? (choose one): <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --BUG REPORT Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"6+", GitVersion:"v1.6.0-beta.3", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-03-11T04:20:22Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.4", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-03-07T23:34:32Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"} Environment: Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: What happened: With kubectl version 1.6.0-beta.3, the output of is not aligned: What you expected to happen: With kubectl version 1.5.4, the output is as follows: How to reproduce it (as minimally and precisely as possible): Run using kubectl version 1.6.0-beta.3. Anything else we need to know:
will take a look
any progress on this?
hi, tony, sorry, I haven't look at this problem yet, maybe tomorrow, just really busy recent day, you can take it if you already find the problem, thanks
Happened to see the same issue on my laptop, I can help fix this issue if you do not have time, thx
thanks, I've been really busy this week, go ahead
for this ticket, want to discuss one more thing with you: at present, for each of the update event, command with option will print out a line showing the status change of this event. But this behavior is different with what we saw from the output of linux's command, which I think is not a very good user experience. For this kind of command, we shall reset all the old output and re-print all the pod's information in the standard output, this will make this command more like a real command. do we have any plans on this right now? welcome your ideas on this.
I think both output format has it's advantage, for current output format, user can copy paste the whole information like checking logs, for linux's command style output which user can stare on only one screen(if it's not too many pods or sth else).
yep, we do have plan to do this, and I really like a more linux style command, you can refer for more detail. Anyway let's just get this fixed at first.

// print the current object if !isWatchOnly { var objsToPrint []runtime.Object <ins> writer := printers.GetNewTabWriter(out) </ins> if isList { objsToPrint, _ = meta.ExtractList(obj) } else { objsToPrint = append(objsToPrint, obj) } for _, objToPrint := range objsToPrint { <del> if err := printer.PrintObj(objToPrint, out); err != nil { </del> <ins> if err := printer.PrintObj(objToPrint, writer); err != nil { </ins> return fmt.Errorf("unable to output the provided object: %v", err) } } <ins> writer.Flush() </ins> } // print watched changes


Question:
This is in 1.6.0-beta.4: Some context: The line number suggests it is here: I think this is because dockershim now uses a protobuf object and Protocol is now an int. And is allowed in go (apparently) - is "A". And the comedic error is because Protocol implements If this is the error, there's something fishy going on because all our UDP tests should fail?
Aha ... I think it is only UDP hostports that would be "broken" by the issue. I wouldn't be surprised if we don't rely on any, and that we might not have any test coverage. I don't think this is a regression - I think this code has been there since the start of dockershim, but maybe CRI is now the default? I don't know how widely UDP hostports are used. Given how close we are to the planned release date, cc and
There is something wrong in dockershim's protocol parsing. Will file an PR to fix this. /assgin
/assign
Opened to fix this.
thanks for filing the issue. Thanks for quickly addressing the bug. We need node e2e tests to capture the kind of regression at the first place.

"//vendor:github.com/blang/semver", "//vendor:github.com/docker/engine-api/types", "//vendor:github.com/docker/engine-api/types/container", <ins> "//vendor:github.com/docker/go-connections/nat", </ins> "//vendor:github.com/golang/mock/gomock", "//vendor:github.com/stretchr/testify/assert", "//vendor:github.com/stretchr/testify/require",


Question:
This is in 1.6.0-beta.4: Some context: The line number suggests it is here: I think this is because dockershim now uses a protobuf object and Protocol is now an int. And is allowed in go (apparently) - is "A". And the comedic error is because Protocol implements If this is the error, there's something fishy going on because all our UDP tests should fail?
Aha ... I think it is only UDP hostports that would be "broken" by the issue. I wouldn't be surprised if we don't rely on any, and that we might not have any test coverage. I don't think this is a regression - I think this code has been there since the start of dockershim, but maybe CRI is now the default? I don't know how widely UDP hostports are used. Given how close we are to the planned release date, cc and
There is something wrong in dockershim's protocol parsing. Will file an PR to fix this. /assgin
/assign
Opened to fix this.
thanks for filing the issue. Thanks for quickly addressing the bug. We need node e2e tests to capture the kind of regression at the first place.

// Some of this port stuff is under-documented voodoo. // See http://stackoverflow.com/questions/20428302/binding-a-port-to-a-host-interface-using-the-rest-api var protocol string <del> switch strings.ToUpper(string(port.Protocol)) { case "UDP": </del> <ins> switch port.Protocol { case runtimeapi.Protocol_UDP: </ins> protocol = "/udp" <del> case "TCP": </del> <ins> case runtimeapi.Protocol_TCP: </ins> protocol = "/tcp" default: glog.Warningf("Unknown protocol %q: defaulting to TCP", port.Protocol)


Question:
This is in 1.6.0-beta.4: Some context: The line number suggests it is here: I think this is because dockershim now uses a protobuf object and Protocol is now an int. And is allowed in go (apparently) - is "A". And the comedic error is because Protocol implements If this is the error, there's something fishy going on because all our UDP tests should fail?
Aha ... I think it is only UDP hostports that would be "broken" by the issue. I wouldn't be surprised if we don't rely on any, and that we might not have any test coverage. I don't think this is a regression - I think this code has been there since the start of dockershim, but maybe CRI is now the default? I don't know how widely UDP hostports are used. Given how close we are to the planned release date, cc and
There is something wrong in dockershim's protocol parsing. Will file an PR to fix this. /assgin
/assign
Opened to fix this.
thanks for filing the issue. Thanks for quickly addressing the bug. We need node e2e tests to capture the kind of regression at the first place.

"github.com/blang/semver" dockertypes "github.com/docker/engine-api/types" <ins> dockernat "github.com/docker/go-connections/nat" </ins> "github.com/stretchr/testify/assert" "github.com/stretchr/testify/require"


Question:
This is in 1.6.0-beta.4: Some context: The line number suggests it is here: I think this is because dockershim now uses a protobuf object and Protocol is now an int. And is allowed in go (apparently) - is "A". And the comedic error is because Protocol implements If this is the error, there's something fishy going on because all our UDP tests should fail?
Aha ... I think it is only UDP hostports that would be "broken" by the issue. I wouldn't be surprised if we don't rely on any, and that we might not have any test coverage. I don't think this is a regression - I think this code has been there since the start of dockershim, but maybe CRI is now the default? I don't know how widely UDP hostports are used. Given how close we are to the planned release date, cc and
There is something wrong in dockershim's protocol parsing. Will file an PR to fix this. /assgin
/assign
Opened to fix this.
thanks for filing the issue. Thanks for quickly addressing the bug. We need node e2e tests to capture the kind of regression at the first place.

assert.Equal(t, test.err, err != nil) } } <ins> func TestMakePortsAndBindings(t *testing.T) { for desc, test := range map[string]struct { pm           []*runtimeapi.PortMapping exposedPorts map[dockernat.Port]struct{} portmappings map[dockernat.Port][]dockernat.PortBinding }{ "no port mapping": { pm:           nil, exposedPorts: map[dockernat.Port]struct{}{}, portmappings: map[dockernat.Port][]dockernat.PortBinding{}, }, "tcp port mapping": { pm: []*runtimeapi.PortMapping{ { Protocol:      runtimeapi.Protocol_TCP, ContainerPort: 80, HostPort:      80, }, }, exposedPorts: map[dockernat.Port]struct{}{ "80/tcp": {}, }, portmappings: map[dockernat.Port][]dockernat.PortBinding{ "80/tcp": { { HostPort: "80", }, }, }, }, "udp port mapping": { pm: []*runtimeapi.PortMapping{ { Protocol:      runtimeapi.Protocol_UDP, ContainerPort: 80, HostPort:      80, }, }, exposedPorts: map[dockernat.Port]struct{}{ "80/udp": {}, }, portmappings: map[dockernat.Port][]dockernat.PortBinding{ "80/udp": { { HostPort: "80", }, }, }, }, "multipe port mappings": { pm: []*runtimeapi.PortMapping{ { Protocol:      runtimeapi.Protocol_TCP, ContainerPort: 80, HostPort:      80, }, { Protocol:      runtimeapi.Protocol_TCP, ContainerPort: 80, HostPort:      81, }, }, exposedPorts: map[dockernat.Port]struct{}{ "80/tcp": {}, }, portmappings: map[dockernat.Port][]dockernat.PortBinding{ "80/tcp": { { HostPort: "80", }, { HostPort: "81", }, }, }, }, } { t.Logf("TestCase: %s", desc) actualExposedPorts, actualPortMappings := makePortsAndBindings(test.pm) assert.Equal(t, test.exposedPorts, actualExposedPorts) assert.Equal(t, test.portmappings, actualPortMappings) } } </ins>


Question:
From https://k8s- I've tried to reproduce locally in an endless loop but no failures so far. cc
Investigating.

return c, informers } <del> func (f *fixture) runExpectError(deploymentName string) { f.run_(deploymentName, true) </del> <ins> func (f *fixture) runExpectError(deploymentName string, startInformers bool) { f.run_(deploymentName, startInformers, true) </ins> } func (f *fixture) run(deploymentName string) { <del> f.run_(deploymentName, false) </del> <ins> f.run_(deploymentName, true, false) </ins> } <del> func (f *fixture) run_(deploymentName string, expectError bool) { </del> <ins> func (f *fixture) run_(deploymentName string, startInformers bool, expectError bool) { </ins> c, informers := f.newController() <del> stopCh := make(chan struct{}) defer close(stopCh) informers.Start(stopCh) </del> <ins> if startInformers { stopCh := make(chan struct{}) defer close(stopCh) informers.Start(stopCh) } </ins> err := c.syncDeployment(deploymentName) if !expectError && err != nil {


Question:
From https://k8s- I've tried to reproduce locally in an endless loop but no failures so far. cc
Investigating.

// Expect to only recheck DeletionTimestamp. f.expectGetDeploymentAction(d) // Sync should fail and requeue to let cache catch up. <del> f.runExpectError(getKey(d, t)) </del> <ins> // Don't start informers, since we don't want cache to catch up for this test. f.runExpectError(getKey(d, t), false) </ins> } // issue: https://github.com/kubernetes/kubernetes/issues/23218


Question:
Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG Kubernetes version (use ): k8s1.3.5. still exist in 1.6 What happened: when exec threadSafeMap.Index func, the returned items contain nil object. That is to say, index is in threadSafeMap but the corresponding has already been deleted. What you expected to happen: when object is deleted from threadSafeMap, all its index must be deleted. How to reproduce it (as minimally and precisely as possible): Suppose we init threadSafeMap with two indexers: indexFunc1 and indexFunc2. Obviously, indexFunc may return error if the object has something wrong or some other reasons. when we add an object, will first add the object to , then compute the index and add the index into threadSafe.indices. when we delete the object, will first delete the index from indices, then delete it from Suppose IndexFunc1(object) return error and indexFunc2(object) works well. I think this circumstance is absolutely possible to happen. Also we know that the order to exec indexFunc is not decided. Feature of iterating map. When we add object, suppose first exec indexFunc2, and then exec indexFunc1. So the index of indexFunc2 is to indices. But the index of indexFunc1 is not , because indexFunc1(obj) return err. Then we delete the object, suppose first exec indexFunc1. It will return error and IndexFunc2 has no chance to run. So the corresponding index indexFunc2(obj) can't be removed. But then the object is removed from items. This problem may result in subtle problems. For example, when we exec threadSafeMap.Index( ) func the returned list may include nil object. Anything else we need to know The reason is that index is still is threadSafeMap but the corresponding object has already been removed. We must clear all corresponding indexs when delete object. And the reason why index is not deleted is that when for-range map, the order is not decided.
Who can help me? Do i miss something important or misunderstand it ?
, can you share a unit test to show it? I'd like to help on it.
Thank you for your help. I think i have decribed the condition resulting in the bug indetail. Be simple, because the order of exec indexFunc is random when update indices, and some function don't care the error returned, the obj is deleted but corresponding indices is still in it. NOTE: the obj is deleted but corresponding indices is still in it So when we run index func, the returned list may contain nil object. Because when get obj from threadsafemap with indice, not care whether the obj exist in this map. Then nil object .
. I am sure this problem exists. And i think we can find a better way to deal with it.
/sig api-machinery threadSafeMap is in client-go
/kind bug
Opened a PR with a fix.
In my understanding, should be deterministic, could you elaborate more how your IndexFunc implemented? If an object can not be called on this , then the question is, how could it exist in at the first beginning?
threadSafeStore should tolerate some bad conditions and deal with it gracefully, instead generating strange and confusing phenomenon. ThreadSafeStore can contain mutiple index funcs. If one of them is not implemented very well, or one of objects we pass to it has something wrong, STRANGE AND CONFUSING phenomenon happens, see what I said above.
That's the point I would like to know more, the call on an object should succeed or fail unconditionally(deterministic).
wechat
I also think index function must be a pure function. See my comment on the pr

// updateIndices modifies the objects location in the managed indexes, if this is an update, you must provide an oldObj // updateIndices must be called from a function that already has a lock on the cache <del> func (c *threadSafeMap) updateIndices(oldObj interface{}, newObj interface{}, key string) error { </del> <ins> func (c *threadSafeMap) updateIndices(oldObj interface{}, newObj interface{}, key string) { </ins> // if we got an old object, we need to remove it before we add it again if oldObj != nil { c.deleteFromIndices(oldObj, key)


Question:
Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG Kubernetes version (use ): k8s1.3.5. still exist in 1.6 What happened: when exec threadSafeMap.Index func, the returned items contain nil object. That is to say, index is in threadSafeMap but the corresponding has already been deleted. What you expected to happen: when object is deleted from threadSafeMap, all its index must be deleted. How to reproduce it (as minimally and precisely as possible): Suppose we init threadSafeMap with two indexers: indexFunc1 and indexFunc2. Obviously, indexFunc may return error if the object has something wrong or some other reasons. when we add an object, will first add the object to , then compute the index and add the index into threadSafe.indices. when we delete the object, will first delete the index from indices, then delete it from Suppose IndexFunc1(object) return error and indexFunc2(object) works well. I think this circumstance is absolutely possible to happen. Also we know that the order to exec indexFunc is not decided. Feature of iterating map. When we add object, suppose first exec indexFunc2, and then exec indexFunc1. So the index of indexFunc2 is to indices. But the index of indexFunc1 is not , because indexFunc1(obj) return err. Then we delete the object, suppose first exec indexFunc1. It will return error and IndexFunc2 has no chance to run. So the corresponding index indexFunc2(obj) can't be removed. But then the object is removed from items. This problem may result in subtle problems. For example, when we exec threadSafeMap.Index( ) func the returned list may include nil object. Anything else we need to know The reason is that index is still is threadSafeMap but the corresponding object has already been removed. We must clear all corresponding indexs when delete object. And the reason why index is not deleted is that when for-range map, the order is not decided.
Who can help me? Do i miss something important or misunderstand it ?
, can you share a unit test to show it? I'd like to help on it.
Thank you for your help. I think i have decribed the condition resulting in the bug indetail. Be simple, because the order of exec indexFunc is random when update indices, and some function don't care the error returned, the obj is deleted but corresponding indices is still in it. NOTE: the obj is deleted but corresponding indices is still in it So when we run index func, the returned list may contain nil object. Because when get obj from threadsafemap with indice, not care whether the obj exist in this map. Then nil object .
. I am sure this problem exists. And i think we can find a better way to deal with it.
/sig api-machinery threadSafeMap is in client-go
/kind bug
Opened a PR with a fix.
In my understanding, should be deterministic, could you elaborate more how your IndexFunc implemented? If an object can not be called on this , then the question is, how could it exist in at the first beginning?
threadSafeStore should tolerate some bad conditions and deal with it gracefully, instead generating strange and confusing phenomenon. ThreadSafeStore can contain mutiple index funcs. If one of them is not implemented very well, or one of objects we pass to it has something wrong, STRANGE AND CONFUSING phenomenon happens, see what I said above.
That's the point I would like to know more, the call on an object should succeed or fail unconditionally(deterministic).
wechat
I also think index function must be a pure function. See my comment on the pr

for name, indexFunc := range c.indexers { indexValues, err := indexFunc(newObj) if err != nil { <del> return err </del> <ins> panic(fmt.Errorf("unable to calculate an index entry for key %q on index %q: %v", key, name, err)) </ins> } index := c.indices[name] if index == nil {


Question:
Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG Kubernetes version (use ): k8s1.3.5. still exist in 1.6 What happened: when exec threadSafeMap.Index func, the returned items contain nil object. That is to say, index is in threadSafeMap but the corresponding has already been deleted. What you expected to happen: when object is deleted from threadSafeMap, all its index must be deleted. How to reproduce it (as minimally and precisely as possible): Suppose we init threadSafeMap with two indexers: indexFunc1 and indexFunc2. Obviously, indexFunc may return error if the object has something wrong or some other reasons. when we add an object, will first add the object to , then compute the index and add the index into threadSafe.indices. when we delete the object, will first delete the index from indices, then delete it from Suppose IndexFunc1(object) return error and indexFunc2(object) works well. I think this circumstance is absolutely possible to happen. Also we know that the order to exec indexFunc is not decided. Feature of iterating map. When we add object, suppose first exec indexFunc2, and then exec indexFunc1. So the index of indexFunc2 is to indices. But the index of indexFunc1 is not , because indexFunc1(obj) return err. Then we delete the object, suppose first exec indexFunc1. It will return error and IndexFunc2 has no chance to run. So the corresponding index indexFunc2(obj) can't be removed. But then the object is removed from items. This problem may result in subtle problems. For example, when we exec threadSafeMap.Index( ) func the returned list may include nil object. Anything else we need to know The reason is that index is still is threadSafeMap but the corresponding object has already been removed. We must clear all corresponding indexs when delete object. And the reason why index is not deleted is that when for-range map, the order is not decided.
Who can help me? Do i miss something important or misunderstand it ?
, can you share a unit test to show it? I'd like to help on it.
Thank you for your help. I think i have decribed the condition resulting in the bug indetail. Be simple, because the order of exec indexFunc is random when update indices, and some function don't care the error returned, the obj is deleted but corresponding indices is still in it. NOTE: the obj is deleted but corresponding indices is still in it So when we run index func, the returned list may contain nil object. Because when get obj from threadsafemap with indice, not care whether the obj exist in this map. Then nil object .
. I am sure this problem exists. And i think we can find a better way to deal with it.
/sig api-machinery threadSafeMap is in client-go
/kind bug
Opened a PR with a fix.
In my understanding, should be deterministic, could you elaborate more how your IndexFunc implemented? If an object can not be called on this , then the question is, how could it exist in at the first beginning?
threadSafeStore should tolerate some bad conditions and deal with it gracefully, instead generating strange and confusing phenomenon. ThreadSafeStore can contain mutiple index funcs. If one of them is not implemented very well, or one of objects we pass to it has something wrong, STRANGE AND CONFUSING phenomenon happens, see what I said above.
That's the point I would like to know more, the call on an object should succeed or fail unconditionally(deterministic).
wechat
I also think index function must be a pure function. See my comment on the pr

set.Insert(key) } } <del> return nil </del> } // deleteFromIndices removes the object from each of the managed indexes // it is intended to be called from a function that already has a lock on the cache <del> func (c *threadSafeMap) deleteFromIndices(obj interface{}, key string) error { </del> <ins> func (c *threadSafeMap) deleteFromIndices(obj interface{}, key string) { </ins> for name, indexFunc := range c.indexers { indexValues, err := indexFunc(obj) if err != nil { <del> return err </del> <ins> panic(fmt.Errorf("unable to calculate an index entry for key %q on index %q: %v", key, name, err)) </ins> } index := c.indices[name]


Question:
Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG Kubernetes version (use ): k8s1.3.5. still exist in 1.6 What happened: when exec threadSafeMap.Index func, the returned items contain nil object. That is to say, index is in threadSafeMap but the corresponding has already been deleted. What you expected to happen: when object is deleted from threadSafeMap, all its index must be deleted. How to reproduce it (as minimally and precisely as possible): Suppose we init threadSafeMap with two indexers: indexFunc1 and indexFunc2. Obviously, indexFunc may return error if the object has something wrong or some other reasons. when we add an object, will first add the object to , then compute the index and add the index into threadSafe.indices. when we delete the object, will first delete the index from indices, then delete it from Suppose IndexFunc1(object) return error and indexFunc2(object) works well. I think this circumstance is absolutely possible to happen. Also we know that the order to exec indexFunc is not decided. Feature of iterating map. When we add object, suppose first exec indexFunc2, and then exec indexFunc1. So the index of indexFunc2 is to indices. But the index of indexFunc1 is not , because indexFunc1(obj) return err. Then we delete the object, suppose first exec indexFunc1. It will return error and IndexFunc2 has no chance to run. So the corresponding index indexFunc2(obj) can't be removed. But then the object is removed from items. This problem may result in subtle problems. For example, when we exec threadSafeMap.Index( ) func the returned list may include nil object. Anything else we need to know The reason is that index is still is threadSafeMap but the corresponding object has already been removed. We must clear all corresponding indexs when delete object. And the reason why index is not deleted is that when for-range map, the order is not decided.
Who can help me? Do i miss something important or misunderstand it ?
, can you share a unit test to show it? I'd like to help on it.
Thank you for your help. I think i have decribed the condition resulting in the bug indetail. Be simple, because the order of exec indexFunc is random when update indices, and some function don't care the error returned, the obj is deleted but corresponding indices is still in it. NOTE: the obj is deleted but corresponding indices is still in it So when we run index func, the returned list may contain nil object. Because when get obj from threadsafemap with indice, not care whether the obj exist in this map. Then nil object .
. I am sure this problem exists. And i think we can find a better way to deal with it.
/sig api-machinery threadSafeMap is in client-go
/kind bug
Opened a PR with a fix.
In my understanding, should be deterministic, could you elaborate more how your IndexFunc implemented? If an object can not be called on this , then the question is, how could it exist in at the first beginning?
threadSafeStore should tolerate some bad conditions and deal with it gracefully, instead generating strange and confusing phenomenon. ThreadSafeStore can contain mutiple index funcs. If one of them is not implemented very well, or one of objects we pass to it has something wrong, STRANGE AND CONFUSING phenomenon happens, see what I said above.
That's the point I would like to know more, the call on an object should succeed or fail unconditionally(deterministic).
wechat
I also think index function must be a pure function. See my comment on the pr

} } } <del> return nil </del> } func (c *threadSafeMap) Resync() error {


Question:
I was confused by the that looks like more as machine readable rather than as human readable. I feel that UX could be better if it will be written in multiple lines and without curly braces/quotes:
Agreed on this one, we should pretty-print this output.
I would like to change that tomorrow if you are not working on it.
I'm not working on it, so feel free to provide a PR. But I wouldn't do anything until we get approval from
cc don't some of the tests rely on this output?
They probably do. There should be a option or something that we can use in test infra.
please have a look at: and related issue: The above is approved and had an lgtm labeled, but removed by the bot because of testing flake.
I think we can close this ?
I don't see that it was fixed. Does the output of was improved?
kubectl now has a option which lets you select between YAML or JSON. However, I believe this is reversed. The more human-friendly output is hidden behind an option and the machine-friendly output is the default. I believe it should be the opposite.

kube::test::version::json_object_to_file() { flags=$1 file=$2 <del> kubectl version $flags --output json | sed -e s/'"'/''/g -e s/'}'/''/g -e s/'{'/''/g -e s/'clientVersion:'/'clientVersion:,'/ -e s/'serverVersion:'/'serverVersion:,'/ | tr , 'n' > "${file}" </del> <ins> kubectl version $flags --output json | sed -e s/' '/''/g -e s/'"'/''/g -e s/'}'/''/g -e s/'{'/''/g -e s/'clientVersion:'/'clientVersion:,'/ -e s/'serverVersion:'/'serverVersion:,'/ | tr , 'n' > "${file}" </ins> } kube::test::version::json_client_server_object_to_file() {


Question:
I was confused by the that looks like more as machine readable rather than as human readable. I feel that UX could be better if it will be written in multiple lines and without curly braces/quotes:
Agreed on this one, we should pretty-print this output.
I would like to change that tomorrow if you are not working on it.
I'm not working on it, so feel free to provide a PR. But I wouldn't do anything until we get approval from
cc don't some of the tests rely on this output?
They probably do. There should be a option or something that we can use in test infra.
please have a look at: and related issue: The above is approved and had an lgtm labeled, but removed by the bot because of testing flake.
I think we can close this ?
I don't see that it was fixed. Does the output of was improved?
kubectl now has a option which lets you select between YAML or JSON. However, I believe this is reversed. The more human-friendly output is hidden behind an option and the machine-friendly output is the default. I believe it should be the opposite.

} cmd.Flags().BoolP("client", "c", false, "Client version only (no server required).") cmd.Flags().BoolP("short", "", false, "Print just the version number.") <del> cmd.Flags().String("output", "", "one of 'yaml' or 'json'") </del> <ins> cmd.Flags().StringP("output", "o", "", "One of 'yaml' or 'json'.") </ins> cmd.Flags().MarkShorthandDeprecated("client", "please use --client instead.") return cmd }


Question:
I was confused by the that looks like more as machine readable rather than as human readable. I feel that UX could be better if it will be written in multiple lines and without curly braces/quotes:
Agreed on this one, we should pretty-print this output.
I would like to change that tomorrow if you are not working on it.
I'm not working on it, so feel free to provide a PR. But I wouldn't do anything until we get approval from
cc don't some of the tests rely on this output?
They probably do. There should be a option or something that we can use in test infra.
please have a look at: and related issue: The above is approved and had an lgtm labeled, but removed by the bot because of testing flake.
I think we can close this ?
I don't see that it was fixed. Does the output of was improved?
kubectl now has a option which lets you select between YAML or JSON. However, I believe this is reversed. The more human-friendly output is hidden behind an option and the machine-friendly output is the default. I believe it should be the opposite.

} fmt.Fprintln(out, string(marshalled)) case "json": <del> marshalled, err := json.Marshal(&versionInfo) </del> <ins> marshalled, err := json.MarshalIndent(&versionInfo, "", "  ") </ins> if err != nil { return err }


Question:
( noticed this) HardPodAffinitySymmetricWeight should be in the , not in KubeSchedulerConfiguration If/when fixing this, we need to be careful to do it in a backward-compatilble way.
Do you mean ? It seems scheduler policy config's args can not passed to FitPredicates; it need to update related interface. For the : mark to be deprecated and set scheduler policy config to HardPodAffinitySymmetricWeight if policy config is empty.
Are you working on this? if no, I can take it.
, nop; please go ahead.
/ / / , When reviewing the PR, there's one question on : the default value of is 1, which means the weight (1) will always add to priorities if term matched (), does it our expectation? And the user has to set it to 0 to disable it. IMO, the default value of should be 0.
The value 1 (instead of 0) was chosen so that if you say A must run with B, there is an automatic preference that B should run with A. This seems like reasonable behavior.
Thanks very much; reasonable case to me. But it seems we can not disable it: if set to 0, the default funcs will set it to default (1); similar result to un-set. Anyway, no user complain this now :).

{"name" : "ServiceSpreadingPriority", "weight" : 1}, {"name" : "EqualPriority", "weight" : 1} ], <del> "extenders":[ </del> <ins> "extenders" : [ </ins> { "urlPrefix": "http://127.0.0.1:12346/scheduler", "apiVersion": "v1beta1",


Question:
( noticed this) HardPodAffinitySymmetricWeight should be in the , not in KubeSchedulerConfiguration If/when fixing this, we need to be careful to do it in a backward-compatilble way.
Do you mean ? It seems scheduler policy config's args can not passed to FitPredicates; it need to update related interface. For the : mark to be deprecated and set scheduler policy config to HardPodAffinitySymmetricWeight if policy config is empty.
Are you working on this? if no, I can take it.
, nop; please go ahead.
/ / / , When reviewing the PR, there's one question on : the default value of is 1, which means the weight (1) will always add to priorities if term matched (), does it our expectation? And the user has to set it to 0 to disable it. IMO, the default value of should be 0.
The value 1 (instead of 0) was chosen so that if you say A must run with B, there is an automatic preference that B should run with A. This seems like reasonable behavior.
Thanks very much; reasonable case to me. But it seems we can not disable it: if set to 0, the default funcs will set it to default (1); similar result to un-set. Anyway, no user complain this now :).

"enableHttps": false, "nodeCacheCapable": false } <del> ] </del> <ins> ], "hardPodAffinitySymmetricWeight" : 10 </ins> }


Question:
( noticed this) HardPodAffinitySymmetricWeight should be in the , not in KubeSchedulerConfiguration If/when fixing this, we need to be careful to do it in a backward-compatilble way.
Do you mean ? It seems scheduler policy config's args can not passed to FitPredicates; it need to update related interface. For the : mark to be deprecated and set scheduler policy config to HardPodAffinitySymmetricWeight if policy config is empty.
Are you working on this? if no, I can take it.
, nop; please go ahead.
/ / / , When reviewing the PR, there's one question on : the default value of is 1, which means the weight (1) will always add to priorities if term matched (), does it our expectation? And the user has to set it to 0 to disable it. IMO, the default value of should be 0.
The value 1 (instead of 0) was chosen so that if you say A must run with B, there is an automatic preference that B should run with A. This seems like reasonable behavior.
Thanks very much; reasonable case to me. But it seems we can not disable it: if set to 0, the default funcs will set it to default (1); similar result to un-set. Anyway, no user complain this now :).

{"name" : "BalancedResourceAllocation", "weight" : 1}, {"name" : "ServiceSpreadingPriority", "weight" : 1}, {"name" : "EqualPriority", "weight" : 1} <del> ] </del> <ins> ], "hardPodAffinitySymmetricWeight" : 10 </ins> }


Question:
( noticed this) HardPodAffinitySymmetricWeight should be in the , not in KubeSchedulerConfiguration If/when fixing this, we need to be careful to do it in a backward-compatilble way.
Do you mean ? It seems scheduler policy config's args can not passed to FitPredicates; it need to update related interface. For the : mark to be deprecated and set scheduler policy config to HardPodAffinitySymmetricWeight if policy config is empty.
Are you working on this? if no, I can take it.
, nop; please go ahead.
/ / / , When reviewing the PR, there's one question on : the default value of is 1, which means the weight (1) will always add to priorities if term matched (), does it our expectation? And the user has to set it to 0 to disable it. IMO, the default value of should be 0.
The value 1 (instead of 0) was chosen so that if you say A must run with B, there is an automatic preference that B should run with A. This seems like reasonable behavior.
Thanks very much; reasonable case to me. But it seems we can not disable it: if set to 0, the default funcs will set it to default (1); similar result to un-set. Anyway, no user complain this now :).

fs.IntVar(&s.HardPodAffinitySymmetricWeight, "hard-pod-affinity-symmetric-weight", api.DefaultHardPodAffinitySymmetricWeight, "RequiredDuringScheduling affinity is not symmetric, but there is an implicit PreferredDuringScheduling affinity rule corresponding "+ "to every RequiredDuringScheduling affinity rule. --hard-pod-affinity-symmetric-weight represents the weight of implicit PreferredDuringScheduling affinity rule.") <ins> fs.MarkDeprecated("hard-pod-affinity-symmetric-weight", "This option was moved to the policy configuration file") </ins> fs.StringVar(&s.FailureDomains, "failure-domains", api.DefaultFailureDomains, "Indicate the "all topologies" set for an empty topologyKey when it's used for PreferredDuringScheduling pod anti-affinity.") fs.MarkDeprecated("failure-domains", "Doesn't have any effect. Will be removed in future version.") leaderelection.BindFlags(&s.LeaderElection, fs)