Question: template: &api.PodTemplateSpec{ Spec: api.PodSpec{ TopologySpreadConstraints: []api.TopologySpreadConstraint{ <del> {TopologyKey: `foo`}, {TopologyKey: `beta.kubernetes.io/arch`}, {TopologyKey: `beta.kubernetes.io/os`}, {TopologyKey: `failure-domain.beta.kubernetes.io/region`}, {TopologyKey: `failure-domain.beta.kubernetes.io/zone`}, {TopologyKey: `beta.kubernetes.io/instance-type`}, </del> <ins> { TopologyKey:   `foo`, LabelSelector: &metav1.LabelSelector{}, }, { TopologyKey:   `beta.kubernetes.io/arch`, LabelSelector: &metav1.LabelSelector{}, }, { TopologyKey:   `beta.kubernetes.io/os`, LabelSelector: &metav1.LabelSelector{}, }, { TopologyKey:   `failure-domain.beta.kubernetes.io/region`, LabelSelector: &metav1.LabelSelector{}, }, { TopologyKey:   `failure-domain.beta.kubernetes.io/zone`, LabelSelector: &metav1.LabelSelector{}, }, { TopologyKey:   `beta.kubernetes.io/instance-type`, LabelSelector: &metav1.LabelSelector{}, }, </ins> }, }, },

Currently, when the Label Selector in the Pod Affinity is empty, no objects will be match. That means if it's in a required PodAffinity, the Pod will never be schedulable. And if it's in a required PodAntiAffinity, that PodAntiAffinity will be no-op. So, I'm thinking PodAffinity with empty Label Selector is very likely mistake. It'd be good if we could notice it for users. To prevent such a misconfiguration. Probably, we cannot filter out such a Pod creation in the validation for a backward compatibility. /assign /sig scheduling
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I'm wondering if it has already discussed somewhere or not. Please just let me know if we have some decisions on it.
I am not sure, where do you propose to put the warning? If the pod is unschedulable, that will show up in the events and pod status.
Do you mean the place to implement this warning? I suspect ↓ is a suitable place:
A warning sounds useful, even if the pod is unschedulable, as it gives you a heads up at the time when you create the Pod. Whereas the Pod scheduling failure might be a bit harder to interpret.
ok, I thought you were referring to something that the scheduler will produce after attempting to schedule the pod. Yeah, if there is already a setup to print warnings on pod creation, it is certainly useful for this case.
Can you please also add it in topology spread?
Sure


Question: import ( "fmt" "reflect" <ins> "strings" </ins> "testing" v1 "k8s.io/api/core/v1"

There is no reason this should be allowed to happen, especially that we can't configure duplicate plugins differently since plugin configs is a separate list . /sig scheduling
/label help-wanted
The label(s) cannot be applied. These labels are supported: <detailsIn response to : /label help-wanted Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/help-wanted
/help /good-first-issue
/assign
/remove-good-first-issue /remove-help


Question: "custom": resource.MustParse("0"), }, Limits: v1.ResourceList{ <del> "memory": resource.MustParse("2G"), </del> <ins> "memory": resource.MustParse("2.5Gi"), </ins> "custom": resource.MustParse("6"), }, }},

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --I found that for with returns a very small value. The value should be almost the same as the one of . My minimal : Result: While the top three lines () are good, the float values in the bottom three lines () might be broken. Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig api-machinery
/assign
/cc /triage accepted


Question: t.Fatalf("unexpected error: %v", err) } if !reflect.DeepEqual(testCase.Limiter, client.rateLimiter) { <del> t.Fatalf("unexpected rate limiter: %#v", client.rateLimiter) </del> <ins> t.Fatalf("unexpected rate limiter: %#v, expected %#v at %s", client.rateLimiter, testCase.Limiter, testCase.Name) </ins> } }) t.Run("Unversioned_"+testCase.Name, func(t *testing.T) {

Reading TestRESTClientLimiter in I noticed a few things. test case struct is defined to have a name field, and its value is used in naming the subtests, but the actual test cases leave the name blank. error statements do not identify which case is failing (do not identify the expected value). Should be clear from the above. Read the code. No response_ <detailsbranch </details<detailsN/A </details<detailsN/A </details<details</details<details</details<details</details
/sig api-machinery
/assign
/triage accepted


Question: inf "gopkg.in/inf.v0" ) <ins> var ( bigMostPositive = big.NewInt(mostPositive) bigMostNegative = big.NewInt(mostNegative) ) </ins> func dec(i int64, exponent int) infDecAmount { // See the below test-- scale is the negative of an exponent. return infDecAmount{inf.NewDec(i, inf.Scale(-exponent))} } <ins> func bigDec(i *big.Int, exponent int) infDecAmount { // See the below test-- scale is the negative of an exponent. return infDecAmount{inf.NewDecBig(i, inf.Scale(-exponent))} } </ins> func decQuantity(i int64, exponent int, format Format) Quantity { return Quantity{d: dec(i, exponent), Format: format} } <ins> func bigDecQuantity(i *big.Int, exponent int, format Format) Quantity { return Quantity{d: bigDec(i, exponent), Format: format} } </ins> func intQuantity(i int64, exponent Scale, format Format) Quantity { return Quantity{i: int64Amount{value: i, scale: exponent}, Format: format} }

I'd like to add a multiplication method for the resource.Quantity. Currently, the resource package provides helper methods to calculate Quantity, such as and : Batch workloads often replicate the same PodTemplate, and controllers calculate the resources needed for all containers. In that case, It is inconvenient to call the function as many times as podTemplate, so I would like to create a helper function for multiplication.
/cc /sig api-machinary /wg batch
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /cc /sig api-machinary /wg batch </details
/sig api-machinery /wg batch
Do you have an example of usage for these utilities? Just curious?
/triage accepted
Sorry for the late response. Taking k-sigs/kueue as an example, I assumed the below: It helps the controllers for batch workloads such as k-sigs/kueue, k-sigs/jobset, kubeflow/mpi-operator, and kubeflow/training-operator. I'm thinking of implementing this function once the code freeze term is finished.
/assign
Hi, may i take this up?
I took this one, and working on this now.


Question: NodeAllocatableChange, NodeConditionChange, NodeLabelChange, <ins> NodeAnnotationChange, </ins> PvcAdd, PvcUpdate, PvAdd,

Due to limited character space, I've stored additional information in annotations on nodes. My custom scheduling plugins read these annotations to schedule Pods. However, when a Pod fails to be scheduled on a node, I want it to be removed from the Unschedulable queue. This can be achieved by modifying the annotations of the node, allowing the Pod to be rescheduled. Unfortunately, the eventHandler of the scheduler only supports five types of node events, and it does not include the nodeAnnotationsChanged event. Consequently, I am currently unable to trigger rescheduling by changing the annotations of nodes. /sig scheduling Unschedulable pods will be rescheduled after annotations of nodes changed. None No response <details</details<details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc
/assign
It's more like a new feature, not bug because it doesn't lead to any bad behavior on the upstream. /remove-kind bug /kind feature
My concern is node annotation is arbitrary, change a node annotation doesn't imply a pod is schedulable. But for other events, like node labels, node taints, they're the constraints we consider in scheduling. They're different, I'm afraid this will lead to invalid scheduling attempts. Who is responsible for change the annotations in you scenario?
This enhancement has no impact on the in-tree plugins / upstream scheduler, right? Only (custom) plugins, that want to reschedule Pods via the node annotation change, will register event to EventsToRegister.
I have a controller capable of modifying node annotations to indicate node's current availability. And my custom plugins need to be aware of these changes.
I guess we can still be impacted when scheduling pod failed because of no fit node rather than failed by the plugin.


Question: NodeAllocatableChange, NodeConditionChange, NodeLabelChange, <ins> NodeAnnotationChange, </ins> PvcAdd, PvcUpdate, PvAdd,

see Can we predict whether the pod is schedulable before running into scheduling cycle via the node annotations, then we can leverage the preEnqueue extension point since you already have a external controller.
Thanks for the advice. But I'm afraid it would not be the best solution for us, because doing that outside the scheduling cycle would cause the custom plugins to malfunction in my scenario.
The concerns raised by make sense to me. In my understanding, most in-tree plugins will classify the event as an event.
No. When no fit Pod, UnschedulablePlugins will have plugin(s) rejected the Pod. (e.g., all nodes don't have enough capacity, UnschedulablePlugins has noderesourcefit. all nodes don't satisfy node affinity, UnschedulablePlugins has nodeaffinity) is a rare case like plugins other than PreFilter/Filter returns non-success. I don't think we should care about it.
But, this is a good point, I think we should refactor all plugins that register event with to have a specific Node update event. For example, . It should have with instead of . event with , that should be the last-resort when we don't know which kind of Node update will make Pods schedulable. (I guess no such case at least in the in-tree plugins.)
1) One possibility is Reserve Plugin, I the capability here 2) Another possibility is PreFilter, but not because failed by the plugin, but no fit node exists Have no idea whether we have other places, I guess no. But yes, this only impacts unschedulable pods has no failed plugins.
~I think this is for intended, like ~ ~1. 1) we have a plugin failed by nodeAffinity~ ~2. 2) we add a new node~ ~3. 3) we receive the node add event, but because node not ready, we failed by nodeTaint plugin as ~ 4) node become ready, we get the node update event, but dismatch with nodeAffinity registered event (if we registered with UpdateNodeLabel event)~ ~5. 5) pod will not be moved to the active or backoff queue~ updated below
weigh in for advices as whether this is a good idea to take nodeAnnotations as a new event.


Question: NodeAllocatableChange, NodeConditionChange, NodeLabelChange, <ins> NodeAnnotationChange, </ins> PvcAdd, PvcUpdate, PvAdd,

As I know, some plugins will read the embedded prometheus metrics in node annotations for scoring, but not for filtering. At least we should mitigate the impacts.
iv. node become ready, we get the node update event, but dismatch with nodeAffinity registered event (if we registered with UpdateNodeLabel event) (3) is not exact. When we receive the node add event, the scheduler retry to schedule the Pod because the nodeaffinity has Node add event. And it's rejected by the nodetaint plugin. (not node affinity) So, in (4), we get the node update event, it matches with node taint registered event, and the Pod is moved to active/backoffQ (node taint registers node update event. Yes I think we can change node taint's node update event to more specific one(s) though)
As I replied in your PR, we acknowledged that in , and yes let's have your patch. Well, that's a good point. So, when all nodes are filtered out by PreFilterResult, then no failedplugin can be returned. That's a problem. I think we should return the plugin name that returns PreFilterResult with not all Node names in this case (= the plugins that contributes to reducing the candidate Nodes via PreFilterResult). What do you think?
Sorry, the previous one is not right, a detailed one: a. we have a pod failed by nodeAffinity. pod will be placed into the unschedulablePods b. node autoscaler triggered, a new node , emitted a node add event c. called -d. node not ready right now and holds a notReady taint, then failed, pod still stays in the unschedulablePods e. node gets ready, emitted a node update event, but dismatch with the registered event(if we register with in nodeAffinity plugin) f. pod will stay in unschedulabelPods I guess this is the right flow. Anyway, glad to see it be more effect in the future.
If we have no performance problems, I think it's ok to take nodeAnnotations into consideration.
Wouldn't this be solved with proposal on ClusterEventWithHint? If so, I prefer we leave the existing code as-is.
whatever we do here, we wouldn't backport, so I think is the right move forward.
We put many stuff here.


Question: NodeAllocatableChange, NodeConditionChange, NodeLabelChange, <ins> NodeAnnotationChange, </ins> PvcAdd, PvcUpdate, PvAdd,

Let me summarize each: Born from from is right. But, that's the problem with preCheck. (Some preCheck depends on some in-tree plugins' logic and it affects badly to other plugins enqueueing ) The preCheck is removed after is completed. After preCheck is removed, will be the correct behavior. So, we can do the optimization that changes node to a specific event. will be fixed in: It's kind of rare case, but I feel we need to have some improvement like Edit: create another issue for this: I'd say (2) and (3) are kind of rare cases. So, the blocker here is (1). (And, (1) is blocked by the removal of preCheck, which is blocked by ) As my conclusion, it's safer to add after (1) is done so that we can make sure the performance won't be affected badly. what do you think?
Instead, can we just keep an event for "node updated", then individual plugins can decide whether a annotations change is worth requeuing or not?
Yes, (1) can be replaced with that idea. But, I'd say changing to a specific is slightly better because the scheduling queue could filter out some Node update events without executing the hint functions.
If rescheduling can be triggered with precise events, that would be great. I haven't revisit since last time, I'll take a look again.
Since we are missing an event for , can we add it? But we also have to modify existing events like so: Because change of such field is also an Update. And in future releases we can get rid of some fine-grained events.
Since we will add hintFuncs for all in-tree plugins, so I think it's ok to add nodeAnnotationsChanged event now, we can filter out unrelated events in hitFuncs. Any other concerns? cc Prefer to keep them as people can register fine-gained events instead a general event.
If by "now" you mean v1.30, then I'm ok with that.


Question: s, ok := h.serviceEndpointsMap.Get(serviceName) if !ok { glog.V(4).Infof("Service %s not found or has no local endpoints", serviceName) <del> sendHealthCheckResponse(rw, http.StatusServiceUnavailable, "No Service Endpoints Not Found") </del> <ins> sendHealthCheckResponse(rw, http.StatusServiceUnavailable, "No Service Endpoints Found") </ins> return } numEndpoints := len(*s.(*serviceEndpointsList).endpoints)

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? Currently, the 4th parameter name of updateServiceListener in kube-proxy is addOrDelete, However, from the func caller, we just pass for AddServiceListener, and for DeleteServiceListener, So, I think the name addOrDelete is unreasonable. I prefer to change the parameter name addOrDelete to add so that avoid making people confused. In addition, there is a typo, We should change addition to deletion! Also, there is a sentence breaking, we should change from No Service Endpoints Not Found =No Service Endpoints are found Thanks!
Fixed!


Question: } } <ins> func TestNodeExpand(t *testing.T) { // FS global path testing tmpFSDir, plug := getNodeExpandablePlugin(t, false) defer os.RemoveAll(tmpFSDir) pvSpec := getTestVolume(false, tmpFSDir, false, nil) resizeOptions := volume.NodeResizeOptions{ VolumeSpec: pvSpec, DevicePath: tmpFSDir, } // Actually, we will do no volume expansion if volume is of type dir resizeDone, err := plug.NodeExpand(resizeOptions) if err != nil { t.Fatal(err) } if !resizeDone { t.Errorf("expected resize to be done") } } </ins> func TestMountUnmount(t *testing.T) { tmpDir, plug := getPlugin(t) defer os.RemoveAll(tmpDir)

There are use cases where users want to expand the local volume (may be managed by LVM or something similar), but even though they can edit underlying PV (and expand the underlying device) - the reported size in pvc.Status keeps reporting older value because in-tree local volumes are non-expandable. We could make this easier for users by - implementing interface for local volumes. This would allow following steps for local volume expansion: /admin edits the pvc - requesting new size. expands the underlying device. edits the PV and sets new size. will expand the filesystem and update This would at least get the user from impossible territory to possible. cc /sig storage
/triage accepted
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
Hi & any news on this issue ? The related PR seems stalled.
merged. So this is fixed now.
Oh right ! I didn't looked close enough and thought it was only for directory based volumes. Thank you.


Question: func (p *podConditionUpdaterImpl) update(pod *v1.Pod, condition *v1.PodCondition) error { klog.V(3).Infof("Updating pod condition for %s/%s to (%s==%s, Reason=%s)", pod.Namespace, pod.Name, condition.Type, condition.Status, condition.Reason) <del> if podutil.UpdatePodCondition(&pod.Status, condition) { _, err := p.Client.CoreV1().Pods(pod.Namespace).UpdateStatus(context.TODO(), pod, metav1.UpdateOptions{}) </del> <ins> oldData, err := json.Marshal(pod) if err != nil { </ins> return err } <del> return nil </del> <ins> if !podutil.UpdatePodCondition(&pod.Status, condition) { return nil } newData, err := json.Marshal(pod) if err != nil { return err } patchBytes, err := strategicpatch.CreateTwoWayMergePatch(oldData, newData, &v1.Pod{}) if err != nil { return fmt.Errorf("failed to create merge patch for pod %q/%q: %v", pod.Namespace, pod.Name, err) } _, err = p.Client.CoreV1().Pods(pod.Namespace).Patch(context.TODO(), pod.Name, types.StrategicMergePatchType, patchBytes, metav1.PatchOptions{}, "status") return err </ins> } type podPreemptorImpl struct {

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: Which test(s) are flaking: Testgrid link: Reason for failure: Unknown yet. Anything else we need to know: links to appreciated links to specific failures in spyglass appreciated /sig scheduling /help
Can I give it a try?
Sure, feel free to take it. /assign ahmad-diaa /remove-help
I keep trying to get it fail locally by running the following command: but it never does, am I doing something wrong? Is there something else that I should be doing and would be more helpful?
Sometimes the flake is hard to reproduce in local env. You may have to read the error log of the flakes thoroughly to get some clues. BTW: you can also run the test multiple times using the directive:
After looking through the logs I came up with some information that might be useful As far as I understand, the reason for the flake is the of the medium priority pod not being cleared That is because the update request for removing returns the following error: So my current guess is that might be outdated in some cases As I am not really familiar with the logic, I would like to know whether or not those are valid points so that I could dig deeper. What do you think?
If that's the case, you can set here: You can raise a PR and manually try for ~20 times to see you can reproduce the flake.
found another issue tracking this too. it seems it has been flaky for a while.
Sorry, didn't mean to take this one for you. I worked the duplicate issue and submitted a PR. Feel free to review that PR, or submit your own if you have a different solution.
This sometimes happens. But we usually take the first PR unless it's clearly worked by someone. In terms of which style to resolve the Conflict error, I'm asking here:
Sorry, didn't mean to take this one for you. I worked the duplicate issue and submitted a PR. Feel free to review that PR, or submit your own if you have a different solution. No worries, I haven't started implementing yet and your PR seems to be following the correct approach as well according to this:


Question: func (p *podConditionUpdaterImpl) update(pod *v1.Pod, condition *v1.PodCondition) error { klog.V(3).Infof("Updating pod condition for %s/%s to (%s==%s, Reason=%s)", pod.Namespace, pod.Name, condition.Type, condition.Status, condition.Reason) <del> if podutil.UpdatePodCondition(&pod.Status, condition) { _, err := p.Client.CoreV1().Pods(pod.Namespace).UpdateStatus(context.TODO(), pod, metav1.UpdateOptions{}) </del> <ins> oldData, err := json.Marshal(pod) if err != nil { </ins> return err } <del> return nil </del> <ins> if !podutil.UpdatePodCondition(&pod.Status, condition) { return nil } newData, err := json.Marshal(pod) if err != nil { return err } patchBytes, err := strategicpatch.CreateTwoWayMergePatch(oldData, newData, &v1.Pod{}) if err != nil { return fmt.Errorf("failed to create merge patch for pod %q/%q: %v", pod.Namespace, pod.Name, err) } _, err = p.Client.CoreV1().Pods(pod.Namespace).Patch(context.TODO(), pod.Name, types.StrategicMergePatchType, patchBytes, metav1.PatchOptions{}, "status") return err </ins> } type podPreemptorImpl struct {

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: pull-kubernetes-integration Which test(s) are flaking: TestNominatedNodeCleanUp Testgrid link: Reason for failure: [TODO] Anything else we need to know: links to appreciated links to specific failures in spyglass appreciated
/sig node These SIGs are my best guesses for this issue. Please comment if I am incorrect about one. I am a bot run by vllry.


Question: func (p *podConditionUpdaterImpl) update(pod *v1.Pod, condition *v1.PodCondition) error { klog.V(3).Infof("Updating pod condition for %s/%s to (%s==%s, Reason=%s)", pod.Namespace, pod.Name, condition.Type, condition.Status, condition.Reason) <del> if podutil.UpdatePodCondition(&pod.Status, condition) { _, err := p.Client.CoreV1().Pods(pod.Namespace).UpdateStatus(context.TODO(), pod, metav1.UpdateOptions{}) </del> <ins> oldData, err := json.Marshal(pod) if err != nil { </ins> return err } <del> return nil </del> <ins> if !podutil.UpdatePodCondition(&pod.Status, condition) { return nil } newData, err := json.Marshal(pod) if err != nil { return err } patchBytes, err := strategicpatch.CreateTwoWayMergePatch(oldData, newData, &v1.Pod{}) if err != nil { return fmt.Errorf("failed to create merge patch for pod %q/%q: %v", pod.Namespace, pod.Name, err) } _, err = p.Client.CoreV1().Pods(pod.Namespace).Patch(context.TODO(), pod.Name, types.StrategicMergePatchType, patchBytes, metav1.PatchOptions{}, "status") return err </ins> } type podPreemptorImpl struct {

Which jobs are flaking: pull-kubernetes-integration Which test(s) are flaking: TestNominatedNodeCleanUp Testgrid link: Reason for failure: /sig scheduling
/assign
any progress on this one?
I was working on it and tried to find the root cause. I will give my solution within this week
Another one
Still happening cc
/assign
/close dupe
Closing this issue. <detailsIn response to : /close dupe Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: }) <del> ginkgo.It("should delete a collection of pods", func() { </del> <ins> /* Release : v1.19 Testname: Pods, delete a collection Description: A set of pods is created with a label selector which MUST be found when listed. The set of pods is deleted and MUST NOT show up when listed by its label selector. */ framework.ConformanceIt("should delete a collection of pods", func() { </ins> podTestNames := []string{"test-pod-1", "test-pod-2", "test-pod-3"} ginkgo.By("Create set of pods")

[x] Original approval issue: 09 July 2020 [x] Two weeks soak end date: 24 July 2020 This issue is created to allow edit by /wip /hold According to this APIsnoop query, there are still an endpoint which is untested. a set of 3 Pods with a static label within a Namespace. that all 3 Pods with the label in the Namespace are created. the set of Namespaced Pods with a label via DeleteCollection. that all Pods with the label have been deleted. Discover useragents: List endpoints hit by the test: Display endpoint coverage change: Two other endpoints tested here have tests in progress to conformance already. If a test with these calls gets merged, test coverage will go up by 1 points This test is also created with the goal of conformance promotion.
/sig testing /sig architecture /area conformance
During Conformance Meeting: Is there a way to write this test so it is resource agnostic? If we are just changing the resource... can be parameterize the resource-type
/assign
GitHub didn't allow me to assign the following users: heyste. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Test to be created using "Polling" and not "Watch tooling"
/reopen Merged Issue open, waiting for Promotion PR
Reopened this issue. <detailsIn response to : /reopen Merged Issue open, waiting for Promotion PR Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
The test appears to be flaking, need to reach out to #sig-node for some feedback.
Can you point us to the link of the flaky tests? Thanks!
The Promotion The testgrid Thank you!


Question: verifyWatchEvent(t, watcher, watch.Added, podFoo) verifyWatchEvent(t, watcher, watch.Modified, podFooPrime) <del> // Check whether we get too-old error. _, err = cacher.Watch(context.TODO(), "pods/ns/foo", "1", storage.Everything) if err == nil { t.Errorf("Expected 'error too old' error") </del> <ins> // Check whether we get too-old error via the watch channel tooOldWatcher, err := cacher.Watch(context.TODO(), "pods/ns/foo", "1", storage.Everything) if err != nil { t.Fatalf("Expected no direct error, got %v", err) </ins> } <ins> defer tooOldWatcher.Stop() // Ensure we get a "Gone" error expectedGoneError := errors.NewGone("").(*errors.StatusError).ErrStatus verifyWatchEvent(t, tooOldWatcher, watch.Error, &expectedGoneError) </ins> initialWatcher, err := cacher.Watch(context.TODO(), "pods/ns/foo", fooCreated.ResourceVersion, storage.Everything) if err != nil {

When attempting to fetch a resource version that is too old, the behavior of watch (for both HTTP and web sockets) changed when watch cache was enabled. The watch cache returns a 410 - without the watch cache, we return 200 and then write an "error" entry to the watch stream. This broke a client that dependent on the watch behavior, and moreover is inconsistent with other errors we set in the registry. Also, for watch over web sockets, most javascript clients will be unable to get at a 410 error from the connection, because browsers don't get access to the data. We should restore the watch cache behavior to be consistent with the previous behavior as this is a regression in API semantics. Resource without watch cache, watching too old: Resource with watch cache, watching too old:
I think this is candidate for back port to 1.2 to restore the previous behavior.
since this broke your javascript client, FYI because of watch cache
+1 for backport
hmm - we enabled watch cache in 1.1 - so this should also be broken in 1.1
It's possible that it was always nondeterministic and just depends on how fast the call to the storage layer executes. I think all of our in-tree clients will handle it either way.
I'm fairly certain this never used to behave this way, because OpenShift hasn't enabled the watch catch and we have never had a report of this until we enabled the watch cache for Kube resources just recently. , Daniel Smith wrote:
I see what you mean about ordering - etcd remote call would almost never be fast enough to beat the watch event. In practice, I suspect no one ever saw this without the watch error. Either way, for web sockets if we don't fix this clients can't get this error (the browser eats the 410 during the websocket negotiation and client code can't get access). So it's at least broken for web sockets. , Clayton Coleman wrote:
Well, it should be easy to just not wait at all, maybe? I'm OK with changing this behavior.
Although, are web clients XSS hardened? I'm not sure that we're optimizing for them currently.
If you mean CSRF, it isn't an issue with bearer token auth.


Question: verifyWatchEvent(t, watcher, watch.Added, podFoo) verifyWatchEvent(t, watcher, watch.Modified, podFooPrime) <del> // Check whether we get too-old error. _, err = cacher.Watch(context.TODO(), "pods/ns/foo", "1", storage.Everything) if err == nil { t.Errorf("Expected 'error too old' error") </del> <ins> // Check whether we get too-old error via the watch channel tooOldWatcher, err := cacher.Watch(context.TODO(), "pods/ns/foo", "1", storage.Everything) if err != nil { t.Fatalf("Expected no direct error, got %v", err) </ins> } <ins> defer tooOldWatcher.Stop() // Ensure we get a "Gone" error expectedGoneError := errors.NewGone("").(*errors.StatusError).ErrStatus verifyWatchEvent(t, tooOldWatcher, watch.Error, &expectedGoneError) </ins> initialWatcher, err := cacher.Watch(context.TODO(), "pods/ns/foo", fooCreated.ResourceVersion, storage.Everything) if err != nil {

Basic auth should already have a big warning around it saying "not safe with browsers"
Maybe I'm missing something, but it seems deterministic to me. This is the uncached watch impl: once the resourceVersion parses, there is no path that returns a direct error, so all errors would have to be returned as an event via the ResultChan(). I don't see us reading from that until after we've already written a 200 and flushed:
Having read the docs, I was very surprised that "type": "ERROR" notices are sent as 200 (even over plain http), with the semantic code e.g. 410 only represented under inside. I see there was good motivation here to stick to 200, but it should be called out in the documentation. doesn't show at all how watch returns errors, and it even says "clients must handle the case by recognizing the status code 410 Gone" which really makes one expect an actual HTTP 410.
I agree this should be called out in the docs. Can you file a bug there and/or send a PR? I don't think this ~4 year old issue is the best place to track that. (Good detective work finding this, though!)


Question: github.com/stretchr/objx v0.1.1/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= github.com/stretchr/testify v1.2.2 h1:bSDNvY7ZPG5RlJ8otE/7V6gMiyenm9RtJ7IUVIAoJ1w= github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs= <del> github.com/vmware/govmomi v0.20.0 h1:+1IyhvoVb5JET2Wvgw9J3ZDv6CK4sxzUunpH8LhQqm4= github.com/vmware/govmomi v0.20.0/go.mod h1:URlwyTFZX72RmxtxuaFL2Uj3fD1JTvZdx59bHWk6aFU= </del> <ins> github.com/vmware/govmomi v0.20.1 h1:7b/SeTUB3tER8ZLGLLLH3xcnB2xeuLULXmfPFqPSRZA= github.com/vmware/govmomi v0.20.1/go.mod h1:URlwyTFZX72RmxtxuaFL2Uj3fD1JTvZdx59bHWk6aFU= </ins> golang.org/x/crypto v0.0.0-20181025213731-e84da0312774 h1:a4tQYYYuK9QdeO/+kEvNYyuR21S+7ve5EANok6hABhI= golang.org/x/crypto v0.0.0-20181025213731-e84da0312774/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4= golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: The vSphere cloud provider is unable to fetch the node failure domain labels when using VCP with SAML auth because of an authentication failure. For context, the SAML token auth is described here - SAML auth support for the zones api was introduced in this PR - , which was cherry-picked to 1.14.1 here - I've tested the credentials I'm using with govc to ensure that I'm able to get a token: The credentials also work fine when using VCP without the node labels feature. Relevant kubelet logs: What you expected to happen: VCP should add failure domain labels to the node object. How to reproduce it (as minimally and precisely as possible): Setup SAML auth for VCP with zone labeling enabled. Anything else we need to know?: There was a previous issue that was raised () but that was before SAML auth support for zones was implemented. This could potentially be an issue with govc handling of SAML tokens since even with the Administrator role assigned to the user, I'm still unable to list tags. The role and keys were generated with the following: Environment: Kubernetes version (use ): 14.1 Cloud provider or hardware configuration: vSphere OS (e.g: ): Linux & Windows
/sig vmware
Would you mind taking a look at this? I can provide more info if needed.
govc's command only works with the SOAP endpoint currently. The REST endpoint has its own session manager and requires a separate login. At the moment, govc's tags commands login/logout when each command is run and don't use a persisted session (which are saved to ~/.govmomi/sessions). However, you can use a token returned by to test SAML auth with the tags commands. For example: Can you give that try? There are some TODOs around this btw:
Those commands work for me. So that would imply the solution user, role, and permissions are set up correctly? Would you expect the to also work?


Question: github.com/stretchr/objx v0.1.1/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= github.com/stretchr/testify v1.2.2 h1:bSDNvY7ZPG5RlJ8otE/7V6gMiyenm9RtJ7IUVIAoJ1w= github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs= <del> github.com/vmware/govmomi v0.20.0 h1:+1IyhvoVb5JET2Wvgw9J3ZDv6CK4sxzUunpH8LhQqm4= github.com/vmware/govmomi v0.20.0/go.mod h1:URlwyTFZX72RmxtxuaFL2Uj3fD1JTvZdx59bHWk6aFU= </del> <ins> github.com/vmware/govmomi v0.20.1 h1:7b/SeTUB3tER8ZLGLLLH3xcnB2xeuLULXmfPFqPSRZA= github.com/vmware/govmomi v0.20.1/go.mod h1:URlwyTFZX72RmxtxuaFL2Uj3fD1JTvZdx59bHWk6aFU= </ins> golang.org/x/crypto v0.0.0-20181025213731-e84da0312774 h1:a4tQYYYuK9QdeO/+kEvNYyuR21S+7ve5EANok6hABhI= golang.org/x/crypto v0.0.0-20181025213731-e84da0312774/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4= golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=

I seem to be getting a 404 when using with the flag - Does seem to work when specifying a tag name: Are there any other govc commands I should check / verify?
yes, the solution user setup looks right. Not sure what the issue is with . Can you share your vCenter build number ()? I'll try to reproduce the issue.
$ govc about -u vcdo Name: VMware vCenter Server Vendor: VMware, Inc. Version: 6.7.0 Build: OS type: linux-x64 API type: VirtualCenter API version: 6.7.2 Product ID: vpx UUID: -55ed-4737-9cc4-
I can reproduce this, but don't know what the issue is yet. The version of Go and vSphere have changed since , but if it was one of those was the cause, I'd expect SAML auth to fail with the govc tags commands too. All I have to go on for now is the vCenter log message below. I'll update when I have more info.
I have it working with the change below, but need to figure out a few things first: Why this is the case when run inside kubelet and not govc (note that the port change to 443 breaks govc) A good way to determine which port to use (depends on the answer to the above) If this is behavior is specific to 6.7U2 or something triggered by a change in K8s I suppose a quick fix would change to try port 443 if 80 fails, but would prefer to avoid that as the failure generates stack traces in the
/assign
This is obvious in retrospect, but the issue is the message signature needs to use the same port as the original request URL. However, in the govc case there was no port set in the request URL. The client defaults to port when connecting (based on the scheme), but does not include any port with the header. In the case, the default is on the server side. This can be reproduced with if includes the port , rather than just as we were using. The VCP tests that use token auth for tags lookup just don't validate the signature, so this wasn't a change related to 6.7U2.


Question: github.com/stretchr/objx v0.1.1/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= github.com/stretchr/testify v1.2.2 h1:bSDNvY7ZPG5RlJ8otE/7V6gMiyenm9RtJ7IUVIAoJ1w= github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs= <del> github.com/vmware/govmomi v0.20.0 h1:+1IyhvoVb5JET2Wvgw9J3ZDv6CK4sxzUunpH8LhQqm4= github.com/vmware/govmomi v0.20.0/go.mod h1:URlwyTFZX72RmxtxuaFL2Uj3fD1JTvZdx59bHWk6aFU= </del> <ins> github.com/vmware/govmomi v0.20.1 h1:7b/SeTUB3tER8ZLGLLLH3xcnB2xeuLULXmfPFqPSRZA= github.com/vmware/govmomi v0.20.1/go.mod h1:URlwyTFZX72RmxtxuaFL2Uj3fD1JTvZdx59bHWk6aFU= </ins> golang.org/x/crypto v0.0.0-20181025213731-e84da0312774 h1:a4tQYYYuK9QdeO/+kEvNYyuR21S+7ve5EANok6hABhI= golang.org/x/crypto v0.0.0-20181025213731-e84da0312774/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4= golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=

I'll open a K8s PR to pull in the fix:
Updating k/vendor to govmomi master adds almost 6K lines for a 3 line fix. I may just create a govmomi branch off of v0.20 to reduce churn in the K8s release branches for this fix, may have to wait until tomorrow.
Really appreciate the insight into this issue. Can confirm that specifying the port (443) in the GOVC_URL reproduces the issue. Thank you!
Unfortunately, I'm seeing the same issue with v1.14.3: cloud_config Works fine when I use the username and password of a vsphere admin, but using cert and key still has authentication issues.
any related errors in vCenter's ?
Seeing a lot of this exception:
whoa! that stack trace is like punch in the gut (brings back memories!) :)
indeed!! apologies, I must have had the change for PR locally, but failed to include it as I had focused on the signature issue and updating vendor/


Question: package foo // Blah is a test. <del> // +k8s:openapi=true </del> <ins> // +k8s:openapi-gen=true // +k8s:openapi-gen=x-kubernetes-type-tag:type_test </ins> type Blah struct { // A simple string String string

Background: kubectl and the Kubernetes reference docs both depend on the spec to operate on Kubernetes resources. The lacks important information required by kubectl and the docs. Usecase: kubectl can identify the patchStrategy and mergeKey for resource fields kubectl can identify the columns to print for on resources docs can parse and display the default value for fields docs can identify top-level objects Solution: Add items to the comment tag: Example: The long term plan is to remove the and struct tags once all code has been updated to use the swagger spec instead. Until then, comments should be for the struct tags and . Some verification process will need to be written to make sure the struct tags and comments remain in sync.
I think this wouldn't fall under sig-apps. cc
I would not do the columns stuff - we're moving get to the server side. Top level objects is already available in some form by the client - specifically the list that we use (should use) to get the swagger doc tells us what the top level type is (or is that for something like explain). Merge key, patch strategy, default values all belong as "real" attributes, although we had not necessarily intended those to be arguments to k8s:openapi-gen, but actual tags in their own right (like "+optional")
Thanks Is there a design proposal / eta for this? This is for generating the docs from swagger. Right now the list of top level resources for the docs to display is hardcoded in a . I would like to move the majority of the metadata out of the config file so that: same tool can generate reference docs for federated apiservers without writing a custom config. don't need to manually maintain and update the config for each release - e.g. adding new types I have no strong opinion about this as long as they appear in the swagger so it is possible right tools the can use this data. Can you comment on why openapi-gen vs specific tags?
Describe in the server makes sense to me. IIUC, just prints out a set of columns. Was the idea that the server would just print out the columns or return a list of column / values or something more complicated? No need to answer if it is in the issue / proposal already.


Question: package foo // Blah is a test. <del> // +k8s:openapi=true </del> <ins> // +k8s:openapi-gen=true // +k8s:openapi-gen=x-kubernetes-type-tag:type_test </ins> type Blah struct { // A simple string String string

is the bigger issue that touches on a rough overview, I can't find my rambling brain dump of the problem, but get would be rough tabular json, you could also get that as csv, and describe could be a richer but not quite full structured key/value structure.
regarding tags, they do not need to be in the same k8s:openapi-gen tag, it can be repeated: they can also be separate tags but still general like this: The idea is to have a general way to add arbitrary extension to a type so we don't need to change code generator for each specific tag. I have no strong feeling about the format, but I would like it to be general vs tags like +optional. The difference is +optional is a defined tag in OpenAPI () but extensions does not have any meaning for code generator. They better to be general so generate can pass them without change.
No, I mean we should define conventions in our API code that match the intent (optional) rather than have tons and tons of repeated code that is setting x-kubernete-describe-columns (or similar)
I spent some time thinking about your comments last night. I don't think this is mutually exclusive with , and could be complementary. IMHO this is general metadata associated with the type, and including it in the doesn't preclude supporting an api endpoint that pre-parses and formats the data. However, including it in the swagger does allow clients using the swagger to take advantage of it. Following is an example of where the swagger would be useful even if get was in the sever. A generated Java client could use the display columns for the "toString" method of the resource. Another use case that had mentioned to me was to allow some kubectl commands to operate on directly on local files. e.g. (strawman) could print the columns. I don't think we want to do this now, but it is an example of why keeping this object metadata private to the server limits what we can do. My hope is that this swagger tags solution will also oprove portable to easily work with ThirdPartyResources as well by writing the display-columns as an annotation on the TPR and keeping the same format. is a can of worms, and I would agree that the solution is capable of providing a richer experience for aggregating information across multiple resource types - e.g.


Question: package foo // Blah is a test. <del> // +k8s:openapi=true </del> <ins> // +k8s:openapi-gen=true // +k8s:openapi-gen=x-kubernetes-type-tag:type_test </ins> type Blah struct { // A simple string String string

how describing Nodes fetches Pod information. I still think that the solution is compatible with the richer solution you proposed and there is not much additional cost in doing it. Howv What would you think of supporting a set of optional ALPHA extensions for "display-columns" and "describe-columns" allowing any consumers of the swagger to print the objects. In theory it could also be used by the UI to provide a summary view without having to do separate lookups. As an ALPHA feature we could try it for supporting sig-service-catalog and revisit it in 1.7. If we consider it successful, the solution outlined in could become another consumer of the display-columns tag? I like this approach. My biggest concern here is being able to get this into 1.6. Since kubectl supports +/-1 version skew, we need will need to maintain the old logic for at least 1 full release after we add support for the new swagger based logic. That puts us at 6 months from now to fully migrate if we don't have any swagger extensions in 1.6.
I think that's a nice to have in the long run - but then people will say "why are the columns different when the server prints them from the local print"? And most of the pod columns in no way can represent the full complexity of what they print with swagger, so the best effort local printing is going to be really, really, really best effort. Also, local print is a really specialized case - if we just did the generic metadata print that would probably be ok. So while I agree it's valid, it seems like a lot of work for very little practical value. Again, not opposed, just don't want to do two things halfway and then have nothing. So just biased towards not adding more client complexity in the near term. So you want to hack this in for service catalog in 1.6 on TPR? I feel like that's a lot of hacks - the generic printer stuff at least gives you name and namespace, and I'd rather do nothing in 1.7 for generic client printer rather than jump ahead. I won't stop you if you want to spend time and effort on it, but it's a lot of specialized work that will then be neglected while we do the server side stuff.


Question: package foo // Blah is a test. <del> // +k8s:openapi=true </del> <ins> // +k8s:openapi-gen=true // +k8s:openapi-gen=x-kubernetes-type-tag:type_test </ins> type Blah struct { // A simple string String string

My intend here was avoiding repeated code for adding each of these tags somewhere (in openapi-gen code or some root level place). Let me distinguish these tags in two categories with respect to OpenAPI code generator: that need some sort of processing that affects OpenAPI spec (e.g. +optional removes the field from OpenAPI field) that need no more processing and will be directly put into OpenAPI spec (e.g. +gvk that we don't have but we can define as a way to define Group/Version/Kind of an object vs register method) My argument is about the second one. I think patch strategy here falls into this category. The question is only about naming it. Naming Option 1: +patch-strategy=... This looks cleaner but we need to define an alias somewhere to tell generator this is the same as x-kubernetes-patch-strategy. The alias definition and the tags are in two places. Putting them in the generator code make the code non-generic and we don't want to do that. Putting these aliases somewhere as an argument to the generator begs for bugs as people that look at the type and add this tag to their types need to know (maybe by reading a doc, but who do that) that it needs to pass the alias to the generator. I, in 6 month, will look at a type and see a tag and copy the tag to my type and expect it to work that will not with this option. Naming Option 2: use something generic for these type of tags. after all generator does not need to do any processing. I suggest +openapi-tag:tag=value that will result in "x-kubernetes-tag: value" in OpenAPI spec. There is no alias defined anywhere and without any code change or alias copying or any knowledge about passing any argument, a tag can be to OpenAPI spec. The value of these tags and their use cases are outside of the scope of what I am arguing about. Anyway a solution is better than no solution, I would suggest we continue with Option 2 as Alpha and change it later if we see fit. Adding an aliasing mechanism to this option later should be easy.
On patch strategy, I had assumed we'd read the value from what's in the struct tag, rather than requiring a net new tag?


Question: package foo // Blah is a test. <del> // +k8s:openapi=true </del> <ins> // +k8s:openapi-gen=true // +k8s:openapi-gen=x-kubernetes-type-tag:type_test </ins> type Blah struct { // A simple string String string

, Mehdy Bohlool <:
The get stuff is less urgent for me as long as we aren't doing even more hacky things to work around the current limitations, and can make a commitment to delivering it in a specific release (sounds like 1.7). For service catalog there was a plan to fork kubectl, compile the service catalog types into the binary, and provide the fork as a separate binary. I have some concerns with this approach. I would rather provide a temporary hack in the code that can provide a consistent user experience from day one and re-implement later vs having users get used to downloading separate binaries for each aggregated apiserver. I have no plans to do any work for on this for the core Kubernetes types, and can punt on it until the scope of the work is understood to be relatively small and easily upgraded to the server-side get.


Question: } }) <del> // Marking this as LinuxOnly because windows tests run against Azure CNI which doesn't publish // a podCIDR for each family. ginkgo.It("should have ipv4 and ipv6 node podCIDRs [LinuxOnly]", func() { // TODO (aramase) can switch to new function to get all nodes nodeList, err := e2enode.GetReadySchedulableNodes(cs) framework.ExpectNoError(err) for _, node := range nodeList.Items { framework.ExpectEqual(len(node.Spec.PodCIDRs), 2) // assert podCIDR is same as podCIDRs[0] framework.ExpectEqual(node.Spec.PodCIDR, node.Spec.PodCIDRs[0]) // assert one is ipv4 and other is ipv6 framework.ExpectEqual(netutils.IsIPv4CIDRString(node.Spec.PodCIDRs[0]) != netutils.IsIPv4CIDRString(node.Spec.PodCIDRs[1]), true) } }) </del> ginkgo.It("should create pod, add ipv6 and ipv4 ip to pod ips", func() { podName := "pod-dualstack-ips"

The test ""should have ipv4 and ipv6 node podCIDRs" depends on a feature that is optional (one of the ipam controllers in the controller manager) We have discussed this previously, but since this is actually affecting downstream CIs I propose to remove it form the e2e framework. For cloud providers, it can be moved to the own external cloud provider code, since the policy is to not host more tests with dependencies. For the feature itself, it can easily be replaced by a unit or an integration test. ref:
we have discussed it before, but now is affecting downstream CIs and is really a controller-manager test. /cc /cc for sig-testing visibility /sig network
Removing the test sounds reasonable as it's specific to cloud providers. I'll review if we have enough unit tests for this and open a PR to remove it from e2e.
We can add more if they don't exist, as a condition to remove the e2e
/assign
/triage accepted
It's not specific to cloud providers. It's specific to different network plugins; some require it and some don't.
I think that the comment is related to the e2e framework skipper, that only skips by cloud provider, but yeah, it is about CNIs


Question: - path: test/conformance/image/Makefile match: BASE_IMAGE_VERSION?= <del> - name: "registry.k8s.io/debian-iptables: dependents" version: bullseye-v1.4.0 </del> <ins> - name: "registry.k8s.io/distroless-iptables: dependents" version: v0.1.1 </ins> refPaths: - path: build/common.sh <del> match: __default_debian_iptables_version= </del> <ins> match: __default_distroless_iptables_version= </ins> - path: test/utils/image/manifest.go <del> match: configs[DebianIptables] = Config{list.BuildImageRegistry, "debian-iptables", "[a-zA-Z]+-v((([0-9]+).([0-9]+).([0-9]+)(?:-([0-9a-zA-Z-]+(?:.[0-9a-zA-Z-]+)*))?)(?:+([0-9a-zA-Z-]+(?:.[0-9a-zA-Z-]+)*))?)"} </del> <ins> match: configs[DistrolessIptables] = Config{list.BuildImageRegistry, "distroless-iptables", "v([0-9]+).([0-9]+).([0-9]+)"} </ins> - name: "registry.k8s.io/go-runner: dependents" version: v2.3.1-go1.18.3-bullseye.0

Istio folks have a iptables image here that is quite small In the past, we did talk about this, for example wrote it well what we could try: We could even do for running the hack script we have for switching the symlinks when vs : kube-proxy image is definitely one of the main causes of the cost we incur in our GCR repository, so we should definitely give this a shot again. /sig network
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I don't think this is a sig/network-only issue. A gives; The looks like this (narroved): If I don't misinterpret this, K8s uses a unified server-image build and it should not be different for . The size of the binary for amd64 in is; It is stripped and can't easily be "made smaller" I guess.
We (Istio) would definitely be interested in sharing ownership (hopefully by upsteaming to distroless mainline) in the image we have if there is interest. Some other projects have expressed similar needs as well
!
kube-proxy make up 9% of current image data traffic
I threw together this little repo (based on our next-gen distroless infrastructure): It pushed which on x64 is roughly another 5x smaller (musl vs. glibc): If folks are interested in this, I'd be happy to create a repo under which is where we've been standing up new images, or you can create your own from pretty easily.
debian based please!! we have KEPs and stuff written that lay down that direction:
For a possible approach + above in that issue and for "distroless" debian based images with arbitrary C/C++ binaries + deps. Note that we need things besides just the distro binaries (see comment linked). and is pretty straightforward for one way to handle the debian binaries part.
I don't even know how we build the kube-proxy image. Where does that happen and form which dockerfile?


Question: - path: test/conformance/image/Makefile match: BASE_IMAGE_VERSION?= <del> - name: "registry.k8s.io/debian-iptables: dependents" version: bullseye-v1.4.0 </del> <ins> - name: "registry.k8s.io/distroless-iptables: dependents" version: v0.1.1 </ins> refPaths: - path: build/common.sh <del> match: __default_debian_iptables_version= </del> <ins> match: __default_distroless_iptables_version= </ins> - path: test/utils/image/manifest.go <del> match: configs[DebianIptables] = Config{list.BuildImageRegistry, "debian-iptables", "[a-zA-Z]+-v((([0-9]+).([0-9]+).([0-9]+)(?:-([0-9a-zA-Z-]+(?:.[0-9a-zA-Z-]+)*))?)(?:+([0-9a-zA-Z-]+(?:.[0-9a-zA-Z-]+)*))?)"} </del> <ins> match: configs[DistrolessIptables] = Config{list.BuildImageRegistry, "distroless-iptables", "v([0-9]+).([0-9]+).([0-9]+)"} </ins> - name: "registry.k8s.io/go-runner: dependents" version: v2.3.1-go1.18.3-bullseye.0

Note we need iptables, iptables-save and iptables-restore but also ipvsadm, shell, conntrack, ipset, and maybe other tools
It used to be in kubernetes/kubernetes under build/images, it's now in (Dockerfile etc there)
That's the base - where is the logic to turn that into a kube-proxy image?
Turning that image into a kube-proxy image is just adding the kube-proxy binary with a dockerfile and entirely too much bash which ultimately just does: (except api-server, where we do a setcap in )
/triage accepted Will take a look between this and next week
It doesn't even need bash; the version of the script in intentionally avoids bash-specific features (and gets tested against Alpine where /bin/sh is busybox (which I guess is "ash"?)). And the version in kubernetes/release appears to be just a stripped-down version of that script. (But actually, you should update the script to be based on the latest version anyway, to get the new logic in , in preparation for .)
Probably per one of the things we don't want to do right now is jump into a musl based image like Alpine, right? So I would take the case that we want to use distroless, and copy all the binaries we need to inside that container (conntrack, ipvs, iptables-*, etc etc). per iptables-wrapper, you said it's not bash specific so maybe we can use distroless + the smaller available shell? I will check some approaches here before jumping into changing the release/dockerfiles/whatever scripts, and check if I can run some comformance tests in a distroless k/proxy. Maybe starting with a good binary list is a way to cross check the binaries + libraries required for this distroless container. I did a list with the installed packages per kproxy needs, and some ldd on it: Will start working on something to assemble an image just with the binaries and required libraries, and check size + libraries
I agree no musl/alpine On Sun, Apr 17, 2022, 11:38 AM Ricardo Katz
If you identify the list of binaries, has logic to handle ldd for a binary (e.g. ignore vdso) + keep the copyright and package version info via dpkg.


Question: - path: test/conformance/image/Makefile match: BASE_IMAGE_VERSION?= <del> - name: "registry.k8s.io/debian-iptables: dependents" version: bullseye-v1.4.0 </del> <ins> - name: "registry.k8s.io/distroless-iptables: dependents" version: v0.1.1 </ins> refPaths: - path: build/common.sh <del> match: __default_debian_iptables_version= </del> <ins> match: __default_distroless_iptables_version= </ins> - path: test/utils/image/manifest.go <del> match: configs[DebianIptables] = Config{list.BuildImageRegistry, "debian-iptables", "[a-zA-Z]+-v((([0-9]+).([0-9]+).([0-9]+)(?:-([0-9a-zA-Z-]+(?:.[0-9a-zA-Z-]+)*))?)(?:+([0-9a-zA-Z-]+(?:.[0-9a-zA-Z-]+)*))?)"} </del> <ins> match: configs[DistrolessIptables] = Config{list.BuildImageRegistry, "distroless-iptables", "v([0-9]+).([0-9]+).([0-9]+)"} </ins> - name: "registry.k8s.io/go-runner: dependents" version: v2.3.1-go1.18.3-bullseye.0

We probably need a few more not in the list above for the other proxy modes?
Thanks this is gonna be very helpful! I will work on it using the script you provided during the week!
/cc
From script and some changes: There's a pretty good reduction of size (50%). But this is without kproxy binary, that is almost 40Mb. I will open a Draft PR in k/release so we can discuss over there the approach. BTW, anyone knows why kube-proxy binary is that big? Never digged into that, but curious about it.
https://dr- !
(updated with conntrack-tools) Why? The binary is statically linked so it does not need or use , it only needs some binaries to be executable and it doesn't matter if they are built with musl. They just have to work. I have sucessfully built and executed my own tests, both with proxy-mode and , using an image built with this : The image size on cluster including the binary (~40MB) is; (I use a private local registry) Compare to;
what about the different iptables-nft vs iptables-legacy binaries?
I have assumed that takes care of that (executed by the Dockerfile).
Re size: Are DWARF2 symbols stripped? Ideally k8s should produce artifacts without DWARF2 symbols and distribute them separatelly. Or produce the optional -debug artifacts. This can cut GCR/GCS costs, but introduce some debug/profiling complexity. The Go symbol table should be always included. Related
IIRC wrapper in prod needs both binaries existing in the container, and will use this in a way to check if the host machine is using iptables-legacy or iptables-nft.
Both exist in Alpine; The first example in is Alpine. can you please confirm that it works with Alpine?
I have altered my testenv so I can switch between legacy and nft and verified that the tests work with both and that shows the same in the container and the K8s node, e.g;
Side remark; to my suprise is not needed in the kube-proxy image
Tested the with K8s e2e;
There are / we're legal reasons to avoid Alpine. Have those been resolved? I haven't heard anything.


Question: - path: test/conformance/image/Makefile match: BASE_IMAGE_VERSION?= <del> - name: "registry.k8s.io/debian-iptables: dependents" version: bullseye-v1.4.0 </del> <ins> - name: "registry.k8s.io/distroless-iptables: dependents" version: v0.1.1 </ins> refPaths: - path: build/common.sh <del> match: __default_debian_iptables_version= </del> <ins> match: __default_distroless_iptables_version= </ins> - path: test/utils/image/manifest.go <del> match: configs[DebianIptables] = Config{list.BuildImageRegistry, "debian-iptables", "[a-zA-Z]+-v((([0-9]+).([0-9]+).([0-9]+)(?:-([0-9a-zA-Z-]+(?:.[0-9a-zA-Z-]+)*))?)(?:+([0-9a-zA-Z-]+(?:.[0-9a-zA-Z-]+)*))?)"} </del> <ins> match: configs[DistrolessIptables] = Config{list.BuildImageRegistry, "distroless-iptables", "v([0-9]+).([0-9]+).([0-9]+)"} </ins> - name: "registry.k8s.io/go-runner: dependents" version: v2.3.1-go1.18.3-bullseye.0

On Tue, Apr 19, 2022, 11:31 AM Lars Ekman
i don't think so. it's status quo on Alpine which is to avoid it totally.
Can I read about those somewhere? Are the legal reasons about Alpine? Or about musl?
Not my personal decision, and I'm not a lawyer let alone an OSS license expert but ... sharing what I've heard on the subject: IIRC for example Alpine does not make available the source code (AFAICT Alpine publishes source code for the packaging scripts, but not the package sources), or even the LICENSE of binary packages, which is problematic for compliance with licenses ... (Various common license require including the license text, and / or making source code available). Normal alpine images don't appear to install these doc packages: So those images don't even ship with license / copyright info files for packages which is ... not a great start. I'm not sure where a good public reference is for this, Google OSPO has some relevant docs internally about following license requirements when producing container images. So far the best public doc on the topic I've found is
Thanks!
TL:DR;
I wasn't suggesting switching to Alpine, I was just saying that I unit test the iptables-wrappers scripts against Alpine, which doesn't use bash, and so I know the scripts don't require bash. Yeah, I don't know the details of how distroless works, but if we're talking about debian packages then it looks like is 1/10 the size of (212k vs 2285k)
In Alpine (and Debian, and probably most distros other than Fedora), the "iptables" package includes both the legacy and nft binaries, so that's the only dependency.
Made a comment on bandwidth that may be better in this issue rahter than the PR;
Want to give a quick update on this (EDIT: I wrote this was going to be quick update, but ended up not being! sorry!): The image generation is working "fine", see: This process generates a temporary that can be used later with and/or changed in Kubernetes code to trigger the tests (see ) Due to the way iptables-wrapper-installer works, it calls iptables-legacy and iptables-nft to check if they are working fine.


Question: - path: test/conformance/image/Makefile match: BASE_IMAGE_VERSION?= <del> - name: "registry.k8s.io/debian-iptables: dependents" version: bullseye-v1.4.0 </del> <ins> - name: "registry.k8s.io/distroless-iptables: dependents" version: v0.1.1 </ins> refPaths: - path: build/common.sh <del> match: __default_debian_iptables_version= </del> <ins> match: __default_distroless_iptables_version= </ins> - path: test/utils/image/manifest.go <del> match: configs[DebianIptables] = Config{list.BuildImageRegistry, "debian-iptables", "[a-zA-Z]+-v((([0-9]+).([0-9]+).([0-9]+)(?:-([0-9a-zA-Z-]+(?:.[0-9a-zA-Z-]+)*))?)(?:+([0-9a-zA-Z-]+(?:.[0-9a-zA-Z-]+)*))?)"} </del> <ins> match: configs[DistrolessIptables] = Config{list.BuildImageRegistry, "distroless-iptables", "v([0-9]+).([0-9]+).([0-9]+)"} </ins> - name: "registry.k8s.io/go-runner: dependents" version: v2.3.1-go1.18.3-bullseye.0

We need to skip it (via --skip-integrity-check) in arch not amd64, as those are really specific kernel features and when called from arm64, as an example the command ends in an error. We need /bin/sleep to be present in distroless as well, due specifically to this image being used in this test: cloud-build timeout needs to be increased, it takes a long time for arm/s390/etc to build this The image can be optimized a bit more, removing some steps like linking the iptables binaries (as pointed by and removing some binaries like wc (as pointed by ) but right now I didn't wanted to change the image build AND the iptables-wrapper script. My take here would be: let's see now with sig-release if the build process is fine, get this image promoted and used before v1.25 All the tests but verify ran fine (verify due to dependency check, etc etc) - Here are the main tests with the distroless image: kind - kind ipv6 - gce ipvs - performance 100 nodes - fyi. I can bring this to next sig-net meeting, wanted to check: from sig/release perspective, is the PR fine? I will make some more changes here, but wanted even to check if this should be called debian-iptables:distroless or something like from sig/net if we want to move forward migrating the iptables-wrapper to shell script this release or not. I tend of agree with Rob that maybe we are getting too many changes in one release, and we can plan this for v1.26 (the PR is already there, I need to check Dan's comments only) Thanks
/assign
/reopen let's close it once we switch to the new image
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hi, Could you suggest if it's ok to use already? That would be super cool to use the image as a base for NodeLocalDNS (as it uses iptables). I already did manual smoke test in cluster - all is working correctly. Thanks!


Question: - path: test/conformance/image/Makefile match: BASE_IMAGE_VERSION?= <del> - name: "registry.k8s.io/debian-iptables: dependents" version: bullseye-v1.4.0 </del> <ins> - name: "registry.k8s.io/distroless-iptables: dependents" version: v0.1.1 </ins> refPaths: - path: build/common.sh <del> match: __default_debian_iptables_version= </del> <ins> match: __default_distroless_iptables_version= </ins> - path: test/utils/image/manifest.go <del> match: configs[DebianIptables] = Config{list.BuildImageRegistry, "debian-iptables", "[a-zA-Z]+-v((([0-9]+).([0-9]+).([0-9]+)(?:-([0-9a-zA-Z-]+(?:.[0-9a-zA-Z-]+)*))?)(?:+([0-9a-zA-Z-]+(?:.[0-9a-zA-Z-]+)*))?)"} </del> <ins> match: configs[DistrolessIptables] = Config{list.BuildImageRegistry, "distroless-iptables", "v([0-9]+).([0-9]+).([0-9]+)"} </ins> - name: "registry.k8s.io/go-runner: dependents" version: v2.3.1-go1.18.3-bullseye.0

Let's introduce it gradually, I suggest to wait until the image is promoted for kube-proxy
I've lost track of this, what do we need to do to promote it for kube-proxy?
I think Ricardo was on it But if someone is busy and others have time, let's sync next monday on slack and organize the work, so someone can get this to the finish line, there is no need to overload one person if there are other wanting to help
I didn’t saw there was a comment there. I can fix that, and wanted to plan with you folks next steps. I agree with Antonio and don’t want to block this on my availability :) I will check suggestion tomorrow and as we get this merged I’m more than happy to handle to someone else! On Sat, 16 Jul 2022 at 07:21 Antonio Ojea
I've opened a new PR to do the proper promotion. As soon as this is promoted I think we can start testing and migrating to distroless :) From here I think I can handle to someone else as I wont have much time and attention to this specific effort and I know this is important :) I'm still willing to migrate iptables-wrapper to go code (we started it in kubernetes-sigs/iptables-wrapper) so I will probably finish fixing Dan's reviews and we can discuss the distroless v0.2.0 being based entirely in Go :)
we should probably nudge a few of the subprojects to migrate, we just migrated KIND but I'm sure we have at least a few more out there like ip-masq-agent
I'm migrating nodelocaldns. I've got a couple of questions: nodelocaldns uses registry . Should I switch it to ? Is considered sort of a prod image registry? a multi-arch image? Because currently we use is it better to keep up with the new tags for ? So that we could quickly upgrade base image version as well? Thanks for the work!
YES! Make happy! This registry is actually a "proxy" that sig-k8s-infra is maintaining so people get directed to the closest Image registry, based on their cloud provider. As an example, with this users in AWS wont need to go to GCR to pull their image Yes, it is. It follows the same build pattern of debian-iptables. See Well, this is a good question.


Question: - path: test/conformance/image/Makefile match: BASE_IMAGE_VERSION?= <del> - name: "registry.k8s.io/debian-iptables: dependents" version: bullseye-v1.4.0 </del> <ins> - name: "registry.k8s.io/distroless-iptables: dependents" version: v0.1.1 </ins> refPaths: - path: build/common.sh <del> match: __default_debian_iptables_version= </del> <ins> match: __default_distroless_iptables_version= </ins> - path: test/utils/image/manifest.go <del> match: configs[DebianIptables] = Config{list.BuildImageRegistry, "debian-iptables", "[a-zA-Z]+-v((([0-9]+).([0-9]+).([0-9]+)(?:-([0-9a-zA-Z-]+(?:.[0-9a-zA-Z-]+)*))?)(?:+([0-9a-zA-Z-]+(?:.[0-9a-zA-Z-]+)*))?)"} </del> <ins> match: configs[DistrolessIptables] = Config{list.BuildImageRegistry, "distroless-iptables", "v([0-9]+).([0-9]+).([0-9]+)"} </ins> - name: "registry.k8s.io/go-runner: dependents" version: v2.3.1-go1.18.3-bullseye.0

AFAIK we don't have an event / notification system that sends a message (in Slack, as an example) whenever a base image gets promoted. It would be cool to have at least for debian-base, debian-iptables and distroless-iptables, probably it's worth to question sig-release on this :) ( maybe know)
Thanks for the answer! Regarding the - we use it as a compile image for dnsmasq. Will this image still be supported or should I find something other to replace this compile image?
Short answer: I imagine/suppose there is no deprecation plan for this image right now, as it is used as the compilation base for a bunch of other things (and it is base of etcd image) Long answer: In case of distroless-iptables the rationale was: we need to reduce costs (and by reducing costs, reducing image size) we need to use something with a smaller footprint and reduce the amount of packages, and as a result the amount of CVEs in image package. kube-proxy exists in all nodes (unless you use a CNI that implements its features) so it is a good candidate I guess right now debian-base will still be used (I don't remember on the top of my head if other kubernetes components are already running using distroless or compiled based on debian-base, as far as I can see at least etcd image depends on debian-base () so probably any effort on this image should be announced.


Question: hostname := "valid-hostname" advertiseIP := "1.2.3.4" cfg := &kubeadmapi.MasterConfiguration{ <del> API:        kubeadmapi.API{AdvertiseAddress: advertiseIP}, Networking: kubeadmapi.Networking{ServiceSubnet: "10.96.0.0/12", DNSDomain: "cluster.local"}, NodeName:   hostname, </del> <ins> API:               kubeadmapi.API{AdvertiseAddress: advertiseIP}, Networking:        kubeadmapi.Networking{ServiceSubnet: "10.96.0.0/12", DNSDomain: "cluster.local"}, NodeName:          hostname, APIServerCertSANs: []string{"10.1.245.94", "10.1.245.95"}, </ins> } altNames, err := getAltNames(cfg)

What happened: I want to use kubeadm config file to extra altnames to use for the API Server serving cert, as we know the --apiserver-cert-extra-sans flag in the command line. but it not worked... What you expected to happen: generate multiple apiserver serving cert using kubeadm config file. Beacause we define extra etcd cluster, so the command line not support.
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. See the and . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig area/kubeadm
I think you yaml is wrong. The right key is
still don't have extra ips...
this is the right way hehe


Question: // createWorker creates a TimedWorker that will execute `f` not earlier than `fireAt`. func createWorker(ctx context.Context, args *WorkArgs, createdAt time.Time, fireAt time.Time, f func(ctx context.Context, args *WorkArgs) error, clock clock.WithDelayedExecution) *TimedWorker { delay := fireAt.Sub(createdAt) <ins> fWithErrorLogging := func() { err := f(ctx, args) if err != nil { klog.Errorf("NodeLifecycle: timed worker failed with error: %q", err) } } </ins> if delay <= 0 { <del> go f(ctx, args) </del> <ins> go fWithErrorLogging() </ins> return nil } <del> timer := clock.AfterFunc(delay, func() { f(ctx, args) }) </del> <ins> timer := clock.AfterFunc(delay, fWithErrorLogging) </ins> return &TimedWorker{ WorkItem:  args, CreatedAt: createdAt,

For 1.25 clusters with PodDIsruptionConditions enabled pods continue to run if the node is tainted with untolerated NoExecute taint. Also, the error is not logged. The pod running on the tainted node should be terminated (prep) Create a 1.25 cluster with the enabled (tested on kind) a job with a single Pod running long enough for the test (say 3min), the node with a custom taint (so that it is not tolerated by the Pod) with the NoExecute effect (nodeName No response <details</details<detailsreproducible on kind </details<details</details<details</details<details</details<details</details
/sig apps /triage accepted /priority important-soon
/assign


Question: err:  "", test: "update pod spec schedulingGates: legal deletion", }, <ins> { old: core.Pod{ Spec: core.PodSpec{ SchedulingGates: []core.PodSchedulingGate{{Name: "baz"}}, }, }, new: core.Pod{ Spec: core.PodSpec{ NodeSelector: map[string]string{ "foo": "bar", }, SchedulingGates: []core.PodSchedulingGate{{Name: "baz"}}, }, }, err:  "Forbidden: pod updates may not change fields other than `spec.containers[*].image", test: "node selector is immutable when AllowMutableNodeSelector is false", }, { old: core.Pod{ Spec: core.PodSpec{ SchedulingGates: []core.PodSchedulingGate{{Name: "baz"}}, }, }, new: core.Pod{ Spec: core.PodSpec{ NodeSelector: map[string]string{ "foo": "bar", }, SchedulingGates: []core.PodSchedulingGate{{Name: "baz"}}, }, }, opts: PodValidationOptions{ AllowMutableNodeSelectorAndNodeAffinity: true, }, test: "adding node selector is allowed for gated pods", }, { old: core.Pod{ Spec: core.PodSpec{ NodeSelector: map[string]string{ "foo": "bar", }, }, }, new: core.Pod{ Spec: core.PodSpec{ NodeSelector: map[string]string{ "foo":  "bar", "foo2": "bar2", }, }, }, opts: PodValidationOptions{ AllowMutableNodeSelectorAndNodeAffinity: true, }, err:  "Forbidden: pod updates may not change fields other than `spec.containers[*].image", test: "adding node selector is not allowed for non-gated pods", }, { old: core.Pod{ Spec: core.PodSpec{ NodeSelector: map[string]string{ "foo": "bar", }, SchedulingGates: []core.PodSchedulingGate{{Name: "baz"}}, }, }, new: core.Pod{ Spec: core.PodSpec{ SchedulingGates: []core.PodSchedulingGate{{Name: "baz"}}, }, }, opts: PodValidationOptions{ AllowMutableNodeSelectorAndNodeAffinity: true, }, err:  "spec.nodeSelector: Invalid value:", test: "removing node selector is not allowed for gated pods", }, { old: core.Pod{ Spec: core.PodSpec{ NodeSelector: map[string]string{ "foo": "bar", }, }, }, new: core.Pod{}, opts: PodValidationOptions{ AllowMutableNodeSelectorAndNodeAffinity: true, }, err:  "Forbidden: pod updates may not change fields other than `spec.containers[*].image", test: "removing node selector is not allowed for non-gated pods", }, { old: core.Pod{ Spec: core.PodSpec{ NodeSelector: map[string]string{ "foo": "bar", }, SchedulingGates: []core.PodSchedulingGate{{Name: "baz"}}, }, }, new: core.Pod{ Spec: core.PodSpec{ NodeSelector: map[string]string{ "foo":  "bar", "foo2": "bar2", }, }, }, opts: PodValidationOptions{ AllowMutableNodeSelectorAndNodeAffinity: true, }, test: "old pod spec has scheduling gate, new pod spec does not, and node selector is added", }, { old: core.Pod{ Spec: core.PodSpec{ NodeSelector: map[string]string{ "foo": "bar", }, SchedulingGates: []core.PodSchedulingGate{{Name: "baz"}}, }, }, new: core.Pod{ Spec: core.PodSpec{ NodeSelector: map[string]string{ "foo": "new value", }, SchedulingGates: []core.PodSchedulingGate{{Name: "baz"}}, }, }, opts: PodValidationOptions{ AllowMutableNodeSelectorAndNodeAffinity: true, }, err:  "spec.nodeSelector: Invalid value:", test: "modifying value of existing node selector is not allowed", }, { old: core.Pod{ Spec: core.PodSpec{ Affinity: &core.Affinity{ NodeAffinity: &core.NodeAffinity{ RequiredDuringSchedulingIgnoredDuringExecution: &core.NodeSelector{ NodeSelectorTerms: []core.NodeSelectorTerm{ { MatchExpressions: []core.NodeSelectorRequirement{ { Key:      "expr", Operator: core.NodeSelectorOpIn, Values:   []string{"foo"}, }, }, }, }, }, }, }, SchedulingGates: []core.PodSchedulingGate{{Name: "baz"}}, }, }, new: core.Pod{ Spec: core.PodSpec{ Affinity: &core.Affinity{ NodeAffinity: &core.NodeAffinity{ RequiredDuringSchedulingIgnoredDuringExecution: &core.NodeSelector{ // Add 1 MatchExpression and 1 MatchField. NodeSelectorTerms: []core.NodeSelectorTerm{ { MatchExpressions: []core.NodeSelectorRequirement{ { Key:      "expr", Operator: core.NodeSelectorOpIn, Values:   []string{"foo"}, }, { Key:      "expr2", Operator: core.NodeSelectorOpIn, Values:   []string{"foo2"}, }, }, MatchFields: []core.NodeSelectorRequirement{ { Key:      "metadata.name", Operator: core.NodeSelectorOpIn, Values:   []string{"foo"}, }, }, }, }, }, }, }, SchedulingGates: []core.PodSchedulingGate{{Name: "baz"}}, }, }, opts: PodValidationOptions{ AllowMutableNodeSelectorAndNodeAffinity: true, }, test: "addition to nodeAffinity is allowed for gated pods", }, { old: core.Pod{ Spec: core.PodSpec{ Affinity: &core.Affinity{ NodeAffinity: &core.NodeAffinity{ RequiredDuringSchedulingIgnoredDuringExecution: &core.NodeSelector{ NodeSelectorTerms: []core.NodeSelectorTerm{ { MatchExpressions: []core.NodeSelectorRequirement{ { Key:      "expr", Operator: core.NodeSelectorOpIn, Values:   []string{"foo"}, }, }, }, }, }, }, }, SchedulingGates: []core.PodSchedulingGate{{Name: "baz"}}, }, }, new: core.Pod{ Spec: core.PodSpec{ Affinity: &core.Affinity{ NodeAffinity: &core.NodeAffinity{}, }, SchedulingGates: []core.PodSchedulingGate{{Name: "baz"}}, }, }, opts: PodValidationOptions{ AllowMutableNodeSelectorAndNodeAffinity: true, }, err:  "spec.affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution.nodeSelectorTerms: Invalid value:", test: "old RequiredDuringSchedulingIgnoredDuringExecution is non-nil, new RequiredDuringSchedulingIgnoredDuringExecution is nil, pod is gated", }, { old: core.Pod{ Spec: core.PodSpec{ Affinity: &core.Affinity{ NodeAffinity: &core.NodeAffinity{ RequiredDuringSchedulingIgnoredDuringExecution: &core.NodeSelector{ NodeSelectorTerms: []core.NodeSelectorTerm{ { MatchExpressions: []core.NodeSelectorRequirement{ { Key:      "expr", Operator: core.NodeSelectorOpIn, Values:   []string{"foo"}, }, }, }, }, }, }, }, }, }, new: core.Pod{ Spec: core.PodSpec{ Affinity: &core.Affinity{ NodeAffinity: &core.NodeAffinity{ RequiredDuringSchedulingIgnoredDuringExecution: &core.NodeSelector{ // Add 1 MatchExpression and 1 MatchField. NodeSelectorTerms: []core.NodeSelectorTerm{ { MatchExpressions: []core.NodeSelectorRequirement{ { Key:      "expr", Operator: core.NodeSelectorOpIn, Values:   []string{"foo"}, }, { Key:      "expr2", Operator: core.NodeSelectorOpIn, Values:   []string{"foo2"}, }, }, MatchFields: []core.NodeSelectorRequirement{ { Key:      "metadata.name", Operator: core.NodeSelectorOpIn, Values:   []string{"foo"}, }, }, }, }, }, }, }, }, }, opts: PodValidationOptions{ AllowMutableNodeSelectorAndNodeAffinity: true, }, err:  "Forbidden: pod updates may not change fields other than `spec.containers[*].image", test: "addition to nodeAffinity is not allowed for non-gated pods", }, { old: core.Pod{ Spec: core.PodSpec{ Affinity: &core.Affinity{ NodeAffinity: &core.NodeAffinity{ RequiredDuringSchedulingIgnoredDuringExecution: &core.NodeSelector{ NodeSelectorTerms: []core.NodeSelectorTerm{ { MatchExpressions: []core.NodeSelectorRequirement{ { Key:      "expr", Operator: core.NodeSelectorOpIn, Values:   []string{"foo"}, }, }, }, }, }, }, }, SchedulingGates: []core.PodSchedulingGate{{Name: "baz"}}, }, }, new: core.Pod{ Spec: core.PodSpec{ Affinity: &core.Affinity{ NodeAffinity: &core.NodeAffinity{ RequiredDuringSchedulingIgnoredDuringExecution: &core.NodeSelector{ // Add 1 MatchExpression and 1 MatchField. NodeSelectorTerms: []core.NodeSelectorTerm{ { MatchExpressions: []core.NodeSelectorRequirement{ { Key:      "expr", Operator: core.NodeSelectorOpIn, Values:   []string{"foo"}, }, { Key:      "expr2", Operator: core.NodeSelectorOpIn, Values:   []string{"foo2"}, }, }, MatchFields: []core.NodeSelectorRequirement{ { Key:      "metadata.name", Operator: core.NodeSelectorOpIn, Values:   []string{"foo"}, }, }, }, }, }, }, }, }, }, opts: PodValidationOptions{ AllowMutableNodeSelectorAndNodeAffinity: true, }, test: "old pod spec has scheduling gate, new pod spec does not, and node affinity addition occurs", }, { old: core.Pod{ Spec: core.PodSpec{ Affinity: &core.Affinity{ NodeAffinity: &core.NodeAffinity{ RequiredDuringSchedulingIgnoredDuringExecution: &core.NodeSelector{ NodeSelectorTerms: []core.NodeSelectorTerm{ { MatchExpressions: []core.NodeSelectorRequirement{ { Key:      "expr", Operator: core.NodeSelectorOpIn, Values:   []string{"foo"}, }, }, }, }, }, }, }, SchedulingGates: []core.PodSchedulingGate{{Name: "baz"}}, }, }, new: core.Pod{ Spec: core.PodSpec{ Affinity: &core.Affinity{ NodeAffinity: &core.NodeAffinity{ RequiredDuringSchedulingIgnoredDuringExecution: &core.NodeSelector{ NodeSelectorTerms: []core.NodeSelectorTerm{ { MatchFields: []core.NodeSelectorRequirement{ { Key:      "metadata.name", Operator: core.NodeSelectorOpIn, Values:   []string{"foo"}, }, }, }, }, }, }, }, SchedulingGates: []core.PodSchedulingGate{{Name: "baz"}}, }, }, opts: PodValidationOptions{ AllowMutableNodeSelectorAndNodeAffinity: true, }, err:  "spec.affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution.nodeSelectorTerms[0]: Invalid value:", test: "nodeAffinity deletion from MatchExpressions not allowed", }, { old: core.Pod{ Spec: core.PodSpec{ Affinity: &core.Affinity{ NodeAffinity: &core.NodeAffinity{ RequiredDuringSchedulingIgnoredDuringExecution: &core.NodeSelector{ NodeSelectorTerms: []core.NodeSelectorTerm{ { MatchExpressions: []core.NodeSelectorRequirement{ { Key:      "expr", Operator: core.NodeSelectorOpIn, Values:   []string{"foo"}, }, }, MatchFields: []core.NodeSelectorRequirement{ { Key:      "metadata.name", Operator: core.NodeSelectorOpIn, Values:   []string{"foo"}, }, }, }, }, }, }, }, SchedulingGates: []core.PodSchedulingGate{{Name: "baz"}}, }, }, new: core.Pod{ Spec: core.PodSpec{ Affinity: &core.Affinity{ NodeAffinity: &core.NodeAffinity{ RequiredDuringSchedulingIgnoredDuringExecution: &core.NodeSelector{ // Add 1 MatchExpression and 1 MatchField. NodeSelectorTerms: []core.NodeSelectorTerm{ { MatchExpressions: []core.NodeSelectorRequirement{ { Key:      "expr", Operator: core.NodeSelectorOpIn, Values:   []string{"foo"}, }, }, }, }, }, }, }, SchedulingGates: []core.PodSchedulingGate{{Name: "baz"}}, }, }, opts: PodValidationOptions{ AllowMutableNodeSelectorAndNodeAffinity: true, }, err:  "spec.affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution.nodeSelectorTerms[0]: Invalid value:", test: "nodeAffinity deletion from MatchFields not allowed", }, { old: core.Pod{ Spec: core.PodSpec{ Affinity: &core.Affinity{ NodeAffinity: &core.NodeAffinity{ RequiredDuringSchedulingIgnoredDuringExecution: &core.NodeSelector{ NodeSelectorTerms: []core.NodeSelectorTerm{ { MatchExpressions: []core.NodeSelectorRequirement{ { Key:      "expr", Operator: core.NodeSelectorOpIn, Values:   []string{"foo"}, }, }, MatchFields: []core.NodeSelectorRequirement{ { Key:      "metadata.name", Operator: core.NodeSelectorOpIn, Values:   []string{"foo"}, }, }, }, }, }, }, }, SchedulingGates: []core.PodSchedulingGate{{Name: "baz"}}, }, }, new: core.Pod{ Spec: core.PodSpec{ Affinity: &core.Affinity{ NodeAffinity: &core.NodeAffinity{ RequiredDuringSchedulingIgnoredDuringExecution: &core.NodeSelector{ // Add 1 MatchExpression and 1 MatchField. NodeSelectorTerms: []core.NodeSelectorTerm{ { MatchExpressions: []core.NodeSelectorRequirement{ { Key:      "expr", Operator: core.NodeSelectorOpIn, Values:   []string{"bar"}, }, }, MatchFields: []core.NodeSelectorRequirement{ { Key:      "metadata.name", Operator: core.NodeSelectorOpIn, Values:   []string{"foo"}, }, }, }, }, }, }, }, SchedulingGates: []core.PodSchedulingGate{{Name: "baz"}}, }, }, opts: PodValidationOptions{ AllowMutableNodeSelectorAndNodeAffinity: true, }, err:  "spec.affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution.nodeSelectorTerms[0]: Invalid value:", test: "nodeAffinity modification of item in MatchExpressions not allowed", }, { old: core.Pod{ Spec: core.PodSpec{ Affinity: &core.Affinity{ NodeAffinity: &core.NodeAffinity{ RequiredDuringSchedulingIgnoredDuringExecution: &core.NodeSelector{ NodeSelectorTerms: []core.NodeSelectorTerm{ { MatchExpressions: []core.NodeSelectorRequirement{ { Key:      "expr", Operator: core.NodeSelectorOpIn, Values:   []string{"foo"}, }, }, MatchFields: []core.NodeSelectorRequirement{ { Key:      "metadata.name", Operator: core.NodeSelectorOpIn, Values:   []string{"foo"}, }, }, }, }, }, }, }, SchedulingGates: []core.PodSchedulingGate{{Name: "baz"}}, }, }, new: core.Pod{ Spec: core.PodSpec{ Affinity: &core.Affinity{ NodeAffinity: &core.NodeAffinity{ RequiredDuringSchedulingIgnoredDuringExecution: &core.NodeSelector{ NodeSelectorTerms: []core.NodeSelectorTerm{ { MatchExpressions: []core.NodeSelectorRequirement{ { Key:      "expr", Operator: core.NodeSelectorOpIn, Values:   []string{"foo"}, }, }, MatchFields: []core.NodeSelectorRequirement{ { Key:      "metadata.name", Operator: core.NodeSelectorOpIn, Values:   []string{"bar"}, }, }, }, }, }, }, }, SchedulingGates: []core.PodSchedulingGate{{Name: "baz"}}, }, }, opts: PodValidationOptions{ AllowMutableNodeSelectorAndNodeAffinity: true, }, err:  "spec.affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution.nodeSelectorTerms[0]: Invalid value:", test: "nodeAffinity modification of item in MatchFields not allowed", }, { old: core.Pod{ Spec: core.PodSpec{ Affinity: &core.Affinity{ NodeAffinity: &core.NodeAffinity{ RequiredDuringSchedulingIgnoredDuringExecution: &core.NodeSelector{ NodeSelectorTerms: []core.NodeSelectorTerm{ { MatchExpressions: []core.NodeSelectorRequirement{ { Key:      "expr", Operator: core.NodeSelectorOpIn, Values:   []string{"foo"}, }, }, MatchFields: []core.NodeSelectorRequirement{ { Key:      "metadata.name", Operator: core.NodeSelectorOpIn, Values:   []string{"foo"}, }, }, }, }, }, }, }, SchedulingGates: []core.PodSchedulingGate{{Name: "baz"}}, }, }, new: core.Pod{ Spec: core.PodSpec{ Affinity: &core.Affinity{ NodeAffinity: &core.NodeAffinity{ RequiredDuringSchedulingIgnoredDuringExecution: &core.NodeSelector{ NodeSelectorTerms: []core.NodeSelectorTerm{ { MatchExpressions: []core.NodeSelectorRequirement{ { Key:      "expr", Operator: core.NodeSelectorOpIn, Values:   []string{"foo"}, }, }, MatchFields: []core.NodeSelectorRequirement{ { Key:      "metadata.name", Operator: core.NodeSelectorOpIn, Values:   []string{"bar"}, }, }, }, { MatchExpressions: []core.NodeSelectorRequirement{ { Key:      "expr", Operator: core.NodeSelectorOpIn, Values:   []string{"foo2"}, }, }, MatchFields: []core.NodeSelectorRequirement{ { Key:      "metadata.name", Operator: core.NodeSelectorOpIn, Values:   []string{"bar2"}, }, }, }, }, }, }, }, SchedulingGates: []core.PodSchedulingGate{{Name: "baz"}}, }, }, opts: PodValidationOptions{ AllowMutableNodeSelectorAndNodeAffinity: true, }, err:  "spec.affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution.nodeSelectorTerms: Invalid value:", test: "nodeSelectorTerms addition on gated pod should fail", }, { old: core.Pod{ Spec: core.PodSpec{ Affinity: &core.Affinity{ NodeAffinity: &core.NodeAffinity{ PreferredDuringSchedulingIgnoredDuringExecution: []core.PreferredSchedulingTerm{ { Weight: 1.0, Preference: core.NodeSelectorTerm{ MatchExpressions: []core.NodeSelectorRequirement{ { Key:      "expr", Operator: core.NodeSelectorOpIn, Values:   []string{"foo"}, }, }, }, }, }, }, }, SchedulingGates: []core.PodSchedulingGate{{Name: "baz"}}, }, }, new: core.Pod{ Spec: core.PodSpec{ Affinity: &core.Affinity{ NodeAffinity: &core.NodeAffinity{ PreferredDuringSchedulingIgnoredDuringExecution: []core.PreferredSchedulingTerm{ { Weight: 1.0, Preference: core.NodeSelectorTerm{ MatchExpressions: []core.NodeSelectorRequirement{ { Key:      "expr", Operator: core.NodeSelectorOpIn, Values:   []string{"foo2"}, }, }, }, }, }, }, }, SchedulingGates: []core.PodSchedulingGate{{Name: "baz"}}, }, }, opts: PodValidationOptions{ AllowMutableNodeSelectorAndNodeAffinity: true, }, test: "preferredDuringSchedulingIgnoredDuringExecution can modified for gated pods", }, { old: core.Pod{ Spec: core.PodSpec{ Affinity: &core.Affinity{ NodeAffinity: &core.NodeAffinity{ PreferredDuringSchedulingIgnoredDuringExecution: []core.PreferredSchedulingTerm{ { Weight: 1.0, Preference: core.NodeSelectorTerm{ MatchExpressions: []core.NodeSelectorRequirement{ { Key:      "expr", Operator: core.NodeSelectorOpIn, Values:   []string{"foo"}, }, }, }, }, }, }, }, SchedulingGates: []core.PodSchedulingGate{{Name: "baz"}}, }, }, new: core.Pod{ Spec: core.PodSpec{ Affinity: &core.Affinity{ NodeAffinity: &core.NodeAffinity{ PreferredDuringSchedulingIgnoredDuringExecution: []core.PreferredSchedulingTerm{ { Weight: 1.0, Preference: core.NodeSelectorTerm{ MatchExpressions: []core.NodeSelectorRequirement{ { Key:      "expr", Operator: core.NodeSelectorOpIn, Values:   []string{"foo"}, }, { Key:      "expr2", Operator: core.NodeSelectorOpIn, Values:   []string{"foo2"}, }, }, MatchFields: []core.NodeSelectorRequirement{ { Key:      "metadata.name", Operator: core.NodeSelectorOpIn, Values:   []string{"bar"}, }, }, }, }, }, }, }, SchedulingGates: []core.PodSchedulingGate{{Name: "baz"}}, }, }, opts: PodValidationOptions{ AllowMutableNodeSelectorAndNodeAffinity: true, }, test: "preferredDuringSchedulingIgnoredDuringExecution can have additions for gated pods", }, { old: core.Pod{ Spec: core.PodSpec{ Affinity: &core.Affinity{ NodeAffinity: &core.NodeAffinity{ PreferredDuringSchedulingIgnoredDuringExecution: []core.PreferredSchedulingTerm{ { Weight: 1.0, Preference: core.NodeSelectorTerm{ MatchExpressions: []core.NodeSelectorRequirement{ { Key:      "expr", Operator: core.NodeSelectorOpIn, Values:   []string{"foo"}, }, }, }, }, }, }, }, SchedulingGates: []core.PodSchedulingGate{{Name: "baz"}}, }, }, new: core.Pod{ Spec: core.PodSpec{ SchedulingGates: []core.PodSchedulingGate{{Name: "baz"}}, }, }, opts: PodValidationOptions{ AllowMutableNodeSelectorAndNodeAffinity: true, }, test: "preferredDuringSchedulingIgnoredDuringExecution can have removals for gated pods", }, { old: core.Pod{ Spec: core.PodSpec{ Affinity: &core.Affinity{ NodeAffinity: &core.NodeAffinity{ RequiredDuringSchedulingIgnoredDuringExecution: &core.NodeSelector{ NodeSelectorTerms: []core.NodeSelectorTerm{ { MatchExpressions: []core.NodeSelectorRequirement{ { Key:      "expr", Operator: core.NodeSelectorOpIn, Values:   []string{"foo"}, }, }, }, }, }, }, }, SchedulingGates: []core.PodSchedulingGate{{Name: "baz"}}, }, }, new: core.Pod{ Spec: core.PodSpec{ Affinity:        &core.Affinity{}, SchedulingGates: []core.PodSchedulingGate{{Name: "baz"}}, }, }, opts: PodValidationOptions{ AllowMutableNodeSelectorAndNodeAffinity: true, }, err:  "spec.affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution.nodeSelectorTerms: Invalid value:", test: "new node affinity is nil", }, { old: core.Pod{ Spec: core.PodSpec{ Affinity: &core.Affinity{ NodeAffinity: &core.NodeAffinity{ PreferredDuringSchedulingIgnoredDuringExecution: []core.PreferredSchedulingTerm{ { Weight: 1.0, Preference: core.NodeSelectorTerm{ MatchExpressions: []core.NodeSelectorRequirement{ { Key:      "expr", Operator: core.NodeSelectorOpIn, Values:   []string{"foo"}, }, }, }, }, }, }, }, SchedulingGates: []core.PodSchedulingGate{{Name: "baz"}}, }, }, new: core.Pod{ Spec: core.PodSpec{ SchedulingGates: []core.PodSchedulingGate{{Name: "baz"}}, }, }, opts: PodValidationOptions{ AllowMutableNodeSelectorAndNodeAffinity: true, }, test: "preferredDuringSchedulingIgnoredDuringExecution can have removals for gated pods", }, { old: core.Pod{ Spec: core.PodSpec{ Affinity: &core.Affinity{ NodeAffinity: &core.NodeAffinity{ RequiredDuringSchedulingIgnoredDuringExecution: &core.NodeSelector{ NodeSelectorTerms: []core.NodeSelectorTerm{ {}, }, }, }, }, SchedulingGates: []core.PodSchedulingGate{{Name: "baz"}}, }, }, new: core.Pod{ Spec: core.PodSpec{ Affinity: &core.Affinity{ NodeAffinity: &core.NodeAffinity{ RequiredDuringSchedulingIgnoredDuringExecution: &core.NodeSelector{ NodeSelectorTerms: []core.NodeSelectorTerm{ { MatchExpressions: []core.NodeSelectorRequirement{ { Key:      "expr", Operator: core.NodeSelectorOpIn, Values:   []string{"foo"}, }, }, }, }, }, }, }, SchedulingGates: []core.PodSchedulingGate{{Name: "baz"}}, }, }, opts: PodValidationOptions{ AllowMutableNodeSelectorAndNodeAffinity: true, }, err:  "spec.affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution.nodeSelectorTerms[0]: Invalid value:", test: "empty NodeSelectorTerm (selects nothing) cannot become populated (selects something)", }, </ins> } for _, test := range tests { test.new.ObjectMeta.ResourceVersion = "1"

See for context. This issue is creating to track work on the API validation changes. Validation relaxation of certain fields in the pod spec to allow for mutable scheduling directives.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/sig scheduling
Docs PR:


Question: return nil } g.Do(""$.$": {n", name) <ins> if err := g.generateExtensions(m.CommentLines); err != nil { return err } </ins> g.Do("SchemaProps: spec.SchemaProps{n", nil) g.generateDescription(m.CommentLines) jsonTags := getJsonTags(m)


Question: return nil } g.Do(""$.$": {n", name) <ins> if err := g.generateExtensions(m.CommentLines); err != nil { return err } </ins> g.Do("SchemaProps: spec.SchemaProps{n", nil) g.generateDescription(m.CommentLines) jsonTags := getJsonTags(m)


Question: return nil } g.Do(""$.$": {n", name) <ins> if err := g.generateExtensions(m.CommentLines); err != nil { return err } </ins> g.Do("SchemaProps: spec.SchemaProps{n", nil) g.generateDescription(m.CommentLines) jsonTags := getJsonTags(m)


Question: return nil } g.Do(""$.$": {n", name) <ins> if err := g.generateExtensions(m.CommentLines); err != nil { return err } </ins> g.Do("SchemaProps: spec.SchemaProps{n", nil) g.generateDescription(m.CommentLines) jsonTags := getJsonTags(m)


Question: return nil } g.Do(""$.$": {n", name) <ins> if err := g.generateExtensions(m.CommentLines); err != nil { return err } </ins> g.Do("SchemaProps: spec.SchemaProps{n", nil) g.generateDescription(m.CommentLines) jsonTags := getJsonTags(m)


Question: //    (localhost) by name. The possible profile names are detailed at //    http://wiki.apparmor.net/index.php/AppArmor_Core_Policy_Reference string apparmor_profile = 9; <ins> // Seccomp profile for the container, candidate values are: // * runtime/default: the default profile for the container runtime // * unconfined: unconfined profile, ie, no seccomp sandboxing // * localhost/<full-path-to-profile>: the profile installed on the node. //   <full-path-to-profile> is the full path of the profile. string seccomp_profile_path = 10; </ins> } // LinuxContainerConfig contains platform-specific configuration for

CRI supports AppArmor, Seccomp and Sysctls via annotations , we should move them to typed and structured fields instead. (refer ) [x] Apparmor () [x] Sysctls () [x] Seccomp () cc
/cc
/sig node
Seccomp is addressed at .