Question: } framework.ExpectNoError(framework.RunRC(controllerRcConfig)) <del> // Make sure endpoints are propagated. // TODO(piosz): replace sleep with endpoints watch. time.Sleep(10 * time.Second) </del> <ins> // Wait for endpoints to propagate for the controller service. framework.ExpectNoError(framework.WaitForServiceEndpointsNum( c, ns, controllerName, 1, startServiceInterval, startServiceTimeout)) </ins> }

https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite} Previous issues for this test:
Builds: ci-kubernetes-e2e-gce-release-1.5 ci-kubernetes-e2e-gci-gce-release-1.5 ci-kubernetes-e2e-gci-gke-release-1.5 ci-kubernetes-e2e-gci-gke-staging-parallel Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
Builds: ci-kubernetes-e2e-gke-release-1.5 ci-kubernetes-e2e-gke-staging-parallel Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
There is a problem w/contacting controller service. This is already fixed by on master. The fix needs to propagated to the 1.5 branch.
ping looks like this is still occasionally happening, PTAL
Looking at the most recent issues here: it looks like there were a lot of instances where it wasn't this bug -- something else failed (like cluster networking, or connection to the API server), which caused this test to fail. There are a couple where there weren't a bunch of simultaneous failures, and on those I see some failures to collect metrics at all (but hard to tell anything w/o getting the Heapster logs). Finally, I do see a test run (https://k8s-) from 9 days ago where it looks like we are getting metrics from Heapster, just perhaps not the right ones. I'll do a bit more digging. P.S. Is there any way we can get the Heapster logs as part of the saved artifacts from test runs
How does heapster run? Is it possible to get the logs into the artifact directory?Here's the script that dumps the logs:
I've inspected https://k8s- run.


Question: } framework.ExpectNoError(framework.RunRC(controllerRcConfig)) <del> // Make sure endpoints are propagated. // TODO(piosz): replace sleep with endpoints watch. time.Sleep(10 * time.Second) </del> <ins> // Wait for endpoints to propagate for the controller service. framework.ExpectNoError(framework.WaitForServiceEndpointsNum( c, ns, controllerName, 1, startServiceInterval, startServiceTimeout)) </ins> }

HPA correctly scales "from 2 to 1" in the previous test case, but there are no HPA logs at all for subsequent test case for "from 1 to 2". Anyway, this problem appeared to happen only once 14 days ago, many bugs have been fixed since then. Most probably the root of the problem was not related to HPA. In my opinion, this should not be a release blocker.
This Issue hasn't been active in 53 days. It will be closed in 36 days (Jun 12, 2017). cc You can add 'keep-open' label to prevent this from happening, or add a comment to keep it open another 90 days
Closing as obsolete.


Question: name := name tc := tc t.Run(name, func(t *testing.T) { <del> if name == "this ran but done, time drifted back, F" { println("hello") } </del> cj := cronJob() cj.Spec.ConcurrencyPolicy = tc.concurrencyPolicy cj.Spec.Suspend = &tc.suspend

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Cronjob should include the last time when a job was successfully completed. Why is this needed: Currently, a Cronjob's only includes the last time when a job was scheduled. This makes monitoring cronjobs much more complicated, because instead of simply comparing to the current timestamp, the monitoring system now has to hunt for the last job related to the cronjob and persist the state somewhere.
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I'd be interesting in picking this up if the apps team thinks this is a good ticket - it seems like it's fairly beginner friendly.
Hi how is this going on? I could take it over if you don't have the time for this issue. :)
Yes please do! I meant to update that I actually wasn't able to pick this up, but it slipped my mind
/assign
Hi do you have any thoughts on this feature?
/unassign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Any updates?
Reiterating the mentions to trigger a notification: <detailsIn response to : Any updates? Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Trying to understand the usage of lastSuccessfulTime. In a normally working case, lastSuccessfulTime lastScheduledTime =last job completed lastSuccessfulTime < lastScheduledTime =last job completed, but a newer one is running Is that the expectation ?
We wanted to use it in monitoring (e.g. fire an alert if is more than X ago). could also mean that the job has been failing for some time, so I don't know what would you get out of that. I guess you could also use as a proxy for "job has failed at least once"?


Question: name := name tc := tc t.Run(name, func(t *testing.T) { <del> if name == "this ran but done, time drifted back, F" { println("hello") } </del> cj := cronJob() cj.Spec.ConcurrencyPolicy = tc.concurrencyPolicy cj.Spec.Suspend = &tc.suspend

Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten
that's exactly what is being asked here. /close
Closing this issue. <detailsIn response to : /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
"successfully scheduled" != "successfully completed", and most of the users care more about the latter.
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign /triage accepted
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at .


Question: name := name tc := tc t.Run(name, func(t *testing.T) { <del> if name == "this ran but done, time drifted back, F" { println("hello") } </del> cj := cronJob() cj.Spec.ConcurrencyPolicy = tc.concurrencyPolicy cj.Spec.Suspend = &tc.suspend

/lifecycle rotten
This is described in and being actively implemented in /close
Closing this issue. <detailsIn response to : This is described in and being actively implemented in /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
It doesn't look like it was actually implemented, though. Searching for only shows API changes, there don't seem to be any changes to the CronJob controller logic. Or am I missing something?
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign /remove-lifecycle rotten


Question: kubectl get services "${kube_flags[@]}" kubectl create -f examples/guestbook/frontend-service.json "${kube_flags[@]}" kubectl get services "${kube_flags[@]}" <ins> output_service=$(kubectl get service frontend -o json --output-version=v1beta3 "${kube_flags[@]}") kubectl delete service frontend "${kube_flags[@]}" echo "${output_service}" | kubectl create -f - "${kube_flags[@]}" kubectl get services "${kube_flags[@]}" </ins> kubectl delete service frontend "${kube_flags[@]}" kube::log::status "Testing kubectl(${version}:replicationcontrollers)"

If I stuff into a file and use that after -f, it works fine. cc
What's on stdin?
Bug, I'm only accepting json because I naively reused watch partitioner and we don't have test cases. Will fix tonight.
If that's the problem, why did it work when I gave it that file not over stdin?
Different code path. Stdin supports streaming json a la watch (multiple json objects in a stream), so you can do Unfortunately I missed yaml input and didn't use a segmenting yaml decoder (returning a stream of yaml objects).
Ah! Thanks for explanation. I think it would be good for us to ask for e2e runs before merging kubectl changes, since there's not quite enough coverage in test- yet.
Yeah, I'm slacking on that for these changes.
Fix incoming.
Fix provided. More tests to test-cmd... :)


Question: // If there's non-daemon pods left on this node, we will create it in the next sync loop var daemonPodsRunning []*v1.Pod for _, pod := range daemonPods { <ins> if pod.DeletionTimestamp != nil { continue } </ins> if pod.Status.Phase == v1.PodFailed { msg := fmt.Sprintf("Found failed daemon pod %s/%s on node %s, will try to kill it", pod.Namespace, node.Name, pod.Name) glog.V(2).Infof(msg)

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: The DaemonSet controller will schedule a pod on a node when the old pod is still in Terminating state. This breaks the expectation that only one pod will be running at a time per node. What you expected to happen: The DaemonSet controller should not let two pods running at the same time on the same node. How to reproduce it (as minimally and precisely as possible): Create a DaemonSet. Set terminationGracePeriodSeconds to be relatively long. Kill a pod. Watch a new pod start on the same node when the old pod is still terminating. Anything else we need to know?: Seems this behavior was introduced in Environment: Kubernetes version (use ): 1.7.2 Cloud provider or hardware configuration**: GKE OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig apps
/cc
cc
cc: I think the expected behavior is correct, the terminating pod could be performing an important step such as 'removing a lock' or 'cleanup' etc., we should allow that to gracefully die before creating a new one. IMHO we should revoke to old behavior, but why did we need this new behavior in the first place? That is not clear to me from the PR and Issue.
We don't currently guarantee at least or at most one scheduling for DaemonSet. Both behaviors are under consideration via issue and in the future work section of the . The reason for is to "Make sure that controllers are not adopting children that are marked for deletion" (). This is necessary for GC to work properly.
Well the behavior may be technically undefined, but can I request that in the future changes like this be mentioned in the release notes?
Yes, certainly.


Question: // If there's non-daemon pods left on this node, we will create it in the next sync loop var daemonPodsRunning []*v1.Pod for _, pod := range daemonPods { <ins> if pod.DeletionTimestamp != nil { continue } </ins> if pod.Status.Phase == v1.PodFailed { msg := fmt.Sprintf("Found failed daemon pod %s/%s on node %s, will try to kill it", pod.Namespace, node.Name, pod.Name) glog.V(2).Infof(msg)

Also "The DaemonSet controller should not let two pods running at the same time on the same node." is not necessarily the same thing as respecting termination prior to launching a new Pod. The former implies a more strict requirement than the latter. I think we should consider at least attempting to wait for a terminating Pod to be deleted prior to creating another Pod on the Node for same DaemonSet as default behavior.
will something like work then?


Question: <ins> /* Copyright 2014 Google Inc. All rights reserved. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package yaml import ( "bufio" "bytes" "encoding/json" "io" "unicode" "github.com/ghodss/yaml" "github.com/golang/glog" ) // YAMLToJSONDecoder decodes YAML documents from an io.Reader by // separating individual documents. It first converts the YAML // body to JSON, then unmarshals the JSON. type YAMLToJSONDecoder struct { scanner *bufio.Scanner } // NewYAMLToJSONDecoder decodes YAML documents from the provided // stream in chunks by converting each document (as defined by // the YAML spec) into its own chunk, converting it to JSON via // yaml.YAMLToJSON, and then passing it to json.Decoder. func NewYAMLToJSONDecoder(r io.Reader) *YAMLToJSONDecoder { scanner := bufio.NewScanner(r) scanner.Split(splitYAMLDocument) return &YAMLToJSONDecoder{ scanner: scanner, } } // Decode reads a YAML document as JSON from the stream or returns // an error. The decoding rules match json.Unmarshal, not // yaml.Unmarshal. func (d *YAMLToJSONDecoder) Decode(into interface{}) error { if d.scanner.Scan() { data, err := yaml.YAMLToJSON(d.scanner.Bytes()) if err != nil { return err } return json.Unmarshal(data, into) } err := d.scanner.Err() if err == nil { err = io.EOF } return err } const yamlSeparator = "n---" // splitYAMLDocument is a bufio.SplitFunc for splitting YAML streams into individual documents. func splitYAMLDocument(data []byte, atEOF bool) (advance int, token []byte, err error) { if atEOF && len(data) == 0 { return 0, nil, nil } sep := len([]byte(yamlSeparator)) if i := bytes.Index(data, []byte(yamlSeparator)); i >= 0 { // We have a potential document terminator i += sep after := data[i:] if len(after) == 0 { // we can't read any more characters if atEOF { return len(data), data[:len(data)-sep], nil } return 0, nil, nil } if j := bytes.IndexByte(after, 'n'); j >= 0 { return i + j + 1, data[0 : i-sep], nil } return 0, nil, nil } // If we're at EOF, we have a final, non-terminated line. Return it. if atEOF { return len(data), data, nil } // Request more data. return 0, nil, nil } // decoder is a convenience interface for Decode. type decoder interface { Decode(into interface{}) error } // YAMLOrJSONDecoder attempts to decode a stream of JSON documents or // YAML documents by sniffing for a leading { character. type YAMLOrJSONDecoder struct { r          io.Reader bufferSize int decoder decoder } // NewYAMLOrJSONDecoder returns a decoder that will process YAML documents // or JSON documents from the given reader as a stream. bufferSize determines // how far into the stream the decoder will look to figure out whether this // is a JSON stream (has whitespace followed by an open brace). func NewYAMLOrJSONDecoder(r io.Reader, bufferSize int) *YAMLOrJSONDecoder { return &YAMLOrJSONDecoder{ r:          r, bufferSize: bufferSize, } } // Decode unmarshals the next object from the underlying stream into the // provide object, or returns an error. func (d *YAMLOrJSONDecoder) Decode(into interface{}) error { if d.decoder == nil { buffer, isJSON := guessJSONStream(d.r, d.bufferSize) if isJSON { glog.V(4).Infof("decoding stream as JSON") d.decoder = json.NewDecoder(buffer) } else { glog.V(4).Infof("decoding stream as YAML") d.decoder = NewYAMLToJSONDecoder(buffer) } } return d.decoder.Decode(into) } // guessJSONStream scans the provided reader up to size, looking // for an open brace indicating this is JSON. It will return the // bufio.Reader it creates for the consumer. func guessJSONStream(r io.Reader, size int) (io.Reader, bool) { buffer := bufio.NewReaderSize(r, size) b, _ := buffer.Peek(size) return buffer, hasPrefix(b, []byte("{")) } // Return true if the first non-whitespace bytes in buf is // prefix func hasPrefix(buf []byte, prefix []byte) bool { buf = bytes.TrimLeftFunc(buf, unicode.IsSpace) return bytes.HasPrefix(buf, prefix) } </ins>


Question: Verb:     "get", Resource: schema.GroupVersionResource{Resource: "resource"}, } <del> c.Invokes(action, nil) </del> <ins> if _, err := c.Invokes(action, nil); err != nil { return nil, err } </ins> for _, resourceList := range c.Resources { if resourceList.GroupVersion == groupVersion { return resourceList, nil

I'm writing a UT for the following function: The code branch where the server returns an error other than could not be tested using 's . This is because does not handle the error returned by and therefore could not return errors other than : Ref: Add a to so simulate errors other than . Run the following UT (should fail):
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig api-machinery
Same issue for some other methods of : -
I'll submit a PR shortly to address this issue. /assign


Question: // Allow subpath environment variable substitution // Only applicable if the VolumeSubpath feature is also enabled VolumeSubpathEnvExpansion utilfeature.Feature = "VolumeSubpathEnvExpansion" <ins> // owner: @vikaschoudhary16 // alpha: v1.11 // // // Enable probe based plugin watcher utility for discovering Kubelet plugins KubeletPluginsWatcher utilfeature.Feature = "KubeletPluginsWatcher" </ins> ) func init() {


Question: fakePlugin.UnmountDeviceHook = func(mountPath string) error { // Act: // 3. While a volume is being unmounted, add it back to the desired state of world <del> t.Logf("UnmountDevice called") generatedVolumeName, err = dsw.AddPodToVolume( </del> <ins> klog.InfoS("UnmountDevice called") var generatedVolumeNameCopy v1.UniqueVolumeName generatedVolumeNameCopy, err = dsw.AddPodToVolume( </ins> podName, pod, volumeSpecCopy, volumeSpec.Name(), "" /* volumeGidValue */) <del> dsw.MarkVolumesReportedInUse([]v1.UniqueVolumeName{generatedVolumeName}) </del> <ins> dsw.MarkVolumesReportedInUse([]v1.UniqueVolumeName{generatedVolumeNameCopy}) </ins> return nil }

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --N/A Triage instead: Data race. links to appreciated links to specific failures in spyglass appreciated specific failure <!-- Please see the deflaking doc () for more guidance! --
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage
previously:
The data race is for this time . fixed only by using a copy.
/assign
GitHub didn't allow me to assign the following users: Haleygo. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Seems noone work here, try to reproduce and fix. /assign
kubernetes# ./stress -p 20 - "TestRunPositive_VolumeMountControllerAttachEnabledRace" 15m25s: runs so far, 0 failures :( I cannot reproduce it, does anyone have better suggestions?
? I have no better idea.
-p 200 ,-p 300, I reproduced this fault (and another DATA RACE). I continued to test after adding deepCopy and found that the stress did not seem to recognize my modification, because the DATA RACE error line was the same as the original, I was a little bit confused....
confused: Use to compile stress, Add Info log to prove that the code has been modified: Call and fail the increased log is not printed (one timeout). As a result, I cannot verify whether I have fixed this problem!
I forgot to update , now the test continues. :-)


Question: UpdateNodeLabel       // 1000 UpdateNodeTaint       // 10000 UpdateNodeCondition   // 100000 <ins> UpdateNodeAnnotation  // 1000000 </ins> <del> All ActionType = 1<<iota - 1 // 111111 </del> <ins> All ActionType = 1<<iota - 1 // 1111111 </ins> // Use the general Update type if you don't either know or care the specific sub-Update type to use. <del> Update = UpdateNodeAllocatable | UpdateNodeLabel | UpdateNodeTaint | UpdateNodeCondition </del> <ins> Update = UpdateNodeAllocatable | UpdateNodeLabel | UpdateNodeTaint | UpdateNodeCondition | UpdateNodeAnnotation </ins> ) // GVK is short for group/version/kind, which can uniquely represent a particular API resource.

Due to limited character space, I've stored additional information in annotations on nodes. My custom scheduling plugins read these annotations to schedule Pods. However, when a Pod fails to be scheduled on a node, I want it to be removed from the Unschedulable queue. This can be achieved by modifying the annotations of the node, allowing the Pod to be rescheduled. Unfortunately, the eventHandler of the scheduler only supports five types of node events, and it does not include the nodeAnnotationsChanged event. Consequently, I am currently unable to trigger rescheduling by changing the annotations of nodes. /sig scheduling Unschedulable pods will be rescheduled after annotations of nodes changed. None No response <details</details<details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc
/assign
It's more like a new feature, not bug because it doesn't lead to any bad behavior on the upstream. /remove-kind bug /kind feature
My concern is node annotation is arbitrary, change a node annotation doesn't imply a pod is schedulable. But for other events, like node labels, node taints, they're the constraints we consider in scheduling. They're different, I'm afraid this will lead to invalid scheduling attempts. Who is responsible for change the annotations in you scenario?
This enhancement has no impact on the in-tree plugins / upstream scheduler, right? Only (custom) plugins, that want to reschedule Pods via the node annotation change, will register event to EventsToRegister.
I have a controller capable of modifying node annotations to indicate node's current availability. And my custom plugins need to be aware of these changes.
I guess we can still be impacted when scheduling pod failed because of no fit node rather than failed by the plugin.


Question: UpdateNodeLabel       // 1000 UpdateNodeTaint       // 10000 UpdateNodeCondition   // 100000 <ins> UpdateNodeAnnotation  // 1000000 </ins> <del> All ActionType = 1<<iota - 1 // 111111 </del> <ins> All ActionType = 1<<iota - 1 // 1111111 </ins> // Use the general Update type if you don't either know or care the specific sub-Update type to use. <del> Update = UpdateNodeAllocatable | UpdateNodeLabel | UpdateNodeTaint | UpdateNodeCondition </del> <ins> Update = UpdateNodeAllocatable | UpdateNodeLabel | UpdateNodeTaint | UpdateNodeCondition | UpdateNodeAnnotation </ins> ) // GVK is short for group/version/kind, which can uniquely represent a particular API resource.

see Can we predict whether the pod is schedulable before running into scheduling cycle via the node annotations, then we can leverage the preEnqueue extension point since you already have a external controller.
Thanks for the advice. But I'm afraid it would not be the best solution for us, because doing that outside the scheduling cycle would cause the custom plugins to malfunction in my scenario.
The concerns raised by make sense to me. In my understanding, most in-tree plugins will classify the event as an event.
No. When no fit Pod, UnschedulablePlugins will have plugin(s) rejected the Pod. (e.g., all nodes don't have enough capacity, UnschedulablePlugins has noderesourcefit. all nodes don't satisfy node affinity, UnschedulablePlugins has nodeaffinity) is a rare case like plugins other than PreFilter/Filter returns non-success. I don't think we should care about it.
But, this is a good point, I think we should refactor all plugins that register event with to have a specific Node update event. For example, . It should have with instead of . event with , that should be the last-resort when we don't know which kind of Node update will make Pods schedulable. (I guess no such case at least in the in-tree plugins.)
1) One possibility is Reserve Plugin, I the capability here 2) Another possibility is PreFilter, but not because failed by the plugin, but no fit node exists Have no idea whether we have other places, I guess no. But yes, this only impacts unschedulable pods has no failed plugins.
~I think this is for intended, like ~ ~1. 1) we have a plugin failed by nodeAffinity~ ~2. 2) we add a new node~ ~3. 3) we receive the node add event, but because node not ready, we failed by nodeTaint plugin as ~ 4) node become ready, we get the node update event, but dismatch with nodeAffinity registered event (if we registered with UpdateNodeLabel event)~ ~5. 5) pod will not be moved to the active or backoff queue~ updated below
weigh in for advices as whether this is a good idea to take nodeAnnotations as a new event.


Question: UpdateNodeLabel       // 1000 UpdateNodeTaint       // 10000 UpdateNodeCondition   // 100000 <ins> UpdateNodeAnnotation  // 1000000 </ins> <del> All ActionType = 1<<iota - 1 // 111111 </del> <ins> All ActionType = 1<<iota - 1 // 1111111 </ins> // Use the general Update type if you don't either know or care the specific sub-Update type to use. <del> Update = UpdateNodeAllocatable | UpdateNodeLabel | UpdateNodeTaint | UpdateNodeCondition </del> <ins> Update = UpdateNodeAllocatable | UpdateNodeLabel | UpdateNodeTaint | UpdateNodeCondition | UpdateNodeAnnotation </ins> ) // GVK is short for group/version/kind, which can uniquely represent a particular API resource.

As I know, some plugins will read the embedded prometheus metrics in node annotations for scoring, but not for filtering. At least we should mitigate the impacts.
iv. node become ready, we get the node update event, but dismatch with nodeAffinity registered event (if we registered with UpdateNodeLabel event) (3) is not exact. When we receive the node add event, the scheduler retry to schedule the Pod because the nodeaffinity has Node add event. And it's rejected by the nodetaint plugin. (not node affinity) So, in (4), we get the node update event, it matches with node taint registered event, and the Pod is moved to active/backoffQ (node taint registers node update event. Yes I think we can change node taint's node update event to more specific one(s) though)
As I replied in your PR, we acknowledged that in , and yes let's have your patch. Well, that's a good point. So, when all nodes are filtered out by PreFilterResult, then no failedplugin can be returned. That's a problem. I think we should return the plugin name that returns PreFilterResult with not all Node names in this case (= the plugins that contributes to reducing the candidate Nodes via PreFilterResult). What do you think?
Sorry, the previous one is not right, a detailed one: a. we have a pod failed by nodeAffinity. pod will be placed into the unschedulablePods b. node autoscaler triggered, a new node , emitted a node add event c. called -d. node not ready right now and holds a notReady taint, then failed, pod still stays in the unschedulablePods e. node gets ready, emitted a node update event, but dismatch with the registered event(if we register with in nodeAffinity plugin) f. pod will stay in unschedulabelPods I guess this is the right flow. Anyway, glad to see it be more effect in the future.
If we have no performance problems, I think it's ok to take nodeAnnotations into consideration.
Wouldn't this be solved with proposal on ClusterEventWithHint? If so, I prefer we leave the existing code as-is.
whatever we do here, we wouldn't backport, so I think is the right move forward.
We put many stuff here.


Question: UpdateNodeLabel       // 1000 UpdateNodeTaint       // 10000 UpdateNodeCondition   // 100000 <ins> UpdateNodeAnnotation  // 1000000 </ins> <del> All ActionType = 1<<iota - 1 // 111111 </del> <ins> All ActionType = 1<<iota - 1 // 1111111 </ins> // Use the general Update type if you don't either know or care the specific sub-Update type to use. <del> Update = UpdateNodeAllocatable | UpdateNodeLabel | UpdateNodeTaint | UpdateNodeCondition </del> <ins> Update = UpdateNodeAllocatable | UpdateNodeLabel | UpdateNodeTaint | UpdateNodeCondition | UpdateNodeAnnotation </ins> ) // GVK is short for group/version/kind, which can uniquely represent a particular API resource.

Let me summarize each: Born from from is right. But, that's the problem with preCheck. (Some preCheck depends on some in-tree plugins' logic and it affects badly to other plugins enqueueing ) The preCheck is removed after is completed. After preCheck is removed, will be the correct behavior. So, we can do the optimization that changes node to a specific event. will be fixed in: It's kind of rare case, but I feel we need to have some improvement like Edit: create another issue for this: I'd say (2) and (3) are kind of rare cases. So, the blocker here is (1). (And, (1) is blocked by the removal of preCheck, which is blocked by ) As my conclusion, it's safer to add after (1) is done so that we can make sure the performance won't be affected badly. what do you think?
Instead, can we just keep an event for "node updated", then individual plugins can decide whether a annotations change is worth requeuing or not?
Yes, (1) can be replaced with that idea. But, I'd say changing to a specific is slightly better because the scheduling queue could filter out some Node update events without executing the hint functions.
If rescheduling can be triggered with precise events, that would be great. I haven't revisit since last time, I'll take a look again.
Since we are missing an event for , can we add it? But we also have to modify existing events like so: Because change of such field is also an Update. And in future releases we can get rid of some fine-grained events.
Since we will add hintFuncs for all in-tree plugins, so I think it's ok to add nodeAnnotationsChanged event now, we can filter out unrelated events in hitFuncs. Any other concerns? cc Prefer to keep them as people can register fine-gained events instead a general event.
If by "now" you mean v1.30, then I'm ok with that.


Question: gvk := schema.GroupVersionKind{ Group:   gv.Group, Version: gv.Version, <del> Kind:    gvks[0].Kind, </del> <ins> Kind:    gvk.Kind, </ins> } exampleOfGVK, err := scheme.New(gvk)

I have an extension apiserver, and I was trying to upgrading the dependency . In our code, we have a serval resources has its , and we expect the storage version is their . for example, we have a resource in both and group, and they are wired up by method, and expected encoded as to storage. However, this seems broken after upgrade to v1.31, so I did some invertgation by myself, and I find the storage version is changed by here: In old behavior honor , but now don't. storage version semantic of would be honored. there is not easy approach, need to create an extension api server and find out. No response <detailsClient Version: v1.30.1 Kustomize Version: v5.0.4-0.- Server Version: v1.31.0-beta.0.25+-dirty </details<detailsNONE </details<detailsKind Cluster run on Darwin Darwin ReficuldeMacBook- 23.0.0 Darwin Kernel Version 23.0.0: Fri Sep 15 14:41:34 PDT 2023; root:xnu-.1.13~1/RELEASEARM64_T8103 arm64 </details<detailskind </details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig api-machinery
PTAL cc


Question: if exists && info.isActive() && info.portalPort == service.Spec.Port && info.portalIP.Equal(serviceIP) { continue } <del> if exists && (info.portalPort != service.Spec.Port || !info.portalIP.Equal(serviceIP) || service.Spec.CreateExternalLoadBalancer != (len(info.publicIP) > 0)) { </del> <ins> if exists && (info.portalPort != service.Spec.Port || !info.portalIP.Equal(serviceIP) || !ipsEqual(service.Spec.PublicIPs, info.publicIP)) { </ins> glog.V(4).Infof("Something changed for service %q: stopping it", service.Name) err := proxier.closePortal(service.Name, info) if err != nil {

I'm not sure if it's the intended behaviour, but when I specify publicIPs for a service, external portals are only opened if createExternalLoadBalancer is set, too (this is checked in the OnUpdate function in ). After reading the docs, I would have expected that IPTables rules for external IPs are independent of the createExternalLoadBalancer flag. Also, we're running kubernetes locally, so setting createExternalLoadBalancer doesn't work since there's no cloud provider specified.
I was about to say it should work, but then I went and re-read the code and found at least one bug. Let me dig a bit more
You're right - the IPs actually propagate to the proxy, but it ignores them if createExternal... is not set. I think it is kosher to just do it. I'm prepping a PR
PR is up - if you feel bold you could test it in your environment...
Thanks, it's working fine with your patch.


Question: kube::test::get_object_assert 'pods --namespace=other' "{{range.items}}{{$id_field}}:{{end}}" 'valid-pod:' # Post-condition: verify shorthand `-n other` has the same results as `--namespace=other` kube::test::get_object_assert 'pods -n other' "{{range.items}}{{$id_field}}:{{end}}" 'valid-pod:' <ins> # Post-condition: a resource cannot be retrieved by name across all namespaces output_message=$(! kubectl get "${kube_flags[@]}" pod valid-pod --all-namespaces 2>&1) kube::test::if_has_string "${output_message}" "a resource cannot be retrieved by name across all namespaces" </ins> ### Delete POD valid-pod in specific namespace # Pre-condition: valid-pod POD exists

The desciption of --all-namespaces is listing the requested object(s) across all namespaces. Namespace in current context is ignored even if specified with --namespace." But when I used the following command: error: namespace may not be empty when retrieving a resource by name no namespace set on resource pods "counter" The error messages are unclear, and the second message isn't labeled as an error even.
And why couldn't use --all-namespaces flag?
You're asking for a specific pod name (counter) in all namespaces, which is a nonsensical query. The pod name is not a grep-like filter, it's a specific name. Yeah, the errors could be better. Describe could work, but potentially generates a LOT of information and API calls.
But I found the following API is supported now. Get The result is: has the more information. Do we need to support the following kubectl command(using the above API) or just return clear error messages? get pods counter --all-namespaces
I'd be ok with this being implemented via the field selector.
The issue is still here :). What's our expected result? I'd like to contribute this.
, after check the code, the URL was built by ; do you mean we should update it; or avoid to use , instead, build url () directly.
After checking the code, it's better to add fieldSelector to , is created for fieldSelector. Will submit a PR based on .
getting an object with a particular name across all namespaces seems like something we shouldn't encourage... it breaks the assumption of namespace independence.
, so we'll only update message for this issue? for example:
Disagree. Need easy way to troubleshouting the problem "why pods is stuck in Pending status". It's tiring for every pod to find who created it and in which namespace. In the logs there is only the name of the pod.
Now, with field selectors we can do To further my understanding, I'd ask:- doesn't itself break independence? One can search by label across all namespaces, it feels funny for by name to be special.


Question: func validateSecretKeySelector(s *api.SecretKeySelector, fldPath *field.Path) field.ErrorList { allErrs := field.ErrorList{} <del> if len(s.Name) == 0 { allErrs = append(allErrs, field.Required(fldPath.Child("name"), "")) </del> <ins> nameFn := ValidateNameFunc(ValidateSecretName) for _, msg := range nameFn(s.Name, false) { allErrs = append(allErrs, field.Invalid(fldPath.Child("name"), s.Name, msg)) </ins> } if len(s.Key) == 0 { allErrs = append(allErrs, field.Required(fldPath.Child("key"), ""))

Scenario: Expected results: pod would not be created as [0] is invalid Actual results: pod is created. Turns out we are not doing any validation of envFrom in container spec today.
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list .
I don't know that we can tighten without failing pods.
The validation code exists, but there was no wiring done from validateContainers.
Pods fail on kubelet already. So those pods were not doing much already. We had a BZ on this.
Fail how - fail to even start, or fail to get the input?