Question: import ( "fmt" <ins> "net/http" </ins> "reflect" "strconv" "strings"

When attempting to fetch a resource version that is too old, the behavior of watch (for both HTTP and web sockets) changed when watch cache was enabled. The watch cache returns a 410 - without the watch cache, we return 200 and then write an "error" entry to the watch stream. This broke a client that dependent on the watch behavior, and moreover is inconsistent with other errors we set in the registry. Also, for watch over web sockets, most javascript clients will be unable to get at a 410 error from the connection, because browsers don't get access to the data. We should restore the watch cache behavior to be consistent with the previous behavior as this is a regression in API semantics. Resource without watch cache, watching too old: Resource with watch cache, watching too old:
I think this is candidate for back port to 1.2 to restore the previous behavior.
since this broke your javascript client, FYI because of watch cache
+1 for backport
hmm - we enabled watch cache in 1.1 - so this should also be broken in 1.1
It's possible that it was always nondeterministic and just depends on how fast the call to the storage layer executes. I think all of our in-tree clients will handle it either way.
I'm fairly certain this never used to behave this way, because OpenShift hasn't enabled the watch catch and we have never had a report of this until we enabled the watch cache for Kube resources just recently. , Daniel Smith wrote:
I see what you mean about ordering - etcd remote call would almost never be fast enough to beat the watch event. In practice, I suspect no one ever saw this without the watch error. Either way, for web sockets if we don't fix this clients can't get this error (the browser eats the 410 during the websocket negotiation and client code can't get access). So it's at least broken for web sockets. , Clayton Coleman wrote:
Well, it should be easy to just not wait at all, maybe? I'm OK with changing this behavior.
Although, are web clients XSS hardened? I'm not sure that we're optimizing for them currently.
If you mean CSRF, it isn't an issue with bearer token auth.


Question: import ( "fmt" <ins> "net/http" </ins> "reflect" "strconv" "strings"

Basic auth should already have a big warning around it saying "not safe with browsers"
Maybe I'm missing something, but it seems deterministic to me. This is the uncached watch impl: once the resourceVersion parses, there is no path that returns a direct error, so all errors would have to be returned as an event via the ResultChan(). I don't see us reading from that until after we've already written a 200 and flushed:
Having read the docs, I was very surprised that "type": "ERROR" notices are sent as 200 (even over plain http), with the semantic code e.g. 410 only represented under inside. I see there was good motivation here to stick to 200, but it should be called out in the documentation. doesn't show at all how watch returns errors, and it even says "clients must handle the case by recognizing the status code 410 Gone" which really makes one expect an actual HTTP 410.
I agree this should be called out in the docs. Can you file a bug there and/or send a PR? I don't think this ~4 year old issue is the best place to track that. (Good detective work finding this, though!)


Question: SecurityContext:               &v1.PodSecurityContext{}, TerminationGracePeriodSeconds: &period, SchedulerName:                 api.DefaultSchedulerName, <del> EnableServiceLinks:            &enableServiceLinks, </del> }, ObjectMeta: metav1.ObjectMeta{ Labels: defaultLabels,

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug The change in a new field () to the Pod spec that is set to true by default. This is causing problems for DaemonSets in some situations. DaemonSets are using ControllerRevisions to keep track of previous configurations and each revision will contain among other things the Pod spec for the DaemonSet. If a DaemonSet is created with the enableServiceLinks property is set to true, the ControllerRevision will also contain this field. If the cluster is rolled back to a previous version of kubernetes, the DeamonSet struct does not have this field and so the ControllerRevision and the DaemonSet will no longer match. This leads to a new ControllerRevision being created and a rolling update of all pods in the DaemonSet. Would it be a better solution to rename the field to and enable service links whenever the field is not set? This would fix the problem with the DaemonSets unless the field is explicitly set before the rollback. This is causing the master upgrade tests to fail as reported in /sig apps /cc /cc /cc
I guess the change in podSpec and the defaulting will break all the workload resources as they are based on comparing the template's podSpec. ~Is the new value defaulted to true also for the DaemonSet's podSpec template? If not, controller might go hot loop like with mutating admission controllers.~ I don't like us making such changes to v1 podSpec. The other way around seems more sensible. /cc
Why is this a bad thing? We will continue to add fields to podSpec - the issue isn't the defaulting (this would happen regardless of which way it defaulted), it's that the controller can't really be version aware. The new value would default to true in the daemonset as well.
I'm not sure that a rollback requiring daemonsets to recreate is a bug. I think that's potentially "the best possible thing that we can do in that case".
/cc SIG Release blocking dashboard with upgrade test failures - https://k8s-
So the problem isn't the fact that a field has been . It the way in which the field was .


Question: SecurityContext:               &v1.PodSecurityContext{}, TerminationGracePeriodSeconds: &period, SchedulerName:                 api.DefaultSchedulerName, <del> EnableServiceLinks:            &enableServiceLinks, </del> }, ObjectMeta: metav1.ObjectMeta{ Labels: defaultLabels,

enableServiceLinks is bool pointer that defaults to true. To be backward compatible it should be something like where the new default behavior is to assume that service links are enabled, and the field should be set on an opt out basis to disable the service links. compatiblity As it is if you create a new DaemonSet and then roll back cluster to a prior version, or if you have a failover in a HA setup, you can trigger a simultaneous rolling restart of all DaemonSets you created even though you didn't explicitly configure (and may not even be aware of) the new feature. If we respect backward compatibility and make the feature explicitly opt-out (that is the semantics we are going for here in either ca So the problem isn't the fact that a field has been . It the way in which the field was . enableServiceLinks is bool pointer that defaults to true. To be backward compatible it should be something like where the new default behavior is to assume that service links are enabled, and the field should be set on an opt out basis to disable the service links. compatiblity As it is if you create a new DaemonSet and then roll back cluster to a prior version, or if you have a failover in a HA setup, you can trigger a simultaneous rolling restart of all DaemonSets you created even though you didn't explicitly configure (and may not even be aware of) the new feature. If we respect backward compatibility and make the feature explicitly opt-out (that is the semantics we are going for here in either case), you won't trigger the rolling restart on rollback or controller manager failoverse), you won't trigger the rolling restart on rollback or controller manager failover unless you explicitly opt out of the behavior. I don't see why the former behavior is more desirable than the latter.
For background/context, the choice between enable/disable was initially discussed here:
Are you saying that the issue is that default value is not the same as the Go "zero value" ? The concept of zero value is interesting but it forces APIs to bend into unnatural shapes to respect it. Since forever Kubernetes has explicitly said that we have version-specific default values which ARE NOT governed by zero values, but instead are governed by the API version. It is safe to assume that if a field is documented as having a default in a version, you can rely on that field being populated with that default, with a few exceptions.


Question: SecurityContext:               &v1.PodSecurityContext{}, TerminationGracePeriodSeconds: &period, SchedulerName:                 api.DefaultSchedulerName, <del> EnableServiceLinks:            &enableServiceLinks, </del> }, ObjectMeta: metav1.ObjectMeta{ Labels: defaultLabels,

We generally ask for boolean fields to be "in the affirmative direction", as in this case. Now, we could revisit this. It would be a significant change in direction, but it's not impossible. But I don't think that solves your problem in general -- if this field were optional string, with a default value of "Hello there", you would be in the same boat right?
So the underlying issue is that we are defaulting into the PodSpec which gets adopted into all of the PodTemplateSpecs of every object. That sort of defaults the users declared intent. Why not just default the value into the Pod itself . Is the intention really to default this in all Templates or to make it optional in the Template and default it in the Pod itself?
/milestone v1.13
I can't speak to the ideology of whether to default the field in all PodSpecs or just the Pod, but your suggestion makes sense to me. I'm not sure of the past reasoning for adding other defaulted fields, most recently SchedulerName in Jan. 2017 (), to all PodSpecs instead of just to the Pod object. Did we not have these same tests in place then? This would effectively be saying we should never add new defaulted fields to PodSpec. If this is the plan, I can create a PR for the fix and I'll add a big warning comment to SetDefaults_PodSpec.
As an example of where we don't default the template we have a CronJob that embeds JobTemplate, we never default that template, only an actual Job gets defaulting. I agree with that not defaulting any templates does make sense.
Agree to only default this in pods. Agree to adding the comment. Most important defaulting happens in pods to keep higher level pod specs simpler. We may need to raise the general pattern in a discussion with api review in general, since this is a change from previous strategy, but the previous strategy was not explicitly chosen (to my recollection at least)
Hi all do we have an agreed path forward for fixing this issue in 1.13? if so can we have an ETA for the fix to land? the release team would like for the tests to get passing in the blocking suites in an effort to unmask any other failures that might show up once these are fixed.


Question: SecurityContext:               &v1.PodSecurityContext{}, TerminationGracePeriodSeconds: &period, SchedulerName:                 api.DefaultSchedulerName, <del> EnableServiceLinks:            &enableServiceLinks, </del> }, ObjectMeta: metav1.ObjectMeta{ Labels: defaultLabels,

/assign
We have a PR with a fix that will hopefully merge today or tomorrow. I will keep an eye on the dashboard to make sure it fixes the failing daemonset tests.
Thanks Can we link the PR here plz. /cc
Reopening, because of the auto-close. Tests are not passing yet. /reopen
Reopening this issue. <detailsIn response to : Reopening, because of the auto-close. Tests are not passing yet. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
PR that should (hopefully) close this issue is here:
DaemonSet tests passed on run 1842, seems to have been fixed: https://k8s- if it doesn't pop up again - /close
Closing this issue. <detailsIn response to : DaemonSet tests passed on run 1842, seems to have been fixed: https://k8s- Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: tc.runTest(t) } <ins> func TestReplicaCalcScaleUpCMObjectIgnoresUnreadyPods(t *testing.T) { tc := replicaCalcTestCase{ currentReplicas:  3, expectedReplicas: 5, // If we did not ignore unready pods, we'd expect 15 replicas. podReadiness:     []v1.ConditionStatus{v1.ConditionFalse, v1.ConditionTrue, v1.ConditionFalse}, metric: &metricInfo{ name:                "qps", levels:              []int64{50000}, targetUtilization:   10000, expectedUtilization: 50000, singleObject: &autoscalingv2.CrossVersionObjectReference{ Kind:       "Deployment", APIVersion: "extensions/v1beta1", Name:       "some-deployment", }, }, } tc.runTest(t) } </ins> func TestReplicaCalcScaleUpCMExternal(t *testing.T) { tc := replicaCalcTestCase{ currentReplicas:  1,

Is this a BUG REPORT or FEATURE REQUEST?: /kind feature What happened: We are starting to use the HPA (horizontal pod autoscaling) for one of our components. What is special about this particular component is that the warm up period is very long (sometimes even more than 30 minutes). During that time, the pod is in Running state but its containers are in Not Ready state. This pod is of course not able to receive any traffic so the metric we use in the HPA is ignoring those pods that are not Ready. The equation used by HPA to calculate the new number of replicas is as follows Our problem is that the equation used by HPA to calculate the new number of replicas is including in the "current replica count" the pods that are "Not Ready" so the value for "new replica count" is not correct. Since this warm up is so long we end up by having a lot of more instances of our services of what it should be necessary. They eventually get deleted once gradually the metrics start going down but that will take a lot of time and a big waste of resources without mentioning that with a changing traffic this HPA will never stabilize What you expected to happen: I would like to have a flag somewhere where I tell HPA to don't include in the "current replica count" the pods that are in "Not Ready" state How to reproduce it (as minimally and precisely as possible): Have a component that takes a lot of time to start and being able to receive traffic. Then define a metric that will ignore the pods that are Not Ready. Finally create a metric that will scale that component based on the mentioned metric Anything else we need to know?: Environment: Kubernetes version 1.7 Cloud provider or hardware configuration: Azure
/sig autoscaling
Hi - thanks for reaching out about this! It looks like this section of the , handles when pods aren't ready. I think you this behavior in , so I'm interested in your thoughts on whats best here. If we do end up adding a flag for this, I'd be happy to write the code :) Also, purely for my own curiosity, I'm curious what type of pod is taking 30 minutes to become ready. If your traffic pattern varies at a rate faster than 30 minutes, it may be difficult for auto-scaling to be helpful.
can you also share a bit more about your HPA?


Question: tc.runTest(t) } <ins> func TestReplicaCalcScaleUpCMObjectIgnoresUnreadyPods(t *testing.T) { tc := replicaCalcTestCase{ currentReplicas:  3, expectedReplicas: 5, // If we did not ignore unready pods, we'd expect 15 replicas. podReadiness:     []v1.ConditionStatus{v1.ConditionFalse, v1.ConditionTrue, v1.ConditionFalse}, metric: &metricInfo{ name:                "qps", levels:              []int64{50000}, targetUtilization:   10000, expectedUtilization: 50000, singleObject: &autoscalingv2.CrossVersionObjectReference{ Kind:       "Deployment", APIVersion: "extensions/v1beta1", Name:       "some-deployment", }, }, } tc.runTest(t) } </ins> func TestReplicaCalcScaleUpCMExternal(t *testing.T) { tc := replicaCalcTestCase{ currentReplicas:  1,

What kind of metric type is it? Based on the description I'm suspecting that it is an Object metric, am I right?
So answer about why our process takes that much time to start. Basically this is a server that in order to do what he needs to do, he needs to load a lot of data in memory (several GBs) and then perform some warm up that are pretty intensive. The idea is that he doesn't need to grab (almost) anything from outside to attend a request so it's latency is extremely low (in the order of millisecond) By the way. Because of this we have another problem which is that there is not an specific flag per HPA object that we could tell k8s to don't try to scale up again passed some time so the new pods have enough time to get ready before the next scale happens. I know that there is a cluster wide flag but we will need to have a different one by resource that we scale. Now to answer You pointed another interesting topic. The metric we use is % Utilization. We are able to calculate the level of utilization of our pods and we want to keep this metric in an specific value. About the flag. I suggested the flag so it doesn't change the existing behavior by default but I honestly think that the default behavior should be to don't include the NonReady pods as part of the calculation
do you think we can add this change to 1.10? it's critical bug/feature for us and we are happy to help with code/testing.
It's really late for 1.10, especially since we don't have any code here, and I'd like to know more about what's going on here, and especially see if it reproduces on Kubernetes 1.9 (we've had HPA controller bugs in the past). FWIW, currently unready pods are treated as consuming 0% of their utilization on scale up, and it was intended that this should prevent such issues. That is, the formula is actually: If the pods are marked as not ready, I find it strange that they would be affecting the scaling that much, unless your other pods are under real heavy load (i.e. I would assume that the unready pods would skew the scaling back down so that the number of pods just stayed constant).
I believe there is different code branch here.


Question: tc.runTest(t) } <ins> func TestReplicaCalcScaleUpCMObjectIgnoresUnreadyPods(t *testing.T) { tc := replicaCalcTestCase{ currentReplicas:  3, expectedReplicas: 5, // If we did not ignore unready pods, we'd expect 15 replicas. podReadiness:     []v1.ConditionStatus{v1.ConditionFalse, v1.ConditionTrue, v1.ConditionFalse}, metric: &metricInfo{ name:                "qps", levels:              []int64{50000}, targetUtilization:   10000, expectedUtilization: 50000, singleObject: &autoscalingv2.CrossVersionObjectReference{ Kind:       "Deployment", APIVersion: "extensions/v1beta1", Name:       "some-deployment", }, }, } tc.runTest(t) } </ins> func TestReplicaCalcScaleUpCMExternal(t *testing.T) { tc := replicaCalcTestCase{ currentReplicas:  1,

metric definition { "type": "Object", "object": { "target": { "kind": "Service", "name": "prometheus-exporter" }, "metricName": "PercentUtilization", "targetValue": "20" } } which should handle as and doesn't check Pod state.
I think proper handling of unready pods is a difficult problem that will require significant work. I suspect we may find out we need to start distinguishing pods that are booting up from pods that are 'broken' and it took us quite a while to solve analogous problem with nodes in Cluster Autoscaler (to be clear: I don't know if we will have to do that, that's just my intuition for now). That being said I agree the formula in GetObjectMetricReplicas will generally lead to scaling-up too much if the pod takes longer to initialize then backoff period in HPA (3 minutes). Assume it takes pod 4 minutes to boot up and we're currently running 2 pods. Metric value is 20 and target is 10. In this case: scales-up to 4 pods (20/10 utilization 2 pods). are starting up, they're not doing anything yet. 3 minutes later HPA scales-up to 8 pods (20/10 utilization 4 pods). If we only counted ready pods the scale-up in 3 would not occur, because the desired replicas would still be 4 (20/10 * 2). My intuition would be that this is the expected behavior of HPA. Also if there is no space in the cluster we will just keep scaling-up until we hit max replicas. The only case where current behavior would be more correct than filtering out unready pods in this formula is if someone uses a metric that is calculated based on total number of replicas. Do we advertise that as the right way to use HPA? If we add to Object metrics in 1.11 this usage pattern will stop making sense. To sum up - I think we will need to change the formula to only count ready pods at some point. I can see how current behavior could be a problem in any use-case except the one described in paragraph above. I'm inclined to agree with and provide some way to change the behavior in 1.10 (a flag or an annotation?), treating it as a bug fix.


Question: tc.runTest(t) } <ins> func TestReplicaCalcScaleUpCMObjectIgnoresUnreadyPods(t *testing.T) { tc := replicaCalcTestCase{ currentReplicas:  3, expectedReplicas: 5, // If we did not ignore unready pods, we'd expect 15 replicas. podReadiness:     []v1.ConditionStatus{v1.ConditionFalse, v1.ConditionTrue, v1.ConditionFalse}, metric: &metricInfo{ name:                "qps", levels:              []int64{50000}, targetUtilization:   10000, expectedUtilization: 50000, singleObject: &autoscalingv2.CrossVersionObjectReference{ Kind:       "Deployment", APIVersion: "extensions/v1beta1", Name:       "some-deployment", }, }, } tc.runTest(t) } </ins> func TestReplicaCalcScaleUpCMExternal(t *testing.T) { tc := replicaCalcTestCase{ currentReplicas:  1,

WDYT
Ah, yeah, I missed that it was object metrics before. I'm not entirely convinced we need a flag here -- it might just make sense to bring Object metrics in line with all the other metrics types. On the other hand, there's a subtlety here -- the other metric types don't quite ignore unready pods -- they just ignore the metrics from unready pods, which causes a heavy bias in the direction of not scaling, but an overwhelming spike can still cause scaling. That behavior is not as achievable using object metrics. I'm inclined to just ignore that subtlety, though, and say that object metrics should ignore unready pods entirely.
I think this is what we should do long term. I was worried about breaking someone who relies on targetValue + averaged metric to get effect similar to targetAverageValue and though about providing something for transitional period, but if you're ok with it I'd rather just change it as you say.
I think it's safe to change as-is -- that feature's still in beta, and it's not an API change, so it shouldn't just hard-break anyone. Plus, we've change stuff like this before (when we did the initial resource metrics change).
As said, we are more than ready to help with the fix/test if required. This since this is a major blocking for us in the implementation of the HPA in our side
is anyone actively working on implementing this? If not, I'm happy to. I think I have a fairly clear picture of the change we need to make and will likely have time early this week to get it done.
(Also, from the conversation above, to we want to tag this as a release blocker for 1.10)?
I probably won't be able to work on this before the end of this week. Happy to do a review if you jump in and implement it.
I don't think this qualifies as a release blocking bug (but I'm willing to have someone try and convince me otherwise ;-)). There doesn't seem to be any behavior here that's explicitly broken -- it's just sub-optimal in some use cases. I'd definitely like to address this in 1.11, though.
it's not release blocker, but to share our experience ;).


Question: tc.runTest(t) } <ins> func TestReplicaCalcScaleUpCMObjectIgnoresUnreadyPods(t *testing.T) { tc := replicaCalcTestCase{ currentReplicas:  3, expectedReplicas: 5, // If we did not ignore unready pods, we'd expect 15 replicas. podReadiness:     []v1.ConditionStatus{v1.ConditionFalse, v1.ConditionTrue, v1.ConditionFalse}, metric: &metricInfo{ name:                "qps", levels:              []int64{50000}, targetUtilization:   10000, expectedUtilization: 50000, singleObject: &autoscalingv2.CrossVersionObjectReference{ Kind:       "Deployment", APIVersion: "extensions/v1beta1", Name:       "some-deployment", }, }, } tc.runTest(t) } </ins> func TestReplicaCalcScaleUpCMExternal(t *testing.T) { tc := replicaCalcTestCase{ currentReplicas:  1,

without this feature we cannot use HPA as is and we need to figure out how to build/run fork (which can be challenging because everything inside controller-manager). also we are planning to upgrade to 1.10 quickly, but not sure for 1.11 (e.g. we are two versions behind now).
Sounds like a plan :) Will try and get this done tomorrow morning!
- I submitted a pr for this - would love your eyes on it so you can see if you think it meets your use case :)
/assign
GitHub didn't allow me to assign the following users: mattjmcnaughton. Note that only and repo collaborators can be assigned. <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I checked the code and it seems good to me. About the comment about scaling down if the readyPods<CurrentPods, we should be fine since our metric is already not including the pods that are not ready.
Is there any advance in this ticket? I'm wondering if this made it to k8s 1.10 or if we need to wait for k8s 1.11?
The PR is lgtm-ed and it will merge once the code freeze is lifted. Meaning it will go to 1.11. To get it into 1.10 you would need to get approval for milestone from (sig-autoscaling lead).


Question: } func (o *ClusterInfoOptions) Run() error { <del> printService(o.Out, "Kubernetes master", o.Client.Host) </del> // TODO use generalized labels once they are implemented (#341) b := o.Builder. WithScheme(legacyscheme.Scheme).

/kind bug What happened: The wording of kubectl cluster-info is misleading. When no cluster is running, it will state (in green) that the kubernetes master is running. What you expected to happen: I would expect a different message, like Or a How to reproduce it (as minimally and precisely as possible): Execute kubectl with no config files, and no local kubernetes cluster running on 8080
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: "reason_cache.go", "runonce.go", "runtime.go", <ins> "time_cache.go", </ins> "util.go", "volume_host.go", ],

These tests have flaked consistently since merged: in many cases, they are the only failure in a test run: Either the functionality introduced by that PR is not reliable, or the test is faulty. /sig node /priority critical-urgent /assign
/milestone v1.19
/assign
Looks like this is failing ~5% of pull-kubernetes-node-e2e runs
FYI
Looks like there might still be a race somewhere looking at the kubelet logs Success case: Fail case:
Looking at it more, it seems like is unexpectedly empty here Success: Fail:
Stacks of interest I think we might be dealing with a stale If there is a call before the container is to the Pod in the runtime and the is called within the 2 second default cache period, it seems possible that the in the could have no containers.
what do you think of this analysis? If so, any obvious solutions occur to you?
If I'm correct, didn't introduce this; just the e2e test exposed an existing bug in the kubelet.
That all looks correct. I'm taking a look at the proposed solutions


Question: } } <ins> func TestStringQuantityAsApproximateFloat64(t *testing.T) { table := []struct { in  string out float64 }{ {"2Ki", 2048}, {"1.1Ki", 1126.4e+0}, {"1Mi", 1.048576e+06}, {"2Gi", 2.147483648e+09}, } for _, item := range table { t.Run(item.in, func(t *testing.T) { in, err := ParseQuantity(item.in) if err != nil { t.Fatal(err) } out := in.AsApproximateFloat64() if out != item.out { t.Fatalf("expected %v, got %v", item.out, out) } if in.d.Dec != nil { if i, ok := in.AsInt64(); ok { q := intQuantity(i, 0, in.Format) out := q.AsApproximateFloat64() if out != item.out { t.Fatalf("as int quantity: expected %v, got %v", item.out, out) } } } }) } } </ins> func benchmarkQuantities() []Quantity { return []Quantity{ intQuantity(1024*1024*1024, 0, BinarySI),

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --I found that for with returns a very small value. The value should be almost the same as the one of . My minimal : Result: While the top three lines () are good, the float values in the bottom three lines () might be broken. Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig api-machinery
/assign
/cc /triage accepted


Question: Subsystem:         KubeletSubsystem, Name:              AssignedConfigKey, Help:              "The node's understanding of intended config. The count is always 1.", <del> DeprecatedVersion: "1.23.0", </del> <ins> DeprecatedVersion: "1.22.0", </ins> StabilityLevel:    metrics.ALPHA, }, []string{ConfigSourceLabelKey, ConfigUIDLabelKey, ConfigResourceVersionLabelKey, KubeletConfigKeyLabelKey},

This shipped in 1.22 as deprecated in 1.22. As a hacky fix to unbreak tests on 1.23 dev, I bumped the deprecation policy one release: We will need to revert that before 1.23 ships and figure out a better solution. /kind cleanup /milestone v1.23 /sig node
/triage accepted
I want to try it. Can I take this issue?
I have explained how to fix this on PR:
/reopen reverted in as fix did not work
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
Hi, Im the bug triage shadow. Since we're in 1.23 milestone code freeze, could we move this issue to milestone 1.24?
this is critical for 1.23 and I think it just got forgotten... we need to get this fixed before test freeze
/priority important-soon


Question: AllowExpandedDNSConfig bool // Allow invalid topologySpreadConstraint labelSelector for backward compatibility AllowInvalidTopologySpreadConstraintLabelSelector bool <ins> // Allow node selector additions for gated pods. AllowMutableNodeSelectorAndNodeAffinity bool </ins> } // validatePodMetadataAndSpec tests if required fields in the pod.metadata and pod.spec are set,

See for context. This issue is creating to track work on the API validation changes. Validation relaxation of certain fields in the pod spec to allow for mutable scheduling directives.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/sig scheduling
Docs PR:


Question: # journal message level, 0 is debug KUBE_LOG_LEVEL=0 <ins> KUBE_ALLOW_PRIV="true" </ins> ``` * Start the appropriate services on fed1:

https://k8s- Failed: [] Density [Feature:Performance] should allow starting 30 pods per node {Kubernetes e2e suite} Previous issues for this test:
https://k8s- Failed: [] Density [Feature:Performance] should allow starting 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Density [Feature:Performance] should allow starting 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Density [Feature:Performance] should allow starting 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Density [Feature:Performance] should allow starting 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Density [Feature:Performance] should allow starting 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Density [Feature:Performance] should allow starting 30 pods per node {Kubernetes e2e suite}
There are 3 different issues in this bug: high resource consumption (or too strict limits on our side) - should fix it to remove namespace: [I looked into it, and it was because events etcd didn't even come up in this test. The problem here is that we are not collecting logs from etcd in kubemark - I will send a PR fixing it later today hopefully. (that I didn't have time to look into yet).
So actually, the non-started events etcd was the pulling image problem: So we may want to retry in case of failures also.
The first one should be fixed by The second one will be fixed by The third problem is actually related to pod-nanny, which is frequently sending list requests to apiserver. It should be rewritten to use watch instead.
Since the main issue is fixed, I'm decreasing the priority.
For the third problem, I've just sent out: for review. Once that one is merged, we should build the new image and bump tags in our config files.


Question: # journal message level, 0 is debug KUBE_LOG_LEVEL=0 <ins> KUBE_ALLOW_PRIV="true" </ins> ``` * Start the appropriate services on fed1:

https://k8s- Failed: [] Density [Feature:Performance] should allow starting 30 pods per node {Kubernetes e2e suite}
So the remaining things here to do are: kubernetes/contrib + update version of pod-nanny in heapster manifests tun kubemark-5 resource constraints
What we should also do, is to add "kubemark-scale" to list of non-blocking suites of merge-queue, so that bugs will be automatically filed for that suite.
https://k8s- Failed: [] Density [Feature:Performance] should allow starting 30 pods per node {Kubernetes e2e suite}
There are three remaining things to do here: tweak limits for kubemark-5 merge add kubemark-scale to non-blocking suites of merge-queue can you please take care of the first one? I will take care of the other two.
https://k8s- Failed: [] Density [Feature:Performance] should allow starting 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Density [Feature:Performance] should allow starting 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Density [Feature:Performance] should allow starting 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Density [Feature:Performance] should allow starting 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Density [Feature:Performance] should allow starting 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Density [Feature:Performance] should allow starting 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Density [Feature:Performance] should allow starting 30 pods per node {Kubernetes e2e suite}
It seems that this PR broke scalability: I'm reverting it and will try to debug it offline.
https://k8s- Failed: [] Density [Feature:Performance] should allow starting 30 pods per node {Kubernetes e2e suite}
All the scalability failures from last day are because of - that PR was reverted, and I fixed the problem in and built a new image. (I did some 100-node cluster tests locally and it worked fine after the change).


Question: err = e2epod.WaitForPodNameRunningInNamespace(ctx, m.cs, pod.Name, pod.Namespace) framework.ExpectNoError(err, "Failed to start pod1: %v", err) <ins> pvc, err = m.cs.CoreV1().PersistentVolumeClaims(pvc.Namespace).Get(ctx, pvc.Name, metav1.GetOptions{}) if err != nil { framework.Failf("failed to get pvc %s, %v", pvc.Name, err) } gomega.Expect(pvc.Spec.VolumeName).ShouldNot(gomega.BeEquivalentTo(""), "while provisioning a volume for resizing") </ins> pv, err := m.cs.CoreV1().PersistentVolumes().Get(ctx, pvc.Spec.VolumeName, metav1.GetOptions{}) if err != nil { framework.Failf("failed to get pv %s, %v", pvc.Spec.VolumeName, err)

Seen in pull-kubernetes-e2e-gce-cos-alpha-features Specific job failure
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
from the error message I don't know if it refers to the sig-storage error or to the sig-auth one
/sig storage
/kind failing-test
/assign From : ... I am not sure if it's related:
is the culprit. cannot be used inside a . This only gets caught when the test actually runs, otherwise Ginkgo never sees the incorrect .
I don’t understand why sometimes it can get the pv by namebut sometimes it failed
but then is a sig-auth test , not storage , sorry /sig auth this seems that was recently in
/remove sig-storage
/assign /priority important-soon
/unassign
is still failed. seems not fixing the whole problem. /reopen feel free to close if the ci is green.
Reopened this issue. <detailsIn response to : is still failed. /reopen feel free to close if the ci is green. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
the failure now looks like it is just a sig storage issue:
cc
/assign looks like the new test is still an issue
I think this line is the problem: When the test creates the object, it may take time for the PV to be provisioned and to be bound to the PVC. So you should do another before this call so you can get the latest PVC object with the bound PV.
I opened to get PVC again before using .
Only do PersistentVolumeClaims.Get() is not enough, whether we need to wait for sometimes to wait it bound? because sometimes the testcase is successful and sometime is failed.
If the Pod was able to run, then the PVC has to be bound.
so I can't understand why sometimes the testcase is successful and sometimes is failed.


Question: err = e2epod.WaitForPodNameRunningInNamespace(ctx, m.cs, pod.Name, pod.Namespace) framework.ExpectNoError(err, "Failed to start pod1: %v", err) <ins> pvc, err = m.cs.CoreV1().PersistentVolumeClaims(pvc.Namespace).Get(ctx, pvc.Name, metav1.GetOptions{}) if err != nil { framework.Failf("failed to get pvc %s, %v", pvc.Name, err) } gomega.Expect(pvc.Spec.VolumeName).ShouldNot(gomega.BeEquivalentTo(""), "while provisioning a volume for resizing") </ins> pv, err := m.cs.CoreV1().PersistentVolumes().Get(ctx, pvc.Spec.VolumeName, metav1.GetOptions{}) if err != nil { framework.Failf("failed to get pv %s, %v", pvc.Spec.VolumeName, err)

By default, the DelayBinding is false, it will wait for PVC bound during creation. This is weird.
It waits for the claim to be bound, but the test case still needs to refresh its pvc object.
Because of the refresh is async, so after the pvc was bound, we need to get the pvc again, isn't it?


Question: csiAttacher := attacher.(*csiAttacher) <del> go func(spec *volume.Spec, id, nodename string, fail bool) { </del> <ins> go func(spec *volume.Spec, nodename string, fail bool) { </ins> attachID, err := csiAttacher.Attach(spec, types.NodeName(nodename)) if !fail && err != nil { t.Errorf("expecting no failure, but got err: %v", err)

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: Followup to In 1.15, we can merge this commit: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
/assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


Question: } } <ins> var ( err                                                  error errorStream, remoteStdin, remoteStdout, remoteStderr httpstream.Stream ) // set up all the streams first </ins> headers := http.Header{} headers.Set(api.StreamType, api.StreamTypeError) <del> errorStream, err := conn.CreateStream(headers) </del> <ins> errorStream, err = conn.CreateStream(headers) </ins> if err != nil { return err } <ins> defer errorStream.Reset() // Create all the streams first, then start the copy goroutines. The server doesn't start its copy // goroutines until it's received all of the streams. If the client creates the stdin stream and // immediately begins copying stdin data to the server, it's possible to overwhelm and wedge the // spdy frame handler in the server so that it is full of unprocessed frames. The frames aren't // getting processed because the server hasn't started its copying, and it won't do that until it // gets all the streams. By creating all the streams first, we ensure that the server is ready to // process data before the client starts sending any. See https://issues.k8s.io/16373 for more info. if e.stdin != nil { headers.Set(api.StreamType, api.StreamTypeStdin) remoteStdin, err = conn.CreateStream(headers) if err != nil { return err } defer remoteStdin.Reset() } if e.stdout != nil { headers.Set(api.StreamType, api.StreamTypeStdout) remoteStdout, err = conn.CreateStream(headers) if err != nil { return err } defer remoteStdout.Reset() } if e.stderr != nil && !e.tty { headers.Set(api.StreamType, api.StreamTypeStderr) remoteStderr, err = conn.CreateStream(headers) if err != nil { return err } defer remoteStderr.Reset() } // now that all the streams have been created, proceed with reading & copying // always read from errorStream </ins> go func() { message, err := ioutil.ReadAll(errorStream) if err != nil && err != io.EOF {

We create the stdin stream and immediately start copying local stdin to the container. We do the same for stdout and stderr (create stream, start copying immediately). In the kubelet, we wait to receive all the streams before we start the exec and copy between the streams and the container. The spdy connection frame handling code has 5 frame workers, each with a queue of up to 50 frames. In the event that we try to push more than 50 frames to a single queue before all streams are ready, the 51st attempt to push a frame blocks until there is space available in the queue. But there won't be any space until the server starts reading from the streams, which won't happen until all the streams have been received, so we're stuck at this point. The fix is to create all the streams first, and only start copying after all the streams have been created and the server is ready to copy between the streams and the container. Originally reported here:
FYI
This seems pretty high priority.
It's an easy fix. has already tested the patch for OpenShift and it works. Are we willing to pull it in before is resolved, as that will take longer to put in place?
Here's the patch, for reference:
5 * 50 = 250 frames before you get stuck? Or did I misunderstand?
Oh, I guess all the frames go to the same worker for some reason?
anyway, I'm fine with taking this and cherry picking it. Order doesn't matter; someone has to rebase either way, right?
frames are assigned to each queue based on stream id, so if you sent 50 frames to the stdin stream, that queue fills up and the loop that processes incoming frames is blocked until something starts reading from the stream. Working on a PR now.
Gotcha, thanks. , Andy Goldstein wrote:
an update: I spent some time trying to create an e2e that would fail more often than not without this fix, but I haven't had any luck so far (I have it stream a copy of the kubectl binary from the local system to the pod). The issue that originally uncovered the problem was from running exec on a Windows system, where it reproduces consistently, but I haven't seen it fail much on Linux. I will keep trying...


Question: "k8s.io/kubernetes/pkg/apis/batch" "k8s.io/kubernetes/pkg/runtime" "k8s.io/kubernetes/pkg/types" <del> hashutil "k8s.io/kubernetes/pkg/util/hash" </del> ) // Utilities for dealing with Jobs and CronJobs and time.

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT Kubernetes version (use ): Environment: Cloud provider or hardware configuration: AWS OS (e.g. from /etc/os-release): Debian GNU/Linux 8 (jessie) Kernel (e.g. ): 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux What happened: Deployed a scheduled job from this manifest: After some time the jobs are no longer executed: What you expected to happen: Jobs are executed as defined in the manifest. How to reproduce it (as minimally and precisely as possible): Deploy a scheduled job and wait for some time.
Some logs from the controller manager:
Ping. More than 2 weeks and no action at all? Anything else i can do?
I'm also encountering this using both and as concurrency policies, fwiw.
I'm hitting this as well. Kube
Do the jobs start to reschedule if you delete old one? Try running . cc
From kube-controller-manager logs: The job is scheduled for 1:00. I think the "Multiple unmet start times" message is a clue of where things start going downhill. Also, notice how at 1:00 the controller wants to create job (for a whole day, it had failed on ).
Deleting (all) old jobs / pods fixes the problem temporarily. After some time the scheduled jobs face the same issue again.
There's a fix () that I've submitted against master to fix the Replace strategy, I'll mark it for inclusion in 1.4.
The other problem that I see here is this: It is possible that our is creating duplicates. I'll try to investigate that a bit more.
sorry for the delay in response time, I must have missed that while going through issues. Next time you're experiencing problems/have a question wrt Jobs/ScheduledJobs, just ping me directly in the issue.
No problem and I will. Thanks!
I'm still hitting this on
Fix in flight , I'll make sure to cherry-pick it back to 1.4
does this ship in to latest kubernetes release?
yes, this should be present in 1.5, 1.4 cherry-pick hasn't been merged yet.
so this issue is still present in v1.4.6?


Question: "k8s.io/kubernetes/pkg/apis/batch" "k8s.io/kubernetes/pkg/runtime" "k8s.io/kubernetes/pkg/types" <del> hashutil "k8s.io/kubernetes/pkg/util/hash" </del> ) // Utilities for dealing with Jobs and CronJobs and time.

yes - it is still present there. My fix was merged to 1.4 yesterday. You should wait for the next 1.4.x release.
thanks!
which according to should happen Dec 8th.
any news on the release?
See !msg/kubernetes-dev/o0At2-4KKQ8/_kPVM1XyDAAJ
so v1.4.7 will be released on that same day?
Look like should be available later today: !msg/kubernetes-dev/00lJpT9pxDc/YZfDzCxMDQAJ


Question: close(monitor.stopCh) } } <ins> // reset monitors so that the graph builder can be safely re-run/synced. gb.monitors = nil </ins> glog.Infof("stopped %d of %d monitors", stopped, len(monitors)) }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: https://k8s- What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig testing /area build-release
A lot of tests in are very flaky. From and , and are the top 3 flaky tests among all kubernetes tests. /assign
fix double close channel panic related to https://k8s-
Is this issue ready to be closed? Did that PR fix the reason for flakes in all these tests?
This is still a problem, these tests are still the flakiest https://k8s-
My fix is double close channel panic. This one is not fixed. May be I shoud edit the title.
we also need a new issue then, I'm a bit overloaded ATM could one of open a new one with the garbagecollector failures?
OK
I'll try to debug some of these.
I created a new issue
Thanks!


Question: pod := newTestPodForQuota(podName, requests, api.ResourceList{}) pod, err = f.Client.Pods(f.Namespace.Name).Create(pod) Expect(err).NotTo(HaveOccurred()) <ins> podToUpdate := pod </ins> By("Ensuring ResourceQuota status captures the pod usage") usedResources[api.ResourceQuotas] = resource.MustParse("1")

The scenario that presented a problem was the following: a) create pods in a namespace that do not make a memory claim b) add a quota that tracks memory c) update one of the pods previously created by reducing its active deadline. Expected result: Since a pod cannot change its resources via update, we should have been allowed to update the pod spec. Actual Result: error since the pod did not make a memory claim, quota rejects the pod.
cc


Question: } } <ins> func TestUpdatePodForTerminatedRuntimePod(t *testing.T) { podWorkers, processed := createPodWorkers() now := time.Now() podWorkers.podSyncStatuses[types.UID("1")] = &podSyncStatus{ startedTerminating: true, terminatedAt:       now.Add(-time.Second), terminatingAt:      now.Add(-2 * time.Second), gracePeriod:        1, } // creates synthetic pod podWorkers.UpdatePod(UpdatePodOptions{ UpdateType: kubetypes.SyncPodKill, RunningPod: &kubecontainer.Pod{ID: "1", Name: "1", Namespace: "test"}, }) drainAllWorkers(podWorkers) if len(processed) != 0 { t.Fatalf("Not all pods processed: %v", processed) } updates := processed["1"] if len(updates) != 0 { t.Fatalf("unexpected updates: %v", updates) } if len(podWorkers.lastUndeliveredWorkUpdate) != 0 { t.Fatalf("Unexpected undelivered work") } } </ins> func TestUpdatePodDoesNotForgetSyncPodKill(t *testing.T) { podWorkers, processed := createPodWorkers() numPods := 20

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --When testing against OpenShift CI, I discovered a panic in the kubelet: Further investigation revealed that it is possible for to panic under certain circumstances: We hit this case when (podStatus remains nil) and (nil will be passed along to ). I'm not sure why exactly that would happen but it seems possible. Don't panic! Unclear, I've been trying to trace the code/find evidence of this happening in k8s CI but haven't found a match yet. ( would be nice :smile:) /sig node /cc /priority important-soon /milestone v1.22 Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/triage accepted
/assign
The bug is that running pods should terminate and then stop, not move to terminated. So something is getting into terminated but should never be in any other work type than terminating. We'll need more debug to determine why but I suspect that an orphaned pod is still in podSyncStatus at the time it UpdatePod is called, which means it's not seen as being in Terminating. isRuntimePod forces podSyncStatus to be a certain state, but perhaps it's should resett a terminated bool. Asked elana to add a v4 min level in the test jobs that reproduced this so we can see.
has bumped, here's a v=4 log with a panic: https://gcsweb-
Elana got the version with more logs and it is indeed the case that the pod is terminating (in this case because a CSI volume hasn't been torn down).


Question: } } <ins> func TestUpdatePodForTerminatedRuntimePod(t *testing.T) { podWorkers, processed := createPodWorkers() now := time.Now() podWorkers.podSyncStatuses[types.UID("1")] = &podSyncStatus{ startedTerminating: true, terminatedAt:       now.Add(-time.Second), terminatingAt:      now.Add(-2 * time.Second), gracePeriod:        1, } // creates synthetic pod podWorkers.UpdatePod(UpdatePodOptions{ UpdateType: kubetypes.SyncPodKill, RunningPod: &kubecontainer.Pod{ID: "1", Name: "1", Namespace: "test"}, }) drainAllWorkers(podWorkers) if len(processed) != 0 { t.Fatalf("Not all pods processed: %v", processed) } updates := processed["1"] if len(updates) != 0 { t.Fatalf("unexpected updates: %v", updates) } if len(podWorkers.lastUndeliveredWorkUpdate) != 0 { t.Fatalf("Unexpected undelivered work") } } </ins> func TestUpdatePodDoesNotForgetSyncPodKill(t *testing.T) { podWorkers, processed := createPodWorkers() numPods := 20

So the bug is: worker for the pod is still running (terminating has completed, but is executing syncTerminatedPod) runs, calls , gets a stale result for a pod (thinks the pod is still running, but that's just a stale cache result) asks pod worker to terminate the pod with a runningPod worker sees that the pod is actually not terminating, but terminated, and so RunningPod is inconsistently handled (UpdatePod is not defensive enough) worker passes the kill pod request through to syncTerminatedPod which has a null pod status, causing the panic In earlier versions of we explicitly refreshed the runtime cache in , but when and I went through review we removed it and I forgot this race was possible. The race becomes more possible here because the particular workload Elana is testing involves CSI drivers that have a bad cleanup loop, so syncTerminatedPod is reaching timeout on volume unmount (3m) before finally succeeding. This means the race window is huge (invoke UpdatePod(runningPod) on an terminated but not yet cleaned up pod). I think the real bug is that UpdatePod is not defensive enough - the difference between this race condition (seeing an older runtime container) and a human somehow starting the container up again before it is terminated is pretty similar. The important difference is that the UpdatePod{... running pod } is intended to clean up orphaned pods - those the pod worker is not tracking. But we only go into terminated when we are certain the workload is stopped, so if we simply ignore until the pod goes away it might be a while before we kill that container. We absolutely need a background worker tracking that, which is best to be a pod worker, but it's a bit more work. So easy fix (remove the possibility of the very common race) is to simply have syncTerminatingPod refresh the pod cache for that one pod. A slightly less easy fix is to have UpdatePod simply ignore the runningPod request until the next housekeeping loop (which in some cases with bad csi drivers could be a few minutes or so and in the worst case might block). The most correct fix is to ensure syncTerminated pod has an opportunity to maintain its invariants (no running containers) or have that be maintained alongside. Pondering variations here of fixes.


Question: } } <ins> func TestUpdatePodForTerminatedRuntimePod(t *testing.T) { podWorkers, processed := createPodWorkers() now := time.Now() podWorkers.podSyncStatuses[types.UID("1")] = &podSyncStatus{ startedTerminating: true, terminatedAt:       now.Add(-time.Second), terminatingAt:      now.Add(-2 * time.Second), gracePeriod:        1, } // creates synthetic pod podWorkers.UpdatePod(UpdatePodOptions{ UpdateType: kubetypes.SyncPodKill, RunningPod: &kubecontainer.Pod{ID: "1", Name: "1", Namespace: "test"}, }) drainAllWorkers(podWorkers) if len(processed) != 0 { t.Fatalf("Not all pods processed: %v", processed) } updates := processed["1"] if len(updates) != 0 { t.Fatalf("unexpected updates: %v", updates) } if len(podWorkers.lastUndeliveredWorkUpdate) != 0 { t.Fatalf("Unexpected undelivered work") } } </ins> func TestUpdatePodDoesNotForgetSyncPodKill(t *testing.T) { podWorkers, processed := createPodWorkers() numPods := 20

The easy fixes will defiintely be "safe", but "slight regression in latency in a few rare cases". Serializing the pod worker caused this, but it's in theory rare.
My understanding is that we make pod worker the source of truth for pod state. In this kind of race condition, as long as the request is sent to the pod worker, and the pod UID matches, it should be able to reconcile the state. Even if we do this, it is not atomic, the race condition still exists in theory. We probably should make the pod worker be able to reconcile the state. I don't quite understand what you mean by "until the next housekeeping loop"? Since the pod is being terminated in , what problem would it cause if we just ignore the runningPod request? If the termination is still in progress, we need to wait for it to finish anyway.
Here's the most concerning scenario: is terminated and then deleted from apiserver kills the container is taking a very long time after 2, housekeeping fires and sees the stale cache, triggers a kill (proposed change) ignore the kill request (since terminated pod is running) restarts container in the pod that references a volume mount can't make progress because volume detach can't make progress because the restarted container is holding a device lock / has a file open / w/e pod worker for that pod retries forever is the minimal fix that ignores (5) but I'm thinking through how to handle 8. I don't think we should do the easy fix (refreshing the cache isn't enough because you are vulnerable to 6-8). I'm potentially considering having the pod worker invoke syncTerminatingPod in the event syncTerminatedPod exits with an error, or simply have syncTerminatedPod do it if it would return an error to address 8.
But this basically means that after , there could still be running containers. In that case, this wouldn't be the only problem, the whole assumption the refactor builds on becomes invalid? BTW, restarting container may happen to docker, but containerd doesn't support it (at least it can't be done easily), not sure about cri-o.
cri-o doesn't support restarting a container, either.
The assumption in the kubelet is there are no more containers started by the kubelet, which is indeed safe.


Question: } } <ins> func TestUpdatePodForTerminatedRuntimePod(t *testing.T) { podWorkers, processed := createPodWorkers() now := time.Now() podWorkers.podSyncStatuses[types.UID("1")] = &podSyncStatus{ startedTerminating: true, terminatedAt:       now.Add(-time.Second), terminatingAt:      now.Add(-2 * time.Second), gracePeriod:        1, } // creates synthetic pod podWorkers.UpdatePod(UpdatePodOptions{ UpdateType: kubetypes.SyncPodKill, RunningPod: &kubecontainer.Pod{ID: "1", Name: "1", Namespace: "test"}, }) drainAllWorkers(podWorkers) if len(processed) != 0 { t.Fatalf("Not all pods processed: %v", processed) } updates := processed["1"] if len(updates) != 0 { t.Fatalf("unexpected updates: %v", updates) } if len(podWorkers.lastUndeliveredWorkUpdate) != 0 { t.Fatalf("Unexpected undelivered work") } } </ins> func TestUpdatePodDoesNotForgetSyncPodKill(t *testing.T) { podWorkers, processed := createPodWorkers() numPods := 20

That doesn’t prevent bugs in CRI implementations, user error, cosmic rays, basically computers. It’s unlikely and not something we really designed for before, but the prerefactor code would survive it. Appears to have gotten past the issue as expected. I’ll look at what it would take to mitigate the possibility of zombie containers such that the pod worker converges even if volume mount can’t be completed.
In general a little defensiveness here to unreliable external components has only minor cost, and some mitigation seems reasonable.
Opened the first part of the fix in
SG. Yeah, that should be considered a rare case. However, adding some defensive cleanup also SGTM as long as it doesn't introduce too much overhead or complexity. :)


Question: // create independent pod informer if required if setPodInformer { <del> podInformer = factory.NewPodInformer(context.clientSet, 12*time.Hour, v1.DefaultSchedulerName) </del> <ins> podInformer = factory.NewPodInformer(context.clientSet, 12*time.Hour) </ins> } else { podInformer = context.informerFactory.Core().V1().Pods() }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Scheduler's pod informer has a selector that . This is incorrect. Scheduler must watch all the pods of the cluster, including those scheduled by other schedulers in order to update its cache of the state of the cluster. Otherwise, it will miss important events and the amount of available resources, pods on nodes, ... which are needed for scheduler to work correctly, will not be updated properly. /sig scheduling /assign
cc
For updating it cache, shouldn't the default scheduler rely on node info/status? Also its not very clear, what information the default scheduler needs from other pods (not to be scheduled by it) that it makes decision on? Just trying to understand the issue more clearly.
Scheduler watches informer events to update its cache. Node status is not enough to update all the information that scheduler caches. Scheduler must know what pods are placed on what nodes. For example, in order to check inter-pod affinity/anti-affinity it needs to know the placement of all the pods, including those scheduled by other schedulers.
But this information is present here: So it seems that your PR will now make sure to get the scheduled pods' information directly from the pod informer.
So when I check the history, I see it was here:
have we seen any specific issues due to this change (like between default scheduler and DS controller)?
Yes, but scheduler relies on its pod informers to keep this location up to date. That's exactly why it is important for the scheduler to receive updates for all the pods in the cluster. Yes. We missed the problem at the time. This is problematic only in clusters that use multiple schedulers. I initially thought that DS controller could cause issues as well, but DS controller does not set , so it must not cause an issue. We have seen instances of scheduler's missing certain events in cluster and as a result scheduler's cache getting stale, but I doubt if it is related to this particular issue. This only affects clusters with more than one scheduler. If you run the test that I have written in with the head, it will fail. That tests simulate a multi-scheduler scenario.


Question: record.StartLogging(glog.Infof) } <del> func SetupEventSending(client *client.Client) { </del> <ins> func SetupEventSending(client *client.Client, hostname string) { </ins> glog.Infof("Sending events to api server.") <del> hostname := util.GetHostname("") </del> record.StartRecording(client.Events(""), api.EventSource{ Component: "kubelet",

Running the pre-built 0.8.0 binaries this flag does not work. Looks like the
I think that code happens only when setting up events, right? The hostnameOverride flag seems to still be there: what do you think about making this function take hostnameOverride as a parameter?
Yeah, it should fix the issue. Will fix it tonight.


Question: // Ok, so we got something back from etcd. Let's set up a watch for new services, and // their endpoints <del> go s.WatchForChanges() </del> <ins> go util.Forever(s.WatchForChanges, 1*time.Second) </ins> for { services, endpoints, err = s.GetServices()

kube-proxy regularly stops running with the below panic / stack-trace. This is when configured with vagrant. (It is also running on a laptop, so my unsupported-by-any-evidence theory is that it happens when the laptop resumes after sleep.) Restarting the service causes the proxy to continue working happily. (Should systemd be restarting the service?) Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: panic: runtime error: invalid memory address or nil pointer dereference Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: [signal 0xb code=0x1 addr=0x0 pc=0x431b51] Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: goroutine 22 [running]: Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: (0x6b7c00, 0xa51de8) Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /usr/local/go/src/pkg/runtime/panic.c:266 +0xb6 Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: (0xc210066600, 0xc210037660, 0xc210037720, 0x77359400, 0x0) Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x841 Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: (0xc210066600, 0xc210037660, 0xc210037720,


Question: // Ok, so we got something back from etcd. Let's set up a watch for new services, and // their endpoints <del> go s.WatchForChanges() </del> <ins> go util.Forever(s.WatchForChanges, 1*time.Second) </ins> for { services, endpoints, err = s.GetServices()

0x77359400) Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x165 Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: created by Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x1b3 Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: goroutine 1 [select (no cases)]: Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: () Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x47a Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: goroutine 3 [chan receive]: Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: (loggingT).flushDaemon(0xa57340) Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x50 Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: created by ·1 Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x276 Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: goroutine 4 [syscall]: Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: () Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /usr/local/go/src/pkg/runtime/proc.c:1394 Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: goroutine 6 [sleep]: Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: time.Sleep(0x12a05f200) Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: +0x31 Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: (0x7aca18,


Question: // Ok, so we got something back from etcd. Let's set up a watch for new services, and // their endpoints <del> go s.WatchForChanges() </del> <ins> go util.Forever(s.WatchForChanges, 1*time.Second) </ins> for { services, endpoints, err = s.GetServices()

0x12a05f200) Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x57 Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: created by Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x99 Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: goroutine 7 [chan receive]: Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: (0xc210077120, 0x7ff2602627f0, 0xc210077060, 0xc210037540) Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x3a Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: created by Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x196 Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: goroutine 8 [chan receive]: Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: (0xc210077210, 0x7ff260262840, 0xc210077150, 0xc2100375a0) Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x3a Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: created by Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x196 Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: goroutine 9 [chan receive]: Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: (Mux).listen(0xc2100770c0, 0x718260, 0x4,


Question: // Ok, so we got something back from etcd. Let's set up a watch for new services, and // their endpoints <del> go s.WatchForChanges() </del> <ins> go util.Forever(s.WatchForChanges, 1*time.Second) </ins> for { services, endpoints, err = s.GetServices()

0xc210037600) Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x3a Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: ·001() Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x43 Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: ·001() Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x40 Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: (0xc21000a920, 0x0) Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x49 Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: created by (*Mux).Channel Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x20a Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: goroutine 10 [chan receive]: Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: ·004() Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x4d Aug 13 12:04:06 kubernetes-minion-1 systemd[1]: kube-proxy.service: main process exited, code=exited, status=2/INVALIDARGUMENT Aug 13 12:04:06 kubernetes-minion-1 systemd[1]: Unit kube-proxy.service entered failed state.
I looked at this one a little bit and I think it's because of This line isn't present at , but I'm not sure which is newer. Presumably results in the channel being closed (and receiving a zero value that it then tries to dereference) on error.
Diagnostic: This line doesn't check for a closed channel, and it needs to. Yeah, how are we supposed to know if our client is going to close the channel or not?


Question: // Ok, so we got something back from etcd. Let's set up a watch for new services, and // their endpoints <del> go s.WatchForChanges() </del> <ins> go util.Forever(s.WatchForChanges, 1*time.Second) </ins> for { services, endpoints, err = s.GetServices()

I think the behavior has changed at least once already. This is part of why I'm trying to (for our code, anyway) standardize all of our watches with a watch.Interface. I took the liberty of assigning this to you since you're more familiar with this code, but feel free to unassign if you can't take it right now.
Got it
can you give a try?
after i do the minion watch for kubelet config i'll come back and gut this and replace it.
Thanks - with merged I haven't seen this happening. Thanks for the super-quick fix!


Question: # Clean up kubectl delete namespace other <del> ############## </del> <ins> ########### </ins> # Secrets # <del> ############## </del> <ins> ########### </ins> ### Create a new namespace # Pre-condition: the test-secrets namespace does not exist

The desciption of --all-namespaces is listing the requested object(s) across all namespaces. Namespace in current context is ignored even if specified with --namespace." But when I used the following command: error: namespace may not be empty when retrieving a resource by name no namespace set on resource pods "counter" The error messages are unclear, and the second message isn't labeled as an error even.
And why couldn't use --all-namespaces flag?
You're asking for a specific pod name (counter) in all namespaces, which is a nonsensical query. The pod name is not a grep-like filter, it's a specific name. Yeah, the errors could be better. Describe could work, but potentially generates a LOT of information and API calls.
But I found the following API is supported now. Get The result is: has the more information. Do we need to support the following kubectl command(using the above API) or just return clear error messages? get pods counter --all-namespaces
I'd be ok with this being implemented via the field selector.
The issue is still here :). What's our expected result? I'd like to contribute this.
, after check the code, the URL was built by ; do you mean we should update it; or avoid to use , instead, build url () directly.
After checking the code, it's better to add fieldSelector to , is created for fieldSelector. Will submit a PR based on .
getting an object with a particular name across all namespaces seems like something we shouldn't encourage... it breaks the assumption of namespace independence.
, so we'll only update message for this issue? for example:
Disagree. Need easy way to troubleshouting the problem "why pods is stuck in Pending status". It's tiring for every pod to find who created it and in which namespace. In the logs there is only the name of the pod.
Now, with field selectors we can do To further my understanding, I'd ask:- doesn't itself break independence? One can search by label across all namespaces, it feels funny for by name to be special.


Question: "k8s.io/apimachinery/pkg/util/uuid" clientset "k8s.io/client-go/kubernetes" coreclientset "k8s.io/client-go/kubernetes/typed/core/v1" "k8s.io/kubernetes/pkg/kubelet/util" "k8s.io/kubernetes/test/e2e/framework" e2epod "k8s.io/kubernetes/test/e2e/framework/pod" testutils "k8s.io/kubernetes/test/utils" <del> "github.com/onsi/ginkgo" "github.com/onsi/gomega" </del> ) var _ = framework.KubeDescribe("NodeProblemDetector [NodeFeature:NodeProblemDetector] [Serial]", func() {

Which jobs are failing: Which test(s) are failing: Since when has it been failing: Last successful run (23 Oct 2020, 10:07 PDT): First unsuccessful run (23 Oct 2020, 12:08 PDT): Testgrid link: Reason for failure: Anything else we need to know: presubmit job in repository also started to fail in the same fashion. Attempted to debug it myself, found out the following stuff: last successful run was accessing the apiserver via HTTP: all unsuccessful runs attempted to access the apiserver via HTTPS but to no avail: perused the history of to find out that the PR is almost certainly the reason of the failures (or rather uncovered some problem with the NPD or its tests): first, it coincides with the start of the failures of , second, we changed the default hostname of the apiserver in node tests in that PR. Judging by the code and the changes in the aforementioned PR I think that a quick mitigation would be to add a flag to the of these jobs. /assign /sig node /cc
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Opened for a mitigation that would make those jobs stop failing.
This won't work, unfortunately. Copying response:
I have filed to address this issue, but it seems I could not run the test in advance to verify my patch, did you know how to manually trigger the presubmit job?
Unfortunately I don't :/. My PR was also a wild guess. could you take a look at the proposed fix? Thanks once again for your help - really appreciated :).
the submission of hasn't fixed the problems, unfortunately :/. Sample failing run: We still have some peculiar problem related to health checking: and now NPD panics, too:
Thanks for your report, I believe the reason is admission controller has been disabled in node e2e tests: We need to figure out a new approach now. /reopen
Reopened this issue.


Question: "k8s.io/apimachinery/pkg/util/uuid" clientset "k8s.io/client-go/kubernetes" coreclientset "k8s.io/client-go/kubernetes/typed/core/v1" "k8s.io/kubernetes/pkg/kubelet/util" "k8s.io/kubernetes/test/e2e/framework" e2epod "k8s.io/kubernetes/test/e2e/framework/pod" testutils "k8s.io/kubernetes/test/utils" <del> "github.com/onsi/ginkgo" "github.com/onsi/gomega" </del> ) var _ = framework.KubeDescribe("NodeProblemDetector [NodeFeature:NodeProblemDetector] [Serial]", func() {

<detailsIn response to : Thanks for your report, I believe the reason is admission controller has been disabled in node e2e tests: We need to figure out a new approach now. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
filed . I hope we could completely address this issue this time.
/milestone v1.20


Question: <ins> // +build !windows /* Copyright 2018 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package dockershim import ( dockertypes "github.com/docker/docker/api/types" dockercontainer "github.com/docker/docker/api/types/container" runtimeapi "k8s.io/kubernetes/pkg/kubelet/apis/cri/runtime/v1alpha2" ) // applySandboxPlatformOptions applies platform specific options to dockercontainer.HostConfig and dockercontainer.ContainerCreateConfig. func (ds *dockerService) applySandboxPlatformOptions(hc *dockercontainer.HostConfig, config *runtimeapi.PodSandboxConfig, createConfig *dockertypes.ContainerCreateConfig, image string, separator rune) error { lc := config.GetLinux() if lc == nil { return nil } // Apply security context. if err := applySandboxSecurityContext(lc, createConfig.Config, hc, ds.network, separator); err != nil { return err } // Set sysctls. hc.Sysctls = lc.Sysctls return nil } </ins>


Question: // AfterEach deletes the namespace, after reading its events. func (f *Framework) AfterEach() { <ins> // If BeforeEach never started AfterEach should be skipped. // Currently some tests under e2e/storage have this condition. if !f.beforeEachStarted { return } </ins> RemoveCleanupAction(f.cleanupHandle) <ins> // This should not happen. Given ClientSet is a public field a test must have updated it! // Error out early before any API calls during cleanup. if f.ClientSet == nil { Failf("The framework ClientSet must not be nil at this point") } </ins> // DeleteNamespace at the very end in defer, to avoid any // expectation failures preventing deleting the namespace. defer func() {

What happened: When running external CSI Driver tests with the Kubernetes E2E: We seem to be getting the following error. From initial investigation it looks like this happens in the during check but only when the test is being skipped. Hunch is that maybe since the test is skipped some resources are never created or they are cleaned up already and they are nil when the is called and expecting them. Kind of low priority since it doesn't make any tests actually fail. /cc /sig storage
see
I think should address this


Question: <rect style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" x="-30" y="416.75" width="364" height="146"/> </g> <g> <del> <rect style="fill: #ffffff" x="418.775" y="551" width="176.225" height="121"/> <rect style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" x="418.775" y="551" width="176.225" height="121"/> <text font-size="12.8" style="fill: #000000;text-anchor:middle;font-family:sans-serif;font-style:normal;font-weight:normal" x="506.887" y="583.4"> <tspan x="506.887" y="583.4">Distributed</tspan> <tspan x="506.887" y="599.4">Watchable</tspan> <tspan x="506.887" y="615.4">Storage</tspan> <tspan x="506.887" y="631.4"></tspan> <tspan x="506.887" y="647.4">(implemented via etcd)</tspan> </del> <ins> <rect style="fill: #ffffff" x="418.774" y="551" width="176.225" height="121"/> <rect style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" x="418.774" y="551" width="176.225" height="121"/> <text font-size="12.8" style="fill: #000000;text-anchor:middle;font-family:sans-serif;font-style:normal;font-weight:normal" x="506.886" y="583.4"> <tspan x="506.886" y="583.4">Distributed</tspan> <tspan x="506.886" y="599.4">Watchable</tspan> <tspan x="506.886" y="615.4">Storage</tspan> <tspan x="506.886" y="631.4"></tspan> <tspan x="506.886" y="647.4">(implemented via etcd)</tspan> </ins> </text> </g> <g>

Architecture picture: says kubecfg is deprecated. The architecture picture should be updated to show kubectl instead of kubecfg.
ref /cc


Question: var map_ResourceSliceList = map[string]string{ "":         "ResourceSliceList is a collection of ResourceSlices.", <del> "listMeta": "Standard list metadata", </del> <ins> "metadata": "Standard list metadata", </ins> "items":    "Items is the list of resource ResourceSlices.", }

Listing s on v1.31.0 returns an object with a field instead of the standard field for (list) objects: Listing s on v1.30.3 behaves correctly: This is also propagated to the Swagger spec (which is how I noticed it, since I maintain the Rust bindings for the client API). v1.31.0: v1.30.3: As explained above. v1.31.0 with feature gate enabled and runtime config . With kind, this is: the API server request as in the above curl command in any way you like as long as it lets you see the raw JSON response. No response <details</details<detailskind 0.23.0 </details<details</details<details</details<details</details<details</details
/sig api-machinery
It seems there may be a discrepancy in the API response format between v1.30.3 and v1.31.0. The object is expected to follow standard Kubernetes API conventions, which typically use the field for list objects.
This deviation might cause compatibility issues for API clients that rely on this structure.
If this deviation from the expected API structure wasn't intentional, it should be aligned with the standard conventions.
The presence of instead of could disrupt client integrations, as you've encountered with the Swagger spec and Rust bindings.
cc
/triage accepted
/assign
this must be fixed to json:"metadata" protobuf:"bytes,1,opt,name=metadata"`` the serialization will not work with standard clients
/priority critical-urgent
Shoot, that's clearly a mistake.


Question: for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { <ins> featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, features.SchedulerQueueingHints, tt.qHintEnabled) </ins> nodeInfo := framework.NewNodeInfo(tt.existingPods...) nodeInfo.SetNode(tt.nodeFn()) preCheckFn := preCheckForNode(nodeInfo) <del> var got []bool </del> <ins> got := make([]bool, 0, len(tt.pods)) </ins> for _, pod := range tt.pods { <del> got = append(got, preCheckFn(pod)) </del> <ins> got = append(got, preCheckFn == nil || preCheckFn(pod)) </ins> } if diff := cmp.Diff(tt.want, got); diff != "" {

In our custom plugin, we want to move pods that is unscheduled by the plugin to activeQ/backoffQ when a new Pod is created. So, we define like this: But, even when a new Pod is created, the Pod that was unscheduled by our custom plugin isn't moved to backoffQ/activeQ. when a new Pod is created, the Pod unscheduled by our custom plugin is moved to backoffQ/activeQ. a custom plugin that has pod events in EventsToRegister. a new Pod that the custom plugin will reject. another new Pod that makes the Pod created in (1) schedulable. can see the Pod created in (1) isn't scheduled immediately (it should've been scheduled immediately after (3) because (3) should move the Pod created in (1) to activeQ/backoffQ.) If I understand correctly and if it's not expected behavior, this bug has a non-small impact, especially in v1.24 because we change to 5 min. v1.23.4 but I think latest master has the same issue. <details</details<details</details<details</details<details</details<details</details
/assign /sig scheduling
Maybe it's only on a newly created unscheduled Pod. We've moved correctly for events related to assigned Pod.
And deleting Node events has the same issue...? I guess we need to run when some Nodes are deleted. In in-tree plugins, only pod topology spread uses the deleting Node event and I guess this bug affects its performance. (Even when some Nodes are deleted, the Pods, that were unscheduled by Pod topology spread, aren't moved to activeQ/backoffQ by that deleting events.)
Hm, it may be a problem that events related to Node and Pod are not always called when an event occurs. (Even if it's intended behavior, at least we need to write this on the comment of .) /retitle scheduler: events in related to Pod and Node don't always work
To summarize: The problem is "not all events related to Pod or Node don't always make move from unschedulableQ to active/backoffQ and it may not be intended for custom-plugin developers because this behavior isn't even documented." I think we should move all unschedulable Pods with all Pod and Node events.


Question: for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { <ins> featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, features.SchedulerQueueingHints, tt.qHintEnabled) </ins> nodeInfo := framework.NewNodeInfo(tt.existingPods...) nodeInfo.SetNode(tt.nodeFn()) preCheckFn := preCheckForNode(nodeInfo) <del> var got []bool </del> <ins> got := make([]bool, 0, len(tt.pods)) </ins> for _, pod := range tt.pods { <del> got = append(got, preCheckFn(pod)) </del> <ins> got = append(got, preCheckFn == nil || preCheckFn(pod)) </ins> } if diff := cmp.Diff(tt.want, got); diff != "" {

If concerned about performance degradation, we can extend or , for example, to have the function like ? This problem has a non-small impact, especially in v1.24+ because we change PodMaxUnschedulableQDuration to 5 min. Let's see the details... ↓ We move the unscheduled Pods (rejected by the plugin that has Pod add/update events in ) in or But, the target Pods against this moving are only unschedulable pods which have any affinity term that matches "pod". So,,,, users cannot move from unschedulableQ to activeQ/backoffQ with all assignedPod events? For example, for Pod Topology Spread (it uses Pod All event in ), the new assigned Pod creation may make some Pods, that are rejected by Pod Topology Spread, schedulable. Regardless of whether that unschedulable Pods have any affinity term that matches a newly created assigned pod or not. Of course this is not just a problem for Pod Topology Spread, it could be a problem for user's custom-plugins. I guess we should pass all unscheduled Pods to . Or at least we should document this behavior. No issue on this. We don't move the unschedulable Pods with those events at all. That's a problem... Even when a plugin uses Pod add/update/delete event in and wants to move the rejected Pods with non-scheduled Pod's add/update/delete events, scheduling queue doesn't move the unschedulable Pods with those events. (In our company, we have the custom plugin for and the plugin wants the scheduler queye to move the Pods rejected by the plugin from unschedulableQ to active/backoffQ when a new non-scheduled Pod is created.) We move only unscheduled Pods that pass . checks the Node can run that unscheduled Pod from the points of view of some in-tree filter plugins like noderesources/nodeport/nodeAffinity/nodename. The problem that arises here is similar to "Assigned Pod's add/update events". For example, for Pod Topology Spread (it uses Node Add event in ), the new Node creation may make some Pods, that are rejected by Pod Topology Spread, schedulable. Regardless of the results of against that Node. (In , we change the skew calculation of Pod Topology Spread to exclude Nodes that don't match the node affinity.


Question: for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { <ins> featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, features.SchedulerQueueingHints, tt.qHintEnabled) </ins> nodeInfo := framework.NewNodeInfo(tt.existingPods...) nodeInfo.SetNode(tt.nodeFn()) preCheckFn := preCheckForNode(nodeInfo) <del> var got []bool </del> <ins> got := make([]bool, 0, len(tt.pods)) </ins> for _, pod := range tt.pods { <del> got = append(got, preCheckFn(pod)) </del> <ins> got = append(got, preCheckFn == nil || preCheckFn(pod)) </ins> } if diff := cmp.Diff(tt.want, got); diff != "" {

But, after is implemented, the calculation can be configured to include those Nodes.) We don't move the unschedulable Pods with Node delete events... That's a problem. Even when the plugin use Node delete event in , scheduling queue doesn't move the unschedulable Pods with any Node delete events.)
/cc Could you please take a look
These are all intended optimizations. These basic checks prevent the scheduler from retrying unschedulable pods over and and over. So far we haven't had the need to make these checks configurable as they check well established scheduling features. +1 on documenting the behavior, but I'm not sure about changing it or making it configurable (that adds complexity that might not be justified). What is it that you are trying to do?
What is it that you are trying to do? We are trying to remove flushing in queue in , right? Without improving like I suggest, some Pods in unschedulableQ cannot move back to activeQ forever in some cases if we remove flushing. (Actually, our company's custom plugin needs to wait flush because non-assigned Pod's events don't let scheduling queue move unschedulable Pods rejected by that plugin.) So, I think we should simply inform all events to queue and let queue move unschedulable Pods with that events. This can happen in some custom plugins like our company's one and also even in in-tree plugins. For example, Pod Topology Spread (after is implemented). When a user uses and , the creation of Node that doesn't match nodeAffinity/nodeSelector may change the scheduling result.
cc who implemented NodeInclusionPolicy. Can you provide a concrete example? The policy just refers to the topology spread calculations. It doesn't mean that the node doesn't have to match the nodeaffinity to be able to accept the pod. I'm asking what is your custom plugin trying to do. Depending on whether it makes sense, we need to evaluate if it's worth the complexity.
For example, for Pod Topology Spread (it uses Pod All event in EventsToRegister), the new assigned Pod creation may make some Pods, that are rejected by Pod Topology Spread, schedulable This is actually a problem here, 5 mins would be too long for background flush. But it doesn't mean that we should simply shorten the interval time.


Question: for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { <ins> featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, features.SchedulerQueueingHints, tt.qHintEnabled) </ins> nodeInfo := framework.NewNodeInfo(tt.existingPods...) nodeInfo.SetNode(tt.nodeFn()) preCheckFn := preCheckForNode(nodeInfo) <del> var got []bool </del> <ins> got := make([]bool, 0, len(tt.pods)) </ins> for _, pod := range tt.pods { <del> got = append(got, preCheckFn(pod)) </del> <ins> got = append(got, preCheckFn == nil || preCheckFn(pod)) </ins> } if diff := cmp.Diff(tt.want, got); diff != "" {

For example, Pod Topology Spread (after is implemented). When a user uses NodeAffinityPolicy: Ignore and WhenUnsatisfiable: DoNotSchedule, the creation of Node that doesn't match nodeAffinity/nodeSelector may change the scheduling result. The skew of a new created node is always 0, so it's still unfit I think.
Our company's custom plugin is like plugin. And like coscheduling's prefilter, our plugin rejects Pods in PreFilter when the number of Pods is less than the defined number. Yes, so, maybe coscheduling plugin has the same issue as our plugin. The coscheduling plugin also uses Pod Add event and I guess it's for non-scheduled Pod's creation. (I don't know much about the coscheduling plugin though.) Oh, that's true Please forget about that example. But, how about like this scenario: user uses and Pod is rejected by Pod Topology Spread because of the skew. e.g. a user uses and TopologyKey is hostname, NodeA/NodeB has 10 Pods each, and NodeC has 8 Pods. adds un-tolerated taint to NodeC skew calculation result should be changed and the Pod rejected in (2) will be allowed by Pod Topology Spread. But the Pod isn't moved from unschedulableQ without flushing because of this issue.
Maybe we can find some pointcuts by the in , e.g. if pod was rejected by PodTopologySpread, we'll when Assigned Pod's add/update events arrived. For customize plugins, we may have to do some refactoring to register the filter func together with the event.
So now NodeC is excluded from calculations. Hence the pod fits in either NodeA or NodeB, which were unchanged. Tricky... I'll have to think about it. We really need those quick checks to pass for performance in most scenarios. /triage accepted
Thanks for bringing this up. Some comments below: You're right. Pods failed by PodTopologySpread may not get re-queued immediately. However, an assigned Pod's add/update event would only impact unschedulable Pods with cross-node scheduling directives. As of today, it's InterPodAffinity and PodTopologySpread.


Question: for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { <ins> featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, features.SchedulerQueueingHints, tt.qHintEnabled) </ins> nodeInfo := framework.NewNodeInfo(tt.existingPods...) nodeInfo.SetNode(tt.nodeFn()) preCheckFn := preCheckForNode(nodeInfo) <del> var got []bool </del> <ins> got := make([]bool, 0, len(tt.pods)) </ins> for _, pod := range tt.pods { <del> got = append(got, preCheckFn(pod)) </del> <ins> got = append(got, preCheckFn == nil || preCheckFn(pod)) </ins> } if diff := cmp.Diff(tt.want, got); diff != "" {

So instead of "pass all unscheduled Pods to movePodsToActiveOrBackoffQueue", I'd like to come up with a solution like: It's pretty rare to move pods depending on Unschedulable Pods' events. In your particular case, maybe a proactive call would behave more efficiently. It doesn't sound correlated, is it? As the check will just pre-verify the basic checks which are orthogonal with PodTopologySpread. In other words, it's impossible that a Pod failed can become schedulable eventually. Updated: The example is a valid point... Can you share some use-cases to move Pods upon node's deletion? In the before, I don't think we have any, until the last release the introduction of feature (as in that case deleting a node would impact the number of domains).
to come up with a solution like: If we do so, also, we need to provide the way to pass some functions to decide which Pods should pass to like for out-of-tree plugins? call would behave more efficiently. Ah~, that's a good idea :) Maybe like this (similar to ) user uses WhenUnsatisfiable: DoNotSchedule. Pod is rejected by Pod Topology Spread because of the skew. e.g. a user uses maxSkew: 1 and TopologyKey is hostname, NodeA/NodeB has 10 Pods each, and NodeC has 8 Pods. deletes NodeC skew calculation result should be changed and the Pod rejected in (2) will be allowed by Pod Topology Spread. But the Pod isn't moved from unschedulableQ without flushing because of this issue. So,,, ignoring some events for performance in scheduler side (without providing a way to change this) could be inconvenient for some plugins. What I thought in the above previous comment () is to define "which kind of resource's events should be ignored" in plugin side. (similar to 's idea? ) We can add pre filter functions in like: Then we can set any pre-filter functions for each plugin. And, we can define some default behavior for PreFilterFn and use them when is empty. So, for example, if most of the plugins are considered to prefer to ignore the event for Node that doesn't match nodeAffinity/nodeSelector, we can define this behavior as the default behavior.


Question: for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { <ins> featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, features.SchedulerQueueingHints, tt.qHintEnabled) </ins> nodeInfo := framework.NewNodeInfo(tt.existingPods...) nodeInfo.SetNode(tt.nodeFn()) preCheckFn := preCheckForNode(nodeInfo) <del> var got []bool </del> <ins> got := make([]bool, 0, len(tt.pods)) </ins> for _, pod := range tt.pods { <del> got = append(got, preCheckFn(pod)) </del> <ins> got = append(got, preCheckFn == nil || preCheckFn(pod)) </ins> } if diff := cmp.Diff(tt.want, got); diff != "" {

What do you think about this idea?
Thanks. It makes sense to me. To summarize, the following cases doesn't properly re-queue relevant pods: Pod's add/update events -scheduled Pod's add/update/delete events 's add/update/delete events For 1 & 2, I've commented above with solutions. (1 needs core changes similar like podAffinity, 2 may leverage the Activate hook) For 3, with 's examples, I'm convinced this is some area we need to improve. One simple and stratightforward solution is to plumb a hook to let the plugin claim "alright, upon Node events, move the pod failed by me anyways". Then once this knob is registered, we move pods failed by that plugin back unconditionally; instead of proceeding with . This hook should be available to in-tree and out-of-tree plugins. In terms of in-tree plugins, it's just PodTopologySpread (we discussed this in today's sig meeting, even InterPodAffinity doesn't fall into this case) So, as a first step, let's ensure we're on the same page and agree on the ^^ summary. Next, we can better evaluate trade-offs and come up with an optimal solution.
Thanks for discussing this topic in the sig meeting. But, I am still believing that all logic related to a plugin should be controllable from the plugin side and we should not create something that can be achieved in in-tree plugins but not in out-of-tree plugins. So... the benefit of the "customizable PreCheck approach" () isn't only to solve this issue. We can use this feature for other optimization: like improving the whole performance of scheduler by moving only Pods that are more likely to be able to be scheduled by defining more detailed events on with customizable PreCheck. Many plugins have a particular scenario that makes rejected Pods schedulable and the current doesn't have an abundant enough ability of expression for this. (So, today many plugins need to say "register this hoge event on foo resource. This event may make rejected Pods schedulable" on comments in . ) Yes, I watched the sig-meeting recording as well and the opinion that the "customizable PreCheck approach" increases some implementation complexity surely makes sense. But,,, isn't this complexity increasing acceptable even if the above is taken into account?