Question: <ins> #!/bin/sh # Copyright 2017 The Kubernetes Authors. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # #     http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # A script encapsulating a common Dockerimage pattern for installing packages # and then cleaning up the unnecessary install artifacts. # e.g. clean-install iptables ebtables conntrack set -o errexit apt-get clean -y rm -rf  /var/cache/debconf/*  /var/lib/apt/lists/*  /var/log/*  /tmp/*  /var/tmp/* </ins>

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: [root EFK]# kubectl logs -f fluentd-es-jrsf7 -n kube-system /var/lib/gems/2.1.0/gems/ffi- block in ffilib': Could not open library '': cannot open shared object file: No such file or directory. (LoadError) Could not open library '': cannot open shared object file: No such file or directory. Could not open library 'libsystemd-': cannot open shared object file: No such file or directory. Could not open library 'libsystemd-': libsystemd- cannot open shared object file: No such file or directory from /var/lib/gems/2.1.0/gems/ffi-map' from /var/lib/gems/2.1.0/gems/ffi- ffilib' from /var/lib/gems/2.1.0/gems/systemd-journal-<module:Native' from /var/lib/gems/2.1.0/gems/systemd-journal- <module:Id128' from /var/lib/gems/2.1.0/gems/systemd-journal-<module:Systemd' from /var/lib/gems/2.1.0/gems/systemd-journal- <top (required)' fromrequire' from require' from /var/lib/gems/2.1.0/gems/systemd-journal-<top (required)' from require' fromrequire' from /var/lib/gems/2.1.0/gems/systemd-journal- <top (required)' fromrequire' from rescue in require' fromrequire' from /var/lib/gems/2.1.0/gems/fluent-plugin-systemd- <top (required)' fromrequire' from require' from /var/lib/gems/2.1.0/gems/fluentd-block in tryloadplugin' from /var/lib/gems/2.1.0/gems/fluentd- each' from /var/lib/gems/2.1.0/gems/fluentd-tryloadplugin' from /var/lib/gems/2.1.


Question: <ins> #!/bin/sh # Copyright 2017 The Kubernetes Authors. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # #     http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # A script encapsulating a common Dockerimage pattern for installing packages # and then cleaning up the unnecessary install artifacts. # e.g. clean-install iptables ebtables conntrack set -o errexit apt-get clean -y rm -rf  /var/cache/debconf/*  /var/lib/apt/lists/*  /var/log/*  /tmp/*  /var/tmp/* </ins>

0/gems/fluentd- newimpl' from /var/lib/gems/2.1.0/gems/fluentd-newinput' from /var/lib/gems/2.1.0/gems/fluentd- addsource' from /var/lib/gems/2.1.0/gems/fluentd-block in configure' from /var/lib/gems/2.1.0/gems/fluentd- each' from /var/lib/gems/2.1.0/gems/fluentd-configure' from /var/lib/gems/2.1.0/gems/fluentd- configure' from /var/lib/gems/2.1.0/gems/fluentd-runconfigure' from /var/lib/gems/2.1.0/gems/fluentd- runconfigure' from /var/lib/gems/2.1.0/gems/fluentd-block in start' from /var/lib/gems/2.1.0/gems/fluentd- call' from /var/lib/gems/2.1.0/gems/fluentd-mainprocess' from /var/lib/gems/2.1.0/gems/fluentd- start' from /var/lib/gems/2.1.0/gems/fluentd-<top (required)' from require' fromrequire' from /var/lib/gems/2.1.0/gems/fluentd-0.12.39/bin/fluentd:5:in <top (required)' from /usr/local/bin/fluentd:23:inload' from /usr/local/bin/fluentd:23:in <main' What you expected to happen: fluentd-es run success How to reproduce it (as minimally and precisely as possible): kubectl create -f fluentd-es- the image is google-containers/fluentd-elasticsearch:v2.0.0 Anything else we need to know?


Question: <ins> #!/bin/sh # Copyright 2017 The Kubernetes Authors. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # #     http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # A script encapsulating a common Dockerimage pattern for installing packages # and then cleaning up the unnecessary install artifacts. # e.g. clean-install iptables ebtables conntrack set -o errexit apt-get clean -y rm -rf  /var/cache/debconf/*  /var/lib/apt/lists/*  /var/log/*  /tmp/*  /var/tmp/* </ins>

: Environment: Kubernetes version (use ):Client Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-07-21T08:23:22Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-07-21T08:08:00Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration**: OS (e.g. from /etc/os-release):centos 7.3 Kernel (e.g. ):Linux node-21.46 3.10.0-514.26.2.el7.x8664 SMP Tue Jul 4 15:04:05 UTC 2017 x8664 x8664 x86_64 GNU/Linux Install tools: Others:
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
Yup, the situation with fluentd on the system with systemd is very sloppy. There's a hack to make it works sometimes, but it seems that it doesn't work. I'll look into it
/sig instrumentation /assign
Facing same issue.


Question: <ins> #!/bin/sh # Copyright 2017 The Kubernetes Authors. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # #     http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # A script encapsulating a common Dockerimage pattern for installing packages # and then cleaning up the unnecessary install artifacts. # e.g. clean-install iptables ebtables conntrack set -o errexit apt-get clean -y rm -rf  /var/cache/debconf/*  /var/lib/apt/lists/*  /var/log/*  /tmp/*  /var/tmp/* </ins>

Environment Details:kubectl version Client Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-07-03T15:31:10Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-07-03T15:31:10Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"} OS (e.g. from /etc/os-release):Centos 7 (Core) Kernel (e.g. uname -a): Linux xxxx 3.10.0-327.el7.x8664 Error for reference /var/lib/gems/2.1.0/gems/ffi- block in ffilib': Could not open library '': cannot open shared object file: No such file or directory. (LoadError) Could not open library '': cannot open shared object file: No such file or directory. Could not open library 'libsystemd-': cannot open shared object file: No such file or directory. Could not open library 'libsystemd-': libsystemd- cannot open shared object file: No such file or directory from /var/lib/gems/2.1.0/gems/ffi-map' from /var/lib/gems/2.1.0/gems/ffi- ffilib' from /var/lib/gems/2.1.0/gems/systemd-journal-<module:Native' from /var/lib/gems/2.1.0/gems/systemd-journal- <module:Id128' from /var/lib/gems/2.1.0/gems/systemd-journal-<module:Systemd' from /var/lib/gems/2.1.0/gems/systemd-journal- <top (required)' fromrequire' from require' from /var/lib/gems/2.1.0/gems/systemd-journal-<top (required)' from require' fromrequire' from /var/lib/gems/2.1.


Question: <ins> #!/bin/sh # Copyright 2017 The Kubernetes Authors. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # #     http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # A script encapsulating a common Dockerimage pattern for installing packages # and then cleaning up the unnecessary install artifacts. # e.g. clean-install iptables ebtables conntrack set -o errexit apt-get clean -y rm -rf  /var/cache/debconf/*  /var/lib/apt/lists/*  /var/log/*  /tmp/*  /var/tmp/* </ins>

0/gems/systemd-journal- <top (required)' fromrequire' from rescue in require' fromrequire' from /var/lib/gems/2.1.0/gems/fluent-plugin-systemd- <top (required)' fromrequire' from require' from /var/lib/gems/2.1.0/gems/fluentd-block in tryloadplugin' from /var/lib/gems/2.1.0/gems/fluentd- each' from /var/lib/gems/2.1.0/gems/fluentd-tryloadplugin' from /var/lib/gems/2.1.0/gems/fluentd- newimpl' from /var/lib/gems/2.1.0/gems/fluentd-newinput' from /var/lib/gems/2.1.0/gems/fluentd- addsource' from /var/lib/gems/2.1.0/gems/fluentd-block in configure' from /var/lib/gems/2.1.0/gems/fluentd- each' from /var/lib/gems/2.1.0/gems/fluentd-configure' from /var/lib/gems/2.1.0/gems/fluentd- configure' from /var/lib/gems/2.1.0/gems/fluentd-runconfigure' from /var/lib/gems/2.1.0/gems/fluentd- runconfigure' from /var/lib/gems/2.1.0/gems/fluentd-block in start' from /var/lib/gems/2.1.0/gems/fluentd- call' from /var/lib/gems/2.1.0/gems/fluentd-mainprocess' from /var/lib/gems/2.1.0/gems/fluentd- start' from /var/lib/gems/2.1.0/gems/fluentd-<top (required)' from require' fromrequire' from /var/lib/gems/2.1.0/gems/fluentd-0.12.39/bin/fluentd:5:in <top (required)' from /usr/local/bin/fluentd:23:inload' from /usr/local/bin/fluentd:23:in <


Question: <ins> #!/bin/sh # Copyright 2017 The Kubernetes Authors. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # #     http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # A script encapsulating a common Dockerimage pattern for installing packages # and then cleaning up the unnecessary install artifacts. # e.g. clean-install iptables ebtables conntrack set -o errexit apt-get clean -y rm -rf  /var/cache/debconf/*  /var/lib/apt/lists/*  /var/log/*  /tmp/*  /var/tmp/* </ins>

main'
You can solve it temporary by removing I'm working on the solution
A solution here might be to base fluentd-es off an image with journald supporting XZ and LZ4 compression. I'm looking :)
Thanks


Question: defaultIntOrString := intstr.FromInt(1) differentIntOrString := intstr.FromInt(5) period := int64(v1.DefaultTerminationGracePeriodSeconds) <del> enableServiceLinks := v1.DefaultEnableServiceLinks </del> defaultTemplate := v1.PodTemplateSpec{ Spec: v1.PodSpec{ DNSPolicy:                     v1.DNSClusterFirst,

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug The change in a new field () to the Pod spec that is set to true by default. This is causing problems for DaemonSets in some situations. DaemonSets are using ControllerRevisions to keep track of previous configurations and each revision will contain among other things the Pod spec for the DaemonSet. If a DaemonSet is created with the enableServiceLinks property is set to true, the ControllerRevision will also contain this field. If the cluster is rolled back to a previous version of kubernetes, the DeamonSet struct does not have this field and so the ControllerRevision and the DaemonSet will no longer match. This leads to a new ControllerRevision being created and a rolling update of all pods in the DaemonSet. Would it be a better solution to rename the field to and enable service links whenever the field is not set? This would fix the problem with the DaemonSets unless the field is explicitly set before the rollback. This is causing the master upgrade tests to fail as reported in /sig apps /cc /cc /cc
I guess the change in podSpec and the defaulting will break all the workload resources as they are based on comparing the template's podSpec. ~Is the new value defaulted to true also for the DaemonSet's podSpec template? If not, controller might go hot loop like with mutating admission controllers.~ I don't like us making such changes to v1 podSpec. The other way around seems more sensible. /cc
Why is this a bad thing? We will continue to add fields to podSpec - the issue isn't the defaulting (this would happen regardless of which way it defaulted), it's that the controller can't really be version aware. The new value would default to true in the daemonset as well.
I'm not sure that a rollback requiring daemonsets to recreate is a bug. I think that's potentially "the best possible thing that we can do in that case".
/cc SIG Release blocking dashboard with upgrade test failures - https://k8s-
So the problem isn't the fact that a field has been . It the way in which the field was .


Question: defaultIntOrString := intstr.FromInt(1) differentIntOrString := intstr.FromInt(5) period := int64(v1.DefaultTerminationGracePeriodSeconds) <del> enableServiceLinks := v1.DefaultEnableServiceLinks </del> defaultTemplate := v1.PodTemplateSpec{ Spec: v1.PodSpec{ DNSPolicy:                     v1.DNSClusterFirst,

enableServiceLinks is bool pointer that defaults to true. To be backward compatible it should be something like where the new default behavior is to assume that service links are enabled, and the field should be set on an opt out basis to disable the service links. compatiblity As it is if you create a new DaemonSet and then roll back cluster to a prior version, or if you have a failover in a HA setup, you can trigger a simultaneous rolling restart of all DaemonSets you created even though you didn't explicitly configure (and may not even be aware of) the new feature. If we respect backward compatibility and make the feature explicitly opt-out (that is the semantics we are going for here in either ca So the problem isn't the fact that a field has been . It the way in which the field was . enableServiceLinks is bool pointer that defaults to true. To be backward compatible it should be something like where the new default behavior is to assume that service links are enabled, and the field should be set on an opt out basis to disable the service links. compatiblity As it is if you create a new DaemonSet and then roll back cluster to a prior version, or if you have a failover in a HA setup, you can trigger a simultaneous rolling restart of all DaemonSets you created even though you didn't explicitly configure (and may not even be aware of) the new feature. If we respect backward compatibility and make the feature explicitly opt-out (that is the semantics we are going for here in either case), you won't trigger the rolling restart on rollback or controller manager failoverse), you won't trigger the rolling restart on rollback or controller manager failover unless you explicitly opt out of the behavior. I don't see why the former behavior is more desirable than the latter.
For background/context, the choice between enable/disable was initially discussed here:
Are you saying that the issue is that default value is not the same as the Go "zero value" ? The concept of zero value is interesting but it forces APIs to bend into unnatural shapes to respect it. Since forever Kubernetes has explicitly said that we have version-specific default values which ARE NOT governed by zero values, but instead are governed by the API version. It is safe to assume that if a field is documented as having a default in a version, you can rely on that field being populated with that default, with a few exceptions.


Question: defaultIntOrString := intstr.FromInt(1) differentIntOrString := intstr.FromInt(5) period := int64(v1.DefaultTerminationGracePeriodSeconds) <del> enableServiceLinks := v1.DefaultEnableServiceLinks </del> defaultTemplate := v1.PodTemplateSpec{ Spec: v1.PodSpec{ DNSPolicy:                     v1.DNSClusterFirst,

We generally ask for boolean fields to be "in the affirmative direction", as in this case. Now, we could revisit this. It would be a significant change in direction, but it's not impossible. But I don't think that solves your problem in general -- if this field were optional string, with a default value of "Hello there", you would be in the same boat right?
So the underlying issue is that we are defaulting into the PodSpec which gets adopted into all of the PodTemplateSpecs of every object. That sort of defaults the users declared intent. Why not just default the value into the Pod itself . Is the intention really to default this in all Templates or to make it optional in the Template and default it in the Pod itself?
/milestone v1.13
I can't speak to the ideology of whether to default the field in all PodSpecs or just the Pod, but your suggestion makes sense to me. I'm not sure of the past reasoning for adding other defaulted fields, most recently SchedulerName in Jan. 2017 (), to all PodSpecs instead of just to the Pod object. Did we not have these same tests in place then? This would effectively be saying we should never add new defaulted fields to PodSpec. If this is the plan, I can create a PR for the fix and I'll add a big warning comment to SetDefaults_PodSpec.
As an example of where we don't default the template we have a CronJob that embeds JobTemplate, we never default that template, only an actual Job gets defaulting. I agree with that not defaulting any templates does make sense.
Agree to only default this in pods. Agree to adding the comment. Most important defaulting happens in pods to keep higher level pod specs simpler. We may need to raise the general pattern in a discussion with api review in general, since this is a change from previous strategy, but the previous strategy was not explicitly chosen (to my recollection at least)
Hi all do we have an agreed path forward for fixing this issue in 1.13? if so can we have an ETA for the fix to land? the release team would like for the tests to get passing in the blocking suites in an effort to unmask any other failures that might show up once these are fixed.


Question: defaultIntOrString := intstr.FromInt(1) differentIntOrString := intstr.FromInt(5) period := int64(v1.DefaultTerminationGracePeriodSeconds) <del> enableServiceLinks := v1.DefaultEnableServiceLinks </del> defaultTemplate := v1.PodTemplateSpec{ Spec: v1.PodSpec{ DNSPolicy:                     v1.DNSClusterFirst,

/assign
We have a PR with a fix that will hopefully merge today or tomorrow. I will keep an eye on the dashboard to make sure it fixes the failing daemonset tests.
Thanks Can we link the PR here plz. /cc
Reopening, because of the auto-close. Tests are not passing yet. /reopen
Reopening this issue. <detailsIn response to : Reopening, because of the auto-close. Tests are not passing yet. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
PR that should (hopefully) close this issue is here:
DaemonSet tests passed on run 1842, seems to have been fixed: https://k8s- if it doesn't pop up again - /close
Closing this issue. <detailsIn response to : DaemonSet tests passed on run 1842, seems to have been fixed: https://k8s- Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: color: bfe5bf - name: do-not-merge color: e11d21 <ins> - name: do-not-merge/work-in-progress color: e11d21 - name: do-not-merge/hold color: e11d21 - name: do-not-merge/cherry-pick-not-approved color: e11d21 - name: do-not-merge/release-note-label-needed color: e11d21 - name: do-not-merge/blocked-paths color: e11d21 </ins> - name: flake-has-meta color: fbca04 - name: for-new-contributors

We should solve this problem in a better way
cc :))
/sig contributor-experience release
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale /lifecycle frozen
/assign


Question: }, ) <del> // Setup cache comparer </del> <ins> // Setup cache debugger </ins> debugger := cachedebugger.New( args.NodeInformer.Lister(), args.PodInformer.Lister(), c.schedulerCache, c.podQueue, ) <del> ch := make(chan os.Signal, 1) signal.Notify(ch, compareSignal) </del> <ins> debugger.ListenForSignal(c.StopEverything) </ins> go func() { <del> for { select { case <-c.StopEverything: c.podQueue.Close() return case <-ch: debugger.Comparer.Compare() debugger.Dumper.DumpAll() } } </del> <ins> <-c.StopEverything c.podQueue.Close() </ins> }() return c

PLEASE COORDINATE WITH SIG-SCHEDULING BEFORE WORKING ON THIS ISSUE. We want to avoid the situation where two people accidentally start work on the same fix. Main tracking issue: Currently, the feature is set up at the end of in . This setup should be moved into its own helper function, preferably in the package. /sig scheduling /kind cleanup /help
/assign /remove-help
I'd like to take this one
/remove-help
the PR I sent should resolve this one. :disappointed: Are any of the others still available?


Question: t.Fatalf("expect TCP connection: %d, actual: %d", tc.expectConnections, len(localAddr)) } <del> // check if watch request is broken by GOAWAY response </del> <ins> // check if watch request is broken by GOAWAY frame </ins> watchTimeout := time.NewTimer(time.Second * 10) <ins> defer watchTimeout.Stop() </ins> for _, watchCh := range watchChs { select { <del> case n := <-watchCh: if n != watchExpectSendBytes { t.Fatalf("in-flight watch was broken by GOAWAY response, expect go bytes: %d, actual got: %d", watchExpectSendBytes, n) </del> <ins> case watchResp := <-watchCh: if watchResp.err != nil { t.Fatalf("watch response got an unexepct error: %v", watchResp.err) } if !reflect.DeepEqual(responseBody, watchResp.body) { t.Fatalf("in-flight watch was broken by GOAWAY frame, expect response body: %s, got: %s", responseBody, watchResp.body) </ins> } case <-watchTimeout.C: t.Error("watch receive timeout")

What happened: We enabled feature in our large-scale cluster, and we found client(Kubelet) may got error(500 http code response) like after received GOAWAY frame. I had already re-produce this bug and open an in golang repo. So far I think this bug only occur with a small probability when client request server in high concurrency use a same httpclient instance, and it's a golang sdk bug in http client. Before I solved this issue, I suggest do NOT enable GOAWAY() feature. As it's a golang sdk bug(currently I known and doubt) in http client, revert GOAWAY feature(codes) is not necessary.
cc
/sig api-machinery
Thanks for the report.
Based on not setting the field, I think it defaults that way, so that's good. We will leave the option and filter so that when golang updates its client, we will have a simple way to turn it on and execute it.
Sure. I am trying to fix the golang http client issue and will update progress at . After golang http client issue fixed, I will add more concurrent test cases to see if there are any other bugs may occur by GOAWAY feature. BTW, we still enable GOAWAY feature in our production cluster, because there is no any evidence told us this error will influence SLO, as all most components(kubelet, controller-manager, scheduler) is controller work mode and will re-try after got error. /assign
Golang version or required, which will release at 07/01/2020.
Kubernetes v1.19(or master branch) now is using golang 1.15, and golang 1.15 has fixed the issue. And is on going. Let me add more test cases to ensure the error will not happen again.
I think we should re-open this issue because of . /re-open


Question: return w.Result(), bytes, decoded } <ins> // completerWorkqueue is a workqueue.RateLimitingInterface that implements // isComplete type completerWorkqueue struct { lock sync.Mutex workqueue.RateLimitingInterface processing map[interface{}]struct{} } var _ = workqueue.RateLimitingInterface(&completerWorkqueue{}) func newCompleterWorkqueue(wq workqueue.RateLimitingInterface) *completerWorkqueue { return &completerWorkqueue{ RateLimitingInterface: wq, processing:            make(map[interface{}]struct{}), } } func (q *completerWorkqueue) Add(item interface{}) { q.lock.Lock() defer q.lock.Unlock() q.processing[item] = struct{}{} q.RateLimitingInterface.Add(item) } func (q *completerWorkqueue) AddAfter(item interface{}, duration time.Duration) { q.Add(item) } func (q *completerWorkqueue) AddRateLimited(item interface{}) { q.Add(item) } func (q *completerWorkqueue) Done(item interface{}) { q.lock.Lock() defer q.lock.Unlock() delete(q.processing, item) q.RateLimitingInterface.Done(item) } func (q *completerWorkqueue) isComplete() bool { q.lock.Lock() defer q.lock.Unlock() return q.Len() == 0 && len(q.processing) == 0 } </ins>

pull-kubernetes-unit sometimes fails with data race. TestDirty I think the test was 3 months ago with the flaking state. There are two issues. First, updating and reading a bool-type variable is not thread-safe. and there is also another problem checking the state before completely syncing it. No response /sig api-machinery
/assign
/triage accepted


Question: . "github.com/onsi/ginkgo" ) <ins> const maxNodes = 100 </ins> var _ = framework.KubeDescribe("SSH", func() { f := framework.NewDefaultFramework("ssh")

On 2k nodes, this test took 53m. We should either do it in batches of nodes, or just subsample the nodes.
And it probably doesn't need to test 4 different variant command sets against all of the nodes, either. :P
When you asked me about the 2k node test, this was the test that was running. I'm not sure why other tests weren't also running in parallel or if this was just the final long pole, but it's a beastly test. PR incoming.
- thanks for looking into it. THe PR is already lgtm-ed by me.


Question: p.unschedulablePods.addOrUpdate(pInfo) return false, nil } <ins> if pInfo.InitialAttemptTimestamp == nil { now := p.clock.Now() pInfo.InitialAttemptTimestamp = &now } </ins> if err := p.activeQ.Add(pInfo); err != nil { logger.Error(err, "Error adding pod to the active queue", "pod", klog.KObj(pInfo.Pod)) return false, err

is recording the time that a Pod is gated. We use the timestamp when the scheduler inserts the pod into the queue: The period of time when a Pod fails PreEnque (like being gated) shouldn't be accounted in the podschedulingdurationseconds. Create a Pod with scheduling gates. Wait some time before removing the gate. Observe the metric No response <details</details<detailsAny </details<details</details<details</details<details</details<details</details
/sig scheduling
/help /triage accepted
This request has been marked as needing help from a contributor. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help /triage accepted Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
would you want to take a stab on this issue?
Yup, I can give it a shot. /assign helayoty
, as we discussed, we can address this issue with different solutions: the pod timestamp to the current time , so the gated time is substituted. timestamp . Use a pointer to the InitialAttemptTimestamp and interpret as the newest timestamp so the pod will go to the back of the queue. I'll go ahead and implement solution no. 3, wdyt?
sgtm. Any concerns ?
I think it should be option 1 + 3, and pInfo.Timestamp should be pInfo.InitialAttemptTimestamp, right? as records the time based on :
I think we should revisit this issue, in terms of its assumption: So is it reasonable to always exclude time spent in PreEnqueue from ? Note that PreEnqueue != SchedulingGates, is one of the PreEnqueue's usecases.


Question: p.unschedulablePods.addOrUpdate(pInfo) return false, nil } <ins> if pInfo.InitialAttemptTimestamp == nil { now := p.clock.Now() pInfo.InitialAttemptTimestamp = &now } </ins> if err := p.activeQ.Add(pInfo); err != nil { logger.Error(err, "Error adding pod to the active queue", "pod", klog.KObj(pInfo.Pod)) return false, err

In other words, some user of PreEnqueue may want to include the time spent in PreEnqueue.
is a signal that cluster providers can use to build SLOs. The time spent before passing PreEnqueue is not under control of kube-scheduler, so considering it would not allow to build an SLO. Are you aware of other users of PreEnqueue?
True, but comparing to state that metric X can or cannot be used to build SLO (which is hard IMO), I'm in favor of giving suggestion on SLI/SLO's semantic definition, for example: the official SLI definition of has a footnote clearly explaining its definition: So if a pod is gated, it's not "immediately schedulable", and hence doesn't fall into the definition of podstartup_latency.
A viable solution might be providing an extra label like to explicitly exclude the time spent on preenqueue.
That is true, but there is no way today to exclude those pods from the metric, or separate the time. Maybe we need an extra metric, but I think the new metric could be the one that includes the time guarded in PreEnqueue. It should probably be part of the KEP. The old metric is already stable, so we can't add labels. And it might already be in use for SLOs.
cc
I think we should be working on that. And treat it as part of GA requirement for PodSchedulingReadiness.
An edge(maybe not edge) case is a pod's scheduling attempts is 0 but the is a long time, so prefer not to account for the waiting time when gated. For users not using PreEnqueue, they will not be impacted, but for users using PreEnqueue, it's a break change for it increases the duration time. And we can add a new metric like to measure how long a pod stays in a scheduler.
I thought of "attempts", it's just I'm not sure it's a good idea to rely on "attempts" to define a SLI/SLO, b/c "attempts" is obtained from the in-memory object and in a HA config env, it's only updated by the primary scheduler instance. In other words, the value is inaccurate when the lease owner switches.
attempt=0 is good enough for most scenarios
do we have a conclusion?


Question: p.unschedulablePods.addOrUpdate(pInfo) return false, nil } <ins> if pInfo.InitialAttemptTimestamp == nil { now := p.clock.Now() pInfo.InitialAttemptTimestamp = &now } </ins> if err := p.activeQ.Add(pInfo); err != nil { logger.Error(err, "Error adding pod to the active queue", "pod", klog.KObj(pInfo.Pod)) return false, err

Should we move on with the solution or remove this issue and the linked PR from the current release?
I would like to move forward with this proposal. ?
To move it forward, I think we should add new metric(s) to explicitly tell it's including or excluding the time spent on PreEnqueue. Given is in stable level: We will need a new metric like "podschedulingexcludingpreenqueuedurationseconds".
28 Bug Triage Shadow here. Just checking-in, if this is still on track for the 1.28 release?
We don't have a conclusion yet, but as a bug, it should qualify for changes after the freezes.


Question: resp, err := http.Post(server.URL+"/apis/company.com/"+version+"/namespaces/default/foos", "application/json", bytes.NewBuffer(data)) if !assert.NoError(err) { <del> t.Errorf("unexpected error: %v", err) return </del> <ins> t.Fatalf("unexpected error: %v", err) </ins> } assert.Equal(http.StatusCreated, resp.StatusCode)

A user report that it's broken in the slack channel:
Also and -- this probably broke with one of our recent refactors. The server should be accepting list options from any version, I thought?
FWIW, I think it's this line barf at the ListOptions: The parameter codec first decodes the query parameter to a versioned object and then convert it to the internal object. The code fails because it cannot create a "" ListOptions.
The server should accept list options in the API version it is sent. ListOptions should be registered in all third party groups.
The thirdPartyResourceDataCreator should handle "ListOptions", which surprises me that it doesn't then respect the group.
I think we should add a wrapper around the api.ParameterCodec here: Do something similar as what does.
Putting together a PR now and will let you guys review.
I guess my point is, the list options are just there from the URL query parameters. The user didn't send a group. Why do we assume that value for the group rather than "/v1" or something?
My understanding is we want to allow options to evolve differently in different groups. We do have v1.ListOptions and v1beta1.ListOptions. And if a client sends requeset to extensions/v1beta1 endpoint, then it's expected to encode the ListOptions to v1beta1 and then convert it to query parameters.
Mostly because the parameter codec is supposed to abstract that. "virtual" groups need to be implemented by the virtual "parameter codec". The statement on floating server objects (List, Delete etc) was that they have a locked API schema, and we have a mechanism to do that via the scheme which is registering them in a group. , Daniel Smith wrote:


Question: func (d dockerOpt) GetKV() (string, string) { return d.key, d.value } <ins> // sharedWriteLimiter limits the total output written across one or more streams. type sharedWriteLimiter struct { delegate io.Writer limit    *int64 } func (w sharedWriteLimiter) Write(p []byte) (int, error) { if len(p) == 0 { return 0, nil } limit := atomic.LoadInt64(w.limit) if limit <= 0 { return 0, errMaximumWrite } var truncated bool if limit < int64(len(p)) { p = p[0:limit] truncated = true } n, err := w.delegate.Write(p) if n > 0 { atomic.AddInt64(w.limit, -1*int64(n)) } if err == nil && truncated { err = errMaximumWrite } return n, err } func sharedLimitWriter(w io.Writer, limit *int64) io.Writer { if w == nil { return nil } return &sharedWriteLimiter{ delegate: w, limit:    limit, } } var errMaximumWrite = errors.New("maximum write") </ins>

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Conformance test failed: xref What you expected to happen: I expected this conformance test to pass. How to reproduce it (as minimally and precisely as possible): We have configured docker's logging driver to use journald, and then ran sonobouy tests against a k8s 1.16.3 cluster. Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: OS (e.g: ): Red Hat Enterprise Linux Server release 7.7 (Maipo) Kernel (e.g. ): 5.3.11- SMP Tue Nov 12 14:45:59 EST 2019 x8664 x8664 x8664 GNU/Linux Install tools: Network plugin and version (if this is a network-related bug): Others:
There are no sig labels on this issue. Please add a sig label by either: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. See the . The in method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/sig cli
/assign
I have a working patch that fixes this, will push a PR shortly.


Question: func TestNewBuilder(t *testing.T) { assert := assert.New(t) <del> plugMgr := newInitializedVolumePlugMgr() </del> <ins> plugMgr, _ := newInitializedVolumePlugMgr(t) </ins> plug, err := plugMgr.FindPluginByName(pluginName) assert.NoError(err)

Unit tests on our CI have apparently been failing for 2 weeks and no one noticed (oops!)... The PR that seems to have caused the failures is . The failing test is - . Full Test Log: I'm not sure why this is failing on our CI and not on shippable. Has anyone else seen something similar? That test only seems to fail on the second run through with . Our unit tests run in a container built specifically for testing k8s-mesos: The Dockerfile for the env: Is anyone else able to repro with the above env or their own?
/cc
that test was around a week ago, it's doesn't seem possible to me that it was failing since 2 weeks ago. Anyway, we have few duplications of this issue: this one And the PR where is working on: It's a little bit difficult to me to follow all the conversations in all the differente places about the same bug, could it be possible to unificate some? Since you provided the environment I will try and take a look to reproduce it locally. Thanks!
I think I've got a working patch in
Closing in favor of &


Question: func TestNewBuilder(t *testing.T) { assert := assert.New(t) <del> plugMgr := newInitializedVolumePlugMgr() </del> <ins> plugMgr, _ := newInitializedVolumePlugMgr(t) </ins> plug, err := plugMgr.FindPluginByName(pluginName) assert.NoError(err)

reports the following failure after syncing with latest code: --- FAIL: TestSetUpAtInternal (0.00s) Error Trace: Error: Not equal: "expected-to-be-set-properly" (expected) != "" (actual) FAIL FAIL 0.067s cc/
CC
Taking a look. What do you mean with failed sometimes? Is it a random failure?
Can you please provide more information. This test seems to be consistently green in our CI runs.
The test is failing on this assertion: but as you can see on the previous line I am checking that there was no error on that call. So the function is returning for the error checking but it's not setting . Only place where it can happen is here: Is the not been clean between runs?
When I said "the only place where it can happen" it was because the other s are being returned after a check. I think that there is no other way to get there.
PS: I run the test suite 100 times and I didn't get any error: (That branch is based on the branch)
can you test to see if the patch in fixes things for you?
See for reproducible test environment and failure logs.
Based on the solution in it sounds like this test creates files the first time through which causes the tests to fail on the second time through.


Question: // EndpointPort is a tuple that describes a single port. type EndpointPort struct { <del> // The name of this port (corresponds to ServicePort.Name). </del> <ins> // The name of this port.  This must match the 'name' field in the // corresponding ServicePort. </ins> // Must be a DNS_LABEL. // Optional only if one port is defined. // +optional

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: When running a cluster service and manually created endpoints, the port name set on the cluster definition acts as an implicit selector - suitable end-points do not get mapped to the cluster service. What you expected to happen: Port name shouldn't be an implicit selector. How to reproduce it (as minimally and precisely as possible): Using the following spec file "": kubectl create -f , then describe the service. Output shows the endpoint as being correctly mapped Now run kubectl delete -f the and uncomment the name: foobar Re-run the kubectl create -f command. Now, checking the service gives the output below (end point isn't mapped): Anything else we need to know?: Adding the name: foobar to the endpoint definition makes the end point to be mapped again into the cluster. I.e. there is an implicit matching of the end-point and cluster port definitions. This is either a bug or a very surprising and undocumented feature Environment: kubectl version Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.7", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-03-25T02:52:13Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.5", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-01-16T18:14:49Z", GoVersion:"go1.10.7", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: OS (e.g: ): root cat /etc/os-release NAME="Ubuntu" VERSION="18.04.1 LTS (Bionic Beaver)" Kernel (e.g.


Question: // EndpointPort is a tuple that describes a single port. type EndpointPort struct { <del> // The name of this port (corresponds to ServicePort.Name). </del> <ins> // The name of this port.  This must match the 'name' field in the // corresponding ServicePort. </ins> // Must be a DNS_LABEL. // Optional only if one port is defined. // +optional

): uname -a Linux 4.15.0-51-generic -Ubuntu SMP Wed May 15 14:27:21 UTC 2019 x8664 x8664 x86_64 GNU/Linux Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig api-machinery
/remove-sig api-machinery /sig network The name mapping to the endpoints is intentional, but needs documentation
/triage unresolved
/assign
It may be under-documented, but it is not UNdocmented.
Also
working on a PR to fix doc


Question: place for the pod, like this: ```console <del> $ kubectl describe pods/frontend | grep -A 3 Events </del> <ins> $ kubectl describe pod frontend | grep -A 3 Events </ins> Events: <del> FirstSeen				LastSeen			Count	From SubobjectPath	Reason			Message Tue, 30 Jun 2015 09:01:41 -0700	Tue, 30 Jun 2015 09:39:27 -0700	128	{scheduler }            failedScheduling	Error scheduling: For each of these fitness predicates, pod frontend failed on at least one node: PodFitsResources. </del> <ins> FirstSeen	LastSeen	 Count	From          Subobject   PathReason			Message 36s		5s		 6	    {scheduler }              FailedScheduling	Failed for reason PodExceedsFreeCPU and possibly others </ins> ``` <del> If a pod or pods are pending with this message, then there are several things to try: </del> <ins> In the case shown above, the pod "frontend" fails to be scheduled due to insufficient CPU resource on the node. Similar error messages can also suggest failure due to insufficient memory (PodExceedsFreeMemory). In general, if a pod or pods are pending with this message and alike, then there are several things to try: </ins> - Add more nodes to the cluster. - Terminate unneeded pods to make room for pending pods. - Check that the pod is not larger than all the nodes.  For example, if all the nodes

did which addresses some of them, but turns up a lot more.
Alright, I will do that.


Question: framework.ExpectNoError(err) } <del> numaNodes, cpusNumPerNUMA = hostCheck() </del> <ins> numaNodes, coresNumPerNUMA, threadsPerCore = hostCheck() cpusNumPerNUMA = coresNumPerNUMA * threadsPerCore </ins> // It is safe to assume that the CPUs are distributed equally across // NUMA nodes and therefore number of CPUs on all NUMA nodes are same // so we just check the CPUs on the first NUMA node framework.Logf("numaNodes on the system %d", numaNodes) <ins> framework.Logf("Cores per NUMA on the system %d", coresNumPerNUMA) framework.Logf("Threads per Core on the system %d", threadsPerCore) </ins> framework.Logf("CPUs per NUMA on the system %d", cpusNumPerNUMA) policy := topologymanager.PolicySingleNumaNode

/kind failing-test <!-- If this is a flake, please add: /kind flake --/sig node started to fail since 09-13.
/cc
/assign
/cc
/triage accepted /priority important-soon it's indeed failing too often
/reopen The test is still failing: Now timing out at 120s i.e. 2 minutes as we increased the default timeout . We need to understand the root cause and get to the bottom of this.
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Snippet from the build log () corresponding from this failure We see from the above logs that: node has two numa nodes, 8 cpus per NUMA so the root cause of the failure is not because the tests are running on a node that doesn't have multi-numa support. is started and the health check fails a couple of times but eventually succeeds. pod is created with 9 cpus (cpusperNUMA +1 i.e. 8+1) which is expected to fail with Topology Affinity Error as we cannot allocate resources for such a pod from a single NUMAnode. We fail to get matching metrics. It is not clear if the check fails because we don't get kubelet metrics at all or the expected metrics but based on point 6 below it seems like it is because of latter. We can add additional logs for more clarity here. each is executed and Kubelet is restarted and health check is done. are collected from the namespace corresponding to the test pod which shows that the pod started even though it was expected to fail with Topology Affinity Error. We need to understand why the pod got successfully created even though it requested CPUs that couldn't be allocated from a single NUMAnode.


Question: pluginDir: pluginDir, watcher:   watcher, fs:        fs, <del> factory:   fakePluginFactory{error: false}, </del> <ins> factory:   fakePluginFactory{}, </ins> } driverPath = filepath.Join(pluginDir, driverName) installDriver(driverName, fs)

When kubelet is running if you create two volume plugin directories here: /usr/libexec/kubernetes/kubelet-plugins/volume/exec /usr/libexec/kubernetes/kubelet-plugins/volume/exec/cifs /usr/libexec/kubernetes/kubelet-plugins/volume/exec/nodeagent~uds You will see these expected logs: Now if you copy over the cifs plugin file to cifs directory and then copy the uds plugin the errors go away. The pods using the uds volume will function but any pods attempting to use the cifs volume will fail with the following error from kubelet: desiredStateOfWorld. err=failed to get Plugin from volumeSpec for volume "pv" err=no volume plugin matched Restarting kubelet fixes this problem as then both plugins are present and properly . A functional volume plugin should be to kubelet even if another volume plugin is missing or invalid. Fixing the invalid volume plugin currently only adds the one that was invalid and fixed and other valid plugins that were previously present are skipped. Start kubelet service on a node. Create two directories for two flex volume plugins in the /usr/libexec/kubernetes/kubelet-plugins/volume/exec/ directory. Eg. nodeagent~uds and cifs Kubelet logs will now start complaining that the plugins are missing. Add the files to the plugin folders and then start a pod using the plugin. Notice that only the last plugin file copied is available to be used by pods. The first plugin will generate this error if used by a volume "err=no volume plugin matched" Performing a touch on the folder of the missing plugin will cause kubelet to properly index it. <details</details<detailsLocal kube cluster </details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig k8s-infra
/sig storage /remove-sig k8s-infra


Question: t.Fatal(err) } <ins> nfs := opts.Flags() for _, f := range nfs.FlagSets { fs.AddFlagSet(f) } if err := fs.Parse(tc.flags); err != nil { t.Fatal(err) } if err := opts.Complete(&nfs); err != nil { t.Fatal(err) } </ins> // use listeners instead of static ports so parallel test runs don't conflict opts.SecureServing.Listener = makeListener(t) defer opts.SecureServing.Listener.Close()

ci-kubernetes-e2e-gci-gce-alpha-features kubetest.Up Jul 02 2021, 01:26:42 UTC+0200 /sig release
based on testgrid probably cc <img width="1203" alt="Screen Shot 2021-07-02 at 4 24 25 PM" src="https://user-"
/cc working on it
/assign
the scheduler fails to start with this error: it seems the feature gates from the command line does not take effect in the defaulting phrase, do you know how to fix it?
ah, found there is a cycle here: The scheduler options , but the default config's default values depend on the feature gates options. I think this disables other feature gate controlled logic in the defaulting too:
I tried to fix it by updating the component config after the CLI flags (including the feature gates) are parsed: PTAL
fixed the flake. is back green now.
thanks!


Question: // return the current replicas if the change would be too small return currentReplicas, utilization, timestamp, nil } <del> replicaCount = int32(math.Ceil(usageRatio * float64(currentReplicas))) </del> <ins> readyPodCount, err := c.getReadyPodsCount(namespace, selector) if err != nil { return 0, 0, time.Time{}, fmt.Errorf("unable to calculate ready pods: %s", err) } replicaCount = int32(math.Ceil(usageRatio * float64(readyPodCount))) </ins> return replicaCount, utilization, timestamp, nil } <ins> // @TODO(mattjmcnaughton) Many different functions in this module use variations // of this function. Make this function generic, so we don't repeat the same // logic in multiple places. func (c *ReplicaCalculator) getReadyPodsCount(namespace string, selector labels.Selector) (int64, error) { podList, err := c.podsGetter.Pods(namespace).List(metav1.ListOptions{LabelSelector: selector.String()}) if err != nil { return 0, fmt.Errorf("unable to get pods while calculating replica count: %v", err) } if len(podList.Items) == 0 { return 0, fmt.Errorf("no pods returned by selector while calculating replica count") } readyPodCount := 0 for _, pod := range podList.Items { if pod.Status.Phase == v1.PodRunning && podutil.IsPodReady(&pod) { readyPodCount++ } } return int64(readyPodCount), nil } </ins> // GetExternalMetricReplicas calculates the desired replica count based on a // target metric value (as a milli-value) for the external metric in the given // namespace, and the current replica count. <del> func (c *ReplicaCalculator) GetExternalMetricReplicas(currentReplicas int32, targetUtilization int64, metricName, namespace string, selector *metav1.LabelSelector) (replicaCount int32, utilization int64, timestamp time.Time, err error) { labelSelector, err := metav1.LabelSelectorAsSelector(selector) </del> <ins> func (c *ReplicaCalculator) GetExternalMetricReplicas(currentReplicas int32, targetUtilization int64, metricName, namespace string, metricSelector *metav1.LabelSelector, podSelector labels.Selector) (replicaCount int32, utilization int64, timestamp time.Time, err error) { metricLabelSelector, err := metav1.LabelSelectorAsSelector(metricSelector) </ins> if err != nil { return 0, 0, time.Time{}, err } <del> metrics, timestamp, err := c.metricsClient.GetExternalMetric(metricName, namespace, labelSelector) </del> <ins> metrics, timestamp, err := c.metricsClient.GetExternalMetric(metricName, namespace, metricLabelSelector) </ins> if err != nil { <del> return 0, 0, time.Time{}, fmt.Errorf("unable to get external metric %s/%s/%+v: %s", namespace, metricName, selector, err) </del> <ins> return 0, 0, time.Time{}, fmt.Errorf("unable to get external metric %s/%s/%+v: %s", namespace, metricName, metricSelector, err) </ins> } utilization = 0 for _, val := range metrics { utilization = utilization + val } <ins> readyPodCount, err := c.getReadyPodsCount(namespace, podSelector) if err != nil { return 0, 0, time.Time{}, fmt.Errorf("unable to calculate ready pods: %s", err) } </ins> usageRatio := float64(utilization) / float64(targetUtilization) if math.Abs(1.0-usageRatio) <= c.tolerance { // return the current replicas if the change would be too small return currentReplicas, utilization, timestamp, nil } <del> return int32(math.Ceil(usageRatio * float64(currentReplicas))), utilization, timestamp, nil </del> <ins> return int32(math.Ceil(usageRatio * float64(readyPodCount))), utilization, timestamp, nil </ins> } // GetExternalPerPodMetricReplicas calculates the desired replica count based on a // target metric value per pod (as a milli-value) for the external metric in the // given namespace, and the current replica count. <del> func (c *ReplicaCalculator) GetExternalPerPodMetricReplicas(currentReplicas int32, targetUtilizationPerPod int64, metricName, namespace string, selector *metav1.LabelSelector) (replicaCount int32, utilization int64, timestamp time.Time, err error) { labelSelector, err := metav1.LabelSelectorAsSelector(selector) </del> <ins> func (c *ReplicaCalculator) GetExternalPerPodMetricReplicas(currentReplicas int32, targetUtilizationPerPod int64, metricName, namespace string, metricSelector *metav1.LabelSelector) (replicaCount int32, utilization int64, timestamp time.Time, err error) { metricLabelSelector, err := metav1.LabelSelectorAsSelector(metricSelector) </ins> if err != nil { return 0, 0, time.Time{}, err } <del> metrics, timestamp, err := c.metricsClient.GetExternalMetric(metricName, namespace, labelSelector) </del> <ins> metrics, timestamp, err := c.metricsClient.GetExternalMetric(metricName, namespace, metricLabelSelector) </ins> if err != nil { <del> return 0, 0, time.Time{}, fmt.Errorf("unable to get external metric %s/%s/%+v: %s", namespace, metricName, selector, err) </del> <ins> return 0, 0, time.Time{}, fmt.Errorf("unable to get external metric %s/%s/%+v: %s", namespace, metricName, metricSelector, err) </ins> } utilization = 0 for _, val := range metrics {

Is this a BUG REPORT or FEATURE REQUEST?: /kind feature What happened: We are starting to use the HPA (horizontal pod autoscaling) for one of our components. What is special about this particular component is that the warm up period is very long (sometimes even more than 30 minutes). During that time, the pod is in Running state but its containers are in Not Ready state. This pod is of course not able to receive any traffic so the metric we use in the HPA is ignoring those pods that are not Ready. The equation used by HPA to calculate the new number of replicas is as follows Our problem is that the equation used by HPA to calculate the new number of replicas is including in the "current replica count" the pods that are "Not Ready" so the value for "new replica count" is not correct. Since this warm up is so long we end up by having a lot of more instances of our services of what it should be necessary. They eventually get deleted once gradually the metrics start going down but that will take a lot of time and a big waste of resources without mentioning that with a changing traffic this HPA will never stabilize What you expected to happen: I would like to have a flag somewhere where I tell HPA to don't include in the "current replica count" the pods that are in "Not Ready" state How to reproduce it (as minimally and precisely as possible): Have a component that takes a lot of time to start and being able to receive traffic. Then define a metric that will ignore the pods that are Not Ready. Finally create a metric that will scale that component based on the mentioned metric Anything else we need to know?: Environment: Kubernetes version 1.7 Cloud provider or hardware configuration: Azure
/sig autoscaling
Hi - thanks for reaching out about this! It looks like this section of the , handles when pods aren't ready. I think you this behavior in , so I'm interested in your thoughts on whats best here. If we do end up adding a flag for this, I'd be happy to write the code :) Also, purely for my own curiosity, I'm curious what type of pod is taking 30 minutes to become ready. If your traffic pattern varies at a rate faster than 30 minutes, it may be difficult for auto-scaling to be helpful.
can you also share a bit more about your HPA?


Question: // return the current replicas if the change would be too small return currentReplicas, utilization, timestamp, nil } <del> replicaCount = int32(math.Ceil(usageRatio * float64(currentReplicas))) </del> <ins> readyPodCount, err := c.getReadyPodsCount(namespace, selector) if err != nil { return 0, 0, time.Time{}, fmt.Errorf("unable to calculate ready pods: %s", err) } replicaCount = int32(math.Ceil(usageRatio * float64(readyPodCount))) </ins> return replicaCount, utilization, timestamp, nil } <ins> // @TODO(mattjmcnaughton) Many different functions in this module use variations // of this function. Make this function generic, so we don't repeat the same // logic in multiple places. func (c *ReplicaCalculator) getReadyPodsCount(namespace string, selector labels.Selector) (int64, error) { podList, err := c.podsGetter.Pods(namespace).List(metav1.ListOptions{LabelSelector: selector.String()}) if err != nil { return 0, fmt.Errorf("unable to get pods while calculating replica count: %v", err) } if len(podList.Items) == 0 { return 0, fmt.Errorf("no pods returned by selector while calculating replica count") } readyPodCount := 0 for _, pod := range podList.Items { if pod.Status.Phase == v1.PodRunning && podutil.IsPodReady(&pod) { readyPodCount++ } } return int64(readyPodCount), nil } </ins> // GetExternalMetricReplicas calculates the desired replica count based on a // target metric value (as a milli-value) for the external metric in the given // namespace, and the current replica count. <del> func (c *ReplicaCalculator) GetExternalMetricReplicas(currentReplicas int32, targetUtilization int64, metricName, namespace string, selector *metav1.LabelSelector) (replicaCount int32, utilization int64, timestamp time.Time, err error) { labelSelector, err := metav1.LabelSelectorAsSelector(selector) </del> <ins> func (c *ReplicaCalculator) GetExternalMetricReplicas(currentReplicas int32, targetUtilization int64, metricName, namespace string, metricSelector *metav1.LabelSelector, podSelector labels.Selector) (replicaCount int32, utilization int64, timestamp time.Time, err error) { metricLabelSelector, err := metav1.LabelSelectorAsSelector(metricSelector) </ins> if err != nil { return 0, 0, time.Time{}, err } <del> metrics, timestamp, err := c.metricsClient.GetExternalMetric(metricName, namespace, labelSelector) </del> <ins> metrics, timestamp, err := c.metricsClient.GetExternalMetric(metricName, namespace, metricLabelSelector) </ins> if err != nil { <del> return 0, 0, time.Time{}, fmt.Errorf("unable to get external metric %s/%s/%+v: %s", namespace, metricName, selector, err) </del> <ins> return 0, 0, time.Time{}, fmt.Errorf("unable to get external metric %s/%s/%+v: %s", namespace, metricName, metricSelector, err) </ins> } utilization = 0 for _, val := range metrics { utilization = utilization + val } <ins> readyPodCount, err := c.getReadyPodsCount(namespace, podSelector) if err != nil { return 0, 0, time.Time{}, fmt.Errorf("unable to calculate ready pods: %s", err) } </ins> usageRatio := float64(utilization) / float64(targetUtilization) if math.Abs(1.0-usageRatio) <= c.tolerance { // return the current replicas if the change would be too small return currentReplicas, utilization, timestamp, nil } <del> return int32(math.Ceil(usageRatio * float64(currentReplicas))), utilization, timestamp, nil </del> <ins> return int32(math.Ceil(usageRatio * float64(readyPodCount))), utilization, timestamp, nil </ins> } // GetExternalPerPodMetricReplicas calculates the desired replica count based on a // target metric value per pod (as a milli-value) for the external metric in the // given namespace, and the current replica count. <del> func (c *ReplicaCalculator) GetExternalPerPodMetricReplicas(currentReplicas int32, targetUtilizationPerPod int64, metricName, namespace string, selector *metav1.LabelSelector) (replicaCount int32, utilization int64, timestamp time.Time, err error) { labelSelector, err := metav1.LabelSelectorAsSelector(selector) </del> <ins> func (c *ReplicaCalculator) GetExternalPerPodMetricReplicas(currentReplicas int32, targetUtilizationPerPod int64, metricName, namespace string, metricSelector *metav1.LabelSelector) (replicaCount int32, utilization int64, timestamp time.Time, err error) { metricLabelSelector, err := metav1.LabelSelectorAsSelector(metricSelector) </ins> if err != nil { return 0, 0, time.Time{}, err } <del> metrics, timestamp, err := c.metricsClient.GetExternalMetric(metricName, namespace, labelSelector) </del> <ins> metrics, timestamp, err := c.metricsClient.GetExternalMetric(metricName, namespace, metricLabelSelector) </ins> if err != nil { <del> return 0, 0, time.Time{}, fmt.Errorf("unable to get external metric %s/%s/%+v: %s", namespace, metricName, selector, err) </del> <ins> return 0, 0, time.Time{}, fmt.Errorf("unable to get external metric %s/%s/%+v: %s", namespace, metricName, metricSelector, err) </ins> } utilization = 0 for _, val := range metrics {

What kind of metric type is it? Based on the description I'm suspecting that it is an Object metric, am I right?
So answer about why our process takes that much time to start. Basically this is a server that in order to do what he needs to do, he needs to load a lot of data in memory (several GBs) and then perform some warm up that are pretty intensive. The idea is that he doesn't need to grab (almost) anything from outside to attend a request so it's latency is extremely low (in the order of millisecond) By the way. Because of this we have another problem which is that there is not an specific flag per HPA object that we could tell k8s to don't try to scale up again passed some time so the new pods have enough time to get ready before the next scale happens. I know that there is a cluster wide flag but we will need to have a different one by resource that we scale. Now to answer You pointed another interesting topic. The metric we use is % Utilization. We are able to calculate the level of utilization of our pods and we want to keep this metric in an specific value. About the flag. I suggested the flag so it doesn't change the existing behavior by default but I honestly think that the default behavior should be to don't include the NonReady pods as part of the calculation
do you think we can add this change to 1.10? it's critical bug/feature for us and we are happy to help with code/testing.
It's really late for 1.10, especially since we don't have any code here, and I'd like to know more about what's going on here, and especially see if it reproduces on Kubernetes 1.9 (we've had HPA controller bugs in the past). FWIW, currently unready pods are treated as consuming 0% of their utilization on scale up, and it was intended that this should prevent such issues. That is, the formula is actually: If the pods are marked as not ready, I find it strange that they would be affecting the scaling that much, unless your other pods are under real heavy load (i.e. I would assume that the unready pods would skew the scaling back down so that the number of pods just stayed constant).
I believe there is different code branch here.


Question: // return the current replicas if the change would be too small return currentReplicas, utilization, timestamp, nil } <del> replicaCount = int32(math.Ceil(usageRatio * float64(currentReplicas))) </del> <ins> readyPodCount, err := c.getReadyPodsCount(namespace, selector) if err != nil { return 0, 0, time.Time{}, fmt.Errorf("unable to calculate ready pods: %s", err) } replicaCount = int32(math.Ceil(usageRatio * float64(readyPodCount))) </ins> return replicaCount, utilization, timestamp, nil } <ins> // @TODO(mattjmcnaughton) Many different functions in this module use variations // of this function. Make this function generic, so we don't repeat the same // logic in multiple places. func (c *ReplicaCalculator) getReadyPodsCount(namespace string, selector labels.Selector) (int64, error) { podList, err := c.podsGetter.Pods(namespace).List(metav1.ListOptions{LabelSelector: selector.String()}) if err != nil { return 0, fmt.Errorf("unable to get pods while calculating replica count: %v", err) } if len(podList.Items) == 0 { return 0, fmt.Errorf("no pods returned by selector while calculating replica count") } readyPodCount := 0 for _, pod := range podList.Items { if pod.Status.Phase == v1.PodRunning && podutil.IsPodReady(&pod) { readyPodCount++ } } return int64(readyPodCount), nil } </ins> // GetExternalMetricReplicas calculates the desired replica count based on a // target metric value (as a milli-value) for the external metric in the given // namespace, and the current replica count. <del> func (c *ReplicaCalculator) GetExternalMetricReplicas(currentReplicas int32, targetUtilization int64, metricName, namespace string, selector *metav1.LabelSelector) (replicaCount int32, utilization int64, timestamp time.Time, err error) { labelSelector, err := metav1.LabelSelectorAsSelector(selector) </del> <ins> func (c *ReplicaCalculator) GetExternalMetricReplicas(currentReplicas int32, targetUtilization int64, metricName, namespace string, metricSelector *metav1.LabelSelector, podSelector labels.Selector) (replicaCount int32, utilization int64, timestamp time.Time, err error) { metricLabelSelector, err := metav1.LabelSelectorAsSelector(metricSelector) </ins> if err != nil { return 0, 0, time.Time{}, err } <del> metrics, timestamp, err := c.metricsClient.GetExternalMetric(metricName, namespace, labelSelector) </del> <ins> metrics, timestamp, err := c.metricsClient.GetExternalMetric(metricName, namespace, metricLabelSelector) </ins> if err != nil { <del> return 0, 0, time.Time{}, fmt.Errorf("unable to get external metric %s/%s/%+v: %s", namespace, metricName, selector, err) </del> <ins> return 0, 0, time.Time{}, fmt.Errorf("unable to get external metric %s/%s/%+v: %s", namespace, metricName, metricSelector, err) </ins> } utilization = 0 for _, val := range metrics { utilization = utilization + val } <ins> readyPodCount, err := c.getReadyPodsCount(namespace, podSelector) if err != nil { return 0, 0, time.Time{}, fmt.Errorf("unable to calculate ready pods: %s", err) } </ins> usageRatio := float64(utilization) / float64(targetUtilization) if math.Abs(1.0-usageRatio) <= c.tolerance { // return the current replicas if the change would be too small return currentReplicas, utilization, timestamp, nil } <del> return int32(math.Ceil(usageRatio * float64(currentReplicas))), utilization, timestamp, nil </del> <ins> return int32(math.Ceil(usageRatio * float64(readyPodCount))), utilization, timestamp, nil </ins> } // GetExternalPerPodMetricReplicas calculates the desired replica count based on a // target metric value per pod (as a milli-value) for the external metric in the // given namespace, and the current replica count. <del> func (c *ReplicaCalculator) GetExternalPerPodMetricReplicas(currentReplicas int32, targetUtilizationPerPod int64, metricName, namespace string, selector *metav1.LabelSelector) (replicaCount int32, utilization int64, timestamp time.Time, err error) { labelSelector, err := metav1.LabelSelectorAsSelector(selector) </del> <ins> func (c *ReplicaCalculator) GetExternalPerPodMetricReplicas(currentReplicas int32, targetUtilizationPerPod int64, metricName, namespace string, metricSelector *metav1.LabelSelector) (replicaCount int32, utilization int64, timestamp time.Time, err error) { metricLabelSelector, err := metav1.LabelSelectorAsSelector(metricSelector) </ins> if err != nil { return 0, 0, time.Time{}, err } <del> metrics, timestamp, err := c.metricsClient.GetExternalMetric(metricName, namespace, labelSelector) </del> <ins> metrics, timestamp, err := c.metricsClient.GetExternalMetric(metricName, namespace, metricLabelSelector) </ins> if err != nil { <del> return 0, 0, time.Time{}, fmt.Errorf("unable to get external metric %s/%s/%+v: %s", namespace, metricName, selector, err) </del> <ins> return 0, 0, time.Time{}, fmt.Errorf("unable to get external metric %s/%s/%+v: %s", namespace, metricName, metricSelector, err) </ins> } utilization = 0 for _, val := range metrics {

metric definition { "type": "Object", "object": { "target": { "kind": "Service", "name": "prometheus-exporter" }, "metricName": "PercentUtilization", "targetValue": "20" } } which should handle as and doesn't check Pod state.
I think proper handling of unready pods is a difficult problem that will require significant work. I suspect we may find out we need to start distinguishing pods that are booting up from pods that are 'broken' and it took us quite a while to solve analogous problem with nodes in Cluster Autoscaler (to be clear: I don't know if we will have to do that, that's just my intuition for now). That being said I agree the formula in GetObjectMetricReplicas will generally lead to scaling-up too much if the pod takes longer to initialize then backoff period in HPA (3 minutes). Assume it takes pod 4 minutes to boot up and we're currently running 2 pods. Metric value is 20 and target is 10. In this case: scales-up to 4 pods (20/10 utilization 2 pods). are starting up, they're not doing anything yet. 3 minutes later HPA scales-up to 8 pods (20/10 utilization 4 pods). If we only counted ready pods the scale-up in 3 would not occur, because the desired replicas would still be 4 (20/10 * 2). My intuition would be that this is the expected behavior of HPA. Also if there is no space in the cluster we will just keep scaling-up until we hit max replicas. The only case where current behavior would be more correct than filtering out unready pods in this formula is if someone uses a metric that is calculated based on total number of replicas. Do we advertise that as the right way to use HPA? If we add to Object metrics in 1.11 this usage pattern will stop making sense. To sum up - I think we will need to change the formula to only count ready pods at some point. I can see how current behavior could be a problem in any use-case except the one described in paragraph above. I'm inclined to agree with and provide some way to change the behavior in 1.10 (a flag or an annotation?), treating it as a bug fix.


Question: // return the current replicas if the change would be too small return currentReplicas, utilization, timestamp, nil } <del> replicaCount = int32(math.Ceil(usageRatio * float64(currentReplicas))) </del> <ins> readyPodCount, err := c.getReadyPodsCount(namespace, selector) if err != nil { return 0, 0, time.Time{}, fmt.Errorf("unable to calculate ready pods: %s", err) } replicaCount = int32(math.Ceil(usageRatio * float64(readyPodCount))) </ins> return replicaCount, utilization, timestamp, nil } <ins> // @TODO(mattjmcnaughton) Many different functions in this module use variations // of this function. Make this function generic, so we don't repeat the same // logic in multiple places. func (c *ReplicaCalculator) getReadyPodsCount(namespace string, selector labels.Selector) (int64, error) { podList, err := c.podsGetter.Pods(namespace).List(metav1.ListOptions{LabelSelector: selector.String()}) if err != nil { return 0, fmt.Errorf("unable to get pods while calculating replica count: %v", err) } if len(podList.Items) == 0 { return 0, fmt.Errorf("no pods returned by selector while calculating replica count") } readyPodCount := 0 for _, pod := range podList.Items { if pod.Status.Phase == v1.PodRunning && podutil.IsPodReady(&pod) { readyPodCount++ } } return int64(readyPodCount), nil } </ins> // GetExternalMetricReplicas calculates the desired replica count based on a // target metric value (as a milli-value) for the external metric in the given // namespace, and the current replica count. <del> func (c *ReplicaCalculator) GetExternalMetricReplicas(currentReplicas int32, targetUtilization int64, metricName, namespace string, selector *metav1.LabelSelector) (replicaCount int32, utilization int64, timestamp time.Time, err error) { labelSelector, err := metav1.LabelSelectorAsSelector(selector) </del> <ins> func (c *ReplicaCalculator) GetExternalMetricReplicas(currentReplicas int32, targetUtilization int64, metricName, namespace string, metricSelector *metav1.LabelSelector, podSelector labels.Selector) (replicaCount int32, utilization int64, timestamp time.Time, err error) { metricLabelSelector, err := metav1.LabelSelectorAsSelector(metricSelector) </ins> if err != nil { return 0, 0, time.Time{}, err } <del> metrics, timestamp, err := c.metricsClient.GetExternalMetric(metricName, namespace, labelSelector) </del> <ins> metrics, timestamp, err := c.metricsClient.GetExternalMetric(metricName, namespace, metricLabelSelector) </ins> if err != nil { <del> return 0, 0, time.Time{}, fmt.Errorf("unable to get external metric %s/%s/%+v: %s", namespace, metricName, selector, err) </del> <ins> return 0, 0, time.Time{}, fmt.Errorf("unable to get external metric %s/%s/%+v: %s", namespace, metricName, metricSelector, err) </ins> } utilization = 0 for _, val := range metrics { utilization = utilization + val } <ins> readyPodCount, err := c.getReadyPodsCount(namespace, podSelector) if err != nil { return 0, 0, time.Time{}, fmt.Errorf("unable to calculate ready pods: %s", err) } </ins> usageRatio := float64(utilization) / float64(targetUtilization) if math.Abs(1.0-usageRatio) <= c.tolerance { // return the current replicas if the change would be too small return currentReplicas, utilization, timestamp, nil } <del> return int32(math.Ceil(usageRatio * float64(currentReplicas))), utilization, timestamp, nil </del> <ins> return int32(math.Ceil(usageRatio * float64(readyPodCount))), utilization, timestamp, nil </ins> } // GetExternalPerPodMetricReplicas calculates the desired replica count based on a // target metric value per pod (as a milli-value) for the external metric in the // given namespace, and the current replica count. <del> func (c *ReplicaCalculator) GetExternalPerPodMetricReplicas(currentReplicas int32, targetUtilizationPerPod int64, metricName, namespace string, selector *metav1.LabelSelector) (replicaCount int32, utilization int64, timestamp time.Time, err error) { labelSelector, err := metav1.LabelSelectorAsSelector(selector) </del> <ins> func (c *ReplicaCalculator) GetExternalPerPodMetricReplicas(currentReplicas int32, targetUtilizationPerPod int64, metricName, namespace string, metricSelector *metav1.LabelSelector) (replicaCount int32, utilization int64, timestamp time.Time, err error) { metricLabelSelector, err := metav1.LabelSelectorAsSelector(metricSelector) </ins> if err != nil { return 0, 0, time.Time{}, err } <del> metrics, timestamp, err := c.metricsClient.GetExternalMetric(metricName, namespace, labelSelector) </del> <ins> metrics, timestamp, err := c.metricsClient.GetExternalMetric(metricName, namespace, metricLabelSelector) </ins> if err != nil { <del> return 0, 0, time.Time{}, fmt.Errorf("unable to get external metric %s/%s/%+v: %s", namespace, metricName, selector, err) </del> <ins> return 0, 0, time.Time{}, fmt.Errorf("unable to get external metric %s/%s/%+v: %s", namespace, metricName, metricSelector, err) </ins> } utilization = 0 for _, val := range metrics {

WDYT
Ah, yeah, I missed that it was object metrics before. I'm not entirely convinced we need a flag here -- it might just make sense to bring Object metrics in line with all the other metrics types. On the other hand, there's a subtlety here -- the other metric types don't quite ignore unready pods -- they just ignore the metrics from unready pods, which causes a heavy bias in the direction of not scaling, but an overwhelming spike can still cause scaling. That behavior is not as achievable using object metrics. I'm inclined to just ignore that subtlety, though, and say that object metrics should ignore unready pods entirely.
I think this is what we should do long term. I was worried about breaking someone who relies on targetValue + averaged metric to get effect similar to targetAverageValue and though about providing something for transitional period, but if you're ok with it I'd rather just change it as you say.
I think it's safe to change as-is -- that feature's still in beta, and it's not an API change, so it shouldn't just hard-break anyone. Plus, we've change stuff like this before (when we did the initial resource metrics change).
As said, we are more than ready to help with the fix/test if required. This since this is a major blocking for us in the implementation of the HPA in our side
is anyone actively working on implementing this? If not, I'm happy to. I think I have a fairly clear picture of the change we need to make and will likely have time early this week to get it done.
(Also, from the conversation above, to we want to tag this as a release blocker for 1.10)?
I probably won't be able to work on this before the end of this week. Happy to do a review if you jump in and implement it.
I don't think this qualifies as a release blocking bug (but I'm willing to have someone try and convince me otherwise ;-)). There doesn't seem to be any behavior here that's explicitly broken -- it's just sub-optimal in some use cases. I'd definitely like to address this in 1.11, though.
it's not release blocker, but to share our experience ;).


Question: // return the current replicas if the change would be too small return currentReplicas, utilization, timestamp, nil } <del> replicaCount = int32(math.Ceil(usageRatio * float64(currentReplicas))) </del> <ins> readyPodCount, err := c.getReadyPodsCount(namespace, selector) if err != nil { return 0, 0, time.Time{}, fmt.Errorf("unable to calculate ready pods: %s", err) } replicaCount = int32(math.Ceil(usageRatio * float64(readyPodCount))) </ins> return replicaCount, utilization, timestamp, nil } <ins> // @TODO(mattjmcnaughton) Many different functions in this module use variations // of this function. Make this function generic, so we don't repeat the same // logic in multiple places. func (c *ReplicaCalculator) getReadyPodsCount(namespace string, selector labels.Selector) (int64, error) { podList, err := c.podsGetter.Pods(namespace).List(metav1.ListOptions{LabelSelector: selector.String()}) if err != nil { return 0, fmt.Errorf("unable to get pods while calculating replica count: %v", err) } if len(podList.Items) == 0 { return 0, fmt.Errorf("no pods returned by selector while calculating replica count") } readyPodCount := 0 for _, pod := range podList.Items { if pod.Status.Phase == v1.PodRunning && podutil.IsPodReady(&pod) { readyPodCount++ } } return int64(readyPodCount), nil } </ins> // GetExternalMetricReplicas calculates the desired replica count based on a // target metric value (as a milli-value) for the external metric in the given // namespace, and the current replica count. <del> func (c *ReplicaCalculator) GetExternalMetricReplicas(currentReplicas int32, targetUtilization int64, metricName, namespace string, selector *metav1.LabelSelector) (replicaCount int32, utilization int64, timestamp time.Time, err error) { labelSelector, err := metav1.LabelSelectorAsSelector(selector) </del> <ins> func (c *ReplicaCalculator) GetExternalMetricReplicas(currentReplicas int32, targetUtilization int64, metricName, namespace string, metricSelector *metav1.LabelSelector, podSelector labels.Selector) (replicaCount int32, utilization int64, timestamp time.Time, err error) { metricLabelSelector, err := metav1.LabelSelectorAsSelector(metricSelector) </ins> if err != nil { return 0, 0, time.Time{}, err } <del> metrics, timestamp, err := c.metricsClient.GetExternalMetric(metricName, namespace, labelSelector) </del> <ins> metrics, timestamp, err := c.metricsClient.GetExternalMetric(metricName, namespace, metricLabelSelector) </ins> if err != nil { <del> return 0, 0, time.Time{}, fmt.Errorf("unable to get external metric %s/%s/%+v: %s", namespace, metricName, selector, err) </del> <ins> return 0, 0, time.Time{}, fmt.Errorf("unable to get external metric %s/%s/%+v: %s", namespace, metricName, metricSelector, err) </ins> } utilization = 0 for _, val := range metrics { utilization = utilization + val } <ins> readyPodCount, err := c.getReadyPodsCount(namespace, podSelector) if err != nil { return 0, 0, time.Time{}, fmt.Errorf("unable to calculate ready pods: %s", err) } </ins> usageRatio := float64(utilization) / float64(targetUtilization) if math.Abs(1.0-usageRatio) <= c.tolerance { // return the current replicas if the change would be too small return currentReplicas, utilization, timestamp, nil } <del> return int32(math.Ceil(usageRatio * float64(currentReplicas))), utilization, timestamp, nil </del> <ins> return int32(math.Ceil(usageRatio * float64(readyPodCount))), utilization, timestamp, nil </ins> } // GetExternalPerPodMetricReplicas calculates the desired replica count based on a // target metric value per pod (as a milli-value) for the external metric in the // given namespace, and the current replica count. <del> func (c *ReplicaCalculator) GetExternalPerPodMetricReplicas(currentReplicas int32, targetUtilizationPerPod int64, metricName, namespace string, selector *metav1.LabelSelector) (replicaCount int32, utilization int64, timestamp time.Time, err error) { labelSelector, err := metav1.LabelSelectorAsSelector(selector) </del> <ins> func (c *ReplicaCalculator) GetExternalPerPodMetricReplicas(currentReplicas int32, targetUtilizationPerPod int64, metricName, namespace string, metricSelector *metav1.LabelSelector) (replicaCount int32, utilization int64, timestamp time.Time, err error) { metricLabelSelector, err := metav1.LabelSelectorAsSelector(metricSelector) </ins> if err != nil { return 0, 0, time.Time{}, err } <del> metrics, timestamp, err := c.metricsClient.GetExternalMetric(metricName, namespace, labelSelector) </del> <ins> metrics, timestamp, err := c.metricsClient.GetExternalMetric(metricName, namespace, metricLabelSelector) </ins> if err != nil { <del> return 0, 0, time.Time{}, fmt.Errorf("unable to get external metric %s/%s/%+v: %s", namespace, metricName, selector, err) </del> <ins> return 0, 0, time.Time{}, fmt.Errorf("unable to get external metric %s/%s/%+v: %s", namespace, metricName, metricSelector, err) </ins> } utilization = 0 for _, val := range metrics {

without this feature we cannot use HPA as is and we need to figure out how to build/run fork (which can be challenging because everything inside controller-manager). also we are planning to upgrade to 1.10 quickly, but not sure for 1.11 (e.g. we are two versions behind now).
Sounds like a plan :) Will try and get this done tomorrow morning!
- I submitted a pr for this - would love your eyes on it so you can see if you think it meets your use case :)
/assign
GitHub didn't allow me to assign the following users: mattjmcnaughton. Note that only and repo collaborators can be assigned. <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I checked the code and it seems good to me. About the comment about scaling down if the readyPods<CurrentPods, we should be fine since our metric is already not including the pods that are not ready.
Is there any advance in this ticket? I'm wondering if this made it to k8s 1.10 or if we need to wait for k8s 1.11?
The PR is lgtm-ed and it will merge once the code freeze is lifted. Meaning it will go to 1.11. To get it into 1.10 you would need to get approval for milestone from (sig-autoscaling lead).


Question: import ( "fmt" <del> "os" "os/signal" </del> "reflect" "time"

PLEASE COORDINATE WITH SIG-SCHEDULING BEFORE WORKING ON THIS ISSUE. We want to avoid the situation where two people accidentally start work on the same fix. Main tracking issue: Currently, the feature is set up at the end of in . This setup should be moved into its own helper function, preferably in the package. /sig scheduling /kind cleanup /help
/assign /remove-help
I'd like to take this one
/remove-help
the PR I sent should resolve this one. :disappointed: Are any of the others still available?


Question: return f } <ins> // BindTransformingStringFlag binds the flag based on the provided info.  If LongName == "", nothing is registered func (f FlagInfo) BindTransformingStringFlag(flags *pflag.FlagSet, target *string, transformer func(string) (string, error)) FlagInfo { // you can't register a flag without a long name if len(f.LongName) > 0 { flags.VarP(newTransformingStringValue(f.Default, target, transformer), f.LongName, f.ShortName, f.Description) } return f } </ins> // BindStringSliceFlag binds the flag based on the provided info.  If LongName == "", nothing is registered func (f FlagInfo) BindStringArrayFlag(flags *pflag.FlagSet, target *[]string) FlagInfo { // you can't register a flag without a long name

Because always return names with a prefix like , it cannot be directly used to iterate over all namespaces to execute other commands. I've compiled from source: Reviewing some scripts, I came across a usage of like this: That line fetches namespace names to later run in a loop: That seemed to be a perfect use case for the flag, but it can't be used without extra hacks to get rid of the prefix (FWIW, is invalid). Changing the current output of might not be possible or desirable, for this could impact existing users that rely on the output format. If I'm wrong about the above limitation, than it would make sense to output just the name when and there is a single resource type, while using the ideas introduced in to show the resource type only when there is more than one type in the output, or when is explicitly set. Before: After:
could you help me ping the appropriate team to triage this? Thanks!
Please re-post your question to . We are trying to consolidate the channels to which questions for help/support are posted so that we can improve our efficiency in responding to your requests, and to make it easier for you to find answers to frequently asked questions and how to address common use cases. We regularly see messages posted in multiple forums, with the full response thread only in one place or, worse, spread across multiple forums. Also, the large volume of support issues on github is making it difficult for us to use issues to identify real bugs. The Kubernetes team scans stackoverflow on a regular basis, and will try to ensure your questions don't go unanswered. Before posting a new question, please search stackoverflow for answers to similar questions, and also familiarize yourself with: Again, thanks for using Kubernetes. The Kubernetes Team
perhaps I didn't do a good job at explaining the problem, but this is not a question of how to do something, but a report on a problem with the CLI design.
Sorry about that :-/. I'll try to route to the proper team.
update: console examples to the description.
the -o format is designed for scripting, the use case is just different from yours. You should be able to paste the output of into . Does do what you want? If this is a common use case we could create another -o flag value.


Question: return f } <ins> // BindTransformingStringFlag binds the flag based on the provided info.  If LongName == "", nothing is registered func (f FlagInfo) BindTransformingStringFlag(flags *pflag.FlagSet, target *string, transformer func(string) (string, error)) FlagInfo { // you can't register a flag without a long name if len(f.LongName) > 0 { flags.VarP(newTransformingStringValue(f.Default, target, transformer), f.LongName, f.ShortName, f.Description) } return f } </ins> // BindStringSliceFlag binds the flag based on the provided info.  If LongName == "", nothing is registered func (f FlagInfo) BindStringArrayFlag(flags *pflag.FlagSet, target *[]string) FlagInfo { // you can't register a flag without a long name

the use cases I've observed doing code reviews: over all namespaces to get certain resources from that namespace: over all namespaces, then iterate of over resource types that produce logs and fetch logs one by one: Above we can use for listing pods because later we'll pass it to that accept the syntax. We cannot do the same for namespaces, because doesn't like the format.
Alternatively we could be kubectl recognize as valid
I'll take a look on it if it's not taken by anyone yet. CC
Have you started this?
Only investigated due to being out of office since Thursday last week. I can work on it in upcoming week, but I'll understand if it's piority issue and you'll reassign it.
Ready :-)
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list .
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: } uidToIDMap := map[string][]string{} for _, container := range containers { <del> _, uid, name, _ := dockertools.ParseDockerName(container.ID) </del> <ins> _, uid, name, _ := dockertools.ParseDockerName(container.Names[0]) </ins> uidName := string(uid) + "." + name uidToIDMap[uidName] = append(uidToIDMap[uidName], container.ID) }

Successfully ran containers with "onFailure" restartPolicy are restarting after purging oldest docker containers. Kubelet purges/removes oldest containers periodically from docker. After purging, we loose the exit code and history of previously ran containers and restart containers as if they were never run.
Thanks & I can help fix this. Let me know. Thanks
Thanks for reporting Just to make sure I understand the issue: You ran some containers with an "onFailure" restartPolicy. When the Kubelet purged old containers the container was restarted (unexpectedly).
Ah, after reading again I think the issue is that after the containers are purged, when the container exits successfully it get restarted although it shouldn't. Is that correct?
After reading through the code I found a bug in the implementation that is most likely causing this error. Should have a PR soon.
Yes. Correct. After initial container run, it should never be run again, since it already ran to completion. Will wait for your PR.
PR is out. This should fix the issue. Feel free to test if you can :)


Question: } available := false for _, pod := range daemonPods { <del> if podutil.IsPodAvailable(pod, ds.Spec.MinReadySeconds, metav1.Now()) { </del> <ins> //for the purposes of update we ensure that the Pod is both available and not terminating if podutil.IsPodAvailable(pod, ds.Spec.MinReadySeconds, metav1.Now()) && pod.DeletionTimestamp == nil { </ins> available = true break }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: The DaemonSet controller will schedule a pod on a node when the old pod is still in Terminating state. This breaks the expectation that only one pod will be running at a time per node. What you expected to happen: The DaemonSet controller should not let two pods running at the same time on the same node. How to reproduce it (as minimally and precisely as possible): Create a DaemonSet. Set terminationGracePeriodSeconds to be relatively long. Kill a pod. Watch a new pod start on the same node when the old pod is still terminating. Anything else we need to know?: Seems this behavior was introduced in Environment: Kubernetes version (use ): 1.7.2 Cloud provider or hardware configuration**: GKE OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig apps
/cc
cc
cc: I think the expected behavior is correct, the terminating pod could be performing an important step such as 'removing a lock' or 'cleanup' etc., we should allow that to gracefully die before creating a new one. IMHO we should revoke to old behavior, but why did we need this new behavior in the first place? That is not clear to me from the PR and Issue.
We don't currently guarantee at least or at most one scheduling for DaemonSet. Both behaviors are under consideration via issue and in the future work section of the . The reason for is to "Make sure that controllers are not adopting children that are marked for deletion" (). This is necessary for GC to work properly.
Well the behavior may be technically undefined, but can I request that in the future changes like this be mentioned in the release notes?
Yes, certainly.


Question: } available := false for _, pod := range daemonPods { <del> if podutil.IsPodAvailable(pod, ds.Spec.MinReadySeconds, metav1.Now()) { </del> <ins> //for the purposes of update we ensure that the Pod is both available and not terminating if podutil.IsPodAvailable(pod, ds.Spec.MinReadySeconds, metav1.Now()) && pod.DeletionTimestamp == nil { </ins> available = true break }

Also "The DaemonSet controller should not let two pods running at the same time on the same node." is not necessarily the same thing as respecting termination prior to launching a new Pod. The former implies a more strict requirement than the latter. I think we should consider at least attempting to wait for a terminating Pod to be deleted prior to creating another Pod on the Node for same DaemonSet as default behavior.
will something like work then?


Question: func (c *csiMountMgr) SetUpAt(dir string, mounterArgs volume.MounterArgs) error { klog.V(4).Infof(log("Mounter.SetUpAt(%s)", dir)) <del> corruptedDir := false mounted, err := isDirMounted(c.plugin, dir) if err != nil { if isCorruptedDir(dir) { corruptedDir = true // leave to CSI driver to handle corrupted mount klog.Warning(log("mounter.SetUpAt detected corrupted mount for dir [%s]", dir)) } else { return errors.New(log("mounter.SetUpAt failed while checking mount status for dir [%s]: %v", dir, err)) } } if mounted && !corruptedDir { klog.V(4).Info(log("mounter.SetUpAt skipping mount, dir already mounted [%s]", dir)) return nil } </del> csi, err := c.csiClientGetter.Get() if err != nil { return volumetypes.NewTransientOperationFailure(log("mounter.SetUpAt failed to get CSI client: %v", err))

What happened: A custom CSI driver implementing returns a failure response, indicating that it did not complete its intended steps to prepare the volume for the mounting pod. Let's say that these steps involve a filesystem at the target directory, data from a service, and that data to a set of files in the mounted filesystem. If the first filesystem mounting step succeeds, but either of the subsequent two steps fail, the volume is still not ready for use by the pod. However, if this CSI driver returns a failure response, and the CSI mounter comes back and, before invoking the RPC again, it finds a filesystem already mounted at the target directory, , and ceases further calls to to allow the CSI driver to try again. What you expected to happen: If a CSI driver returns a failure response to a RPC, Kubernetes should honor that declaration of failure and invoke the RPC again, disallowing the mounting pod from starting until this RPC succeeds. Even though it's possible to partially work around the current behavior by having the driver mount the filesystem as late as possible, and then attempting to unmount it if fails to populate the volume with its intended data, doing so makes writing drivers more complicated and forces them to do extra work. How to reproduce it (as minimally and precisely as possible): In a CSI driver (even one that serves just ephemeral, inline volumes), implement to mount a filesystem at the target path, then return a failure response. Confirm that Kubernetes does not invoke again, because there's now a filesystem mounted at the target path. Anything else we need to know?: This topic came up for discussion in the "Kubernetes" Slack team. There, helped investigate this complaint, and to discuss changing this behavior. Environment: Kubernetes version (use ):Cloud provider or hardware configuration: AWS EC2 OS (e.g: ):Kernel (e.g. ):Install tools: kubeadm Others: csi-node-driver-registrar container image tag: v1.0.2
/sig storage
While the plugin should try to clean up as much as possible, I agree that kubelet should not be assuming that if there's a mount point, the publish is successful. NodePublish is supposed to be idempotent so it should theoretically be safe to call it when it's already mounted, although there is the possibility that plugins may have written against the K8s behavior.