Question:
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite} Previous issues for this test:
This is pretty much the same failure, that are happening in "gce-enormous" suite and we currently debugging them with The main reason of those failures is that we are terminating all watchers for a particular resource type from time to time, and some of them are actually causing a snowball effect. After more looking into it, my hypothesis is that our machinery for aggregating watchers in apiserver sometimes is just to slow. In particular, I some log in my local branch and what I'm observing is that in etcd_watcher, we are: accumulating objects in incoming channel but also accumulating objects in outgoing channel, and the accumulation there is usually higher [I will send a PR to put those logs into master - they seem to be pretty useful for debugging such issues.] So my current hypothesis is that dispatching watch events to watchers is what is too slow. We will continue debugging.
The logs that I seem to confirm my hypothesis - outgoing channel is generally more full then the incoming channel in etcd_watcher: vs I'm working on the PR that should at least improve the situation.
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
Please apply the label to this issue if it does not need to be fixed before we launch 1.4. Otherwise please advise on how to proceed with respect to 1.4.
This was a regression - I fixed it by reverting an offending PR: This seems to be fixed now.
I have 33 pods in my container (1 is substantive, the other 32 just run on log files). I get this all the time now. I'm running 1.5.4.
- to be honest, I don't undertsand what your wrote. what is "container" in "33 pods in my container"? Also, if you believe there is a bug somewhere, please open a new issue instead of adding comment to more than half-year-old one.

// OK, block sending, but only for up to 5 seconds. // cacheWatcher.add is called very often, so arrange // to reuse timers instead of constantly allocating. <ins> startTime := time.Now() </ins> const timeout = 5 * time.Second t, ok := timerPool.Get().(*time.Timer) if ok {


Question:
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite} Previous issues for this test:
This is pretty much the same failure, that are happening in "gce-enormous" suite and we currently debugging them with The main reason of those failures is that we are terminating all watchers for a particular resource type from time to time, and some of them are actually causing a snowball effect. After more looking into it, my hypothesis is that our machinery for aggregating watchers in apiserver sometimes is just to slow. In particular, I some log in my local branch and what I'm observing is that in etcd_watcher, we are: accumulating objects in incoming channel but also accumulating objects in outgoing channel, and the accumulation there is usually higher [I will send a PR to put those logs into master - they seem to be pretty useful for debugging such issues.] So my current hypothesis is that dispatching watch events to watchers is what is too slow. We will continue debugging.
The logs that I seem to confirm my hypothesis - outgoing channel is generally more full then the incoming channel in etcd_watcher: vs I'm working on the PR that should at least improve the situation.
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
Please apply the label to this issue if it does not need to be fixed before we launch 1.4. Otherwise please advise on how to proceed with respect to 1.4.
This was a regression - I fixed it by reverting an offending PR: This seems to be fixed now.
I have 33 pods in my container (1 is substantive, the other 32 just run on log files). I get this all the time now. I'm running 1.5.4.
- to be honest, I don't undertsand what your wrote. what is "container" in "33 pods in my container"? Also, if you believe there is a bug somewhere, please open a new issue instead of adding comment to more than half-year-old one.

c.forget(false) c.stop() } <ins> glog.V(2).Infof("cacheWatcher add function blocked processing for %v", time.Since(startTime)) </ins> } func (c *cacheWatcher) sendWatchCacheEvent(event watchCacheEvent) {


Question:
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite} Previous issues for this test:
This is pretty much the same failure, that are happening in "gce-enormous" suite and we currently debugging them with The main reason of those failures is that we are terminating all watchers for a particular resource type from time to time, and some of them are actually causing a snowball effect. After more looking into it, my hypothesis is that our machinery for aggregating watchers in apiserver sometimes is just to slow. In particular, I some log in my local branch and what I'm observing is that in etcd_watcher, we are: accumulating objects in incoming channel but also accumulating objects in outgoing channel, and the accumulation there is usually higher [I will send a PR to put those logs into master - they seem to be pretty useful for debugging such issues.] So my current hypothesis is that dispatching watch events to watchers is what is too slow. We will continue debugging.
The logs that I seem to confirm my hypothesis - outgoing channel is generally more full then the incoming channel in etcd_watcher: vs I'm working on the PR that should at least improve the situation.
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
Please apply the label to this issue if it does not need to be fixed before we launch 1.4. Otherwise please advise on how to proceed with respect to 1.4.
This was a regression - I fixed it by reverting an offending PR: This seems to be fixed now.
I have 33 pods in my container (1 is substantive, the other 32 just run on log files). I get this all the time now. I'm running 1.5.4.
- to be honest, I don't undertsand what your wrote. what is "container" in "33 pods in my container"? Also, if you believe there is a bug somewhere, please open a new issue instead of adding comment to more than half-year-old one.

import ( "fmt" "net/http" <ins> "reflect" </ins> "sync" "sync/atomic" "time"


Question:
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite} Previous issues for this test:
This is pretty much the same failure, that are happening in "gce-enormous" suite and we currently debugging them with The main reason of those failures is that we are terminating all watchers for a particular resource type from time to time, and some of them are actually causing a snowball effect. After more looking into it, my hypothesis is that our machinery for aggregating watchers in apiserver sometimes is just to slow. In particular, I some log in my local branch and what I'm observing is that in etcd_watcher, we are: accumulating objects in incoming channel but also accumulating objects in outgoing channel, and the accumulation there is usually higher [I will send a PR to put those logs into master - they seem to be pretty useful for debugging such issues.] So my current hypothesis is that dispatching watch events to watchers is what is too slow. We will continue debugging.
The logs that I seem to confirm my hypothesis - outgoing channel is generally more full then the incoming channel in etcd_watcher: vs I'm working on the PR that should at least improve the situation.
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
Please apply the label to this issue if it does not need to be fixed before we launch 1.4. Otherwise please advise on how to proceed with respect to 1.4.
This was a regression - I fixed it by reverting an offending PR: This seems to be fixed now.
I have 33 pods in my container (1 is substantive, the other 32 just run on log files). I get this all the time now. I'm running 1.5.4.
- to be honest, I don't undertsand what your wrote. what is "container" in "33 pods in my container"? Also, if you believe there is a bug somewhere, please open a new issue instead of adding comment to more than half-year-old one.

// Injectable for testing. Send the event down the outgoing channel. emit func(watch.Event) <ins> // HighWaterMarks for performance debugging. incomingHWM HighWaterMark outgoingHWM HighWaterMark </ins> cache etcdCache }


Question:
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite} Previous issues for this test:
This is pretty much the same failure, that are happening in "gce-enormous" suite and we currently debugging them with The main reason of those failures is that we are terminating all watchers for a particular resource type from time to time, and some of them are actually causing a snowball effect. After more looking into it, my hypothesis is that our machinery for aggregating watchers in apiserver sometimes is just to slow. In particular, I some log in my local branch and what I'm observing is that in etcd_watcher, we are: accumulating objects in incoming channel but also accumulating objects in outgoing channel, and the accumulation there is usually higher [I will send a PR to put those logs into master - they seem to be pretty useful for debugging such issues.] So my current hypothesis is that dispatching watch events to watchers is what is too slow. We will continue debugging.
The logs that I seem to confirm my hypothesis - outgoing channel is generally more full then the incoming channel in etcd_watcher: vs I'm working on the PR that should at least improve the situation.
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
Please apply the label to this issue if it does not need to be fixed before we launch 1.4. Otherwise please advise on how to proceed with respect to 1.4.
This was a regression - I fixed it by reverting an offending PR: This seems to be fixed now.
I have 33 pods in my container (1 is substantive, the other 32 just run on log files). I get this all the time now. I'm running 1.5.4.
- to be honest, I don't undertsand what your wrote. what is "container" in "33 pods in my container"? Also, if you believe there is a bug somewhere, please open a new issue instead of adding comment to more than half-year-old one.

cancel:   nil, } w.emit = func(e watch.Event) { <ins> if curLen := int64(len(w.outgoing)); w.outgoingHWM.Update(curLen) { // Monitor if this gets backed up, and how much. glog.V(1).Infof("watch (%v): %v objects queued in outgoing channel.", reflect.TypeOf(e.Object).String(), curLen) } </ins> // Give up on user stop, without this we leak a lot of goroutines in tests. select { case w.outgoing <- e:


Question:
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite} Previous issues for this test:
This is pretty much the same failure, that are happening in "gce-enormous" suite and we currently debugging them with The main reason of those failures is that we are terminating all watchers for a particular resource type from time to time, and some of them are actually causing a snowball effect. After more looking into it, my hypothesis is that our machinery for aggregating watchers in apiserver sometimes is just to slow. In particular, I some log in my local branch and what I'm observing is that in etcd_watcher, we are: accumulating objects in incoming channel but also accumulating objects in outgoing channel, and the accumulation there is usually higher [I will send a PR to put those logs into master - they seem to be pretty useful for debugging such issues.] So my current hypothesis is that dispatching watch events to watchers is what is too slow. We will continue debugging.
The logs that I seem to confirm my hypothesis - outgoing channel is generally more full then the incoming channel in etcd_watcher: vs I'm working on the PR that should at least improve the situation.
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
Please apply the label to this issue if it does not need to be fixed before we launch 1.4. Otherwise please advise on how to proceed with respect to 1.4.
This was a regression - I fixed it by reverting an offending PR: This seems to be fixed now.
I have 33 pods in my container (1 is substantive, the other 32 just run on log files). I get this all the time now. I'm running 1.5.4.
- to be honest, I don't undertsand what your wrote. what is "container" in "33 pods in my container"? Also, if you believe there is a bug somewhere, please open a new issue instead of adding comment to more than half-year-old one.

incoming <- &copied } <del> var ( watchChannelHWM HighWaterMark ) </del> // translate pulls stuff from etcd, converts, and pushes out the outgoing channel. Meant to be // called as a goroutine. func (w *etcdWatcher) translate() {


Question:
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite} Previous issues for this test:
This is pretty much the same failure, that are happening in "gce-enormous" suite and we currently debugging them with The main reason of those failures is that we are terminating all watchers for a particular resource type from time to time, and some of them are actually causing a snowball effect. After more looking into it, my hypothesis is that our machinery for aggregating watchers in apiserver sometimes is just to slow. In particular, I some log in my local branch and what I'm observing is that in etcd_watcher, we are: accumulating objects in incoming channel but also accumulating objects in outgoing channel, and the accumulation there is usually higher [I will send a PR to put those logs into master - they seem to be pretty useful for debugging such issues.] So my current hypothesis is that dispatching watch events to watchers is what is too slow. We will continue debugging.
The logs that I seem to confirm my hypothesis - outgoing channel is generally more full then the incoming channel in etcd_watcher: vs I'm working on the PR that should at least improve the situation.
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
Please apply the label to this issue if it does not need to be fixed before we launch 1.4. Otherwise please advise on how to proceed with respect to 1.4.
This was a regression - I fixed it by reverting an offending PR: This seems to be fixed now.
I have 33 pods in my container (1 is substantive, the other 32 just run on log files). I get this all the time now. I'm running 1.5.4.
- to be honest, I don't undertsand what your wrote. what is "container" in "33 pods in my container"? Also, if you believe there is a bug somewhere, please open a new issue instead of adding comment to more than half-year-old one.

return case res, ok := <-w.etcdIncoming: if ok { <del> if curLen := int64(len(w.etcdIncoming)); watchChannelHWM.Update(curLen) { </del> <ins> if curLen := int64(len(w.etcdIncoming)); w.incomingHWM.Update(curLen) { </ins> // Monitor if this gets backed up, and how much. <del> glog.V(2).Infof("watch: %v objects queued in channel.", curLen) </del> <ins> glog.V(1).Infof("watch: %v objects queued in incoming channel.", curLen) </ins> } w.sendResult(res) }


Question:
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite} Previous issues for this test:
This is pretty much the same failure, that are happening in "gce-enormous" suite and we currently debugging them with The main reason of those failures is that we are terminating all watchers for a particular resource type from time to time, and some of them are actually causing a snowball effect. After more looking into it, my hypothesis is that our machinery for aggregating watchers in apiserver sometimes is just to slow. In particular, I some log in my local branch and what I'm observing is that in etcd_watcher, we are: accumulating objects in incoming channel but also accumulating objects in outgoing channel, and the accumulation there is usually higher [I will send a PR to put those logs into master - they seem to be pretty useful for debugging such issues.] So my current hypothesis is that dispatching watch events to watchers is what is too slow. We will continue debugging.
The logs that I seem to confirm my hypothesis - outgoing channel is generally more full then the incoming channel in etcd_watcher: vs I'm working on the PR that should at least improve the situation.
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
Please apply the label to this issue if it does not need to be fixed before we launch 1.4. Otherwise please advise on how to proceed with respect to 1.4.
This was a regression - I fixed it by reverting an offending PR: This seems to be fixed now.
I have 33 pods in my container (1 is substantive, the other 32 just run on log files). I get this all the time now. I'm running 1.5.4.
- to be honest, I don't undertsand what your wrote. what is "container" in "33 pods in my container"? Also, if you believe there is a bug somewhere, please open a new issue instead of adding comment to more than half-year-old one.

if res == nil { continue } <ins> if len(wc.resultChan) == outgoingBufSize { glog.Warningf("Fast watcher, slow processing. Number of buffered events: %d."+ "Probably caused by slow dispatching events to watchers", outgoingBufSize) } </ins> // If user couldn't receive results fast enough, we also block incoming events from watcher. // Because storing events in local will cause more memory usage. // The worst case would be closing the fast watcher.


Question:
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite} Previous issues for this test:
This is pretty much the same failure, that are happening in "gce-enormous" suite and we currently debugging them with The main reason of those failures is that we are terminating all watchers for a particular resource type from time to time, and some of them are actually causing a snowball effect. After more looking into it, my hypothesis is that our machinery for aggregating watchers in apiserver sometimes is just to slow. In particular, I some log in my local branch and what I'm observing is that in etcd_watcher, we are: accumulating objects in incoming channel but also accumulating objects in outgoing channel, and the accumulation there is usually higher [I will send a PR to put those logs into master - they seem to be pretty useful for debugging such issues.] So my current hypothesis is that dispatching watch events to watchers is what is too slow. We will continue debugging.
The logs that I seem to confirm my hypothesis - outgoing channel is generally more full then the incoming channel in etcd_watcher: vs I'm working on the PR that should at least improve the situation.
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
Please apply the label to this issue if it does not need to be fixed before we launch 1.4. Otherwise please advise on how to proceed with respect to 1.4.
This was a regression - I fixed it by reverting an offending PR: This seems to be fixed now.
I have 33 pods in my container (1 is substantive, the other 32 just run on log files). I get this all the time now. I'm running 1.5.4.
- to be honest, I don't undertsand what your wrote. what is "container" in "33 pods in my container"? Also, if you believe there is a bug somewhere, please open a new issue instead of adding comment to more than half-year-old one.

func (wc *watchChan) sendEvent(e *event) { if len(wc.incomingEventChan) == incomingBufSize { <del> glog.V(2).Infof("Fast watcher, slow processing. Number of buffered events: %d."+ </del> <ins> glog.Warningf("Fast watcher, slow processing. Number of buffered events: %d."+ </ins> "Probably caused by slow decoding, user not receiving fast, or other processing logic", incomingBufSize) }


Question:
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite} Happened on a presubmit run in . Previous issues for this test:
failed in https://k8s-
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite} Happened on a presubmit run in .
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
Any updates on this issue If you could you change the milestone to 1.5 and add a label for the responsible team that would be a great help as we triage flakes for the 1.5 release. Thanks!
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
There is a problem with fetching images on GKE. It seems that pod is unable to fetch image for more than 2 minutes. It is strange, as AFAIK the image should be pre-fetched by image-puller:
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
, all issues must be labeled either release blocker or non release blocking by end of day 18 November 2016 PST. If you are not the correct owner please identify the correct owner and add them to this thread for assignment. Thanks for your help. cc:
WRT https://k8s- There was a problem w/removing namespace. This is a duplicate of and is not related to HPA.
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
Could you please help triage this issue as blocking or non-blocking for 1.5 (or help find someone who can). If there is a real issue here, we need to find out ASAP, or else we risk the 1.5 release date.
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
WRT https://k8s- build: kubelet logs for pod rc-light-qyowb : Kubelet is not starting pod for 2 minutes, reason unknown. The same problem is in issue Can you take a look or assign someone from kubelet team?
I suspect that the logging level is set to in GKE tests and so most of the useful logs are not available. Let me figure out a means to bump the logging level to further debug this issue. you might still have to triage the non v1.4 release failures to address concern .
how can we update the environment variables for GKE test clusters? For OSS, we up the kubelet logging for test clusters
I still don't agree with test environmental changes, which is one of the reasons we've never made this a priority. A GKE test cluster looks the same as a GKE prod cluster. How would you debug this in production? Always ask that, and you come back to test == prod.
cc
In my opinion it is release blocker: sometimes pods are not started
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
The pod startup failure you posted in is for release 1.4 branch. Why do you think it is a release blocker? We need to triage other failures on HEAD and/or 1.5 branch which could be release blockers. If the intent is to keep prod and test in sync, then let's up the logging level in prod as well. When issues happen in prod, we might need those missing log lines anyways. I suspect that we might have to have GCI rotate logs more aggressively as a side effect.
Why is the argument to increase the log level rather than making sure that all logs necessary for debugging production issues are at level 2 or above? Levels 3/4 shouldn't be necessary to figure out why a key control loop isn't behaving correctly.
What you suggest is a good fix for the future. We need to reexamine logging levels across kubelet. However, that might not happen in the next few days. How do we make progress now?
Would isolating logging level changes to specific test suites be acceptable? That way, most of our tests would match production setup.
That's what I'd suggested on the other (internal) code review.
Do we have consensus on how to get past this as a release-blocker?
We are waiting on to give a go ahead from the release perspective. The kubelet failures that and I have been discussing about have not been on v1.5 AFAIK.
Sorry, I didn't notice that this problem is on 1.4 branch. If so, the issue is not release-blocker. So, I'm removing the label.
Ack. Marking as non-release-blocker
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
Is it appropriate to move this to the next milestone? (and remove the non-release-blocker tag as well)
PR should be cherry-picked to 1.5 This is not a release-blocker as it affects e2e tests only.
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}

} framework.ExpectNoError(framework.RunRC(controllerRcConfig)) <del> // Make sure endpoints are propagated. // TODO(piosz): replace sleep with endpoints watch. time.Sleep(10 * time.Second) </del> <ins> // Wait for endpoints to propagate for the controller service. framework.ExpectNoError(framework.WaitForServiceEndpointsNum( c, ns, controllerName, 1, startServiceInterval, startServiceTimeout)) </ins> }


Question: https://k8s- Failed: BeforeSuite {Kubernetes e2e suite} Previous issues for this test:

} // errorBadPodsStates create error message of basic info of bad pods for debugging. <del> func errorBadPodsStates(badPods []api.Pod, desiredPods int, ns string, timeout time.Duration) string { errStr := fmt.Sprintf("%d / %d pods in namespace %q are NOT in the desired state in %vn", len(badPods), desiredPods, ns, timeout) </del> <ins> func errorBadPodsStates(badPods []api.Pod, desiredPods int, ns, desiredState string, timeout time.Duration) string { errStr := fmt.Sprintf("%d / %d pods in namespace %q are NOT in %s state in %vn", len(badPods), desiredPods, ns, desiredState, timeout) </ins> // Pirnt bad pods info only if there are fewer than 10 bad pods if len(badPods) > 10 { return errStr + "There are too many bad pods. Please check log for details."


Question: https://k8s- Failed: BeforeSuite {Kubernetes e2e suite} Previous issues for this test:

// pods have been created. func WaitForPodsSuccess(c *client.Client, ns string, successPodLabels map[string]string, timeout time.Duration) error { successPodSelector := labels.SelectorFromSet(successPodLabels) <del> start, badPods := time.Now(), []api.Pod{} </del> <ins> start, badPods, desiredPods := time.Now(), []api.Pod{}, 0 </ins> if wait.PollImmediate(30*time.Second, timeout, func() (bool, error) { podList, err := c.Pods(ns).List(api.ListOptions{LabelSelector: successPodSelector})


Question: https://k8s- Failed: BeforeSuite {Kubernetes e2e suite} Previous issues for this test:

return true, nil } badPods = []api.Pod{} <ins> desiredPods = len(podList.Items) </ins> for _, pod := range podList.Items { if pod.Status.Phase != api.PodSucceeded { badPods = append(badPods, pod)


Question: https://k8s- Failed: BeforeSuite {Kubernetes e2e suite} Previous issues for this test:

}) != nil { logPodStates(badPods) LogPodsWithLabels(c, ns, successPodLabels) <del> return fmt.Errorf("Not all pods in namespace %q are successful within %v", ns, timeout) </del> <ins> return errors.New(errorBadPodsStates(badPods, desiredPods, ns, "SUCCESS", timeout)) </ins> } return nil }


Question: https://k8s- Failed: BeforeSuite {Kubernetes e2e suite} Previous issues for this test:

logPodStates(badPods) return false, nil }) != nil { <del> return errors.New(errorBadPodsStates(badPods, desiredPods, ns, timeout)) </del> <ins> return errors.New(errorBadPodsStates(badPods, desiredPods, ns, "RUNNING and READY", timeout)) </ins> } wg.Wait() if waitForSuccessError != nil {


Question:
Another flake i found earlier yesterday, just linking to it for posterity, not blocking me right now. TestContainerStatus
Could you post some error log here? I can't reproduce it with stress test
Hm, got one reproduce:

} // Create the container. <del> fClock.SetTime(time.Now()) </del> <ins> fClock.SetTime(time.Now().Add(-1 * time.Hour)) </ins> *expected.CreatedAt = fClock.Now().Unix() id, err := ds.CreateContainer("sandboxid", config, sConfig) // Set the id manually since we don't know the id until it's created.


Question:
Another flake i found earlier yesterday, just linking to it for posterity, not blocking me right now. TestContainerStatus
Could you post some error log here? I can't reproduce it with stress test
Hm, got one reproduce:

assert.Equal(t, expected, status) // Advance the clock and stop the container. <del> fClock.SetTime(time.Now()) </del> <ins> fClock.SetTime(time.Now().Add(1 * time.Hour)) </ins> *expected.FinishedAt = fClock.Now().Unix() *expected.State = runtimeApi.ContainerState_EXITED *expected.Reason = "Completed"


Question:
Another flake i found earlier yesterday, just linking to it for posterity, not blocking me right now. TestContainerStatus
Could you post some error log here? I can't reproduce it with stress test
Hm, got one reproduce:

) func newTestDockerSevice() (*dockerService, *dockertools.FakeDockerClient, *clock.FakeClock) { <del> c := dockertools.NewFakeDockerClient() </del> fakeClock := clock.NewFakeClock(time.Time{}) <ins> c := dockertools.NewFakeDockerClientWithClock(fakeClock) </ins> return &dockerService{client: c}, c, fakeClock }


Question:
Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"4+", GitVersion:"v1.4.0-beta.11", GitCommit:"", GitTreeState:"clean", BuildDate:"2016-09-23T22:53:01Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2016-09-26T18:10:32Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"} Environment: AWS - CentOS 7.2 x8664 with cloud-init (HVM) (ami-) OS (e.g. from /etc/os-release): NAME="CentOS Linux" VERSION="7 (Core)" ID="centos" IDLIKE="rhel fedora" VERSIONID="7" PRETTYNAME="CentOS Linux 7 (Core)" ANSICOLOR="0;31" CPENAME="cpe:/o:centos:centos:7" HOMEURL="" BUGREPORTURL="" CENTOSMANTISBTPROJECT="CentOS-7" CENTOSMANTISBTPROJECTVERSION="7" REDHATSUPPORTPRODUCT="centos" REDHATSUPPORTPRODUCTVERSION="7" Linux ip-172-31-35-103 3.10.0-327.el7.x8664 SMP Thu Nov 19 22:10:57 UTC 2015 x8664 x8664 x86_64 GNU/Linux Install tools: Others: What happened: when selinux is enabled i was getting following error on discovery container What you expected to happen: i expected node to be able to join master, but nothing was running on 9898 port of master i.e. discovery due the error mentioned above How to reproduce it (as minimally and precisely as possible): use the api which i gave you and do normal kubeadm init Anything else do we need to know: This issue got fixed with setenforce 0
This issue got fixed with setenforce 0 but want to track it
cc
BTW the issue was on latest Centos iso too downloaded from here
cc
this is to do with secrets, and kubelet log is the place to look for more info, could you please grab and paste it in a gist (it's gonna be fairly large)? Also, the kernel audit log would be handy, do you know how to get this (I forgot)?
not sure if this will help
the lines are incomplete, it's best if you redirect it to a file and upload it somewhere with a link here.
could you also provide the output of ?

// `HostIP: s.API.AdvertiseAddrs[0]`, if there is only one address` {Name: "http", ContainerPort: 9898, HostPort: 9898}, }, <ins> SecurityContext: &api.SecurityContext{ SELinuxOptions: &api.SELinuxOptions{ // TODO: This implies our discovery container is not being restricted by // SELinux. This is not optimal and would be nice to adjust in future // so it can read /tmp/secret, but for now this avoids recommending // setenforce 0 system-wide. Type: "unconfined_t", }, }, </ins> }}, Volumes: []api.Volume{{ Name: kubeDiscoverySecretName,


Question:
Kubernetes version (use ): Environment: Cloud provider or hardware configuration: Any OS (e.g. from /etc/os-release): Any Kernel (e.g. ): Any Install tools: Any What happened: If I use a template with the seperator, and if the line with the separator has trailing whitespace characters, then kubectl considers that to be the EOF instead of reading further. e.g. template What you expected to happen: It should not stop reading if there are trailing whitespace characters, it should just trim the separator line and ignore the whitespaces after the . This is how it worked until v1.3.0 kubectl version. How to reproduce it (as minimally and precisely as possible): Just use the template from above and you'll notice that it won't create the Service, but it'll create the Replication controller. Anything else do we need to know: This is on kubectl version 1.4.0
kubectl regression. cc
This bug is general (can be found on get, apply, delete, ...). May be somewhere in resource builder.
Might be , Janet Kuo wrote:
Confirmed - is the cause. would you take a look?
We may need to cherrypick the fix
Yaml rules are separator must be at beginning of line, three dashes, and new line. Did we change how strict we were in that PR? On Sep 27, 2016, at 7:46 PM, Janet Kuo wrote: We may need to cherrypick the fix — You are receiving this because you are on a team that was mentioned. Reply to this email directly, view it on GitHub , or mute the thread .
Yes, and the test was changed too (originally tests allow with trailing spaces)
According to spec (if my spec-alese is correct) newline + three dashes is the only requirement to terminate a document. Whitespace or prefixes are allowed afterwards, but for it to be valid YAML those must be valid at the beginning of a doc as well (as far as I can tell). So we should split and remove on any line starting with a newline and three dashes - but not consume the remaining whitespace (I believe) , Janet Kuo wrote:
We should probably exit non-0 and have an error message if we see something in the file that we don't expect. E.g. is neither a separator nor parseable yaml.
We should add a test that uses a YAML document prefix rule and verify that it correctly passes / fails when YAML goes to parse. From reading the spec it appears that anything that is invalid when n--- is present is also invalid in a single document context, and vice versa. , Phillip Wittrock wrote:
I have rechecked the code, and I have located the root cause, since I am outing for a training these days, I will fix this right now after the training on Friday.
I am quite sorry about the regression. :(
should we allow trailing whitespace characters after , if so we can trim the whitespaces when we read line by line?
is it possible to include this fix to one of the 1.4 releases? We'd really like our users to have of k8s alpha features (Scheduled jobs is one of them) that can be configured only using 1.4 client; w/o them potentially hitting this regression.
I will double check this regression, and open a cherry-pick PR.
I am thinking create a cherry-pick PR for this to include it in 1.4, wdyt?
I would like to see this in 1.4 since as you note it may break scripts. On Oct 19, 2016, at 8:19 PM, TonyAdo wrote: I am thinking create a cherry-pick PR for this to include it in 1.4, wdyt? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub , or mute the thread .
Do we have a test case now? :) On Oct 19, 2016, at 8:19 PM, TonyAdo wrote: I am thinking create a cherry-pick PR for this to include it in 1.4, wdyt? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub , or mute the thread .
we already test this, you can see

} const yamlSeparator = "n---" <del> const separator = "---n" </del> <ins> const separator = "---" </ins> // splitYAMLDocument is a bufio.SplitFunc for splitting YAML streams into individual documents. func splitYAMLDocument(data []byte, atEOF bool) (advance int, token []byte, err error) {


Question:
Kubernetes version (use ): Environment: Cloud provider or hardware configuration: Any OS (e.g. from /etc/os-release): Any Kernel (e.g. ): Any Install tools: Any What happened: If I use a template with the seperator, and if the line with the separator has trailing whitespace characters, then kubectl considers that to be the EOF instead of reading further. e.g. template What you expected to happen: It should not stop reading if there are trailing whitespace characters, it should just trim the separator line and ignore the whitespaces after the . This is how it worked until v1.3.0 kubectl version. How to reproduce it (as minimally and precisely as possible): Just use the template from above and you'll notice that it won't create the Service, but it'll create the Replication controller. Anything else do we need to know: This is on kubectl version 1.4.0
kubectl regression. cc
This bug is general (can be found on get, apply, delete, ...). May be somewhere in resource builder.
Might be , Janet Kuo wrote:
Confirmed - is the cause. would you take a look?
We may need to cherrypick the fix
Yaml rules are separator must be at beginning of line, three dashes, and new line. Did we change how strict we were in that PR? On Sep 27, 2016, at 7:46 PM, Janet Kuo wrote: We may need to cherrypick the fix — You are receiving this because you are on a team that was mentioned. Reply to this email directly, view it on GitHub , or mute the thread .
Yes, and the test was changed too (originally tests allow with trailing spaces)
According to spec (if my spec-alese is correct) newline + three dashes is the only requirement to terminate a document. Whitespace or prefixes are allowed afterwards, but for it to be valid YAML those must be valid at the beginning of a doc as well (as far as I can tell). So we should split and remove on any line starting with a newline and three dashes - but not consume the remaining whitespace (I believe) , Janet Kuo wrote:
We should probably exit non-0 and have an error message if we see something in the file that we don't expect. E.g. is neither a separator nor parseable yaml.
We should add a test that uses a YAML document prefix rule and verify that it correctly passes / fails when YAML goes to parse. From reading the spec it appears that anything that is invalid when n--- is present is also invalid in a single document context, and vice versa. , Phillip Wittrock wrote:
I have rechecked the code, and I have located the root cause, since I am outing for a training these days, I will fix this right now after the training on Friday.
I am quite sorry about the regression. :(
should we allow trailing whitespace characters after , if so we can trim the whitespaces when we read line by line?
is it possible to include this fix to one of the 1.4 releases? We'd really like our users to have of k8s alpha features (Scheduled jobs is one of them) that can be configured only using 1.4 client; w/o them potentially hitting this regression.
I will double check this regression, and open a cherry-pick PR.
I am thinking create a cherry-pick PR for this to include it in 1.4, wdyt?
I would like to see this in 1.4 since as you note it may break scripts. On Oct 19, 2016, at 8:19 PM, TonyAdo wrote: I am thinking create a cherry-pick PR for this to include it in 1.4, wdyt? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub , or mute the thread .
Do we have a test case now? :) On Oct 19, 2016, at 8:19 PM, TonyAdo wrote: I am thinking create a cherry-pick PR for this to include it in 1.4, wdyt? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub , or mute the thread .
we already test this, you can see

return nil, err } <del> if string(line) == separator || err == io.EOF { </del> <ins> sep := len([]byte(separator)) if i := bytes.Index(line, []byte(separator)); i == 0 { // We have a potential document terminator i += sep after := line[i:] if len(strings.TrimRightFunc(string(after), unicode.IsSpace)) == 0 { if buffer.Len() != 0 { return buffer.Bytes(), nil } if err == io.EOF { return nil, err } } } if err == io.EOF { </ins> if buffer.Len() != 0 { <ins> // If we're at EOF, we have a final, non-terminated line. Return it. </ins> return buffer.Bytes(), nil } <del> if err == io.EOF { return nil, err } } else { buffer.Write(line) </del> <ins> return nil, err </ins> } <ins> buffer.Write(line) </ins> } }


Question:
Kubernetes version (use ): Environment: Cloud provider or hardware configuration: Any OS (e.g. from /etc/os-release): Any Kernel (e.g. ): Any Install tools: Any What happened: If I use a template with the seperator, and if the line with the separator has trailing whitespace characters, then kubectl considers that to be the EOF instead of reading further. e.g. template What you expected to happen: It should not stop reading if there are trailing whitespace characters, it should just trim the separator line and ignore the whitespaces after the . This is how it worked until v1.3.0 kubectl version. How to reproduce it (as minimally and precisely as possible): Just use the template from above and you'll notice that it won't create the Service, but it'll create the Replication controller. Anything else do we need to know: This is on kubectl version 1.4.0
kubectl regression. cc
This bug is general (can be found on get, apply, delete, ...). May be somewhere in resource builder.
Might be , Janet Kuo wrote:
Confirmed - is the cause. would you take a look?
We may need to cherrypick the fix
Yaml rules are separator must be at beginning of line, three dashes, and new line. Did we change how strict we were in that PR? On Sep 27, 2016, at 7:46 PM, Janet Kuo wrote: We may need to cherrypick the fix — You are receiving this because you are on a team that was mentioned. Reply to this email directly, view it on GitHub , or mute the thread .
Yes, and the test was changed too (originally tests allow with trailing spaces)
According to spec (if my spec-alese is correct) newline + three dashes is the only requirement to terminate a document. Whitespace or prefixes are allowed afterwards, but for it to be valid YAML those must be valid at the beginning of a doc as well (as far as I can tell). So we should split and remove on any line starting with a newline and three dashes - but not consume the remaining whitespace (I believe) , Janet Kuo wrote:
We should probably exit non-0 and have an error message if we see something in the file that we don't expect. E.g. is neither a separator nor parseable yaml.
We should add a test that uses a YAML document prefix rule and verify that it correctly passes / fails when YAML goes to parse. From reading the spec it appears that anything that is invalid when n--- is present is also invalid in a single document context, and vice versa. , Phillip Wittrock wrote:
I have rechecked the code, and I have located the root cause, since I am outing for a training these days, I will fix this right now after the training on Friday.
I am quite sorry about the regression. :(
should we allow trailing whitespace characters after , if so we can trim the whitespaces when we read line by line?
is it possible to include this fix to one of the 1.4 releases? We'd really like our users to have of k8s alpha features (Scheduled jobs is one of them) that can be configured only using 1.4 client; w/o them potentially hitting this regression.
I will double check this regression, and open a cherry-pick PR.
I am thinking create a cherry-pick PR for this to include it in 1.4, wdyt?
I would like to see this in 1.4 since as you note it may break scripts. On Oct 19, 2016, at 8:19 PM, TonyAdo wrote: I am thinking create a cherry-pick PR for this to include it in 1.4, wdyt? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub , or mute the thread .
Do we have a test case now? :) On Oct 19, 2016, at 8:19 PM, TonyAdo wrote: I am thinking create a cherry-pick PR for this to include it in 1.4, wdyt? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub , or mute the thread .
we already test this, you can see

s := NewYAMLToJSONDecoder(bytes.NewReader([]byte(`--- stuff: 1 <del> --- </del> <ins> --- </ins> `))) obj := generic{} if err := s.Decode(&obj); err != nil {


Question:
Kubernetes version (use ): Environment: Cloud provider or hardware configuration: Any OS (e.g. from /etc/os-release): Any Kernel (e.g. ): Any Install tools: Any What happened: If I use a template with the seperator, and if the line with the separator has trailing whitespace characters, then kubectl considers that to be the EOF instead of reading further. e.g. template What you expected to happen: It should not stop reading if there are trailing whitespace characters, it should just trim the separator line and ignore the whitespaces after the . This is how it worked until v1.3.0 kubectl version. How to reproduce it (as minimally and precisely as possible): Just use the template from above and you'll notice that it won't create the Service, but it'll create the Replication controller. Anything else do we need to know: This is on kubectl version 1.4.0
kubectl regression. cc
This bug is general (can be found on get, apply, delete, ...). May be somewhere in resource builder.
Might be , Janet Kuo wrote:
Confirmed - is the cause. would you take a look?
We may need to cherrypick the fix
Yaml rules are separator must be at beginning of line, three dashes, and new line. Did we change how strict we were in that PR? On Sep 27, 2016, at 7:46 PM, Janet Kuo wrote: We may need to cherrypick the fix — You are receiving this because you are on a team that was mentioned. Reply to this email directly, view it on GitHub , or mute the thread .
Yes, and the test was changed too (originally tests allow with trailing spaces)
According to spec (if my spec-alese is correct) newline + three dashes is the only requirement to terminate a document. Whitespace or prefixes are allowed afterwards, but for it to be valid YAML those must be valid at the beginning of a doc as well (as far as I can tell). So we should split and remove on any line starting with a newline and three dashes - but not consume the remaining whitespace (I believe) , Janet Kuo wrote:
We should probably exit non-0 and have an error message if we see something in the file that we don't expect. E.g. is neither a separator nor parseable yaml.
We should add a test that uses a YAML document prefix rule and verify that it correctly passes / fails when YAML goes to parse. From reading the spec it appears that anything that is invalid when n--- is present is also invalid in a single document context, and vice versa. , Phillip Wittrock wrote:
I have rechecked the code, and I have located the root cause, since I am outing for a training these days, I will fix this right now after the training on Friday.
I am quite sorry about the regression. :(
should we allow trailing whitespace characters after , if so we can trim the whitespaces when we read line by line?
is it possible to include this fix to one of the 1.4 releases? We'd really like our users to have of k8s alpha features (Scheduled jobs is one of them) that can be configured only using 1.4 client; w/o them potentially hitting this regression.
I will double check this regression, and open a cherry-pick PR.
I am thinking create a cherry-pick PR for this to include it in 1.4, wdyt?
I would like to see this in 1.4 since as you note it may break scripts. On Oct 19, 2016, at 8:19 PM, TonyAdo wrote: I am thinking create a cherry-pick PR for this to include it in 1.4, wdyt? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub , or mute the thread .
Do we have a test case now? :) On Oct 19, 2016, at 8:19 PM, TonyAdo wrote: I am thinking create a cherry-pick PR for this to include it in 1.4, wdyt? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub , or mute the thread .
we already test this, you can see

obj := generic{} err := s.Decode(&obj) if err == nil { <del> t.Fatal("expected error with yaml: prefix, got no error") </del> <ins> t.Fatal("expected error with yaml: violate, got no error") </ins> } fmt.Printf("err: %sn", err.Error()) <del> if !strings.HasPrefix(err.Error(), "yaml: line 1:") { t.Fatalf("expected %q to have 'yaml: line 1:' prefix", err.Error()) </del> <ins> if !strings.HasPrefix(err.Error(), "yaml: line 2:") { t.Fatalf("expected %q to have 'yaml: line 2:' found a tab character", err.Error()) </ins> } }


Question:
Just saw this warning today: Not sure if this is only limited to kubemark, or if there are other places we need to fix, too.
Sorry if this sounds stupid, but as expected, won't show this warning - what needs to be fixed here?
we have a bunch of scripts and Makefiles which are omitting the currently. We need to fix these by adding the missing s.

docker build -t $(PREFIX)/logs-generator:$(TAG) . push: <del> gcloud docker push $(PREFIX)/logs-generator:$(TAG) </del> <ins> gcloud docker -- push $(PREFIX)/logs-generator:$(TAG) </ins> clean: rm -f logs-generator  No newline at end of file


Question:
See
CI has built and uploaded unofficial prerelease darwin binaries for 1.4 using go1.7.1: cc Official binaries will ship as soon as we cut 1.4.1.
We have official v1.4.1 binaries available now, too.
Hooray

RUN mkdir -p ${K8S_PATCHED_GOROOT}  && curl -sSL https://github.com/golang/go/archive/go${K8S_PATCHED_GOLANG_VERSION}.tar.gz | tar -xz -C ${K8S_PATCHED_GOROOT} --strip-components=1 <ins> # We need a patched go1.7.1 for linux/arm (https://github.com/kubernetes/kubernetes/issues/29904) # We need go1.7.1 for all darwin builds (https://github.com/kubernetes/kubernetes/issues/32999) </ins> COPY golang-patches/CL28857-go1.7.1-luxas.patch ${K8S_PATCHED_GOROOT}/ RUN cd ${K8S_PATCHED_GOROOT}  && patch -p1 < CL28857-go1.7.1-luxas.patch  && cd src  && GOROOT_FINAL=${K8S_PATCHED_GOROOT} GOROOT_BOOTSTRAP=/usr/local/go ./make.bash  <del> && for platform in linux/arm; do GOOS=${platform%/*} GOARCH=${platform##*/} GOROOT=${K8S_PATCHED_GOROOT} go install std; done </del> <ins> && for platform in linux/arm darwin/386 darwin/amd64; do GOOS=${platform%/*} GOARCH=${platform##*/} GOROOT=${K8S_PATCHED_GOROOT} go install std; done </ins>


Question:
See
CI has built and uploaded unofficial prerelease darwin binaries for 1.4 using go1.7.1: cc Official binaries will ship as soon as we cut 1.4.1.
We have official v1.4.1 binaries available now, too.
Hooray

<del> v1.6.3-7 </del> <ins> v1.6.3-8 </ins>


Question:
See
CI has built and uploaded unofficial prerelease darwin binaries for 1.4 using go1.7.1: cc Official binaries will ship as soon as we cut 1.4.1.
We have official v1.4.1 binaries available now, too.
Hooray

if [[ ${platform} == "linux/arm" ]]; then export CGO_ENABLED=1 export CC=arm-linux-gnueabi-gcc <ins> # See https://github.com/kubernetes/kubernetes/issues/29904 </ins> export GOROOT=${K8S_PATCHED_GOROOT} elif [[ ${platform} == "linux/arm64" ]]; then export CGO_ENABLED=1


Question:
See
CI has built and uploaded unofficial prerelease darwin binaries for 1.4 using go1.7.1: cc Official binaries will ship as soon as we cut 1.4.1.
We have official v1.4.1 binaries available now, too.
Hooray

elif [[ ${platform} == "linux/ppc64le" ]]; then export CGO_ENABLED=1 export CC=powerpc64le-linux-gnu-gcc <ins> elif [[ ${platform} == "darwin/"* ]]; then # See https://github.com/kubernetes/kubernetes/issues/32999 export GOROOT=${K8S_PATCHED_GOROOT} </ins> fi fi }


Question:
See
CI has built and uploaded unofficial prerelease darwin binaries for 1.4 using go1.7.1: cc Official binaries will ship as soon as we cut 1.4.1.
We have official v1.4.1 binaries available now, too.
Hooray


Question:
See
CI has built and uploaded unofficial prerelease darwin binaries for 1.4 using go1.7.1: cc Official binaries will ship as soon as we cut 1.4.1.
We have official v1.4.1 binaries available now, too.
Hooray


Question:
See
CI has built and uploaded unofficial prerelease darwin binaries for 1.4 using go1.7.1: cc Official binaries will ship as soon as we cut 1.4.1.
We have official v1.4.1 binaries available now, too.
Hooray


Question:
See
CI has built and uploaded unofficial prerelease darwin binaries for 1.4 using go1.7.1: cc Official binaries will ship as soon as we cut 1.4.1.
We have official v1.4.1 binaries available now, too.
Hooray


Question:
<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): No What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): watch leak Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): 0 Environment: Cloud provider or hardware configuration: vagrant on Mac OS X OS (e.g. from /etc/os-release): CoreOS 1164.1.0 Kernel (e.g. ): Install tools: Others: What happened: We ran controller which handles our own ThirdPartyResource. When we hit "watch of ... ended with: 401: The event in requested index is outdated and cleared (the requested history has been cleared" error, "forcing resync" log is duplicated. This log message comes from . It looks like this goroutine does not exit when we hit the above mentioned error, and continued to work, and running "resync" more frequently. What you expected to happen: The resync goroutine must be terminated when watch is closed (i.e., ListAndWatch function returned), instead of leaking it. How to reproduce it (as minimally and precisely as possible): I just run our controller for ~15 minutes. I'm not sure why "401: The event in requested index is outdated and cleared (the requested history has been cleared" happens, but from the issues I read, it should be harmless. Anything else do we need to know:
I think I see what is going on here. If exits for any reason beyond an error from the resync loop no one will tell the resync loop to exit. Furthermore, there is no "local" way to do this as it would be unexpected for this function to signal the Done channel as other things outside of it may be relying on it. There are some useful functions around Context () that help to create a "local" copy of context that can be used in this situation.
Thank you for reply. Isn't is possible to create local channel in ListAndWatch, and defer close it, and then make resync goroutine "select" it too?
This issue was labelled as component/kubectl, but actually it is not limited to kubectl. All controllers suffer from this. Only people notice this if they enable V(4) logging.
The really annoying thing caused by this issue is that eventually resync is done in every few seconds due to the leaked goroutine. Even if you set 30 second resync period in controller, leaked goroutines call resync too, and you end up having resync call very very frequently. If you have many resources in cache, it really performs unnecessary sync loop. This is the last thing you want.
this looks like a real issue. Could be related to Could you help triage? looks like a proper fix.
I agree - this looks like a real issue. And indeed LGTM as a fix for it.

r.setLastSyncResourceVersion(resourceVersion) resyncerrc := make(chan error, 1) <ins> cancelCh := make(chan struct{}) defer close(cancelCh) </ins> go func() { for { select { case <-resyncCh: case <-stopCh: return <ins> case <-cancelCh: return </ins> } glog.V(4).Infof("%s: forcing resync", r.name) if err := r.store.Resync(); err != nil {


Question:
Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Master Environment: Cloud provider or hardware configuration: OpenStack OS (e.g. from /etc/os-release): n/a Kernel (e.g. ): n/a Install tools: n/a Others: What happened: now can update a LB instead of deleting it and recreating it. However the security group code is still set to try and create security group rules, and if it gets an error, delete the LB and start over. What you expected to happen: Updating a service does not delete the LBaaS LB How to reproduce it (as minimally and precisely as possible): Enable in the section of Restart a Anything else do we need to know:
you're absolutely right, this is a critical issue (if you enable security group management). The dangerousness of aggressive cleanups has already been discussed in detail in this thread: I hope you guys can fix this.
Yup - I have a larger refactor, that ensures this, and a few other edge cases we have found in our development. It is in massive need of a rebase though, so I will need a day or 2 of uninterrupted time to get it together
Are you still working on this? If not, I can take a look.
ping :)
/close
We just ran into this where Kubernetes recreated an LB with a new Floating IP because the OpenStack API was not available for a few seconds while the LB was updated. Could this issue be re-opened? Since this is a major bug for us, we will start working on a fix for this today.
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
There are no sig labels on this issue. Please add a sig label by either: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. See the . The in method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

if lbaas.opts.ManageSecurityGroups { err := lbaas.ensureSecurityGroup(clusterName, apiService, nodes, loadbalancer) if err != nil { <del> // cleanup what was created so far _ = lbaas.EnsureLoadBalancerDeleted(ctx, clusterName, apiService) return status, err </del> <ins> return status, fmt.Errorf("Error reconciling security groups for LB service %v/%v: %v", apiService.Namespace, apiService.Name, err) </ins> } }


Question:
PetSets currently support an annotation in their PodTemplateSpec field. From the user documentation: Debugging: If one want to bring up the pets in my petset one after the other, while pausing after each step, confirming that the state is as expected, one may use this functionality. For example, a petset where each pet is a replica of a distributed database; we set the to in the petset template initially. This leads to the following progression: PetSetController creates pet-0 Manually intervene and check that it has initialized as expected. Set annotation to true (equivalent to marking that pet as healthy) PetSetController creates pet-1 Manually intervene and check that it has initialized as expected. ... Blocking the PetSet controller: We may want to ensure that the petset controller is not performing scale operations when a different user-initiated action is in progress (for example, where pets are leaving due to a kubectl drain). This may be used as a mechanism to orphan a pod from a petset controller (for a closer look), while ensuring that a replacement is not created. The use case 1 above can be realized using and is probably easier to do that way rather than use the annotation. I don't see a reason to have that case supported when we can have users scale up a PetSet gradually and manually in an identical manner. It may have been of more importance before v1.4 when the scaler & reaper were not implemented. Due to the second use-case, if it would make sense to continue to have this ability, I propose that we move it from the PodTemplateSpec to the PetSetSpec, where it will eventually reside. This seems to me, like the logical place for such a debug hook to reside. In order to run/pause the scaling operations of the PetSet controller, we could have annotations along the lines of with the annotation defaulting to when absent. /cc
Should we just support Pause on PetSet (for part of these cases)?
Do users have to catch the roll out at the right time in order for this to work? For example if I want to pause between nodes 2 and 3 coming up, do I need to be precise in my timing when editing the resource? How does this look from the users perspective?
Note that Pause allows scaling in Deployments, it just blocks upgrades. That being said, I think Pause will be useful for PetSets in general. Not sure though if we can use it somehow for this specific issue.
I like the debugging hook for PetSets. If it's optional I don't see any reason to get rid of it.
pauses petset controller if necessary and inspects the state of the system_ With the previous flow, the petset controller was paused after each pet's creation automatically, and here it would be an extra step to pause the controller it if necessary. I don't know how important it is though, but to me it seems easier to understand than a per-pod annotation.
and I had several discussions with the lifecycle of what would be useful. There are a bunch of use cases that overlap: not a primary concern for most users but helpful (prevent changes) - very important if you are triaging failures. Technically scaling is itself a user action, so if you don't want your pet set autoscaled while triaging failures you need to take action on the HPA like deployments. Since PetSets and ReplicaSets are nothing alike w.r.t. scaling I'm not terribly concerned that pause means slightly different things if for end users it Does The Right Thing control by a third party controller - this is the babysitter controller wanting to reuse the petset controller for core stuff but still have the hook angle. Not thought through enough to design around.
Note that Pause does not prevent scaling in Deployments, only upgrades , Clayton Coleman wrote:
Just to summarize: The current behavior with the annotation It is set on a per-pet basis. Setting it to false on any single pod blocks all operations on the petset; scaling, creation of new pets, etc. All pets must contain the annotation, and have its value set to true in order to make progress. By default, it is set to false and makes the petset controller pause after creating one pet. The smallest change we can make for beta: Rename it to (or some better name) Make it optional (with the absence of the annotation meaning that the petset can scale) Surface the reason better when we block on scaling because of the annotation. I still think there is value in moving it to be an annotation on the PetSet resource, rather than having it be in each pod, which is confusing to users. wdyt?
Putting it on the pod template is clever, but surprising. How many examples are dependent on it? , Anirudh Ramanathan < :
Last I checked, all the examples had set to true. We should just get rid of it from all examples in my opinion. If we make the default (or the absence of any such annotation) to mean that the petset controller can make progress, we can safely do that. was going to do a pass over the examples after the PetSet renaming. This could be an additional step. Does that sound ok?
If it's not found, we assume you want to pause. You're proposing that if it's not found, we default to "forge ahead". SG. It should only be used in debugging/testing today. Basically if you don't know how to write your , you'd create a petset with it set, kubectl exec and write it out, scale the petset by 1, kubectl exec ..., and when you have a working cluster, put what you did in an init container and create the petset without it. This is the absolute min. You're also proposing that we move it to the petset. This makes sense overall, but I initially didn't do that because I didn't know how we would end up doing pause.
We could leave it there as "alpha" and then remove it in 1.6 once we have more time to talk about how upgrades and debugging and pause should work? , Prashanth B wrote:
Yeah the minimum work is to flip the behavior of the controller when the annotation is not found:
Agree on flipping behavior. , Prashanth B wrote:
Flipping SGTM. are you suggesting we leave the annotation-name as it is, and don't change it to ? If we are changing it, we should probably change the name from as well, since it doesn't make as much sense anymore. Probably ?
or healthy rather.
Flipping default behavior (when not present). Or did I misunderstand beeps proposal? On Oct 27, 2016, at 7:11 PM, Anirudh Ramanathan wrote: Flipping SGTM. are you suggesting we leave the annotation-name as it is, and don't change it to ? If we are changing it, we should probably change the name from initialized as well, since it doesn't make as much sense anymore. Probably unhealthy? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub , or mute the thread .
Renaming it to beta is effectively moving it to beta, so I think given that we're still confused about how this ties in to the big picture, that might be premature. If it's either move to beta or drop, I'd probably vote for drop, it's a debug hook after all. Renaming it to something else but keeping in beta makes sense.
sorry, keeping in alpha
whats going on with this tag? it's used in the docs here should i use it going forward?
It's alpha and may be removed at any time. On Nov 2, 2016, at 3:11 PM, Zachary Adam Kaplan wrote: whats going on with this tag? it's used in the docs here should i use it going forward? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub , or mute the thread .

// It doesn't update, probe or get the pod. type defaultPetHealthChecker struct{} <del> // isHealthy returns true if the pod is running and has the // "pod.alpha.kubernetes.io/initialized" set to "true". </del> <ins> // isHealthy returns true if the pod is ready & running. If the pod has the // "pod.alpha.kubernetes.io/initialized" annotation set to "false", pod state is ignored. </ins> func (d *defaultPetHealthChecker) isHealthy(pod *api.Pod) bool { if pod == nil || pod.Status.Phase != api.PodRunning { return false } <ins> podReady := api.IsPodReady(pod) // User may have specified a pod readiness override through a debug annotation. </ins> initialized, ok := pod.Annotations[StatefulSetInitAnnotation] <del> if !ok { glog.Infof("StatefulSet pod %v in %v, waiting on annotation %v", api.PodRunning, pod.Name, StatefulSetInitAnnotation) return false } b, err := strconv.ParseBool(initialized) if err != nil { return false </del> <ins> if ok { if initAnnotation, err := strconv.ParseBool(initialized); err != nil { glog.Infof("Failed to parse %v annotation on pod %v: %v", StatefulSetInitAnnotation, pod.Name, err) } else if !initAnnotation { glog.Infof("StatefulSet pod %v waiting on annotation %v", pod.Name, StatefulSetInitAnnotation) podReady = initAnnotation } </ins> } <del> return b && api.IsPodReady(pod) </del> <ins> return podReady </ins> } // isDying returns true if the pod has a non-nil deletion timestamp. Since the


Question:
PetSets currently support an annotation in their PodTemplateSpec field. From the user documentation: Debugging: If one want to bring up the pets in my petset one after the other, while pausing after each step, confirming that the state is as expected, one may use this functionality. For example, a petset where each pet is a replica of a distributed database; we set the to in the petset template initially. This leads to the following progression: PetSetController creates pet-0 Manually intervene and check that it has initialized as expected. Set annotation to true (equivalent to marking that pet as healthy) PetSetController creates pet-1 Manually intervene and check that it has initialized as expected. ... Blocking the PetSet controller: We may want to ensure that the petset controller is not performing scale operations when a different user-initiated action is in progress (for example, where pets are leaving due to a kubectl drain). This may be used as a mechanism to orphan a pod from a petset controller (for a closer look), while ensuring that a replacement is not created. The use case 1 above can be realized using and is probably easier to do that way rather than use the annotation. I don't see a reason to have that case supported when we can have users scale up a PetSet gradually and manually in an identical manner. It may have been of more importance before v1.4 when the scaler & reaper were not implemented. Due to the second use-case, if it would make sense to continue to have this ability, I propose that we move it from the PodTemplateSpec to the PetSetSpec, where it will eventually reside. This seems to me, like the logical place for such a debug hook to reside. In order to run/pause the scaling operations of the PetSet controller, we could have annotations along the lines of with the annotation defaulting to when absent. /cc
Should we just support Pause on PetSet (for part of these cases)?
Do users have to catch the roll out at the right time in order for this to work? For example if I want to pause between nodes 2 and 3 coming up, do I need to be precise in my timing when editing the resource? How does this look from the users perspective?
Note that Pause allows scaling in Deployments, it just blocks upgrades. That being said, I think Pause will be useful for PetSets in general. Not sure though if we can use it somehow for this specific issue.
I like the debugging hook for PetSets. If it's optional I don't see any reason to get rid of it.
pauses petset controller if necessary and inspects the state of the system_ With the previous flow, the petset controller was paused after each pet's creation automatically, and here it would be an extra step to pause the controller it if necessary. I don't know how important it is though, but to me it seems easier to understand than a per-pod annotation.
and I had several discussions with the lifecycle of what would be useful. There are a bunch of use cases that overlap: not a primary concern for most users but helpful (prevent changes) - very important if you are triaging failures. Technically scaling is itself a user action, so if you don't want your pet set autoscaled while triaging failures you need to take action on the HPA like deployments. Since PetSets and ReplicaSets are nothing alike w.r.t. scaling I'm not terribly concerned that pause means slightly different things if for end users it Does The Right Thing control by a third party controller - this is the babysitter controller wanting to reuse the petset controller for core stuff but still have the hook angle. Not thought through enough to design around.
Note that Pause does not prevent scaling in Deployments, only upgrades , Clayton Coleman wrote:
Just to summarize: The current behavior with the annotation It is set on a per-pet basis. Setting it to false on any single pod blocks all operations on the petset; scaling, creation of new pets, etc. All pets must contain the annotation, and have its value set to true in order to make progress. By default, it is set to false and makes the petset controller pause after creating one pet. The smallest change we can make for beta: Rename it to (or some better name) Make it optional (with the absence of the annotation meaning that the petset can scale) Surface the reason better when we block on scaling because of the annotation. I still think there is value in moving it to be an annotation on the PetSet resource, rather than having it be in each pod, which is confusing to users. wdyt?
Putting it on the pod template is clever, but surprising. How many examples are dependent on it? , Anirudh Ramanathan < :
Last I checked, all the examples had set to true. We should just get rid of it from all examples in my opinion. If we make the default (or the absence of any such annotation) to mean that the petset controller can make progress, we can safely do that. was going to do a pass over the examples after the PetSet renaming. This could be an additional step. Does that sound ok?
If it's not found, we assume you want to pause. You're proposing that if it's not found, we default to "forge ahead". SG. It should only be used in debugging/testing today. Basically if you don't know how to write your , you'd create a petset with it set, kubectl exec and write it out, scale the petset by 1, kubectl exec ..., and when you have a working cluster, put what you did in an init container and create the petset without it. This is the absolute min. You're also proposing that we move it to the petset. This makes sense overall, but I initially didn't do that because I didn't know how we would end up doing pause.
We could leave it there as "alpha" and then remove it in 1.6 once we have more time to talk about how upgrades and debugging and pause should work? , Prashanth B wrote:
Yeah the minimum work is to flip the behavior of the controller when the annotation is not found:
Agree on flipping behavior. , Prashanth B wrote:
Flipping SGTM. are you suggesting we leave the annotation-name as it is, and don't change it to ? If we are changing it, we should probably change the name from as well, since it doesn't make as much sense anymore. Probably ?
or healthy rather.
Flipping default behavior (when not present). Or did I misunderstand beeps proposal? On Oct 27, 2016, at 7:11 PM, Anirudh Ramanathan wrote: Flipping SGTM. are you suggesting we leave the annotation-name as it is, and don't change it to ? If we are changing it, we should probably change the name from initialized as well, since it doesn't make as much sense anymore. Probably unhealthy? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub , or mute the thread .
Renaming it to beta is effectively moving it to beta, so I think given that we're still confused about how this ties in to the big picture, that might be premature. If it's either move to beta or drop, I'd probably vote for drop, it's a debug hook after all. Renaming it to something else but keeping in beta makes sense.
sorry, keeping in alpha
whats going on with this tag? it's used in the docs here should i use it going forward?
It's alpha and may be removed at any time. On Nov 2, 2016, at 3:11 PM, Zachary Adam Kaplan wrote: whats going on with this tag? it's used in the docs here should i use it going forward? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub , or mute the thread .

ObjectMeta: api.ObjectMeta{ Name:      name, Namespace: ns, <ins> Annotations: map[string]string{ "pod.alpha.kubernetes.io/initialized": "false", }, </ins> }, Spec: apps.StatefulSetSpec{ Selector: &unversioned.LabelSelector{


Question:
Kubernetes version (use ): HEAD Environment: GCE What happened: When I set up I can see the following message: What you expected to happen: master should not be registered
cc
Looks like the env var isn't being respected any longer by the master scripts (but the logic for # of expected nodes did adjust correctly).
/cc I think this may have broken in because it which unconditionally sets the field that tells a kubelet to register with the master outside of the guard that checks .
Fix in progress: . There was a problem caused by unconditional setting on the master in .

create-static-ip "${MASTER_NAME}-ip" "${REGION}" MASTER_RESERVED_IP=$(gcloud compute addresses describe "${MASTER_NAME}-ip"  --project "${PROJECT}" --region "${REGION}" -q --format='value(address)') <del> KUBELET_APISERVER="${MASTER_RESERVED_IP}" </del> <ins> if [[ "${REGISTER_MASTER_KUBELET:-}" == "true" ]]; then KUBELET_APISERVER="${MASTER_RESERVED_IP}" fi </ins> KUBERNETES_MASTER_NAME="${MASTER_RESERVED_IP}" create-certs "${MASTER_RESERVED_IP}"


Question:
From . During the firewall e2e test we need to change the service type from to to avoid leaking LB resources. But found it is not working because the field. Error I got as below: If don't delete this field, got instead another error:
Just saw the TODO comment . Working on it.

} // TODO(freehan): allow user to update loadbalancerSourceRanges <del> allErrs = append(allErrs, ValidateImmutableField(service.Spec.LoadBalancerSourceRanges, oldService.Spec.LoadBalancerSourceRanges, field.NewPath("spec", "loadBalancerSourceRanges"))...) </del> <ins> // Only allow removing LoadBalancerSourceRanges when change service type from LoadBalancer // to non-LoadBalancer or adding LoadBalancerSourceRanges when change service type from // non-LoadBalancer to LoadBalancer. if service.Spec.Type != api.ServiceTypeLoadBalancer && oldService.Spec.Type != api.ServiceTypeLoadBalancer || service.Spec.Type == api.ServiceTypeLoadBalancer && oldService.Spec.Type == api.ServiceTypeLoadBalancer { allErrs = append(allErrs, ValidateImmutableField(service.Spec.LoadBalancerSourceRanges, oldService.Spec.LoadBalancerSourceRanges, field.NewPath("spec", "loadBalancerSourceRanges"))...) } </ins> allErrs = append(allErrs, validateServiceFields(service)...) allErrs = append(allErrs, validateServiceAnnotations(service, oldService)...)


Question: The logic to account for all pods being ready is incorrectly counting some pods as being controlled by a replication controller. This is causing the 1.3-1.5 upgrade tests to fail. See https://k8s- as an example. Since , the pre-flight checks fail and the rest of the tests never run. I think this was exposed by which changed some things from ReplicationControllers to Deployments. There's a pod being created by a deployment that also matches the selector of a replication controller, and is therefore being incorrectly counted by cc

return errStr + buf.String() } <del> // check if a Pod is controlled by a Replication Controller in the List func hasReplicationControllersForPod(rcs *api.ReplicationControllerList, pod api.Pod) bool { for _, rc := range rcs.Items { selector := labels.SelectorFromSet(rc.Spec.Selector) if selector.Matches(labels.Set(pod.ObjectMeta.Labels)) { return true } } return false } </del> // WaitForPodsSuccess waits till all labels matching the given selector enter // the Success state. The caller is expected to only invoke this method once the // pods have been created.


Question: The logic to account for all pods being ready is incorrectly counting some pods as being controlled by a replication controller. This is causing the 1.3-1.5 upgrade tests to fail. See https://k8s- as an example. Since , the pre-flight checks fail and the rest of the tests never run. I think this was exposed by which changed some things from ReplicationControllers to Deployments. There's a pod being created by a deployment that also matches the selector of a replication controller, and is therefore being incorrectly counted by cc

// WaitForPodsRunningReady waits up to timeout to ensure that all pods in // namespace ns are either running and ready, or failed but controlled by a <del> // replication controller. Also, it ensures that at least minPods are running // and ready. It has separate behavior from other 'wait for' pods functions in // that it requires the list of pods on every iteration. This is useful, for </del> <ins> // controller. Also, it ensures that at least minPods are running and // ready. It has separate behavior from other 'wait for' pods functions in // that it requests the list of pods on every iteration. This is useful, for </ins> // example, in cluster startup, because the number of pods increases while // waiting. // If ignoreLabels is not empty, pods matching this selector are ignored and


Question: The logic to account for all pods being ready is incorrectly counting some pods as being controlled by a replication controller. This is causing the 1.3-1.5 upgrade tests to fail. See https://k8s- as an example. Since , the pre-flight checks fail and the rest of the tests never run. I think this was exposed by which changed some things from ReplicationControllers to Deployments. There's a pod being created by a deployment that also matches the selector of a replication controller, and is therefore being incorrectly counted by cc

}() if wait.PollImmediate(Poll, timeout, func() (bool, error) { <del> // We get the new list of pods and replication controllers in every // iteration because more pods come online during startup and we want to // ensure they are also checked. </del> <ins> // We get the new list of pods, replication controllers, and // replica sets in every iteration because more pods come // online during startup and we want to ensure they are also // checked. replicas, replicaOk := int32(0), int32(0) </ins> rcList, err := c.Core().ReplicationControllers(ns).List(api.ListOptions{}) if err != nil { Logf("Error getting replication controllers in namespace '%s': %v", ns, err) return false, nil } <del> replicas := int32(0) </del> for _, rc := range rcList.Items { replicas += rc.Spec.Replicas <ins> replicaOk += rc.Status.ReadyReplicas } rsList, err := c.Extensions().ReplicaSets(ns).List(api.ListOptions{}) if err != nil { Logf("Error getting replication sets in namespace %q: %v", ns, err) return false, nil } for _, rs := range rsList.Items { replicas += rs.Spec.Replicas replicaOk += rs.Status.ReadyReplicas </ins> } podList, err := c.Core().Pods(ns).List(api.ListOptions{})


Question: The logic to account for all pods being ready is incorrectly counting some pods as being controlled by a replication controller. This is causing the 1.3-1.5 upgrade tests to fail. See https://k8s- as an example. Since , the pre-flight checks fail and the rest of the tests never run. I think this was exposed by which changed some things from ReplicationControllers to Deployments. There's a pod being created by a deployment that also matches the selector of a replication controller, and is therefore being incorrectly counted by cc

Logf("Error getting pods in namespace '%s': %v", ns, err) return false, nil } <del> nOk, replicaOk := int32(0), int32(0) </del> <ins> nOk := int32(0) </ins> badPods = []api.Pod{} desiredPods = len(podList.Items) for _, pod := range podList.Items {


Question: The logic to account for all pods being ready is incorrectly counting some pods as being controlled by a replication controller. This is causing the 1.3-1.5 upgrade tests to fail. See https://k8s- as an example. Since , the pre-flight checks fail and the rest of the tests never run. I think this was exposed by which changed some things from ReplicationControllers to Deployments. There's a pod being created by a deployment that also matches the selector of a replication controller, and is therefore being incorrectly counted by cc

} if res, err := testutils.PodRunningReady(&pod); res && err == nil { nOk++ <del> if hasReplicationControllersForPod(rcList, pod) { replicaOk++ } </del> } else { if pod.Status.Phase != api.PodFailed { Logf("The status of Pod %s is %s (Ready = false), waiting for it to be either Running (with Ready = true) or Failed", pod.ObjectMeta.Name, pod.Status.Phase) badPods = append(badPods, pod) <del> } else if !hasReplicationControllersForPod(rcList, pod) { Logf("Pod %s is Failed, but it's not controlled by a ReplicationController", pod.ObjectMeta.Name) </del> <ins> } else if _, ok := pod.Annotations[api.CreatedByAnnotation]; !ok { Logf("Pod %s is Failed, but it's not controlled by a controller", pod.ObjectMeta.Name) </ins> badPods = append(badPods, pod) } <del> //ignore failed pods that are controlled by a replication controller </del> <ins> //ignore failed pods that are controlled by some controller </ins> } }