We have found that on 1k nodes cluster scheduling takes 60-200 ms and becomes the bottleneck. We want to have a standard way to reproduce scheduling latency metrics result and benchmark scheduler as simple and fast as possible. Only with that can we: 1. figure out underlying behaviors and potential bottlenecks quickly; 2. publish results in a standard way and prevent future performance degradation. We have the following goals: Goal 1: Save time on testing. More accurate scheduling latency. We mock up kubelets and only measure scheduling latency and do not care about the latency to start the pod. This is different from Kubemark -- scheduled pods won’t be run by mocked kubelets. We only need to set up the master components. Goal 2: Profiling runtime metrics and find out performance bottleneck. Write scheduler integration test. Take advantage of go test and profiling tools, like cpu-profiling, memory-profiling and block-profiling. This is very fine-grained metrics. Goal 3: Reproduce test result easily. We want to have a known place to do the performance related test for scheduler. Developers should just run one script to collect all the information they need. We have the following plans for writing scheduler tests: density test (by adding a new Go test) schedule 30k pods on 1000 (fake) nodes and 3k pods on 100 (fake) nodes print out scheduling rate every second let you learn the rate changes vs number of scheduled pods benchmark (by modifying the existing benchmark in integration or simply add a new one) make use of and report nanosecond/op. schedule b.N pods when the cluster has N nodes and P scheduled pods. Since it takes relatively long time to finish one round, b.N is small -- level of 10. We are new to k8s codebase. We also want to know where is the best place (directory) to commit in these tests? Any suggestions are welcomed!
/cc
This extends
So density already does this... the one option it looks like you may want is custom control , real or fake is irrelevant as the e2e's just need to point to an api-server. re: benchmarks, seems like we could simply add to the unit-tests.
After chatting with lets add this as an agenda item for the SIG meeting tomorrow.
+1 for adding benchmark (howeveer this should be integration test, not unit-test in my opinion)
cc The next SIG on scheduler meeting is on 12.07, isn't it? If there is one tomorrow, can I have the information please?
retract, +1 to . I was adding this to the list given the context, but it crosses over into both.
Slides from today's meeting:
Here are are the action item notes from the SIG meeting: Everyone agrees this is a good idea and long overdue, but the question is where would this live: New category of tests? (isolated/sandbox’d/component) The current testing structure allows to easily append. create new directory (component) + hack/script to execute Future "we could" gate on known good data, and fail performance regressions
/cc
/cc


I'm running application with several env variables. One of them has a key ending with ==. Another variable has a comma separated list of email addresses. While perfectly valid on *nix and windows, we can't pass these values into Kubernetes run command. It fails with "error: invalid env:" Example command: The parsing in should be more robust
I volunteer myself to fix this.
Maybe they can be passed from file? I mean s'rsly.. you need to lits 20 variables by hand?? I've specified them in yaml file already, but kubectl-run have no way to specify it.. without this, deployment is totally impossible when you need to run migrations first!


Have not reproduced locally even after a few tens of runs.
which version of Go are you using locally? This probably only repros on 1.5.1.
This was from Jenkins runs.
This is caused because an attempts counter is being read/written to without synchronization. I think this is caused by the event creation order not being guaranteed. I am looking a little deeper to track this down.
Got a fix with 130k+ runs and no failures. Will send out PR shortly.


Currently scheduler latency are observed regardless of failure. For example, binding latency is like: This could be a potential confusion if latency spikes happen because of failure. We should better handle it like separating error case from normal
/cc
I completely agree - are you going to change this?
Yes. I will do it.


I'll take a look at the failed builds and triage.
I went through the recent reboot failures in kubernetes-e2e-gke-ci-reboot and here is a summary. [6481, 6486, 6488] Kubelet never came back from the reboot, and stopped posting node status. Throughout the test suite, the failed node stayed silent: It's hard to dig further due to the lack of clarity. Gathering log would at least help us rule out some possible causes . [6485] Heapster seems to crash loop quite frequently, and sometimes it failed the test (due to being not ready).. cc [6487] kube-proxy pod is running but not ready before the reboot test even started: This is not really a reboot test failure since the cluster was already in a bad state. We should make sure all the pods are ready after each test (e.g. "should work after restarting kube-proxy" ran before the failed reboot test). Another thing that would help is to increase the log level of the gke cluster. The log level of a gce cluster is 4, but I think gke is at the regular 2. Due to the low log level, it's unclear when kube-proxy started failing the readiness probe.
Filed for (3).
The reboot tests basically haven't worked since 9pm last night: Interestingly, this lines up with the GKE equivalent for , but in that same window, has had no trouble. cc
Digging into logs from the runs linked.
All of the gke-ci failures in are heapster crash-looping after a reboot.
Changes to the heapster config in the last day: , . Nothing sticks out to me in those PRs.
Heapster could be in crashloop if it cannot reach influxdb. But that is a bad design, and should fix it in today's heapster with image 18. I double checked them, it is image version 18.
From , could be the num_nodes is unset, we give heapster no resource to run? If that is true, it could be point to the same version screw issue of our test infra.
Heapster v0.18.4 will crashloop if InfluxDB isn't available. However we can update HEAD to v0.19.0, which will not crash if the backend goes down. I will send a PR to update InfluxDB version. , Dawn Chen wrote:
GKE doesn't use InfluxDB, right?
Heapster is the only kube-system pod that's not ready in the tests. If it relies on influxdb, shouldn't influxdb be not ready too?
I like theory. My salt suggestion might've been horrible for unset ? I thought I did that correctly, but maybe my Python/Jinja was bad.
Yes, GKE doesn't use influxDB. My point is that if the backend storage is not ready, heapster is in crashloop. Vishnu made some changes to make heapster can run standalone, but looks like that fix is not included in heapster 18 based on his comment.
Okay. I'm still trying to catch up. Why is heapster on GKE waiting on backend storage? And if is unset (e.g. we haven't finished running configure-), it looks like we just give the
Catching others up: is seeing nodes that are actually hung with CPU pegged on reboot, so the heapster conversation may be a red herring.
There are plenty of tests that failed because kubelet didn't come back (and post status) after the reboot. There are a few that failed only because heapster was crash looping. They may be related or not...
I spun up a GKE test cluster: and the heapster manifest looks fine, so I think I ruled out anything regarding
I think is the winner. The reboot tests are a red herring.
I'm actually just seeing heapster flap on a fresh cluster, no reboots.
Heapster crashlooping is one problem. There are other cases, however, where kubelet just stops posting status all together. If heapster crashlooping can render kubelet unresponsive, that's a bug.
is the serial console output from the node that got pegged (in ). I rebooted the node to see if it would repro, and it came back up fine. EDIT: the link to the jenkins run.
I guess the serial console you posted is from build , the very recent one. I saw a similar one and the node never successfully come back. At this moment, I ignore those failures. The reboot failure caused by heapster crashloop is similar to what I observed 2 days ago for influxdb in gce cluster. The comment is at Can we retitle this one for (1)? Reboot test flakiness is caused by many reasons: version screw which is handled by test-infra, heapster and influxdb crashloop is handled by , kube-proxy not ready is handled by cluster team.
Ok. So we think we have the heapster flakes, versioning stuff, and kube-proxy stuff all taken care of elsewhere? I'm fine with targeting this issue at the "node doesn't come back after reboot" case. Is that what you were recommending?
Yes. 1) heapster in crashloop if backend is not ready is a known issue for a while, and fix is merged to heapster repositry for a while. We just need to upgrade our heapster after more testing (PR: ) 2) influxdb one requires more information, but we can handle them through 3) version screw is handled by test-infra through . already test/flake label for it 4) kube-proxy readiness issue is handled through
where else should I be looking for hints on why that VM didn't come back to life until I kicked it? Or did I destroy any evidence we might have had?
Fun fact: I thought this might be related the timing of when I made the ContainerVM change on the GKE side, but I made that change for all releases 1.1.4. That picks up this test suite, too, because it's at , and it's green: So this is a divergence from master to 1.1, or between projects.
That suite does not run any reboot tests...
Dammit, I thought I scanned for that. Good call. We need to fix that.
Here's the set of suites that run the reboot tests (thanks kubernetes-e2e-gce kubernetes-e2e-gce-reboot kubernetes-e2e-gke-ci kubernetes-e2e-gke-ci-reboot
Shouldn't we decided only X-reboot test suites run those reboot tests since they are disruptive?
Hm. builds 6478-6480 are all green and all after the (GKE-side) ContainerVM change, so we're looking for something else.
The container-vm change was merged at , which lines up with kubernetes-e2e-gke-ci-reboot 6481 (the beginning of the failures).
6481 is the one with the new containervm change. It includes kubernetes-build/7822, which has the change.
/ That's the OSS side change, which is irrelevant. GKE flipped at 19:45:55.
Ahhh. Fair enough.
Actually, and I just determined that the GKE-side change didn't make it out until 01:45:44, it looks like, so indeed nothing to do with ContainerVM.
I took a more careful look at the serial output pasted, and this is interesting: We're actually in the middle of a clean shutdown, trying to unmount and failing. This same stacktrace repeats every 120s.
I spun up an e2e cluster and with minimal testing have a node in the state of It's the same kernel hang on shutdown. ... and then I accidentally deleted it when I went to bisect, but ohwell. I'm sure I'll be able to get one back if I need.
Okay, so I've confirmed that at the build level at , I can spin up a GKE cluster and reboot tests fail. I'm going to peel back to and see if they pass (because that matches ). ETA: For reference, I'm able to tickle this with just:
7819 had a kernel hang on the second run as well, though slightly different signature (still aufs in the end):
For my next test, I'm peeling back to v1.1.4. We have no test coverage on reboots on the release branch of GKE right now, and I just want to guarantee this isn't some weird GCE hypervisor issue. And v1.1.4 is clean, at least for two runs worth.
This failed quickly on v1.1.5. Hrm. I'm wondering if my v1.1.4 test was a fluke, peeling back to try it there again. v1.1.5 only differs by ContainerVM (and go 1.4.3 vs 1.4.2), but all of our other timelines show that not being directly related. (Alternately, it's a lot easier on the newer kernel.) (Still no idea why the tests aren't triggered, but is.)
was very solid. I'm now trying on GKE test. This is a GKE unique thing, because after Friday morning at 1am, anything = will get deployed using the new ContainerVM. So I'm ret-conning a way to effectively try with the new ContainerVM, just to see if it's a contributing factor. It doesn't look like it should be. One of the other commits in the stream is a heapster version bump (from / ). If there's additional flapping or something caused by that (unique to GKE), it may be causing the hang. But that timeline doesn't match the line (relevant to , etc.)
Bad news bears: failed reboot tests. I'm having trouble rectifying this information with what we know about the timeline. But it does suggest there's a problem with ContainerVM at the very least, so there might be two problems here, so I'm rolling one back on GKE and we'll see what happens.
I've manually confirmed that by reverting deployment of on GKE, I can run reboot tests at and a latest build. Using a special build of latest that will still trigger deployment of , reboot tests fail. Well, damn.
GKE has been repushed and should, in theory, go back green soon. There's a lot I don't understand about this bug, and it's effectively going to stop us shipping and until we can figure it out.
Okay, I think that reverting was not all that's necessary. We unfortunately haven't managed to collect a green on yet. That gives me some comfort, because it means that the timeline-discomfort I had previously was founded.
Heh, I lied and right right writing that we collected our first one. Woo. I suspect will follow soon.
Haven't we updated go to 1.5 on Friday as well? I don't know if we use the same version of go for GKE as we do in GKE, but in general we know it causes trouble. cc for Heapster thing.
The timeline there is off, too. The golang1.5.3 PR went in at , which is well after the trouble started.
Morning status: Looks like we made a little greener, and didn't touch , so the patient had two viruses.
Okay, this is my suspicion, but I can't quite piece together what's going on: For some reason, something in (? but that busted the Heapster config, so it's confusing me as to why), caused this. I'm basing this on: the timeline of the timeline of If you suppose that we injected a bug in build 7820 that caused about a 60-70% cointoss as to whether we regress (which is similar to the rate we're seeing now that has been rolled back on GKE), it's not unreasonable that was green for , and that's the only one you have to explain away on this theory. After that, it goes kind of red, and then at about 1am, comes in and wrecks completely as well. Pulling it back much further, to, say, build 7819, and you have to explain the entire range from 6477-6480 as green, which is a much lower probability. So my money is on build 7820. (7821 is revision-equivalent, so boring to talk about.)
Oh. It is Heapster. I spun up a 7819 cluster and a 7820 cluster, and the difference is that even though still had slightly busted Salt syntax, it was functional enough to create the replication controller, whereas on 7819, there's no Heapster replication controller at all. So I think the Heapster flapping is entirely the cause of the remaining issues, so we really need to get in ASAP.
So yesterday afternoon I was wondering why build 7820 triggered such a large regression when, say, the bump to heapster in didn't regress us at all. Build 7820 finally let the the fix in , which was actually important because GKE at HEAD wasn't deploying (until last night), it was deploying only , so the pillar wasn't configured. I'm still not exactly sure what's going on here, and I had intentionally left blank for a while during this investigation because it didn't seem like it should matter, but it's obvious that if 7820 is the culprit that it's interesting. I pushed a fix last night to GKE that adds on 1.2 clusters as well, and we've collected our first green dot. I don't know why is still flakey, though.
Er, correction to my last comment. My change didn't make it out last night and won't be relevant until . The green is a fluke. Sad panda.
PR introduced syntax error, PR fixed the error although it introduced another syntax issue which was finally fixed in . if you are testing anything make sure you are using version with merged. I agree with that the problem is not with Heapster accessing InfluxDB. Also the same problem takes places in autoscaling suite (described here ). We started observing such problems around Jan 19th. AFAIK once the container crashes kubelet invoke docker inspect on the failing container. How about invoking also docker logs then? It would be very useful for debugging purposes.
Yes, I understand that fixed the issue fully, but at build 7820 the cluster that's created actually spins up a cluster with a replication controller .yaml with resources. At build 7819, it doesn't. The syntax in is busted, but it's apparently not so busted. Previous to , in build 7819 (and builds previous), there was no controller .yaml at all, so there was a window where GKE wasn't spinning up Heapster at all. Hence my conclusion.
The syntax in is correct from Salt point of view, although the generated file has empty line in the begging and is not accepted by Api Server, so the RC is not created.
That's inconsistent with the fact that Heapster is starting on a 7820 (GKE) cluster.
I have a new theory, which is that Heapster hasn't been running on GKE in 1.2 since , until .
It's not clear to me that upgrading heapster will solve this issue, since heapster doesn't rely on any backend on gke. Instead of asking kubelet to do that, it makes more sense for the test itself to get the logs through the apiserver. I think we should add that to the e2e test.
I spun up a cluster from just prior to and this is the set of RCs I see: So basically, when went in, Heapster was enabled for the first time in GKE (master, not the release branches) in months and we started flipping out. This is a serious test coverage issue, obviously.
I edited the above comment, but we would obviously catch this in the release branches because we have monitoring tied to Heapster related things. We need to be better here for CI.
I made some investigation. The problem with crashlooping Heapster is due to: The issue was fixed in I'll create a patch release and bump the version.
It looks like is green. I'm hoping that , which is the first build past , will go green as well. (The cycle time on that build is waaaaay too long.)
It looks like we're good! is green now!
for node problem detector.


I noticed this flake by looking at the submit queue. The test is simple, but when it fails, we have no good way to debug it. Below is the test in question. When the test fails, we should print the lines we got in each case.


hack/build-cross sets it without looking at the current value. And the value has to be an array, which is a bit awkward to use.
This could be improved indeed. Maybe something like and Haven't tested it though... But why would one want to change the value?
I just wanted to build one, so I did the "obvious" thing: hack/dev-build-and-` It failed for multiple reasons :) , Lucas Käldström wrote:
This issue has been a major downer for me. It's much harder right now than it ought to be to articulate the few platforms for which you'd like to build each of server binaries, client binaries, test binaries, etc. Every time I've tried to do this "the right way," I've spent hours looking at scripts and feeling frustrated that build options aren't documented better. Every time, I end up cutting my losses and resorting to temporary build script hacks instead, just to get the job done.
Doesn't fix this issue exactly, but I've found that you can easily build only if you set .
Taking a look at this today, I also thought that setting seems to be the most reasonable.
/sig release


The delete command for a replicaset hangs forever. This then manifests as phantom RSs when I delete and recreate a namespace.
- delete is the gift that keeps on giving
cc
Had you already deleted the corresponding Deployment?
I am still seeing this problem. Deployments and replica sets are sticking.
From what I can tell, calling DELETE_COLLECTION on Deployments is returning: And since the controller assumes not found errors are equivalent to already deleted, it thinks the content is deleted. When I manually inspect, I still see the deployment and replica set laying around... //cc
holding on this while i debug some more...
This was a false alarm, apologies.


http://pr- is another failure.
Also seen in http://pr-
Again. Not assigned to anyone, so I am sending to for routing
Again: Is there anyone working on this? I will spend some time looking into this.
I think that the name of this issue is misleading - the issue seems to be unrelated to etcd_watcher Basically, if you look into this part of log: So it seems that, the WaitGroup that is problematic is in httptest.Server. So my feeling is that we are simply talking to a server that is already being close and this is ~roughly duplicate of We should just find where exactly the problem is and try to work-around it.
I think this is exactly what we had before. I will send a PR with the fix today.


I'm following step by step tutorial on how to create resource quotas per namespace () looks like this: { "apiVersion": "v1", "kind": "ResourceQuota", "metadata": { "name": "quota", }, "spec": { "hard": { "memory": "1Gi", "cpu": "20", "pods": "10", "services": "5", "replicationcontrollers":"20", "resourcequotas":"1", }, } when i'm about to create quota it fails with: kubectl create -f --namespace=prod invalid character '}' looking for beginning of object key string Namespace "prod" exists: kubectl get namespace NAME LABELS STATUS AGE default <noneActive 26d dev name=development Active 8d kube-system <noneActive 26d prod name=production Active 8d kubectl version Client Version: version.Info{Major:"1", Minor:"1", GitVersion:"v1.1.2", GitCommit:"", GitTreeState:"clean"} Server Version: version.Info{Major:"1", Minor:"1", GitVersion:"v1.1.2", GitCommit:"", GitTreeState:"clean"}
Those trailing commas are wrong. should look like: I'll submit a docs patch
sweet. ta. That seem to do the trick: kubectl create -f --namespace=prod resourcequota "quota" created
I'm going to reopen until the PR which fixes this merges, when it should autoclose...
I'm following this tutorial on installing percona with kubectl, and I'm getting a similar error: When I run: kubectl create -f pmm-server- I get: error: json: line 1: invalid character ‘Â’ looking for beginning of object key string


The says this about the new GUI: It does not what? Seems important!
Hmm... That's interesting. Will fix.


When creating a new cluster on AWS with the 1.2 version I always get a failure when checking docker on the nodes, but after the check loop ends everything seems to work fine and all nodes are marked as ready. The instance types for nodes is
That is very surprising. Are you using any particular settings (to help me try to reproduce the problem)?
These are the exports I make before running kube-up:
The problem is the instances take a very long time to start up. I had to increase the timeout to 19.
Hey how did you increase the timeout? Didn't see a setting in config- that I could override. Getting the same issue as the poster.
Thanks , that did the trick!


I have been having the kube-controller-manager (running in a container) crash quite frequently with this error: I cant fathom what seems to cause it. It seems to happen consistently for a while (CrashLoopBackoff occurs) and then fixes itself after an undetermined amount of time. Kubectl version: Docker version: Networking: Calico (with the calico k8s policy agent) Please do let me know if there is any other information that I can supply. I appreciate I haven't given much to go off of. I am not sure what the steps to reproduce are...
cc
It looks like a bug. I'll take a look. Thanks for reporting it.
Weirdly, the description showed up in the email github sent me but it appears absent on the github issue?
The description was there when I read it but disappeared now.
Well thats very odd! If you have it on your email, could you post it on here please (and I will update the description)? I unfortunately haven't got the error message to hand now...
I took the liberty of just editing the OP.
thanks a lot! Not sure what happened there!
could you paste the returned output at "server-ip:port/api" and "server-ip:port/apis"?
Sure.
Thanks "" has an empty preferred version and that's causing the error you reported. Is "" created via the thirdparty resource api? ()
aha! Good spot! Yes, it was created via that API. This is the Yaml file I used to create it: Here is the output of
Thanks for the confirmation Your configuration is correct. The preferred version is left empty by the thirdparty resource controller, while our discovery client has an assumption that the preferred version won't be empty. I'll patch the discovery client to tolerate an empty preferred version, and will discuss with on if we should make a default preferred version for a thirdparty resource.
Thanks a lot Great job on working it out :)
Sent to fix.
What would be a workaround for this before the release, how can one specify the preferredVersion?
AFAIK, there is no workaround. There is no way to specify the preferredVersion for the thirdPartyResource.
I have a quick workaround to fix this : 1) Change the kubernetes to reflect the following : from : thirdpartyresources=true to : thirdpartyresources=false 2) Then restart 'kubelet' service OR reboot the K8S Master server systemctl restart kubelet 3) Revert back the changes done to kube- from : thirdpartyresources=false to : thirdpartyresources=true 4) Verify whether the above change on the above manifest file has been automatically picked up by the kubernetes API ps aux | grep -i thirdpartyresources 5) Finally you may restart kubelet service systemctl restart kubelet 6) Now verify whether the 'kube-controller-manager' is in a 'Running' state kubectl get pods --all-namespaces | grep -i controller-manager Hope this fix works for you guys as well !!


Minor improvement in UX maybe?
We have semi-ambiguous syntax because 'get pod/foo' is also valid. We should have define syntax for this early on. On Apr 8, 2016 7:13 PM, "Prashanth B" wrote:
+1 for the UX improvement.
get pod/foo != get pod ns/foo ?
.... LOL
This would make SO much easier to use :+1:
I dont disagree, but it is somewhat complicated to transition now. Maybe something like 'kind/ns:name' which would be less ambiguous and strictly compatible (ns:name is not a valid name, nor is it confusing wrt kind/name). Need to define precedence of this with --namespace flag and in-file namespace yaml. I also don't see it as a huge win, personally. It would have been nice from day1 but relatively minor now, IMO. But this is a decision for UX team - if someone sent a patch implementing something like it, or at least a proposal that worked through all the details, it would be a better starting place. On Apr 10, 2016 12:33 PM, "Lucas Käldström" wrote:
This is more like P0 to me. Somewhere else I complained about having to type too much and the kind/name syntax. My vote if for kind/ns:name, too.
cc
What is the scenario you're trying to simplify? You often need to use resources from multiple namespaces on the same command line?
cc
ns:name is not a valid name for namespaces and our core resources, but it can be a valid name for extension resources in the Kube ecosystem, like users, roles, authorizations. We'd have to add more slashes
Attaching the namespace to the "name" part won't work well because then you have to chose whether is "list all resources in namespace token" or "get the resource named token in the current namespace". To allow both cases, you must attach the information as a discretely tokenized part or as part of the "resource" part. I think that parses unambiguously since resource can't have dots groups can't have colons and namespaces can't have slashes.
I can assign short names to namespaces and end up with a small kubectl get line, --namespace= just feels unnecessary. I guess it would feel more natural if I could also do . This isn't a huge deal for me personally because I just bash alias my way out of typing.
In my case, I work with multiple namespaces, not on the same command line (I use --all-namespaces frequently, too). I guess I could create bash aliases, but I also tend to reinstall the machines I run kubectl from on a regular basis.
--all-namespaces is so long anyway that it makes me cringe. , Rudi C wrote:
What api group syntax did we settle on? I don't see it documented anywhere. cc
With Ubernetes, we'll also have multi-cluster scenarios coming.
Syntax is: <resource[[.<version].<group] , Brian Grant wrote:
I think either a short option, like or could be a solution or suffix the namespace using " .
was hoping to save for indicating a for the day when we'd be able to retrieve a historical version of an object via the API. Does not alias to like openshift?
No it does not.
would be an improvement for me: it's noticeably shorter than . Does introduce ambiguities with other flags? or introduce some other issue? or is the hope that there's a nicer unambiguous solution (e.g., something similar to the original proposal)?
I don't know of any conflicts. I'm in favor of the short flag because it gets used all the time.
I am fine with , really appreciate if you could help with this.
We originally had -n in Kube but took it out while we sorted out the other config. We went and it in OpenShift because it was used all the time, but forgot to add it in in kubectl. I think it's time to bring it back to kubectl. Other folks who care about this in kubectl should v oice objections in the next day or so, otherwise I think we have consensus to add as the alias for . Can someone scan over the flags in use today in kubectl and verify that no one has introduced a ? , TonyAdo wrote:
Already did, just client generation. has opened to add this. I've got a minor comments unrelated to the code and tagged to give him a chance to object.
I'm also in favor for adding as a shorthand for


I'd like to be able to sort by a timestamp, e.g. but that fails with an exception: I appreciate that the root of the problem is probably outside of kubernetes, yet I hope you'd agree that sorting by a timestamp is very useful functionality.
Agree with sorting by timestamp.. Just make it clear, is of type: while only these types are allowed to be sorted:


During
any chance this is related to pass: https://k8s- fail: https://k8s- https://k8s-
This is because for some reason the submit queue rebuild didn't catch the flake before merging. Protobuf landed first, merging a new api object like petset should have failed.
Quick fix is to rebuild the proto marshalers and push.
Also, hack/verify-generated- is broken, which probably is the root cause.
Looking at it now
Three problems: -generated-protobuf wasn't updated when we made update-generated-protobuf use docker, so it didn't fail out script was verifying the old API groups, not any of the new API groups that have been merged without protobuf serializations (although pkg/api/serialization_test should have failed, possibly the job didn't build with the latest master?), which caused the tests to fail
Something about the merge bot is broken - it merged without all the tests being green: . That's why this broke -
Spawned since there is something wrong with update-generated-protobuf - will merge that to fix the failing test and then debug the generator in the morning.
The bot can race with pushes between when it starts testing and when it merges...


Hi, Deployment patching with doesn't show up in the deployment rollout history: first: after: Is it a bug? Should it show up? Thanks, Peter
cc
Which kubernetes and kubectl releases? Did the patch actually trigger a rollout?
I also run the command from CI with the latest . So the problem affects the v1.2.3 kubectl as well
We should also fix the way we sort revisions - 10 should be after 9, not 1.
Can confirm this. The issue is that the patch isn't updating , so the rollout history will just copy the value from the previous version. If you started with you get that for the patches. In my case I have a different line repeated. Builds 7 and 8 were built with different . I'm not sure why the patch worked once. Command I am calling is:
If no one is working on patch - i can try to make one.
I'll like to work on this issue if you haven't started yet
Actually never mind I can look at something else :)
Hi, i see that you are the author of the following comment. I think that currently every valid change cause is discarded because of version conflict. Maybe you have some information why it happens?
command instead generates a patch to update then applies that.
It was the same for patch, but for some reason was changed.
The intent was two fold. First, prevent the change-cause from ever failing patch the CLI command. change-cause isn't authoritative and the patch took hold, so a non-zero return code is inappropriate. Second, if the resource version change post-patch, before the update, the change-cause is unknown, something else touched it after the fact. A command that issues a instead of an could very easily break the history since patches will attempt to reapply themselves regardless of resource version in the face of conflicts. The object returned from the patch operation is the one that we use to avoid hitting a resource conflict with ourselves. Truth be told, I don't feel very strongly about getting the history right since it can be changed anyone with write access to the object, includes secrets that should be elided (--token for instance), ignores environment variables which affect outcome, ignores the actual path used to locate the binary you're running, and references paths that are only good from one location on one machine, so if you want to switch it back to a second patch, I don't mind as long as a failure to update the object doesn't cause a non-zero return from the command.
Could there be a race between the call and the deployment updating itself to action the changes? On 19 May 2016 9:47 pm, "David Eads" wrote:


Running show failures at times: The expected queue is while got , see code here: I would give an idea that these cases have something similar, so referring and /cc
cc
I think so as well.
The debugging I to the test is obviously not correct - I'll fix that separately.
I wonder if on very short intervals we're not getting a goroutine schedule swap, which causes this race to manifest (based on comments in the other thread).
Does heap insertion guarantee order if the heap value is equivalent? I don't think so.
This seems like it is that now() is not monotonically increasing, so we violate the add ordering condition. Since we already have a lock in most cases, we may want to either wrap now() to make it monotonic inc, or have the queue check for equality in the heap and add one nanosecond until there is no matching heap item. On Apr 29, 2016, at 7:12 PM, Marek Grabowski wrote: cc — You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub
Thanks for clarification, so it looks like we need to give some time break between doing , I'll see whether that helps.
I believe this is a bug in rate limited queue, so we definitely need to fix it (esp. since it's happening in the real world)
you wrote Would you be a good person to pick an owner for this bug? Giving P1 since we think happens in real world and I don't understand severity.
and I are probably the two who have touched it, I think as well but may be misremembering. It should be a straightforward fix to make now monotonic, I don't know when I can get to it
Me neither. If it's P1 I most likely won't have time before v1.3.
I think override the function call in as you did would resolve this issue, what do you think, if you agree with this, I am gonna to try that.
It would fix it, although I think this is broken in the core code, not just in the test. We suspect real workloads have been impacted by this.
Opened to determine whether we actually need to fix the RLQ
Assigning to per


When i doing an elasticsearch aggs with the following statement: then, i got this: Actually, i want the pod names like this:
I have solved this problem by adding a index template for k8s logs. Now, I can aggregation pod names of the same namespace.
How are you sending your logs to elasticsearch? Using the default fluentd configuration?
First, I use the default fluentd configuration. Secondly, My application is written with nodejs language, using library.


is and does not honor the platform it is invoked on. I noticed this when I needed to install etcd on my Mac (which, in turn, was required by and transitively by the pre-commit hooks) and later failed to execute it due to an incompatible binary.
Oh wow, that explains my failures. I ended up using homebrew.


I started the federation-apiserver and went to and got the following output: Expected: I should have got an empty v1.ServiceList Any idea on what we are doing wrong? Have you guys seen this error?
I think your deep copiers or conversion functions are not getting registered properly. If you look at my change in , some places are inappropriately registering with api.Scheme instead of the passed in scheme. Maybe you have that problem or a similar one, since I think you guys copied some code. , Nikhil Jindal wrote:
Deep copy generation was slightly broken as well - you may want to double check that they are registered. On May 23, 2016, at 5:50 PM, Daniel Smith wrote: I think your deep copiers or conversion functions are not getting registered properly. If you look at my change in , some places are inappropriately registering with api.Scheme instead of the passed in scheme. Maybe you have that problem or a similar one, since I think you guys copied some code. , Nikhil Jindal wrote: — You are receiving this because you are on a team that was mentioned. Reply to this email directly or view it on GitHub
I got the same error with latest branch, as I was swamping on service controller, so I use a previous version and worked around it.
deep copies were indeed missing, but adding them hasnt fixed the issue. Will debug more. is my PR to add the deep copies cc Please feel free to file issues for any such problem you face. We need to fix all these issues. Its good if we know about them early. Thanks!
fixes this


https://k8s- Failed: [] KubeletManagedEtcHosts should test kubelet managed /etc/hosts file {Kubernetes e2e suite}
https://k8s- Failed: [] KubeletManagedEtcHosts should test kubelet managed /etc/hosts file {Kubernetes e2e suite}
https://k8s- Failed: [] KubeletManagedEtcHosts should test kubelet managed /etc/hosts file {Kubernetes e2e suite}
https://k8s- Failed: [] KubeletManagedEtcHosts should test kubelet managed /etc/hosts file {Kubernetes e2e suite}
Abhishek ought not be responsible for this any more.
https://k8s- Failed: [] KubeletManagedEtcHosts should test kubelet managed /etc/hosts file {Kubernetes e2e suite}
https://k8s- Failed: [] KubeletManagedEtcHosts should test kubelet managed /etc/hosts file {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention...
[FLAKE-PING] This flaky-test issue would love to have more attention...
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] KubeletManagedEtcHosts should test kubelet managed /etc/hosts file {Kubernetes e2e suite}
https://k8s- Failed: [] KubeletManagedEtcHosts should test kubelet managed /etc/hosts file {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] KubeletManagedEtcHosts should test kubelet managed /etc/hosts file {Kubernetes e2e suite}
https://k8s- Failed: [] KubeletManagedEtcHosts should test kubelet managed /etc/hosts file {Kubernetes e2e suite}
https://k8s- Failed: [] KubeletManagedEtcHosts should test kubelet managed /etc/hosts file {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] KubeletManagedEtcHosts should test kubelet managed /etc/hosts file {Kubernetes e2e suite}
https://k8s- Failed: [] KubeletManagedEtcHosts should test kubelet managed /etc/hosts file {Kubernetes e2e suite}
https://k8s- Failed: [] KubeletManagedEtcHosts should test kubelet managed /etc/hosts file {Kubernetes e2e suite}
https://k8s- Failed: [] KubeletManagedEtcHosts should test kubelet managed /etc/hosts file {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] KubeletManagedEtcHosts should test kubelet managed /etc/hosts file {Kubernetes e2e suite}
https://k8s- Failed: [] KubeletManagedEtcHosts should test kubelet managed /etc/hosts file {Kubernetes e2e suite}
what is the status of this issue?
https://k8s- Failed: [] KubeletManagedEtcHosts should test kubelet managed /etc/hosts file {Kubernetes e2e suite}
No idea. This should have nothing to do with me.
https://k8s- Failed: [] KubeletManagedEtcHosts should test kubelet managed /etc/hosts file {Kubernetes e2e suite}
Ah kk just thought you might know something given the label.
https://k8s- Failed: [] KubeletManagedEtcHosts should test kubelet managed /etc/hosts file {Kubernetes e2e suite}
I know it didn't get any attention last release :)
Status is that I'm trying to fix it... I some code to the test to help debug why the flake was occurring as I wasn't able to repro it on my local cluster.
https://k8s- Failed: [] KubeletManagedEtcHosts should test kubelet managed /etc/hosts file {Kubernetes e2e suite} Happened on a presubmit run in .
https://k8s- Failed: [] KubeletManagedEtcHosts should test kubelet managed /etc/hosts file {Kubernetes e2e suite}
https://k8s- Failed: [] KubeletManagedEtcHosts should test kubelet managed /etc/hosts file {Kubernetes e2e suite}
This seems to be related to issue


Hello Kubernetes, The order of the columns when you get events confused me. I was thinking, Turns out it's just a visual problem where these events are sorted correctly (by ), but the first column is . I think the first column should be because that's the order the events are listed.
This also confused me when I first saw it - I think this ordering would be more intuitive and help users discover the aggregation feature logically, instead of first having to grapple with why events are "out of order"
Should the order be always LASTSEEN,FIRSTSEEN or just when using -w? I have an ugly PR for the latter, but perhaps we should just go for the former.


is failing with: Examples:
This is the failing line of code...
PASS: FAIL: Seems like this disappeared: Also this is flipped: we run the following
my leading theory is this is something in
The issue is that the above PR changed a bunch of to be and thus we no longer wait for getArchive() to complete before starting tests. And unfortunately the previous PR cannot be reverted
After merging, kubelet-gce-e2e-ci still failed with the same issue. reopen the issue. Currently the failure blocked the merge-bot.
did stop the bleeding. Now kubelet-gce-e2e-ci is green: and merge-bot is unblocked.
I think the "go arc" piece was a red herring. I suspect the issue is that the same PR tries to copy the binary into the archive directory, but PR stops building ginkgo, so the copy fails. That is why this line then fails...


When replacing an existing pod, the command fails as it tries to create a pod with a name that is still used by terminating pod. How does the option works? deletes a pod (e.g. mypod) creates a new pod with the same name (mypod) What happens? As the command tries to recreate the pod, it fails as the is in state. Thus it can not be created. The problem is the does not wait for a pod to be deleted. Reproducer: run mypod --image=yapei/hello-openshift --generator=run-pod/v1 get pod mypod -o yaml replace -f --force The last command ends with:
option introduce by
With set to positive value, what should be the default behaviour of the command? When set to , should the command hang for and then create the pod? Or is there a way to create a callback (inside a scheduler?) that creates the pod once the current one is deleted?
You could watch for the resource to be deleted and create after you see the deletion happen
have you ever tried to set , I think this will do a non graceful deletion. should make sure the old resource is deleted before it creates a new resource, if not, that's a bug.
Though the help says the is to be used only with , it returns the error above.
Then I think there are two issues we need to track: 1. option should be ignored or just returns error if users are not using option. 2. should always make sure the old resource has been deleted before create the new resource.


As mentioned in , the federation etcd will lose its state any time the pod dies, unless we modify to add a PVC. We'll also need to create a PV and modify the turnup script to create the backing volume in the cloud provider.
FYI I believe this is a showstopper.
OK. implied I could use to get this provisioned on-demand. AFAICT, , and since it makes our life easier I am trying to get this to work in my test cluster, but so far no joy. Any obvious mistake?
I needed to not specify a .
whew! , Matt Liggett wrote:


https://k8s- Failed: [] Container Runtime Conformance Test container runtime conformance blackbox test when running a container with a new image should not be able to pull from private registry without secret {E2eNode Suite}
Same with , the fix is under review
https://k8s- Failed: [] Container Runtime Conformance Test container runtime conformance blackbox test when running a container with a new image should not be able to pull from private registry without secret {E2eNode Suite}
https://k8s- Failed: [] Container Runtime Conformance Test container runtime conformance blackbox test when running a container with a new image should not be able to pull from private registry without secret {E2eNode Suite}
The fix has been merged , close this one. Feel free to reopen if this happens again.


Either include non-arch specific stuff in every tar or break that out too. ref cc
interested in helping with this one?
Yes, I may help
OK, I did not actually understand what was in , apparently. Alternate proposal: [ ] update all references that need just server binaries to download the server tarball directly [ ] same for places that just need (related to ) [ ] remove all arch-specific binary artifacts from and replace with a script which downloads the client/server tarballs as necessary. The alternative option, including all of the non-arch-specific code and docs in an arch-specific starts to get very tricky, since we run into a large cross product of client x server platforms/archs.


The code in doesn't work for me: does not pass on SIGTERM when the caller is in the same process group. Then the knocks out the process without giving it a chance to kill its child. The bit doesn't help: either the process has already been killed as above, or it gets killed now with SIGKILL; either way the process is still running. Oddly, it does work about one time in ten. I have no explanation for that. I am running from the command-line like this: , on 0-41-generic -Ubuntu SMP Fri Jun 24 11:28:43 UTC 2016 x8664 x8664 x86_64 GNU/Linux CC
The manual says "This is the signal that the calling process will get when its parent dies."; it is applied to the process, so that will get killed if the process dies. That isn't what we are looking for:


https://k8s- Failed: [] [Feature:Example] [] Storm should create and stop Zookeeper, Nimbus and Storm worker servers {Kubernetes e2e suite}
https://k8s- Failed: [] [Feature:Example] [] Storm should create and stop Zookeeper, Nimbus and Storm worker servers {Kubernetes e2e suite}
https://k8s- Failed: [] [Feature:Example] [] Storm should create and stop Zookeeper, Nimbus and Storm worker servers {Kubernetes e2e suite}
https://k8s- Failed: [] [Feature:Example] [] Storm should create and stop Zookeeper, Nimbus and Storm worker servers {Kubernetes e2e suite}
https://k8s- Failed: [] [Feature:Example] [] Storm should create and stop Zookeeper, Nimbus and Storm worker servers {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention...
This has been marked with the "needs attention" label and needs to be addressed for the 1.4 release. Could you please close if it's fixed update on your progress to fix it make a case for it not to block the release or find a new owner who can do one of the above soon. Once you or the new owner update this issue and are actively working on it, you can remove the "needs attention" label.
the following message looks suspicious, looks a DNS issue?
Actually more like a network connectivity issue
do you happen to have the Dockerfile for these images? We better push them into
I don't think this is a blocking issue. It looks a transient issue. Since the example is largely dependent on whether the image can be fetched and how long it takes to pull the image, it is very environment sensitive. I would recommend removing label. Longer term, we should move the images used in examples to gcr.
https://k8s- Failed: [] [Feature:Example] [] Storm should create and stop Zookeeper, Nimbus and Storm worker servers {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
for the last flake, based on , image pull (mattf/zookeeper:latest) started at 42:40, finished at 48:31, about 5min51sec, well passed 5 min
storm-nimbus, storm-worker and zookeeper are from they're all automated builds from github (quick link: and )
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] [Feature:Example] [] Storm should create and stop Zookeeper, Nimbus and Storm worker servers {Kubernetes e2e suite}


I think got confused between e.g. KUBEMASTEROSDISTRIBUTION and MASTEROS_DISTRIBUTION; it definitely doesn't match GCE. PR coming in once I've tested.
cc


Description of problem Create a petset, when pod is evicted, the petset can't create a new pod on other node and make sure pod is running. Kubernetes version (use ): v1.4.0-alpha.2.1427+ How to reproduce it (as minimally and precisely as possible): kubele eviction-hard="memory.available<${value}" a peset and wait all pod is running node MemoryPressure=true all pod status What happened: is evicted and petset can't create new pod and make sure pod is running. What you expected to happen: create new pod on other node and make sure pod is running.
Also when node come back with sufficient memory, petset can't make pod running again.
FYI
cc
Was able to reproduce it. Pet set controller just loops with the following logs I0830 12:32:52. 9 ] Syncing PetSet default/cockroachdb with 3 pets I0830 12:32:52. 9 ] PetSet cockroachdb blocked from scaling on pet cockroachdb-0 I0830 12:32:52. 9 ] PetSet cockroachdb waiting on unhealthy pet cockroachdb-0 I0830 12:32:52. 9 ] PetSet cockroachdb waiting on unhealthy pet cockroachdb-1 I0830 12:32:52. 9 ] PetSet cockroachdb waiting on unhealthy pet cockroachdb-2
-- do you have the petset YAML that we can evaluate? What is the quality of service tier for the pods produced by the petset? Can you include the of an evicted petset pod so we can see the final resource requirements spec?
I am able to reproduce it with nginx pet set in e2e test. Here is yaml output of it Also it seems that i have working fix and almost finished debugging e2e test (patchset is linked above). Will be grateful for review.
Ah, this helps clear things up... the linked pr shows the problem in petset controller and not kubelet...
Does it make sense to set milestone 1.4 for this issue? I think the fix is ready for review


https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite} Previous issues for this test:
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
I'm pretty sure, those amount of failure are related to: (I don't know why yet).
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}


<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? Currently, the 4th parameter name of updateServiceListener in kube-proxy is addOrDelete, However, from the func caller, we just pass for AddServiceListener, and for DeleteServiceListener, So, I think the name addOrDelete is unreasonable. I prefer to change the parameter name addOrDelete to add so that avoid making people confused. In addition, there is a typo, We should change addition to deletion! Also, there is a sentence breaking, we should change from No Service Endpoints Not Found =No Service Endpoints are found Thanks!
Fixed!


In practice this means that it doesn't get to update and thereby may refuse to admit pods which had been assigned to it by the scheduler. For example, if you update labels for that given node, and try to use affinity or on a pod, it won't get to run.
This looks like a sufficient fix for me, but I'm happy to add instead, if that'd help getting it accepted for 1.4.0: cc
cc
That commit makes sense to me.
Thanks, I'll open a PR.


While following this to setup a kube-mesos integration env on my laptop, I found the default service account and default token are not created when start the cluster. As a result, I can not create a pod or deployment in this environment. Reproduce steps: the latest Mesos code, build and start 1 single master with 1 single agent. the latest Kubernetes code, use to build. the following commands to start apiserver, controller-manager and scheduler respectively: all the components are started, run command and see that there is no default token created for the default namespace. At this moment, run "kubectl create -f " will see the failure of creating a pod.
This introduced to node controller and now when we start a node controller, if is not nil, we will start this Informer controller as well. But the start of the will block main thread from starting other controllers later in the main thread. we can see that in the kube-mesos integration, we started some other controllers, including serviceaccount controller, after we started node controller. I will submit a patch later to run the in a go routine to solve the issue.


https://k8s- Failed: ThirdParty resources Simple Third Party creating/deleting thirdparty objects works [Conformance] {Kubernetes e2e suite}
cc
Seems like the namespace creation/deletion process is flaky. IIRC there have been some recent PRs touching this area, is that right?
We recently code to try to expose the real underlying error for why the ns deletion fails. This is hopefully now showing it.
The issue is with TPR not integrated properly with the namespace controller. Creation of a new type requires a restart of the controller-manager given current state of the code. The test must be marked flaky until the state of the art is improved (which is not happening in 1.4).


https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite} Previous issues for this test:
This is pretty much the same failure, that are happening in "gce-enormous" suite and we currently debugging them with The main reason of those failures is that we are terminating all watchers for a particular resource type from time to time, and some of them are actually causing a snowball effect. After more looking into it, my hypothesis is that our machinery for aggregating watchers in apiserver sometimes is just to slow. In particular, I some log in my local branch and what I'm observing is that in etcd_watcher, we are: accumulating objects in incoming channel but also accumulating objects in outgoing channel, and the accumulation there is usually higher [I will send a PR to put those logs into master - they seem to be pretty useful for debugging such issues.] So my current hypothesis is that dispatching watch events to watchers is what is too slow. We will continue debugging.
The logs that I seem to confirm my hypothesis - outgoing channel is generally more full then the incoming channel in etcd_watcher: vs I'm working on the PR that should at least improve the situation.
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
Please apply the label to this issue if it does not need to be fixed before we launch 1.4. Otherwise please advise on how to proceed with respect to 1.4.
This was a regression - I fixed it by reverting an offending PR: This seems to be fixed now.
I have 33 pods in my container (1 is substantive, the other 32 just run on log files). I get this all the time now. I'm running 1.5.4.
- to be honest, I don't undertsand what your wrote. what is "container" in "33 pods in my container"? Also, if you believe there is a bug somewhere, please open a new issue instead of adding comment to more than half-year-old one.


https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite} Happened on a presubmit run in . Previous issues for this test:
failed in https://k8s-
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite} Happened on a presubmit run in .
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
Any updates on this issue If you could you change the milestone to 1.5 and add a label for the responsible team that would be a great help as we triage flakes for the 1.5 release. Thanks!
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
There is a problem with fetching images on GKE. It seems that pod is unable to fetch image for more than 2 minutes. It is strange, as AFAIK the image should be pre-fetched by image-puller:
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
, all issues must be labeled either release blocker or non release blocking by end of day 18 November 2016 PST. If you are not the correct owner please identify the correct owner and add them to this thread for assignment. Thanks for your help. cc:
WRT https://k8s- There was a problem w/removing namespace. This is a duplicate of and is not related to HPA.
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
Could you please help triage this issue as blocking or non-blocking for 1.5 (or help find someone who can). If there is a real issue here, we need to find out ASAP, or else we risk the 1.5 release date.
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
WRT https://k8s- build: kubelet logs for pod rc-light-qyowb : Kubelet is not starting pod for 2 minutes, reason unknown. The same problem is in issue Can you take a look or assign someone from kubelet team?
I suspect that the logging level is set to in GKE tests and so most of the useful logs are not available. Let me figure out a means to bump the logging level to further debug this issue. you might still have to triage the non v1.4 release failures to address concern .
how can we update the environment variables for GKE test clusters? For OSS, we up the kubelet logging for test clusters
I still don't agree with test environmental changes, which is one of the reasons we've never made this a priority. A GKE test cluster looks the same as a GKE prod cluster. How would you debug this in production? Always ask that, and you come back to test == prod.
cc
In my opinion it is release blocker: sometimes pods are not started
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
The pod startup failure you posted in is for release 1.4 branch. Why do you think it is a release blocker? We need to triage other failures on HEAD and/or 1.5 branch which could be release blockers. If the intent is to keep prod and test in sync, then let's up the logging level in prod as well. When issues happen in prod, we might need those missing log lines anyways. I suspect that we might have to have GCI rotate logs more aggressively as a side effect.
Why is the argument to increase the log level rather than making sure that all logs necessary for debugging production issues are at level 2 or above? Levels 3/4 shouldn't be necessary to figure out why a key control loop isn't behaving correctly.
What you suggest is a good fix for the future. We need to reexamine logging levels across kubelet. However, that might not happen in the next few days. How do we make progress now?
Would isolating logging level changes to specific test suites be acceptable? That way, most of our tests would match production setup.
That's what I'd suggested on the other (internal) code review.
Do we have consensus on how to get past this as a release-blocker?
We are waiting on to give a go ahead from the release perspective. The kubelet failures that and I have been discussing about have not been on v1.5 AFAIK.
Sorry, I didn't notice that this problem is on 1.4 branch. If so, the issue is not release-blocker. So, I'm removing the label.
Ack. Marking as non-release-blocker
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
Is it appropriate to move this to the next milestone? (and remove the non-release-blocker tag as well)
PR should be cherry-picked to 1.5 This is not a release-blocker as it affects e2e tests only.
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}


https://k8s- Failed: BeforeSuite {Kubernetes e2e suite} Previous issues for this test:


Another flake i found earlier yesterday, just linking to it for posterity, not blocking me right now. TestContainerStatus
Could you post some error log here? I can't reproduce it with stress test
Hm, got one reproduce:


Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"4+", GitVersion:"v1.4.0-beta.11", GitCommit:"", GitTreeState:"clean", BuildDate:"2016-09-23T22:53:01Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2016-09-26T18:10:32Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"} Environment: AWS - CentOS 7.2 x8664 with cloud-init (HVM) (ami-) OS (e.g. from /etc/os-release): NAME="CentOS Linux" VERSION="7 (Core)" ID="centos" IDLIKE="rhel fedora" VERSIONID="7" PRETTYNAME="CentOS Linux 7 (Core)" ANSICOLOR="0;31" CPENAME="cpe:/o:centos:centos:7" HOMEURL="" BUGREPORTURL="" CENTOSMANTISBTPROJECT="CentOS-7" CENTOSMANTISBTPROJECTVERSION="7" REDHATSUPPORTPRODUCT="centos" REDHATSUPPORTPRODUCTVERSION="7" Linux ip-172-31-35-103 3.10.0-327.el7.x8664 SMP Thu Nov 19 22:10:57 UTC 2015 x8664 x8664 x86_64 GNU/Linux Install tools: Others: What happened: when selinux is enabled i was getting following error on discovery container What you expected to happen: i expected node to be able to join master, but nothing was running on 9898 port of master i.e. discovery due the error mentioned above How to reproduce it (as minimally and precisely as possible): use the api which i gave you and do normal kubeadm init Anything else do we need to know: This issue got fixed with setenforce 0
This issue got fixed with setenforce 0 but want to track it
cc
BTW the issue was on latest Centos iso too downloaded from here
cc
this is to do with secrets, and kubelet log is the place to look for more info, could you please grab and paste it in a gist (it's gonna be fairly large)? Also, the kernel audit log would be handy, do you know how to get this (I forgot)?
not sure if this will help
the lines are incomplete, it's best if you redirect it to a file and upload it somewhere with a link here.
could you also provide the output of ?


Kubernetes version (use ): Environment: Cloud provider or hardware configuration: Any OS (e.g. from /etc/os-release): Any Kernel (e.g. ): Any Install tools: Any What happened: If I use a template with the seperator, and if the line with the separator has trailing whitespace characters, then kubectl considers that to be the EOF instead of reading further. e.g. template What you expected to happen: It should not stop reading if there are trailing whitespace characters, it should just trim the separator line and ignore the whitespaces after the . This is how it worked until v1.3.0 kubectl version. How to reproduce it (as minimally and precisely as possible): Just use the template from above and you'll notice that it won't create the Service, but it'll create the Replication controller. Anything else do we need to know: This is on kubectl version 1.4.0
kubectl regression. cc
This bug is general (can be found on get, apply, delete, ...). May be somewhere in resource builder.
Might be , Janet Kuo wrote:
Confirmed - is the cause. would you take a look?
We may need to cherrypick the fix
Yaml rules are separator must be at beginning of line, three dashes, and new line. Did we change how strict we were in that PR? On Sep 27, 2016, at 7:46 PM, Janet Kuo wrote: We may need to cherrypick the fix — You are receiving this because you are on a team that was mentioned. Reply to this email directly, view it on GitHub , or mute the thread .
Yes, and the test was changed too (originally tests allow with trailing spaces)
According to spec (if my spec-alese is correct) newline + three dashes is the only requirement to terminate a document. Whitespace or prefixes are allowed afterwards, but for it to be valid YAML those must be valid at the beginning of a doc as well (as far as I can tell). So we should split and remove on any line starting with a newline and three dashes - but not consume the remaining whitespace (I believe) , Janet Kuo wrote:
We should probably exit non-0 and have an error message if we see something in the file that we don't expect. E.g. is neither a separator nor parseable yaml.
We should add a test that uses a YAML document prefix rule and verify that it correctly passes / fails when YAML goes to parse. From reading the spec it appears that anything that is invalid when n--- is present is also invalid in a single document context, and vice versa. , Phillip Wittrock wrote:
I have rechecked the code, and I have located the root cause, since I am outing for a training these days, I will fix this right now after the training on Friday.
I am quite sorry about the regression. :(
should we allow trailing whitespace characters after , if so we can trim the whitespaces when we read line by line?
is it possible to include this fix to one of the 1.4 releases? We'd really like our users to have of k8s alpha features (Scheduled jobs is one of them) that can be configured only using 1.4 client; w/o them potentially hitting this regression.
I will double check this regression, and open a cherry-pick PR.
I am thinking create a cherry-pick PR for this to include it in 1.4, wdyt?
I would like to see this in 1.4 since as you note it may break scripts. On Oct 19, 2016, at 8:19 PM, TonyAdo wrote: I am thinking create a cherry-pick PR for this to include it in 1.4, wdyt? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub , or mute the thread .
Do we have a test case now? :) On Oct 19, 2016, at 8:19 PM, TonyAdo wrote: I am thinking create a cherry-pick PR for this to include it in 1.4, wdyt? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub , or mute the thread .
we already test this, you can see


Just saw this warning today: Not sure if this is only limited to kubemark, or if there are other places we need to fix, too.
Sorry if this sounds stupid, but as expected, won't show this warning - what needs to be fixed here?
we have a bunch of scripts and Makefiles which are omitting the currently. We need to fix these by adding the missing s.


See
CI has built and uploaded unofficial prerelease darwin binaries for 1.4 using go1.7.1: cc Official binaries will ship as soon as we cut 1.4.1.
We have official v1.4.1 binaries available now, too.
Hooray


<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): No What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): watch leak Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): 0 Environment: Cloud provider or hardware configuration: vagrant on Mac OS X OS (e.g. from /etc/os-release): CoreOS 1164.1.0 Kernel (e.g. ): Install tools: Others: What happened: We ran controller which handles our own ThirdPartyResource. When we hit "watch of ... ended with: 401: The event in requested index is outdated and cleared (the requested history has been cleared" error, "forcing resync" log is duplicated. This log message comes from . It looks like this goroutine does not exit when we hit the above mentioned error, and continued to work, and running "resync" more frequently. What you expected to happen: The resync goroutine must be terminated when watch is closed (i.e., ListAndWatch function returned), instead of leaking it. How to reproduce it (as minimally and precisely as possible): I just run our controller for ~15 minutes. I'm not sure why "401: The event in requested index is outdated and cleared (the requested history has been cleared" happens, but from the issues I read, it should be harmless. Anything else do we need to know:
I think I see what is going on here. If exits for any reason beyond an error from the resync loop no one will tell the resync loop to exit. Furthermore, there is no "local" way to do this as it would be unexpected for this function to signal the Done channel as other things outside of it may be relying on it. There are some useful functions around Context () that help to create a "local" copy of context that can be used in this situation.
Thank you for reply. Isn't is possible to create local channel in ListAndWatch, and defer close it, and then make resync goroutine "select" it too?
This issue was labelled as component/kubectl, but actually it is not limited to kubectl. All controllers suffer from this. Only people notice this if they enable V(4) logging.
The really annoying thing caused by this issue is that eventually resync is done in every few seconds due to the leaked goroutine. Even if you set 30 second resync period in controller, leaked goroutines call resync too, and you end up having resync call very very frequently. If you have many resources in cache, it really performs unnecessary sync loop. This is the last thing you want.
this looks like a real issue. Could be related to Could you help triage? looks like a proper fix.
I agree - this looks like a real issue. And indeed LGTM as a fix for it.


Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Master Environment: Cloud provider or hardware configuration: OpenStack OS (e.g. from /etc/os-release): n/a Kernel (e.g. ): n/a Install tools: n/a Others: What happened: now can update a LB instead of deleting it and recreating it. However the security group code is still set to try and create security group rules, and if it gets an error, delete the LB and start over. What you expected to happen: Updating a service does not delete the LBaaS LB How to reproduce it (as minimally and precisely as possible): Enable in the section of Restart a Anything else do we need to know:
you're absolutely right, this is a critical issue (if you enable security group management). The dangerousness of aggressive cleanups has already been discussed in detail in this thread: I hope you guys can fix this.
Yup - I have a larger refactor, that ensures this, and a few other edge cases we have found in our development. It is in massive need of a rebase though, so I will need a day or 2 of uninterrupted time to get it together
Are you still working on this? If not, I can take a look.
ping :)
/close
We just ran into this where Kubernetes recreated an LB with a new Floating IP because the OpenStack API was not available for a few seconds while the LB was updated. Could this issue be re-opened? Since this is a major bug for us, we will start working on a fix for this today.
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
There are no sig labels on this issue. Please add a sig label by either: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. See the . The in method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


PetSets currently support an annotation in their PodTemplateSpec field. From the user documentation: Debugging: If one want to bring up the pets in my petset one after the other, while pausing after each step, confirming that the state is as expected, one may use this functionality. For example, a petset where each pet is a replica of a distributed database; we set the to in the petset template initially. This leads to the following progression: PetSetController creates pet-0 Manually intervene and check that it has initialized as expected. Set annotation to true (equivalent to marking that pet as healthy) PetSetController creates pet-1 Manually intervene and check that it has initialized as expected. ... Blocking the PetSet controller: We may want to ensure that the petset controller is not performing scale operations when a different user-initiated action is in progress (for example, where pets are leaving due to a kubectl drain). This may be used as a mechanism to orphan a pod from a petset controller (for a closer look), while ensuring that a replacement is not created. The use case 1 above can be realized using and is probably easier to do that way rather than use the annotation. I don't see a reason to have that case supported when we can have users scale up a PetSet gradually and manually in an identical manner. It may have been of more importance before v1.4 when the scaler & reaper were not implemented. Due to the second use-case, if it would make sense to continue to have this ability, I propose that we move it from the PodTemplateSpec to the PetSetSpec, where it will eventually reside. This seems to me, like the logical place for such a debug hook to reside. In order to run/pause the scaling operations of the PetSet controller, we could have annotations along the lines of with the annotation defaulting to when absent. /cc
Should we just support Pause on PetSet (for part of these cases)?
Do users have to catch the roll out at the right time in order for this to work? For example if I want to pause between nodes 2 and 3 coming up, do I need to be precise in my timing when editing the resource? How does this look from the users perspective?
Note that Pause allows scaling in Deployments, it just blocks upgrades. That being said, I think Pause will be useful for PetSets in general. Not sure though if we can use it somehow for this specific issue.
I like the debugging hook for PetSets. If it's optional I don't see any reason to get rid of it.
pauses petset controller if necessary and inspects the state of the system_ With the previous flow, the petset controller was paused after each pet's creation automatically, and here it would be an extra step to pause the controller it if necessary. I don't know how important it is though, but to me it seems easier to understand than a per-pod annotation.
and I had several discussions with the lifecycle of what would be useful. There are a bunch of use cases that overlap: not a primary concern for most users but helpful (prevent changes) - very important if you are triaging failures. Technically scaling is itself a user action, so if you don't want your pet set autoscaled while triaging failures you need to take action on the HPA like deployments. Since PetSets and ReplicaSets are nothing alike w.r.t. scaling I'm not terribly concerned that pause means slightly different things if for end users it Does The Right Thing control by a third party controller - this is the babysitter controller wanting to reuse the petset controller for core stuff but still have the hook angle. Not thought through enough to design around.
Note that Pause does not prevent scaling in Deployments, only upgrades , Clayton Coleman wrote:
Just to summarize: The current behavior with the annotation It is set on a per-pet basis. Setting it to false on any single pod blocks all operations on the petset; scaling, creation of new pets, etc. All pets must contain the annotation, and have its value set to true in order to make progress. By default, it is set to false and makes the petset controller pause after creating one pet. The smallest change we can make for beta: Rename it to (or some better name) Make it optional (with the absence of the annotation meaning that the petset can scale) Surface the reason better when we block on scaling because of the annotation. I still think there is value in moving it to be an annotation on the PetSet resource, rather than having it be in each pod, which is confusing to users. wdyt?
Putting it on the pod template is clever, but surprising. How many examples are dependent on it? , Anirudh Ramanathan < :
Last I checked, all the examples had set to true. We should just get rid of it from all examples in my opinion. If we make the default (or the absence of any such annotation) to mean that the petset controller can make progress, we can safely do that. was going to do a pass over the examples after the PetSet renaming. This could be an additional step. Does that sound ok?
If it's not found, we assume you want to pause. You're proposing that if it's not found, we default to "forge ahead". SG. It should only be used in debugging/testing today. Basically if you don't know how to write your , you'd create a petset with it set, kubectl exec and write it out, scale the petset by 1, kubectl exec ..., and when you have a working cluster, put what you did in an init container and create the petset without it. This is the absolute min. You're also proposing that we move it to the petset. This makes sense overall, but I initially didn't do that because I didn't know how we would end up doing pause.
We could leave it there as "alpha" and then remove it in 1.6 once we have more time to talk about how upgrades and debugging and pause should work? , Prashanth B wrote:
Yeah the minimum work is to flip the behavior of the controller when the annotation is not found:
Agree on flipping behavior. , Prashanth B wrote:
Flipping SGTM. are you suggesting we leave the annotation-name as it is, and don't change it to ? If we are changing it, we should probably change the name from as well, since it doesn't make as much sense anymore. Probably ?
or healthy rather.
Flipping default behavior (when not present). Or did I misunderstand beeps proposal? On Oct 27, 2016, at 7:11 PM, Anirudh Ramanathan wrote: Flipping SGTM. are you suggesting we leave the annotation-name as it is, and don't change it to ? If we are changing it, we should probably change the name from initialized as well, since it doesn't make as much sense anymore. Probably unhealthy? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub , or mute the thread .
Renaming it to beta is effectively moving it to beta, so I think given that we're still confused about how this ties in to the big picture, that might be premature. If it's either move to beta or drop, I'd probably vote for drop, it's a debug hook after all. Renaming it to something else but keeping in beta makes sense.
sorry, keeping in alpha
whats going on with this tag? it's used in the docs here should i use it going forward?
It's alpha and may be removed at any time. On Nov 2, 2016, at 3:11 PM, Zachary Adam Kaplan wrote: whats going on with this tag? it's used in the docs here should i use it going forward? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub , or mute the thread .


Kubernetes version (use ): HEAD Environment: GCE What happened: When I set up I can see the following message: What you expected to happen: master should not be registered
cc
Looks like the env var isn't being respected any longer by the master scripts (but the logic for # of expected nodes did adjust correctly).
/cc I think this may have broken in because it which unconditionally sets the field that tells a kubelet to register with the master outside of the guard that checks .
Fix in progress: . There was a problem caused by unconditional setting on the master in .


From . During the firewall e2e test we need to change the service type from to to avoid leaking LB resources. But found it is not working because the field. Error I got as below: If don't delete this field, got instead another error:
Just saw the TODO comment . Working on it.


The logic to account for all pods being ready is incorrectly counting some pods as being controlled by a replication controller. This is causing the 1.3-1.5 upgrade tests to fail. See https://k8s- as an example. Since , the pre-flight checks fail and the rest of the tests never run. I think this was exposed by which changed some things from ReplicationControllers to Deployments. There's a pod being created by a deployment that also matches the selector of a replication controller, and is therefore being incorrectly counted by cc


Similar to this is to test the first contributor experience fails on ubuntu 16.04, docker version 1.12.3:
Got the same error on Linux ubuntu 4.2.0-27-generic ~14.04.1-Ubuntu SMP Fri Jan 22 15:32:26 UTC 2016 x8664 x8664 x86_64 GNU/Linux Docker Client version: 1.6.2 Client API version: 1.18 Go version (client): go1.2.1 Git commit (client): OS/Arch (client): linux/amd64 Server version: 1.6.2 Server API version: 1.18 Go version (server): go1.2.1 Git commit (server): OS/Arch (server): linux/amd64
What git sha of were you building from?
I have cloned the head. I think the reason might be missing package: gccgo-4.8-aarch64-linux-gnu I'm compiling right now and update on result.
were you running or ? runs inside docker, so the gcc error is confusing.
I run: sudo make release in Ubuntu VM running in VMWare workstation if that matters. Are you in kubernetes-users slack channel?
does fail in the same way?
Do you need to run , instead of just ? Can you try running the following:
well, if I don't do sudo docker does not work :) I need to run as root. I'll try the commands as soon as quick-release ends.
ah, gotcha. you can probably fix the docker access issue by following in the docker manual. that's probably not why it's failing, though.
make release still does not work :(
can you please run and then paste the results? ( turns up the debugging verbosity.)
for me: works (using your PR to fixes the gobindata dep) -local Go build- works (docker build) works if you have Docker running. does not work: on OSX did not seem to give more verbosity.
V=4 make release did not help :(
oops, that was not the correct environment variable, sorry. I meant .
here we go:
Aha, I think I figured it out. Sorry for making you rebuild so many times; the issue still isn't super obvious from the logs. Basically, the bug only occurs if you don't have enough memory to do the docker builds in parallel; if that happens, you fall into this . On the parallel side of the branch, we set the platform env vars within a subshell, so each of the cross-compilation builds won't affect each other. On the non-parallel side of the branch, we don't use a subshell. Inside , there's no overrides done for . As a result, we end up incorrectly reusing the value for , , which doesn't recognize the flag being applied for the darwin build. The fix should be easy; I'll get that sent out ASAP. we probably want this fix for 1.5, since if you fall back to non-parallel crossbuilds we're doing the wrong thing.
Ack. Will keep an eye out for the fix. Please mark it as cherrypick-candidate
Thank you guys!
I think it is almost there. Got much further in the build (4 GB of RAM). Towards the end there is a failed test that makes the build fail: This is probably a different bug, as your PR got passed the misconfigured compilation.
FWIW, the worked on ubuntu 16.06. It did ran some tests apparently but the build succeeded.
I just tried with and it worked fine
since we have a PR ready and working, can we please switch this to non-release-blocker?


On kubernetes 1.4.6 taken from releases. Using CentOS 7.2 Kubernetes worked with a 1.2 version. I just uninstalled and redo installation with 1.4.6. SELinux is disabled Everyhting seems to works (controller-manager, kubelet, and so on...) excepting kube-proxy that returns an error: "sysfs not mounted". API is responding (internally and externally), and I can see nodes The command that is launched by systemd is (--v=8 to get a maximum of information): But sysfs is mounted on /sys: I also tried to set /sys as writable (only to be sure, with chmod +x), and also tried to use userspace proxy mode. Note: 1.2.6 version was ok I think that the problem is that sysfs mount "none" as label, not "sysfs".
Note: I can now confirm that the problem comes from the "label". I did this: Then kube-proxy successfully started.
before setting . This logic was introduced in v1.3 by to workaround a known docker issue That's why "1.2.6 version was ok". Because sysfs is mounted as , kubeproxy couldn't find it then reported error. But why sysfs is mounted as ?
I installed kubernetes on a "scaleway" (a very nice cloud provider) VPS cluster. Their VPS uses a init file that mounts fs like that. I opened an issue on their github. But anyway you may mount "none" for tmpfs or any ramfs, sysfs, bugfs. What I think is that the mount listing function that is used in kube-proxy code tries to list the mount "source" but not the "type". For that kind of mountpoint you can use "none" as source.
the problem is here: You should check .Type and not .Device. "Device" is what I named "label" or "source" because I didn't remember the name ;) Sysfs can mount a "none" device as type "sysfs", so maybe you can change. See:
I'll do a PR
SGTM.