Question:
Currently a Docker bug may cause KubeProxy to crashloop thus making the Node unable to use cluster network, which in turn makes Pods scheduled on this Node unreachable. To mitigate this issue we need to surface the information about problems with KubeProxy, that will allow scheduler to ignore given node when scheduling new Pods. ProblemAPI is going to address this kind of problems in the future, but we need a fix for the 1.3. Either Kubelet or KubeProxy needs to update a NodeStatus with the information that Node networking is down. I suggests that we reuse NodeReady Condition for this, as this would make the rest of ControlPlane work out of the box. We will replace this fix with the ProblemAPI use in the future. Another option, which requires more work, is creating a completely new NodeCondition type to handle this case. This would require making Scheduler aware of this new Condition.
Does kubelet monitor kube-proxy health?
Kube-proxy today runs as a static pod (daemonset), kubelet treats it the same as other static pods. Yes, Kubelet reports kube-proxy podstatus back to master, that is all. The detail information on kube-proxy is in crashloop at
cc/ Brandon, this issue explains my long position against running kubelet, kube-proxy, etc. critical daemons as docker container. I am ok to packaging them into docker image, and running as linux containers, but they shouldn't depend on docker daemon.
I think we don't want to run them as docker containers as well, we want to run them as rkt containers.
The initial proposal I reviewed was docker container, and I did raise my concerns to and coreos developers then. Running it as rkt container should be ok since rkt is daemon-less today, and shouldn't have this chicken egg issue at the first place. But we still have the problem on the nodes without rkt as runtime. At the meeting, I suggested that kube-proxy update NodeStatus with a newly introduced NodeCondition, like what we plan to do with kernel issue at: The reason of not using NodeReady is that kubelet might override that. Another very hacky way is before crashing, Kube-proxy logs the read-only filesystem problem somewhere, like /var/lib/kube-proxy/... Kubelet will pick up the information, then update NodeReady Condition with detail error message prefixing with "kube-proxy:" kube-proxy's responsibility to make the information up-to-date.
Can I have more description of the bug? Is there a related docker issue? Is there a repro? Does this only affect kube-proxy or will it affect other pods on the node? We need to support running critical applications (including system daemons) in pods in order to allow easy deployment and integration with vendors (e.g. storage and network vendors). cc since we were just talking about integrating through addons. What features are we missing that we can't support this right now? Is it just that our default container runtime is buggy and flaky and unreliable? Do we need the NodeProblem API? Do we need taints/tolerations?
- I don't know about any way to repro it other than running huge clusters. Pretty much every run of a 1000 Node cluster has this problem on at least one Node.
If you want to know the detail, see It is a docker issue, and pretty rare.
Can we regroup on this issue this week? Like tomorrow? , Dawn Chen wrote:
I will happily make kube-proxy write to or offer up pretty much any API we deem necessary to detect this issue. To me it still feels like a docker health-check suite is needed, and one of the tests is that sysfs gets mounted rw. BUt in the mean time, how about this: make be the API. Any component which feels that it has a node problem can write a JSON file into that directory. For this case, kube-proxy would write a file: Kubelet can turn that into Node status reports. , Tim Hockin wrote:
cc
PR of introducing NodeProblemDetector was merged last week, and the pr to enable it by default for Kubernetes cluster was merged over the weekend. So far everything works fine. Even it is an alpha feature, GKE folks agreed to enable it for GKE. To workaround this particular issue before we have proper docker fix, we can easily extend today's KernelMonitor module of NodeProblemDetector to monitor kube-proxy log, and report a new NodeCondition called SysfsReadonly to the issue visible. But we don't have remedy system yet, and pr of converting node problem (events, conditions etc.) to taints is still under the debate / review: 1) Should NodeController respect this new Condition, and simply mark node NotReady? 2) Should repair system pickup this issue, and restart docker?
Do we still observe the problem with docker 1.11.X? If no, I want to close this; otherwise, we are going to reconfigure NodeProblemDetector to make the issue visible.
cc/
Agree, I'd love to not "fix" this..
I think we've seen it the day before yesterday during our tests - can you please confirm?
I can't recall - we certainly did saw some failed nodes during the test, but I'm not sure we made sure it was a kube-proxy issue.
But what I can tell for sure that it's definitely less frequent - we started 2000-node cluster 10+ times during this week and we've seen it at most once (which makes it at least 99.995% reliable).
Not enough data. GKE is still deploying the old ContainerVM and isn't running Docker 1.11.1 yet, unless I'm mistaken.
aah ok - I didn't know that...
I'm going to tentatively close this until we show that it's a problem on Docker 1.11.2, now that Docker 1.11.2 is in on both GCE and GKE.
(Obviously feel free to reopen if I'm wrong.)
Another occurrence:
So it's still happening, can we teach NPD to detect and surface this as node unschedulable?
Yes, we can. But I found that the error only occurred once with docker 1.11 till now. At least, this should be a known issue, right? XRef Do we want to fix this in 1.3 or just claim known issues in release notes? :)
Doesn't this render the node unusable (i.e no services will work because kube-proxy is broken)? if so, we should surface regardless of frequency IMO. Whether we fix or paper over it is a different issue.
+1
Because it's code freeze now, I can add a temporary short hack in node problem detector to detect this issue. Should we do that? :)
Removed the priority and it back to 1.3 milestone, so that we can discuss at burndown meeting.
Yeah i think codefreeze just means no features
It is very rare after we upgrade to docker 1.11.X. To extend KernelMontor in NodeProblemDetector to detect the issue is totally overkill, since we have to allocate more compute resource to NodeProblemDetector to process tons of kube-proxy logs. Here is what I suggested for this issue for 1.3 and 1.4+: Document this as known issue for docker releases: docker 1.10.X (), docker 1.11.X (). Also document it as Kubernetes 1.3 release known issue at is also updated admin guide for NodeProblemDetector, and the suggestion on how to handle this issue will be included there. For 1.4, I think we should change kube-proxy. Instead of logging the error then crashing, kube-proxy should report its problem to NodeProblemDector, which aggregates the node issues and propagate them to upstream layers, so that the admin / remedy system & control-plane can take proper action upon the issues. I am removing this from 1.3 milestone, and unassigning myself. cc/ to find the right owner from cluster team for next release here.
On a 2k node cluster, I just saw 4 nodes come up this way on Docker 1.11.2. I'm not saying it's going to stop the release, but it does stop bringup on large clusters pretty frequently. For some reason we went through a "nice" patch where it was rare, but today it was just bad.
We can have a quick fix in node problem detector to catch this kind of problem by parsing the kube-proxy log. The only problem is that, kube-proxy generates logs relatively fast. If we let node problem detector do that, it has to waste significant cpu even on the good nodes. A better solution is to make node problem detector expose an endpoint and let other system components report problems. However, we are not there yet for 1.3. If we'd still want to solve this for 1.3, we could: 1) Let kube-proxy have some kind of error log, and only print error into it. Then we can easily change node problem detector to parse the error log and report problems. By this, we can at least make these errors visible to users for 1.3. 2) Or let kube-proxy report the node condition itself. /cc
Comment at is still valid. We cannot do much for 1.3 release here, and extending NodeProblemDetector to parse the application logs is wrong move. Also NodeProblemDetector is not running on GKE nodes yet.
I'm seeing people say they reproduced on 1.11.2, is there a 1.11 docker issue? I know that was closed. If I'm going to get my docker guys to dig, I need a docker bug report...
Can you please ramp up on this and come up with a way to bring node and kube proxy teams together to resolve? Even if we don't land in 1.3.0, I propose this is at least a P1 for 1.3.1.
FYI and others In I a hacky way of resolving those issues to our test framework. In the last run of gke-large-cluster run it seemed to work (there was 1 broken kube-proxy, this triggered restarting docker, which in turn fixed the problem). This is obviously not the solution, but: it seems to prove that 'docker restart' solves the problem it should stop the bleeding in our tests
& I copied & paste what I wrote in another internal thread to give you some background: No, it is not a blocker for 1.3 release: It is not a regression from 1.2 release. It is way less frequently after we upgrade to docker 1.11.2 from docker 1.9.1 NodeProblemDetector is not designed for processing arbitrary logs. It is an overkill solution given such low frequency and relative high resource overhead. By default NodeProblemDetector is not running on gke node due to the extra overhead per node introduced to our customer. Meanwhile Since GKE already has repair system built, it should be very easy to extend GKE repair system to check if kube-proxy is in crashloop. If yes, then kubectl logs to retrieve the last terminated container log to see if the issue is caused by readonly root filesystem. If yes, the repair system can issue a docker restart.
agreed not a blocker. This is likely only fixable in docker. We should keep it up the priority so we dig into and fix docker. can try next week to get our docker people talking to whoever on your side you want to get this fixed (in 1.11.3 or later probably)
Can we add a node readiness or liveness probe to solve this? (I remember there is a issue for this, but can't find it now) One of the default behaviours of the probe is to make sure the key node components are running normally, and the behaviour could be extended overtime. Probably adding a node probe controller or node health controller to perform the probing. Or just use the health probe of GCE MIG, if its functionality is enough to handle this. This could be part of the remedy system. Just random thoughts, not for 1.3 and even may not work. :)
All are good ideas. But let's move that discussion to a separate issue, and use this one to focus providing the workaround for KubeProxy panic within 1.3 timeframe. Thanks!
Also what you suggested probe controller can be easily achieved by GKE repair system which I suggested above.
Agree. needs quite a lot of work, definitely not for 1.3 :)
Keeping in v1.3 until we decide on Docker 1.11 for sure.
The issue seems to be present on old and new docker alike, sadly.
I opened a docker issue here Feel free to supply information on that issue.
So what's the right kube-proxy fix? , Lantao Liu wrote:
- didn't we decided that we just want to discover this problem in NodeProblemDetector and then mark this node as not-healthy (whatever that means - adding a new condition or extending NodeReady or ...)? In my opinion if it would be clear to user why this node is not healthy, this should be good enough. FYI in I worked around this in our tests :)
It wasn't clear to me if kube-proxy should do something different or not. , Wojciech Tyczynski < :
yeah this is a docker bug (), unsure if there's a pod level workaround, couldn't make out if it was specific to hostNetwork. the work around is restarting docker. It happened more frequently in 1.9, but still happens on 1.11 (node team empirical data). To surface it to the NPD we need to either send some http event to a NPD server (vaporware) or write to a special hostPath termination log type file (parsing kube-proxy logs form NPD is too expensive).
This feels like a hot potato but I am not sure if it still needs to be addressed or not. If it does, let's do so. I apologize for not being able to understand what the current status is. It looks like we have a . Reading comment from 12 days ago she states: While folks may want the node problem detector extended to parse logs to detect it, that is not a good fit for that system and it doesn't run on all nodes anyway. Also GKE can be extended to possibly repair this problem. Questions: Is this an issue that we need to still address for production? Or can we wait for docker fix? If we haven't fixed the kube-proxy panic (which we should) how do ensure we still provide a signal that this is happening?
I am leaving this on 1.3 until we know the best thing to do.
If we can DIRECTLY tell the node problem system about a problem, I have no qualms about doing that. drop a file in a dir, for example. , Michael Rubin wrote:
Node problem detector is not running on GKE cluster now. :) and should be a better fix for this. If we still want a fix in the node problem detector, letting kubeproxy drop a file could be enough for now.
If you can give me a spec of what dir to mount and what sort of file to write, I'll do it. , Lantao Liu wrote:
For temporary fix, something like should be enough. The kernel monitor of node problem detector could be ported to parse other logs. The only reason we can't do that for kube- is that the log is too spammy. should be enough for now. Only the kubeproxy panic error is printed directly into : Even though the default  of glog is , I checked the node in and cluster, normally only few lines are at Error level.
I thought that Dawn said we DO NOT want to parse logs. Parsing logs is, in general, a total hack and very very brittle. we should have a real push API. , Lantao Liu wrote:
I agree with I also understood thought we should not parse logs. Is that common practise for the node problem detector? It feels fragile to me and to be avoided.
Yeah, I agree parsing log is fragile, that's why I call it a temporary or a quick fix. :) The only good thing is that it doesn't need significant change on both sides, although I don't think it's a good solution, either. :( For a better fix, it will need some more design and work. Let me think about it a little more~ Anyway, I still think and is needed. As a key component running on each node, it's wired that we don't have a component to monitor whether it is ready or not.
Re making kube-proxy "not panic", it currently fails in resizing the hash table for conntrack entries, for which it needs to write to /sys. I'm not sure if there are more locations (we moved /sys/module/br_netfilter loading into kubelet IIUC), or if conntrack itself needs write access to /sys. I think Dawn/Liu are more worried about the NPD not parsing kube-proxy logs, because its memory usage would balloon. If kube-proxy can detect the situtation internally, either by parsing its own logs (which is brittle) or poking someting in /sys for example (like the hash table resizing it's currently failing on), it can drop a token into a hostPath (hoping the docker bug doesn't manifest as ro hostPath). Kube-proxy will still remain dysfunctional, though we might be able to: the issue/mark node unusable a feedback loop that restarts docker We just need to be careful to cleanup the hostPath so it doesn't end up in a restart loop.
My understanding of 1.3's state based on a brief chat with : something like ~1 in 1000 nodes will be affected by this issue and if nodes are affected, pods scheduled on those nodes have {broken,little,no} networking. We hope Docker will eventually fix it. But because that's not the case for 1.3, and we don't want to make excessive changes to 1.3, perhaps that means that we should fix it expediently. Summarizing my understanding of previous discussion: expedient thing would be some mechanism to detect ( proposal like kube-proxy detecting and writing to /tmp/kube-proxy-claims-docker-issue-) and $something (kubelet, node problem detector, a bash script in a loop) that checks for existence of that file followed by some form of remediation {kill docker, sudo reboot, .....} to bash the node into working again. Maybe an ugly hack is okay for now and we hope that docker fixes it for the next release so in 1.4 we can rm the code? Or for 1.4 we can do something that is More Elegant And Less Offensive To Engineering Sensibilities?
Bounded-lifetime hacks are fine, but they need to come with giant disclaimers and named-assignees to cleanup the mess after a specific event. , Alex Mohr wrote:
Agree, re: bounded-lifetime hacks needing care and we shouldn't do them indiscriminately. We and our users have a problem today -- and we've had a problem for the past N months without traction. I'd like the problem solved for 1.3 because I'd like us to have a great product and I don't think the state of broken nodes is there. Aside; we've spent N hours discussing the issue and could likely have worked around the issue in less aggregate people time than we spent discussing? Apologies for the leaked frustration: this issue seems to be stuck in either some form of analysis paralysis or cracks between teams or perfect-is-the-enemy-of-the-good state. Again, I don't care about implementation details. We also need more than just tech or a piece of code. Exit criteria is that users don't get broken nodes shipped to them.
We're not going to solve this problem for as long as we depend on a lagging docker release cycle, and they don't backport or do intermediate releases to help us. This is just a one off docker bug that's left us in a bad state. Such issues come up every release, get documented in the release notes and we ship anyway. The easiest fix (disclaimers and all) is to redirect kube-proxy stderr and parse it out from NPD. Even then, the node will remain broken unless we restart docker. the NDP will only surface the error.
I'm working on a PR yesterday night, will send it out soon.
FYI, a fix is here . In , I let kube-proxy update a node condition with specific reason, message and hit to the administrator about the remediation. The specific reason why we want to do this is discussed in the PR description
FWIW I agree 100% with using node condition SGTM, but your PR should probably also modify the scheduler here to prevent the system from sending new pods to a node that has RuntimeUnhealthy condition. I realize there will still be a time window when we might send some pods to that node (before the problem is detected) but at least the time window will be bounded this way.
Yeah, we can do that. But before that, we should make sure that the node is really unusable without setting conntrack. In fact, we only setting conntrack to increase the max connection limit on the node (default 64k-256k) Without this, is the node really considered to be unusable? If so, we should prevent scheduling pods to the node. If not, maybe we should just surface the problem to the administrator and let the node keep working. I'll run e2e test without conntrack set to see whether there is any problem. Yeah, that's why I think we should not restart docker ourselves to try to remedy the problem. There may still be workloads running the node. We'd better leave this to the user to decide. :)
is merged, and hopefully, it could solve this issue. I've also sent a PR to revert the walkaround in the test framework to verify whether the issue is really fully solved. Feel free to reopen this if the issue or relative issue happens again.
thanks for the help!
I just ran into this, and I am able to repro consistently. It might be due to the craziness of my experiment, but I still wanted to contribute the data point. I am experimenting with docker inside docker. All I am about to describe is itself running inside a privileged CentOS 7 container that has systemd and Docker installed. The CentOS 7 container is running on my dev machine (Docker for Mac). I have a kubelet running fine, and all control plane components running as static pods. When I attempt to run kube-proxy as a static pod (privileged) it crashes with the following logs: sysfs is mounted as rw: sysfs is mounted as rw on containers: Using busybox to further inspect:
I'm seeing the same issue when running kubeadm-dind-cluster, where kube-proxy is failing to come up, because it is trying to write to /sys/module/nf_conntrack/parameters/hashsize. This is on the filesystem sysfs, which "mount" shows is rw (so it passes the R/W check in kube-proxy code), but the filesystem is not writeable. Within kube-proxy: I'm also running docker containers inside of docker containers. FYI, others have not seen this issue, when just running docker on bare-metal.

glog.Infof("Setting nf_conntrack_tcp_timeout_established to %d", seconds) return sysctl.SetSysctl("net/netfilter/nf_conntrack_tcp_timeout_established", seconds) } <ins> // isSysFSWritable checks /proc/mounts to see whether sysfs is 'rw' or not. func isSysFSWritable() (bool, error) { const permWritable = "rw" const sysfsDevice = "sysfs" m := mount.New() mountPoints, err := m.List() if err != nil { glog.Errorf("failed to list mount points: %v", err) return false, err } for _, mountPoint := range mountPoints { if mountPoint.Device != sysfsDevice { continue } // Check whether sysfs is 'rw' if len(mountPoint.Opts) > 0 && mountPoint.Opts[0] == permWritable { return true, nil } glog.Errorf("sysfs is not writable: %+v", mountPoint) break } return false, nil } </ins>


Question:
Currently a Docker bug may cause KubeProxy to crashloop thus making the Node unable to use cluster network, which in turn makes Pods scheduled on this Node unreachable. To mitigate this issue we need to surface the information about problems with KubeProxy, that will allow scheduler to ignore given node when scheduling new Pods. ProblemAPI is going to address this kind of problems in the future, but we need a fix for the 1.3. Either Kubelet or KubeProxy needs to update a NodeStatus with the information that Node networking is down. I suggests that we reuse NodeReady Condition for this, as this would make the rest of ControlPlane work out of the box. We will replace this fix with the ProblemAPI use in the future. Another option, which requires more work, is creating a completely new NodeCondition type to handle this case. This would require making Scheduler aware of this new Condition.
Does kubelet monitor kube-proxy health?
Kube-proxy today runs as a static pod (daemonset), kubelet treats it the same as other static pods. Yes, Kubelet reports kube-proxy podstatus back to master, that is all. The detail information on kube-proxy is in crashloop at
cc/ Brandon, this issue explains my long position against running kubelet, kube-proxy, etc. critical daemons as docker container. I am ok to packaging them into docker image, and running as linux containers, but they shouldn't depend on docker daemon.
I think we don't want to run them as docker containers as well, we want to run them as rkt containers.
The initial proposal I reviewed was docker container, and I did raise my concerns to and coreos developers then. Running it as rkt container should be ok since rkt is daemon-less today, and shouldn't have this chicken egg issue at the first place. But we still have the problem on the nodes without rkt as runtime. At the meeting, I suggested that kube-proxy update NodeStatus with a newly introduced NodeCondition, like what we plan to do with kernel issue at: The reason of not using NodeReady is that kubelet might override that. Another very hacky way is before crashing, Kube-proxy logs the read-only filesystem problem somewhere, like /var/lib/kube-proxy/... Kubelet will pick up the information, then update NodeReady Condition with detail error message prefixing with "kube-proxy:" kube-proxy's responsibility to make the information up-to-date.
Can I have more description of the bug? Is there a related docker issue? Is there a repro? Does this only affect kube-proxy or will it affect other pods on the node? We need to support running critical applications (including system daemons) in pods in order to allow easy deployment and integration with vendors (e.g. storage and network vendors). cc since we were just talking about integrating through addons. What features are we missing that we can't support this right now? Is it just that our default container runtime is buggy and flaky and unreliable? Do we need the NodeProblem API? Do we need taints/tolerations?
- I don't know about any way to repro it other than running huge clusters. Pretty much every run of a 1000 Node cluster has this problem on at least one Node.
If you want to know the detail, see It is a docker issue, and pretty rare.
Can we regroup on this issue this week? Like tomorrow? , Dawn Chen wrote:
I will happily make kube-proxy write to or offer up pretty much any API we deem necessary to detect this issue. To me it still feels like a docker health-check suite is needed, and one of the tests is that sysfs gets mounted rw. BUt in the mean time, how about this: make be the API. Any component which feels that it has a node problem can write a JSON file into that directory. For this case, kube-proxy would write a file: Kubelet can turn that into Node status reports. , Tim Hockin wrote:
cc
PR of introducing NodeProblemDetector was merged last week, and the pr to enable it by default for Kubernetes cluster was merged over the weekend. So far everything works fine. Even it is an alpha feature, GKE folks agreed to enable it for GKE. To workaround this particular issue before we have proper docker fix, we can easily extend today's KernelMonitor module of NodeProblemDetector to monitor kube-proxy log, and report a new NodeCondition called SysfsReadonly to the issue visible. But we don't have remedy system yet, and pr of converting node problem (events, conditions etc.) to taints is still under the debate / review: 1) Should NodeController respect this new Condition, and simply mark node NotReady? 2) Should repair system pickup this issue, and restart docker?
Do we still observe the problem with docker 1.11.X? If no, I want to close this; otherwise, we are going to reconfigure NodeProblemDetector to make the issue visible.
cc/
Agree, I'd love to not "fix" this..
I think we've seen it the day before yesterday during our tests - can you please confirm?
I can't recall - we certainly did saw some failed nodes during the test, but I'm not sure we made sure it was a kube-proxy issue.
But what I can tell for sure that it's definitely less frequent - we started 2000-node cluster 10+ times during this week and we've seen it at most once (which makes it at least 99.995% reliable).
Not enough data. GKE is still deploying the old ContainerVM and isn't running Docker 1.11.1 yet, unless I'm mistaken.
aah ok - I didn't know that...
I'm going to tentatively close this until we show that it's a problem on Docker 1.11.2, now that Docker 1.11.2 is in on both GCE and GKE.
(Obviously feel free to reopen if I'm wrong.)
Another occurrence:
So it's still happening, can we teach NPD to detect and surface this as node unschedulable?
Yes, we can. But I found that the error only occurred once with docker 1.11 till now. At least, this should be a known issue, right? XRef Do we want to fix this in 1.3 or just claim known issues in release notes? :)
Doesn't this render the node unusable (i.e no services will work because kube-proxy is broken)? if so, we should surface regardless of frequency IMO. Whether we fix or paper over it is a different issue.
+1
Because it's code freeze now, I can add a temporary short hack in node problem detector to detect this issue. Should we do that? :)
Removed the priority and it back to 1.3 milestone, so that we can discuss at burndown meeting.
Yeah i think codefreeze just means no features
It is very rare after we upgrade to docker 1.11.X. To extend KernelMontor in NodeProblemDetector to detect the issue is totally overkill, since we have to allocate more compute resource to NodeProblemDetector to process tons of kube-proxy logs. Here is what I suggested for this issue for 1.3 and 1.4+: Document this as known issue for docker releases: docker 1.10.X (), docker 1.11.X (). Also document it as Kubernetes 1.3 release known issue at is also updated admin guide for NodeProblemDetector, and the suggestion on how to handle this issue will be included there. For 1.4, I think we should change kube-proxy. Instead of logging the error then crashing, kube-proxy should report its problem to NodeProblemDector, which aggregates the node issues and propagate them to upstream layers, so that the admin / remedy system & control-plane can take proper action upon the issues. I am removing this from 1.3 milestone, and unassigning myself. cc/ to find the right owner from cluster team for next release here.
On a 2k node cluster, I just saw 4 nodes come up this way on Docker 1.11.2. I'm not saying it's going to stop the release, but it does stop bringup on large clusters pretty frequently. For some reason we went through a "nice" patch where it was rare, but today it was just bad.
We can have a quick fix in node problem detector to catch this kind of problem by parsing the kube-proxy log. The only problem is that, kube-proxy generates logs relatively fast. If we let node problem detector do that, it has to waste significant cpu even on the good nodes. A better solution is to make node problem detector expose an endpoint and let other system components report problems. However, we are not there yet for 1.3. If we'd still want to solve this for 1.3, we could: 1) Let kube-proxy have some kind of error log, and only print error into it. Then we can easily change node problem detector to parse the error log and report problems. By this, we can at least make these errors visible to users for 1.3. 2) Or let kube-proxy report the node condition itself. /cc
Comment at is still valid. We cannot do much for 1.3 release here, and extending NodeProblemDetector to parse the application logs is wrong move. Also NodeProblemDetector is not running on GKE nodes yet.
I'm seeing people say they reproduced on 1.11.2, is there a 1.11 docker issue? I know that was closed. If I'm going to get my docker guys to dig, I need a docker bug report...
Can you please ramp up on this and come up with a way to bring node and kube proxy teams together to resolve? Even if we don't land in 1.3.0, I propose this is at least a P1 for 1.3.1.
FYI and others In I a hacky way of resolving those issues to our test framework. In the last run of gke-large-cluster run it seemed to work (there was 1 broken kube-proxy, this triggered restarting docker, which in turn fixed the problem). This is obviously not the solution, but: it seems to prove that 'docker restart' solves the problem it should stop the bleeding in our tests
& I copied & paste what I wrote in another internal thread to give you some background: No, it is not a blocker for 1.3 release: It is not a regression from 1.2 release. It is way less frequently after we upgrade to docker 1.11.2 from docker 1.9.1 NodeProblemDetector is not designed for processing arbitrary logs. It is an overkill solution given such low frequency and relative high resource overhead. By default NodeProblemDetector is not running on gke node due to the extra overhead per node introduced to our customer. Meanwhile Since GKE already has repair system built, it should be very easy to extend GKE repair system to check if kube-proxy is in crashloop. If yes, then kubectl logs to retrieve the last terminated container log to see if the issue is caused by readonly root filesystem. If yes, the repair system can issue a docker restart.
agreed not a blocker. This is likely only fixable in docker. We should keep it up the priority so we dig into and fix docker. can try next week to get our docker people talking to whoever on your side you want to get this fixed (in 1.11.3 or later probably)
Can we add a node readiness or liveness probe to solve this? (I remember there is a issue for this, but can't find it now) One of the default behaviours of the probe is to make sure the key node components are running normally, and the behaviour could be extended overtime. Probably adding a node probe controller or node health controller to perform the probing. Or just use the health probe of GCE MIG, if its functionality is enough to handle this. This could be part of the remedy system. Just random thoughts, not for 1.3 and even may not work. :)
All are good ideas. But let's move that discussion to a separate issue, and use this one to focus providing the workaround for KubeProxy panic within 1.3 timeframe. Thanks!
Also what you suggested probe controller can be easily achieved by GKE repair system which I suggested above.
Agree. needs quite a lot of work, definitely not for 1.3 :)
Keeping in v1.3 until we decide on Docker 1.11 for sure.
The issue seems to be present on old and new docker alike, sadly.
I opened a docker issue here Feel free to supply information on that issue.
So what's the right kube-proxy fix? , Lantao Liu wrote:
- didn't we decided that we just want to discover this problem in NodeProblemDetector and then mark this node as not-healthy (whatever that means - adding a new condition or extending NodeReady or ...)? In my opinion if it would be clear to user why this node is not healthy, this should be good enough. FYI in I worked around this in our tests :)
It wasn't clear to me if kube-proxy should do something different or not. , Wojciech Tyczynski < :
yeah this is a docker bug (), unsure if there's a pod level workaround, couldn't make out if it was specific to hostNetwork. the work around is restarting docker. It happened more frequently in 1.9, but still happens on 1.11 (node team empirical data). To surface it to the NPD we need to either send some http event to a NPD server (vaporware) or write to a special hostPath termination log type file (parsing kube-proxy logs form NPD is too expensive).
This feels like a hot potato but I am not sure if it still needs to be addressed or not. If it does, let's do so. I apologize for not being able to understand what the current status is. It looks like we have a . Reading comment from 12 days ago she states: While folks may want the node problem detector extended to parse logs to detect it, that is not a good fit for that system and it doesn't run on all nodes anyway. Also GKE can be extended to possibly repair this problem. Questions: Is this an issue that we need to still address for production? Or can we wait for docker fix? If we haven't fixed the kube-proxy panic (which we should) how do ensure we still provide a signal that this is happening?
I am leaving this on 1.3 until we know the best thing to do.
If we can DIRECTLY tell the node problem system about a problem, I have no qualms about doing that. drop a file in a dir, for example. , Michael Rubin wrote:
Node problem detector is not running on GKE cluster now. :) and should be a better fix for this. If we still want a fix in the node problem detector, letting kubeproxy drop a file could be enough for now.
If you can give me a spec of what dir to mount and what sort of file to write, I'll do it. , Lantao Liu wrote:
For temporary fix, something like should be enough. The kernel monitor of node problem detector could be ported to parse other logs. The only reason we can't do that for kube- is that the log is too spammy. should be enough for now. Only the kubeproxy panic error is printed directly into : Even though the default  of glog is , I checked the node in and cluster, normally only few lines are at Error level.
I thought that Dawn said we DO NOT want to parse logs. Parsing logs is, in general, a total hack and very very brittle. we should have a real push API. , Lantao Liu wrote:
I agree with I also understood thought we should not parse logs. Is that common practise for the node problem detector? It feels fragile to me and to be avoided.
Yeah, I agree parsing log is fragile, that's why I call it a temporary or a quick fix. :) The only good thing is that it doesn't need significant change on both sides, although I don't think it's a good solution, either. :( For a better fix, it will need some more design and work. Let me think about it a little more~ Anyway, I still think and is needed. As a key component running on each node, it's wired that we don't have a component to monitor whether it is ready or not.
Re making kube-proxy "not panic", it currently fails in resizing the hash table for conntrack entries, for which it needs to write to /sys. I'm not sure if there are more locations (we moved /sys/module/br_netfilter loading into kubelet IIUC), or if conntrack itself needs write access to /sys. I think Dawn/Liu are more worried about the NPD not parsing kube-proxy logs, because its memory usage would balloon. If kube-proxy can detect the situtation internally, either by parsing its own logs (which is brittle) or poking someting in /sys for example (like the hash table resizing it's currently failing on), it can drop a token into a hostPath (hoping the docker bug doesn't manifest as ro hostPath). Kube-proxy will still remain dysfunctional, though we might be able to: the issue/mark node unusable a feedback loop that restarts docker We just need to be careful to cleanup the hostPath so it doesn't end up in a restart loop.
My understanding of 1.3's state based on a brief chat with : something like ~1 in 1000 nodes will be affected by this issue and if nodes are affected, pods scheduled on those nodes have {broken,little,no} networking. We hope Docker will eventually fix it. But because that's not the case for 1.3, and we don't want to make excessive changes to 1.3, perhaps that means that we should fix it expediently. Summarizing my understanding of previous discussion: expedient thing would be some mechanism to detect ( proposal like kube-proxy detecting and writing to /tmp/kube-proxy-claims-docker-issue-) and $something (kubelet, node problem detector, a bash script in a loop) that checks for existence of that file followed by some form of remediation {kill docker, sudo reboot, .....} to bash the node into working again. Maybe an ugly hack is okay for now and we hope that docker fixes it for the next release so in 1.4 we can rm the code? Or for 1.4 we can do something that is More Elegant And Less Offensive To Engineering Sensibilities?
Bounded-lifetime hacks are fine, but they need to come with giant disclaimers and named-assignees to cleanup the mess after a specific event. , Alex Mohr wrote:
Agree, re: bounded-lifetime hacks needing care and we shouldn't do them indiscriminately. We and our users have a problem today -- and we've had a problem for the past N months without traction. I'd like the problem solved for 1.3 because I'd like us to have a great product and I don't think the state of broken nodes is there. Aside; we've spent N hours discussing the issue and could likely have worked around the issue in less aggregate people time than we spent discussing? Apologies for the leaked frustration: this issue seems to be stuck in either some form of analysis paralysis or cracks between teams or perfect-is-the-enemy-of-the-good state. Again, I don't care about implementation details. We also need more than just tech or a piece of code. Exit criteria is that users don't get broken nodes shipped to them.
We're not going to solve this problem for as long as we depend on a lagging docker release cycle, and they don't backport or do intermediate releases to help us. This is just a one off docker bug that's left us in a bad state. Such issues come up every release, get documented in the release notes and we ship anyway. The easiest fix (disclaimers and all) is to redirect kube-proxy stderr and parse it out from NPD. Even then, the node will remain broken unless we restart docker. the NDP will only surface the error.
I'm working on a PR yesterday night, will send it out soon.
FYI, a fix is here . In , I let kube-proxy update a node condition with specific reason, message and hit to the administrator about the remediation. The specific reason why we want to do this is discussed in the PR description
FWIW I agree 100% with using node condition SGTM, but your PR should probably also modify the scheduler here to prevent the system from sending new pods to a node that has RuntimeUnhealthy condition. I realize there will still be a time window when we might send some pods to that node (before the problem is detected) but at least the time window will be bounded this way.
Yeah, we can do that. But before that, we should make sure that the node is really unusable without setting conntrack. In fact, we only setting conntrack to increase the max connection limit on the node (default 64k-256k) Without this, is the node really considered to be unusable? If so, we should prevent scheduling pods to the node. If not, maybe we should just surface the problem to the administrator and let the node keep working. I'll run e2e test without conntrack set to see whether there is any problem. Yeah, that's why I think we should not restart docker ourselves to try to remedy the problem. There may still be workloads running the node. We'd better leave this to the user to decide. :)
is merged, and hopefully, it could solve this issue. I've also sent a PR to revert the walkaround in the test framework to verify whether the issue is really fully solved. Feel free to reopen this if the issue or relative issue happens again.
thanks for the help!
I just ran into this, and I am able to repro consistently. It might be due to the craziness of my experiment, but I still wanted to contribute the data point. I am experimenting with docker inside docker. All I am about to describe is itself running inside a privileged CentOS 7 container that has systemd and Docker installed. The CentOS 7 container is running on my dev machine (Docker for Mac). I have a kubelet running fine, and all control plane components running as static pods. When I attempt to run kube-proxy as a static pod (privileged) it crashes with the following logs: sysfs is mounted as rw: sysfs is mounted as rw on containers: Using busybox to further inspect:
I'm seeing the same issue when running kubeadm-dind-cluster, where kube-proxy is failing to come up, because it is trying to write to /sys/module/nf_conntrack/parameters/hashsize. This is on the filesystem sysfs, which "mount" shows is rw (so it passes the R/W check in kube-proxy code), but the filesystem is not writeable. Within kube-proxy: I'm also running docker containers inside of docker containers. FYI, others have not seen this issue, when just running docker on bare-metal.

// Tune conntrack, if requested if s.Conntracker != nil { if s.Config.ConntrackMax > 0 { <del> if err := s.Conntracker.SetMax(int(s.Config.ConntrackMax)); err != nil { return err </del> <ins> err := s.Conntracker.SetMax(int(s.Config.ConntrackMax)) if err != nil { if err != readOnlySysFSError { return err } // readOnlySysFSError is caused by a known docker issue (https://github.com/docker/docker/issues/24000), // the only remediation we know is to restart the docker daemon. // Here we'll send an node event with specific reason and message, the // administrator should decide whether and how to handle this issue, // whether to drain the node and restart docker. // TODO(random-liu): Remove this when the docker bug is fixed. const message = "DOCKER RESTART NEEDED (docker issue #24000): /sys is read-only: can't raise conntrack limits, problems may arise later." s.Recorder.Eventf(s.Config.NodeRef, api.EventTypeWarning, err.Error(), message) </ins> } } if s.Config.ConntrackTCPEstablishedTimeout.Duration > 0 {


Question: Slow conformance test has caused issues like We should make runtime/image conformance test run faster without breaking the test coverage, possibly by: Shorten consistently check interval . Remove redundant test cases such as ... /cc

return ConformanceContainer{containers[0], cc.Client, pod.Spec.RestartPolicy, pod.Spec.Volumes, pod.Spec.NodeName, cc.Namespace, cc.podName}, nil } <del> func (cc *ConformanceContainer) GetStatus() (api.ContainerStatus, api.PodPhase, error) { </del> <ins> func (cc *ConformanceContainer) IsReady() (bool, error) { </ins> pod, err := cc.Client.Pods(cc.Namespace).Get(cc.podName) if err != nil { <del> return api.ContainerStatus{}, api.PodUnknown, err </del> <ins> return false, err </ins> } <ins> return api.IsPodReady(pod), nil } func (cc *ConformanceContainer) GetPhase() (api.PodPhase, error) { pod, err := cc.Client.Pods(cc.Namespace).Get(cc.podName) if err != nil { return api.PodUnknown, err } return pod.Status.Phase, nil } func (cc *ConformanceContainer) GetStatus() (api.ContainerStatus, error) { pod, err := cc.Client.Pods(cc.Namespace).Get(cc.podName) if err != nil { return api.ContainerStatus{}, err } </ins> statuses := pod.Status.ContainerStatuses <del> if len(statuses) != 1 { return api.ContainerStatus{}, api.PodUnknown, errors.New("Failed to get container status") </del> <ins> if len(statuses) != 1 || statuses[0].Name != cc.Container.Name { return api.ContainerStatus{}, fmt.Errorf("unexpected container statuses %v", statuses) </ins> } <del> return statuses[0], pod.Status.Phase, nil </del> <ins> return statuses[0], nil </ins> } func (cc *ConformanceContainer) Present() (bool, error) {


Question: Slow conformance test has caused issues like We should make runtime/image conformance test run faster without breaking the test coverage, possibly by: Shorten consistently check interval . Remove redundant test cases such as ... /cc

return false, err } <del> type ContainerState uint32 </del> <ins> type ContainerState int </ins> const ( <del> ContainerStateWaiting ContainerState = 1 << iota </del> <ins> ContainerStateWaiting ContainerState = iota </ins> ContainerStateRunning ContainerStateTerminated ContainerStateUnknown


Question: Slow conformance test has caused issues like We should make runtime/image conformance test run faster without breaking the test coverage, possibly by: Shorten consistently check interval . Remove redundant test cases such as ... /cc

pollInterval           = time.Second * 5 ) <del> type testStatus struct { </del> <ins> type testCase struct { </ins> Name             string RestartPolicy    api.RestartPolicy Phase            api.PodPhase


Question: Slow conformance test has caused issues like We should make runtime/image conformance test run faster without breaking the test coverage, possibly by: Shorten consistently check interval . Remove redundant test cases such as ... /cc

Ready            bool } <del> var _ = Describe("[FLAKY] Container runtime Conformance Test", func() { </del> <ins> var _ = Describe("Container Runtime Conformance Test", func() { </ins> var cl *client.Client BeforeEach(func() {


Question: Slow conformance test has caused issues like We should make runtime/image conformance test run faster without breaking the test coverage, possibly by: Shorten consistently check interval . Remove redundant test cases such as ... /cc

}) Describe("container runtime conformance blackbox test", func() { <del> var testCContainers []ConformanceContainer </del> namespace := "runtime-conformance" <del> BeforeEach(func() { testCContainers = []ConformanceContainer{} }) Context("when start a container that exits successfully", func() { </del> <ins> Context("when starting a container that exits", func() { </ins> It("it should run with the expected status [Conformance]", func() { <ins> restartCountVolumeName := "restart-count" restartCountVolumePath := "/restart-count" </ins> testContainer := api.Container{ Image: ImageRegistry[busyBoxImage], VolumeMounts: []api.VolumeMount{ { <del> MountPath: "/restart-count", Name:      "restart-count", </del> <ins> MountPath: restartCountVolumePath, Name:      restartCountVolumeName, </ins> }, }, <del> ImagePullPolicy: api.PullIfNotPresent, </del> } testVolumes := []api.Volume{ { <del> Name: "restart-count", </del> <ins> Name: restartCountVolumeName, </ins> VolumeSource: api.VolumeSource{ HostPath: &api.HostPathVolumeSource{ Path: os.TempDir(),


Question: Slow conformance test has caused issues like We should make runtime/image conformance test run faster without breaking the test coverage, possibly by: Shorten consistently check interval . Remove redundant test cases such as ... /cc

}, }, } <del> testCount := int32(3) testStatuses := []testStatus{ {"terminate-cmd-rpa", api.RestartPolicyAlways, api.PodRunning, ContainerStateWaiting | ContainerStateRunning | ContainerStateTerminated, ">", testCount, false}, {"terminate-cmd-rpof", api.RestartPolicyOnFailure, api.PodSucceeded, ContainerStateTerminated, "==", testCount, false}, {"terminate-cmd-rpn", api.RestartPolicyNever, api.PodSucceeded, ContainerStateTerminated, "==", 0, false}, </del> <ins> testCases := []testCase{ {"terminate-cmd-rpa", api.RestartPolicyAlways, api.PodRunning, ContainerStateRunning, "==", 2, true}, {"terminate-cmd-rpof", api.RestartPolicyOnFailure, api.PodSucceeded, ContainerStateTerminated, "==", 1, false}, {"terminate-cmd-rpn", api.RestartPolicyNever, api.PodFailed, ContainerStateTerminated, "==", 0, false}, </ins> } <del> for _, testStatus := range testStatuses { </del> <ins> for _, testCase := range testCases { </ins> tmpFile, err := ioutil.TempFile("", "restartCount") Expect(err).NotTo(HaveOccurred()) defer os.Remove(tmpFile.Name()) <del> // It fails in the first three runs and succeeds after that. tmpCmd := fmt.Sprintf("echo 'hello' >> /restart-count/%s ; test $(wc -l /restart-count/%s| awk {'print $1'}) -ge %d", path.Base(tmpFile.Name()), path.Base(tmpFile.Name()), testCount+1) testContainer.Name = testStatus.Name </del> <ins> // It failed at the 1st run, then succeeded at 2nd run, then run forever cmdScripts := ` f=%s count=$(echo 'hello' >> $f ; wc -l $f | awk {'print $1'}) if [ $count -eq 1 ]; then exit 1 fi if [ $count -eq 2 ]; then exit 0 fi while true; do sleep 1; done ` tmpCmd := fmt.Sprintf(cmdScripts, path.Join(restartCountVolumePath, path.Base(tmpFile.Name()))) testContainer.Name = testCase.Name </ins> testContainer.Command = []string{"sh", "-c", tmpCmd} terminateContainer := ConformanceContainer{ Container:     testContainer, Client:        cl, <del> RestartPolicy: testStatus.RestartPolicy, </del> <ins> RestartPolicy: testCase.RestartPolicy, </ins> Volumes:       testVolumes, NodeName:      *nodeName, Namespace:     namespace, } <del> err = terminateContainer.Create() Expect(err).NotTo(HaveOccurred()) testCContainers = append(testCContainers, terminateContainer) </del> <ins> Expect(terminateContainer.Create()).To(Succeed()) defer terminateContainer.Delete() </ins> <del> Eventually(func() api.PodPhase { _, phase, _ := terminateContainer.GetStatus() return phase }, retryTimeout, pollInterval).ShouldNot(Equal(api.PodPending)) var status api.ContainerStatus </del> By("it should get the expected 'RestartCount'") <del> Eventually(func() int32 { status, _, _ = terminateContainer.GetStatus() return status.RestartCount }, retryTimeout, pollInterval).Should(BeNumerically(testStatus.RestartCountOper, testStatus.RestartCount)) </del> <ins> Eventually(func() (int32, error) { status, err := terminateContainer.GetStatus() return status.RestartCount, err }, retryTimeout, pollInterval).Should(BeNumerically(testCase.RestartCountOper, testCase.RestartCount)) </ins> <del> By("it should get the expected 'Ready' status") Expect(status.Ready).To(Equal(testStatus.Ready)) </del> <ins> By("it should get the expected 'Phase'") Eventually(terminateContainer.GetPhase, retryTimeout, pollInterval).Should(Equal(testCase.Phase)) </ins> <del> By("it should get the expected 'State'") Expect(GetContainerState(status.State) & testStatus.State).NotTo(Equal(0)) </del> <ins> By("it should get the expected 'Ready' condition") Expect(terminateContainer.IsReady()).Should(Equal(testCase.Ready)) </ins> <del> By("it should be possible to delete [Conformance]") err = terminateContainer.Delete() Expect(err).NotTo(HaveOccurred()) Eventually(func() bool { isPresent, err := terminateContainer.Present() return err == nil && !isPresent }, retryTimeout, pollInterval).Should(BeTrue()) } }) }) Context("when start a container that keeps running", func() { It("it should run with the expected status [Conformance]", func() { testContainer := api.Container{ Image:           ImageRegistry[busyBoxImage], Command:         []string{"sh", "-c", "while true; do echo hello; sleep 1; done"}, ImagePullPolicy: api.PullIfNotPresent, } testStatuses := []testStatus{ {"loop-cmd-rpa", api.RestartPolicyAlways, api.PodRunning, ContainerStateRunning, "==", 0, true}, {"loop-cmd-rpof", api.RestartPolicyOnFailure, api.PodRunning, ContainerStateRunning, "==", 0, true}, {"loop-cmd-rpn", api.RestartPolicyNever, api.PodRunning, ContainerStateRunning, "==", 0, true}, } for _, testStatus := range testStatuses { testContainer.Name = testStatus.Name runningContainer := ConformanceContainer{ Container:     testContainer, Client:        cl, RestartPolicy: testStatus.RestartPolicy, NodeName:      *nodeName, Namespace:     namespace, } err := runningContainer.Create() Expect(err).NotTo(HaveOccurred()) testCContainers = append(testCContainers, runningContainer) Eventually(func() api.PodPhase { _, phase, _ := runningContainer.GetStatus() return phase }, retryTimeout, pollInterval).Should(Equal(api.PodRunning)) var status api.ContainerStatus var phase api.PodPhase Consistently(func() api.PodPhase { status, phase, err = runningContainer.GetStatus() return phase }, consistentCheckTimeout, pollInterval).Should(Equal(testStatus.Phase)) Expect(err).NotTo(HaveOccurred()) By("it should get the expected 'RestartCount'") Expect(status.RestartCount).To(BeNumerically(testStatus.RestartCountOper, testStatus.RestartCount)) By("it should get the expected 'Ready' status") Expect(status.Ready).To(Equal(testStatus.Ready)) </del> <ins> status, err := terminateContainer.GetStatus() Expect(err).ShouldNot(HaveOccurred()) </ins> By("it should get the expected 'State'") <del> Expect(GetContainerState(status.State) & testStatus.State).NotTo(Equal(0)) </del> <ins> Expect(GetContainerState(status.State)).To(Equal(testCase.State)) </ins> By("it should be possible to delete [Conformance]") <del> err = runningContainer.Delete() Expect(err).NotTo(HaveOccurred()) Eventually(func() bool { isPresent, err := runningContainer.Present() return err == nil && !isPresent }, retryTimeout, pollInterval).Should(BeTrue()) } }) }) Context("when start a container that exits failure", func() { It("it should run with the expected status [Conformance]", func() { testContainer := api.Container{ Image:           ImageRegistry[busyBoxImage], Command:         []string{"false"}, ImagePullPolicy: api.PullIfNotPresent, } testStatuses := []testStatus{ {"fail-cmd-rpa", api.RestartPolicyAlways, api.PodRunning, ContainerStateWaiting | ContainerStateRunning | ContainerStateTerminated, ">", 0, false}, {"fail-cmd-rpof", api.RestartPolicyOnFailure, api.PodRunning, ContainerStateTerminated, ">", 0, false}, {"fail-cmd-rpn", api.RestartPolicyNever, api.PodFailed, ContainerStateTerminated, "==", 0, false}, } for _, testStatus := range testStatuses { testContainer.Name = testStatus.Name failureContainer := ConformanceContainer{ Container:     testContainer, Client:        cl, RestartPolicy: testStatus.RestartPolicy, NodeName:      *nodeName, Namespace:     namespace, } err := failureContainer.Create() Expect(err).NotTo(HaveOccurred()) testCContainers = append(testCContainers, failureContainer) Eventually(func() api.PodPhase { _, phase, _ := failureContainer.GetStatus() return phase }, retryTimeout, pollInterval).ShouldNot(Equal(api.PodPending)) var status api.ContainerStatus By("it should get the expected 'RestartCount'") Eventually(func() int32 { status, _, _ = failureContainer.GetStatus() return status.RestartCount }, retryTimeout, pollInterval).Should(BeNumerically(testStatus.RestartCountOper, testStatus.RestartCount)) By("it should get the expected 'Ready' status") Expect(status.Ready).To(Equal(testStatus.Ready)) By("it should get the expected 'State'") Expect(GetContainerState(status.State) & testStatus.State).NotTo(Equal(0)) By("it should be possible to delete [Conformance]") err = failureContainer.Delete() Expect(err).NotTo(HaveOccurred()) Eventually(func() bool { isPresent, err := failureContainer.Present() return err == nil && !isPresent }, retryTimeout, pollInterval).Should(BeTrue()) </del> <ins> Expect(terminateContainer.Delete()).To(Succeed()) Eventually(terminateContainer.Present, retryTimeout, pollInterval).Should(BeFalse()) </ins> } }) })


Question: Slow conformance test has caused issues like We should make runtime/image conformance test run faster without breaking the test coverage, possibly by: Shorten consistently check interval . Remove redundant test cases such as ... /cc

Context("when running a container with invalid image", func() { It("it should run with the expected status [Conformance]", func() { testContainer := api.Container{ <del> Image:           "foo.com/foo/foo", Command:         []string{"false"}, ImagePullPolicy: api.PullIfNotPresent, </del> <ins> Image:   "foo.com/foo/foo", Command: []string{"false"}, </ins> } <del> testStatus := testStatus{"invalid-image-rpa", api.RestartPolicyAlways, api.PodPending, ContainerStateWaiting, "==", 0, false} testContainer.Name = testStatus.Name </del> <ins> testCase := testCase{"invalid-image-rpa", api.RestartPolicyAlways, api.PodPending, ContainerStateWaiting, "==", 0, false} testContainer.Name = testCase.Name </ins> invalidImageContainer := ConformanceContainer{ Container:     testContainer, Client:        cl, <del> RestartPolicy: testStatus.RestartPolicy, </del> <ins> RestartPolicy: testCase.RestartPolicy, </ins> NodeName:      *nodeName, Namespace:     namespace, } <del> err := invalidImageContainer.Create() Expect(err).NotTo(HaveOccurred()) testCContainers = append(testCContainers, invalidImageContainer) </del> <ins> Expect(invalidImageContainer.Create()).To(Succeed()) defer invalidImageContainer.Delete() </ins> <del> var status api.ContainerStatus var phase api.PodPhase </del> <ins> Eventually(invalidImageContainer.GetPhase, retryTimeout, pollInterval).Should(Equal(testCase.Phase)) Consistently(invalidImageContainer.GetPhase, consistentCheckTimeout, pollInterval).Should(Equal(testCase.Phase)) </ins> <del> Consistently(func() api.PodPhase { if status, phase, err = invalidImageContainer.GetStatus(); err != nil { return api.PodPending } else { return phase } }, consistentCheckTimeout, pollInterval).Should(Equal(testStatus.Phase)) </del> <ins> status, err := invalidImageContainer.GetStatus() </ins> Expect(err).NotTo(HaveOccurred()) By("it should get the expected 'RestartCount'") <del> Expect(status.RestartCount).To(BeNumerically(testStatus.RestartCountOper, testStatus.RestartCount)) </del> <ins> Expect(status.RestartCount).To(BeNumerically(testCase.RestartCountOper, testCase.RestartCount)) </ins> By("it should get the expected 'Ready' status") <del> Expect(status.Ready).To(Equal(testStatus.Ready)) </del> <ins> Expect(status.Ready).To(Equal(testCase.Ready)) </ins> By("it should get the expected 'State'") <del> Expect(GetContainerState(status.State) & testStatus.State).NotTo(Equal(0)) </del> <ins> Expect(GetContainerState(status.State)).To(Equal(testCase.State)) </ins> By("it should be possible to delete [Conformance]") <del> err = invalidImageContainer.Delete() Expect(err).NotTo(HaveOccurred()) Eventually(func() bool { isPresent, err := invalidImageContainer.Present() return err == nil && !isPresent }, retryTimeout, pollInterval).Should(BeTrue()) </del> <ins> Expect(invalidImageContainer.Delete()).To(Succeed()) Eventually(invalidImageContainer.Present, retryTimeout, pollInterval).Should(BeFalse()) </ins> }) }) <del> AfterEach(func() { for _, cc := range testCContainers { cc.Delete() } }) </del> }) })


Question:
Ref and should sort revisions correctly (10 should be after 9, not 1).
I can help with this bug. Looks like trivial fix in PrintRolloutHistory
Thanks
are you still taking care of this one ? I am searching for some easy to do PR to just get familiar with the process and put my first PR in.
I created pr some time ago to close this issue
great thanks :-)
you can look for help-wanted issues or check and
thanks

import ( "fmt" "io" <del> "sort" "strconv" </del> "k8s.io/kubernetes/pkg/api" "k8s.io/kubernetes/pkg/api/meta"


Question:
Ref and should sort revisions correctly (10 should be after 9, not 1).
I can help with this bug. Looks like trivial fix in PrintRolloutHistory
Thanks
are you still taking care of this one ? I am searching for some easy to do PR to just get familiar with the process and put my first PR in.
I created pr some time ago to close this issue
great thanks :-)
you can look for help-wanted issues or check and
thanks

clientset "k8s.io/kubernetes/pkg/client/clientset_generated/internalclientset" "k8s.io/kubernetes/pkg/runtime" deploymentutil "k8s.io/kubernetes/pkg/util/deployment" <del> "k8s.io/kubernetes/pkg/util/errors" </del> <ins> sliceutil "k8s.io/kubernetes/pkg/util/slice" </ins> ) const (


Question:
Ref and should sort revisions correctly (10 should be after 9, not 1).
I can help with this bug. Looks like trivial fix in PrintRolloutHistory
Thanks
are you still taking care of this one ? I am searching for some easy to do PR to just get familiar with the process and put my first PR in.
I created pr some time ago to close this issue
great thanks :-)
you can look for help-wanted issues or check and
thanks

return fmt.Sprintf("No rollout history found in %s %q", resource, name), nil } // Sort the revisionToChangeCause map by revision <del> var revisions []string for k := range historyInfo.RevisionToTemplate { revisions = append(revisions, strconv.FormatInt(k, 10)) </del> <ins> var revisions []int64 for r := range historyInfo.RevisionToTemplate { revisions = append(revisions, r) </ins> } <del> sort.Strings(revisions) </del> <ins> sliceutil.SortInts64(revisions) </ins> return tabbedString(func(out io.Writer) error { fmt.Fprintf(out, "%s %q:n", resource, name) fmt.Fprintf(out, "REVISIONtCHANGE-CAUSEn") <del> errs := []error{} </del> for _, r := range revisions { // Find the change-cause of revision r <del> r64, err := strconv.ParseInt(r, 10, 64) if err != nil { errs = append(errs, err) continue } changeCause := historyInfo.RevisionToTemplate[r64].Annotations[ChangeCauseAnnotation] </del> <ins> changeCause := historyInfo.RevisionToTemplate[r].Annotations[ChangeCauseAnnotation] </ins> if len(changeCause) == 0 { changeCause = "<none>" } <del> fmt.Fprintf(out, "%st%sn", r, changeCause) </del> <ins> fmt.Fprintf(out, "%dt%sn", r, changeCause) </ins> } <del> return errors.NewAggregate(errs) </del> <ins> return nil </ins> }) }


Question:
Ref and should sort revisions correctly (10 should be after 9, not 1).
I can help with this bug. Looks like trivial fix in PrintRolloutHistory
Thanks
are you still taking care of this one ? I am searching for some easy to do PR to just get familiar with the process and put my first PR in.
I created pr some time ago to close this issue
great thanks :-)
you can look for help-wanted issues or check and
thanks

} return shuffled } <ins> // Int64Slice attaches the methods of Interface to []int64, // sorting in increasing order. type Int64Slice []int64 func (p Int64Slice) Len() int           { return len(p) } func (p Int64Slice) Less(i, j int) bool { return p[i] < p[j] } func (p Int64Slice) Swap(i, j int)      { p[i], p[j] = p[j], p[i] } // Sorts []int64 in increasing order func SortInts64(a []int64) { sort.Sort(Int64Slice(a)) } </ins>


Question:
Ref and should sort revisions correctly (10 should be after 9, not 1).
I can help with this bug. Looks like trivial fix in PrintRolloutHistory
Thanks
are you still taking care of this one ? I am searching for some easy to do PR to just get familiar with the process and put my first PR in.
I created pr some time ago to close this issue
great thanks :-)
you can look for help-wanted issues or check and
thanks

} } } <ins> func TestSortInts64(t *testing.T) { src := []int64{10, 1, 2, 3, 4, 5, 6} expected := []int64{1, 2, 3, 4, 5, 6, 10} SortInts64(src) if !reflect.DeepEqual(src, expected) { t.Errorf("func Ints64 didnt sort correctly, %v !- %v", src, expected) } } </ins>


Question: vSphere volume spec fails with the following error, as its not been in api/validation. "spec.volumes[0]: Required value: must specify a volume type" Fixed in PR

numVolumes++ allErrs = append(allErrs, validateAzureFile(source.AzureFile, fldPath.Child("azureFile"))...) } <ins> if source.VsphereVolume != nil { if numVolumes > 0 { allErrs = append(allErrs, field.Forbidden(fldPath.Child("vsphereVolume"), "may not specify more than 1 volume type")) } else { numVolumes++ allErrs = append(allErrs, validateVsphereVolumeSource(source.VsphereVolume, fldPath.Child("vsphereVolume"))...) } } </ins> if numVolumes == 0 { allErrs = append(allErrs, field.Required(fldPath, "must specify a volume type")) }


Question: vSphere volume spec fails with the following error, as its not been in api/validation. "spec.volumes[0]: Required value: must specify a volume type" Fixed in PR

return allErrs } <ins> func validateVsphereVolumeSource(cd *api.VsphereVirtualDiskVolumeSource, fldPath *field.Path) field.ErrorList { allErrs := field.ErrorList{} if len(cd.VolumePath) == 0 { allErrs = append(allErrs, field.Required(fldPath.Child("volumePath"), "")) } return allErrs } </ins> // ValidatePersistentVolumeName checks that a name is appropriate for a // PersistentVolumeName object. var ValidatePersistentVolumeName = NameIsDNSSubdomain


Question: vSphere volume spec fails with the following error, as its not been in api/validation. "spec.volumes[0]: Required value: must specify a volume type" Fixed in PR

numVolumes++ allErrs = append(allErrs, validateAzureFile(pv.Spec.AzureFile, specPath.Child("azureFile"))...) } <ins> if pv.Spec.VsphereVolume != nil { if numVolumes > 0 { allErrs = append(allErrs, field.Forbidden(specPath.Child("vsphereVolume"), "may not specify more than 1 volume type")) } else { numVolumes++ allErrs = append(allErrs, validateVsphereVolumeSource(pv.Spec.VsphereVolume, specPath.Child("vsphereVolume"))...) } } </ins> if numVolumes == 0 { allErrs = append(allErrs, field.Required(specPath, "must specify a volume type")) }


Question:
PersistentVolume integration tests fail sometimes: There are some interesting reports in the log like: Thans for report.
It turns out that volume integration tests have no logs at all, was not checked and I don't really know what's going on there. adds error checks to help me find out what's wrong there. Still, I suspect there are some troubles with etcd, not related to the test itself.
This flake affects also TestPersistentVolumeMultiPVs:
Extended logging got merged just now, please post links to logs when it breaks again!
Marking as P1 as per
I have no real usable logs, but I think it could be fixed by , it hasn't flaked since it was merged. And I have (in merge queue) to further stabilize the tests.
Not blocking this on 1.3.
It hasn't flaked for almost a month, can we close this?
Closing this issue since previous fixes made the flake go away.

pvc := createPVC("fake-pvc", "5G", []api.PersistentVolumeAccessMode{api.ReadWriteOnce}) <del> w, _ := testClient.PersistentVolumes().Watch(api.ListOptions{}) </del> <ins> w, err := testClient.PersistentVolumes().Watch(api.ListOptions{}) if err != nil { t.Errorf("Failed to watch PersistentVolumes: %v", err) } </ins> defer w.Stop() <del> _, _ = testClient.PersistentVolumes().Create(pv) _, _ = testClient.PersistentVolumeClaims(api.NamespaceDefault).Create(pvc) </del> <ins> _, err = testClient.PersistentVolumes().Create(pv) if err != nil { t.Errorf("Failed to create PersistentVolume: %v", err) } _, err = testClient.PersistentVolumeClaims(api.NamespaceDefault).Create(pvc) if err != nil { t.Errorf("Failed to create PersistentVolumeClaim: %v", err) } </ins> // wait until the controller pairs the volume and claim waitForPersistentVolumePhase(w, api.VolumeBound)


Question:
PersistentVolume integration tests fail sometimes: There are some interesting reports in the log like: Thans for report.
It turns out that volume integration tests have no logs at all, was not checked and I don't really know what's going on there. adds error checks to help me find out what's wrong there. Still, I suspect there are some troubles with etcd, not related to the test itself.
This flake affects also TestPersistentVolumeMultiPVs:
Extended logging got merged just now, please post links to logs when it breaks again!
Marking as P1 as per
I have no real usable logs, but I think it could be fixed by , it hasn't flaked since it was merged. And I have (in merge queue) to further stabilize the tests.
Not blocking this on 1.3.
It hasn't flaked for almost a month, can we close this?
Closing this issue since previous fixes made the flake go away.

// change the reclamation policy of the PV for the next test pv.Spec.PersistentVolumeReclaimPolicy = api.PersistentVolumeReclaimDelete <del> w, _ = testClient.PersistentVolumes().Watch(api.ListOptions{}) </del> <ins> w, err = testClient.PersistentVolumes().Watch(api.ListOptions{}) if err != nil { t.Errorf("Failed to watch PersistentVolumes: %v", err) } </ins> defer w.Stop() <del> _, _ = testClient.PersistentVolumes().Create(pv) _, _ = testClient.PersistentVolumeClaims(api.NamespaceDefault).Create(pvc) </del> <ins> _, err = testClient.PersistentVolumes().Create(pv) if err != nil { t.Errorf("Failed to create PersistentVolume: %v", err) } _, err = testClient.PersistentVolumeClaims(api.NamespaceDefault).Create(pvc) if err != nil { t.Errorf("Failed to create PersistentVolumeClaim: %v", err) } </ins> waitForPersistentVolumePhase(w, api.VolumeBound)


Question:
PersistentVolume integration tests fail sometimes: There are some interesting reports in the log like: Thans for report.
It turns out that volume integration tests have no logs at all, was not checked and I don't really know what's going on there. adds error checks to help me find out what's wrong there. Still, I suspect there are some troubles with etcd, not related to the test itself.
This flake affects also TestPersistentVolumeMultiPVs:
Extended logging got merged just now, please post links to logs when it breaks again!
Marking as P1 as per
I have no real usable logs, but I think it could be fixed by , it hasn't flaked since it was merged. And I have (in merge queue) to further stabilize the tests.
Not blocking this on 1.3.
It hasn't flaked for almost a month, can we close this?
Closing this issue since previous fixes made the flake go away.

pvc := createPVC("pvc-2", strconv.Itoa(maxPVs/2)+"G", []api.PersistentVolumeAccessMode{api.ReadWriteOnce}) <del> w, _ := testClient.PersistentVolumes().Watch(api.ListOptions{}) </del> <ins> w, err := testClient.PersistentVolumes().Watch(api.ListOptions{}) if err != nil { t.Errorf("Failed to watch PersistentVolumes: %v", err) } </ins> defer w.Stop() for i := 0; i < maxPVs; i++ { <del> _, _ = testClient.PersistentVolumes().Create(pvs[i]) </del> <ins> _, err = testClient.PersistentVolumes().Create(pvs[i]) if err != nil { t.Errorf("Failed to create PersistentVolume %d: %v", i, err) } </ins> } <del> _, _ = testClient.PersistentVolumeClaims(api.NamespaceDefault).Create(pvc) </del> <ins> _, err = testClient.PersistentVolumeClaims(api.NamespaceDefault).Create(pvc) if err != nil { t.Errorf("Failed to create PersistentVolumeClaim: %v", err) } </ins> // wait until the controller pairs the volume and claim waitForPersistentVolumePhase(w, api.VolumeBound)


Question:
PersistentVolume integration tests fail sometimes: There are some interesting reports in the log like: Thans for report.
It turns out that volume integration tests have no logs at all, was not checked and I don't really know what's going on there. adds error checks to help me find out what's wrong there. Still, I suspect there are some troubles with etcd, not related to the test itself.
This flake affects also TestPersistentVolumeMultiPVs:
Extended logging got merged just now, please post links to logs when it breaks again!
Marking as P1 as per
I have no real usable logs, but I think it could be fixed by , it hasn't flaked since it was merged. And I have (in merge queue) to further stabilize the tests.
Not blocking this on 1.3.
It hasn't flaked for almost a month, can we close this?
Closing this issue since previous fixes made the flake go away.

pvc := createPVC("pvc-rwm", "5G", []api.PersistentVolumeAccessMode{api.ReadWriteMany}) <del> w, _ := testClient.PersistentVolumes().Watch(api.ListOptions{}) </del> <ins> w, err := testClient.PersistentVolumes().Watch(api.ListOptions{}) if err != nil { t.Errorf("Failed to watch PersistentVolumes: %v", err) } </ins> defer w.Stop() <del> _, _ = testClient.PersistentVolumes().Create(pv_rwm) _, _ = testClient.PersistentVolumes().Create(pv_rwo) </del> <ins> _, err = testClient.PersistentVolumes().Create(pv_rwm) if err != nil { t.Errorf("Failed to create PersistentVolume: %v", err) } _, err = testClient.PersistentVolumes().Create(pv_rwo) if err != nil { t.Errorf("Failed to create PersistentVolume: %v", err) } </ins> <del> _, _ = testClient.PersistentVolumeClaims(api.NamespaceDefault).Create(pvc) </del> <ins> _, err = testClient.PersistentVolumeClaims(api.NamespaceDefault).Create(pvc) if err != nil { t.Errorf("Failed to create PersistentVolumeClaim: %v", err) } </ins> // wait until the controller pairs the volume and claim waitForPersistentVolumePhase(w, api.VolumeBound)


Question:
removed this test. We need to continue testing this. This wasn't about testing upgrades, the prefix has nothing to do with upgrades. It's just there so people can share etcd. We maybe don't need to double everything but we do need to make sure that keeps working.
To be honest - I think that what we was testing before didn't give us completely any information. We didn't testing what information we are storing in etcd and where. That said, if we had a hardcoded prefix somewhere in our code, we wouldn't discover it at all, and those tests would still be passing. We should have a test that is directly checking etcd contents. I think it should be possible to write a simple unit test for it. Will try to put something together next week.

ETCD_HOST=${ETCD_HOST:-127.0.0.1} ETCD_PORT=${ETCD_PORT:-4001} <ins> ETCD_PREFIX=${ETCD_PREFIX:-randomPrefix} </ins> API_PORT=${API_PORT:-8080} API_HOST=${API_HOST:-127.0.0.1} KUBE_API_VERSIONS=""


Question:
removed this test. We need to continue testing this. This wasn't about testing upgrades, the prefix has nothing to do with upgrades. It's just there so people can share etcd. We maybe don't need to double everything but we do need to make sure that keeps working.
To be honest - I think that what we was testing before didn't give us completely any information. We didn't testing what information we are storing in etcd and where. That said, if we had a hardcoded prefix somewhere in our code, we wouldn't discover it at all, and those tests would still be passing. We should have a test that is directly checking etcd contents. I think it should be possible to write a simple unit test for it. Will try to put something together next week.

--bind-address="${API_HOST}"  --insecure-port="${API_PORT}"  --etcd-servers="http://${ETCD_HOST}:${ETCD_PORT}"  <ins> --etcd-prefix="${ETCD_PREFIX}"  </ins> --runtime-config="${RUNTIME_CONFIG}"  --cert-dir="${TMPDIR:-/tmp/}"  --service-cluster-ip-range="10.0.0.0/24"


Question:
removed this test. We need to continue testing this. This wasn't about testing upgrades, the prefix has nothing to do with upgrades. It's just there so people can share etcd. We maybe don't need to double everything but we do need to make sure that keeps working.
To be honest - I think that what we was testing before didn't give us completely any information. We didn't testing what information we are storing in etcd and where. That said, if we had a hardcoded prefix somewhere in our code, we wouldn't discover it at all, and those tests would still be passing. We should have a test that is directly checking etcd contents. I think it should be possible to write a simple unit test for it. Will try to put something together next week.

### END TEST DEFINITION CUSTOMIZATION ### ####################################################### # Step 1: Start a server which supports both the old and new api versions, # but KUBE_OLD_API_VERSION is the latest (storage) version.


Question:
removed this test. We need to continue testing this. This wasn't about testing upgrades, the prefix has nothing to do with upgrades. It's just there so people can share etcd. We maybe don't need to double everything but we do need to make sure that keeps working.
To be honest - I think that what we was testing before didn't give us completely any information. We didn't testing what information we are storing in etcd and where. That said, if we had a hardcoded prefix somewhere in our code, we wouldn't discover it at all, and those tests would still be passing. We should have a test that is directly checking etcd contents. I think it should be possible to write a simple unit test for it. Will try to put something together next week.

old_storage_version=${test_data[4]} kube::log::status "Verifying ${resource}/${namespace}/${name} has storage version ${old_storage_version} in etcd" <del> curl -s http://${ETCD_HOST}:${ETCD_PORT}/v2/keys/registry/${resource}/${namespace}/${name} | grep ${old_storage_version} </del> <ins> curl -s http://${ETCD_HOST}:${ETCD_PORT}/v2/keys/${ETCD_PREFIX}/${resource}/${namespace}/${name} | grep ${old_storage_version} </ins> done killApiServer


Question:
removed this test. We need to continue testing this. This wasn't about testing upgrades, the prefix has nothing to do with upgrades. It's just there so people can share etcd. We maybe don't need to double everything but we do need to make sure that keeps working.
To be honest - I think that what we was testing before didn't give us completely any information. We didn't testing what information we are storing in etcd and where. That said, if we had a hardcoded prefix somewhere in our code, we wouldn't discover it at all, and those tests would still be passing. We should have a test that is directly checking etcd contents. I think it should be possible to write a simple unit test for it. Will try to put something together next week.

new_storage_version=${test_data[5]} kube::log::status "Verifying ${resource}/${namespace}/${name} has updated storage version ${new_storage_version} in etcd" <del> curl -s http://${ETCD_HOST}:${ETCD_PORT}/v2/keys/registry/${resource}/${namespace}/${name} | grep ${new_storage_version} </del> <ins> curl -s http://${ETCD_HOST}:${ETCD_PORT}/v2/keys/${ETCD_PREFIX}/${resource}/${namespace}/${name} | grep ${new_storage_version} </ins> done killApiServer


Question: If the first scheduling failed (whatever the reason is), the Error() func is called. However, the default Error function is wrong and in case of some errors may drop the pod and never requeue it for scheduling again. The problem is that if the Get() function: return an error (for me it was "dial tcp 127.0.0.1:8080: connect: cannot assign requested address", but it doesn't really matter), then we simply forget about a pod and will never schedule it again. We should retry those requests (with some backoff).

const ( SchedulerAnnotationKey = "scheduler.alpha.kubernetes.io/name" <ins> initialGetBackoff      = 100 * time.Millisecond maximalGetBackoff      = time.Minute </ins> ) // ConfigFactory knows how to fill out a scheduler config with its support functions.


Question: If the first scheduling failed (whatever the reason is), the Error() func is called. However, the default Error function is wrong and in case of some errors may drop the pod and never requeue it for scheduling again. The problem is that if the Get() function: return an error (for me it was "dial tcp 127.0.0.1:8080: connect: cannot assign requested address", but it doesn't really matter), then we simply forget about a pod and will never schedule it again. We should retry those requests (with some backoff).

} // Get the pod again; it may have changed/been scheduled already. pod = &api.Pod{} <del> err := factory.Client.Get().Namespace(podID.Namespace).Resource("pods").Name(podID.Name).Do().Into(pod) if err != nil { if !errors.IsNotFound(err) { glog.Errorf("Error getting pod %v for retry: %v; abandoning", podID, err) </del> <ins> getBackoff := initialGetBackoff for { if err := factory.Client.Get().Namespace(podID.Namespace).Resource("pods").Name(podID.Name).Do().Into(pod); err == nil { break </ins> } <del> return </del> <ins> if errors.IsNotFound(err) { glog.Warning("A pod %v no longer exists", podID) return } glog.Errorf("Error getting pod %v for retry: %v; retrying...", podID, err) if getBackoff = getBackoff * 2; getBackoff > maximalGetBackoff { getBackoff = maximalGetBackoff } time.Sleep(getBackoff) </ins> } if pod.Spec.NodeName == "" { podQueue.AddIfNotPresent(pod)


Question:
For example
This is with recent kubectl:
I think this is a duplicate, but just in case.
I will double check this.
Thanks - it isn't a big deal, because I obviously didn't intend to rolling update to an empty manifest :-) (I was experimenting with templating my manifests and made an error) But let me know if you need any more info to reproduce!

if len(list.Items) > 1 { return cmdutil.UsageError(cmd, "%s specifies multiple items", filename) } <ins> if len(list.Items) == 0 { return cmdutil.UsageError(cmd, "please make sure %s exists and is not empty", filename) } </ins> obj = list.Items[0] } newRc, ok = obj.(*api.ReplicationController)


Question:
Observed weird memory spikes for both the dns healthz and kube-dns pods in production. Kube-dns didn't oom because there's enough headroom, but healthz did. I can't explain the spikes yet. I tried a bunch of different repros: no , bad nslookup command, sharing the node with cpu hogs etc, but was unable to trigger it. The memory grows, but stays under 10Mi. As a short term mitigation, we stole some ram from kube-dns, because it was always comfortably within limits. Related b/
can be closed -- healthz has been removed
-- close issue?

apiVersion: v1 kind: ReplicationController metadata: <del> name: kube-dns-v18 </del> <ins> name: kube-dns-v19 </ins> namespace: kube-system labels: k8s-app: kube-dns <del> version: v18 </del> <ins> version: v19 </ins> kubernetes.io/cluster-service: "true" spec: replicas: __PILLAR__DNS__REPLICAS__ selector: k8s-app: kube-dns <del> version: v18 </del> <ins> version: v19 </ins> template: metadata: labels: k8s-app: kube-dns <del> version: v18 </del> <ins> version: v19 </ins> kubernetes.io/cluster-service: "true" spec: containers:


Question:
Observed weird memory spikes for both the dns healthz and kube-dns pods in production. Kube-dns didn't oom because there's enough headroom, but healthz did. I can't explain the spikes yet. I tried a bunch of different repros: no , bad nslookup command, sharing the node with cpu hogs etc, but was unable to trigger it. The memory grows, but stays under 10Mi. As a short term mitigation, we stole some ram from kube-dns, because it was always comfortably within limits. Related b/
can be closed -- healthz has been removed
-- close issue?

# keep request = limit to keep this container in guaranteed class limits: cpu: 10m <del> memory: 20Mi </del> <ins> memory: 50Mi </ins> requests: cpu: 10m <del> memory: 20Mi </del> <ins> # Note that this container shouldn't really need 50Mi of memory. The # limits are set higher than expected pending investigation on #29688. # The extra memory was stolen from the kubedns container to keep the # net memory requested by the pod constant. memory: 50Mi </ins> args: - -cmd=nslookup kubernetes.default.svc.__PILLAR__DNS__DOMAIN__ 127.0.0.1 >/dev/null && nslookup kubernetes.default.svc.__PILLAR__DNS__DOMAIN__ 127.0.0.1:10053 >/dev/null - -port=8080


Question:
Observed weird memory spikes for both the dns healthz and kube-dns pods in production. Kube-dns didn't oom because there's enough headroom, but healthz did. I can't explain the spikes yet. I tried a bunch of different repros: no , bad nslookup command, sharing the node with cpu hogs etc, but was unable to trigger it. The memory grows, but stays under 10Mi. As a short term mitigation, we stole some ram from kube-dns, because it was always comfortably within limits. Related b/
can be closed -- healthz has been removed
-- close issue?

apiVersion: v1 kind: ReplicationController metadata: <del> name: kube-dns-v18 </del> <ins> name: kube-dns-v19 </ins> namespace: kube-system labels: k8s-app: kube-dns <del> version: v18 </del> <ins> version: v19 </ins> kubernetes.io/cluster-service: "true" spec: replicas: {{ pillar['dns_replicas'] }} selector: k8s-app: kube-dns <del> version: v18 </del> <ins> version: v19 </ins> template: metadata: labels: k8s-app: kube-dns <del> version: v18 </del> <ins> version: v19 </ins> kubernetes.io/cluster-service: "true" spec: containers:


Question:
Observed weird memory spikes for both the dns healthz and kube-dns pods in production. Kube-dns didn't oom because there's enough headroom, but healthz did. I can't explain the spikes yet. I tried a bunch of different repros: no , bad nslookup command, sharing the node with cpu hogs etc, but was unable to trigger it. The memory grows, but stays under 10Mi. As a short term mitigation, we stole some ram from kube-dns, because it was always comfortably within limits. Related b/
can be closed -- healthz has been removed
-- close issue?

# keep request = limit to keep this container in guaranteed class limits: cpu: 10m <del> memory: 20Mi </del> <ins> memory: 50Mi </ins> requests: cpu: 10m <del> memory: 20Mi </del> <ins> # Note that this container shouldn't really need 50Mi of memory. The # limits are set higher than expected pending investigation on #29688. # The extra memory was stolen from the kubedns container to keep the # net memory requested by the pod constant. memory: 50Mi </ins> args: - -cmd=nslookup kubernetes.default.svc.{{ pillar['dns_domain'] }} 127.0.0.1 >/dev/null && nslookup kubernetes.default.svc.{{ pillar['dns_domain'] }} 127.0.0.1:10053 >/dev/null - -port=8080


Question:
Observed weird memory spikes for both the dns healthz and kube-dns pods in production. Kube-dns didn't oom because there's enough headroom, but healthz did. I can't explain the spikes yet. I tried a bunch of different repros: no , bad nslookup command, sharing the node with cpu hogs etc, but was unable to trigger it. The memory grows, but stays under 10Mi. As a short term mitigation, we stole some ram from kube-dns, because it was always comfortably within limits. Related b/
can be closed -- healthz has been removed
-- close issue?

apiVersion: v1 kind: ReplicationController metadata: <del> name: kube-dns-v18 </del> <ins> name: kube-dns-v19 </ins> namespace: kube-system labels: k8s-app: kube-dns <del> version: v18 </del> <ins> version: v19 </ins> kubernetes.io/cluster-service: "true" spec: replicas: $DNS_REPLICAS selector: k8s-app: kube-dns <del> version: v18 </del> <ins> version: v19 </ins> template: metadata: labels: k8s-app: kube-dns <del> version: v18 </del> <ins> version: v19 </ins> kubernetes.io/cluster-service: "true" spec: containers:


Question:
Observed weird memory spikes for both the dns healthz and kube-dns pods in production. Kube-dns didn't oom because there's enough headroom, but healthz did. I can't explain the spikes yet. I tried a bunch of different repros: no , bad nslookup command, sharing the node with cpu hogs etc, but was unable to trigger it. The memory grows, but stays under 10Mi. As a short term mitigation, we stole some ram from kube-dns, because it was always comfortably within limits. Related b/
can be closed -- healthz has been removed
-- close issue?

# "burstable" category so the kubelet doesn't backoff from restarting it. limits: cpu: 100m <del> memory: 200Mi </del> <ins> memory: 170Mi </ins> requests: cpu: 100m <del> memory: 100Mi </del> <ins> memory: 70Mi </ins> livenessProbe: httpGet: path: /healthz


Question:
Observed weird memory spikes for both the dns healthz and kube-dns pods in production. Kube-dns didn't oom because there's enough headroom, but healthz did. I can't explain the spikes yet. I tried a bunch of different repros: no , bad nslookup command, sharing the node with cpu hogs etc, but was unable to trigger it. The memory grows, but stays under 10Mi. As a short term mitigation, we stole some ram from kube-dns, because it was always comfortably within limits. Related b/
can be closed -- healthz has been removed
-- close issue?

# keep request = limit to keep this container in guaranteed class limits: cpu: 10m <del> memory: 20Mi </del> <ins> memory: 50Mi </ins> requests: cpu: 10m <del> memory: 20Mi </del> <ins> # Note that this container shouldn't really need 50Mi of memory. The # limits are set higher than expected pending investigation on #29688. # The extra memory was stolen from the kubedns container to keep the # net memory requested by the pod constant. memory: 50Mi </ins> args: - -cmd=nslookup kubernetes.default.svc.$DNS_DOMAIN 127.0.0.1 >/dev/null && nslookup kubernetes.default.svc.$DNS_DOMAIN 127.0.0.1:10053 >/dev/null - -port=8080


Question:
Observed weird memory spikes for both the dns healthz and kube-dns pods in production. Kube-dns didn't oom because there's enough headroom, but healthz did. I can't explain the spikes yet. I tried a bunch of different repros: no , bad nslookup command, sharing the node with cpu hogs etc, but was unable to trigger it. The memory grows, but stays under 10Mi. As a short term mitigation, we stole some ram from kube-dns, because it was always comfortably within limits. Related b/
can be closed -- healthz has been removed
-- close issue?

apiVersion: v1 kind: ReplicationController metadata: <del> name: kube-dns-v17 </del> <ins> name: kube-dns-v17.1 </ins> namespace: kube-system labels: k8s-app: kube-dns <del> version: v17 </del> <ins> version: v17.1 </ins> kubernetes.io/cluster-service: "true" spec: replicas: __PILLAR__DNS__REPLICAS__ selector: k8s-app: kube-dns <del> version: v17 </del> <ins> version: v17.1 </ins> template: metadata: labels: k8s-app: kube-dns <del> version: v17 </del> <ins> version: v17.1 </ins> kubernetes.io/cluster-service: "true" spec: containers:


Question:
Observed weird memory spikes for both the dns healthz and kube-dns pods in production. Kube-dns didn't oom because there's enough headroom, but healthz did. I can't explain the spikes yet. I tried a bunch of different repros: no , bad nslookup command, sharing the node with cpu hogs etc, but was unable to trigger it. The memory grows, but stays under 10Mi. As a short term mitigation, we stole some ram from kube-dns, because it was always comfortably within limits. Related b/
can be closed -- healthz has been removed
-- close issue?

# keep request = limit to keep this container in guaranteed class limits: cpu: 10m <del> memory: 20Mi </del> <ins> memory: 50Mi </ins> requests: cpu: 10m <del> memory: 20Mi </del> <ins> # Note that this container shouldn't really need 50Mi of memory. The # limits are set higher than expected pending investigation on #29688. # The extra memory was stolen from the kubedns container to keep the # net memory requested by the pod constant. memory: 50Mi </ins> args: - -cmd=nslookup kubernetes.default.svc.__PILLAR__DNS__DOMAIN__ 127.0.0.1 >/dev/null - -port=8080


Question:
Observed weird memory spikes for both the dns healthz and kube-dns pods in production. Kube-dns didn't oom because there's enough headroom, but healthz did. I can't explain the spikes yet. I tried a bunch of different repros: no , bad nslookup command, sharing the node with cpu hogs etc, but was unable to trigger it. The memory grows, but stays under 10Mi. As a short term mitigation, we stole some ram from kube-dns, because it was always comfortably within limits. Related b/
can be closed -- healthz has been removed
-- close issue?

apiVersion: v1 kind: ReplicationController metadata: <del> name: kube-dns-v17 </del> <ins> name: kube-dns-v17.1 </ins> namespace: kube-system labels: k8s-app: kube-dns <del> version: v17 </del> <ins> version: v17.1 </ins> kubernetes.io/cluster-service: "true" spec: replicas: {{ pillar['dns_replicas'] }} selector: k8s-app: kube-dns <del> version: v17 </del> <ins> version: v17.1 </ins> template: metadata: labels: k8s-app: kube-dns <del> version: v17 </del> <ins> version: v17.1 </ins> kubernetes.io/cluster-service: "true" spec: containers:


Question:
Observed weird memory spikes for both the dns healthz and kube-dns pods in production. Kube-dns didn't oom because there's enough headroom, but healthz did. I can't explain the spikes yet. I tried a bunch of different repros: no , bad nslookup command, sharing the node with cpu hogs etc, but was unable to trigger it. The memory grows, but stays under 10Mi. As a short term mitigation, we stole some ram from kube-dns, because it was always comfortably within limits. Related b/
can be closed -- healthz has been removed
-- close issue?

# keep request = limit to keep this container in guaranteed class limits: cpu: 10m <del> memory: 20Mi </del> <ins> memory: 50Mi </ins> requests: cpu: 10m <del> memory: 20Mi </del> <ins> # Note that this container shouldn't really need 50Mi of memory. The # limits are set higher than expected pending investigation on #29688. # The extra memory was stolen from the kubedns container to keep the # net memory requested by the pod constant. memory: 50Mi </ins> args: - -cmd=nslookup kubernetes.default.svc.{{ pillar['dns_domain'] }} 127.0.0.1 >/dev/null - -port=8080


Question:
Observed weird memory spikes for both the dns healthz and kube-dns pods in production. Kube-dns didn't oom because there's enough headroom, but healthz did. I can't explain the spikes yet. I tried a bunch of different repros: no , bad nslookup command, sharing the node with cpu hogs etc, but was unable to trigger it. The memory grows, but stays under 10Mi. As a short term mitigation, we stole some ram from kube-dns, because it was always comfortably within limits. Related b/
can be closed -- healthz has been removed
-- close issue?

apiVersion: v1 kind: ReplicationController metadata: <del> name: kube-dns-v17 </del> <ins> name: kube-dns-v17.1 </ins> namespace: kube-system labels: k8s-app: kube-dns <del> version: v17 </del> <ins> version: v17.1 </ins> kubernetes.io/cluster-service: "true" spec: replicas: $DNS_REPLICAS selector: k8s-app: kube-dns <del> version: v17 </del> <ins> version: v17.1 </ins> template: metadata: labels: k8s-app: kube-dns <del> version: v17 </del> <ins> version: v17.1 </ins> kubernetes.io/cluster-service: "true" spec: containers:


Question:
Observed weird memory spikes for both the dns healthz and kube-dns pods in production. Kube-dns didn't oom because there's enough headroom, but healthz did. I can't explain the spikes yet. I tried a bunch of different repros: no , bad nslookup command, sharing the node with cpu hogs etc, but was unable to trigger it. The memory grows, but stays under 10Mi. As a short term mitigation, we stole some ram from kube-dns, because it was always comfortably within limits. Related b/
can be closed -- healthz has been removed
-- close issue?


Question:
Observed weird memory spikes for both the dns healthz and kube-dns pods in production. Kube-dns didn't oom because there's enough headroom, but healthz did. I can't explain the spikes yet. I tried a bunch of different repros: no , bad nslookup command, sharing the node with cpu hogs etc, but was unable to trigger it. The memory grows, but stays under 10Mi. As a short term mitigation, we stole some ram from kube-dns, because it was always comfortably within limits. Related b/
can be closed -- healthz has been removed
-- close issue?

# keep request = limit to keep this container in guaranteed class limits: cpu: 10m <del> memory: 20Mi </del> <ins> memory: 50Mi </ins> requests: cpu: 10m <del> memory: 20Mi </del> <ins> # Note that this container shouldn't really need 50Mi of memory. The # limits are set higher than expected pending investigation on #29688. # The extra memory was stolen from the kubedns container to keep the # net memory requested by the pod constant. memory: 50Mi </ins> args: - -cmd=nslookup kubernetes.default.svc.$DNS_DOMAIN 127.0.0.1 >/dev/null - -port=8080