in there is a generic error variable defined that is used when a failure occurs and no other error is surfaced, it is a catch-all when something goes wrong. Think this should be reworded to something that is more reflective of the true issue and not so confusing for users. This error mostly comes when a builder can not be created and it is typically not true that the volume type is unsupported, but rather some other underlying issue, such as missing secrets or endpoints or etc... propose something like: or
are you working on this?
- yes, I would like to submit the PR for this


GCE PD docs so unused blocks can be freed back into the cloud and performance gains realized. Rummaging through the source of the GCE PD and mounter stuff in Kube, it does not appear that that's being done (the old shell script version of safeformatand_mount does, but I don't think that's used). I haven't confirmed that it's the case at runtime for sure, though, although I will be able to shortly.
I LGTMed the PR and then realized it needed more work. Reverting it and assigning it to ciwang. We need to make sure that we only use the "discard" mount option when the fs supports it. Assigning to ciwang for now. If anyone else wants to work on it feel free to grab it.
Assigning to This fell through the cracks as ciwang was working in other areas. This did not make 1.4 either moving to next-candidate.
Issues go stale after 30d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close


There was a naive attempt to do this in , but _output it hardcoded throughout the code base so we need a more comprehensive solution.
Do we still need this ?
/sig release
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close


The e2e test is flaky. It often fails with the error in the test suite. Sample failures: Marking it P0 since it frequently blocks the merge queue.
cc
In the flaky test, the pod is deleted with grace-period=30, but there is no guarantee that the pod is going to be deleted from etcd in 30s, thus the flake. This piece of test is copied from the pod e2e test: In that test, the total timeout is actually larger than 30s, because there is some extra logic (< 30s) between the time when the deletion is requested, and when the timer is started. Given that there is no flake reported for test in , I will fix this flake by extending the timeout to 60s.
This flake should have the same root cause as .
ETA on fix? This is the most frequent cause of test flakes and submit queue blockage.
sent out as a temporary fix.
Hmm, happened in: Which looks like we're just not handling the watch cancellation properly in the test.
does the timeout really fix the watch cancellation? (didn't look too deep at the pr)
I think so. see bprashanth's last comment, the timeout occurs again.
As of 3:20pm Apr.8, last failure occurs at 3:13:18pm, Apr. 7, Mike's fix is merged at 15:04:42, Apr. 7.


When trying to use kubectl from hyperkube I'm met with "unknown flag" errors. It looks as if the flags aren't being passed through to kubectl but hyperkube is trying to handle them itself (and subsequently giving an error when you give it flags it doesn't understand). You can reproduce simply by running: And you'll see the error:
I have a fix up in PR .


The recently merged PR a version cache in dockertools, which makes the test  flaky. The reason is that injected error into fakedockerclient and then expected to see the error from right after that (See ). However, the error was consumed by the goroutine which periodically updates the version cache (See ). The docker daemon version is rarely updated, and we only need version once for each container creating, during which we also do container create, start, inspect #etc. 1) Do we care about the cost of the call so much that we have to introduce the complexity? 2) Kubelet periodically calls , can we rely on that to update the cache? Anyhow, at least, we should disable the cache or have a fake one in the unit test. :) /cc
I once considered move this cache to runtime level, while thus may need to change runtime interface. What if only cache apiversion?
We can only get via , so there will still be the data race. :) One option is to update the version cache in , now kubelet periodically call to check the runtime status. Another better option is to cache the version in layer, now is under our control, we can refactor it with an additional argument indicating whether to use cached value or get directly from the daemon. If we cache the version only for performance, I prefer the second option which is cleaner and more efficient. :)
OK, will look into this tonight
The unit test itself should not start a goroutine to update the cache (unless you're actually testing the goroutine). We should disable it anyway, as we often do in our unit tests. It also sounds like we don't need a goroutine dedicated for this. If we can tolerate the additional overhead from time to time, we can set a expiration time (e.g., 1 minute), and only call the runtime if the entry expires.
Totally agree. :) Maybe a new function in wrapping the update logic, or wrapping the logic in the .
expiration time sounds good in this case
This is happening quite frequently - increasing priority to P0.


Turns our that when running a container in host network, its that is passed to dockertools.SyncPod() will always contain empty IP, as a result, the environment becomes empty. Maybe we want to use the instead of in this case to ? Since the apiPodStatus has the valid PodIP, which is when the container runs in host network. Or just populate the podIP with hostIP in when the container runs on host network. Ref cc
Why not just set the same with before ? Ideally, we want to remove from SyncPod() in the future if possible, it may be better to get it from . WDYT?
Do you mean "set the podStatus.IP to apiPodStatus.IP before SyncPod?" IMO, we should call only once. Why don't we populate set the before calling ?
We also use . However, because we don't support host network bandwidth limit, it is fine to leave the IP unset here. However, I think it would be good to have IP always correctly set in .
I must admit, I have been thinking about this in the background since created this issue, and the discussion hasn't up for me. I finally sat down and spent some time getting my numerous fixes for podIP via downward API back into my brain, and now the problem seems very obvious to me. So, after some thought, I believe that for the docker runtime, the problem is that , which is responsible for generating the which is used to supply the pod IP, doesn't check to see whether the host network is being used and return the correct IP address if so. However, does, which is why the obvious fix seems to be to use the value from this method. I think the right thing to do instead is to change the following call stack to correctly determine the IP when the host network is being used.
Today doesn't pass the pod spec, so it cannot know if the pod is running on host. This could be solved if we serialize the pod spec into docker labels. BTW, I think we have a similar issue here for rkt.
As said, GetPodStatus doesn't have full pod spec today. And I don't think docker manager has knowledge about the host ip now.
I think we should think about adding the plbuming. I have not had a good experience storingthis information in multiple places :)
The code was written so that it'd work in the absence of the pod specs (which we don't always have). I agree with you, but adding the dependency today without some form of checkpointing (e.g., container annotations) of the pod specs may be problematic now. We can revisit this after we store pod specs in the annotations. What do you think? :)
I'm cool with revisiting but if we do that we should drop this to p2
Reopen the issue to capture
Is this fixed?
hi, also interested in knowing if this is fixed.
I don't think anyone is tacking on this one right now since people are mostly working on the new CRI implementation for different runtimes. cc
thanks. it also looks like there might be a specific issue with rkt here. Looking at and , it looks like rkt does not handle the case of status.PodIP when using the host network.
Thanks for the heads up. I will take a look to see if we can quickly work around this.
Issues go stale after 30d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close


I'm not sure where the defaults come from but they sometimes drop all packets causing container IPs to become unreachable. ARP fails. I'm not sure if this is related but I also noticed on this machine that the default pfifo qdisc drops all packets. pfifo is the default qdisc for an htb node so it's used in the unconfigured htb qdisc: This is because the default value for queue size is zero. On other machines I've seen the default value set to 1 packet and 1000 packets. When I set the limit explicitly the container becomes reachable again. The broken configuration is applied by kubelet to implement traffic shapping and there is no option to turn this off. This patch fixes the kubelet on broken machines by disabling traffic shapping: cc
does this happen with --configure-cbr0 and packet shaping too? The kubenet implementation just uses the existing kubernetes shaping code that --configure-cbr0 does.
It seems the *fifo qdiscs default their packet limit to the txquelen of the interface (man tc-pfifo). It also seems most virtual interface types (bridge, etc) set a dev-txqueuelen of 0. It further seems that relies on the default packet limit. I could swear this worked before with kubenet, and it must have worked with --configure-cbr0 when the code was originally . I'll check around to figure out what the default limit should be for interfaces that don't specify a tx queue length.
Note that to test with --configure-cbr0 you'd also need --reconcile-cidr.
Upstream CNI fix is here: and we'll also work around it in kubenet for now.
FWIW, the upstream CNI fix got merged for CNI 0.3.0


At least twice now, we've created a namespace and seen that the corresponding service account default secret token is never created. (The service account itself is, but not the default secret.) Namespace creation + apiserver logs logstash-ingester: namespace creation triggered service account creation but no secret logstash-broker: namespace creation triggered service account creation and secret creation Resulting service accounts Resulting secrets Controller-manager logs for broken case Controller-manager logs for working case Version Info:
Dupe of , fixed in
Actually, not a dupe. This may be a consequence of other refactors to controller logic - possibly we are missing / dropping events out of the queue. I would expect it to get fixed on the sync interval though.
We are also seeing this happening from time to time. In our case, it wasn't fixed in the next sync interval but it took actually a little bit more of 24 hours.
can we resolve this and remove it from the 1.6 milestone?
fixed in in 1.4
We just hit this exact issue running 1.5.2. Maybe the fix didn't get the root cause or there was a regression since 1.4? This should be reopened. I waited about an hour to see if the sync interval would resolve this but it did not. I fixed by deleting and recreating the namespace. I can share logs, but they are identical to what the OP noted. Version info:
Working with this seems like it's not fixed in v1.5.2 -- what should we do here? We don't have permission to re-open the ticket, but filing the same ticket with the same logs (just fresher dates) seems passive-aggressive. :-) What's your preference?
no problem, just getting back to this. can reopen this issue.
can you recreate with the controller manager running at and include controller manager log lines from ?
Fresh repo, offending namespace "foo96", with "--v=5" since we already have that logging level in place on my dev stack. This took 96 repros of a "create namespace / monitor for token creation / delete namespace" loop. Our script's commands, starting just after 16:36:43: Controller manager log lines from The first reference to the offending namespace (foo96) and the one other entry afterwards: All controller manager log lines citing the offending namespace: For comparison, the previous successfully-created namespace: Status as of 20 minutes later:
Also, here's the apiserver logs + controller manager logs, for more action/reaction context: Namespace: foo96 (problem) Namespace: foo95 (success)
that's helpful, thanks. it looks like while the token controller is processing the creation event for the service account, it does a live lookup against etcd and fails to find the service account. what version of etcd are you running with? are you running HA etcd? do you have quorum read enabled?
We run etcd version 2.3.7 (previously saw this on 2.2.x as well). We run etcd HA, with a 3 node cluster. We do not run the apiserver with the --etcd-quorum-read flag enabled.
(Note: My specific logs above from this morning are on etcd 2.2.5, but yeah we've seen repro on 2.3.7 as well.)
yeah, the non-quorum HA etcd config is the issue. I can look into updating the token controller to tolerate that, but I'm positive there are similar issues in many controllers that assume live lookups are reliable
Ahh excellent, glad to have an explanation! So -- now that we know what to google for, we found -- where you and clayton addressed what appears to be this same problem for service accounts, and specifically noted that you wanted to avoid recommending quorum reads. Why is the answer different here? Is it because the downstream impact of non-quorum reads is too diverse to reliably retry every relevant action? (Totally fine if that's the answer, just trying to make sure we understand the reason for what appears to be an entirely opposite conclusion to the same type of problem.)
I still think controllers should be built to tolerate weak reads, and I'm willing to fix the token controller bug here, but three factors make me think this position is a losing battle: quorum read got a lot cheaper with etcd3 there are strong opinions that etcd quorum read should be on () which means that very few people are writing controllers designed to tolerate weak reads all the CI tests I'm aware of run with a single etcd or with quorum reads on
Perfect, works for us. As for fixing this particular token controller bug, I no longer have skin in that game -- your call entirely. :-)


When users try to label/annotate multiple resources, we should tell them to add . Current output: Expected output:
and too
Instead of "--all is false", say "multiple resources were provided, but no label selector or --all flag specified"
Description updated based on suggestion
I think "resource(s) were provided, but no name, label selector, or --all flag specified" is better. I will send a pull request for this as my first PR.


Currently a Docker bug may cause KubeProxy to crashloop thus making the Node unable to use cluster network, which in turn makes Pods scheduled on this Node unreachable. To mitigate this issue we need to surface the information about problems with KubeProxy, that will allow scheduler to ignore given node when scheduling new Pods. ProblemAPI is going to address this kind of problems in the future, but we need a fix for the 1.3. Either Kubelet or KubeProxy needs to update a NodeStatus with the information that Node networking is down. I suggests that we reuse NodeReady Condition for this, as this would make the rest of ControlPlane work out of the box. We will replace this fix with the ProblemAPI use in the future. Another option, which requires more work, is creating a completely new NodeCondition type to handle this case. This would require making Scheduler aware of this new Condition.
Does kubelet monitor kube-proxy health?
Kube-proxy today runs as a static pod (daemonset), kubelet treats it the same as other static pods. Yes, Kubelet reports kube-proxy podstatus back to master, that is all. The detail information on kube-proxy is in crashloop at
cc/ Brandon, this issue explains my long position against running kubelet, kube-proxy, etc. critical daemons as docker container. I am ok to packaging them into docker image, and running as linux containers, but they shouldn't depend on docker daemon.
I think we don't want to run them as docker containers as well, we want to run them as rkt containers.
The initial proposal I reviewed was docker container, and I did raise my concerns to and coreos developers then. Running it as rkt container should be ok since rkt is daemon-less today, and shouldn't have this chicken egg issue at the first place. But we still have the problem on the nodes without rkt as runtime. At the meeting, I suggested that kube-proxy update NodeStatus with a newly introduced NodeCondition, like what we plan to do with kernel issue at: The reason of not using NodeReady is that kubelet might override that. Another very hacky way is before crashing, Kube-proxy logs the read-only filesystem problem somewhere, like /var/lib/kube-proxy/... Kubelet will pick up the information, then update NodeReady Condition with detail error message prefixing with "kube-proxy:" kube-proxy's responsibility to make the information up-to-date.
Can I have more description of the bug? Is there a related docker issue? Is there a repro? Does this only affect kube-proxy or will it affect other pods on the node? We need to support running critical applications (including system daemons) in pods in order to allow easy deployment and integration with vendors (e.g. storage and network vendors). cc since we were just talking about integrating through addons. What features are we missing that we can't support this right now? Is it just that our default container runtime is buggy and flaky and unreliable? Do we need the NodeProblem API? Do we need taints/tolerations?
- I don't know about any way to repro it other than running huge clusters. Pretty much every run of a 1000 Node cluster has this problem on at least one Node.
If you want to know the detail, see It is a docker issue, and pretty rare.
Can we regroup on this issue this week? Like tomorrow? , Dawn Chen wrote:
I will happily make kube-proxy write to or offer up pretty much any API we deem necessary to detect this issue. To me it still feels like a docker health-check suite is needed, and one of the tests is that sysfs gets mounted rw. BUt in the mean time, how about this: make be the API. Any component which feels that it has a node problem can write a JSON file into that directory. For this case, kube-proxy would write a file: Kubelet can turn that into Node status reports. , Tim Hockin wrote:
cc
PR of introducing NodeProblemDetector was merged last week, and the pr to enable it by default for Kubernetes cluster was merged over the weekend. So far everything works fine. Even it is an alpha feature, GKE folks agreed to enable it for GKE. To workaround this particular issue before we have proper docker fix, we can easily extend today's KernelMonitor module of NodeProblemDetector to monitor kube-proxy log, and report a new NodeCondition called SysfsReadonly to the issue visible. But we don't have remedy system yet, and pr of converting node problem (events, conditions etc.) to taints is still under the debate / review: 1) Should NodeController respect this new Condition, and simply mark node NotReady? 2) Should repair system pickup this issue, and restart docker?
Do we still observe the problem with docker 1.11.X? If no, I want to close this; otherwise, we are going to reconfigure NodeProblemDetector to make the issue visible.
cc/
Agree, I'd love to not "fix" this..
I think we've seen it the day before yesterday during our tests - can you please confirm?
I can't recall - we certainly did saw some failed nodes during the test, but I'm not sure we made sure it was a kube-proxy issue.
But what I can tell for sure that it's definitely less frequent - we started 2000-node cluster 10+ times during this week and we've seen it at most once (which makes it at least 99.995% reliable).
Not enough data. GKE is still deploying the old ContainerVM and isn't running Docker 1.11.1 yet, unless I'm mistaken.
aah ok - I didn't know that...
I'm going to tentatively close this until we show that it's a problem on Docker 1.11.2, now that Docker 1.11.2 is in on both GCE and GKE.
(Obviously feel free to reopen if I'm wrong.)
Another occurrence:
So it's still happening, can we teach NPD to detect and surface this as node unschedulable?
Yes, we can. But I found that the error only occurred once with docker 1.11 till now. At least, this should be a known issue, right? XRef Do we want to fix this in 1.3 or just claim known issues in release notes? :)
Doesn't this render the node unusable (i.e no services will work because kube-proxy is broken)? if so, we should surface regardless of frequency IMO. Whether we fix or paper over it is a different issue.
+1
Because it's code freeze now, I can add a temporary short hack in node problem detector to detect this issue. Should we do that? :)
Removed the priority and it back to 1.3 milestone, so that we can discuss at burndown meeting.
Yeah i think codefreeze just means no features
It is very rare after we upgrade to docker 1.11.X. To extend KernelMontor in NodeProblemDetector to detect the issue is totally overkill, since we have to allocate more compute resource to NodeProblemDetector to process tons of kube-proxy logs. Here is what I suggested for this issue for 1.3 and 1.4+: Document this as known issue for docker releases: docker 1.10.X (), docker 1.11.X (). Also document it as Kubernetes 1.3 release known issue at is also updated admin guide for NodeProblemDetector, and the suggestion on how to handle this issue will be included there. For 1.4, I think we should change kube-proxy. Instead of logging the error then crashing, kube-proxy should report its problem to NodeProblemDector, which aggregates the node issues and propagate them to upstream layers, so that the admin / remedy system & control-plane can take proper action upon the issues. I am removing this from 1.3 milestone, and unassigning myself. cc/ to find the right owner from cluster team for next release here.
On a 2k node cluster, I just saw 4 nodes come up this way on Docker 1.11.2. I'm not saying it's going to stop the release, but it does stop bringup on large clusters pretty frequently. For some reason we went through a "nice" patch where it was rare, but today it was just bad.
We can have a quick fix in node problem detector to catch this kind of problem by parsing the kube-proxy log. The only problem is that, kube-proxy generates logs relatively fast. If we let node problem detector do that, it has to waste significant cpu even on the good nodes. A better solution is to make node problem detector expose an endpoint and let other system components report problems. However, we are not there yet for 1.3. If we'd still want to solve this for 1.3, we could: 1) Let kube-proxy have some kind of error log, and only print error into it. Then we can easily change node problem detector to parse the error log and report problems. By this, we can at least make these errors visible to users for 1.3. 2) Or let kube-proxy report the node condition itself. /cc
Comment at is still valid. We cannot do much for 1.3 release here, and extending NodeProblemDetector to parse the application logs is wrong move. Also NodeProblemDetector is not running on GKE nodes yet.
I'm seeing people say they reproduced on 1.11.2, is there a 1.11 docker issue? I know that was closed. If I'm going to get my docker guys to dig, I need a docker bug report...
Can you please ramp up on this and come up with a way to bring node and kube proxy teams together to resolve? Even if we don't land in 1.3.0, I propose this is at least a P1 for 1.3.1.
FYI and others In I a hacky way of resolving those issues to our test framework. In the last run of gke-large-cluster run it seemed to work (there was 1 broken kube-proxy, this triggered restarting docker, which in turn fixed the problem). This is obviously not the solution, but: it seems to prove that 'docker restart' solves the problem it should stop the bleeding in our tests
& I copied & paste what I wrote in another internal thread to give you some background: No, it is not a blocker for 1.3 release: It is not a regression from 1.2 release. It is way less frequently after we upgrade to docker 1.11.2 from docker 1.9.1 NodeProblemDetector is not designed for processing arbitrary logs. It is an overkill solution given such low frequency and relative high resource overhead. By default NodeProblemDetector is not running on gke node due to the extra overhead per node introduced to our customer. Meanwhile Since GKE already has repair system built, it should be very easy to extend GKE repair system to check if kube-proxy is in crashloop. If yes, then kubectl logs to retrieve the last terminated container log to see if the issue is caused by readonly root filesystem. If yes, the repair system can issue a docker restart.
agreed not a blocker. This is likely only fixable in docker. We should keep it up the priority so we dig into and fix docker. can try next week to get our docker people talking to whoever on your side you want to get this fixed (in 1.11.3 or later probably)
Can we add a node readiness or liveness probe to solve this? (I remember there is a issue for this, but can't find it now) One of the default behaviours of the probe is to make sure the key node components are running normally, and the behaviour could be extended overtime. Probably adding a node probe controller or node health controller to perform the probing. Or just use the health probe of GCE MIG, if its functionality is enough to handle this. This could be part of the remedy system. Just random thoughts, not for 1.3 and even may not work. :)
All are good ideas. But let's move that discussion to a separate issue, and use this one to focus providing the workaround for KubeProxy panic within 1.3 timeframe. Thanks!
Also what you suggested probe controller can be easily achieved by GKE repair system which I suggested above.
Agree. needs quite a lot of work, definitely not for 1.3 :)
Keeping in v1.3 until we decide on Docker 1.11 for sure.
The issue seems to be present on old and new docker alike, sadly.
I opened a docker issue here Feel free to supply information on that issue.
So what's the right kube-proxy fix? , Lantao Liu wrote:
- didn't we decided that we just want to discover this problem in NodeProblemDetector and then mark this node as not-healthy (whatever that means - adding a new condition or extending NodeReady or ...)? In my opinion if it would be clear to user why this node is not healthy, this should be good enough. FYI in I worked around this in our tests :)
It wasn't clear to me if kube-proxy should do something different or not. , Wojciech Tyczynski < :
yeah this is a docker bug (), unsure if there's a pod level workaround, couldn't make out if it was specific to hostNetwork. the work around is restarting docker. It happened more frequently in 1.9, but still happens on 1.11 (node team empirical data). To surface it to the NPD we need to either send some http event to a NPD server (vaporware) or write to a special hostPath termination log type file (parsing kube-proxy logs form NPD is too expensive).
This feels like a hot potato but I am not sure if it still needs to be addressed or not. If it does, let's do so. I apologize for not being able to understand what the current status is. It looks like we have a . Reading comment from 12 days ago she states: While folks may want the node problem detector extended to parse logs to detect it, that is not a good fit for that system and it doesn't run on all nodes anyway. Also GKE can be extended to possibly repair this problem. Questions: Is this an issue that we need to still address for production? Or can we wait for docker fix? If we haven't fixed the kube-proxy panic (which we should) how do ensure we still provide a signal that this is happening?
I am leaving this on 1.3 until we know the best thing to do.
If we can DIRECTLY tell the node problem system about a problem, I have no qualms about doing that. drop a file in a dir, for example. , Michael Rubin wrote:
Node problem detector is not running on GKE cluster now. :) and should be a better fix for this. If we still want a fix in the node problem detector, letting kubeproxy drop a file could be enough for now.
If you can give me a spec of what dir to mount and what sort of file to write, I'll do it. , Lantao Liu wrote:
For temporary fix, something like should be enough. The kernel monitor of node problem detector could be ported to parse other logs. The only reason we can't do that for kube- is that the log is too spammy. should be enough for now. Only the kubeproxy panic error is printed directly into : Even though the default  of glog is , I checked the node in and cluster, normally only few lines are at Error level.
I thought that Dawn said we DO NOT want to parse logs. Parsing logs is, in general, a total hack and very very brittle. we should have a real push API. , Lantao Liu wrote:
I agree with I also understood thought we should not parse logs. Is that common practise for the node problem detector? It feels fragile to me and to be avoided.
Yeah, I agree parsing log is fragile, that's why I call it a temporary or a quick fix. :) The only good thing is that it doesn't need significant change on both sides, although I don't think it's a good solution, either. :( For a better fix, it will need some more design and work. Let me think about it a little more~ Anyway, I still think and is needed. As a key component running on each node, it's wired that we don't have a component to monitor whether it is ready or not.
Re making kube-proxy "not panic", it currently fails in resizing the hash table for conntrack entries, for which it needs to write to /sys. I'm not sure if there are more locations (we moved /sys/module/br_netfilter loading into kubelet IIUC), or if conntrack itself needs write access to /sys. I think Dawn/Liu are more worried about the NPD not parsing kube-proxy logs, because its memory usage would balloon. If kube-proxy can detect the situtation internally, either by parsing its own logs (which is brittle) or poking someting in /sys for example (like the hash table resizing it's currently failing on), it can drop a token into a hostPath (hoping the docker bug doesn't manifest as ro hostPath). Kube-proxy will still remain dysfunctional, though we might be able to: the issue/mark node unusable a feedback loop that restarts docker We just need to be careful to cleanup the hostPath so it doesn't end up in a restart loop.
My understanding of 1.3's state based on a brief chat with : something like ~1 in 1000 nodes will be affected by this issue and if nodes are affected, pods scheduled on those nodes have {broken,little,no} networking. We hope Docker will eventually fix it. But because that's not the case for 1.3, and we don't want to make excessive changes to 1.3, perhaps that means that we should fix it expediently. Summarizing my understanding of previous discussion: expedient thing would be some mechanism to detect ( proposal like kube-proxy detecting and writing to /tmp/kube-proxy-claims-docker-issue-) and $something (kubelet, node problem detector, a bash script in a loop) that checks for existence of that file followed by some form of remediation {kill docker, sudo reboot, .....} to bash the node into working again. Maybe an ugly hack is okay for now and we hope that docker fixes it for the next release so in 1.4 we can rm the code? Or for 1.4 we can do something that is More Elegant And Less Offensive To Engineering Sensibilities?
Bounded-lifetime hacks are fine, but they need to come with giant disclaimers and named-assignees to cleanup the mess after a specific event. , Alex Mohr wrote:
Agree, re: bounded-lifetime hacks needing care and we shouldn't do them indiscriminately. We and our users have a problem today -- and we've had a problem for the past N months without traction. I'd like the problem solved for 1.3 because I'd like us to have a great product and I don't think the state of broken nodes is there. Aside; we've spent N hours discussing the issue and could likely have worked around the issue in less aggregate people time than we spent discussing? Apologies for the leaked frustration: this issue seems to be stuck in either some form of analysis paralysis or cracks between teams or perfect-is-the-enemy-of-the-good state. Again, I don't care about implementation details. We also need more than just tech or a piece of code. Exit criteria is that users don't get broken nodes shipped to them.
We're not going to solve this problem for as long as we depend on a lagging docker release cycle, and they don't backport or do intermediate releases to help us. This is just a one off docker bug that's left us in a bad state. Such issues come up every release, get documented in the release notes and we ship anyway. The easiest fix (disclaimers and all) is to redirect kube-proxy stderr and parse it out from NPD. Even then, the node will remain broken unless we restart docker. the NDP will only surface the error.
I'm working on a PR yesterday night, will send it out soon.
FYI, a fix is here . In , I let kube-proxy update a node condition with specific reason, message and hit to the administrator about the remediation. The specific reason why we want to do this is discussed in the PR description
FWIW I agree 100% with using node condition SGTM, but your PR should probably also modify the scheduler here to prevent the system from sending new pods to a node that has RuntimeUnhealthy condition. I realize there will still be a time window when we might send some pods to that node (before the problem is detected) but at least the time window will be bounded this way.
Yeah, we can do that. But before that, we should make sure that the node is really unusable without setting conntrack. In fact, we only setting conntrack to increase the max connection limit on the node (default 64k-256k) Without this, is the node really considered to be unusable? If so, we should prevent scheduling pods to the node. If not, maybe we should just surface the problem to the administrator and let the node keep working. I'll run e2e test without conntrack set to see whether there is any problem. Yeah, that's why I think we should not restart docker ourselves to try to remedy the problem. There may still be workloads running the node. We'd better leave this to the user to decide. :)
is merged, and hopefully, it could solve this issue. I've also sent a PR to revert the walkaround in the test framework to verify whether the issue is really fully solved. Feel free to reopen this if the issue or relative issue happens again.
thanks for the help!
I just ran into this, and I am able to repro consistently. It might be due to the craziness of my experiment, but I still wanted to contribute the data point. I am experimenting with docker inside docker. All I am about to describe is itself running inside a privileged CentOS 7 container that has systemd and Docker installed. The CentOS 7 container is running on my dev machine (Docker for Mac). I have a kubelet running fine, and all control plane components running as static pods. When I attempt to run kube-proxy as a static pod (privileged) it crashes with the following logs: sysfs is mounted as rw: sysfs is mounted as rw on containers: Using busybox to further inspect:
I'm seeing the same issue when running kubeadm-dind-cluster, where kube-proxy is failing to come up, because it is trying to write to /sys/module/nf_conntrack/parameters/hashsize. This is on the filesystem sysfs, which "mount" shows is rw (so it passes the R/W check in kube-proxy code), but the filesystem is not writeable. Within kube-proxy: I'm also running docker containers inside of docker containers. FYI, others have not seen this issue, when just running docker on bare-metal.


Slow conformance test has caused issues like We should make runtime/image conformance test run faster without breaking the test coverage, possibly by: Shorten consistently check interval . Remove redundant test cases such as ... /cc


Ref and should sort revisions correctly (10 should be after 9, not 1).
I can help with this bug. Looks like trivial fix in PrintRolloutHistory
Thanks
are you still taking care of this one ? I am searching for some easy to do PR to just get familiar with the process and put my first PR in.
I created pr some time ago to close this issue
great thanks :-)
you can look for help-wanted issues or check and
thanks


vSphere volume spec fails with the following error, as its not been in api/validation. "spec.volumes[0]: Required value: must specify a volume type" Fixed in PR


PersistentVolume integration tests fail sometimes: There are some interesting reports in the log like: Thans for report.
It turns out that volume integration tests have no logs at all, was not checked and I don't really know what's going on there. adds error checks to help me find out what's wrong there. Still, I suspect there are some troubles with etcd, not related to the test itself.
This flake affects also TestPersistentVolumeMultiPVs:
Extended logging got merged just now, please post links to logs when it breaks again!
Marking as P1 as per
I have no real usable logs, but I think it could be fixed by , it hasn't flaked since it was merged. And I have (in merge queue) to further stabilize the tests.
Not blocking this on 1.3.
It hasn't flaked for almost a month, can we close this?
Closing this issue since previous fixes made the flake go away.


removed this test. We need to continue testing this. This wasn't about testing upgrades, the prefix has nothing to do with upgrades. It's just there so people can share etcd. We maybe don't need to double everything but we do need to make sure that keeps working.
To be honest - I think that what we was testing before didn't give us completely any information. We didn't testing what information we are storing in etcd and where. That said, if we had a hardcoded prefix somewhere in our code, we wouldn't discover it at all, and those tests would still be passing. We should have a test that is directly checking etcd contents. I think it should be possible to write a simple unit test for it. Will try to put something together next week.


If the first scheduling failed (whatever the reason is), the Error() func is called. However, the default Error function is wrong and in case of some errors may drop the pod and never requeue it for scheduling again. The problem is that if the Get() function: return an error (for me it was "dial tcp 127.0.0.1:8080: connect: cannot assign requested address", but it doesn't really matter), then we simply forget about a pod and will never schedule it again. We should retry those requests (with some backoff).


For example
This is with recent kubectl:
I think this is a duplicate, but just in case.
I will double check this.
Thanks - it isn't a big deal, because I obviously didn't intend to rolling update to an empty manifest :-) (I was experimenting with templating my manifests and made an error) But let me know if you need any more info to reproduce!


Observed weird memory spikes for both the dns healthz and kube-dns pods in production. Kube-dns didn't oom because there's enough headroom, but healthz did. I can't explain the spikes yet. I tried a bunch of different repros: no , bad nslookup command, sharing the node with cpu hogs etc, but was unable to trigger it. The memory grows, but stays under 10Mi. As a short term mitigation, we stole some ram from kube-dns, because it was always comfortably within limits. Related b/
can be closed -- healthz has been removed
-- close issue?


https://k8s- Failed: [] SchedulerPredicates [Serial] validates that InterPodAntiAffinity is respected if matching 2 {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that InterPodAntiAffinity is respected if matching 2 {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that InterPodAntiAffinity is respected if matching 2 {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that InterPodAntiAffinity is respected if matching 2 {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that InterPodAntiAffinity is respected if matching 2 {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that InterPodAntiAffinity is respected if matching 2 {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that InterPodAntiAffinity is respected if matching 2 {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that InterPodAntiAffinity is respected if matching 2 {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that InterPodAntiAffinity is respected if matching 2 {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that InterPodAntiAffinity is respected if matching 2 {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that InterPodAntiAffinity is respected if matching 2 {Kubernetes e2e suite}
Looking into this...
trying to track this flake down. It looks like the pod is launched despite AntiAffinity in . In my local setup I saw the same, but noticed that the new admission controller was not activate. I have activate it and now get this:
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that InterPodAntiAffinity is respected if matching 2 {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that InterPodAntiAffinity is respected if matching 2 {Kubernetes e2e suite}
Should be fixed by


https://k8s- Failed: [] Services should preserve source pod IP for traffic thru service cluster IP {Kubernetes e2e suite} Happened on a presubmit run in .
Sorry, it is a new e2e test I am still working on. It is not merged yet.
Don't worry about it, our flake detection isn't quite perfect yet :)
[FLAKE-PING] This flaky-test issue would love to have more attention...
[FLAKE-PING] This flaky-test issue would love to have more attention.
This new e2e test has been merged. Let's observe it few more days and close this issue if nothing bad happen.
[FLAKE-PING] This flaky-test issue would love to have more attention.
This does't look like a failure in new test, rather its a generic failure with serviceaccounts.
Thanks for the investigation. Yeah I think it is the serviceaccounts issue. I wonder when could we close this flaky issue. Seems like it is stable if serviceaccounts issue is fixed.
You can close it with a reference to the serviceaccounts issue - do you know the # for that one ?
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] Services should preserve source pod IP for traffic thru service cluster IP {Kubernetes e2e suite}
https://k8s- Failed: [] Services should preserve source pod IP for traffic thru service cluster IP {Kubernetes e2e suite}
https://k8s- Failed: [] Services should preserve source pod IP for traffic thru service cluster IP {Kubernetes e2e suite}
https://k8s- Failed: [] Services should preserve source pod IP for traffic thru service cluster IP {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
Have a fix for this, I'm now keeping running tests on my own cluster. Will send out PR once verify the fix works.


On a cluster using GCI, I'm not able to create docker containers using the docker CLI. On further introspection, it looks like bridge is being deleted. I consider this a regression. K8s used to allow creating docker containers out of band and in-fact it is a feature that is taken into account as part of k8s node features. To further exacerbate this issue, exists on containervm. We need to match the behavior between GCI and container-vm and ideally stop deleting docker0 bridge. cc
During GCI start up, got deleted to avoid overlapping cidr with . ref:
there's no need to delete docker0 afaik. It was deleted on gci master to fix the "overlapping cidr" problem, i.e we were supplying to kubelet on GKE master, and this resulted in 2 bridges with the same cidr. At the time I'd suggested passing the correct cidr. Don't remember where this discussion ended. Note that containers on docker0 with/without cni are broken anyway, we used to start docker with --bridge=cbr0, so would result in an ip from cbr0. This is no longer the case. We use kubenet to put containers on cbr0, so will result in an ip from docker0, which is not connected to nanything else.
This shouldn't cause docker crashes? you should still be able to create containers, just no networking. And networking is broken through docker CLI anyway because of the cni issue mentioned, or are you observing something different?
will return error. Have to add
Since bridge has been deleted, docker run by default will fail unless one overrides the network ns to be host or none.
so what we're really losing is the ability to docker run then talk to others (external internet, other pods) without net=host on GCI. This is useful and we should fix it, as mentioned earlier we probably don't need to delete docker0. docker run + others talking to us is broken because of the cni change, and will be broken even with docker0 (IMO this is fine).
cc
Yeah. I'd prefer ignoring docker0 as long as it doesn't affect kubelet functionality. , Prashanth B wrote:
Vish pointed out that of docker dies, docker0 comes back. Given that we should be consistent. Either force docker not to create a bridge at all, EVER, or just ignore docker0. We may even want to consider assigning --bip= 169.254.123.0/24 or something, so we can reliably use 172.* for clusters eventually. , Vish Kannan wrote:


When I tried to use the following command to build kubernetes: I encountered the following errors: I'm using Ubuntu 14.04 and the code from master branch. will submit a patch then to fix this problem.
whoa, I just landed in the exact same place, I am trying to build k8s on a CentOS7 box. Tried with removing the unused utils and works like a charm !


liveness probe was introduced in Heapster seems to work correctly so we should either fix it or revert liveness probe.
cc
port seems to be wrong


introduced a new requirement for node names to be DNS1123Labels. In case of hack/local-up-, an IP address is used as node name and thus endpoints aren't created for services, e.g. kube-dns doesn't work anymore. There are other projects that use IP addresses as node names for running simplified k8s dev environment: Probably there are other projects where it's convenient to have just an IP address as a node name. Verified against . On affected installs, service latency e2e test hangs, see e.g. Example ( some newlines):


During a manual upgrade test, described in , a kubelet failed to communicate via SSL to its master. This is currently blocking the 1.4 release. There is not a repro for this yet. The cluster was running 1.3, and was upgraded to 1.4 successfully, but upon downgrading to 1.3, the kubelets one one or more machines are not able to talk to the master. Current kubelet version is . (EDITED) The error when talking to the master is (lots like this): I think this means that the kubelet cannot verify the master's cert. I can't find the master's cert. Since there is not flag to kubelet to tell it where to find its config, I assume it is using the default location, which is . It seems wrong that clusters[0].cluster.certificate-authority-data is empty. That should have the cert for the master, right? I see this thing: but I don't understand what process generates , and if, presumably, from that template, why it isn't setting the env var CA_CERT.
in case this is related to kubelet bootstrap changes in 1.4.
Currently node is :
Correction. The node is running 1.4.0-beta.5
If the kubelet hasn't set the then it will not go through the bootstrap code path. Is the kubeconfig provided through the bootstrap flag or the kubeconfig flag?
Not set. For reference, args are:
Question: Could you please tell me why your nodes are running with GCI image at the first place? The test is starting with 1.3 release, which is using old debian-based containervm image, and I thought doesn't change the image at all (separate topic).
It does change to gci on gce with the default upgrade scripts, i brought this up on an email thread. I'm guessing something was dropped when we translated salt grains (containervm) to gci env vars.
What the default upgrade scripts you were referring to? I just double-checked with test cluster which was upgrade to 1.4 through , the node is stay with debian-based containervm, but Kubernetes version is bumped up to 1.4-beta.X
Ok, I am assigning this to to confirm. If upgrade logic from 1.3 to 1.4 changes the underneath os, that is wrong. I strongly believe this is just a simple operational error.
My understanding is that ignores base images altogether. , Dawn Chen wrote:
Hmm
I've tried and the node image did get updated to GCI. Before upgrade to v1.4: After upgrade to v1.4: After downgrade to v1.3: /cc
helped me debug this. He found that the metadata for the instance did not include the CA_CERT in it. That seems wrong.
Renamed from "x509 failure for kubelet after upgrade test"
Yeah, that's the error you'd get if the proper ca isn't loaded.
Will we get this problem if we don't upgrade to GCI (i.e. we stay with CVM) in -N? I guess we can try it and find out.
Manual upgrades have been successful if we don't switch image types. , David Oppenheimer < :
re: If you don't change the underneath image during the upgrade, the issue shouldn't occur.


./hack/e2e- line 1192: local: `replica-pd=jenkins-us-central1-b-master-pd': not a valid identifier FYI - this looks related to your recent work? I don't see any obvious PR to blame, but I seem to remember you guys touching this part of the code recently? BTW, I deleted a bunch of leaked resources in the relevant test project on Friday (9/16), assuming that the test scripts would re-create whatever they needed. It's possible (but unlikely) that this has anything to do with that.
Here are some example logs to aid debugging: https://k8s-
I believe this to be the source of dysfunction: Which is turn caused by what you found, which causes the disk to not be cleaned up ahead of time: You can try this out in your local shell: git blames leads us to this recent commit: cc the author The root problem is unfortunatley very annoying: this is how bash tells you that variable names (in this case, $replica-pd) cannot contain dashes!
fix is up in
Awesome many thanks. I'll make sure it gets merged ASAP.


After PR , I did a bit experiments on kube-dns graceful termination and got some unintended result as below: I guess it is because the other container was terminated before . DNS query from clients failed because is the actual connection point. After a bit discussion with and two solutions for this are: another program wrap up the dnsmasq and capture the SIGTERM signal. out a way to modify the sig mask of process without touching the original codes.
Something I have been meaning to try: Put in Dockerfile Build and push image We should deliver SIGCONT rather than SIGTERM. I did a quick test and it seems to work. Otherwise trapping SIGTERM in the shell and ignoring it is well documented... , Zihong Zheng wrote:
Thanks, I will take a look and test these.


Kubernetes version (use ): 1.3.7 Environment: Cloud provider or hardware configuration: Bare-metal Ubuntu cluster OS (e.g. from /etc/os-release): Ubuntu 14.04.3 LTS Kernel (e.g. ): 3.19.0-47-generic Install tools: KUBERNETESPROVIDER=ubuntu ./kube- Others: Docker version 1.12.1 What happened: appends the same copy of each time we start and stop Kubernetes cluster (using Ubuntu's & ). For example, if we start Kubernetes cluster the first time and restart it again, the content of is doubled: This cause the following error when starting Kubernetes as seen in : What you expected to happen: content should not be concatenated every time Kubernetes is restarted. How to reproduce it (as minimally and precisely as possible): Start Kubernetes, stop it, and start it again, using and . Anything else do we need to know: One possible fix is to remove the in the following line in :
This is a bug and easy to fix. I'll open a PR to fix it
Bug has been fixed.


https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite} Previous issues for this test:
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
We understand the problem here - I'm working on the fix.
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}


What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): vSphere, "message": "no endpoints available for service "kube-dns"", kube-dns, etc -- similar issues include: however it's not quite the same Is this a BUG REPORT or FEATURE REQUEST? (choose one): Bug Kubernetes version (use ): Environment: Cloud provider or hardware configuration: vSphere OS (e.g. from /etc/os-release): Using provided vmdk Kernel (e.g. ): and Install tools: GOVC + the script What happened: Our environment utilizes the RFC1918 10.0.0.0/8 block, so I edited to utilize an address space we are not using. This is my current configuration: After the deployment is finished, I am able to login to the dashboard via the URL provided via , however, when I check the DNS server I see: When I list services, I see: When I start a container and try to ping kube-dns, I am unable to: I can ping external IP addresses: Here's the configuration on each minion and the master: What you expected to happen: I expect containers can reach ping eachother. How to reproduce it (as minimally and precisely as possible): I've reproduced this problem using both 172.16.0.0/16 addresses as well as the 192.168.0.0/16 addresses. Anything else do we need to know: I followed the documentation here: and the only change I made was to the network configuration.
I just noticed that doesn't include any routes for the service IPs I defined (e.g. has no routing table entries on any of the minions, only ). I'm not sure how to fix it, but it looks like the DNS server can make requests to the upstream DNS, it just can't get responses back.
cc
Some additional info I probably should have included the first time:
I just noticed that while from inside a container I can ping google: I cannot ping our internal DNS which is at IP . So I think I'm getting closer to where the problem actually is. I also think the 192.168.120.120 was not actually unreachable, just not pingable. For example: Returns right away, while doing the same to IP hangs. I'm not sure why when using our DNS in the of my master/minions doesn't seem to give them the ability to resolve internal assets on our network.
Can you provide your output for iptables also
Hi imkin, Sure thing! Minion 1 - 10.248.30.38 Minion 2 - 10.248.30.40 Minion 3 - 10.248.30.37 Minion 4 - 10.248.30.39 Master - 10.248.30.36 Let me know what else I can provide, I haven't had much time to continue troubleshooting this week, but getting this to work would be a really great template for building our bare metal deployment. Thanks! ZC
Can you try with DNSSERVERIP="192.168.128.120"
Hi - I will edit that in the initial configuration and re-up my cluster this morning to see if that helps things work out of the box so to speak. Thanks! ZC
Hey I have edited my configuration as follows: I have re-run the script and everything came up and validated. I am still experiencing the same behavior, however. I now see this in the logs for the DNS service via the dashboard: I see that the service can't be reached by kubectl either (it doesn't even return kube-dns like it did previously): I also notice that on the minions, none of them seem to have routes for the higher IP ranges:
This looks to be the same as posting a PR shortly.
Awesome!! Thanks - I'm grabbing the from your pull request and will re-open this issue if that did not resolve it (once it's closed by the merge).
In the second case [when your kube-dns had the ip 192.168.128.120] where was your kube-dns?
based on the logs you posted you were definitely impacted by the issue in but I am not sure if that is the only issue on going on your setup. Please let us know how the fix goes for your environment.
There is no route for 192.168.3.0/24


Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT Kubernetes version (use ): Environment: Cloud provider or hardware configuration: AWS OS (e.g. from /etc/os-release): Debian GNU/Linux 8 (jessie) Kernel (e.g. ): 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux What happened: Deployed a scheduled job from this manifest: After some time the jobs are no longer executed: What you expected to happen: Jobs are executed as defined in the manifest. How to reproduce it (as minimally and precisely as possible): Deploy a scheduled job and wait for some time.
Some logs from the controller manager:
Ping. More than 2 weeks and no action at all? Anything else i can do?
I'm also encountering this using both and as concurrency policies, fwiw.
I'm hitting this as well. Kube
Do the jobs start to reschedule if you delete old one? Try running . cc
From kube-controller-manager logs: The job is scheduled for 1:00. I think the "Multiple unmet start times" message is a clue of where things start going downhill. Also, notice how at 1:00 the controller wants to create job (for a whole day, it had failed on ).
Deleting (all) old jobs / pods fixes the problem temporarily. After some time the scheduled jobs face the same issue again.
There's a fix () that I've submitted against master to fix the Replace strategy, I'll mark it for inclusion in 1.4.
The other problem that I see here is this: It is possible that our is creating duplicates. I'll try to investigate that a bit more.
sorry for the delay in response time, I must have missed that while going through issues. Next time you're experiencing problems/have a question wrt Jobs/ScheduledJobs, just ping me directly in the issue.
No problem and I will. Thanks!
I'm still hitting this on
Fix in flight , I'll make sure to cherry-pick it back to 1.4
does this ship in to latest kubernetes release?
yes, this should be present in 1.5, 1.4 cherry-pick hasn't been merged yet.
so this issue is still present in v1.4.6?
yes - it is still present there. My fix was merged to 1.4 yesterday. You should wait for the next 1.4.x release.
thanks!
which according to should happen Dec 8th.
any news on the release?
See !msg/kubernetes-dev/o0At2-4KKQ8/_kPVM1XyDAAJ
so v1.4.7 will be released on that same day?
Look like should be available later today: !msg/kubernetes-dev/00lJpT9pxDc/YZfDzCxMDQAJ


It fails with: cc
Looking at the generated I see that FEDERATIONIMAGETAG is "" which is resulting in this error. I see that before , FEDERATIONIMAGETAG was being set to but that docker tag file does not exist for hyperkube. I see the following exist in my : but does not exist there. Trying to figure out why.
cc
make sure you are running the scripts with env var exported. That is what controls whether gets created.
With , we are not generating the federation-apiserver binary and instead using the hyperkube binary.
sorry, i copied the wrong line. The droid you're looking for: File conditionally written during process File consumed when templating federation control plane yaml
Right. My is also empty. I am trying to run federation e2e tests. The command I am running is: which is what I have always used. I guess something changed in
that should definitely work. Please and then run the step, capture the output and email to me? I was literally doing exactly this process for another PR today, and I verified the image tag file was getting written correctly as well.
I figured out this is happening because after make clean or a fresh clone, kubectl will not be available when we are trying to get the semanticimagetag_version() in here.
Awesome! Thanks


Implement unit (eventually, integration) tests for . This is an effort for Q4 2016 to be lead by and Refs
Has been implemented very much thanks to ;)


Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
, it seems we forgot to add to ; I'll create a PR for that.
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!


Similar to
is a dup of this. PR is just a quick try. Will spend more time on it to cover all cases.
status?
I sent PR which is waiting for reviewing from . Also, that PR is blocked by issue which is self-assigned by you but not fixed yet :)
That issue might be not hard to fix, but I didn't find how.So I need your help. IMO, we mignt need to change CI jobs configure in some files or images.
After discussing with reviewer ( this might be better suited to merge but not for 1.7.0, as it touches a part of the code that is quite old and stable. In other words, we probably need to give this change some time to soak in the e2e system as opposed to put right away into the release.


<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): Is this a BUG REPORT or FEATURE REQUEST? (choose one): FEATURE REQUEST <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): all :smile: Environment: Cloud provider or hardware configuration: GCE OS (e.g. from /etc/os-release): all Kernel (e.g. ): all Install tools: Others: What happened: uses the cli a lot and does a (in the commonly used function ) before doing things like . updates can introduce breaking changes (for example in the recent 132.0.0 release it removed a bunch of flags). This has caused cross-the-board test failures like . But this is not limited to just tests. Any kubernetes GCE users who rely on the and commands could have been broken. What you expected to happen: Stop updating gcloud automatically. We could either warn user to have the required dependencies installed, or pin to specific versions of dependencies like gcloud. How to reproduce it (as minimally and precisely as possible): download 1.4.4 release tarball run and Anything else do we need to know:
cc
/cc
cc I think it's reasonable to remove gcloud update calls from scripts, as I believe gcloud is updated out of band by k8s continuous testing. The scripts will still need to work with latest, so maybe they should check for a minimum version.
Good idea.
Yeah, we definitely want to continue using latest in CI so that we know if things break, but I think it's reasonable to remove the update itself from the kubernetes cluster util scripts.


Is this a BUG REPORT or FEATURE REQUEST? (choose one): Bug Report Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2016-10-10T18:19:49Z", GoVersion:"go1.7.1", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"3", GitVersion:"v1.3.9", GitCommit:"", GitTreeState:"clean", BuildDate:"2016-10-20T20:37:18Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"linux/amd64"} Environment: Cloud provider or hardware configuration: GKE OS (e.g. from /etc/os-release): Local: El Capitan, Remote: GCI Kernel (e.g. ): Local: Darwin ... 15.6.0 Darwin Kernel Version 15.6.0: Thu Sep 1 15:01:16 PDT 2016; root:xnu-3248.60.11~2/RELEASEX8664 x86_64 Install tools: gcloud Others: n/a What happened: Provisioned 3 clusters each in different regions (us-central1, asia-east1, europe-west1). Each cluster is in 2 zones (w/ gcloud --zone... --additional-zones...) Provisioned Federation API Server and Controller (1.4.4) based on Verified clusters were ready. Provisioned federated service & deployed backends into each of the cluster. Verified the federated services were created, and all of the regional LB were created, and verified all the LB IPs were working w/ direct requests. DNS provider updated DNS w/ most of the service/zone CNAME records. But A records did not complete. Further investigation for the log points to multiple entries in rrdata w/ the same IP due to the multi-zone deployment: I1029 19:24:24. 1 ] Have recordset <(clouddns) "buttonmasher-" type=CNAME rrdatas=["buttonmasher-"] ttl=180. Need recordset <(clouddns) "buttonmasher-" type=A rrdatas=["130.211.203.196" "130.211.203.196"] ttl=180I1029 19:24:24. 1 ] Existing recordset <(clouddns) "buttonmasher-" type=CNAME rrdatas=["buttonmasher-"] ttl=180is not equivalent to needed recordset <(clouddns) "buttonmasher-" type=A rrdatas=["130.211.203.196" "130.211.203.196"] ttl=180, removing existing and adding needed. After creating the clusters w/ only one zone, the DNS provider worked properly. What you expected to happen: Cloud DNS Provider should update the DNS record properly w/ multi-zone clusters. How to reproduce it (as minimally and precisely as possible): 3 x multi-zone clusters Federation API Server and Controller clusters in to federation federated service and backends Anything else do we need to know: Using GKE and Cloud DNS
/cc
Also hit this. Any workarounds?
Sorry, no direct workarounds available yet. You could convert your multi-zone clusters into multiple single-zone clusters, and federate all of those.
FYI, although I'm sure you're aware - also affects AWS multi AZ clusters (1.6.2)
This seems bad. I'll try to take a look into it.
Fixed by


I just found a cluster like this cc/
do you know if anyone else is currently working on a fix for this? I've already started looking into a fix, would you be willing to review my PR soon?
I don't know of anyone who is working on it.
(I see your PR, and I'll review it.)


https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
should this issue be moved into the 1.5 milestone and is it a release blocker? The test seems to flake fairly infrequently but we need to get the issue triaged as to whether it's a 1.5 release blocker. Please update this issue by 1PM PST 30 November 2016. Thanks! cc:
Since the feature it is testing is introduced for 1.5, so it should have the 1.5 milestone? This test is flaking because it is not well written, hence I don't think it is a release blocker. And I already sent out a which got LGTM before but recently rebased.
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
Thanks for triaging
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}


Is this a request for help? No What keywords did you search in Kubernetes issues before filing this one? http proxy, httpproxy, httpsproxy, AWS, kubeadm, kubectl, cloud-provider Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT Kubernetes version (use ): v1.4.4 Environment: Cloud provider or hardware configuration: kubeadm on EC2 instances on AWS OS (e.g. from /etc/os-release): Ubuntu 16.04.1 LTS Kernel (e.g. ): Linux ip-10-0-133-99 4.4.0-45-generic -Ubuntu SMP Wed Oct 19 14:12:37 UTC 2016 x8664 x8664 x8664 GNU/Linux Install tools: kubeadm Others: httpproxy, httpsproxy and noproxy set in /etc/environment, Docker config as well on startup scripts that run What happened: won't complete and gets stuck on it's first deployment. When one looks through the logs it shows that the pods are downloading fine but we are stuck due to the kube-controller-manager continually crashing as it can't contact AWS. Elsewhere in kubernetes I see references to and if the AWS cloud provider could use that it would be able to connect as well. From the POD if I manually set httpproxy and httpsproxy and no_proxy I am able to use the AWS CLI without any issues. What you expected to happen: I would expect to successfully complete and use the environment proxies set up on the system so as not to depend on a default Internet Gateway being connected to the VPC just for AWS access. How to reproduce it (as minimally and precisely as possible): Set up a public VPC with Squid set up as a proxy Set up a private VPC peered to the public VPC Set up a master node and run Set up a regular node and run will never fully initialize. If you remove the --cloud-provider aws` then it will complete and nodes will register. Anything else do we need to know:
what do you have in proxy variables ? (env | grep -i proxy) it tries to connect to (tcp 176.32.118.53:443) directly. You're expecting to connect there via proxy ? then do you have something like in your no_proxy ?
Unfortunately on the pod itself for the the environment variables aren't passed through. As this is created using I don't have control over the pod definition. So basically none of the *_proxy variables are set inside the pod in the docker image, however if I and export the variables and try to wget that URL it works. Basically I'm hoping we can update to respect and pass the environment variables down to the system pods if they exist, eg, add: Without it I can't stand up a cluster with AWS integration on a completely private VPC and for such a small change it would make a rather big difference.
ok, so request is actually to pass proxy variables into containers. interesting. I'll check how it can be done. BTW, for noproxy value in your example: things like 10.0.0.0/8 would not work on 99% of software that respects noproxy variable. it must be always exact IP match.
you're right that what I'm actually asking for is to pass proxy variables through either from a commandline setting on kubeadm or pick it up from /etc/environment or shell environment that launches kubeadm. I didn't realize it till I dug in to try and think about how one might go about fixing this. At first I thought we would just pick up the variables from the system environment but then I realized kube-controller-manager is deployed in a container and so we'd need to update the yaml file that launches it to contain the environment variables. As I'd like to go to production with this soon hopefully this is a small change that can be .
This issue was moved to kubernetes/kubeadm


Kubernetes version: HEAD What happened: Using interface to delete Job throws an error . What you expected to happen: Job should be deleted. We've found that issue after dependency update on dashboard: We are using to delete resources. It works fine for all other resources. I've investigated that and it appears to be the core issue. Other groups have registered as a known type. however batch is missing it After adding I was able to successfully delete Job using our code. I can quickly create PR for that. Just need a confirmation that this is indeed a bug. cc
+1 apis/autoscaling is similar
I thought we only needed to register it in the external versions (). I'll double check. cc
Registering api.DeleteOptions in batch looks fine. In the future, I think it's more proper to register DeleteOptions and ListOptions as unversioned, like how we treat .
DeleteOptions and ListOptions are expected to be in their group. They have to be in internal version and external version. Most of this code will be fixed in 1.6 (we'll standardize registering the "common" things). , Chao Xu wrote:
I've created PR to fix this issue. Could anyone check that?


Docker daemon complains "invalid hostname format" if the truncated host name ends with "." The truncation occurs .
Docker checks hostname with the regexp (See ): It means that the hostname must end with an alnum, not or . This issue has been here for a long time. This usually happens to static pod, because we compose a looooong name for static pod. We didn't hit this just because we are lucky. :P Will send a PR to fix this. /cc
: Needed for v1.5? please add milestone if so
Thanks for reminding. :) This is a bug fix in kubelet. Mark 1.5. /cc
Are you working on a fix?
Yeah, the fix is here


The error reporting functions here all repeat a URI without sanitizing it. An XSS attacker could use this to get the server to embed a reference to a malicious script. (As always, please note that kubernetes apiserver is NOT xss hardened and we do NOT recommend using the api directly from a browser.)
The error correctly sets the content type to , so this is only an issue using a browser that does content sniffing. Would it be sufficient to just add the header here?
Yes, that will work for browsers that honor the header. It's probably sufficient-- we could do that globally, we set Content-Type correctly everywhere, I think.


Failed: https://k8s- Run so broken it didn't make JUnit output!
Sen what is the plan for migrating these to bootstrap?
isn't it a manual run test?
I guess I'll move it over with empty cron-string?
Why did run this test? Please don't run large and enormous tests - this requires coordination w SREs etcd and may potentially break GCE.
I don't think we've run these test ourselves, someone ran it and failed(10 days ago) thus this test failure occurred.


Failed: https://k8s- Run so broken it didn't make JUnit output!
Bootstrap?
Why did run this test? Please don't run large and enormous tests - this requires coordination w SREs etcd and may potentially break GCE.
/close hummm I just see the index is -1


Kubernetes version (use ): 6 Environment: GCE What happened: I created a cluster for Heapster testing purposes to debug failing test in It turned out that Summary API does not export system/misc stats in SystemContainers section (though there are kubelet and docker stats). Everything works as expected on nodes. See also https://k8s-
What node OS were you using? We don't create the system (misc) container on systemd, so it's expected that GCI nodes don't report those stats.
The default configuration for 1.4, which is: master: GCI nodes: Container VM
Are the nodes running 1.4.6 too? There was a bug with this in 1.4.5, but it should have been fixed in 1.4.6.
Yes, both are running in 1.4.6. If you want to reproduce from Heapster current head run . It will take some time and as a side effect you should have a cluster with the mentioned problem. This was catch by integration tests and I temporary disabled the check here
if you have spare cycles, can you look into this?
friendly ping
Looking at it now
I have found a couple things so far. On the containerVM nodes, cadvisor is exporting a container with "name": "/system" On the GCI Master, cadvisor is exporting a container with "name": "" The has this: "{% set system_container = "--system-cgroups=/system" -%}" (/system is the default misc container name) So the real question is: Should cadvisor export "/system" instead of "" on GCI? Or should we be configuring GCI to use "--system-"?
I thought we fix this very issue for 1.4 branch by a while back. Are you running into the same issue with 1.5?
I think I told you the wrong thing earlier. We don't support the misc (/system) cgroup on systemd systems (I can't remember why, but it was an explicit decision - or might know). The is something different, I believe a base systemd cgroup (slice). As for the misc container on ContainerVM nodes, we've been testing for the misc container in 1.5+ in the summary api test (https://k8s-) for a while, and I haven't seen this issue. So if this is still reproducible, it's limited to 1.4. See this the bug was created for the post-fix.
Found the explanation here: The should use the flag to know the location that has the processes that are associated with , but it should not modify the cgroups of existing processes on the system during bootstrapping of the node. This is because is the on the host and it has not delegated authority to the to change how it manages .
IIUC this is intended behavior on systemd nodes. Based on the explanation provided, the kubelet doesnt have the authority to control "misc" processes on systemd.
+1. "misc" is just a bandaid for cvm. , David Ashpole <:
Ok, so I'm closing this issue.


https://k8s- Failed: [] Network Partition [Disruptive] [Slow] [] [ReplicationController] should eagerly create replacement pod during network partition when termination grace is non-zero {Kubernetes e2e suite}
https://k8s- seems to be the first failure here. It seems to have gone in with a bunch of RC/RS changes () cc/


Is this a BUG REPORT or FEATURE REQUEST? BUG REPORT Environment: Cloud provider or hardware configuration: GCE OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: What happened: $ kubectl describe pods test-ipj93 Name: test-ipj93 Namespace: default Node: kubernetes-minion-group-yxra/10.128.0.3 Start Time: Mon, 28 Nov 2016 14:39:28 -0800 Labels: <noneStatus: Pending IP: Controllers: <noneContainers: busybox: Container ID: Image: busybox Image ID: Port: Command: sh -c while true; do echo (hostname)-(cat /proc/stat cut -f 4 -d" "); sleep 30; done QoS Tier: cpu: Burstable memory: BestEffort Requests: cpu: 100m State: Waiting Reason: ContainerCreating Ready: False Restart Count: 0 Environment Variables: Conditions: Type Status Initialized True Ready False PodScheduled True Volumes: data: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: test ReadOnly: false default-token-kjoqg: Type: Secret (a volume populated by a Secret) SecretName: default-token-kjoqg Events: FirstSeen LastSeen Count From SubobjectPath Type Reason Message --------- -------- ----- ---- ------------- -------- ------ ------- 57m 57m 1 {default-scheduler } Normal Scheduled Successfully assigned test-ipj93 to kubernetes-minion-group-yxra 57m 57m 1 {kubelet kubernetes-minion-group-yxra} spec.containers{busybox} Normal Pulling pulling image "busybox" 57m 57m 1 {kubelet kubernetes-minion-group-yxra} spec.containers{busybox} Normal Pulled Successfully pulled image "busybox" 57m 57m 1 {kubelet kubernetes-minion-group-yxra} spec.containers{busybox} Normal Created Created container with docker id ; Security:[seccomp=unconfined] 57m 57m 1 {kubelet kubernetes-minion-group-yxra} spec.containers{busybox} Normal Started Started container with docker id 47m 47m 6 {kubelet kubernetes-minion-group-yxra} Warning FailedMount MountVolume.SetUp failed for volume "" (spec.Name: "default-token-kjoqg") pod "-b5bb-11e6-bab0-" (UID: "-b5bb-11e6-bab0-") with: Get https://kubernetes-master/api/v1/namespaces/default/secrets/default-token-kjoqg: dial tcp: lookup kubernetes-master on 169.254.169.254:53: no such host 46m 46m 1 {kubelet kubernetes-minion-group-yxra} Warning FailedMount MountVolume.SetUp failed for volume "" (spec.Name: "default-token-kjoqg") pod "-b5bb-11e6-bab0-" (UID: "-b5bb-11e6-bab0-") with: Get https://kubernetes-master/api/v1/namespaces/default/secrets/default-token-kjoqg: dial tcp 10.128.0.2:443: i/o timeout 45m 45m 1 {kubelet kubernetes-minion-group-yxra} Warning FailedMount MountVolume.SetUp failed for volume "" (spec.Name: "default-token-kjoqg") pod "-b5bb-11e6-bab0-" (UID: "-b5bb-11e6-bab0-") with: Get https://kubernetes-master/api/v1/namespaces/default/secrets/default-token-kjoqg: dial tcp 10.128.0.2:443: getsockopt: connection refused 11m 11m 2 {controller-manager } Warning FailedMount Failed to attach volume "test" on node "kubernetes-minion-group-yxra" with: error getting instance "kubernetes-minion-group-yxra" 9m 9m 1 {kubelet kubernetes-minion-group-yxra} Warning FailedSync Error syncing pod, skipping: network is not ready: [Kubenet does not have netConfig. This is most likely due to lack of PodCIDR] 9m 9m 2 {kubelet kubernetes-minion-group-yxra} Warning FailedSync Error syncing pod, skipping: network is not ready: [Kubenet does not have netConfig. This is most likely due to lack of PodCIDR] 6m 14s 4 {kubelet kubernetes-minion-group-yxra} Warning FailedMount Unable to mount volumes for pod "test-ipj93_default(-b5bb-11e6-bab0-)": timeout expired waiting for volumes to attach/mount for pod "test-ipj93"/"default". list of unattached/unmounted volumes=[data] 6m 14s 4 {kubelet kubernetes-minion-group-yxra} Warning FailedSync Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod "test-ipj93"/"default". list of unattached/unmounted volumes=[data] What you expected to happen: GCE PD to attach and mount successfully post the upgrade How to reproduce it (as minimally and precisely as possible): Running the upgrade steps in the doc for upgrade of manual test from 1.4 to 1.5 Anything else do we need to know:
Ran the upgrade again. This time the .
kube-controller-
Current theory is because kubelet deletes and recreates the node API object during upgrade (because externalID changed), see the new object does not contain the correct status (it is written by the Attach/Detach controller to the old object but not subseqently updated). If this proves to be the issue, the proposed fix is to force the attach/detach controller to update the node status when the node's external ID changes. is working to verify this theory by executing this code path (without a full upgrade). Once he has validated the theory, he will work on the fix, verify it, merge it to master, and get it cherry-picked to 1.5.
: is expected to fix this problem?
: Yes


Slack user "nalin" reported an issue Looking at the code this makes sense. Will fix.
For anyone following along, the fix for this bug unfortunately didn't get cherrypicked because of a process issue, and is broken until 1.5.2. cc
I just the new lines from As a temperatur fix ONLY. If anyone like me dont want to wait. Worked like at charm for AWS cluster setup. :)
A quick note, just as confirmation: the above temporary fix does resolve the issue for AWS - even though it's a change to a GCE-related file (cluster/gce/configure-). It seems that the GCE files are used for certain aws functions.
In based AWS deployment, the script is shared between AWS and GCE.
Also, this fix was cherry-picked into the branch a while ago, so instead of manually adding the lines, you can also just checkout the branch and run from there.


Is this a BUG REPORT or FEATURE REQUEST? (choose one): bug Kubernetes version (use ): master Environment: Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: What happened: When testing , I found kube-proxy can not record event with following errors. What you expected to happen: No errors How to reproduce it (as minimally and precisely as possible): cc
I am still getting this with the latest build
, thanks for your feedback, I'll check it.


kubectl edit is not working on HEAD. It's working on 1.5.


<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): no What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): subpath, emptyDir, mode, permissions Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-01-12T04:57:25Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-01-12T04:52:34Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"} Environment: Cloud provider or hardware configuration: Bare metal, SSD, but also VMs OS (e.g. from /etc/os-release): Debian GNU/Linux 8 (jessie) Kernel (e.g. ): 4.8.0-0.bpo.2-amd64 Install tools: Others: Docker v1.12.6 What happened: When I mount an volume using a subpath, I'm unable to write into it if my process is not running as . When I leave out the subpath, I can. What you expected to happen: I can write to the directory in both cases. How to reproduce it (as minimally and precisely as possible): Output is: Anything else we need to know: This issue is related to . I know this is caused by the Docker daemon creating those directories on a bind mount. Nevertheless, I expect Kubernetes to abstract this away, since this behavior is neither documented nor IMHO intuitive.
Could someone from please take a look? Thanks! cc:
cc/
attempted fix created here
I created a cherrypick PR for 1.6 here: since this bug is against 1.6. Could someone (who agrees this could be cherrypicked) please add the cherrypick-candidate label + milestone to the original PR , thank you! edit: sorry, it's against 1.5. Should it be cherrypicked back to that as well?
I'm still facing this issue with Kubernetes version 1.12.8 on a AKS cluster when I mount a Azure Storage as azureFile share. Did anyone know if the issue is fixed for all types of mounts and not only just the emptyDir?
same issue on GCP @ k8s 1.14
k8s 1.17 via eks as well
So what's the recommendation? Not using subPath?


getNodeConditionPredicate() in makes our code hard to understand because it hides the node-condition-based filtering in the node lister, which is totally non-obvious. We should get rid of this function and have NodeController add NoSchedule taints for these situations instead. (Alas, we might not actually be able to get rid of getNodeConditionPredicate() completely until we get rid of the Unschedulable field of PodSpec, but at least we can get rid of all the other code in this function.) If there are pods that should still be able to schedule in any of these situations (e.g. DaemonSet pods?) we should add tolerations in admission control for them (e.g. see ). cc/ cc/ Sub-tasks according to the design doc: [x] Add node taints label and feature flag () [x] In the node controller, taint Nodes according to the Node Condition () [x] In DaemonSet, update DaemonSetController to Tolerant new Taints () [x] In admissionController, add MemoryPressure/DiskPressure toleration for no BestEffort pod () [x] In scheduler, disable the current behaviour of filtering out Nodes. Instead, pods will not be scheduled to tainted nodes if a toleration does not exist in its PodSpec () [x] Add e2e test for this feature () [x] Update doc for this feature ()
I think you are referring Unschedulable field of NodeSpec?
Yes sorry, that was a typo.
- please CC me to all PRs
See also
Note that memory and disk pressure are checked in separate predicates, not in the NodeConditionPredicate. These should be represented as NoSchedule taints too.
Setting NoSchedule taints corresponding to the node conditions that today cause pods not to schedule onto a node, should also allow us to use tolerations rather than special-case code in the DaemonSet controller (and thereby make it easier to have DaemonSet pods be scheduled by the default scheduler.)
ref/
The task list was moved to the issue description .
/cc
/reopen
you can't re-open an issue/PR unless you authored it or you are assigned to it. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/reopen
/reopen
Currently we already have toleration on dsc for taint related to node problems (e.g NodeNotReady and NodeUnreachable). But to me ds should respect the new taints (outOfDisk, NodeMemoryPressure etc..), any insights ?
/reopen
Yes, it's one of sub-task of this project. This issue is closed by 'partially fixes' :(, I had updated PRs to 'part of #'.
- I can handle the dsc tolerations if you want. cc to keep you on the loop.
, sure, thanks very much :).
- just saw that you moved node conditions into predicate func, once adding node taiting and the according tolerations to dsc, do we'll still need to have node conditions checking ? (since the we're tainting by node condition). Also, correct me if I'm wrong, but from what I understood: dsc tolerates new taints with effect noExec since we don't wanna evict any already-running pods ?
We won't, but it'll take time for us to get there. We'll roll out Tainting as alpha (obviously), later move it to beta and GA in 1.10 at the earliest. We can't remove Condition checking logic before that. dsc is a daemon set right? New taints are NoSchedule only, so running Pods won't be affected (at least on the control plane level). Kubelet can still decide to evict them based on problem it observes.
- seems logic for the timeline. From sub-tasks Since new Taints are NoSchedule Only, Daemon set shouldn't have toleration for these ? It should instead respect the new taints when applied and not schedule on these nodes, or I'm I missing something ?
The task for DS in my mind: For all DS pods, it should tolerant MemoryPressure & DiskPressure For critical DS pods, it should tolerant OutOfDisk And if the taint by condition feature is enabled, we should check OutOfDisk by taints instead of predicates; the refactor is trying to make this simpler :).
- You're right, I'll send a PR after yours get merged :)
Do we have a to do list so folks can help out with I'd like to help when I get some time. I have a question about timeline. If alpha makes it into v1.8, will I be able to schedule to NotReady nodes (no CNI) if I reference the alpha tolerations? Or do you mean it won't be functional at all until v1.10?
It means you'll need to flip feature gate flag for it to work.
, here is the task list: I'm working on related PRs for them; please ping me when you got time :).
- i anticipate we will also add a CPUPressure condition which will also mean that no BestEffort pod can be scheduled to that node if the kubelet is running with a static cpu assignment policy. /cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
I think it's done (as alpha). - do we want to keep this issue open?
We can close this.