https://k8s- Pod has a restart count 1 FYI, all relevant messages about this pod in the From the sync loop timeline, some containers died. Check the log again: The sandbox was GC'd right after it was created. This led to kubelet killing all the containers in the pod since the sandbox needed to be recreated. Need to check whether this was caused by the container state emulated by fakedockerclient, or by a real bug. /cc
/cc FYI.
kuberuntime GC routine has a minAge protection such that it can't GC any pod sandbox that was created no more than 30 seconds ago). This doesn't seem to have be respected from the log. I suspect that the fakedockerclient was not returning a proper timestamp.
GC logic might be faulty. I see a similar failure in https://k8s- EDIT: The node name is The sandbox was created and removed within 2 seconds
- by looking at the logs, can you tell if there's an easy repro?
So given that this happens also in non-kubemark suites, it seems like a release blocker to me. Anyway - - do you think it's releated to number of pod on a node, or it's just that it happens from time to time, and in scalability suites (no matter if it's kubemark or not) we have more of those tries (due to scale)?
I think the timestamp comparison logic might be wrong in the sandbox garbage collector. This would only happen (in my theory) when the the snapshot of all containers/sandboxes on the node are taken when: sandbox is created but not running (started) yet. other containers in the pod haven't not been created. In this case, the fail-safe (minAge) kicks in, and apparently failed in the tests discussed above. This would more likely to happen when during batch creation of more pods. kubelet would usually heal itself by restarting all container in the pod (which has the undesirable side-effect of killing the user containers of course). I think other tests might not catch this because 1) they don't perform large batch creation, and 2) they don't check the restart count.
Oh the above comment was completely based on my theory. Reproducing this deterministically on a working node may need some hacking in the kubelet code (e.g., inject a long enough sleep after sandbox creation). I will try writing a unit test first to verify the behavior.
Can't reproduce this in my cluster with artificially injected latency... Adding more debug information in For people who are interested in checking the logs and identify the victim pod, here are the instructions: Download the artifacts: E.g., Find the (density) pod with restart count 0:
Didn't reproduce the problem either. I think so. GC can't distinguish just-created sandboxes from failed sandboxes. Especially when the time from sandbox creation to ready is longer than GC's minAge. I'm wondering should runtime (e.g. dockershim) filter out those created-but-not-ready sandboxes from sandbox list? or introduce another sandbox state in CRI?
has been merged 11 hours ago and scalability suite hasn't flaked since. Will monitor and see if there is any need to reopen.
OK. But because , there are still potential problems under big pressure.
First, this issue is no longer a release blocker after the fix. Let's open a separate issue to discuss potential improvement, instead of commenting on a closed issue :) Also, FWIW, we used the same logic in the old docker manager, so this is also not a regression.
Agreed. Opened for this.


<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): Is this a BUG REPORT or FEATURE REQUEST? (choose one): <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --BUG REPORT Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"6+", GitVersion:"v1.6.0-beta.3", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-03-11T04:20:22Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.4", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-03-07T23:34:32Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"} Environment: Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: What happened: With kubectl version 1.6.0-beta.3, the output of is not aligned: What you expected to happen: With kubectl version 1.5.4, the output is as follows: How to reproduce it (as minimally and precisely as possible): Run using kubectl version 1.6.0-beta.3. Anything else we need to know:
will take a look
any progress on this?
hi, tony, sorry, I haven't look at this problem yet, maybe tomorrow, just really busy recent day, you can take it if you already find the problem, thanks
Happened to see the same issue on my laptop, I can help fix this issue if you do not have time, thx
thanks, I've been really busy this week, go ahead
for this ticket, want to discuss one more thing with you: at present, for each of the update event, command with option will print out a line showing the status change of this event. But this behavior is different with what we saw from the output of linux's command, which I think is not a very good user experience. For this kind of command, we shall reset all the old output and re-print all the pod's information in the standard output, this will make this command more like a real command. do we have any plans on this right now? welcome your ideas on this.
I think both output format has it's advantage, for current output format, user can copy paste the whole information like checking logs, for linux's command style output which user can stare on only one screen(if it's not too many pods or sth else).
yep, we do have plan to do this, and I really like a more linux style command, you can refer for more detail. Anyway let's just get this fixed at first.


This is in 1.6.0-beta.4: Some context: The line number suggests it is here: I think this is because dockershim now uses a protobuf object and Protocol is now an int. And is allowed in go (apparently) - is "A". And the comedic error is because Protocol implements If this is the error, there's something fishy going on because all our UDP tests should fail?
Aha ... I think it is only UDP hostports that would be "broken" by the issue. I wouldn't be surprised if we don't rely on any, and that we might not have any test coverage. I don't think this is a regression - I think this code has been there since the start of dockershim, but maybe CRI is now the default? I don't know how widely UDP hostports are used. Given how close we are to the planned release date, cc and
There is something wrong in dockershim's protocol parsing. Will file an PR to fix this. /assgin
/assign
Opened to fix this.
thanks for filing the issue. Thanks for quickly addressing the bug. We need node e2e tests to capture the kind of regression at the first place.


From https://k8s- I've tried to reproduce locally in an endless loop but no failures so far. cc
Investigating.


Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG Kubernetes version (use ): k8s1.3.5. still exist in 1.6 What happened: when exec threadSafeMap.Index func, the returned items contain nil object. That is to say, index is in threadSafeMap but the corresponding has already been deleted. What you expected to happen: when object is deleted from threadSafeMap, all its index must be deleted. How to reproduce it (as minimally and precisely as possible): Suppose we init threadSafeMap with two indexers: indexFunc1 and indexFunc2. Obviously, indexFunc may return error if the object has something wrong or some other reasons. when we add an object, will first add the object to , then compute the index and add the index into threadSafe.indices. when we delete the object, will first delete the index from indices, then delete it from Suppose IndexFunc1(object) return error and indexFunc2(object) works well. I think this circumstance is absolutely possible to happen. Also we know that the order to exec indexFunc is not decided. Feature of iterating map. When we add object, suppose first exec indexFunc2, and then exec indexFunc1. So the index of indexFunc2 is to indices. But the index of indexFunc1 is not , because indexFunc1(obj) return err. Then we delete the object, suppose first exec indexFunc1. It will return error and IndexFunc2 has no chance to run. So the corresponding index indexFunc2(obj) can't be removed. But then the object is removed from items. This problem may result in subtle problems. For example, when we exec threadSafeMap.Index( ) func the returned list may include nil object. Anything else we need to know The reason is that index is still is threadSafeMap but the corresponding object has already been removed. We must clear all corresponding indexs when delete object. And the reason why index is not deleted is that when for-range map, the order is not decided.
Who can help me? Do i miss something important or misunderstand it ?
, can you share a unit test to show it? I'd like to help on it.
Thank you for your help. I think i have decribed the condition resulting in the bug indetail. Be simple, because the order of exec indexFunc is random when update indices, and some function don't care the error returned, the obj is deleted but corresponding indices is still in it. NOTE: the obj is deleted but corresponding indices is still in it So when we run index func, the returned list may contain nil object. Because when get obj from threadsafemap with indice, not care whether the obj exist in this map. Then nil object .
. I am sure this problem exists. And i think we can find a better way to deal with it.
/sig api-machinery threadSafeMap is in client-go
/kind bug
Opened a PR with a fix.
In my understanding, should be deterministic, could you elaborate more how your IndexFunc implemented? If an object can not be called on this , then the question is, how could it exist in at the first beginning?
threadSafeStore should tolerate some bad conditions and deal with it gracefully, instead generating strange and confusing phenomenon. ThreadSafeStore can contain mutiple index funcs. If one of them is not implemented very well, or one of objects we pass to it has something wrong, STRANGE AND CONFUSING phenomenon happens, see what I said above.
That's the point I would like to know more, the call on an object should succeed or fail unconditionally(deterministic).
wechat
I also think index function must be a pure function. See my comment on the pr


I was confused by the that looks like more as machine readable rather than as human readable. I feel that UX could be better if it will be written in multiple lines and without curly braces/quotes:
Agreed on this one, we should pretty-print this output.
I would like to change that tomorrow if you are not working on it.
I'm not working on it, so feel free to provide a PR. But I wouldn't do anything until we get approval from
cc don't some of the tests rely on this output?
They probably do. There should be a option or something that we can use in test infra.
please have a look at: and related issue: The above is approved and had an lgtm labeled, but removed by the bot because of testing flake.
I think we can close this ?
I don't see that it was fixed. Does the output of was improved?
kubectl now has a option which lets you select between YAML or JSON. However, I believe this is reversed. The more human-friendly output is hidden behind an option and the machine-friendly output is the default. I believe it should be the opposite.


( noticed this) HardPodAffinitySymmetricWeight should be in the , not in KubeSchedulerConfiguration If/when fixing this, we need to be careful to do it in a backward-compatilble way.
Do you mean ? It seems scheduler policy config's args can not passed to FitPredicates; it need to update related interface. For the : mark to be deprecated and set scheduler policy config to HardPodAffinitySymmetricWeight if policy config is empty.
Are you working on this? if no, I can take it.
, nop; please go ahead.
/ / / , When reviewing the PR, there's one question on : the default value of is 1, which means the weight (1) will always add to priorities if term matched (), does it our expectation? And the user has to set it to 0 to disable it. IMO, the default value of should be 0.
The value 1 (instead of 0) was chosen so that if you say A must run with B, there is an automatic preference that B should run with A. This seems like reasonable behavior.
Thanks very much; reasonable case to me. But it seems we can not disable it: if set to 0, the default funcs will set it to default (1); similar result to un-set. Anyway, no user complain this now :).


<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): build--5-v1.7.5-3 Environment: Cloud provider or hardware configuration: Laptop (bare metal) OS (e.g. from /etc/os-release): Fedora 26 Kernel (e.g. ): Linux localhost.localdomain 4.11.0-0.rc6.git0.1.fc26.x8664 SMP Mon Apr 10 19:45:50 UTC 2017 x8664 x8664 x8664 GNU/Linux Install tools: make && hack/local-cluster- -o _output/bin What happened: built from and ran with local-cluster- All commands gave the following error: Running showed the output: What you expected to happen: for commands to work, and Content-type to show How to reproduce it (as minimally and precisely as possible):
ccing
I see the text plain content type, but not kubectl errors
i was using the wrong version of kubectl. updated to new and i'm not getting an error any longer.
Text/plain coming back from the server is still an issue
cc
I'm not seeing against head and the pastebin is gone. Did you reproduce?
I did last week. Insecure and secured port behaved differently
You can reproduce this using against a Kubernetes cluster built from HEAD
Hi all, I think we could use write Content-type correctly. I have made a PR to fix this.


Currently if one wants to view secrets stored in k8s they must follow the process described here: This involves copy&pasting. It seems to me that kubectl could provide a more helpful interface to base64 decrypt the users on behalf of the user saving them the trouble.
The kubelet does decode secrets before exposing them to the pod. Also the title doesn't really make sense. Secrets aren't encrypted so there's nothing to decrypt.
/kind feature
Yes, though this is low priority since we mostly don't want end users to view these once they get into the system.
Why can't that be an API permissions issue? I mean, if I really want to see it, I can just launch a pod to tell me.
I've update the issue title with what I understand as the request. To the best of my understanding this is just a request for us to avoid the need to pipe into base64 when inspecting secrets.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
I would like to try to make this work before it rots. cc Any guidance on a preferred implementation?
What do you think about adding support for functions in templates? This would allow someone to use the existing and a template that uses the function -- or a wide variety of other interesting functions as a side effect. I believe (but have not yet tested) that the following might be sufficient to enable the requested behaviour (plus the change to the dependencies list, of course):
Hi I suppose my thoughts on this issue have changed a little since I wrote it. The situation isn't quite as bad as I described as it is possible to decode a secret in one go without then need to copy and paste. i.e. That said, that is a bit of a mouthful. Also, it might be nice to more easily 'work' with individual secrets and set them as well as get them. Should a user really need to pull down an entire group of secrets and edit a file just to be able to update a single password? For example something like this might be possible: What do people think? PS: Should this have been an issue on [kubectl] instead? PPS: I think some of the ugliness of this comes from configmaps being very similar to secrets except for the base64 encoding. Have there been any thoughts around bringing them into line? If memory serves, it might actually be nicer, due to YAML syntax-shenanigans, if configmaps were base64 encoded too.) [kubectl]:
Hi ... yes, I was thinking that it would be handy to be able to do something like: which is essentially the same as what you did without the overhead of the base64 process, or embed the information in a larger template like or in theory be able to use any of the other Sprig functions if someone found them useful. I think that having the Sprig methods available for generic formatting would just be useful on its own, really. On the topic of a or a , that seems pretty cool. There seem to be quite a lot of options for that I think people might expect to see mirrored in a if it were to exist. Thoughts?
It's also come to my attention that works great on MacOS but on Linux it's , so if someone, say, has a Helm chart where the notes tell the user to retrieve the secret data using one or the other of these, it won't work on the other one. would clearly be a simpler experience for the user than Hmm... thinking further: does does the idea conflict with the documentation in ? Perhaps we'd have to do something like: or similar?
/remove-lifecycle rotten
Small tip: you can use which works on both Linux and Mac.
Thank you
Update: I've looked at the huge amount of additional code that would be imported as a result of importing sprig and decided to go with a much smaller change instead, which is simply to implement a template function using . See PR with code and test.
please see the linked PR which addresses the original issue in a minimal fashion -- it simply provides a template function. It has passed all the tests and needs review; if it's good enough to solve the original problem I'm happy to squash the commits and can hope to get it merged. As far as solving evolved description of the problem, I'm not sure where to start or if you really want me to go in that direction. Would you consider taking this change and splitting the additional work for being able to get / set the contents of an individual key in a secret into a separate issue? If you have any guidance on what you'd like to see with that I'd be happy to take a look, but it might be out of my current grasp.


<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): Is this a BUG REPORT or FEATURE REQUEST? (choose one): <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): master Environment: Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: What happened: Check , it is still using , we should move it to . What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know:
/assign
probably will want to move the init container also to the PodSpec
I think the init container still depend on annotations , can I specify in ? If so, I want to file another issue to track it
FYI Init containers have graduated to GA and now appear as a field. The beta annotation value will still be respected and overrides the field value. (,
Seems like that the doc should be updated. EDIT: NOTE: This feature is beta in Kubernetes 1.5. Once the feature exits beta, Init Containers will be specified in the PodSpec alongside the app containers array.
Thanks I filed an issue here
What about merge this first and then handle init-container in a separate PR?
Sounds good to me, thanks.


I'm seeing the following in my logs: There was an errors, and we silently skipped it.
Good catch! Thanks for pointing.


Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG OR Do i miss something important? Kubernetes version (use ): k8s 1.3.5. still exists in 1.6.2 What happened: HA of kube-controller-manager and kube-scheduler is achieved by create endpoint and update it periodically. However, endpoint controller deletes endpoints without corresponding services which is implemented in checkLeftoverEndpoints func: That will result in leader transition. What you expected to happen: checkLeftoverEndpoints func shouldn't delete endpoints used by LeaderElection. How to reproduce it (as minimally and precisely as possible): turn on LeaderElection for kube-controller-manager and kube-scheduler. After five minutes, endpoint controller will delete orphan endpoints. Anything else we need to know: we can add some logic in checkLeftoverEndpoints, for example, not add endpoint in kube-system namespace to queue.
Fixed by
Thanks


Found in I can reproduce locally using
/assign (via git blame)
cc
Thanks. I will take a look.
Got a fix locally. Testing it.


Is this a BUG REPORT or FEATURE REQUEST? (choose one): bug Kubernetes version (use ): Environment: Cloud provider or hardware configuration: N/A OS (e.g. from /etc/os-release): RHE 7.2 Kernel (e.g. ): What happened: The node resource is empty when start local cluster by . What you expected to happen: There's one node in this local cluster. How to reproduce it (as minimally and precisely as possible):
Just check the script/code, it seems kubelet will not register until dynamic config is there ( enabled all alpha features by default).
I met same problem
What do you mean "dynamic config" ?
I think there's a PR that make kubelet can update configuration dynamically, similar to what we had done in scheduler. But did not find PR :(. I'm checking the code.
Should be this one: /cc /sig node
that shouldn't be the case. As long as you can get a client and a nodeName you should be able to register. What specific error are you seeing that suggests it's dynamic config? Also why does local-up-cluster enable all alpha features by default? That seems wrong to me.
Not sure why, but it really does:
I think this should be caused by /cc
, sorry for the disturbing :). Regarding all alpha features, I prefer to enable is by default: local-up-cluster is a kind of test environment for developer, enabling all alpha features maybe helpful to catch up potential feature interactions.
I would not expect all alpha features on by default in local up cluster
Neither would I. Developers, by default, shouldn't have to wonder if bugs are their fault or if they come from unstable alpha components. Users, by default, shouldn't have to deal with unstable alpha features either.
/ , got that; alpha components is unstable, do NOT cover everything for it.
met same problem


We have , but there isn't a label in the k/k repo. There is only . Do we need to add a label or rename to ?
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list .
/sig ui :smiley:
/sig contributor-experience
/cc
I just it. Fixed?
Yeah. BTW, the page has a bad md format. It also should be fixed.
Feel free to send a PR to fix :)
As Piotr is the owner, I thought let him fix:) If it's Okay, I will do. Thanks Christoph!
anyone can contribute to the kubernetes/community repo too :) the PR would be reviewed like any other (usually the sig-lead for that sig would review)
Okay:)
Yep just fix things when you can. That's just faster :)
FWIW this lead me to open


<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a BUG REPORT or FEATURE REQUEST? (choose one): FEATURE REQUEST <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Environment: Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: What happened: the psp admission controller iterates through applicable psps for both the user and the service account. This leads to at least one RBAC DENY message for each pod. What you expected to happen: "RBAC DENY" should be a more meaningful message How to reproduce it (as minimally and precisely as possible): enable psp w/rbac
closing in favor of


5 tests failed, 6 jobs failed, 151 builds failed. Failure stats cover 1 day time range '2 Jun 2017 20:57 UTC' to '3 Jun 2017 20:57 UTC'. <details<summaryRationale for assignments:</summary </details<br
/unassign I am probably not the right owner for these tests. Leaving this to to triage it.
This seems pretty likely a quota issue. Many of them started after the cleanuped the old gce resources. Probably some tests are leaking resources. Though it doesn't occur on other CIs like . I will dig into it a bit more.
The root cause is ~2 years old from We are parsing human readable gcloud output, which recently changed. Additionally, when we fail to create the LB we leak the static IP address. It's failing on GKE but not GCE because GKE uses a pre-release version of gcloud as a canary.
/priority failing-test
/reopen ci-kubernetes-e2e-gci-gke-slow-release-1.5 and ci-kubernetes-e2e-gke-slow-release-1.5 will continue failing until we cherry pick the fix. Reopen for now.
Forgot to update: ci-kubernetes-e2e-gci-gke-slow starts passing after was merged.
Can you cherrypick to both 1.5 and 1.6 branch? But this issue is not 1.7 blocking issue anymore.
Sure, working on it.
Sent cherry picks: release-1.6: release-1.5: release-1.4: Cherrypicking the fix into 1.4 as well because we also have gce-slow-1.4 and gci-slow-1.4 CIs running.
remove the milestone, this is not a 1.7 issue
Since all issues listed here are addressed separately. Close this one.


We should just use GOFLAGS
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
/sig contributor-experience
/assign


There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
/sig azure
fixed by
/label official-cve-feed (Related to kubernetes/sig-security)


<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: BUG REPORT What happened: When using IPv6 addresses, the kubelet passes the address to client-go, which does not correctly format the address for use in API requests. For example: Failed to list v1.Pod: Get https://2001:129::136:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dnode136&resourceVersi on=0: dial tcp: too many colons in address 2001:129::136:6443 What you expected to happen: The URI should have https://[2001:129::136]:6443, so that SplitHostPort() will correctly be able to separate the host and port components. How to reproduce it (as minimally and precisely as possible): Create cluster using IPv6 address (e.g. kubeadm init --apiserver-advertise-address=2001:129::136). When the kubelet starts up, it'll log this error. Anything else we need to know?: This is being handled as part of effort to support IPv6 in Kubernetes. Was found by a co-worker, I'll work on a commit to fix. Environment: Kubernetes version (use ): Using latest, via KubeAdm Cloud provider or hardware configuration*: bare-metal OS (e.g. from /etc/os-release): Ubuntu 16.04 Kernel (e.g. ): 4.4.0-81-generic Install tools: Using KubeAdm Others:
/area client-libraries /area kubelet /area ipv6
/assign
The point of failure is here: SplitHostPort will accept IPv4 and IPv6 addresses, but the latter must have square brackets around the IPv6 address.
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <bre.g., for API Machinery<br(2) specifying the label manually: <bre.g., for sig/scalability<br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
/sig api-machinery because client-go
Should we open this on the repository?
I see in the client-go repo read me that it states: Does that confirm that I should do a PR here?
One thing that I was wonder, was whether or not the hostname/address provided in the call to getListener, has been validated. If so, and it can only be a hostname, IPv4, or IPv6 address as a string, we could just check if there is a ":" in the name to determine IPv6. Otherwise, I could use net.ParseIP(), and net.To4() to determine if IPv6. I'll start working with the latter assumption, but please let me know, if the former could be used.
Also, I'm making the assumption that we'd want to make client-go be able to handle IPv6 addresses, rather than having each user of client-go (e.g. kubelet) from ensuring that the IP address is of the right form. IOW, have client-go add the needed [] around IPv6 addresses, rather than have kubelet do that. My thought is that there may be cases where we want just the IP address and not the modified IP. Does that sound OK?


We want kubemark setups to be as similar to real clusters as possible. Since we are launching kube-aggregator in real clusters, we should add a possibility (and probably also enable by default) to have kube-aggregator in front of apiserver(s).
was working on being able to use it in e2es.
With kube-aggregator integrated into kube-apiserver, is this still relevant?
There are a few flags which need to be set up for kubemark to actually be properly testing with the aggregator. I am digging around in that code this week anyway. I will try to get to it and make sure its set up properly.
Thanks for looking. Let me know if I you need something.
[MILESTONENOTIFIER] Milestone Labels Incomplete Action required: This issue requires label changes. If the required changes are not made within 3 days, the issue will be moved out of the v1.8 milestone. kind: Must specify at most one of ['kind/bug', 'kind/feature', 'kind/cleanup']. priority: Must specify at most one of ['priority/critical-urgent', 'priority/important-soon', 'priority/important-longterm']. <detailsAdditional instructions available <a href=""here</a</details


It looks like a test configuration failure:
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <bre.g., for API Machinery<br(2) specifying the label manually: <bre.g., for sig/scalability<br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
/assign
last passing run: https://k8s- first failing run: https://k8s-
https://k8s- /close


/kind bug What happened: We constantly log message like at V(2) that don't tell any useful information.


<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --BUG REPORT: What happened: I'm running a simple Kubernetes cluster with master and one node. All basic PODs are deploying successfully (ie. weave-net) but when I try to deploy GlusterFS as a POD - container creation fails with error: This is a result of failing to perform from docker file. The weird part is that if I run this image directly on node through docker it runs smoothly. What you expected to happen: I'd expect to be able to deploy GlusterFS through Kubernetess properly. How to reproduce it (as minimally and precisely as possible): Setup minimal cluster and try to deploy the follwing POD Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration**: virtualbox OS (e.g. from /etc/os-release):Kernel (e.g. ):Docker versionInstall tools: vagrant Others:
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig node
Reiterating the mentions to trigger a notification: <detailsIn response to : /sig node Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
One update - I can deploy this POD with no problems on version of k8s.
I am hitting the same bug on a freshly installed k8s cluster running on installed via kubeadm plus weave-net with a currently unsupported docker version. For me, downgrading to a supported docker version helped to resolve this issue (v1.12):
- Thanks for the info! - Well we can confirm that kubernetes is not working properly with Docker 1.29 and 1.3
Reiterating the mentions to trigger a notification: <detailsIn response to : - Thanks for the info! - Well we can confirm that kubernetes is not working properly with Docker 1.29 and 1.3 Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This is not really a bug since kubernetes doesn't support the docker version you use yet, but feel free to send a patch if you find the problem.
I was thinking about your comment and I think it is still a bug since downgrading Kubernetes to 1.6 fixes the problem while having the same setup as described above. So something have changed in 1.7. But yeah. Will try to investigate on my own.
I'm actually getting this same error with 1.7.1 and docker 1.13.1. Anyone have other thoughts on what the issue might be here? I found this so question () but it looks like everything is configured correctly.
In kubernetes 1.7, under pid # 1, there is a "/ pause" process, and since init takes pid 1, it is no longer available. In earlier versions of kubernetes, pid # 1 was free . PID USER TIME COMMAND 1 root 0:00 /pause 7 root 3:41 /tiller Kubernetes now shares a single PID namespace among all containers in a pod when running with docker = 1.13.1. This means processes can now signal processes in other containers in a pod, but it also means that the pattern will cause the pod to be restarted rather than a single container.
I can work around the problem using the option in the kubelet, however this not the desired solution of course.
just FYI, looks like some images have to be run as PID 1.
In 1.7 release, we enabled shared pid namespace if the docker version is docker 1.13 and beyond for debug pod feature. Do anyone know why those images, such as GlusterFS have to run as PID 1? we need to figure out how to resolve this conflict with shared pid namespace. If we cannot, we have to disable that setting for 1.8 since we scheduled to support docker 1.13 in 1.8 timeframe. I am currently assigning you this bug to come up a solution.
cc/
gluster/gluster-centos is a container image that runs systemd so that it can run ntpd, crond, gssproxy, glusterd & sshd inside a single container. This isn't how Kubernetes is intended to be used and I don't know how much we should go out of our way to support it (this is a question for & seems to be a good write up of the challenges of running systemd inside of docker. It also prevents Kubernetes from doing process management like: reporting state of processes signaling individual processes for graceful restart ingesting and display daemon output (to stdout/err) Ideally we could change this many-processes-in-a-container to be many-containers-in-a-pod, something like this: This doesn't need a privileged container and doesn't run sshd or ntpd, neither of which are needed in Kubernetes. This also doesn't run crond, but that may actually be needed by gluster. I've never used gluster. (I used flags so I didn't have to investigate how to make these processes log to stdout) If you must run systemd in a container, adding this to the container config will get systemd to run: This tells systemd to run in system mode even though it's not PID 1 and disables its chroot detection (which is unrelated to PID 1 but checked when invoked as "systemd", I guess).
afaict, running in a container is well recommended ( and ) at certain scenarios and a common practice. iic, there are enough containers who make use of as its . If shared namespace which introduced in 1.7 breaks this, I believe its a backward compatibility break :(. I would invite here to share his thoughts as well. :)
Those articles are about how to work around shortcomings in vanilla docker that Kubernetes solves natively: resource sharing via pods, containing via SELinux, reaping orphaned zombies, etc. My earlier question was along the lines of how much effort should go into supporting use cases that are not using Kubernetes as intended. (I'm actually asking, not trying to make a point.) Does Kubernetes commit to continuing support for every container image that's ever run on Kubernetes? For how long? Indefinitely? or can we change it over the course of multiple releases? Anyway, this change is compatible with running an init system in a container. The problem is with images that assume they will only ever have , which has so far been the case with docker due to a technical limitation, but it was never the case with other runtimes such as rkt, hyperv, etc. Brainstorming other options: shared pid namespace and require users of systemd to set an environment variable in their Kubernetes config (or port the multi-process container to a multi-container pod). shared pid namespace and enable it in a future release where we have the same problem. a solution where pid namespace is configurable per-pod, but not supported indefinitely. It would probably be implemented using annotations, and we will have to modify the Container Runtime Interface to support it. Over the course of multiple releases the default behavior would change but could be reverted based on the presence of an annotation. In a future release, users will have to change an annotation in their config to get the old behavior (rather than the environment variable). In a future future release, the annotation will stop working and users will have to set the environment variable. to supporting per-pod shared vs isolated pid namespaces indefinitely and adding it to the Kubernetes API. It doesn't make sense in some runtimes, so we probably can't require all runtimes to implement it, so it'll likely only see use with Docker. Last time we discussed it in SIG Node, and didn't want to support isolated pid namespaces indefinitely.
Yes running systemd in a container is a critical need for us. There are lots of situations where it is and will be used. Biggest use case if for people simply moving current workloads that run in a VM into a container. I have a service that runs fun under systemd in a VM. If I can simply take the init script and move it whole sale into a container, why wouldn't I want to do this. It's simple and does require a great deal of knowledge by the end user. service containers. Not every project can easily be moved into microservices. But they probably can run inside of containers. If we drop support for systemd running inside of containers, then we loose a big group of users. Getting access to journald information inside of the container. Applications that write to syslog and directly to the journal. Currently in container world without systemd, this information is dropped on the floor. if I run systemd and journald, I can actually retrieve and use this information.
If the intension is to force all containers inside of a POD to run with the same PID Namespace? I think this is a mistake since it eliminates another really cool use case. I would love to be able to run two containers inside of a pod, and INIT container with escallated privileges and a locked down container, with no privileges. I want to make sure the locked down container can not see the processes of the privileged container. An example of this, might be a container that loads a kernel module that is needed by the locked down container. Another use case would be a buildah container. The INIT container mounts up a COW file system onto a directory on a location that the locked down container can write, The locked down container writes out the data for his image, when the container completes, the INIT container commits the data to the repo. I believe allowing different containers inside of a POD to run with different security and isolated PIDs should not be removed.
none of the things you've mentioned are removed by this change, save using docker to hide other processes in the pod, and I'm not sure why that's a requirement. Is anyone currently using the "privileged sidecar" pattern you describe or is it hypothetical? It just occurred to me that an interesting compromise here might be to only enable shared pid for multi-container pods. This would enable pod semantics for PID when it would make a difference while drop-in docker monoliths keep the PID 1 behavior they're expecting in docker. I'd have to double check, but I'm pretty sure that behavior could be implemented entirely in the dockershim with no API or CRI changes.
Hypothetical, but I am pushing it for a solution for problems I hear from users fairly frequently, where then need big privileges at start and stop of the pod, but not for the time is is available to the network.
ok, thanks. ftr, I think we should continue to support this pattern as well, and I don't think a shared PID namespace prohibits it. I will bring up this issue at Tuesday's sig node meetup if you'd like to join us. Does the config I mentioned in fix the issue with systemd?
Copying from a comment made elsewhere:
Update from SIG Node Meetup: Everyone wants to support these use cases, but we're not yet sure the requirements we want to support medium to long term. is going to sync up with and then we'll figure out what we want to support in a future release. For 1.8 the current plan is to leave docker shared pid namespace enabled by default. Cluster administrators can disable it per-node with the --docker-disable-shared-pid flag. I'll see about whether we should add something back to the release notes.
This seems somewhat backwards. Wouldn't it make more sense to go back to supporting systemd use cases by default NOW (by making the shared PID namespace behavior optional or something) until a final decision is reached and can be advertised to current users?
absolutely ! thank you very much
I think that is the best solution is to make the PID Namespace sharing optional.
- optional at what scope - cluster, node, pod. let's sync up before next sig-node and we can revisit if need be before 1.8 goes out. The main question right now is if pid namespace sharing is opt-in or opt-out. We need to weigh pro/cons of both.
I would expect it at the POD, since pods would have different requirements, not sure why you would want to have this at the cluster or node level.
yes it works :)
awesome, thanks! I'd like to better understand the requirements of people objecting to this change ( I think there's a bit of confusion about what has changed, so let's start with some facts: This container image runs on Kubernetes without changes to the image, but it requires (non-obvious) pod configuration. is nothing inherently incompatible between a shared pid namespace and a process manager like systemd. systemd and other init systems work with pid != 1, but may require special invocation. in this issue is specific to Kubernetes. That is, this container image wouldn't work with vanilla docker PID namespace sharing, either. change does not affect securityContext I can think of a few things that are objectionable about the change, but let's state them as requirements. For example, are these requirements for anyone? Kubernetes must run all containers published on docker hub with no additional pod configuration Kubernetes must run all systemd containers, regardless of how they are built, with zero additional pod configuration. Kubernetes must run this specific container image with zero additional configuration Kubernetes must officially support isolated PID namespaces between containers in a single pod across all runtimes (docker, rkt, hyperv, etc). Configuration changes are ok, but the flag to systemd cannot be used because the documentation says its not useful. I want to use systemd as intended. Rather than having Kubernetes work around this container image, should we patch the Dockerfile to make it compatible with even the native docker shared-pid, unrelated to Kubernetes?
Configuration changes are ok, but the --system flag to systemd cannot be used because the documentation says its not useful. I want to use systemd as intended. We can ask the systemd guys why they recommend never using the --system flag, but I would guess one of the reasons is that processes showing up that were not created in the ancestry of systemd, could cause confusion. I would also guess that running more then one container with systemd --system would not be supported. Kubernetes must officially support isolated PID namespaces between containers in a single pod. This is required so that you can run different containers with different security constraints in the same pod at the same time. Your last comment makes the seem like it is the default, when actually it is the exception. almost no containers run with shared-pid normally. I guess in a side car container, it makes sense to support this scenario, but forcing this as the ONLY way to run containers is limiting.
This is the part I don't understand. Why? What does PID visibility have to do with security constraints? What am I missing? nor do they run with a shared network namespace, but we impose that on docker containers because that's the pod model.
An update from the last SIG Node: we'll disable this by default in 1.8 and revisit for 1.9 I'll use to track that.
if I have a privileged container process in the same pid namespace as a pid in a non priv container, then the container process in the non priv can attack the priv process. Having them in separate pid namespaces gives you better security. Bottom line, you have a potential change where containers that expect to be run as PID 1, whether legitimately or not, could break when you make this change. Kubernetes in my opinion should make it easy to opt out of this behavior.
/priority critical-urgent
[MILESTONENOTIFIER] Milestone Labels Complete Issue label settings: sig/node: Issue will be escalated to these SIGs if needed. priority/critical-urgent: Never automatically move out of a release milestone; continually escalate to contributor and SIG through all available channels. kind/bug: Fixes a bug discovered during the current release. <detailsAdditional instructions available <a href=""here</a</details
Update: is pending review and should be to the v1.8 Milestone to resolve this issue.


Is this a BUG REPORT or FEATURE REQUEST?: /kind cleanup What happened: It seems kubelet did not update inode pressure; and only test codes are using . So prefer to deprecate or remove it directly. (seems no node-misc group :). )


Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: I have a transient local disk mounted under /var/lib/docker in order to take advantage of faster IO on Azure VMs. This means that all that data is lost when a reboot occurs, but the rest of the data outside that mountpoint persists. After the reboot, the existing pods are relaunched, and kubelet is unable to create the log file symlinks because they already exist, but they are pointing to an invalid place, that container no longer exists. Here are the relevant log lines: The old symlinks are there: What you expected to happen: The correct symlinks should overwrite the old ones. How to reproduce it (as minimally and precisely as possible): Erase /var/lib/docker between reboots with existing pods. Anything else we need to know?: No. Environment: Kubernetes version (use ): Server Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.0+coreos.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-06-30T17:46:00Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration**: Azure VMs OS (e.g. from /etc/os-release): Container Linux by CoreOS 1409.6.0 (Ladybug) Kernel (e.g. ): Linux node-2-vm 4.11.9-coreos SMP Thu Jul 6 00:54:42 UTC 2017 x86_64 Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40GHz GenuineIntel GNU/Linux Install tools:
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
I think this falls under
Reiterating the mentions to trigger a notification: <detailsIn response to : I think this falls under Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


/kind bug /sig storage
/assign
It also fails on debian GKE. However, they all pass on debian and GCI GCE. Error msg is mounting timed out. This leads me to suspect that there is some difference in networking for GKE vs GCE.
I do strongly suspect there is some networking issue on GKE. Many other tests are also failing with "connection timed out" errors. 1.7 GKE tests are looking healthy, so it seems like something is broken on master. /sig network
Fixed by /close


What happened: Wrong HPA condition - ScalingLimited have one rc with 1 replica(current have 1 pod) have one hpa with spec.minReplicas=1 and spec.maxReplicas=10 HPA condition ScalingLimited is true and described by below, I expect ScalingLimited=false at that time. What you expected to happen: I expect ScalingLimited=false at above time How to reproduce it (as minimally and precisely as possible): kubernetes with DNS api server with --runtime-config="api/all=true" binaries and images, deploy heapster, influxdb and grafana in kubernetes a rc a v2 hpa hpa conditions by I expect ScalingLimited of hpa conditions should be: Addition info:
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig scalability
could you please take a look.
/sig autoscaling
It looks, at first glance, like what's happening is that since CPU usage is zero, we're calculating a desired replica count of 0, which is less that the minimum (1), so we're getting . We might want to detect this condition and provide another status message that's clearer. does that make sense on your end? Does the problem persist if CPU usage is non-zero?
Yes, the problem only persist if CPU usage is zero, and not exist when CPU usage is non-zero. I will keep this issue open if you want to improve and let it more clearer while CPU usage is zero. Thank you.
If this is still an open issue, I'd love to take a shot at submitting a pr for it. To confirm, the desired behavior is that if CPU usage is zero, and the desired replica count is 0, which is less than the minimum (1), should still be , but the status message should make it clear that it is because the CPU usage is zero?
I'd be happy to review such a PR :-). Yep! Having zero usage is a bit of a special case, so it's probably worth letting people know explicitly.
Wonderful! Working on it :)


<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?:BUG REPORT What happened: the err message in addNamespaceRoleBinding is wrong. ref to
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig auth


/kind feature What happened: In playing with the nginx ingress controller deployment, I noticed the following pod name: nginx-ingress-controller--c0ck5 I figure we should possibly put something in place to prevent accidental l33t profanity ;-). What you expected to happen: The following pod name: nginx-ingress-controller--r0ck5
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
My first github (k8s) issue. I've no idea which SIG would cover this topic. /sig product-management
/sig api-machinery
Previously we removed vowels in . We can 3xt3nd the idea with 0, 1 (can stand for ) and 3. would you like to send a PR? If not, I can work on this.
/assign jamesawebb1
Reassign to me if you do not want to contribute a PR.


Starting a large data set (32gb allocated, 4gb resident) in the api server causes about 7.5gb allocations of . The tree for that shows: This appears to be due to openapi aggregation. If there is an easy fix in 1.7 it might be worth it. Otherwise this should be addressed for 1.8.
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
This is all API server startup - for small clusters it's about 20-60% of startup time
Most of those are that is the library we used to convert our spec to proto. The key is this line: Do you have any idea how we can get a better space time/complexity here?
I have a PR for that now.
Fixed in
Even without aggregation, starting and serving OpenAPI docs was a huge chunk of startup time. The linked change makes it disappear.
This should not be related to aggregation at all (as far as I can tell from that tree). Thanks for the fix
Thanks, it's in gnostic master now.


Both https://k8s- and https://k8s- started failing recently. Their 1.7 counterparts are passing. The difference between the last passing and the first failing tests is: Probably something in broke the tests. /sig scheduling /cc /kind bug
It looks like there isn't enough cpu on the master to run the cos-nvidia-installer daemonset. Did we add any addons recently?
My previous comment is a red herring. It is an issue with . Ill post a fix soon. From the scheduler logs on the master:
that's a benign error. The daemonset is not expected to run on the master. But there is no easy way to express that request in our APIs.
Yes, the pod not scheduling on the master is benign. The actual error is that we do not report node condition , so it is left as . The fix is to report that is false.
The tests are still failing. /reopen
/assign
I will be updating to track progress.
tests are green again! /close


<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug /priority failing-test /area test-infra What happened: job is failing. See https://k8s- https://k8s- https://k8s- https://k8s- What you expected to happen: test should pass
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/assign
Reiterating the mentions to trigger a notification: <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I'm guessing that something got messed up in the bazel cache when reverting from 0.5.2 to 0.4.5. I'm not sure how best to fix.
How about updating everything to bazel 0.5.3 at once? I know you're trying to do a step-by-step approach to avoid breakages, but I suspect it will be cheaper to just deal with a brief outage while we update? (aka update the bazelbuild image to 0.5.3 and then merge your PR with updated and working BUILD rules?)
there's at least one other issue I've discovered with the latest go rules - the binary no longer works, due to changes introduced in that might be arguably better (at least everything still builds?), though it's frustrating that it makes things more broken in a different way.


All controllers that use the field have a default value of 10 for the field, except for , which has a default of 2. We should increase it to 10 for consistency on its default value across controllers. /sig apps


<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: I was attempting to filter pods with a using client-go selecting by its . I tried multiple variations until I ended up with . When filtering by , list of pod(s) came back empty even if pod with the IP we are filtering by was running. Then I tried reproducing it by calling the Kubernetes API directly which had the same result, list of pods came back empty: What you expected to happen: When passing the IP of the previously retrieved running Pod, I expected to get that Pod returned in the result. How to reproduce it (as minimally and precisely as possible): create a simple , get its IP (For example using - query Kubernetes: Anything else we need to know?: If I leave out the pod IP, I get all the pods back. I also tried using different formats when doing the curl, not one of them return anything. Also it seems like field is . As a side note I retrieved the URL which I used from debugging which calls client-go made. Environment: Kubernetes version (use ):Cloud provider or hardware configuration**: minikube version: v0.20.0 OS (e.g. from /etc/os-release): minikube OS Kernel (e.g. ): minikube kernel: Install tools: minikube Others:
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig api-machinery
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Currently the for only support , , , and . I've already submitted a PR in . PTAL.
cc


Is this a BUG REPORT or FEATURE REQUEST?: /kind cleanup /sig scheduling Descriptions: According to the discussion at , we'll move the node condition filter into a new predicates; and use feature gate to turn-off/on it. /cc
/assign


<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration**: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals


There are two issues: doesn't have merge key The validation code regards it as an error if both Initializers.Result and Initializers.Pending are nil. So if multiple initializer controllers sending patches removing themselves from Initializers.Pending at the same time, the last patch that gets applied will get a validation error.
/sig api-machinery


Seen in https://k8s- Aug 26 09:35:45.074: Couldn't delete ns: "e2e-tests-containers-9cscj": an error on the server ("missing route (service "sample-api" not found)") has prevented the request from succeeding (get ) (&errors.StatusError{ErrStatus:v1.Status{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ListMeta:v1.ListMeta{SelfLink:"", ResourceVersion:""}, Status:"Failure", Message:"an error on the server ("missing route (service "sample-api" not found)") has prevented the request from succeeding (get )", Reason:"InternalError", Details:(*v1.StatusDetails)(0xc4217a4900), Code:500}})
https://k8s-


We should solve this problem in a better way
cc :))
/sig contributor-experience release
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale /lifecycle frozen
/assign


https://k8s- https://k8s- cc: /kind bug /priority failing-test
Thanks a lot for reporting this! I think I have a fix PR for this issue. Or should we roll PR back?
I've kicked off testing of your PR. If it fixes the cross build, then that may be the easier path than reverting. cc: (sig-node leads)
I also fixed it in since it also touches the imports in that file. Would love to not rebase it yet again for this.
That call would be up to or they would have to determine which approach they prefer (fix on it's own in a small PR, or fix as part of a larger PR to prevent rebase).
Smaller is probably better; that way if the combined one needs to be reverted it doesn't re-break cross as a result. Will rebase if necessary (will depend on queue order.)


A couple of gce-scale-performance runs failed due to failing to delete the network, which in turn was due to a couple of leaked firewall-rules (I manually deleted the network later to fix it): Ref: https://k8s- cc (assigning to you, feel free to reassign as apt)
I digged up a bit and seems like the firewall is created by the service-controller as part of , which should've been deleted in but wasn't. Feel free to close the bug if you think it's just a one-off failure.
Took a quick look, both and runs have these firewall rules initially but doesn't, which indicates something happen in the run. But oddly there is no build log, logs in artifacts are empty as well. Likely that run is badly broken so cluster got terminated forcibly in the middle, and resource are leaked. Will observe this suite a bit longer. /assign /unassign
I think it wasn't caused by this job. We're running this one () and on alternating days in the project. And they both create clusters (network, etc) with the same name. And I think of gce-scale-correctness leaked the firewall (and hence the network) as it runs some correctness tests around load-balancer. That likely caused this job to fail.
Ref The resource leak was likely caused by test timeout --- GCE operations took too long to finish and the cluster was terminated ahead. Because heathcheck and firewall cleanup in gce controller are coupled with targetpool deletion, deleting cluster while targetpool deletion is pending will leak the resources.
Fyi - this affected gce-scale-performance again (ref: run 35 of https://k8s-): Is there a way to make sure they get deleted even in case of a timeout? I have a dirty hack in my mind - of deleting those as part of kube-down. Maybe you have some better suggestions?
Consider the frequency this happens, I think it makes sense to implement some cleanup logic within the e2e test. Will give it a try.
Turned out we already have such cleanup logic, it is just we use an incorrect firewall name and forget to cleanup node health check resources as well. Sent to fix.
Thanks for fixing!
Sure thing, thanks for reporting!
Reopening as it seems like the issue is not fixed yet. For e.g run-16 of https://k8s- failed due to this: This is also affecting our job, both of them being release-blocking jobs. Could you PTAL?
By any chance we could make and use different names so that it will be easier to target the leaking point? Also, in that way one leaking job will not affect the other. Or maybe we could also make different runs use different network names? The scalability job is expensive so it might be great if newer runs wouldn't be affected by the leaked resources from previous run. I haven't figured out how exactly and when the resource are leaked, but given that the operation cycle of service controller and e2e test process proceed in an asynchronous manner (plus always fail ESIPP tests ), there might be chances we will leak resource even after we put cleanup logic in e2e test. Will take a deeper look later. Also what you suggested before --- deleting those as part of kube-down --- may also be a viable solution.
I thought about this option, but it's better for them to share the same name for network and other resources (instance-prefix, etc) because that way we ensure that if one of them leaks some resources, the other will delete them during it's startup. Else we'll run out of quota in the scale-testing project (for e.g wrt routes or VMs) by trying to start another 5k-node cluster while the previous one still exists. I went to pantheon and deleted the network and that went through successfully (though fails). Wonder if the delete command in kube-down can be modified to delete all sub-resources of the network (like firewalls, etc) before deleting it?
Sounds reasonable to me. We may need yet another env to trigger it? As deleting everything correlated to that network sounds a bit dangerous.
Well, took another look, seems like the actual thing that prevented us from deleting the network was mostly firewall rules, and deleting those sounds safe enough if we are going to delete the network anyway. Will try to make a PR for this.
Thanks for fixing the issue with firewalls. However seems like the latest run of gce-scale-correctness still leaked the network (but doesn't seem due to firewall this time): ref: https://k8s-
I thought we're deleting the subnets in this case too when we're using custom subnets mode.
Yep, we are supposed to. I think the scale CI hit the same issue as described. I've sent a PR for fixing it
For the time-being I manually deleted the subnets and then the network to unblock the next scale job run. But it would be good to have this fixed soon. cc
Closing this issue as we're now deleting all remaining firewall rules in the network before deleting the network itself in kube-down. For other networking resources in general, we have issue .


/kind bug /sig node When the is set to and a container that was granted exclusive CPUs terminates with a normal exit status, the CPU Manager never receives a call. Kuberuntime should be updated to invoke the container manager's hook on detecting normal termination for these containers. This method is already idempotent since it is currently invoked both from kuberuntime's and (the latter of which is invoked by GC.) How to reproduce it (as minimally and precisely as possible): Start singleton cluster with static CPU manager policy enabled. Start a guaranteed pod with a single container that consumes 1 CPU (to get an exclusive allocation) with restartPolicy never that simply exits . Try to start a similar container that won't fit on the node unless the first container's CPU is reaped. Observe the second container fails with insufficient CPUs, and this problem does not resolve even after an extended period of time. Environment: Kubernetes version (use ): v1.8 beta Cloud provider or hardware configuration**: all OS (e.g. from /etc/os-release): all cc
/assign
Yikes, that's a fun issue. I guess we need to listen for PLEG events?
There's a spot in the kuberuntime sync where we think we can make the call.
well, i knew i would miss something ;-) good catch.
All thanks to workload perf testing
By the way, where is of CPU manager called if is set to or ? invoked by GC?
Hi, Besides, I wonder do we have a mechanism to recover things in state of CPU manager when kubelet restarts? Here I have a trial, seems it doesn't work as expected, that is: a few logs before and after the action of addcontainer and removecontainer in policy, to see the content of state: a pod that declares an exclusive CPU, and saw logs: kubelet the pod, to compare the state with that , I think if recovered, the content of them should be the same. but saw logs: Seems that the content in state did not recovered, this way, the CPU manager would behavior far from what we expected if kubelet restart. Did I miss something important?
is called as a part of internal container lifecycle pre-stop and post-stop hooks (see and ). Regarding Kubelet restarts, it is a known issue and captured in
priority/important-soon
[MILESTONENOTIFIER] Milestone Labels Complete Issue label settings: sig/node: Issue will be escalated to these SIGs if needed. priority/important-soon: Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. kind/bug: Fixes a bug discovered during the current release. <detailsAdditional instructions available <a href=""here</aThe commands available for adding these labels are documented <a href=""here</a</details
Well, I understand business in , what I wonder is upper behavor, that is, the behavior of in containerRuntime, When a container with restartStrategy exits, It seems that the runtime try to restart new container directly, without calling things like . So I failed to see the place where remove related hooks called in this case, so here I have a trial: a pod whose container is in restartStrategy of and would exit after a perid exited after a peiod: try to restart it but failed: restart after quite a preiod: So I think the resource is not recycled synchronously, and the remove related hook is called in GC. Am I right? Does it works in the way we expected?
Yes, that's exactly correct. The linked PR with the fix for this issue also helps the situation you described to converge faster.


Follow-up of Docker 1.13 introduced a change into the logging mechanism (), that split the log lines longer than 16K into chunks of no more than that. Current default logging integrations (fluentd-gcp and fluentd-es addons) do not support that and this might result in broken ingestion, i.e. when JSON parsing is broken for entries longer than 16K. One possible solution is introducing a fluentd plugin to the configuration to concat such entries in the similar way does it (assuming there is one out there, cc for that) /cc
Presumably this should be done in the Docker fluentd logging driver... The discussion on seems to be leaning that way...
In fluentd logging driver side, journald driver approach is better: I'm not sure buffering partial message in logging driver is good or not... In fluentd side, implementing or is needed.
do we need to block 1.8 release? can we please move this out of 1.8 if it's not?
Moved to 1.9, since it's not specific for 1.8
Is there any way this can be fixed by running a custom fluentd? Or is there a fluentd docker driver github issue I can follow for updates on when they expect to have this fixed? Txh
[MILESTONENOTIFIER] Milestone Issue Needs Approval Action required: This issue must have the label applied by a SIG maintainer. <details<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
Problem is that there's no fluentd plugin to fix the problem, so running a custom fluentd won't help, there's nothing to add to the configuration to make it work (unless you want to write your own solution) They don't think it's a docker problem and don't plan to fix it. However, it will be fixed in GKE
That seems fair from Docker's perspective. I'd rather have stable containers as well :) So you're thinking of switching from the current log drive - at least that's what my Google Container Optimized image in GKE 1.8.1 uses - to the journald driver to circumvent the problem? And then use a plugin like to recombine the partial messages?
No, that's not the case. Solution is TBD, but if you export your logs to Stackdriver (that has a limit of 100KB per entry) you won't see the difference
I think there's still a difference in that the 100KB stackdriver logging limit is larger than the 16KB docker limit at which the partial messaging starts, so without proper recombination of those chunked messages you'll lose the ability to send log lines larger than 16KB into stackdriver. However we're also running our own fluentd that sends messages straight into pub/sub, which has a limit of 10MB per message. And although we're trying to keep the messages much smaller than 16KB it would be nice to be able to send bigger messages without instrumenting our applications to push the messages into pub/sub themselves. I hope you find a nice solution that will make this possible :)
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
/remove-lifecycle rotten Still relevant, hasn't been fixed
I've been reading up on this issue, it's really too bad docker would not allow the configurable log-max-size option, this would be an easy temporary workaround to the issue until fully resolved. Regardless, I also wanted to point out a 2nd issue that the log splitting causes which might help increase the priority of this issue. If you are using json logging and have MCD or Structured Arguements attributes in your log messages, what ends up happening unfortunately is the attributes are lost on the split messages. This is a huge drawback right now. In my case I am using spring cloud sleuth which adds span and traceid's to all log messages allowing you to easily filter and correlate all related logging messages. When the messages are split, the traceid's are not part of the message in elasticsearch making filtering by traceid impossible. My application logging is setup as follows: is it still the plan to have fluentd put the logs back together into a single message? can anyone update if any progress is being made? thanks for your help.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten Still relevant, hasn't been fixed
Would using the "concat" fluentd plugin be relevant? It seems to have explicit support for our specific use case. Edit: For EFK, Adding fluentd-plugin-concat to fluentd-es image (, ) and adding to the Works well. Would a PR with those changes be acceptable?
trying to solve this as well. Having the same issue as where long lines are being truncated and MDC gone.
Please see The big problem here is concurrency. Just concatenating all partial log messages as they arrive may be wrong. Let's say your app logs two events, called and here. Then the Docker log messages may be written to the JSON log file (or whatever Docker logging driver is used, e.g. in plain Docker or Docker Swarm) in this order: The fluentd concat plugin cannot decide what to do. What records belong together? The good message is: This can be solved (similar to ) as Docker logs contain all the needed information here: So you just have to extend these methods and the struct by the partial log meta data: Then you have to fix Mind the ID and ordinal of the partial log metadata so that you concatenate the right records in the correct order.
I just edited my above comment because I am not sure how the case of linebreaks (i.e. those written by the application) in log messages relate to this issue. However, often you can control the logging of your application and let it write JSON to STDOUT (e.g. for Java / Spring Boot applications) so you do not have linebreaks. Back to the original issue: I have serious doubts that will do it under load with a lot of concurrent partial messages. I observed this in a Docker Swarm setup with fluentd and the concat plugin configured analogously.
Could you please reopen this issue? The concurrency is the problem here: Let your container simultaneously produce two log messages of 16KB (let's say 20 KB). Then fluentd will concatenate all four messages to one large (16+16+4+4 KB = 40 KB) message. The fluentd (or whatever you use) needs to know the ID of the partial log metadata to concatenate only those messages with identical IDs. So this needs to be implemented in the (default) local JSON file log driver in Docker (moby).
Problem is that ID comes with Docker 18.06 if I'm not wrong, this is why I didn't use it: do we want to require a docker version which is 2 months old?
is no longer much involved with the project. if you would like to open this back up to alter the implementation, please file a new issue
If you do so, please ping me as well.
I have the concat plugin but still it is not concatenating, Any Idea?
No idea why it does not work. For up-to-date information, I have inserted: Just after the kubernetes_metadate injection.


The following that have been failing on our 5k-node tests (https://k8s-): [HPA] Horizontal pod autoscaling (scale resource: CPU) [sig-autoscaling] ReplicationController light Should scale from 1 pod to 2 pods [HPA] Horizontal pod autoscaling (scale resource: CPU) [sig-autoscaling] ReplicationController light Should scale from 2 pods to 1 pod This is part of the correctness suite and large cluster tests are release-blocking. cc
[MILESTONENOTIFIER] Milestone Labels Complete Issue label settings: sig/autoscaling sig/scalability: Issue will be escalated to these SIGs if needed. priority/critical-urgent: Never automatically move out of a release milestone; continually escalate to contributor and SIG through all available channels. kind/bug: Fixes a bug discovered during the current release. <detailsAdditional instructions available <a href=""here</aThe commands available for adding these labels are documented <a href=""here</a</details
Could you PTAL into this or redirect as apt? It's a release-blocker.
xRef
The tests seem to be passing on the normal correctness job - https://k8s- (except the latest run) but they're failing on large cluster and the reason seems to be timeout waiting for scaling to happen.
If i have to guess it is heapster who is failing to provide metrics. HPA itself has no direct dependency on the number of nodes in the cluster. Assigning to .
is going to take a look.
From the logs it seems like Heapster is unable to get metrics from nodes. It appears to be a separate issue from addon resizer restarts.
Thanks for checking it. I have got the same logs for smaller cluster (10 nodes), so at least we know that the problem is not related to big number of nodes/pods.
- I'm seeing the test pass on our regular (small cluster) gce job - https://k8s- And seems like it was passing before and then started failing on our 2k-node job - https://k8s-
thanks for pointing on it. The main difference between runs on small and big cluster - special node that we are creating for heapster container. And my local experiment have shown that without this node everything works, but with it - fails. So I assume that it's more network related. I'll try to figure out what's going on.
Removing this as release-blocker and downing the priority as it's a setup issue specific to use of heapster node. Would continue to work on it (thanks!).


When creating a service passing the annotation , two IPs are created in the service, a private and a public: Having a private IP is not a desired scenario for service discovery in federated global environments. If we have, for example, a cluster in GKE and join a local OpenStack cluster, this private IP is written in the DNS zone but the GKE cluster won't be able to access it. We could add an annotation to say we don't want the private IP to be registered in service's LoadBalancer Ingress field, like: or And when it's false, only the public IP would be written in the service, therefore in the DNS zone /kind feature
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig openstack /sig federation
'' sounds better.
I don't like adding annotations just to make things work. I counter-propose just not publishing the private IP whenever there is also a floating-ip (without a configuration option to control this behaviour).


<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: FC plugin returns TargetWWNs at GetVolumeName(), but TargetWWNs are not unique identifier if user has multiple volumes from same target wwn. As a result, after introducing attach/detach functions, FC plugin always fails to attach volumes if a pod requests 2+ luns. The GetVolumeName() is called from GetUniqueVolumeNameFromSpec() during Pod creation. FC plugin should return unique identifier in the GetVolumeName(). If user uses WWID to specify FC volume, this problem doesn't happen because the WWID is unique identifier. What you expected to happen: All FC volumes are attached to pod. How to reproduce it (as minimally and precisely as possible): Create two PVs and PVCs which uses same target wwn. Create Pod with these two PVCs Anything else we need to know?: Environment: Kubernetes version (use ):Install tools: local-up- /cc
/sig storage
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals


Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: If a pod is already running in the default namespace with the expected name (i.e. ), the PV recycle method just assumes it's a recycler pod. However, if it is not a recycler, it could result in the volume being reused without the data being cleared. What you expected to happen: The recycled volume should not be reused until the data has been verifiably deleted. The easiest approach would be to make it an error to not run the recycler pod successfully, i.e. delete these lines: I think that should be safe, since the recycler will retry. Other errors in that method should also be considered fatal, e.g.:
/cc
How about generating a unique name for each recycler pod so that it doesn't matter what existing pods are named?
Could someone from your team pick this up?
Ack
OpenShift runs the recycler pod in a dedicated namespace, not visible to users. Maybe it could be a good idea to do so also in Kubernetes. Anyway, killing the old recycler is probably the best option - it could be forged by user to block recycling and cheat quota. /assign
Agree with running in a dedicated namespace, but it sounds like this feature is just being deprecated, so it might not be worth the effort. Thanks for picking this up.


Is this a BUG REPORT or FEATURE REQUEST?: What happened: Kubeadm E2E test is currently broken due to the commit I report the same issue on my local cluster. If I revert PR , everything works fine. We believe the current weave-net plugin doesn't work with the CNI version change. Right now all kubeadm related development is blocked on this. Could we consider reverting this change from head until we figure out what's wrong in weave-net/CNI? More context in
/assign
Please reopen this in kubernetes/kubeadm. There is an issue as e2e tests & PR job are failing, but locally the CNI bit seem to work for me, so I think it has something with building the debs to do. Not technically. We just don't have any coverage right now. Not IMO. We just need to forward-fix the issue. Closing this; you can reopen in kubernetes/kubeadm
kubeadm e2e test still red. Do you have any progress or hint in terms how to fix it forward? Just to clarify, If I install k8s and kubeadm locally using deb, I can repro the issue on my test box. I could believe it's sth wrong with debs. In the meantime, CNI and plug-in like weave-net change is a big deal. We clearly didn't test it thoroughly for this PR. Reverting the change seems to be the clean solution.
FYI kubernetes/release bumps new CNI releases for deb/rpm packages.
Thanks.
Is this the fix needed to fix these tests?
I actually think that will take us in the other direction. During the kubeadm e2e tests, we use the deb packages built by bazel. During our official release process, we use the debs built by anago (in the release repo). Bumping CNI in our bazel build broken our e2e tests. Bumping CNI in the release repo scripts will similarly break our official release builds as well. If I understand the problem correctly, the only way to fix both of them is for the CNI providers like weave-net to be updated to handle the changes of upgrading CNI to 0.6.0.
I think this issue worth some more discussion. Re-open the case as it might not be affecting kubeadm only
sorry, i cannot reopen the issue as it's closed by But let's continue to discuss the option here.
Yes, I think so
This will be fixed by ; this was due to the CNI binaries being extracted in the wrong place. It is kind of a kubeadm packaging issue. (Reopening just to make the PR close this)
nice find!


/kind bug Right now we do not validate PV updates so admin can update at any time. Kubernetes has no facility to reflect this update - it won't magically re-mount an existing volume to e.g. a different NFS server. IMO the PersistentVolumeSource to be read-only and set to stone once created.
/assign
I think it is a good idea to make the VolumeSource read-only but I am afraid ship has sailed in v1 and existing code/tooling that relies on the ability to modify PVs will break as a result of this change.
- do you have any advice on how we should proceed on fixing this issue?


I also hit when running cri-containerd as the container runtime. The reason is that we don't populate stats like in cri-containerd integration now. is adding the containerd support in cadvisor, after that we should have full summary metrics. However, heapster should not panic in this case anyway. Can we cut a heapster patch release and update heapster in the main repo?
+1. Many new CRI integration would run into this issue.
In fact, as far as I am aware, not a single CRI implementation currently implements any of the stats calls.
I beg to differ: That's the reason why filed the issue.
Yeah does implement the CRI stats. Currently it only support core metrics, which is only part of summary api. We are working on integration with cadvisor, before that some fields in summary api such as will be nil. Current heapster version will crash when is nil, which shouldn't happen. :)
/cc
My bad, thanks for the pointer :slightlysmilingface:. I'll correct my statement: I believe no CRI implementation except for cri-containerd implements the stats calls. The ones I explicitly checked are: dockershim, cri-o, rktlet.
Something I'm wondering about though: The intention of the stats calls is in order to be able to get rid of cAdvisor. Making the stats calls depend on cAdvisor seems to cover up the issues it has. Re:
Here is the reason that we still have to integrate with cadvisor for now.
/cc
If I understand correctly this is meant as a migration mean, so that both are possible for some time, and then we drop cAdvisor once a reasonable migration period has been granted.
[MILESTONENOTIFIER] Milestone Issue Current <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
I will take a look at Heapster and try to fix this. Can you advise how to reproduce the issue? This will help a lot with debugging and verifying a fix once it's ready.
this is already fixed at head and 1.4.3 version of heapster The problem that is using kubernetes version from head and we have there 1.5.0-beta.0: I see two possible solutions: version of heapster to 1.4.3 at head of kubernetes. Piotr mentioned that 1.5.0 has still a lot of problems, so it maybe makes sense. a new release of heapster (1.5.0-beta.1) and after that bump version at head.
And thanks for taking this issue.
Not directly related to this issue: a more general discussion is at We'd like to understand which fields are required and optional in heapster, so that we won't crash heapster with partial result.
We've merged a work around for now Hopefully it works. But anyway, we should fix this in 1.9.
what needs to be done for 1.9 here?
we'll need an assignee as well
We just need to update heapster version.


Is this a BUG REPORT or FEATURE REQUEST?: Bug Report /kind bug What happened: improved disk eviction testing to make it more precise. For example, it expects a pod which uses 299 Mb of disk not to be evicted if it sets its limit to 300 Mi. As shown on , this test passes for all OS distributions except CoreOS. I have ruled out an issue with the storage driver and backing filesystem since both COS and CoreOS use overlay backed by extfs. What you expected to happen: I expected the pod using 299Mb not to be evicted when its limit is 300 Mb. I expected not to see this event: How to reproduce it (as minimally and precisely as possible): The container runs this command to use 299Mb: Each iteration writes 1Mb to disk, producing a total of 299 Mb of disk usage. The pod has set to 300Mi. The test has the LocalStorageCapacityIsolation feature gate enabled. Anything else we need to know?: This may be a discrepancy in the (which we use to measure container disk usage) or , which is used by the test to write exact quantities of disk.
/sig node
cc can you take a look at this after thanksgiving?
I bumped the coreos image... going to see if it clears up the flake
It appears the root filesystem is being included in the disk space calculation . Do you think the coreos image might be slightly larger than the other images under test?
One more code snippet that is interesting:
thats the container's root filesystem
correct. the feature appears to add and compare to the quota: The reproducible steps do not take into account the rootfs space. If the quota is set to 300Mi, then the under loop count would have to take into account the rootfs space to not exceed the limit. I do not see anywhere in the tests or the reproduce steps this part of the calculation.
The refers only to the writable layer of the container, and excludes base (image) layers. With the provided command, the only files that are in the container's rootfs are the ones written by the command, and should amount to 299Mb. Additionally, the container does not write any logs (through stdout), so that value should be 0.
I ran a quick test on a vm with the current image, and found that both and correctly wrote and measured disk space correctly. I also tested as it inside a container, and verified that we are measuring the container's writable layer correctly. My guess now is that it has something to do with how we calculate disk usage by logging with journald. I am looking into it
Actually, the previous comment was incorrect. When I ran the test, I found that each file was 1028 kb instead of 1024 kb, which resulted in the command writing 300.3 Mb instead of 299 Mb. I will change the test to require slightly less precision. This appears to be because an empty file in coreos takes up 4 kb. Test this by running . On other OS we test on, empty files use 0 kb. This is what accounts for the difference.
I tracked down an issue to the dd call. The stdout and stderr need a redirect to /dev/null to suppress the following messages: Problem Line: Fix:
I PRed that fix and changed it to stderr, since the other tests seem to use stderr.
leaving this open until we get a green run
I'll continue to test locally to see if I find anything else.
opened PR with the help of and CoreOS defaults to using the actual size instead of the apprent size.


We should have e2e feature for resizing in GCE environment. Also add unit tests for node restrictions.
/sig storage /assign


<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: kubelet throws the exception when --cloud-provider=cloudstack is included: What you expected to happen: There shouldn't be an exception. How to reproduce it (as minimally and precisely as possible): Add the following line to /etc/systemd/system/kubelet.service.d/10- Environment="KUBELETEXTRAARGS=--cloud-provider=cloudstack --cloud-" and also configure the file. Anything else we need to know?: The condition check in the InstanceType() function in is wrong. It currently reads: but the correct code is (condition negated): Environment: Kubernetes version (use ): 1.8 Cloud provider or hardware configuration: cloudstack OS (e.g. from /etc/os-release): Ubuntu 16.04.3 LTS (Xenial Xerus) Kernel (e.g. ): 4.4.0-98-generic Install tools: kubeadm Others: /sig cloudstack
I will have a look
/sig cluster-lifecycle /area cloudprovider


<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: The test has been failing since Nov 22nd. Looks like it is because the test restarts Kubelet and then calls getLocalNode() which checks for node readiness. Not sure why this problem only started to show up recently. What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): 1.9 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): COS Kernel (e.g. ): Install tools: Others:
/sig node
Hi, Just FYI, . Maybe it is helpful for you.


We should not attempt resizing of file system if volume is mounted as read-only. /sig storage cc