This issue is meant to track planned work items for pre-allocated hugepages across releases. This issue captures the plan of record discussed in the wg-resource-mgmt meetings. [x] HugePages should be restricted via ResourceQuota [Owner: , fixed ] Assess ability to graduate feature to beta. [x] HugePages should require the user to request cpu or memory [Owner: fixed ] [x] HugePages should not be tied to QoS [Owner: fixed ] [ ] HugePages promoted to beta [Owner: ][ ] HugePages isolated at container and not just pod boundary [Owner: TBD] [ ] HugePages should be restricted via LimitRange [Owner: TBD] cc
/cc
If no one is doing it, I'd like to work on the ResourceQuota part.
would this be discussed in the meeting recently? i would like to try on the "via LimitRange" but seems more details need to be settled.
/cc
Regarding to the ResourceQuota issue, all I could think is to add support for resource HugePages in podEvaluator of quota, so that it could be aware of the hugepage declared in resourceQuota, and that requested in podSpec. It is thought to be a small change, The only thing that needs attention is the way to address resource HugePages, as it's name is not fixed, like CPU or memory, but in a specific format. Correct me if I miss something :)
-- the quota change is relatively small, if you want to pick up the work, i would be happy to shephard it.
-- you as owner, thanks for volunteering! let me know when you have the pr up
are you ready for the LimitRange part? :)
yes, I've worked on this for some time and will commit the pr ASAP.
/cc
hi, I am confused by the tests I've made, which seems the "HugePages should be restricted via LimitRange" has already been done. I created the LimitRange as below: apiVersion: v1 kind: LimitRange metadata: name: mytest spec: limits: default: hugepages-2Mi: 40Mi defaultRequest: hugepages-2Mi: 40Mi type: Container and the pods created by the following yaml: apiVersion: v1 kind: Pod metadata: generateName: hugepages-volume- spec: containers: image: tomcatjava:v6.0 command: name: example and the output of the pod in yaml is like: spec: containers: command: image: tomcatjava:v6.0 imagePullPolicy: IfNotPresent name: example resources: limits: hugepages-2Mi: 40Mi memory: 250Mi requests: hugepages-2Mi: 40Mi memory: 125Mi Is this the LimitRange part already been done? Otherwise anything wrong with my understanding? Also, I found an subtle error in the error info when creating pods restricted by the hugepage , would you please take a look?
-- thanks for the feedback, i have been distracted looking at other topics, but hope to return to this later this week.
I have filed an issue related to hugepages, would you please check when you are free? link:
you can't re-open an issue/PR unless you authored it or you are assigned to it. <detailsIn response to : /reopen </details
hi, I've worked on the HugePages isolated at container and not just pod boundary issue and would like to confirm some details. My first though was to implement this issue via docker, just like what cpu and memory did. But docker seems not support hugepage limit in present. Which means we have to get it done by coding in kubernetes or libcontainer, is that right? Please let me know if there are any other instructions that may help :)
-- to support isolation at container boundary, we would need support from the container runtime. that would require support in the CRI to expose the resource, and support in individual container runtimes (contianerd, cri-o, docker, rkt, etc.) to actually make use of it. in practice though, i have not seen a major requirement to support container boundary isolation. do you have use cases in mind that would require it?
I have no any other use cases as well, for now all the achievements fit well for our requirements. BTW, does the still need to move on? I found it seems already being supported by default.
I don't think the intent was for to close this /reopen
you can't re-open an issue/PR unless you authored it or you are assigned to it. <detailsIn response to : I don't think the intent was for to close this /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Pods with volumeMount gitRepo volumes in 1.8.0 see the following error in kubelet logs (also in ) It is important to not that the git repo clone is done via ssh. I think what is happening is that hyperkube image now used to have command in it and now it does not have it. So the git clone command that kubelet runs fails with the above stacktrace since it cannot find . What you expected to happen: gitRepo is volume mounted. How to reproduce it (as minimally and precisely as possible): Reproducible in kubernetes 1.8.0+ hyperkube images. Anything else we need to know?: This probably is due to missing ssh binary in the new 1.8.0 hyperkube images. The commit might be an issue. Will comment further if I find otherwise. Environment: Kubernetes version (use ): =1.8.0 Cloud provider or hardware configuration**: Not Application. OS (e.g. from /etc/os-release): hyperkube base image Kernel (e.g. ): Install tools: Others:
How did you installed Kubernetes may I ask?
how does this help?
Normally, it is suggest to install kubelet as a daemon on host instead of using hyperkube container. cc you may be interested in this issue.
Need your help in identifying the fix here. I can send something for review. Suggestions are welcome.
The problem is that ssh isn't installed inside of the container. I don't know if it should be either, as the hyperkube image grows so large. If you install the openssh-client, it should work
Yes I did identify that as the problem if you see the description I mentioned. I pointed to a git commit that might have caused this. I am not sure what package installation has changed in the hyperkube container. "ssh" used to be available and now it is not. For the gitrepo feature to work it should be available.
Ah okay. If it was there before, we might want to add it back. SGTM.
Yeah, I switched the hyperkube image base from to our with some additional dependencies back. Apparently due to lack of volume + hyperkube test coverage, I've missed a number of dependencies needed. cc
we could use dropbear as a replacement for just the ssh client?
I would prefer to use the openssh client over the dropbear client. It has wider usage and a stronger security track record IMO. The dropbear client also has somewhat different usage and configuration, so arguably using it is still a regression over the previous state. Saving a couple MB isn't worth using worse utilities, however much fun "dockerfile-golfing" is.
to fix
cherrypick to release-1.8:
This should be fixed in v1.8.3.


<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: I'm running a Redis pod with a PV provisioned as an Azure Managed Disk. It's got an exec readiness probe that calls in the container. At some point it receives a SIGTERM. The kubelet then tries to sync the pod and restart the container, but it panics in (looking over the code, that's the only place where the error from is ignored) As a result, the pod hangs in a state, since the readiness probe cannot be ran in a non existing container. What you expected to happen: The container would successfully restart. How to reproduce it (as minimally and precisely as possible): It's the 2nd time it happens but I haven't yet figured out the conditions to reproduce. Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration**: Azure w/ Managed Disks OS (e.g. from /etc/os-release):Kernel (e.g. ):Install tools:Others:
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hi could you also share the storageclass, pvc & pod yaml files? I would like to repro, thx
I may insert a nil check statment in this func: But I need to reproduce this bug anyway, so that I could find the root cause of this issue, thx.
Yes, an error check is what I'm hoping for :) I've used the chart. The PV/PVC were created manually, they're set to an empty storage class. I should mention that I've also noticed other pods being terminated/recreated at the time this one failed, however I didn't trace the cause of this yet.
thanks for the reporting, I may have found the root cause, just need a few more debuging time, thx.
GitHub didn't allow me to assign the following users: xiazhang. Note that only can be assigned. <detailsIn response to : /assign xiazhang Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign andyzhangx
- Thanks for the fix!


Our 5k-node cluster jobs recently started failing even before the e2e tests start. The reason is the step which checks that all kube-system pods are running is timing out: cc
From the scheduler logs I'm seeing that the pods are being scheduled quite slowly, getting worse with time: 2-3 pods/min looks quite bad. cc
Also finding such traces: Seems like most of the time is being spent in priority calculation.
From the pod spec, I'm seeing soft inter-pod anti-affinity at node-level topology: I'm almost sure this is the one causing the mischief. We've been having performance issues with anti-affinity traditionally :p
Ok... PR is the cause. It introduced anti-affinity to kube-dns. I think we should revert this change given known issues with anti-affinity. - Opinions?
FYI - it is taking 80m until the last pod is scheduled when I'm running 5k-node cluster locally.
That's quite unfortunate. It looks like we should revert the change until the anti-affinity change is scalable enough to be used more broadly, unless the scheduler folks have a target fix for this use case.
cc
We are very well aware of the performance issues in our affinity/anti-affinity functions, but I am not aware of a concrete solution at this point. This is partly due to the flexible design of affinity/anti-affinity concept. My humble opinion at this point is to drop some of the flexibility as they are rarely used, in order to streamline the algorithm and improve performance. Anyway, I think the proper solution to this problem needs more time than your requirements to address the issue in hand. So, you may want to change your configuration and drop the anti-affinity rule.
would it make sense to add a note to the anti-affinity user documentation to hint at performance problems at large scale, until this is fixed?
Re-opening this issue as this problem has struck again. The following runs of our 5k-node scalability jobs failed to bring up the cluster on time: https://k8s- https://k8s- https://k8s- https://k8s- And the reason is kube-dns pods pending to be scheduled for a long time:
These failures started happening about 4 days ago and I'm pretty sure it's this one: Add self anti-affinity to kube-dns pods Sadly, we'll have to revert it again for now :( cc - Seems like the improvements to anti-affinity performance made under weren't enough?
We need to find other improvements for the predicate. If we cannot find major improvement opportunities, we may have to change its design to be more scalable at the cost of the reducing its flexibility.
- my personal opinion is that we need to change the design anyway. Having it work both ways (i.e. existing pods everywhere impacting placing of our pod) will always be painful (especially for things like cluster autoscaler)
I agree. The symetrical application of anti-affinity is problematic. I also think we need to change it.
Great. - do you agree with that too?
, I'm wondering after some affinity/anti-affinity optimization in scheduler, do you think if it's fine to test coreDNS with anti-affinity policy in large cluster with 5000 nodes?
We probably should try to see where we are - but this requires testing before enabling that.
I agree with - Could we first try checking with a benchmark integration test () as a proxy for real cluster test?
I'm wondering how many real nodes are required for testing?
Since the integration tests fake out the nodes, I think it's safe to crank it up to 5000 (that's the current largest integration test benchmark size) I'm also hoping to refactor our benchmarks soon, so let me know if you have any thoughts about them.


BUG REPORT: /kind bug /sig aws What happened: Adding this annotation to an existing service: did not trigger an update on AWS ELB (no new tags). What you expected to happen: Expected to see those 2 new tags under the existing ELB: How to reproduce it (as minimally and precisely as possible): a service without annotation the annotation tags on AWS Anything else we need to know?: Creating a new service WITH the annotation will create the expected tags on AWS. Environment: Kubernetes version (use ): 1.7.9 Cloud provider or hardware configuration**: AWS OS (e.g. from /etc/os-release): Debian GNU/Linux 8 (jessie) Kernel (e.g. ): Linux ip-xxxxx 4.4.78-k8s SMP Fri Jul 28 01:28:39 UTC 2017 x86_64 GNU/Linux Install tools: Others: Using Kops
+1
+1
+1
+1
+1
I've done some work on a fix for this, will try and get a PR submitted later today.


On AWS, mutable API calls such as AttachVolume or DeleteVolume have separate API quota from API calls that do not modify the resource. Currently k8s makes a high number of AttachVolume and DeleteVolume API calls even though k8s can check if the operation is going to succeed.
/sig aws


We should make the CRI log parsing code in kubelet to be a library. So that container runtimes and external tools don't need to copy the whole code ( vendor it instead ). Referring
/assign
/sig node
I can help on it.
/assign
GitHub didn't allow me to assign the following users: yanxuean. Note that only can be assigned. <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Thanks. I have a branch ready for this. Will open a PR later and ping you when it is out.
ok


<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT: What happened: When a statefulset object is created, a lot of controllerrevisions are created at the same time. the controllerrevisions have differenct hash value but the same Data.Raw. How to fix it: two controllerrevisions are seen as the same , when [ControllerRevisionHashLabel] , Controllerrevision.Data.Raw , Controllerrevision.Data.Object are strictly the same // this line shoud be to fix the bug like below: Anything else we need to know?: Environment: Kubernetes version (use ): 1.7.8 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
I would help fix it
/kind bug /sig apps
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
StatefulSet uses the following code to construct its history object (controllerrevision): However, collisionCount shouldn't be blindly increased when there's name collision. Because controller uses a cache store, the name collision could be due to a controllerrevision created but not yet observed by the controller. The right way to handle it is to first check if the controllerrevision with the same name has the same content as the one we're going to create, like how DaemonSet controller handles it:


/kind bug /sig cli What happened: Tried to list the age of pods in the namespace in an otherwise empty cluster $ kubectl --namespace kube-system plugin aging The Magnificent Aging Plugin. No pods What you expected to happen: That the pods in the namespace would have been listed How to reproduce it (as minimally and precisely as possible): the and from into . the command: Anything else we need to know?: No Environment: Kubernetes version (use ): v1.8.2 Cloud provider or hardware configuration: minikube & GKE OS (e.g. from /etc/os-release): Mac OSX El Capitan Kernel (e.g. ): Darwin Kernel Version 15.6.0 Install tools: Google Cloud SDK
hi for cli issue, could you please file it in this repo next time ? thanks !!
Will the k8s github bots be able to resolve references in commit messages made in kubernetes/kubernetes PRs be able to find the corresponding issue ticket in that other project? Semi-OT but related: I know there's an overall intention to refactoring k8s proper into many distinct projects, so it would be helpful if either the contributor guide contained a mapping of each and every module making up a k8s release with its corresponding github project and protocol governing the component. Or if each directory (module) in kubernetes/kubernetes contains its own explaining the contribution protocol. Right now documentation and information about ownership, contribution and protocols of various project modules are a sprawl and a bit of a mess, creating a significant barrier to entry (contribution). What's the proper place and form to discuss these sort of things?
thanks, I think those are some good suggestions, as far as I know you could send an email to our mailling-list(google-groups) or create an new issue then cc


<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --BUG REPORT /kind bug What happened: I tried to submit a Service Replace () to the kube-apiserver and this results the kube-apiserver container crashing and restarting. PUT localhost:8001/api/v1/namespaces/<namespace/services/<servicenameWhat you expected to happen: I expected the server not to crash and to get validation errors if the request was invalid. How to reproduce it (as minimally and precisely as possible): Create a service, get the resourceVersion and clusterIP from the created resource GET localhost:8001/api/v1/namespaces/<namespace/services/<servicenameSubmit a replace command to PUT localhost:8001/api/v1/namespaces/<namespace/services/<servicenamewith body like If the same request is submitted with the nodePort fields in the ports array then the request succeeds. I get that I may be sending an invalid request without a node port (though shouldn't the merge deal with that?) but it shouldn't take down the API server. Anything else we need to know?: Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.3", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-05-10T23:29:08Z", GoVersion:"go1.8.1", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-10-11T23:16:41Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: Bare metal cluster OS (e.g. from /etc/os-release): Ubuntu 16.04.3 LTS (Xenial Xerus) Kernel (e.g. ): 4.4.0-98-generic Install tools: Others:
thanks for reporting this! Seems like this is similar to and should be fixed by . can you PTAL that PR to see if it's good to merge?
/sig api-machinery
Duplicate of


When one of the aggregated API servers returns a different definition of an object than what the other servers return, the merged openapi schema may contain multiple identical copies of the object definition (they differ only in the name). Is this a BUG REPORT or FEATURE REQUEST?:BUG /kind bug What happened: After changing the definition of the RawExtension type in one of the aggregated API servers, the aggregator returned the following openapi schema (note that v2, v3 and _v4 are identical; don't mind the actual new definition of RawExtension - it's incorrect): What you expected to happen: The merged schema should only contain two different RawExtension object definitions instead of 4. Environment: Kubernetes version (use ):
/sig api-machinery
/assign


https://k8s- https://k8s- cc: /kind bug /priority failing-test
This is at least twice in a week we've had cross break due to a public method signature change not being brought to all platforms () We should do:
I'm lost - the PR in question adds an argument and fixes the call-site - what is broken?
Ahh, I see.
How did that PR get through, if it is so obviously broken on Cross?
Cross build doesn't run 100% of the time as a pre-submit (because it takes so long), and it can't block. It's something sig-testing is tracking, but there isn't a quick fix yet.
Looks like was skipped for that PR: "pull-kubernetes-cross Skipped" not sure why yet.
Cross runs when a file matching: changes, (which needs to be fixed, opening a test-infra issue to expand the regex), but also some files could have build tags without the file name matching () Additionally cross is not blocking (), the underlying bug should be fixed now though so once we get the fix in to k/k we can make it blocking when it does run.
I find has already sent a PR. Thanks!


Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: When kube-apiserver's livenessProbe suggests that pod is unhealthy, kubelet tries to kill its container with 30 seconds grace period. The container is killed ungracefully with SIGKILL when grace period passes. This happens each time. What you expected to happen: I expected that SIGTERM will be passed to kube-apiserver which will try to gracefully stop before 30 seconds pass (at least in some cases). How to reproduce it (as minimally and precisely as possible): Inject kube-apiserver failure (e.g. ) Watch kubelet logs for container's death You will see that container is killed after 30 seconds each time Anything else we need to know?: I believe the problem is that in kube-apiserver.manifest uses '/bin/sh -c CMD' as a PID1 process which doesn't pass SIGTERM signals to child process. See Environment: Kubernetes version (use ): v1.7.11
/sig node
Looks like without the pipe command will have to be removed which will make the logs hard to track. There is indeed some which makes kube-apiserver run in background and trapping SIGTERM then send a SIGTERM to apiserver's PID.
I think this is the case for all our components, not just kube-apiserver. - FYI
How about using 'exec'-based solution suggested here: In our case it means prepending 'exec': I did changed this in my cluster and the process seems to meet requirements: fd/1 and fd/2 are redirected to /var/log/kube- container is killed instantaneously If everyone is fine with such a solution, I can prepare PR adding 'exec ' in front of all kubernetes/cluster/saltbase/salt/kube-apiserver/*.manifest using '/bin/sh -c CMD' pattern.
Not sure if understand what you mean. Before there was no logs, so /bin/sh -c wasn't needed. -based solution seems to have all required properties. What is your concern about using it?
Umm. just ignore my previous comment. I just tried prepending and it works fine for me.


<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: Now, benchmark test make docker as the only runtime. What you expected to happen: benchmark test should be generic to all CRI container runtimes Related to How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/area test /area node-e2e
/sig node


<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Priority admission controller of mirror pods. This is an error and should be removed. What you expected to happen: Integer value of Priority should be populated by the admission controller. How to reproduce it (as minimally and precisely as possible): Create a static pod with PriorityClassName set. The created mirror pod will not get the integer value of priority. /sig scheduling xref
what is the kubelet expected to do with a low priority static pod if it gets evicted?
also doesn't this mean a user can claim their pod is a mirror pod via an annotation and get it admitted with any priority they want?
knows better about kubelet evictions, but in my opinion a static low priority pod should be evicted similar to other pods. Whether it makes sense for static pods to be low priority or not is a relevant question. I would say it doesn't make sense, but one might find a reason to have static low priority pod. Not really. This check effectively bypasses integer resolution of priority causing the priority of all those mirror pods to be considered zero. This if statement is wrong and should be removed.
It also bypasses the check that prevents the pod creator from setting an integer priority themselves. In any case, agree removing the bypass is good
Static pods that fail for any reason (eviction is just one) are not re-run. To avoid evictions entirely as a static pod, users can use the critical pod annotation.
, it seems we just log error if creating mirror pod failed; refer to . Does it make sense?
I am not sure I understand the question entirely. The mirror pod exists only to show the status of the static pod. If the mirror pod doesn't exist, the static pod should still function correctly. Since that is in , it will be retried periodically.
The scheduler dependent on mirror pods for resource usage :). If creating mirror pod failed (e.g. wrong priority class name), there'll be ping-pong between scheduler and kubelet: scheduler send pod to node, kubelet reject it. But we can-t just return error, kubelet can start in standalone mode.
However critical pod annotation is ignored if pod priorities are in use.
This is not the case. See in your issue.
However critical pod annotation is ignored if pod priorities are in use. the intent of pod priorities was to replace the critical pod annotation with something more principled. is that not how it was implemented?
Priority replaces most of the functionality that the critical pod annotation provides. However, the critical pod annotation is still useful for static pods, as it can prevent kubelet evictions. But for all other use-cases, I agree that priority is a much better way to manage which workloads are run, and which remain pending. I am not sure if the scheduler translates a "critical" pod to a "system-priority" pod for scheduling purposes, but the kubelet still prevents the eviction of static critical pods as an escape hatch for important static pods which cannot tolerate kubelet evictions.
Why would that not be modeled as a system priority? I expected critical pod annotations to not be present once the priority feature was completed.
I agree with this. do you think we should not evict static pods which are system priority, so that even users of static pods can move from using the critical pod annotation to use pod priority?
I think Kubelet should not evict static system pods, but even with the current logic of eviction, if these Pods do not use more resources than requested, kubelet never evicts them as their priority is higher than other pods.
I think as a best practice users that write static pod manifests should apply a system priority to prevent eviction, but I do not think the kubelet should do anything magical about that source.


<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /assign What happened: The tests are failing again because the scheduler messages have changed yet again. I'm going to remove the check for the exact messages because this is at least the 3rd time this has happened. Instead depend on unit/integration tests to check for specific failures. Also there's no need for these tests to run across all types of local volumes because it's just testing the scheduler part.


<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: I tried to use pvs provisioned by the cephfs storageclass build from PV and pvc are OK but it failed when mounting. here is the output of the ceph-fuse mount command: I try the ceph-fuse mounting outside kubernetes environment, and found that I have to pass the --id argument to make ceph-fuse work correctly: so I add the --id option in file on line 401: and cephfs volumes can mount corretly with ceph-fuse then. Anything else we need to know?: My kubernetes cluster version is v1.6.7, running on arm64 servers, with ceph version 10.1.2, the kernel version is 4.4.58 . I backported the latest version to mount cephfs volumes with ceph-fuse.
/sig storage
Does this problem still exist in the latest k8s version?
yes, I've tested on kubernetes v1.10.0-alpha.3 with ceph version 10.2.10 and the problem still exists
OK, let me see it ,and feedback soon. =)
Hi, one question is that do you use 'admin' as user or other user to mount cephfs by fuse ?
when no "--id" specified, the default user is admin, which will result in bug when user is not admin.
/assign


/kind test-flake https://k8s-
/kind flake
cc
Saw this on
Sorry, I haven't had time to look into this. Let's see if can get to this earlier than I do.
I'm seeing this a lot in the rush to code freeze.
if this is actually a 1.10 bug, can you add status/approved-for-milestone to it? Thanks.
Something suspicious from the , but probably it's not related to the failure:
The garbage collector's behavior looks right, it does delete the dependent configmap: The tested start at 16:44:27., and failed with "test timed out after 5m0s" at 16:44:50.. I'll check if other failed cases have the same causes.
The "Kubernetes Aggregated Failures" shows there is no "TestCRDDeletionCascading" failure in the last tests (). Maybe I used the wrong tool? cc
cc because triage does not support pull-kubernetes-unit? I cannot find in as well
gave a new failure sample: https://k8s- (And the triage dashboard can't find it). Anyway, the test showed similar behavior. The tested started at I0226 20:57:02.. The gc behaved correctly: And the 5min timeout occurred 23s after the test began. I'll check what's the 5min timeout for.
The 5 min timeout is for all the tests in the test/integration/garbagecollector package. The tests are run sequentially, so is the last one to run, and unfortunately uses up the 5 min quota. The two logs I looked into showed the test itself was healthy. We just need to increase the timeout limit for this package.
found a lot of failure samples here: https://k8s- The first 11 of them failed because the 5m timeout.
Indicates this is still present and severe.
I was trying to get recent failures, it seems we recently moved intergration tests out of the unit test suites, https://k8s-, where there is no failure yet. I'll try again tomorrow.
(we rename the job this afternoon, it was named kubernetes-unit but only runs integration test)
https://k8s- that looks all green so far
Can we also check run time of the tests in that package over the past couple weeks? It seems like this just spiked. Did our tests slow down, or did we add tests to that package?
as far as I can tell the runtime of the test didn't spike ( shows the test running in just over 300s and also included the original bump of the timeout from 300s to 600s), but the timeout was accidentally decreased from 600s back to 300s as an unintended side effect of
ah, ok
Since it has passed the last 26 times, is this safe to reclose?
/close


Is this a BUG REPORT or FEATURE REQUEST?: What happened: docs recommend using '--leader-elect' to have HA setup for kube-scheduler and kube-controller-manager. However, these two components approach differently what is default value for the flag. . . Seems that setting the value to true in kube-scheduler code was lost in refactoring scheduler API config in . The code before refactoring had the value . What you expected to happen: kube-scheduler set true as a default value for '--leader-elect' Environment:
not sure I picked right SIG
/sig api-machinery
/assign
/sig scheduling


<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: When useInstanceMetadata is set to true for vmss nodes, their providerID is wrong. What you expected to happen: Their providerID should be in format azure:///subscriptions/<subscription-id/resourceGroups/<rg-name/providers/Microsoft.Compute/virtualMachineScaleSets/<scaleset-name/virtualMachines/<instance-id How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig azure
adding priority important-soon so that this issue can stay in 1.10. Please update if incorrect. /priority important-soon


Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: When creating an IPv6 cluster, the --pod-network-cidr needs to be a /66, it cannot be for a larger network (smaller value) What you expected to happen: Should be able to use values smaller then /66, /64 or a /48 How to reproduce it (as minimally and precisely as possible): Create a cluster with Kubeadm, and set the podSubnet as below networking: podSubnet: "fd00:1:2:3::/64" Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: /area ipv6 /sig network
/assign rpothier


In following with history we will be curating action items here, but it's still a WIP, atm: All non-bug-fixes for this release cycle should be merged, but there is still an ongoing list of bugs that folks may execute against for the remainder of the freeze and we will evaluating on a case-by-case basis. /cc
Thank you for creating this issue!
/cc
Adding required tags. Please use this issue to supply a frequent summary update of the issues you are tracking in the kubeadm repo and their status. Especially, please call out and link to any bugs that are blockers for 1.10. /kind feature /priority critical-urgent
can we have a general status update on the kubeadm issues? The issues themselves don't generally have status updates. This is the last week before release, where are you with the kubeadm changes?
We're all green except for the 1.9-1.10 upgrade tests which appear to have other issues that are not kubeadm specific. Final default update PR is which will auto-close this issue.
^ I don't know why we have this policy but it's ridiculous...
are there any 1.9-1.10 upgrade issues which aren't already represented by issues against kubernetes/kubernetes? that is, are there any in the kubadm repo only?
Not that I see atm, I still need to triage the inbound issues.
/status in-progress
/status in-progress
[MILESTONENOTIFIER] Milestone Issue: Up-to-date for process Note: This issue is marked as , and must be updated every 1 day during code freeze. Example update: <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : New functionality. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details


This test is failing in both gce serial suite: http://k8s- /sig network /priority failing-test /priority critical-urgent /kind bug /status approved-for-milestone cc /assign PTAL
xref
/milestone v1.10
this also fails in http://k8s-
Failing to connect to Services at all. I can't even ping pod IPs:
So I can ping pods same-node and cross-node right up until it kills the apiserver. Rebooting fixes a given node. The apiserver demonstrably comes back - I can query it for info...
- I could use help - have to shift attention soon.
FWIW, I am testing on a Calico-enabled node
I'll take a look at this as well.
Weird, now even after a reboot, this condition is sticky. I have a cluster in this state, if you want to poke.
ugh, now it is resolved.
I just repro'ed without the e2e - have a cluster, proved ping works, killed apiserver ping fails.
Interesting - docker shows the pods not running, but apiserver says they are alive.
Update: docker sees no running pods. API server thinks kubelet is running them. kubelet is getting "" from apiserver, even after kubelet restart.
Update and recap: Bring up a cluster Run some pods Prove that pods are running and services work pkill -9 apiserver Wait. Watch kubelets "lose" pods that were assigned to them Observe that pods are not running and services do not work Observe that apiserver says the pods are running pkill -9 kubelet Observe that it is still broken kubectl delete one pod Observe that kubelet is fixed for that one pod, but not others Q: Why is kubelet not getting a full listing when it restarts?
What exactly does it mean for it to be fixed for a deleted pod?
Kubelet started running the pod that was deleted - it got the update for that one.
The cluster I was testing was a week or 2 old. I built a cluster from HEAD and I can not repro - said the same. Is this test running against head of master?
I also ran the test manually against my new cluster and it passed
FWIW I was also unable to reproduce this yesterday on a cluster from HEAD.
Did some investigations and had some findings for this specific failure: Traced down a bit on the history, the first occurrence was https://k8s- From the , seems suspicious because it modified . Also found reported a relevant symptom: The fix () for that was merged a day ago and may explained why we can't repro from HEAD anymore.
nice, is this issue reproducible without the fix ()?
Sorry should have had confirmed this, bringing up a cluster now.
Quick update: I was able to repro the same failure () from HEAD with reverted. Though this test is now flaky on some other errors: https://k8s- https://k8s- They all seem to failed while communicating with apiserver.
Thanks ! So fixed the issue and I am curious about what introduced the issue. Do you happen to know when did this test start to fail? Just trying to find out what went wrong in between. We have some other possibly related issues so finding out the root cause of this one would be super helpful.
/assign
The first occurrence (for that exact failure) I noticed was https://k8s-, which happened on 2018-02-15 09:49 PST.
You all rock! I see a couple greens on it now, but still mostly red?
The test is now flaky mostly on creating an exec pod after restarting apiserver. Some sample failures: To elaborate a bit more, a snippet of the test procedure is: ssh to master and (sending a SIGTERM) Keep polling on /healthz on apiserver until 200 is returned. Create an exec pod for later steps (and failed on various erros --- EOF, connection refused, apiserver terminating). According to above test failures, my suspicion is that the test is assuming the apiserver is restarted too soon. It is possible that apiserver captured SIGTERM and did a graceful termination, so that polling on /healthz may succeed even in the termination phase. I'm not surprised trying to create a pod via a terminating apiserver will hit those errors. Could anyone confirme the SIGTERM theory?
Spoke with folks, sounds like after SIGTERM apiserver does wait a bit for remaining requests to finish. I will take a stab to refine the test. cc
Sent .
Test is now passing on gce but failing on gke. Apparently we don't have a way to trigger apiserver restart on gke at this moment. Sending a PR to skip this test on gke.
[MILESTONENOTIFIER] Milestone Issue Needs Attention Action required: During code freeze, issues in the milestone should be in progress. If this issue is not being actively worked on, please remove it from the milestone. If it is being worked on, please add the label so it can be tracked with other in-flight issues. Note: This issue is marked as , and must be updated every 1 day during code freeze. Example update: <details<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
Commented too quick. On gke we do a same-version master upgrade. Will send another fix :/


/kind bug /sig storage What happened: When mounting iSCSI PV kubelet doesn't give multipathd enough time to create multipath device and fallbacks to mounting single SCSI device What you expected to happen: it uses multipath device to mount How to reproduce it (as minimally and precisely as possible): create ~5 pods on a single node, each with dynamically provisioned PV (we use Trident for that), observe how some mounts are /dev/sd and not /dev/mapper/ Environment: Kubelet 1.9.3 Baremetal + NetApp ONTAP ove iSCSI + Trident dynamic volume provisioner CoreOS (kernel 4.14.24, multipath-tools 0.6.4)
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
I can reproduce this issue fairly often (although not reliably). The problem is in in the AttachDisk() function. The basic problem is that the method that finds the multipath device looks at each device path one time, and if none of them have an associated multipath device, it just falls back to formatting and mounting the first device path. There is no wait loop for the multipath device to appear. It would be easy to add a wait loop, but that would slow down the non-multipath use case. A possible optimization would be to only wait when 2 or more paths are known, but that still opens up the possibility of incorrect behavior when attaching while all but one paths are down, or when only one path is discovered (for whatever reason).
After thinking about this over lunch, I think a possible fix would be: add a 10 second wait loop for the multipath device if and only if the PV are 2 or more target portals specified. This puts control in the hands of the provisioner without modifying the API at all, and not breaking backwards compatibility. The one corner case I can see is what would happen the PV has 2 portals, but the system is only able to log in to one. This would occur when multipath was correctly configured on the storage controller and in the provisioner, but one of the paths was down (for whatever reason) at attach time. I'm inclined to make the attachment FAIL if the multipath device can't be found within 10 seconds, which would be a change to current behavior, but this can easily be worked around by the cluster administrator setting up multipathd to create devices for single path disks. Provisioners would want to document the recommended way to do that (i.e. set find_multipaths = no, and configure the blacklist appropriately).
is vaguely described as "target portal ips for high availability", without describing what the "high availability" means. Should we assume that it means multipath? Then Kubernetes can always wait for multipath device to appear when more that one is specified. Would it break anything? Does anyone run multiple portals for the same volume without multipath? Is it something we should support? any opinion?
yes, the iscsi portals were made possible by for HA/mpio use case (see ). The lack of concrete description makes it less obvious though.
was it working perfectly with older kube versions and the issue started to pop up recently or from ? Why I am asking is, multipath support was in kube () and atleast I havent seen this issue reported till this report or very recently. Gluster team were also using multipathing feature for quite some time where we always got mpath device mounted reliably, but noticed this issue very recently or with latest OCP builds. iic, There are good number of changes happened in between especially on device discovery/attach ( ex: ) disk paths. I am not concretely saying its a regression or a behavioral change or side effect of some other patch, but trying to figure out if thats the case. any thoughts here ? my understanding was/is that with more target portal IPs means ( atleast) in ISCSI context, but we could have mention as well in the description. Suppose the backup portal IPs are filled by provisioner/admin but initiator havent enabled ( dm_mulitpath is not loaded or deamon service is not running) software in the kubelet/node, what will happen ? Is it going to be infinite wait for failure ? my understanding is that, if multipath is enabled, even if it failed ( for some reason) for backup paths, a mpath device will be created for single path in the system. If thats the case above shouldnt be a concern and device mapper layer will take care of the path discovery later when path becomes stable/available. I dont think so.
we started with 1.9.x on kubelets, so can't tell much about mounting behaviour of previous versions. it is going to fail, but it is fine, nodes should be correctly configured. Lets say you specify fsType, but given FS module is not loaded by kernel, or now mkfs.<FSbinary is available to the kubelet, mount is going to fail and thats OK IMHO. Kubelet shouldn't try to fallback to anything less than was asked, mounting via single portal when multiple portals are specified is similar to trying "default FS" if required fsType can't be used. Yes, default configuration for multipathd is option (note, that default in Linux distributions might be different), which creates multipath device for every non-blacklisted block device.
I'm fairly confident that adding a wait loop for the multipath device in the case when there are 2 or more portals in the PV spec won't cause any regressions. The thing that gives me pause is that there are cases when there's a single portal in the PV spec, but the device still supports multipath and the existing sendtargets command is the mechanism for finding those other paths. I'm wondering if it's possible to detect that case (reliably) and use a wait loop for the multipath device in those cases too.
is the default on RHEL 7, see
Did you mean Multiple Connection Per Session (MC/S)? I don't think this is supported in open-iscsi. Open-iSCSI only supports MPIO (multiple portals)
I don't understand the move by RedHat to default to findmultipaths = no. It basically guarantees bad behavior in cloud-oriented use cases when there's a temporary path failure. My reading of the linked document is that the main motivation for changing the default is to avoid the need to setup a correct blacklist. It seems to me that correct behavior is more important that avoiding some work up front when configuring multipath. No, I'm referring to the . If a storage backend has multiple target portals, but the dynamic provisioner only puts one portal in the PV spec, then as long as that one portal is accessible at attach time, open-iscsi will discover the other portals and log into them. The only weakness to this approach is what happens when that one path is inaccessible at attach time due to a network switch/network port/NIC failure, while the other ports are still available.
, AFAIK multipathd (binary) in RHEL 7 defaults to , but configuration utility they ship writes config with , that utility needs to be called explicitly , IMHO that should be direct scsi mount unconditionally, without multipath involved
The thing that gives me pause is that there are cases when there's a single portal in the PV spec I would be inclined to agree, except that someone the sendtargets logic to the iscsi code. This code exists for no reason other than to allow finding multiple path given a single path. In fact the existing NetApp dynamic provisioner (Trident) relies on that behavior currently to get multipath support (this is something I've fixed and will upstream shortly). If we decide that PVs with exactly one portal should not support multipath, then we should remove that sendtargets logic from the iscsi module in kubelet.


Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Service loadbalancer source range don't work in ipvs proxy mode. Which result in gci-gce-ipvs CI failed. http://k8s- How to reproduce it (as minimally and precisely as possible): In gce environment. a service with backend. service type to service util service external IP is specified. the service from pod inside and pod out of the . Environment: Kubernetes version (use ): master Cloud provider or hardware configuration: gce OS (e.g. from /etc/os-release): ubuntu Kernel (e.g. ): Install tools: cluster/kube- Others:
/kind bug /sig network /label ipvs
/area ipvs
The test seems to be flaky now. Can you take a look please?
I don't know why, but I think this flaky have the similiar reason with other normal service-test cases, like Those flaky are pretty annoying, problem is, I have run those cases in my own environment many times, never fail... : ( But still, I'll do my best to find the reason.


New tests for subpath consistently fail in ci-kubernetes-e2e-gce-multizone: https://k8s- /kind bug /sig storage
Is this a blocker for 1.10?
I would expect so, yes. /milestone v1.10
I'm on it /assign
The problem is likely due to inline GCE PD volumes not having enough zone information. Considering changing volumes to PVs instead.
fix was in progress in to change to a PV
/unassign
[MILESTONENOTIFIER] Milestone Issue: Up-to-date for process Note: This issue is marked as , and must be updated every 1 day during code freeze. Example update: <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details


<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: e2e test "gci-gce" is running in 2 versions: with kube-dns as DNS Service : https://k8s- with CoreDNS as DNS Service : https://k8s- In CoreDNS use case the test is failing What you expected to happen: This e2e tests should not depend on what DNS service is running as default in K8s. CoreDNS is currently BETA, will move to GA soon (after v1.10 though) See : How to reproduce it (as minimally and precisely as possible): look at logs of dashboard : https://k8s- go run -v --test --test_args='--[sig-autoscaling]sDNSshorizontalsautoscalings[DisabledForLargeClusters]skube-dns-autoscalersshouldsscaleskube-dnsspodssinsbothsnonfaultysandsfaultysscenarios$' Anything else we need to know?: Environment: Kubernetes version (use ): branch master Cloud provider or hardware configuration: GCE Install tools: deployment intra-test Others: see https://k8s-
/sig autoscaling
/unsig cli
Looks like the test is searching specifically for a deployment labeled "kube-dns". There are several references to "kube-dns" scattered around it, actually.
Right now the tool "dns-horizontal-autoscaller" seems dedicated for kube-dns. This tools is deployed as "add-on" when enabled by this ENV variable : I guess we should have the definition of another dns-horizontal-autoscaller, dedicated to coredns (unless we generate the yaml file). and the choice of deployment is based on the ENV variable . see
Currently, the translation of the deployment of "dns-horizontal-autoscaller" for CoreDNS is in progress by It is expected that the corresponding e2e test will work as is (as the artifacts are selected by labels, which are the same for CroeDNS and kube-dns).


/sig gcp /sig cluster-life-cycle /kind flake /milestone v1.10 cc https://k8s- starting last night /assign can one of you verify?
/kind bug /priority important-soon
/priority critical-urgent /priority failing-test /status approved-for-milestone
[MILESTONENOTIFIER] Milestone Issue Needs Attention Action required: During code freeze, issues in the milestone should be in progress. If this issue is not being actively worked on, please remove it from the milestone. If it is being worked on, please add the label so it can be tracked with other in-flight issues. Note: This issue is marked as , and must be updated every 1 day during code freeze. Example update: <details<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
/assign according to , gonna move those jobs to us-central1-c instead


/kind bug What happened: We had a pod with persistent storage. We drained the node. The node VM was deleted. The PV was not attached to the node that the Pod migrated too. The kube-controller-manager logs contained the following. What you expected to happen: The PV can be attached to the node it was migrated to. How to reproduce it (as minimally and precisely as possible): Create a 2 node cluster on vSphere Deploy a pod with persistent storage attached Drain the node Delete the node VM immediately after draining Anything else we need to know?: The issue seems to be in the call to getVSphereInstance. This will return an error if the Node doesn't exist. Precluding the logic on line 804-813 from being executed. A restart of the kube-controller-manager resolved the issue. The following e2e test is not triggered in vSphere environments - Which might expose the issue. Environment: Kubernetes version (use ): 1.9.5 Cloud provider or hardware configuration: vSphere OS (e.g. from /etc/os-release): ubuntu Kernel (e.g. ): Install tools: CFCR Others: /sig storage /sig vmware
More logs:
/assign


<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: Pods stuck in terminating state on Windows node after deleting it What you expected to happen: Pods get terminated and should disappear from kubectl How to reproduce it (as minimally and precisely as possible): Run a pod with windows container Delete it Pods are still terminating Anything else we need to know?: Could you pls help to resolve this issue. K8s pods stuck as Terminating after getting deleted on windows node with windows container(IIS). Docker ps -a not showing this container, so it is completely deleted, but in kubectl it is marked as Terminating. For networking we using OVN. kubectl logs: kubelet logs: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: Hyper-V VMs OS (e.g. from /etc/os-release): Windows Node: Master Node: NAME="Ubuntu" VERSION="16.04.4 LTS (Xenial Xerus)" ID=ubuntu IDLIKE=debian PRETTYNAME="Ubuntu 16.04.4 LTS" VERSIONID="16.04" HOMEURL="" SUPPORTURL="" BUGREPORTURL="" VERSIONCODENAME=xenial UBUNTUCODENAME=xenial Kernel (e.g. ): Linux esd-kbrnts-lnx-1 4.13.0-37-generic ~16.04.1-Ubuntu SMP Wed Mar 7 16:03:28 UTC 2018 x8664 x8664 x8664 GNU/Linux Install tools: OVN 2.7 on windows OVN 2.7.2 on linux Others: Docker Version on Windows node: Server Version: 17.06.2-ee-6
/sig windows
It needs sometime for kubelet to remove the pod. Could you wait a while and check whether the Pod removed?
Hi actually I have waited for 1 day, and it is not deleting. Can delete it only using --grace-period=0 --force. But I don't want to use this manual approach). Could you pls give some advice on this? Thanks in advance!
is a reference to this issue. Checking kubelet logs will always help to troubleshoot such problem.
Actually, I have attached kubelet log in the ticket and think the main reason is Also as there any option for kubelet to automatically force delete Terminating pods after some period? And --containerized is also not solving my issue. Thanks!
Kubelet won't delete the pod from apiserver. Since it is already not exist on the node, you could force delete it by kubectl.
It is clear, so how we can resolve this error: ? I think this is the core item of our issue. Thanks.
I'm not using OVN. Have you met this problem when using OVN?
adding Bob to comment on this. i have seen this issue before, but it has been quite a while.
Hi, Guys will be very grateful if you will provide some solution for it. We have tried to add to deployment spec but it doesn't help. Thanks.
The error message of is actually a warning message. It shouldn't block the removal of pods. Could you recheck the following things: the node, check whether the pod is still existing, e.g. by the pod exists, check if there are other error or warning message in kubelet logs; or else, the pod could be force removed safely from apiserver
Hi, I had posted this info in the ticket description. not showing this container, so it is completely deleted, but in kubectl it is displayed as Terminating.
i think you may get more support and faster responses in our slack channel for sig-windows.
So it could be safely force removed from API. And agreed with slack is a more faster way to such troubleshooting issues.
The issue seems to be related to . The only platform specific implementation for PodContainerManager is the one for . There is a default implementation (), that always returns true in the Exists call. Disabling the CgroupsPerQOS option on the Windows nodes resolves the issue.
Windows doesn't support cgroups. You should always disable it.
it is clear enough that Windows does not support cgroups). Unfortunately, it's not specified in the documentation. So to disable CgroupsPerQOS you should need to add this flags to kubelet: Maybe it will help someone... Thank you all for discussion!


Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: In the kubelet, when using the kubelet creates static pods but doesn't update the status accordingly in the PodList (kubelet ). You can find that kind of Pods with the following annotation: The status will stay like that: What you expected to happen: The status is updated. How to reproduce it (as minimally and precisely as possible): Use a static pod. Anything else we need to know?: I tried to fix it in but the test-grid continuously failed at the reboot phase (see ). I had to revert it (, ). I'm creating this issue to track this bug. The problem in the e2e tests are probably from . This is an old output of the failure: https://k8s-
I don't think this is a bug. only exposes what kubelet sees from the sources, and doesn't include statuses known only to kubelet. Since for static pods, their statuses are not written back to the source, they do not show up in . The fact that it has the condition is unintentional and a bug. It has just been fixed recently.
I confirm that this behaviour is still happening. Having pods with status where they are actually . The kubernetes API shows them as when Kubelet endpoint as .
/sig node
I agree with that this is working as intended.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
I understand that the source of truth is currently the API server and that details can be retrieved there. We have however found this to be a scaling challenge. In our use case, we are a monitoring application that runs as a DaemonSet. Monitoring systems such as ours need to continuously refresh the PodList because Kubernetes operators want a real time feedback on application health via metrics, logs and traces. Having each node that is monitoring query the Kubernetes apiserver to get each of the Pod statuses results in high load on the API server. This is exacerbated and becomes difficult to scale as clusters grow to a large number of nodes. We have however found that having each node query its own Kubelet API locally is quite scaleable and allows this work to be distributed across the cluster. Ideally we would be able to use this API for static pods as well since the state of the static Pods is stored inside the kubelet. Other benefits of using the kublet API are that: It provides the link to the container runtime in the section, which is helpful for metadata collection and association. Is easily managed the RBAC system through webhooks If there is another approach or API that you’d suggest we’re happy to rethink how we collect this data.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
I also meet this issue.
Reporting the status as "Pending" might be consistent, but it's incorrect. If the pod is created statically and is health checking OK, it's not "Pending". The Kubelet API is almost very useful but the status flag seems to make it much less so since it's not possible to tell from the API endpoint if a static Pod is actually up successfully or not.
Hi team, I'm reaching out to see if we can revisit this issue. We need this to be able to monitor pods locally without having agents reach out to the apiserver. Currently the pod is in a wrong state (Pending when it's actually Running), and we're missing all status information (pod IP, container statuses, etc.). We would be happy and willing to update to solve the issue if we get the green light that this will be accepted. cc
I think as long as we fix the test failures the change caused, we can re-introduce it. Feel free to assign me on a PR which accomplishes this, and make sure the reboot test passes.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale


Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: says: What you expected to happen: Don't suggest that -- as it doesn't actually do that when used. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: (This was reported in but that issue was closed) Environment: Kubernetes version (use ): (This is indeed head from today) Cloud provider or hardware configuration: macOS OS (e.g. from /etc/os-release): Kernel (e.g. ):Install tools:Others:
/sig cli
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


https://k8s- Please work on finding a fix for the flake. /priority failing-test /priority important-soon /sig api-machinery /kind flake
/assign
I think this was fixed by
we had a failure on 3/28 (after the above fix merged). https://k8s- Looking at the logs this seems to be a different error than the one fixed by . Can you confirm plz?
Hmm that is different, and I will look into the reason.
There hasn't been a failure since 3/28, So while the test might have flakes it is definitely less flaky than 12 failures per week now
Yes the latest runs look good. Please close the issue
/close


Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Many service related test cases in test case are flaky. Especially related cases. CI Link: http://k8s- What you expected to happen: IPVS should sync rules within . How to reproduce it (as minimally and precisely as possible): a cluster with ipvs proxy mode, and specify . a service with backends. check the delay from create service to sync ipvs rules. (This issue is probabilistic, about 50% possibility to happened.)
/sig network /area ipvs
/assign


Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Deploying kube-proxy with mode ipvs on a system with the required ipvs support compiled into the kernel rather than as modules causes the kube-proxy deployment to fall back to using iptables. tryIPVSProxy is only checking for the presence of kernel modules and not if the running kernel has the required support compiled in. What you expected to happen: tryIPVSProxy should succeed if either the required modules are found or support is compiled into the running kernel. How to reproduce it (as minimally and precisely as possible): Boot a system using a kernel with ipvs support compiled in rather than as a module (such as the kernel used by and a host without iptables installed Add --config=/etc/kubernetes/kubelet- to the kubelet arguments using the following kubelet- contents: Install using kubeadm with the following config: Check the output of the kube-proxy pod: Anything else we need to know?: From , it should be possible to check for the existence of modules.builtin and if present use that in addition to using modprobe and the contents of /proc/modules. Environment: Kubernetes version (use ): v1.10.2 Cloud provider or hardware configuration: local kvm host OS (e.g. from /etc/os-release): custom linuxkit build Kernel (e.g. ): Linux linuxkit- 4.14.40-linuxkit SMP Thu May 10 10:06:57 UTC 2018 x86_64 Linux Install tools: kubeadm
/sig network
/cc
cc
cc please take a look.


both in PR and CI runs see /assign
/assign
Interesting fact: is not flaky, only . The test codepath for these two are almost identical. I wonder if this is a real bug not a flaky test. still investigating.
still seeing flakes like https://k8s- the loop that ensures the watch cache is primed goes version-by-version, and creates and observes the delete event for that version. it does not ensure the delete event is drained from the watch caches for the other versions.
a better approach would be probably be to:
why the poll? Because the watch is not reliable? Otherwise, we have that structure now.
Moves the version loop inside the poll Today we loop over versions and poll per version, which can leave unconsumed delete events in prior versions
Cannot follow. Now we have the following without a poll:
At minimum, I would move the version loop inside the poll so that when we exit we know the deletion events from isWatchCachePrimed are consumed by the watch caches for all versions. If we wanted, we could move the create/delete up out of the isWatchCachePrimed function or pass versions into it to avoid creating/deleting an object per version
Ok, so we were talking about different code. good. To understand: we do a call per version, each creating and deleting an object, and waiting for the delete event with a watch call. a following watch will not see the events of the calls of itself (because they use the same versioned watch cache) and of any version in front of it in the version slice (is this true?), but it might see events of calls of versions later in the version slice. So with your outer poll and inner version loop we would make sure that we drain every version's cache for following rv=0 watch requests. Sounds plausible.
Where do we have the official watch axioms?
I think I know what is the problem here. We make sure the watch is primed for . then we move on to and make sure the watch is primed by creating another object in . The watch cache is drained for but the watch cache for is not and that is the problem here. will send a fix.
yep, same thing I described above. The solution looks good.
I didn't see your comment when I commented. Your description is accurate.


<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: During k8s cluster launch, is checked and if any etcd binary is found regardless of current OS Distribution, expected etcd binary is not downloaded and etcd soft link is not updated to point desired etcd binary, so cluster creation fails. What you expected to happen: Along with expected etcd version, OS Distribution should also be checked for successfully and efficiently installing etcd during setting up local cluster. How to reproduce it (as minimally and precisely as possible): Checkout k8s codbase on MacOS and launch cluster. working k8s codebase under virtual machine to launch cluster in VM. (Cluster creation would fail, prompting error that etcd needs to be installed and update in {KUBEROOT}/thirdparty/etcd-v{ETCDVERSION}-linux-* Anything else we need to know?: It's an optimization on managing thirdparty etcd binary for local cluster setup. Environment: Kubernetes version (use ):Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
do you have a patch that you are working on? we don't really support local-up-cluster on macos yet. /sig testing
is fixing this issue. Though I use Mac as my workstation but sometimes need local-cluster to test codebase so I mapped (softlinked) my GOPATH in virtual machine keeping separate bin directory. Everything is running perfectly fine after making changes mentioned in this PR. So thought to submit small optimization on etcd installation logic.
/close


<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: In order to migrate to a new version of kubernetes on GKE. I cordoned(Kubectl cordon <node) all nodes in the old node pool(1.8.9) in order to drain the node pool(pods on the node) with an intention to cause the pods to run on a new version of the node pool(1.10.2). As soon as I cordoned all the nodes in the old node pool, it caused the downtime of ingress(nginx). The nginx controller pods where running fine on the cordoned node pool. But the cordon caused the nodes to be removed from the target pool of ingress load balancer. Below are the documents that suggest the above same method that I followed as best practice for zero downtime upgrade. 1). 2). What you expected to happen: I expected "kubectl cordon <node" not to have the node/s removed from the target pool of the loadbalancer because the nginx controller pods where running absolutely fine on these nodes. How to reproduce it (as minimally and precisely as possible): 1).Cordon all the nodes in an old node pool that runs nginx controller. Anything else we need to know?: Environment: Kubernetes version (use ): 1.10.2 Cloud provider or hardware configuration: GKE OS (e.g. from /etc/os-release): Container OS Kernel (e.g. ): 4.14.22+ Install tools: Others: Cordon causes to remove the nodes from the target pool of the ingress load balancer. The health checks fail on the nodes of the target pool, which should not be the case. Because the nginx controller pods run fine on these nodes.
Technically, this isn't a Kubernetes bug as this is working as intended; however, it does beg the question why we remove unschedulable nodes if they're marked as ready since we perform the to determine LB backend nodes. is it scheduable? is it not a master? is it not marked for exclusion? is it marked as ready? If any of those are a no, then it'll be removed from the pool on the next sync. Perhaps someone from can shed some light. /sig network
cc This problem extends beyond GCP/ingress controller and to anyone using pools to upgrade nodes. A typical workflow I see is you'd cordon a node pool so as you drain each node the pods don't go to another node in the pool that'll shortly be drained as well. The current logic does not support that and complicates cluster administration because cordoning does not respect PDB's.
/remove-sig apps /remove-sig gcp (since the service controller falls under sig networking)
Those labels are not set on the issue: <detailsIn response to : /remove-sig apps /remove-sig gcp Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Ref I thought a general practice for draining the old pool is to move over one node at a time in a rolling fashion. Shouldn't there always be certain backend pods being able to receive & serve traffic? cc
WRT draining, it is recommended to do it one node at a time ; however, in real world scenarios doing it one node at a time can cause a lot of avoidable container churn if you don't cordon off the nodes you at least plan to drain so that pods aren't being scheduled on a node you'll be draining after the next one. According to the comments in the service controller it seems the unschedulable part exists because it was how we differentiated masters in the past; however, we have labels for that now. We also have the service exclusion label (can't find when/if that'll be promoted though as i don't think there's a ticket on it).
I'm having problems with this issue as well. The fact that nodes are removed from target pools when cordoned means that I sometimes get brief interruptions of the traffic to a service when I cordon a node. I'm guessing the node isn't gracefully removed from the load balancer which results in some traffic being cut off trying to route through the recently cordoned node. The same problem occasionally happens when deleting nodes / node pools or when the cluster autoscaler deletes a node. We are running GKE ver. 1.10.2-gke.1. Our best defence against these issue have so far been to limit node changes as much as possible and not use any auto scaling.
I've posted a question regarding this issue in SO some weeks before it was created. The suggested action was to use node taints () instead of cordon, that way nodes will still be marked as .
Thank you for the link, it Certainly helped. Perhaps, if thats the case(as mentioned in the accepted answer of the above link), then to cordon all the nodes of old node pools should not be a recommended action, at least in the GKE upgrade docs.
only says: And I can't find what more it does in the documentation. So I would expect that it has exactly the same behaviour as a taint. Why is removing running pods from the balancer working as intended? If it isn't a bug then the documentation is certainly unclear in this regard.
This just caused a major outage for us too. I agree with the previous posts, while this may be working as the code intended, it does not follow the patterns and documentation k8s provides. When I cordon a node it should not impact the traffic going to the node or any pods on it. I should be able to gradually remove traffic by draining nodes/pods and respecting PDBs.
this issue touches on the fact we are removing masters as well and inadvertently removing cordoned nodes. The logic in the service controller mentions we only filter out unschedulable nodes since that's how masters were setup.
We wanted to replace all of the kubernetes cluster nodes to different node instance types, so we started off by cordoning all of the nodes, but it surprisingly caused an outage for us. As pointed out, the only documentation around cordoning is , which I took to mean that new pods can not be scheduled on those nodes. It was surprising behavior to find out that, not only does it not allow new pods to be scheduled on those nodes, but that it also causes the service controller to remove unschedulable nodes from load balancing pools. I understand that it's working as intended, but if that was also documented as part of the cordon operation, I would have been able to avoid an outage.
I finally caught up to this bug. Fun one. I agree that this is working-as-intended and that it stinks. We can do better. As a short-term workaround, you can taint the nodes before cordoning one-by-one. That will stop new work from arriving, but will leave old work there. If we agree on an appropriately named taint, we can have nginx tolerate it by default. Am I missing any reason why this would not work? If that flies, we should fix docs ( those are GCP docs). I am happy to help with that. Longer term maybe we can do something smarter. I am not sure what that would be, just now. Ideas?
Is there any reason we can't just leave the nodes in the load balancer pool when the node is cordoned? And when the pods are terminated the nodes can be removed? I may be missing some context here?
The presumption is that a node that is unschedulable is going away, or needs some other repair. Using it as a target for load-balancers means there could be sessions routed THROUGH that node. Even as the pods are drained. Right up until the node suddenly dies, causing an outage of a different flavor. What should really happen is that a node which becomes unschedulable starts draining connections. Once all the pods are moved off, the node can be removed from the pool entirely. I don't think any cloud providers, even GCP, handle this just right today. David Radcliffe <:
Wait, if you brought up a new nodepool, all those new nodes should be in the TargetPool. Cordoning the old nodepool should only terminate the existing connections, new connections should be fine. Something doesn't add up...
Was this using
Yes
Right, well, that explains it. So even draining won't do what you want - you need new connections to be allowed until you have capacity in the new pool. The good news is that you can do this entirely yourself, with taints. We just need nginx to tolerate the taint, which you can also specify yourself. The only real alternative I see would be, as you said, to leave nodes in the LB pool until the node dies. Given that we never specified this, it is probably different on every provider, which would make it hard to change (especially if the provider already provides clean drains while unschedulable).
IIUC, we don't want new pods to land on the tainted/cordoned nodes. In that case, we can place "NoSchedule" taint on the nodes. We don't need any tolerations on nginx. This will prevent new instances of nginx from landing on the tainted nodes, but the existing ones will keep running there.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
This just surprised us also. I understand the technical reasons behind it but the command name and associated documentation is misleading. To me it isn't intuitive that the command does not remove compute (pods) but does remove some networking. I would expect cordon to keep new pods from getting scheduled to it and keep it from getting to new services/ELBs/etc. The command would then cause pods to be removed and also take the node out of networking paths. Almost need a new state for a node of that is clearly defined.
My main issue is with the docs, and also GKE docs. They should explain this pitfall. After I got aware of the issue I understand why it works as it does. A simple workaround is to taint the nodes you are going to rotate first.
Just chiming in; We as a GKE customer have today been bitten by this issue in production. The documentation of (in the context of GKE) is not extensive enough so we could be aware of this issue. This should be addressed. Luckily the outage was not severe because we were using the node-by-node approach (and accepted the container churn this would cause). We explicitly took this approach because our previous nodepool upgrade was a total outage also caused by in a different way. (for those interested in our total outage, not related to this issue): Due to the fact that also results in the re-calculating the amount of schedulable nodes and thus scaled the amount of pods in our cluster (in our case: from 110 back to 2) resulting for us in major DNS outage internally. Be warned that for a nodepool upgrade can have a lot of unwanted & unexpected side effects.
It looks like this happens only to AWS and GCP but not on Azure? Can anyone confirm this? I just tested in both AWS and Azure, and while I see logs like in AWS Kubernetes, I do not see any logs like that in Azure. Is it intended? If so isn't behavior different for each cloud provider? i.e.) cordoning a node in AWS Kubernetes prevents loadbalancer from routing traffic to the affected node but cordoning a node in Azure Kubernetes will still let the loadbalancer route traffic to the affected node. Is it intentional?
Besides running a test like the above, how would I determine what the behavior is on AWS? I haven't found anything in the node administration docs or the Service that would imply that would shift traffic away from a Pod. In fact, the doc for explicitly states, emphasis mine: If I hadn't come across this Issue due to reports from GKE, I suppose it would only have been a matter of time until we had a high-impact production outage. There's a disconnect between Kubernetes API and the LoadBalancer as to whether Pods running on a 'd node are Ready. From all the docs I've seen, I would expect that either a) would evict Pods over to a Node before setting the instance as , or b) LoadBalancer would not equate with
Hi All, What approaches people using too workaround this? From reading the comments I was hoping to follow the custom taint approach. However, I've noticed executes a cordon making it difficult to drain the node safely with this approach.
I was lucky in that I experienced this issue while cordoning our testing environment, so we didn't experience a production outage. I think what I'm going to try is the following... it's going to be time-consuming but I think it's the safest bet. Double the number of nodes in the server group to bring online the new nodes first Place a custom taint on each "legacy" node As you said, executes a first, so that's a no-go. This is the time-consuming part... for each "legacy" node I am going to each pod, and let my PDBs take care of maintaining minimum number of pods. I'm hopeful to try this approach this afternoon, I'll comment back here if I do!
Same as myself! Pity neither of us have a story for Since my comment this morning I've been trying a few approaches, I've ended up with the following: Create a new node pool of equal size in GKE with the upgraded kube version Set the current node pool to not be able to grow beyond its current size - if your using regional nodes this can be annoying, let it bring up new nodes anyways and just have them taint as NoSchedule straight away Taint all old node pool nodes with NoSchedule Go through each old node pool node executing a taint of NoExecute which will cause the node to drain and go onto available nodes i.e. your newly provisioned node pool Ensure your nginx ingress is deployed with tolerances:
Hi All, I'm at a bit of a loss on this one. I've started using custom taints to drain the nodes without getting them removed from the load balancer. My application can hold a HTTP connection open for up to 600 seconds. I'm using nginx as a daemonset, have it configured with a keepalive timeout of 620s, and have . I drain the application first using a taint on each node. I have nginx on each node tolerate this taint and remove it separately at a later point using a different taint. I'm waiting 660s between drains of each node for each taint to ensure nothing is dropped - this all works fine as expected. However, once I kill the node pool or do a kubectl drain after the node is empty of all my stuff plus nginx ingress I start seeing connection issues. I'm a bit lost where to go at debugging them, best guess is something to do with kube-proxy. Can anyone shed some light on this? is there anything on the Google Layer 4 load balancer I'm missing here? Is there some way to correctly drain kube-proxy (or does that even make sense?)
This doesn't address the network issues you're seeing, but for those trying to "roll their own cordon" using taints: Using a NoExecute taint to "drain" the node won't be orderly and won't respect any PDBs. You want to replicate the logic from which tries to do evictions of every pod in a loop. (I realize this is a burdensome idea, but it is also the safest idea.)
To add on to ,: One approach that does not use node taints to force migrate, is to change for the requirements of your app. A change in node selector (e.g. based on instance type), along with a PDB (e.g. 66% / out of 3 replicas availability) should migrate you over fine. This worked out better for me than to drain or add taints to nodes to force push apps out. After the migration completes, you should be able to the nodes, and delete the node pool. FWIW, I too saw connection issues during load testing of a migration, and at least on my setup it was due to using an AWS NLB for the NGINX Ingress Controller. Switching to a classic ELB instead worked, as the NGINX LB Service seems to hit some sort of by my accounts when using NLBs. self-plug: I happened to recently cover a similar topic in a that you may find relevant. Hope it helps
For better or worse, for in my case pods that are behind an ingress resource are using node affinity to preferably not schedule them onto the same node. Originally this was done to ensure traffic is balanced across the different nginx ingresses. It has had a positive affect in this case too. Very interesting! Thank you for sharing, I'm not sure that swapping out the Layer4 GCE load balancer is an option here, but you have me thinking. I'm wondering if the TCP keepalive between the LB and nginx is potentially keeping connections open. , and suggest that there may be a 600 second TCP keepalive going on. Will retest doing a sleep of 620s after nginx removal and before node drain.
Has anyone come up with a strategy to safely move workloads between two pools? Reading this bug, it seems there's no way to reliably do this without causing an outage if you follow the recommendation of + . In my case I'm not using an nginx ingress but services with type . From what I can see, adding to a node causes the Service Controller to consider the node not ready and thus removes it from a target group. This would mean that between and there is no way to add a taint to the node that will leave it in the LB until it's drained. Using doesn't work in this case since the thing I'm draining might just come back and I'm not sure if the service controller will still consider the node ready in that case, but I believe it does. The one strategy I can think of is using labels on the node and an anti-affinity on the pod. You'd then add the label to a node that would push the pod away and then delete the pod. As long as the anti affinity uses this should be safe to do in preparation of a migration. Once the pods have been deleted you could then the nodes.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/lifecycle frozen /remove-lifecycle stale /remove-lifecycle rotten
- As I mentioned in I set up a script that first taints all nodes in the old node pool and then cordon+taint one by one node. If your ingress services are redundant this should work well.
Is there a plan for cordon behavior to improve on GCP? We have relied on this behavior for years on our AWS clusters and it was very surprising to find out.
This is a bug with service load balancers - not something we should document. We defined a label in that cluster admins can use - Unschedulable has nothing to do with SLB inclusion. which is beta and on by default in 1.19.
Disagree. Unschedulable means "no new workloads". It has nothing to do with existing workloads, that is why we have drain (but we should have something in kube that communicates nodes are draining). I have opened to remove this check - we have an alternate approach now and the old logic is wrong (the comment in the code even says "this is for masters!" and that's wrong now too as part of the kep).
A node status condition?
Possibly, or a taint, or a label. Something concrete.
/milestone v1.19


Is this a BUG REPORT or FEATURE REQUEST?: /kind feature What happened: I need to know to which Pod(s) a PVC is being mounted to, without having to go through each Pod one by one checking the Volume mounts and the PersistentVolumeClaim. The kubectl describe pvc and get pvc, only tell you to which Persistent Volume the PVC is bound to, but not if is being used by a Pod(s) What you expected to happen: Add a new field in the describe or get that lists the pods that are using the PVC as a volume mount How to reproduce it (as minimally and precisely as possible): Have a PV/PVC bound and create a pod using a volume mount Anything else we need to know?: Not sure if this is only for Azure Disk, which is the Storage Class I am using. Environment: Kubernetes version (use ): 7 Cloud provider or hardware configuration: Azure OS (e.g. from /etc/os-release): Ubuntu 16.04 Kernel (e.g. ): 4 Install tools: Azure Kubernetes Services Others: -n/a
/sig storage
/sig cli
I think the kubectl describe can just construct this on the fly. Another angle is should we add this information to PVC object itself
That will be great, as right now the only way to correlate a PVC to a Pod is via a "describe" on the Pod.
I can work on a PR for this!
GitHub didn't allow me to assign the following users: clandry94. Note that only and repo collaborators can be assigned. <detailsIn response to : I can work on this! Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
+1 for the feature. cc
How to find "what pods are using particular PVC through command line"
+1 for the feature.
this feature will be in Kubernetes 1.12
ok kubernetes 1.12 released, how to? which command?
Can someone point to the documentation or paste command to get the pods which have the PVC in use
see the associated PR to see an example usage of this The line in should show it.
Workaround for the meantime:
Thanks , this workaround is great for those people who still use old kubernetes version. However it will meet an error if the pod have no volumes, so I improves it by adding a to check whether the volumes exists or not.
python script to list pods with pvc in namespace(s): Is a part of a cli to set pods with velero/restic annotations. Hope it helps someone :)
For posterity... There is a Used By: field in the output of describe.
client-go didn't give this api... ,PodPVCIndex can solve the problem
Hi thanks worked for me ;) but I want to make it reverse my workarounds with adding after didn't worked do you have any ideas
$ kubectl describe pvc -A Used" Name: ocs-deviceset-lso-volumeset-0-data-1b45dn Namespace: openshift-storage StorageClass: lso-volumeset Used By: rook-ceph-osd-1--8fwpd Also useful ^
the reason it didn't work is because each return does, in point of fact, have a key defined for claimName. That key is just set to null value. So, what you'd actually want to do is test for nulls. As nice as answer is, it's a PITA to parse unstructured human-readable command output, as opposed to command output in a markup language that well known shell parsers like jq and yq can work with. Towards that end, because I had use of this, I came up with the following: That simple not null test at the end plucks out all your nulls. Obviously invert the last conditional if you want all the nulls, but you'll get all sorts of temp things that way that you probably won't care about.


Is this a BUG REPORT or FEATURE REQUEST?: /kind bug /sig network /area kube-proxy /area ipvs /assign I'll send a fixing pull request tomorrow. What happened: On centos 7.2 with ipset 6.29, kube-proxy in ipvs mode failed to create ipset sets. After some dig, it turns out to be problem of ipset on centos/rh. See I tried ipset 6.29 on ubuntu, it's OK. But when I tried ipset 6.29 on centos, I got errors: What you expected to happen: Since the problematic ipset versions have very large installation base, we could remove ipset commet, or remove ipset commet with problematic ipset version. The problematic ipset also says it support comment by --help. The problem could be detected by a test run of "ipset create foo hash:ip comment". How to reproduce it (as minimally and precisely as possible): Create some service in kubernetes, run kube-proxy with ipvs mode on Centos 7.x node. Anything else we need to know?: Environment: Kubernetes version (use ): commit (actually all commits after ) Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): CentOS Linux release 7.2.1511 (Core) Kernel (e.g. ): Linux VM34centos 3.10.0-514.26.2.el7.x8664 SMP Tue Jul 4 15:04:05 UTC 2017 x8664 x8664 x8664 GNU/Linux Install tools: Others: ipset v6.29, protocol version: 6
could you test it before commit , that's ok?
, yes, I did. It's OK. But I think the we don't need to revert the iptables comments.
if ipset package not installed, will set module or match-set option in rule be weird and now seems not.


<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: After merged, the leader election is calling too many api calls. Previously it calls with a 2s interval by default. Now it calls with 0 interval. Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig api-machinery
/assign
/kind bug


What happened: Data race in tests in an unrelated PR Logs https://k8s- /kind bug /area platform/vsphere /area cloudprovider /sig cloud-provider
/kind flake
Another time it flaked in an unrelated PR https://k8s-
bringing this to your attention as a vsphere dev.
/assign
Looks like the race was in Feb 2017 here: But that code path wasn't covered by go test until vcsim support was here: I'll work on a fix.


Is this a BUG REPORT or FEATURE REQUEST?: What happened: The output of my kubeadm: When I pull the image with docker pull, I get such output: But other images are successfully pulled as we can see from previous log. Why?
/sig cluster-lifecycle cc
Ugh can you fix this please? I missed this in the PR that just got merged, should be without the arch suffix as this is a manifest list
Getting on with this ASAP.
fyi ,


Implement proposal for Azure We should implement interface in Azure cloud provider. returns true if the instance is in stopped state to detach volumes. But if instance is deleted we should not return error. Related features: Related proposal: Related PRs: /kind feature /sig azure /cc
- can you assign me on this one ? I already have a PR in flight.
That's great! /assign just loop in for this RP review, thanks.
GitHub didn't allow me to assign the following users: yastij. Note that only and repo collaborators can be assigned. <detailsIn response to : That's great! /assign just loop in for this RP review, thanks. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
o, you are not a k8s member yet, anyway, you could send out the PR for this feature, related to this issue, thanks.
Thanks, looking forward to your PR.