Question: out.Min = (*int32)(unsafe.Pointer(in.Min)) out.TCPEstablishedTimeout = (*v1.Duration)(unsafe.Pointer(in.TCPEstablishedTimeout)) out.TCPCloseWaitTimeout = (*v1.Duration)(unsafe.Pointer(in.TCPCloseWaitTimeout)) <ins> out.TCPBeLiberal = in.TCPBeLiberal </ins> out.UDPTimeout = in.UDPTimeout out.UDPStreamTimeout = in.UDPStreamTimeout return nil

What happened: When kube-proxy is configured with a cluster-cidr range, it drops packets with an INVALID conntrack state for all sources on the KUBE-FORWARD chain. This is meant to prevent spurious retransmits in long-lived TCP connections to a service IP, but it prevents clusters from having asymmetrical routing on private subnets with services outside of a cluster. What you expected to happen: Packets on the KUBE-FORWARD chain with an INVALID conntrack state should be dropped for only sources inside of the cluster-cidr if set. How to reproduce it (as minimally and precisely as possible): N/A Anything else we need to know?: Change to drop INVALID conntrack packets was in Backing out that commit and compiling is my current workaround. Environment: Kubernetes version (use ): 1.17.2 Cloud provider or hardware configuration: OS (e.g: ):Kernel (e.g. ):
/sig network
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
I am reading but I don't fully grok what caused the problem in the first place, or why we think "only drop if it was in-cluster" is a correct fix? Edit, OK I get the original problem, but still don't understand why packets from outside the pod range would not exhibit , thereby requiring yet another revisit of this problem. Is this a conntrack bug? Should it be applying the DNAT?
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
I can try to give a bit more context to our setup, but the knowledge I have around this all dates back to Feb, unfortuately, so I may have forgotten details. We were replacing our mesos cluster with a kubernetes cluster in Feb and using a custom ingress solution (HAProxy w/an outside process configuring it for pod ips / mesos ip-ports) for both. The ingress nodes lived in the private subnet, but outside of the kubernetes cluster cidr range. When we upgraded from kubernetes 1.14 to kubernetes 1.15 (the first version with PR ) our ingress could no longer talk to any pod ips.


Question: out.Min = (*int32)(unsafe.Pointer(in.Min)) out.TCPEstablishedTimeout = (*v1.Duration)(unsafe.Pointer(in.TCPEstablishedTimeout)) out.TCPCloseWaitTimeout = (*v1.Duration)(unsafe.Pointer(in.TCPCloseWaitTimeout)) <ins> out.TCPBeLiberal = in.TCPBeLiberal </ins> out.UDPTimeout = in.UDPTimeout out.UDPStreamTimeout = in.UDPStreamTimeout return nil

I discovered that the KUBE-FORWARD chain had a rule to drop invalid packets, which was in Compiling kubernetes 1.15 without this commit fixed our problems, and we have been carrying that patch forward ever since. I would like to get rid of our forked kubernetes, hence this ticket. When I was writing this ticket up, I also found also mentions the same issue with the iptables dropping INVALID conntrack state packets near the end, regarding asymmetrical routing. Edit, OK I get the original problem, but still don't understand why packets from outside the pod range would not exhibit , thereby requiring yet another revisit of this problem. I think you are correct, I can imagine packets from outside of a cluster could exhibit the same behavior as packets inside the cluster and cause spurious resets. However, I am wondering if kube-proxy should be managing packets from outside of the cluster on a node... That seems outside of the scope of kube-proxy.
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
I took a stab at implementing what I think is the correct behavior for kube-proxy in .
/assign
I think that another option is to set
I don't understand how that will help me here. The packets with a ctstate INVALID are not generated due to conntrack saturation (), but due to asymmetric routing in our network. See for an example of a similar setup.
are the packets on your asymmetric routing setup marked as invalid if you set ? it should only mark the ones that are out of window per the description
/triage accepted
I'd love to pin this down, but I don't have the means to set up a repro. Is there a way I can set up a way to experiment?
We were bit by this too after revisiting an old service that had been unused since the heady days of pre-, and would also like to see the proposed patch merged (or a configuration option to prevent insertion of the INVALID state drop rule - either suits us). I unfortunately don't have any useful input as to setting up for reproduction. In our case we have a pretty odd anycast set up with a small pool of (public) addresses managed by calico.


Question: out.Min = (*int32)(unsafe.Pointer(in.Min)) out.TCPEstablishedTimeout = (*v1.Duration)(unsafe.Pointer(in.TCPEstablishedTimeout)) out.TCPCloseWaitTimeout = (*v1.Duration)(unsafe.Pointer(in.TCPCloseWaitTimeout)) <ins> out.TCPBeLiberal = in.TCPBeLiberal </ins> out.UDPTimeout = in.UDPTimeout out.UDPStreamTimeout = in.UDPStreamTimeout return nil

However I do seem to have a dirty workaround for small clusters. By setting notrack (ctstate UNTRACKED?) in the raw table for packets in our specific ranges that would otherwise match ctstate INVALID, the packets aren't dropped. Which may be of help to someone else who finds this issue.
We are seeing something very similar on large file downloads and our issue exactly fits this one's subject. These RESET (corrected after are happening intermittently and I am seeing similar behavior mentioned in the blog mentioned below. I am unsure if someone has found a workaround for this issue or not, so yeah my +1 to this issue being looked up by the community.
We're using as a solution for the connection reset we were seeing and might fix this issue.
Let me try to avoid that this thread derails and see if I can get current status: -proxy install rules to drop INVALID cstate connections, this solves the ~REJECT~ RESET problems are some scenarios, with asymmetric routing per example, that this rules may drop unrelated traffic. The request is to remove the rule that drops those INVALID states or to try to restrict it, and that can cause that people start to have the ~REJECT~ RESET problems and drops again. If we have a reproducer we can be more precise in the final solution, but removing one rule without any strong evidence in kube-proxy is very risky As a workaround for the asymetric routing problem, I suggested to set the nfconntracktcpbeliberal on the kernel: I think that will solve the problems, it will be good if someone can test it and provide feedback
No sadly, its my bad, it is not REJECT, it is RESET that's happening and conntrack has nothing to do with it, mostly (I hope). Kubeproxy is doing what is intended but specifically when there are external agents in play, the packet doesn't sends an ACK back to the response of SYN-ACK. Our case, we hit A to initiate a download, A calls B to get the action completed to generate download data for the user finally via their browser. A and B both are on k8s, we saw A starts and a complete cycle goes for sometime until there multiple parallel downloads requested where few gets completed and few gets a RESET.


Question: out.Min = (*int32)(unsafe.Pointer(in.Min)) out.TCPEstablishedTimeout = (*v1.Duration)(unsafe.Pointer(in.TCPEstablishedTimeout)) out.TCPCloseWaitTimeout = (*v1.Duration)(unsafe.Pointer(in.TCPCloseWaitTimeout)) <ins> out.TCPBeLiberal = in.TCPBeLiberal </ins> out.UDPTimeout = in.UDPTimeout out.UDPStreamTimeout = in.UDPStreamTimeout return nil

On primary debugging, it appears that RESET was getting invoked by the pod but pod is just sending a RST-ACK not RST itself. The missing packet is never found in the default route, unless we see that there is a chance where ACK is missed and we found some lines in tcpdump with missing sequence for ACKs. Thats why i doubt my issue is related here. One more weird coincidence is few downloads have spurious transmission.
Thankyou, I will take a look, but mostly I would like to wait before using it, debug more to understand the actual cause and then act on it.
sorry , I wanted to say RESET, thanks for catching it. If you think is different and you can provide information and how to reproduce it, I suggest you to open a new issue
I will sure do, when it is different. For now I would like to watch and submit my findings if it is related. PS: sysctl -w net.netfilter.nfconntracktcpbeliberal=1 is already implemented and we are testing if it does something helpful. so far it doesn't looks like it. thanks for your help.
/area kube-proxy
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle rotten
I have the feeling we are also affected by this with a standard calico installation: every node announces a /122 IPv6 network to the upstream router the upstream router uses ECMP to route the packet back packets that are sent from* the pod and returned via asymmetric path from outside seem to be dropped
and if you set ?
Same behaviour with set on all nodes, the outgoing pod traffic does not return. And I instantly see packets going back into the drop rule:
:thinking:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.


Question: out.Min = (*int32)(unsafe.Pointer(in.Min)) out.TCPEstablishedTimeout = (*v1.Duration)(unsafe.Pointer(in.TCPEstablishedTimeout)) out.TCPCloseWaitTimeout = (*v1.Duration)(unsafe.Pointer(in.TCPCloseWaitTimeout)) <ins> out.TCPBeLiberal = in.TCPBeLiberal </ins> out.UDPTimeout = in.UDPTimeout out.UDPStreamTimeout = in.UDPStreamTimeout return nil

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hi, are you still working on this? Same problem happened in our clusters, just because of asymmetrical routing.
/reopen
Reopened this issue. <detailsIn response to : /reopen </details
Also the command suggested by is helping us, but I still think this should be a bug of kube-proxy.
The problem here is: what happens if the solutions are exclusive or we regress? The addition of this rule fixed a legit bug that was documented and there is also a nice blog explaining the details Removing this rule without a repro and understanding what is the root cause of this issue is not really the best way to proceed. I think that people affected should try to find the root cause and explain the problem or just provide a way to reproduce it, so we can fix it
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at .


Question: out.Min = (*int32)(unsafe.Pointer(in.Min)) out.TCPEstablishedTimeout = (*v1.Duration)(unsafe.Pointer(in.TCPEstablishedTimeout)) out.TCPCloseWaitTimeout = (*v1.Duration)(unsafe.Pointer(in.TCPCloseWaitTimeout)) <ins> out.TCPBeLiberal = in.TCPBeLiberal </ins> out.UDPTimeout = in.UDPTimeout out.UDPStreamTimeout = in.UDPStreamTimeout return nil

/close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The same problem happened in our cluster, the same as because of asymmetrical routing. the command suggested by doesn't work for us, and I delete the iptables rule and add new rule(refer to ), it works I think that makes sense I'll put up a reproduction of the gist link later on /reopen
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The main problem is how to be completely sure we don't regress, this comment from Tim is spot on
I'm trying to check that non-local connections will still reset without the INVALID Rule
I think the patch in is wrong; it changes the rule from "if INVALID then drop" to "if INVALID and from a pod, then drop" so that it doesn't affect non-kubernetes traffic, but I think the rule we actually need is "if INVALID and to a service, then drop". In particular, I believe we need to make sure that the rule applies to traffic with a cluster-external source and an in-cluster destination. It's possible that changing it to "if INVALID and to a pod, then drop" would also work. I skimmed and some linked bugs elsewhere but I was unable to confirm whether that rule would cover the reporters' use cases.
I think that is only INVALID and to a pod
In my case, cluster-external source to service is both ok with the INVALID rule or without the INVALID rule. It seems the rule doesn't affect on access service.


Question: out.Min = (*int32)(unsafe.Pointer(in.Min)) out.TCPEstablishedTimeout = (*v1.Duration)(unsafe.Pointer(in.TCPEstablishedTimeout)) out.TCPCloseWaitTimeout = (*v1.Duration)(unsafe.Pointer(in.TCPCloseWaitTimeout)) <ins> out.TCPBeLiberal = in.TCPBeLiberal </ins> out.UDPTimeout = in.UDPTimeout out.UDPStreamTimeout = in.UDPStreamTimeout return nil

I add this rule but it doesn't work
So, based on that, the bug happens when a reply packet fails to get un-DNAT'ed, causing the client to then send a RST to the server, which matches the source/destination IP that the server knows of. But this could happen if you had a service with external IPs too; in that case the reply packet would be from the external endpoint IP to the node IP (because the packet would have been masqueraded), and if conntrack didn't unmap it correctly then the node would send a RST to the external IP, breaking the connection in just the same way. In this case there is no good way kube-proxy can recognize the packet as being its own, other than by using conntrack, which presumably won't work if the packet is INVALID?
It seems that there is no better way to solve this for external sources accessing the cluster CIDR, is it right? If so, Can we have a flag to decide whether to create The invalid rule? It would be nice that It's up to the user to decide whether to create the rule.
There's related discussion in . I think we should create whatever iptables rules we can that will fix up our own traffic without interfering with other traffic, and also document the sysctl that admins can use to change conntrack behavior to work around the problem. So probably, we should change the "drop if INVALID" rule to "drop if INVALID and to a pod", and also address the problem in , but not try to solve the problem with external IPs I mentioned in because we don't have a good way of fixing that without potentially breaking other things. (And it's kind of way more of an edge case anyway.) Note that can currently output rules for "from a pod", but not for "to a pod", so that will need some tweaking. (FWIW will also eventually need the ability to output nftables rules rather than iptables rules so that's another thing to think about if someone is going to change its API.)
So in the absence of a way to fix the problem with external IPs, can we have a flag to decide whether to create this rule or not? If true (default), create the drop if INVALID and to a pod rule. Otherwise, We do not create the rule.


Question: out.Min = (*int32)(unsafe.Pointer(in.Min)) out.TCPEstablishedTimeout = (*v1.Duration)(unsafe.Pointer(in.TCPEstablishedTimeout)) out.TCPCloseWaitTimeout = (*v1.Duration)(unsafe.Pointer(in.TCPCloseWaitTimeout)) <ins> out.TCPBeLiberal = in.TCPBeLiberal </ins> out.UDPTimeout = in.UDPTimeout out.UDPStreamTimeout = in.UDPStreamTimeout return nil

I believe that even without this rule we can solve this problem by way of sysctl, right? I think we should do the following things We need a flag to disable the rule We should change the "drop if INVALID" rule to "drop if INVALID and to a pod" We document sysctl as a way to solve the problem.
If the rule is "drop if INVALID and to a pod", we shouldn't need a flag to disable it; it won't affect non-pod traffic, and it won't interfere with users that want to set the sysctl instead. (Setting the sysctl will cause packets to not be marked as INVALID so the rule we add just won't get hit.)
OK, drop if INVALID and to a pod is help us identify cluster traffic, right? We only drop cluster traffic if it's invalid, but this will not solve the problem of external IP. So we need a flag to disable the rule is drop if INVALID?
The same problem happened in our cluster, the same as because of asymmetrical routing None of the above methods solve the problem in our ENV. Are there any other workarounds?
Can you execute the following command on each node? I think this is a workaround.
Thank you for your reply I also tried removing this rule. However, k8s will reconfigure the deleted rules to the system at some point in time.
I worded that badly: I meant "external IP" as in "the endpoints are cluster-external IPs rather than pod IPs", not as in . So that would apply only to people using kube-proxy to load-balance traffic from clients outside the cluster to servers outside the cluster, which is a pretty weird use case. And nobody ever reported the packet RST bug for that case anyway. So I was arguing that we can just ignore that case. So: change the existing "drop if invalid" rule to "drop if invalid and to a pod" document the sysctl We don't need an option to disable the drop rule, because it won't affect non-k8s traffic, and it will automatically become a no-op if someone decides to use the sysctl instead.
Thank you for your response. I'd love to take this one. But I have a question, How to define k8s traffic?


Question: out.Min = (*int32)(unsafe.Pointer(in.Min)) out.TCPEstablishedTimeout = (*v1.Duration)(unsafe.Pointer(in.TCPEstablishedTimeout)) out.TCPCloseWaitTimeout = (*v1.Duration)(unsafe.Pointer(in.TCPCloseWaitTimeout)) <ins> out.TCPBeLiberal = in.TCPBeLiberal </ins> out.UDPTimeout = in.UDPTimeout out.UDPStreamTimeout = in.UDPStreamTimeout return nil

If a pod in a cluster, we call it an underlay pod, its IP is not allocated from cluster CIDR, this is a real underlay IP, when it accesses other underlay pods in the cluster or pods that allocate IP from cluster CIDR(We call it overlay pod), is this k8s traffic? In our cluster, multiple types of CNI pods are implemented through multus running in a cluster, because of the problem of asymmetrical routing and the rule that makes them inaccessible, it seems that the above method will not solve this problem.
For kube-proxy, this is done by the . If you are in a cluster where there is no simple way to discriminate pod from non-pod traffic, then you'd have to run kube-proxy with , and then kube-proxy will see that returns and so it won't try to distinguish local from non-local traffic, and will do some things (like internal/external traffic policy) slightly differently as a result. So in this case, if there was no functioning localDetector, it would also not create the "drop invalid" rule
Thanks for the details, this is probably the case we want.
/assign
I did not understand how you could get asymmetric routing in K8s, but reading up more carefully I saw: If I understand this correctly, this is a load-balancer outside K8s that load-balance directly to POD addresses. To get asymmetric routing some kind of Direct Server Return (DSR) must also be used. So, to get asymmetric routing: from an external source is directed to an external LB LB is configured with POD-IPs as targets and selects one packet is routed to a K8s node, e.g. with ECMP. But is may not be the node where the target POD is executing packet is handled by the POD, but the response packet may be directed to some other GW This will give asymmetric routing. But it's not the every-day K8s network setup for sure. Given the above it's fairly easy to reproduce the problem. You don't even have to setup a load-balancer. The biggest problem may be that you need two ways in to the cluster. Example: ! I setup direct routes that ensures asymmetric routing, which can be checked with on vm-201 and vm-202.


Question: out.Min = (*int32)(unsafe.Pointer(in.Min)) out.TCPEstablishedTimeout = (*v1.Duration)(unsafe.Pointer(in.TCPEstablishedTimeout)) out.TCPCloseWaitTimeout = (*v1.Duration)(unsafe.Pointer(in.TCPCloseWaitTimeout)) <ins> out.TCPBeLiberal = in.TCPBeLiberal </ins> out.UDPTimeout = in.UDPTimeout out.UDPStreamTimeout = in.UDPStreamTimeout return nil

From I can now directly ping POD-IPs, but due to the problem in this issue, I can't make a TCP connection: The POD with address runs on . A similar setup as above can be used to verify a PR. Now, there is a simple work-around: use proxy-mode=ipvs The update with the rule never made it to the ipvs proxier (a rather common case :cry: ), but in this case it may save your day. The same as above with proxy-mode=ipvs; The server on 5001 responds with the POD name.
Yeah, Our production environment also solved this problem by switching to IPVS mode. In IPVS mode, there is no the DROP rule.
The sysctl solves the problem in , but not the problem in this issue. Those problems are kind of related, but rather distant.
The same affects outgoing traffic if packets are not masqueraded to the node-ip, as should be a common setup for ipv6, please see For Calico I think you can setup BGP for clusterCIDRs and route incoming packets to PODs to the correct K8s node, instead of using ECMP. This would solve the problem, and save the extra hop.
I met the same issue. KUBE-FORWARD drop my packages.
temptatively solution
The issue fixed by , I'll close it, you can feel free to open it if have any questions. /close
Closing this issue. <detailsIn response to : The issue fixed by , I'll close it, you can feel free to open it if have any questions. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: } return nil } <ins> // WaitForContainerRunning waits for the given Pod container to have a state of running func WaitForContainerRunning(c clientset.Interface, namespace, podName, containerName string, timeout time.Duration) error { return wait.PollImmediate(poll, timeout, isContainerRunning(c, namespace, podName, containerName)) } </ins>

This issue is to discuss an e2e testing plan for Ephemeral Containers. /sig node
As a first step towards this issue, I am planning to create node-e2e tests (under ) to verify the following: With a running pod client adds a valid EphemeralContainer resource, and the ephemeral container starts from the ephemeral container, list processes of itself and other containers, and then signal other processes (if is ) from the ephemeral container, ping & curl other containers try to read/write to stdin/stdout of the ephemeral container verify resource request/limit values against calculated expectations check oom scores of processes in the ephemeral container verify volume mounts restarts according to policy, when process exits client tries to add EphemeralContainer resource with ports (should be rejected) client tries to add EphemeralContainer resource with probes (should be rejected) client tries to add EphemeralContainer resource with resources (should be rejected) client tries to add EphemeralContainer resource with securityContext (should be rejected) removes the ephemeral container, and the ephemeral container is stopped and deleted adds multiple ephemeral containers, should all startWith pod of non-running status (TBD)
Awesome, thanks! This is a great start, here are some things to look out for: Visibility should work, but I expect signaling to fail without which is not currently allowed. iirc was interested in how this would be tested for ephemeral containers. Can you point us towards any examples that would help us get started? Ephemeral containers will never restart (and the policy should always be never) These will be denied by the api server without ever making it to the kubelet, so they are better tested in the apiserver integration tests (and I think they are already). Removing ephemeral containers is not currently allowed. /assign
Validation of ephemeralContainers is well tested in Testing of removing ephemeral containers is in ,
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with .


Question: } return nil } <ins> // WaitForContainerRunning waits for the given Pod container to have a state of running func WaitForContainerRunning(c clientset.Interface, namespace, podName, containerName string, timeout time.Duration) error { return wait.PollImmediate(poll, timeout, isContainerRunning(c, namespace, podName, containerName)) } </ins>

Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen /assign /priority important-soon
Reopened this issue. <detailsIn response to : /reopen /assign /priority important-soon Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-contributor-experience at . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Rotten issues close after 30d of inactivity.


Question: } return nil } <ins> // WaitForContainerRunning waits for the given Pod container to have a state of running func WaitForContainerRunning(c clientset.Interface, namespace, podName, containerName string, timeout time.Duration) error { return wait.PollImmediate(poll, timeout, isContainerRunning(c, namespace, podName, containerName)) } </ins>

Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-contributor-experience at . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen /remove-lifecycle rotten
Reopened this issue. <detailsIn response to : /reopen /remove-lifecycle rotten Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: appsv1 "k8s.io/api/apps/v1" v1 "k8s.io/api/core/v1" schedulingv1 "k8s.io/api/scheduling/v1" <ins> apiequality "k8s.io/apimachinery/pkg/api/equality" </ins> apierrors "k8s.io/apimachinery/pkg/api/errors" "k8s.io/apimachinery/pkg/api/resource" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" "k8s.io/apimachinery/pkg/labels" <ins> "k8s.io/apimachinery/pkg/runtime" "k8s.io/apimachinery/pkg/runtime/schema" </ins> "k8s.io/apimachinery/pkg/types" "k8s.io/apimachinery/pkg/util/intstr" utilrand "k8s.io/apimachinery/pkg/util/rand" "k8s.io/apimachinery/pkg/util/wait" <ins> watch "k8s.io/apimachinery/pkg/watch" quota "k8s.io/apiserver/pkg/quota/v1" </ins> clientset "k8s.io/client-go/kubernetes" <ins> clientscheme "k8s.io/client-go/kubernetes/scheme" "k8s.io/client-go/tools/cache" watchtools "k8s.io/client-go/tools/watch" "k8s.io/client-go/util/retry" </ins> "k8s.io/kubernetes/pkg/quota/v1/evaluator/core" "k8s.io/kubernetes/test/e2e/framework" "k8s.io/kubernetes/test/utils/crd"

[X] Test approval issue: [X] Test PR: 19 Oct 2022 [x] Two weeks soak end date: 2 Nov 2022 [x] Test promotion PR: According to this APIsnoop query, there are still three remaining ResourceQuota status endpoints which are untested. patchCoreV1NamespacedResourceQuotaStatusreplaceCoreV1NamespacedResourceQuotaStatusreadCoreV1NamespacedResourceQuotaStatus Using a number of existing e2e test practices a new has been created for 3 ResourceQuota Status endpoints. The e2e logs for this test are listed below. This query shows which Resource Status endpoints are hit within a short period of running this e2e test If a test with these calls gets merged, test coverage will go up by 3 point This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/triage accepted
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: <del> #!/usr/bin/env bash # Copyright 2016 The Kubernetes Authors. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # #     http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # This script checks whether removal of bazel related files is still needed. # If so: run `hack/update-bazel.sh` to actually remove them. # # Usage: `hack/verify-bazel.sh`. set -o errexit set -o nounset set -o pipefail KUBE_ROOT=$(dirname "${BASH_SOURCE[0]}")/.. export KUBE_ROOT source "${KUBE_ROOT}/hack/lib/init.sh" _tmpdir="$(kube::realpath "$(mktemp -d -t verify-bazel.XXXXXX)")" kube::util::trap_add "chmod -R u+rw ${_tmpdir} && rm -rf ${_tmpdir}" EXIT _tmp_gopath="${_tmpdir}/go" _tmp_kuberoot="${_tmp_gopath}/src/k8s.io/kubernetes" mkdir -p "${_tmp_kuberoot}/.." cp -a "${KUBE_ROOT}" "${_tmp_kuberoot}/.." cd "${_tmp_kuberoot}" GOPATH="${_tmp_gopath}" PATH="${_tmp_gopath}/bin:${PATH}" ./hack/update-bazel.sh diff=$(diff -Naupr -x '_output' "${KUBE_ROOT}" "${_tmp_kuberoot}" || true) if [[ -n "${diff}" ]]; then echo "${diff}" >&2 echo >&2 echo "Run ./hack/update-bazel.sh" >&2 exit 1 fi </del>

This is intended to be removed once there are no active branches with bazel. 1.19 was still supported last I checked. We have some other cleanup pending the final turndown of bazel branches.
19 just had it's last patch release a week ago , these are removed now.


Question: <ins> // +build windows /* Copyright 2018 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package dockershim import ( dockertypes "github.com/docker/docker/api/types" dockercontainer "github.com/docker/docker/api/types/container" runtimeapi "k8s.io/kubernetes/pkg/kubelet/apis/cri/runtime/v1alpha2" ) // applySandboxPlatformOptions applies platform specific options to dockercontainer.HostConfig and dockercontainer.ContainerCreateConfig. func (ds *dockerService) applySandboxPlatformOptions(hc *dockercontainer.HostConfig, config *runtimeapi.PodSandboxConfig, createConfig *dockertypes.ContainerCreateConfig, image string, separator rune) error { dnsConfig := config.GetDnsConfig() if dnsConfig == nil { return nil } // Setup DNS. hc.DNS = dnsConfig.GetServers() hc.DNSSearch = dnsConfig.GetSearches() hc.DNSOptions = dnsConfig.GetOptions() return nil } </ins>


Question: KubeProxyVersion:        version.Get().String(), }, Capacity: v1.ResourceList{ <del> v1.ResourceCPU:     *resource.NewMilliQuantity(2000, resource.DecimalSI), v1.ResourceMemory:  *resource.NewQuantity(20E9, resource.BinarySI), v1.ResourceStorage: *resource.NewQuantity(500*mb, resource.BinarySI), v1.ResourcePods:    *resource.NewQuantity(0, resource.DecimalSI), </del> <ins> v1.ResourceCPU:    *resource.NewMilliQuantity(2000, resource.DecimalSI), v1.ResourceMemory: *resource.NewQuantity(20E9, resource.BinarySI), v1.ResourcePods:   *resource.NewQuantity(0, resource.DecimalSI), </ins> }, Allocatable: v1.ResourceList{ <del> v1.ResourceCPU:     *resource.NewMilliQuantity(1800, resource.DecimalSI), v1.ResourceMemory:  *resource.NewQuantity(19900E6, resource.BinarySI), v1.ResourceStorage: *resource.NewQuantity(500*mb, resource.BinarySI), v1.ResourcePods:    *resource.NewQuantity(0, resource.DecimalSI), </del> <ins> v1.ResourceCPU:    *resource.NewMilliQuantity(1800, resource.DecimalSI), v1.ResourceMemory: *resource.NewQuantity(19900E6, resource.BinarySI), v1.ResourcePods:   *resource.NewQuantity(0, resource.DecimalSI), </ins> }, Addresses: []v1.NodeAddress{ {Type: v1.NodeInternalIP, Address: "127.0.0.1"},

Local storage is not expected to be available as a scheduling resource in v1.7. Kubelet is incorrectly exposing a resource named . This is incorrect because the feature is alpha and so it requires opt-in via feature flags and, the resource names are invalid. The expected resource names are listed - and . cc This is v1.7 blocker because we cannot expose a new resource in the API and rename it in a subsequent release.
CC I'll find someone on the storage team to address this.
/assign
GitHub didn't allow me to assign the following users: verult. Note that only can be assigned. <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: "net.ipv4.ip_local_port_range", "net.ipv4.tcp_syncookies", "net.ipv4.ping_group_range", <ins> "net.ipv4.ip_unprivileged_port_start", </ins> ) )

Is the sysctl namespaced? If so, can we include this in the set of "safe" sysctls? If we allow this, we would: add to permit in PodSecurity baseline profile cc So that containers can bind to low ports without needing to bake CAPNETBINDSERVICE into the files in the image doesn't help hostNetwork pods since they don't have their own namespace and you likely wouldn't want to change the system-wide sysctl also, pods that schedule to arbitrary nodes likely shouldn't start taking advantage of this until 2+ releases after it merges into the kubelet, because of potential node skew
/sig network security node auth
xref: for actually potentially setting this by default even.
/assign
Yes, it is network-namespaced and :+1: to adding it to safe list. From
/triage accepted


Question: // This is particularly useful in external cloud providers where the kubelet // does not initialize node data. func (gce *GCECloud) GetZoneByProviderID(providerID string) (cloudprovider.Zone, error) { <del> return cloudprovider.Zone{}, errors.New("GetZoneByProviderID not implemented") </del> <ins> _, zone, _, err := splitProviderID(providerID) if err != nil { return cloudprovider.Zone{}, err } region, err := GetGCERegion(zone) if err != nil { return cloudprovider.Zone{}, err } return cloudprovider.Zone{FailureDomain: zone, Region: region}, nil </ins> } // GetZoneByNodeName implements Zones.GetZoneByNodeName // This is particularly useful in external cloud providers where the kubelet // does not initialize node data. func (gce *GCECloud) GetZoneByNodeName(nodeName types.NodeName) (cloudprovider.Zone, error) { <del> return cloudprovider.Zone{}, errors.New("GetZoneByNodeName not imeplemented") </del> <ins> instanceName := mapNodeNameToInstanceName(nodeName) instance, err := gce.getInstanceByName(instanceName) if err != nil { return cloudprovider.Zone{}, err } region, err := GetGCERegion(instance.Zone) if err != nil { return cloudprovider.Zone{}, err } return cloudprovider.Zone{FailureDomain: instance.Zone, Region: region}, nil </ins> } // ListZonesInRegion returns all zones in a GCP region

Is this a BUG REPORT or FEATURE REQUEST?: As part of the initiative for external cloud providers (), we GetZoneByProviderID and GetZoneByNodeName. See for more details. The previous PR did not actually implement those methods for existing cloud providers in . These methods should be implemented by all cloud providers in the future. cc
Could someone add the label and the label for this issue please?
/sig cluster-lifecycle
/area cloudprovider
/assign wlan0
cc We need the following label - help-wanted and add it to the 1.8(?) milestone cc to implement it for their respective clouds.
I'll try to have a look by the end of the week.
Ok, I will take care of the Rackspace and OpenStack.
Thank you !
- Is there any listing where to see the already implemented cloud provider ?
No providers have this yet. Current PRs: Openstack: Rackspace: Please contribute if you have something ready :)
Sorry for the delay guys :)
Have you implemented them for gce?
Intending to work on it tonight, but if you have something I can work on azure
Yes, I have prepared for GCE.
Azure .
Cool, than you all! Any other providers we should implement this for in time for v1.8?
I'll take a look tomorrow to see if there's something left
Let's see which providers are left before closing this...
- I guess this one should be closed ?
If you say that all providers have this now, thanks, great!


Question: package tools import ( <ins> "errors" </ins> "fmt" "reflect" "sync"

When you setup TLS assets, you need to make sure each master API server is using the same ones, unless you were running a single master. But this is a super common mistake, and it often breaks things. Because the masters validating tokens using that key, the controller-manager also has this service-account key, and it’s responsible for creating tokens for service accounts. So, depending on which apiserver the pod is talking to, that service-account token may, or may not work. This most commonly surfaces in kube2sky (the dns addon). Result: Tokens are invalidated regularly (~10 minutes) Expected: Warning about system having different certs (or some alert). Would this ever be a valid configuration?
Is this specifically for an HA setup (since you mention multiple apiservers)? Or do you mean that each master component should have the same cert?
Correct - it's the HA deployment.
I don't think that the apiservers should necessarily cross-check each other (and all other system components like the controller manager) and alert. If you are running an HA deployment, you should have something else making sure that your deployment is correct. On cluster creation, the startup scripts should do it.
I guess what I'm saying is that we know this is going to be a problem. There's almost no way to make this NOT a problem, yet it is a very easy bug to get into. So... with that in mind, what's the path for doing this? When you add a second master to your HA config, how does it establish trust with the first HA? Could we check at that point?
Yup, we do this on provision time with Ansible.


Question: authProviderArgs         map[string]string authProviderArgsToRemove []string <ins> execCommand     cliflag.StringFlag execAPIVersion  cliflag.StringFlag execArgs        []string execEnv         map[string]string execEnvToRemove []string </ins> } const ( flagAuthProvider    = "auth-provider" flagAuthProviderArg = "auth-provider-arg" <ins> flagExecCommand    = "exec-command" flagExecAPIVersion = "exec-api-version" flagExecArg        = "exec-arg" flagExecEnv        = "exec-env" </ins> ) var (

Now it's possible to configure third-party auth providers using command with option. It's nice to have the same possibility for the recently . Example: To specify the executable path we can run This will generate a new config for the user: Various auxiliary configuration options are also could be . For instance: to define an argument for the executable; to set an environment variable; to specify a desired API version. /kind feature
I'm going to start working on this when the API is promoted to beta.
/sig auth
/assign
Can someone share why you can't do : Is there a way to set map values (e.g. something that is -name: value ; versus something that is -context: )
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
I would like to work on it, if this is still persist.
cc
This still seems useful and is outstanding. any objections to taking it over?
No, it's okay, I'm working on it. I'll post the code this week, sorry for the delay
I think this would help out with generating valid kubectl configurations for our EKS developers. I'm using Pulumi to build EKS environments and if all the flags are available, I think I can generate the necessary entries to assume a role. Thanks! Ex: name: k8s.developer user: exec: apiVersion: args: token -i cluster-eksCluster- -r arn:aws:iam:::role/k8sDevAdminRole- command: aws-iam-authenticator env: null
done :)
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
/priority important-longterm


Question: // Note: The user still retains the ability to explicitly set feature-gates and that value will overwrite this base value. if enabled, present := cfg.FeatureGates[features.IPv6DualStack]; present { defaultArguments["feature-gates"] = fmt.Sprintf("%s=%t", features.IPv6DualStack, enabled) <ins> } else if cfg.Networking.PodSubnet != "" { // TODO(Arvinderpal): Needs to be fixed once PR #73977 lands. Should be a list of maskSizes. maskSize := calcNodeCidrSize(cfg.Networking.PodSubnet) defaultArguments["node-cidr-mask-size"] = maskSize </ins> } command := []string{"kube-controller-manager"}

What happened: I use the following configuration file the last generation of ,There are three parameters in yaml which looks like default. The result is that the kube-controller cannot start. When I delete it, the kube-controller will work fine What you expected to happen: In this case, the default parameter should not be Anything else we need to know?: kubeadm version kubectl version
/sig cluster-lifecycle /area kubeadm /milestone v1.17


Question: ) err := admissiontesting.WithReinvocationTesting(t, ctrl).Admit(context.TODO(), attrs, nil) klog.Infof("Got %v", err) if !test.expectError { if err != nil { t.Errorf("Test %q: unexpected error received: %v", test.name, err)

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Following the documentation at the , I tried enabling quota for a namespace other than but I was still not able to create a pod with priorityClass in my namespace with following error: Error from server (Forbidden): error when creating "": pods "memory-demo" is forbidden: pods with system-cluster-critical priorityClass is not permitted in my-namespace namespace What you expected to happen: Expected pod to be created with namespace where I allowed quota. I can see that both ResourceQuota and Priority Plugins are enabled from logs. I0408 17:15:08. ] Loaded 8 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,MutatingAdmissionWebhook. I0408 17:15:08. ] Loaded 6 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,ValidatingAdmissionWebhook,ResourceQuota. How to reproduce it (as minimally and precisely as possible): Follow the steps at Anything else we need to know?: I think, the issue is with priorityPlugin where we are rejecting instead of letting resourceQuota plugin to handle the admission when is enabled. Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Others:
/sig scheduling /cc
Same happens to me. Correction, it does not allow to use the default permissions outside of kube-system
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
See also:
/remove-lifecycle stale
/assign Per offline discussion with
Same issue here.


Question: ) err := admissiontesting.WithReinvocationTesting(t, ctrl).Admit(context.TODO(), attrs, nil) klog.Infof("Got %v", err) if !test.expectError { if err != nil { t.Errorf("Test %q: unexpected error received: %v", test.name, err)

We are using namespaces other than kube-system for system pods as it is getting too crowded, therefore we would like to run cluster-node-critical pods in other namespaces.
Waiting for this to be available myself. No reason my Node Exporter dataset has to run in the namespace instead of my namespace.


Question: fi if [[ -n "${GCE_ALPHA_FEATURES:-}" ]]; then use_cloud_config="true" <del> cat <<EOF >>/etc/gce.conf alpha-features = ${GCE_ALPHA_FEATURES} </del> <ins> # split GCE_ALPHA_FEATURES into an array by comma. IFS=',' read -r -a alpha_features <<< ${GCE_ALPHA_FEATURES} for feature in ${alpha_features[@]}; do cat <<EOF >>/etc/gce.conf alpha-features = ${feature} </ins> EOF <ins> done </ins> fi if [[ -n "${SECONDARY_RANGE_NAME:-}" ]]; then use_cloud_config="true"


Question: // `iptables -t nat -A <from> -m set --match-set <name> <matchType> -j <to>` // example: iptables -t nat -A KUBE-SERVICES -m set --match-set KUBE-NODE-PORT-TCP dst -j KUBE-NODE-PORT // ipsets with other match rules will be created Individually. <ins> // Note: kubeNodePortLocalSetTCP must be prior to kubeNodePortSetTCP, the same for UDP. </ins> var ipsetWithIptablesChain = []struct { name          string from          string

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug /assign I'll make the pull request tomorrow. What happened: start kube-proxy in ipvs mode, create some NodePort services. Then look at the iptables rules. match-set of KUBE-NODE-PORT-TCP is inadequate to be qualified as NodePort traffic. "--dst-type LOCAL" is needed. What you expected to happen: since we have also KUBE-NODE-PORT-UDP, another way would be like this: How to reproduce it (as minimally and precisely as possible): start kube-proxy in ipvs mode, create some NodePort services. Then look at the iptables rules. Anything else we need to know?: Environment: Kubernetes version (use ): commit Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): CentOS Linux release 7.2.1511 (Core) Kernel (e.g. ): Linux VM34centos 3.10.0-514.26.2.el7.x8664 SMP Tue Jul 4 15:04:05 UTC 2017 x8664 x8664 x86_64 GNU/Linux Install tools: Others:
/sig network /area kube-proxy /area ipvs /assign
/cc i remember this bug fixed, that's right?
, Thanks, seems like necessary. But you have any use case that ipvs proxier will have bugs in current mach rules?
It's already fixed by ? Can we close this issue now?
I don't see how fixed this issue. How about this scenario: k8s cluster A uses node port for service A k8s cluster B uses node port for service B pod (or host) in cluster A tries to access node port on k8s cluster B local node port rule matches the traffic and NATs it to the local node port, pod (or host) cannot reach cluster B's node port. Also, the remote node port could be replaced by any non-k8s host that happens to use port .
Then I think can solve this problem.


Question: PROXY_PORT_FILE= } # Starts "kubect proxy" to test the client proxy. $1: api_prefix function start-proxy() {

I disabled a block of code in . The code was flaky (). Suggest one of the following solutions: this type of test coverage to an e2e, or VM or docker based test, where there are no port conflicts. kubectl proxy command to have a mode where it will pick a port for you, trying until it finds one. the test loop to allow failure of kubectl proxy command when it exits with a certain error code that means it could not get a port, and retries until it gets one.
Fixed in ?
No, not fixed:
We still need to reenable this code. I was unaware that it'd been disabled.
I made a reversion PR.


Question: if exists { // For non overcommitable resources, not only requests can't exceed limits, they also can't be lower, i.e. must be equal. if quantity.Cmp(limitQuantity) != 0 && !helper.IsOvercommitAllowed(resourceName) { <del> allErrs = append(allErrs, field.Invalid(reqPath, quantity.String(), fmt.Sprintf("must be equal to %s limit", resourceName))) </del> <ins> allErrs = append(allErrs, field.Invalid(reqPath, quantity.String(), fmt.Sprintf("must be equal to %s limit of %s", resourceName, limitQuantity.String()))) </ins> } else if quantity.Cmp(limitQuantity) > 0 { <del> allErrs = append(allErrs, field.Invalid(reqPath, quantity.String(), fmt.Sprintf("must be less than or equal to %s limit", resourceName))) </del> <ins> allErrs = append(allErrs, field.Invalid(reqPath, quantity.String(), fmt.Sprintf("must be less than or equal to %s limit of %s", resourceName, limitQuantity.String()))) </ins> } } else if !helper.IsOvercommitAllowed(resourceName) { allErrs = append(allErrs, field.Required(limPath, "Limit must be set for non overcommitable resources"))

Created limit range object with this parameters then tried to create this pod got message: Error from server (Invalid): error when creating "pod-to-be-": Pod "nginx-to-be-removed-4" is invalid: spec.containers[0].resources.requests: Invalid value: "700m": must be less than or equal to cpu limit Pod should be created with Requests & Limits of cpu equal to 700m (since it is in range of min/max cpu limitrange object) create limitrange object and pod as in description No response <details</details<detailson premise/AKS </details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
You have a default CPU limit of 10m, and no explicit CPU limit on the container, but a request larger than the limit. The request must always be less than or equal to the limit. Your pod needs to also set a higher limit, or the limit range needs to not set a default limit.
/remove-kind bug /kind support
/cc
but this is not obvious behavior since there is min & max set in limit range. This is limits (not min/max) that should be overwritten by pod. limits should be set to requests in this case imho
limit range limits what the user can set, it doesn't perform any mutations other than the defaulting and doesn't claim to. Min/max enforce that pods cannot set higher / lower. The default ensures that if the pod doesn't set a limit it will have one. personally I would remove the default limit, especially at 10m, this will aggressively limit pods. it would be, if the pod set a limit that would be a substantial breaking change, I don't think we could do that short of a v2 API and I'm not sure it would be correct behavior even then. we could make it opt-in, but then, why not just set them to match on the pod to begin with?


Question: <ins> /* Copyright 2024 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ // This is an empty file to avoid "build constraints exclude all Go files" error while running unit tests on non-linux platforms. package testing </ins>

Tried running make test against local latest fork of kubernetes master Go version Go env Successfully able to run the test Try running make test on darwin os. No response <details</detailsLatest master <details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
Bro i think there might be problem with your setup , its not an issue .
test cases are working fine
Thanks for verifying, I will double check from my side, I hope you verified it on darwin os.
/sig release
Also seeing this when trying to run on mac


Question: if err != nil { return err } <del> if err := o.updateTaints(obj); err != nil { </del> <ins> operation, err := o.updateTaints(obj) if err != nil { </ins> return err } newData, err := json.Marshal(obj)

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"6+", GitVersion:"v1.6.0-alpha.3.1185+-dirty Server Version: version.Info{Major:"1", Minor:"6+", GitVersion:"v1.6.0-alpha.3.1185+-dirty" What happened: Kubectl is always reporting as node <XYZis tainted irrespective of operation when used with local cluster. taint nodes XYZ dedicated- node "XYZ" tainted What you expected to happen: It should have shown exact operation like tainted, untainted modified and overwritten but because of it is always returning tainted. How to reproduce it (as minimally and precisely as possible): taint nodes XYZ dedicated-
cc/
Regarding Yes I agree we should provide more clearer output/feedback. But providing something that makes output more confusing than existing is something would be even more concerning. So as I stated , it would be good to discuss/decide how the output/feedback should like for various operations for one and multi nodes scenarios.
I think having 1 return values is an exponential improvement on what we have now, even if they aren't perfect for all cases. Rather then a disproportionately long debate over a relatively simple improvement.... Could you guys suggest something simple that you're confident it will be easy to agree on, and we can implement it as a second pass ? we are happy to implement anything with n 1 return values :).
I think we'd be happy to consider any reasonable PR. I'm not sure why was closed; I didn't look at it though so I don't know if it solves the problem.
(hit close accidentally)
, I am not sure why was closed. But I raised a PR around 15 days ago() to address this.


Question: - list - watch - apiGroups: <ins> - apps attributeRestrictions: null resources: - statefulsets verbs: - get - list - watch - apiGroups: </ins> - policy attributeRestrictions: null resources:

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): No. What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail* the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Environment: Cloud provider or hardware configuration: GCE OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: What happened: The is constantly flooded with the following error once per second: What you expected to happen: Log not flooded with error. How to reproduce it (as minimally and precisely as possible): Start a cluster at the above version. Check the on master. Anything else do we need to know: I didn't create any StatefulSets or PDBs. I only created some RCs.
Attn
cc This looks related to
Looks like an authorization issue. It doesn't seem to happen on a local cluster. From APIServer logs:


Question: <ins> /* Copyright 2018 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package debugger import "os" // compareSignal is the signal to trigger cache compare. For windows, // it's SIGINT. var compareSignal = os.Interrupt </ins>

PLEASE COORDINATE WITH SIG-SCHEDULING BEFORE WORKING ON THIS ISSUE. We want to avoid the situation where two people accidentally start work on the same fix. Main tracking issue: Currently, the feature is set up at the end of in . This setup should be moved into its own helper function, preferably in the package. /sig scheduling /kind cleanup /help
/assign /remove-help
I'd like to take this one
/remove-help
the PR I sent should resolve this one. :disappointed: Are any of the others still available?


Question: existingNode.Status.Addresses = append(existingNode.Status.Addresses, existingNodeAddress) } <del> nodeIP := testCase.nodeIP </del> nodeIPValidator := func(nodeIP net.IP) error { return nil }

When adding to kubelet and to the OpenStack external cloud provider, which is based on kubernetes v1.27.1, the node annotation only shows one address and consequently the node addresses also only displays one address. The kubelet logs shows that node-ip contains dual-stack addresses. The annotation on the node would contain both IPv4 and IPv6 addresses and the node addresses would list them. Deploy a cluster with to kubelet and external cloud provider. Also make sure the external cloud provider is based on Kubernetes 1.27. Kubelet logs: one of CCMs Pods: one of the nodes annotation: <details</details<details</details<details</details<details</details<details</details<details</details
/sig network /sig cloud-provider /cc
/assign
/triage accepted


Question: return false, fmt.Errorf("multiple instances found for instance: %s", instanceID) } <del> state := instances[0].State.Name if *state != "running" { glog.Warningf("the instance %s is not running", instanceID) return false, nil } </del> return true, nil } // InstanceShutdownByProviderID returns true if the instance is in safe state to detach volumes func (c *Cloud) InstanceShutdownByProviderID(ctx context.Context, providerID string) (bool, error) { <del> return false, cloudprovider.NotImplemented </del> <ins> instanceID, err := kubernetesInstanceID(providerID).mapToAWSInstanceID() if err != nil { return false, err } request := &ec2.DescribeInstancesInput{ InstanceIds: []*string{instanceID.awsString()}, } instances, err := c.ec2.DescribeInstances(request) if err != nil { return false, err } if len(instances) == 0 { glog.Warningf("the instance %s does not exist anymore", providerID) return true, nil } if len(instances) > 1 { return false, fmt.Errorf("multiple instances found for instance: %s", instanceID) } instance := instances[0] if instance.State != nil { state := aws.StringValue(instance.State.Name) // valid state for detaching volumes if state == ec2.InstanceStateNameStopped || state == ec2.InstanceStateNameTerminated { return true, nil } } return false, nil </ins> } // InstanceID returns the cloud provider ID of the node with the specified nodeName.

Is this a BUG REPORT or FEATURE REQUEST?: FEATURE /kind feature Implement InstanceShutdownByProviderID to AWS. InstanceShutdownByProviderID returns true if the instance is in safe state to detach volumes. Note: if instance is deleted we should not return error.
/sig aws
This is only when using the cloud-controller-manager, right?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close


Question: return "", err } ip = ip.Mask(ipnet.Mask) <del> hexIP := hex.EncodeToString([]byte(ip.To4())) </del> <ins> hexIP := hex.EncodeToString([]byte(ip)) </ins> hexMask := ipnet.Mask.String() return hexIP + "/" + hexMask, nil }

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): No What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): hexCIDR and CIDR Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): latest on master Environment: Cloud provider or hardware configuration: Bare-metal, 3 node config using KubeAdm OS (e.g. from /etc/os-release): CentOS 7 Kernel (e.g. ): Linux devstack-77 3.10.0-514.10.2.el7.x8664 SMP Fri Mar 3 00:04:05 UTC 2017 x8664 x8664 x8664 GNU/Linux Install tools: KubeAdm Others: What happened: Investigating the code, hexCIDR() expects the CIDR passed to be for an IPv4 address and doesn't support an IPv6 CIDR. What you expected to happen: Support either IPv4 or IPv6 CIDRs. How to reproduce it (as minimally and precisely as possible): Anything else we need to know: Will be working on a solution for this. Note: In some of the call paths, a PodIP string is used and passed in as a CIDR with a "/32" appended. At some point, this would need to be updated as well, to use "/128" if an IPv6 PodIP was used.
I see there is an asciiCIDR() method too that needs addressing, but only in the UTs.


Question: return "", err } ip = ip.Mask(ipnet.Mask) <del> hexIP := hex.EncodeToString([]byte(ip.To4())) </del> <ins> hexIP := hex.EncodeToString([]byte(ip)) </ins> hexMask := ipnet.Mask.String() return hexIP + "/" + hexMask, nil }

Appears that it is expecting to reverse hex back to string, but test is not showing case of mask being properly applied. For example, "2001::5/64" when converted to hex and then back to string will be "2001::/64". Will update test to be more accurate.
/area networking
These labels do not exist in this repository: . <detailsIn response to : /area networking Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. I understand the commands that are listed . </details
/sig network
One question I have... Can the callers (bandwidth?) of the function work with an IPv6 based hex CIDR? In the future, if we support pod IPs that are IPv6 addresses, and we modify the code to add /128 vs /32, as is done currently, this would create an IPv6 based hex CIDR.
The only caller is currently the ingress/egress shaping function in kubelet. That code also needs to be updated for IPv6. For example, assumes IPv4 when adding CIDRs to the shaper list.
Agree.I wanted to separate this out, partly to make the change smaller, but also, when I looked at the callers (this shaping functionality) I wasn't sure if it would support IPv6. I looked a bit at the "tc" command and saw some notes that indicate that it doesn't support IPv6 yet. Is that still accurate? Would it be ok to tackle that separately?


Question: - createCoreV1NamespacedServiceAccountToken - createCoreV1Node - createStorageV1CSINode <del> - createStorageV1VolumeAttachment </del> - deleteCoreV1Node - deleteStorageV1CollectionCSINode <del> - deleteStorageV1CollectionVolumeAttachment </del> - deleteStorageV1CSINode <del> - deleteStorageV1VolumeAttachment </del> - getInternalApiserverAPIGroup - getResourceAPIGroup - listStorageV1CSINode <del> - listStorageV1VolumeAttachment </del> - patchStorageV1CSINode <del> - patchStorageV1VolumeAttachment </del> - patchStorageV1VolumeAttachmentStatus - readStorageV1CSINode <del> - readStorageV1VolumeAttachment </del> - readStorageV1VolumeAttachmentStatus - replaceStorageV1CSINode <del> - replaceStorageV1VolumeAttachment </del> - replaceStorageV1VolumeAttachmentStatus

[x] two weeks soak end date : 14 Mar 2024 : According to following APIsnoop query, there are six VolumeAttachment endpoints that are untested. Using a number of existing e2e test practices a new has been created to provide future Conformance coverage for the 7 endpoints. The e2e logs for this test are listed below. This query shows the following VolumeAttachment endpoints are hit within a short period of running this e2e test. If a test with these calls gets merged, test coverage will go up by 7 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The Kubernetes project currently lacks enough contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: const dummyRegistryEmail = "name@contoso.com" <ins> var containerRegistryUrls = []string{"*.azurecr.io", "*.azurecr.cn", "*.azurecr.de", "*.azurecr.us"} </ins> // init registers the various means by which credentials may // be resolved on Azure. func init() {


Question: const dummyRegistryEmail = "name@contoso.com" <ins> var containerRegistryUrls = []string{"*.azurecr.io", "*.azurecr.cn", "*.azurecr.de", "*.azurecr.us"} </ins> // init registers the various means by which credentials may // be resolved on Azure. func init() {

After my repro and investigation(really spent a lot of time...), I found the issue lies here, would return empty even after granting sp access to azure acr, and after around 15min, it returns correct loginServer list: Looks like it's a SDK or acr list with specific sp issue, workaround is wait around 15min or more time, it could recover by itself. Paste my investigation logs here, I grant sp access to acr in around , you could see it returns empty registry list first and then after a few more minutes, it returns the correct list with loginServer:
BTW, in v1.12, acr list() uses new API, I will check whether in v1.12 would have same issue. Acutally I have already implemented LazyProvide func(check here: ), while it won't fix the original issue.
Update: After a few more tries, I found it's related to this code snippet, before , we need to generate a new ServicePrincipalToken every time: I will move this code snippet into Provide() func, and send out a PR! The failure recover time is within 2min now.


Question: if len(name) < 1 || len(name) > 2 { klog.Warningf("Unexpected EBS DeviceName: %q", aws.StringValue(blockDevice.DeviceName)) } <ins> if blockDevice.Ebs != nil && blockDevice.Ebs.VolumeId != nil { volumeStatus[EBSVolumeID(*blockDevice.Ebs.VolumeId)] = aws.StringValue(blockDevice.Ebs.Status) } </ins> deviceMappings[mountDevice(name)] = EBSVolumeID(aws.StringValue(blockDevice.Ebs.VolumeId)) }

E2e tests occasionally flake on AWS. It happens randomly to all tests that create a pod with a volume, write something to the volume, delete the pod and create a new one that reads from the volume. For example: /D controller detaches a volume from a node (= the old pod died) after that (under 1s) it tries to attach the same volume to the same node (=the new pod is scheduled and coincidentally it uses the same node). In most cases this works, however, few times per day it happens that the second attach (in Kubernetes terms, see below) does not succeed and the test times out. AWS cloud provider in Kubernetes heavily uses DescribeInstances and DescribeVolumes to check that a volume has been attached or detached: In step 1. above, Kubernetes issues AWS API call and polls until it reports that the volume has been detached. In step 2. above, Kubernetes issues AWS API call to check what devices are used on a node and then it tries to find a free device for the volume that should be attached. In this bug, in step 2 reports the volume as detaching, even that reported the volume as detached in step 1. There was no call in between! Kubernetes does not expect that this could ever happen and does not try attach the volume (it's not reported as detached, so why bother, right?). This is the code that finds out that the volume is not detached ("detaching" != "detached"): And few lines below we skip : We (OpenShift) run lot of Kubernetes e2e tests on AWS. We run them in a single AWS project and I guess we hit some timing issues / races on AWS side. We hit also AWS API quota pretty hard, but it does not seem to be related to this bug, I mention it only to illustrate the amount of things that run in parallel and that AWS may not like that load. Environment: Kubernetes v1.13.4 /kind bug /sig aws /sig storage
Flow from AWS API perspective: So far so good. = volume vol- is back detaching: All timing is related to a response received, processed and logged by Kubernetes, which may be different to time corresponding request was sent. Everything in this report happens in sequence, there is no pararellism for a single volume.


Question: if len(name) < 1 || len(name) > 2 { klog.Warningf("Unexpected EBS DeviceName: %q", aws.StringValue(blockDevice.DeviceName)) } <ins> if blockDevice.Ebs != nil && blockDevice.Ebs.VolumeId != nil { volumeStatus[EBSVolumeID(*blockDevice.Ebs.VolumeId)] = aws.StringValue(blockDevice.Ebs.Status) } </ins> deviceMappings[mountDevice(name)] = EBSVolumeID(aws.StringValue(blockDevice.Ebs.VolumeId)) }

(Other volumes may be and often are attached / detached in parallel to this flow).
this looks like a bug in AWS itself, seems to return old data randomly. Can you please check AWS side of things? And we probably need fix / workaround on Kubernetes side. How can we check that a volume has been 100% detached if returns inconsistent data? We can check if a volume is in and not report is as attached here: This will cause Kubernetes to re-use the same device name and call . How bulletproof is it going to be? And most importantly, what other parts of Kubernetes depend on being consistent?
I confirmed internally. This is expected behavior for and since ec2 API follows a where resource mutation might be not reflected immediately. This could explain why returns a volume in detaching state even after the previous call returns without the volume (detached) I think this won't work in the case when the volume is actually stuck in detaching state, since attaching a stuck volume could simply make it worse. Given that, how about, in call, we return error when notice the volume is still being returned from and we rely on the AD controller retry to check the state next time, and hopefully the state converges.
If can return stale data, depending on it in (or anywhere else) probably won't help, because we may again get new or stale data "randomly". We call in today to get fresh list of used/free devices. Depending on timing, two cases happen: Sometimes it returns that the volume is detaching and I can see in logs. Handling as you suggest would help here. Sometimes it returns that the volume is detached. We find a new device name and call AWS and start polling for finished attach. This then returns that the volume is detaching from the node with the old device name. So we're actually attaching a volume that's being detached and you wrote that's a bad thing. In my logs, I can see that Kubernetes recovers from this error: Volume plugin AttachVolume() fails with A/D controller re-tries AttachVolume() calls AttachDisk() that calls to find a free device. returns that the volume is actually being attached to the node (I can see in logs) AttachDisk() polls until the volume is attached. -it works, but to me it feels like it's just an accident. Checking can't help in this case.


Question: if len(name) < 1 || len(name) > 2 { klog.Warningf("Unexpected EBS DeviceName: %q", aws.StringValue(blockDevice.DeviceName)) } <ins> if blockDevice.Ebs != nil && blockDevice.Ebs.VolumeId != nil { volumeStatus[EBSVolumeID(*blockDevice.Ebs.VolumeId)] = aws.StringValue(blockDevice.Ebs.Status) } </ins> deviceMappings[mountDevice(name)] = EBSVolumeID(aws.StringValue(blockDevice.Ebs.VolumeId)) }

So, is there another way how to check that a volume is really detached? Or should we promote the error flow described above to official fix and add bunch of comments around so we won't remove / tweak the flow in future? What bad can happen when we call AWS on a volume that's detaching? AWS at least once reported that the volume is detached, so the volume probably is not be stuck.
I think it is only bad if the volume was previously stuck. But feels acceptable for me if it is being detached and the previous call in at least confirmed once. Confirming internally. But I suspect there isn't any. As long as the volume is not stuck (unmounted properly), we should be able to attach a detaching volume especially when it is at least confirmed being detached once from call in . Even if the first time it doesn't work, retry should help recover the situation. How do you think?
According pointed, AWS recommends two actions to manage the eventual consistency model (k8s already does 1, but AFAIK not 2, except when there's an exponential backoff among calls: 1) Run the appropriate call before modifying a resource 2) Add wait time between subsequent commands, even if returns a green response (this does infer that might return stale data) Currently, k8s assumes a state based on a single green response, e.g., is called repeatedly with an exponential backoff until it returns a given volume state. What if it required at least two positive responses instead? What are the chances that would return stale data twice? I'd assume that two calls returning a detached volume (with exponential backoff in between, which enforces point 2 above) would be reliable enough. The downside is that this would increase the API usage, which is already high.
do you know if the native AWS calls may be more stale-data-proof?
Looking at implementation, it tries with constant backoff of 15 seconds (which I consider pretty large) and it returns when the first returns that the volume is in use, so basically the same as what Kubernetes does.
From what I saw, sometimes it happens that reports a volume as detached, subsequent reports that the volume is not attached, succeeds and only after that shows that the volume is (from the old device name) instead of . So Kubernetes got at least two confirmations that the volume is detached.


Question: if len(name) < 1 || len(name) > 2 { klog.Warningf("Unexpected EBS DeviceName: %q", aws.StringValue(blockDevice.DeviceName)) } <ins> if blockDevice.Ebs != nil && blockDevice.Ebs.VolumeId != nil { volumeStatus[EBSVolumeID(*blockDevice.Ebs.VolumeId)] = aws.StringValue(blockDevice.Ebs.Status) } </ins> deviceMappings[mountDevice(name)] = EBSVolumeID(aws.StringValue(blockDevice.Ebs.VolumeId)) }

What would be the correct delay to wait for second (third, ...) confirmation that the volume is really detached? IMO it would hurt our performance a lot.
The Wait function simply calls DescribeXX under the hood and uses some hard coded timeout. So it's similar to what kubernetes does like said.


Question: } return fresh, nil }) // Use ControllerRefManager to adopt/orphan as needed. <del> cm := controller.NewPodControllerRefManager(dsc.podControl, ds, selector, controllerKind, canAdoptFunc) </del> <ins> cm := controller.NewPodControllerRefManager(dsc.podControl, ds, selector, controllerKind, dsNotDeleted) </ins> return cm.ClaimPods(pods) }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: The DaemonSet controller will schedule a pod on a node when the old pod is still in Terminating state. This breaks the expectation that only one pod will be running at a time per node. What you expected to happen: The DaemonSet controller should not let two pods running at the same time on the same node. How to reproduce it (as minimally and precisely as possible): Create a DaemonSet. Set terminationGracePeriodSeconds to be relatively long. Kill a pod. Watch a new pod start on the same node when the old pod is still terminating. Anything else we need to know?: Seems this behavior was introduced in Environment: Kubernetes version (use ): 1.7.2 Cloud provider or hardware configuration**: GKE OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig apps
/cc
cc
cc: I think the expected behavior is correct, the terminating pod could be performing an important step such as 'removing a lock' or 'cleanup' etc., we should allow that to gracefully die before creating a new one. IMHO we should revoke to old behavior, but why did we need this new behavior in the first place? That is not clear to me from the PR and Issue.
We don't currently guarantee at least or at most one scheduling for DaemonSet. Both behaviors are under consideration via issue and in the future work section of the . The reason for is to "Make sure that controllers are not adopting children that are marked for deletion" (). This is necessary for GC to work properly.
Well the behavior may be technically undefined, but can I request that in the future changes like this be mentioned in the release notes?
Yes, certainly.


Question: } return fresh, nil }) // Use ControllerRefManager to adopt/orphan as needed. <del> cm := controller.NewPodControllerRefManager(dsc.podControl, ds, selector, controllerKind, canAdoptFunc) </del> <ins> cm := controller.NewPodControllerRefManager(dsc.podControl, ds, selector, controllerKind, dsNotDeleted) </ins> return cm.ClaimPods(pods) }

Also "The DaemonSet controller should not let two pods running at the same time on the same node." is not necessarily the same thing as respecting termination prior to launching a new Pod. The former implies a more strict requirement than the latter. I think we should consider at least attempting to wait for a terminating Pod to be deleted prior to creating another Pod on the Node for same DaemonSet as default behavior.
will something like work then?


Question: default: *loopbackClientConfig = secureLoopbackClientConfig <del> // Write to the front of SNICerts so that this overrides any other certs with the same name (*secureServingInfo).SNICerts = append([]dynamiccertificates.SNICertKeyContentProvider{certProvider}, (*secureServingInfo).SNICerts...) </del> } return nil

The code block does these actions: the self-signed cert for loopback connection the loopbackClientConfig the self-signed cert to the If the main cert () is empty, the loopbackClientConfig will not be created due to this check, loopbackClientConfig shouldn't be left empty, because we have created the SNICert, it's just that we haven't executed action 3 to add the SNICert to the list. Switching the order of action 2 and action 3 will make the loopbackClientConfig successfully created. /kind bug /sig auth api-machinery


Question: } function upgrade-master() { <ins> local num_masters num_masters=$(get-master-replicas-count) if [[ "${num_masters}" -gt 1 ]]; then echo "Upgrade of master not supported if more than one master replica present. The current number of master replicas: ${num_masters}" exit 1 fi </ins> echo "== Upgrading master to '${SERVER_BINARY_TAR_URL}'. Do not interrupt, deleting master instance. ==" # Tries to figure out KUBE_USER/KUBE_PASSWORD by first looking under

Script does not support upgrade of clusters with HA master setup.
CC
PR does not allow to run upgrade scripts on cluster with HA masters. The upgrade scenario for HA clusters from 1.5 to 1.6 may be the following: all master replicas but one, master upgrade, master replicas again.
I haven't reviewed the PR yet, but the scenario you are proposing defeats the purpose of HA master! I think it's ok to say we do not support HA master upgrades yet, but we should document how this can be done manually and fix this properly later by adding support in
It's too late to add this in for 1.6.0, but we should consider backporting a fix so that users can update an HA master to 1.6.x in the near future. We should also put in a big warning in the release notes warning users not to try and upgrade if they deployed an HA master using kube-
/sig cluster-lifecycle
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale /lifecycle frozen
/remove-lifecycle frozen /sig cloud-provider /area provider/gcp changing with ownership of ""
/remove-sig cluster-lifecycle
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or .


Question: } function upgrade-master() { <ins> local num_masters num_masters=$(get-master-replicas-count) if [[ "${num_masters}" -gt 1 ]]; then echo "Upgrade of master not supported if more than one master replica present. The current number of master replicas: ${num_masters}" exit 1 fi </ins> echo "== Upgrading master to '${SERVER_BINARY_TAR_URL}'. Do not interrupt, deleting master instance. ==" # Tries to figure out KUBE_USER/KUBE_PASSWORD by first looking under

/close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: statustest "k8s.io/kubernetes/pkg/kubelet/status/testing" ) <del> func TestRemoveTerminatedContainerInfo(t *testing.T) { </del> <ins> func TestFilterTerminatedContainerInfoAndAssembleByPodCgroupKey(t *testing.T) { </ins> const ( seedPastPod0Infra      = 1000 seedPastPod0Container0 = 2000

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --For single node with hundreds and thousands of pods and containers, the cpu usage of kubelet becomes high, one bottleneck is the cadvisor stats provider as below: the method iterate all containerInfos for every podStats cause high cpu the method should not iterate allcontainerInfos for every podStats, use map instead Kubernetes version (use ): v1.19.4 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig node
/triage accepted
/assign
I’ve done an analysis on the code and this is what I have found, I believe we have two issues to address. There are multiple range loops being performed over v2.ContainerInfo / PodStat maps within the ListPodStatus method which can be contributing to the high cpu utilisation. The allInfos argument which is supplied to the getCadvisorContainerInfo function is a map which uses a cgroup path as the key, in order to find matching container info to podUID, processing of the keys in the map needs to be performed until a match is found. Examples of keys are as follows: /kubepods/burstable/podf9e3a173-1470-451a-9ca2-/d8426b82582fbcfae16085bc794c49769d3a9648bd5463976aca4071f9342754 /docker/2dfa17fc81e781212bcc26e189ebb724721fe097394b4de3cbeb7e069bf23290 My recommendation would be to apply the following updates: Refactor the getCadvisorContainerInfo function to allow for a filter and key type parameter to be supplied by the caller, this enables reducing the number of range loops being performed over the container info map by removing elements that are not required and the ability to set the key being used in the returned container infos map to enable fast lookup of container info.


Question: statustest "k8s.io/kubernetes/pkg/kubelet/status/testing" ) <del> func TestRemoveTerminatedContainerInfo(t *testing.T) { </del> <ins> func TestFilterTerminatedContainerInfoAndAssembleByPodCgroupKey(t *testing.T) { </ins> const ( seedPastPod0Infra      = 1000 seedPastPod0Container0 = 2000

My initial thoughts are to use an enumeration which has two values KEYCGROUPPATH to provide backwards compatibility and KEYPODUID to enable fast lookups via podUID Refactor ListPodStats method to be more efficient in constructing the PodStats slice and taking into account the fast lookup via podUID I think this is the best approach instead of pushing the data structure implementation for the container info upstream to the cadviser clients, this will at least initially provide a better optimisation scoped within the kubelet/stats package and reduce other possible impacts to changing the cadvisor clients Please advise.
Just realized this looks like a duplicate issue and looking as fixed by PR Both this issue and were raised by Zac
/unassign
/close
You can't close an active issue/PR unless you authored it or you are a collaborator. <detailsIn response to : /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Didn't think I could close this, leaving this with
this will be automatically closed once the PR merges


Question: } type nodeHolder struct { <ins> lock sync.Mutex </ins> node *v1.Node } <ins> func (n *nodeHolder) setNode(node *v1.Node) { n.lock.Lock() defer n.lock.Unlock() n.node = node } </ins> func (n *nodeHolder) getNode(name string) (*v1.Node, error) { <ins> n.lock.Lock() defer n.lock.Unlock() </ins> return n.node, nil }

What happened: A Pod was scheduled for deletion because a taint w/ effect was observed in the Node the Pod was assigned to, and after the taint is removed from the Node, the Pod scheduled deletion isn't canceled. After scheduling pod deletion, the only cancels pod deletion if all taints are removed from the node, including any taints that the user relies on for their use-cases (example below). What you expected to happen: After the taint that triggered scheduling pod deletion is removed, cancels the scheduled pod deletion. How to reproduce it (as minimally and precisely as possible): A node is tainted w/ , so any workloads already assigned to it which don't tolerate this taint are evicted. A pod is created w/ tolerations: w/ unspecified and w/ . is assigned to . At some point in time after the previous steps took place, is tainted as . gets an update event for and observes two taints. It proceeds to calculate the minimum time (in seconds) that tolerates for the two taints which at this point is 300 seconds, and marks for deletion in ~300 seconds - this happens in-memory, in a timed-queue. At some point in time after the previous steps took place, and before the 300 seconds trigger happen, taint is removed from . gets an update event for and observes only one taint, . It proceeds to calculate the minimum time (in seconds) that tolerates for the this taint which is infinity, and returns, not canceling the previous deletion. It completely ignores the fact that the taint that triggered deletion is no longer observed. Anything else we need to know?: Worked w/ to figure this out after all pods in a node got evicted without apparent reason. Environment: Kubernetes version (use ): 1.17.3 (but looking at the code seems to affect all of 1.17.x and 1.18.x and master. Cloud provider or hardware configuration: N/A OS (e.g: ): N/A Kernel (e.g. ): 5.x Install tools: N/A Network plugin and version (if this is a network-related bug): N/A Others: N/A cc (owners of this code)
There seems to be at least .


Question: } type nodeHolder struct { <ins> lock sync.Mutex </ins> node *v1.Node } <ins> func (n *nodeHolder) setNode(node *v1.Node) { n.lock.Lock() defer n.lock.Unlock() n.node = node } </ins> func (n *nodeHolder) getNode(name string) (*v1.Node, error) { <ins> n.lock.Lock() defer n.lock.Unlock() </ins> return n.node, nil }

You can sort of get a feel for what's going on here by looking at these two code snippets: This is what happens in when the node does not have any NoExecute taints: If you have any NoExecute taints on the node processPodOnNode gets called and this routine is really, really eager to evict pods. So : We are currently working around this issue by changing over our NoExecute taints to NoSchedule.
/sig scheduling node
/cc
/remove-sig scheduling scheduler isn't involved in this case.
Which SIG owns the controller-manager or is the node lifecycle manager (and its NoExecuteTaintManager) owned by sig-node alone?
I think node lifecycle is jointly owned by sig-node and sig-cloud-provider
API machinery owns the mechanics of the controller manager (controller loop setup and management), but the individual controllers are SIG-specific
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Looks like we lost some momentum here and I'd like to avoid this falling into the abyss of bugs that never get fixed. When this bug is triggered the outcome is pretty terrible and once it does happen it is very difficult to dig into and understand what actually happened. Did we ever get this routed to the right owner?
adding sig-node leads for triage/routing. this seems like a clearly reproducible bug with significant correctness issues. /assign
thanks for the detailed report and reproducer. to help accelerate this, do you have a form of your reproducer we could add in the e2e suite (even if its failing initially)?
custom taints are considered disruptive, so it couldn't run in the main suite
It's been a while since I wrote tests for the e2e suite but if you are available over slack in case guidance is needed, I'm willing to put the effort.
Do all suites need to pass for a fix to be merged? I'd assume it is the case, otherwise how would a regression be prevented after the fix lands?