Question: }, }, }, <del> Required: []string{"maxPerCore", "min", "tcpEstablishedTimeout", "tcpCloseWaitTimeout", "udpTimeout", "udpStreamTimeout"}, </del> <ins> Required: []string{"maxPerCore", "min", "tcpEstablishedTimeout", "tcpCloseWaitTimeout", "tcpBeLiberal", "udpTimeout", "udpStreamTimeout"}, </ins> }, }, Dependencies: []string{

What happened: When kube-proxy is configured with a cluster-cidr range, it drops packets with an INVALID conntrack state for all sources on the KUBE-FORWARD chain. This is meant to prevent spurious retransmits in long-lived TCP connections to a service IP, but it prevents clusters from having asymmetrical routing on private subnets with services outside of a cluster. What you expected to happen: Packets on the KUBE-FORWARD chain with an INVALID conntrack state should be dropped for only sources inside of the cluster-cidr if set. How to reproduce it (as minimally and precisely as possible): N/A Anything else we need to know?: Change to drop INVALID conntrack packets was in Backing out that commit and compiling is my current workaround. Environment: Kubernetes version (use ): 1.17.2 Cloud provider or hardware configuration: OS (e.g: ):Kernel (e.g. ):
/sig network
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
I am reading but I don't fully grok what caused the problem in the first place, or why we think "only drop if it was in-cluster" is a correct fix? Edit, OK I get the original problem, but still don't understand why packets from outside the pod range would not exhibit , thereby requiring yet another revisit of this problem. Is this a conntrack bug? Should it be applying the DNAT?
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
I can try to give a bit more context to our setup, but the knowledge I have around this all dates back to Feb, unfortuately, so I may have forgotten details. We were replacing our mesos cluster with a kubernetes cluster in Feb and using a custom ingress solution (HAProxy w/an outside process configuring it for pod ips / mesos ip-ports) for both. The ingress nodes lived in the private subnet, but outside of the kubernetes cluster cidr range. When we upgraded from kubernetes 1.14 to kubernetes 1.15 (the first version with PR ) our ingress could no longer talk to any pod ips.


Question: }, }, }, <del> Required: []string{"maxPerCore", "min", "tcpEstablishedTimeout", "tcpCloseWaitTimeout", "udpTimeout", "udpStreamTimeout"}, </del> <ins> Required: []string{"maxPerCore", "min", "tcpEstablishedTimeout", "tcpCloseWaitTimeout", "tcpBeLiberal", "udpTimeout", "udpStreamTimeout"}, </ins> }, }, Dependencies: []string{

I discovered that the KUBE-FORWARD chain had a rule to drop invalid packets, which was in Compiling kubernetes 1.15 without this commit fixed our problems, and we have been carrying that patch forward ever since. I would like to get rid of our forked kubernetes, hence this ticket. When I was writing this ticket up, I also found also mentions the same issue with the iptables dropping INVALID conntrack state packets near the end, regarding asymmetrical routing. Edit, OK I get the original problem, but still don't understand why packets from outside the pod range would not exhibit , thereby requiring yet another revisit of this problem. I think you are correct, I can imagine packets from outside of a cluster could exhibit the same behavior as packets inside the cluster and cause spurious resets. However, I am wondering if kube-proxy should be managing packets from outside of the cluster on a node... That seems outside of the scope of kube-proxy.
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
I took a stab at implementing what I think is the correct behavior for kube-proxy in .
/assign
I think that another option is to set
I don't understand how that will help me here. The packets with a ctstate INVALID are not generated due to conntrack saturation (), but due to asymmetric routing in our network. See for an example of a similar setup.
are the packets on your asymmetric routing setup marked as invalid if you set ? it should only mark the ones that are out of window per the description
/triage accepted
I'd love to pin this down, but I don't have the means to set up a repro. Is there a way I can set up a way to experiment?
We were bit by this too after revisiting an old service that had been unused since the heady days of pre-, and would also like to see the proposed patch merged (or a configuration option to prevent insertion of the INVALID state drop rule - either suits us). I unfortunately don't have any useful input as to setting up for reproduction. In our case we have a pretty odd anycast set up with a small pool of (public) addresses managed by calico.


Question: }, }, }, <del> Required: []string{"maxPerCore", "min", "tcpEstablishedTimeout", "tcpCloseWaitTimeout", "udpTimeout", "udpStreamTimeout"}, </del> <ins> Required: []string{"maxPerCore", "min", "tcpEstablishedTimeout", "tcpCloseWaitTimeout", "tcpBeLiberal", "udpTimeout", "udpStreamTimeout"}, </ins> }, }, Dependencies: []string{

However I do seem to have a dirty workaround for small clusters. By setting notrack (ctstate UNTRACKED?) in the raw table for packets in our specific ranges that would otherwise match ctstate INVALID, the packets aren't dropped. Which may be of help to someone else who finds this issue.
We are seeing something very similar on large file downloads and our issue exactly fits this one's subject. These RESET (corrected after are happening intermittently and I am seeing similar behavior mentioned in the blog mentioned below. I am unsure if someone has found a workaround for this issue or not, so yeah my +1 to this issue being looked up by the community.
We're using as a solution for the connection reset we were seeing and might fix this issue.
Let me try to avoid that this thread derails and see if I can get current status: -proxy install rules to drop INVALID cstate connections, this solves the ~REJECT~ RESET problems are some scenarios, with asymmetric routing per example, that this rules may drop unrelated traffic. The request is to remove the rule that drops those INVALID states or to try to restrict it, and that can cause that people start to have the ~REJECT~ RESET problems and drops again. If we have a reproducer we can be more precise in the final solution, but removing one rule without any strong evidence in kube-proxy is very risky As a workaround for the asymetric routing problem, I suggested to set the nfconntracktcpbeliberal on the kernel: I think that will solve the problems, it will be good if someone can test it and provide feedback
No sadly, its my bad, it is not REJECT, it is RESET that's happening and conntrack has nothing to do with it, mostly (I hope). Kubeproxy is doing what is intended but specifically when there are external agents in play, the packet doesn't sends an ACK back to the response of SYN-ACK. Our case, we hit A to initiate a download, A calls B to get the action completed to generate download data for the user finally via their browser. A and B both are on k8s, we saw A starts and a complete cycle goes for sometime until there multiple parallel downloads requested where few gets completed and few gets a RESET.


Question: }, }, }, <del> Required: []string{"maxPerCore", "min", "tcpEstablishedTimeout", "tcpCloseWaitTimeout", "udpTimeout", "udpStreamTimeout"}, </del> <ins> Required: []string{"maxPerCore", "min", "tcpEstablishedTimeout", "tcpCloseWaitTimeout", "tcpBeLiberal", "udpTimeout", "udpStreamTimeout"}, </ins> }, }, Dependencies: []string{

On primary debugging, it appears that RESET was getting invoked by the pod but pod is just sending a RST-ACK not RST itself. The missing packet is never found in the default route, unless we see that there is a chance where ACK is missed and we found some lines in tcpdump with missing sequence for ACKs. Thats why i doubt my issue is related here. One more weird coincidence is few downloads have spurious transmission.
Thankyou, I will take a look, but mostly I would like to wait before using it, debug more to understand the actual cause and then act on it.
sorry , I wanted to say RESET, thanks for catching it. If you think is different and you can provide information and how to reproduce it, I suggest you to open a new issue
I will sure do, when it is different. For now I would like to watch and submit my findings if it is related. PS: sysctl -w net.netfilter.nfconntracktcpbeliberal=1 is already implemented and we are testing if it does something helpful. so far it doesn't looks like it. thanks for your help.
/area kube-proxy
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle rotten
I have the feeling we are also affected by this with a standard calico installation: every node announces a /122 IPv6 network to the upstream router the upstream router uses ECMP to route the packet back packets that are sent from* the pod and returned via asymmetric path from outside seem to be dropped
and if you set ?
Same behaviour with set on all nodes, the outgoing pod traffic does not return. And I instantly see packets going back into the drop rule:
:thinking:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.


Question: }, }, }, <del> Required: []string{"maxPerCore", "min", "tcpEstablishedTimeout", "tcpCloseWaitTimeout", "udpTimeout", "udpStreamTimeout"}, </del> <ins> Required: []string{"maxPerCore", "min", "tcpEstablishedTimeout", "tcpCloseWaitTimeout", "tcpBeLiberal", "udpTimeout", "udpStreamTimeout"}, </ins> }, }, Dependencies: []string{

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hi, are you still working on this? Same problem happened in our clusters, just because of asymmetrical routing.
/reopen
Reopened this issue. <detailsIn response to : /reopen </details
Also the command suggested by is helping us, but I still think this should be a bug of kube-proxy.
The problem here is: what happens if the solutions are exclusive or we regress? The addition of this rule fixed a legit bug that was documented and there is also a nice blog explaining the details Removing this rule without a repro and understanding what is the root cause of this issue is not really the best way to proceed. I think that people affected should try to find the root cause and explain the problem or just provide a way to reproduce it, so we can fix it
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at .


Question: }, }, }, <del> Required: []string{"maxPerCore", "min", "tcpEstablishedTimeout", "tcpCloseWaitTimeout", "udpTimeout", "udpStreamTimeout"}, </del> <ins> Required: []string{"maxPerCore", "min", "tcpEstablishedTimeout", "tcpCloseWaitTimeout", "tcpBeLiberal", "udpTimeout", "udpStreamTimeout"}, </ins> }, }, Dependencies: []string{

/close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The same problem happened in our cluster, the same as because of asymmetrical routing. the command suggested by doesn't work for us, and I delete the iptables rule and add new rule(refer to ), it works I think that makes sense I'll put up a reproduction of the gist link later on /reopen
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The main problem is how to be completely sure we don't regress, this comment from Tim is spot on
I'm trying to check that non-local connections will still reset without the INVALID Rule
I think the patch in is wrong; it changes the rule from "if INVALID then drop" to "if INVALID and from a pod, then drop" so that it doesn't affect non-kubernetes traffic, but I think the rule we actually need is "if INVALID and to a service, then drop". In particular, I believe we need to make sure that the rule applies to traffic with a cluster-external source and an in-cluster destination. It's possible that changing it to "if INVALID and to a pod, then drop" would also work. I skimmed and some linked bugs elsewhere but I was unable to confirm whether that rule would cover the reporters' use cases.
I think that is only INVALID and to a pod
In my case, cluster-external source to service is both ok with the INVALID rule or without the INVALID rule. It seems the rule doesn't affect on access service.


Question: }, }, }, <del> Required: []string{"maxPerCore", "min", "tcpEstablishedTimeout", "tcpCloseWaitTimeout", "udpTimeout", "udpStreamTimeout"}, </del> <ins> Required: []string{"maxPerCore", "min", "tcpEstablishedTimeout", "tcpCloseWaitTimeout", "tcpBeLiberal", "udpTimeout", "udpStreamTimeout"}, </ins> }, }, Dependencies: []string{

I add this rule but it doesn't work
So, based on that, the bug happens when a reply packet fails to get un-DNAT'ed, causing the client to then send a RST to the server, which matches the source/destination IP that the server knows of. But this could happen if you had a service with external IPs too; in that case the reply packet would be from the external endpoint IP to the node IP (because the packet would have been masqueraded), and if conntrack didn't unmap it correctly then the node would send a RST to the external IP, breaking the connection in just the same way. In this case there is no good way kube-proxy can recognize the packet as being its own, other than by using conntrack, which presumably won't work if the packet is INVALID?
It seems that there is no better way to solve this for external sources accessing the cluster CIDR, is it right? If so, Can we have a flag to decide whether to create The invalid rule? It would be nice that It's up to the user to decide whether to create the rule.
There's related discussion in . I think we should create whatever iptables rules we can that will fix up our own traffic without interfering with other traffic, and also document the sysctl that admins can use to change conntrack behavior to work around the problem. So probably, we should change the "drop if INVALID" rule to "drop if INVALID and to a pod", and also address the problem in , but not try to solve the problem with external IPs I mentioned in because we don't have a good way of fixing that without potentially breaking other things. (And it's kind of way more of an edge case anyway.) Note that can currently output rules for "from a pod", but not for "to a pod", so that will need some tweaking. (FWIW will also eventually need the ability to output nftables rules rather than iptables rules so that's another thing to think about if someone is going to change its API.)
So in the absence of a way to fix the problem with external IPs, can we have a flag to decide whether to create this rule or not? If true (default), create the drop if INVALID and to a pod rule. Otherwise, We do not create the rule.


Question: }, }, }, <del> Required: []string{"maxPerCore", "min", "tcpEstablishedTimeout", "tcpCloseWaitTimeout", "udpTimeout", "udpStreamTimeout"}, </del> <ins> Required: []string{"maxPerCore", "min", "tcpEstablishedTimeout", "tcpCloseWaitTimeout", "tcpBeLiberal", "udpTimeout", "udpStreamTimeout"}, </ins> }, }, Dependencies: []string{

I believe that even without this rule we can solve this problem by way of sysctl, right? I think we should do the following things We need a flag to disable the rule We should change the "drop if INVALID" rule to "drop if INVALID and to a pod" We document sysctl as a way to solve the problem.
If the rule is "drop if INVALID and to a pod", we shouldn't need a flag to disable it; it won't affect non-pod traffic, and it won't interfere with users that want to set the sysctl instead. (Setting the sysctl will cause packets to not be marked as INVALID so the rule we add just won't get hit.)
OK, drop if INVALID and to a pod is help us identify cluster traffic, right? We only drop cluster traffic if it's invalid, but this will not solve the problem of external IP. So we need a flag to disable the rule is drop if INVALID?
The same problem happened in our cluster, the same as because of asymmetrical routing None of the above methods solve the problem in our ENV. Are there any other workarounds?
Can you execute the following command on each node? I think this is a workaround.
Thank you for your reply I also tried removing this rule. However, k8s will reconfigure the deleted rules to the system at some point in time.
I worded that badly: I meant "external IP" as in "the endpoints are cluster-external IPs rather than pod IPs", not as in . So that would apply only to people using kube-proxy to load-balance traffic from clients outside the cluster to servers outside the cluster, which is a pretty weird use case. And nobody ever reported the packet RST bug for that case anyway. So I was arguing that we can just ignore that case. So: change the existing "drop if invalid" rule to "drop if invalid and to a pod" document the sysctl We don't need an option to disable the drop rule, because it won't affect non-k8s traffic, and it will automatically become a no-op if someone decides to use the sysctl instead.
Thank you for your response. I'd love to take this one. But I have a question, How to define k8s traffic?


Question: }, }, }, <del> Required: []string{"maxPerCore", "min", "tcpEstablishedTimeout", "tcpCloseWaitTimeout", "udpTimeout", "udpStreamTimeout"}, </del> <ins> Required: []string{"maxPerCore", "min", "tcpEstablishedTimeout", "tcpCloseWaitTimeout", "tcpBeLiberal", "udpTimeout", "udpStreamTimeout"}, </ins> }, }, Dependencies: []string{

If a pod in a cluster, we call it an underlay pod, its IP is not allocated from cluster CIDR, this is a real underlay IP, when it accesses other underlay pods in the cluster or pods that allocate IP from cluster CIDR(We call it overlay pod), is this k8s traffic? In our cluster, multiple types of CNI pods are implemented through multus running in a cluster, because of the problem of asymmetrical routing and the rule that makes them inaccessible, it seems that the above method will not solve this problem.
For kube-proxy, this is done by the . If you are in a cluster where there is no simple way to discriminate pod from non-pod traffic, then you'd have to run kube-proxy with , and then kube-proxy will see that returns and so it won't try to distinguish local from non-local traffic, and will do some things (like internal/external traffic policy) slightly differently as a result. So in this case, if there was no functioning localDetector, it would also not create the "drop invalid" rule
Thanks for the details, this is probably the case we want.
/assign
I did not understand how you could get asymmetric routing in K8s, but reading up more carefully I saw: If I understand this correctly, this is a load-balancer outside K8s that load-balance directly to POD addresses. To get asymmetric routing some kind of Direct Server Return (DSR) must also be used. So, to get asymmetric routing: from an external source is directed to an external LB LB is configured with POD-IPs as targets and selects one packet is routed to a K8s node, e.g. with ECMP. But is may not be the node where the target POD is executing packet is handled by the POD, but the response packet may be directed to some other GW This will give asymmetric routing. But it's not the every-day K8s network setup for sure. Given the above it's fairly easy to reproduce the problem. You don't even have to setup a load-balancer. The biggest problem may be that you need two ways in to the cluster. Example: ! I setup direct routes that ensures asymmetric routing, which can be checked with on vm-201 and vm-202.


Question: }, }, }, <del> Required: []string{"maxPerCore", "min", "tcpEstablishedTimeout", "tcpCloseWaitTimeout", "udpTimeout", "udpStreamTimeout"}, </del> <ins> Required: []string{"maxPerCore", "min", "tcpEstablishedTimeout", "tcpCloseWaitTimeout", "tcpBeLiberal", "udpTimeout", "udpStreamTimeout"}, </ins> }, }, Dependencies: []string{

From I can now directly ping POD-IPs, but due to the problem in this issue, I can't make a TCP connection: The POD with address runs on . A similar setup as above can be used to verify a PR. Now, there is a simple work-around: use proxy-mode=ipvs The update with the rule never made it to the ipvs proxier (a rather common case :cry: ), but in this case it may save your day. The same as above with proxy-mode=ipvs; The server on 5001 responds with the POD name.
Yeah, Our production environment also solved this problem by switching to IPVS mode. In IPVS mode, there is no the DROP rule.
The sysctl solves the problem in , but not the problem in this issue. Those problems are kind of related, but rather distant.
The same affects outgoing traffic if packets are not masqueraded to the node-ip, as should be a common setup for ipv6, please see For Calico I think you can setup BGP for clusterCIDRs and route incoming packets to PODs to the correct K8s node, instead of using ECMP. This would solve the problem, and save the extra hop.
I met the same issue. KUBE-FORWARD drop my packages.
temptatively solution
The issue fixed by , I'll close it, you can feel free to open it if have any questions. /close
Closing this issue. <detailsIn response to : The issue fixed by , I'll close it, you can feel free to open it if have any questions. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: k8s-app: fluentd-es kubernetes.io/cluster-service: "true" version: v1.22 <del> # This annotation ensures that fluentd does not get evicted if the node # supports critical pod annotation based priority scheme. # Note that this does not guarantee admission on the nodes (#40573). </del> <ins> # This annotation ensures that fluentd does not get evicted if the node # supports critical pod annotation based priority scheme. # Note that this does not guarantee admission on the nodes (#40573). </ins> annotations: scheduler.alpha.kubernetes.io/critical-pod: '' <ins> scheduler.alpha.kubernetes.io/tolerations: '[{"key": "node.alpha.kubernetes.io/ismaster", "effect": "NoSchedule"}]' </ins> spec: containers: - name: fluentd-es

Problem showed, that despite e2e tests for container's logs, we still can encounter regressions for system logs. We need to establish an e2e test for ingesting system components's logs: kubelet docker CC
broke ingesting logs from master and we didn't notice it at once Test should check that logs are present from all nodes, including master (when possible)
Moving to 1.7 as late to happen in 1.6. Feel free to switch back if this is incorrect.
/sig instrumentation do you want this in v1.7? (since you're listed as sig-instrumentation leads at and I can't find a team to ping)
No, thanks for noticing!
Fixed by


Question: metrics = []resettableCollector{ deprecatedRequestGauge, requestCounter, <ins> longRunningRequestsGauge, </ins> longRunningRequestGauge, requestLatencies, responseSizes,

apiserverlongrunninggauge and apiserverregisterwatchers are kinda redundant . We discussed this in our SIG instrumentation meeting and while it's clear that apiserverlongrunninggauge, the name does not conform to instrumentation guidelines (it was created before those guidelines existed). What we want to do is deprecate both metrics and create a new one which actually does conform to our naming conventions probably something like . has volunteered to undertake this. /cc /sig instrumentation
/assign
GitHub didn't allow me to assign the following users: yan-lgtm. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/triage accepted
Hello! Have you still working on this issue? I am new to the community, and can I try it?
Hey I the deprecate in If you want to add the new metric, feel free to do so.
Can I add the new metric? If I can, can you give me some advice about the new metric? Thank you~
/assign


Question: } } <ins> // TestApplyNoExecuteTaintsToNodesEnqueueTwice ensures we taint every node with NoExecute even if enqueued twice func TestApplyNoExecuteTaintsToNodesEnqueueTwice(t *testing.T) { fakeNow := metav1.Date(2017, 1, 1, 12, 0, 0, 0, time.UTC) evictionTimeout := 10 * time.Minute fakeNodeHandler := &testutil.FakeNodeHandler{ Existing: []*v1.Node{ // Unreachable Taint with effect 'NoExecute' should be applied to this node. { ObjectMeta: metav1.ObjectMeta{ Name:              "node0", CreationTimestamp: metav1.Date(2012, 1, 1, 0, 0, 0, 0, time.UTC), Labels: map[string]string{ v1.LabelTopologyRegion:          "region1", v1.LabelTopologyZone:            "zone1", v1.LabelFailureDomainBetaRegion: "region1", v1.LabelFailureDomainBetaZone:   "zone1", }, }, Status: v1.NodeStatus{ Conditions: []v1.NodeCondition{ { Type:               v1.NodeReady, Status:             v1.ConditionUnknown, LastHeartbeatTime:  metav1.Date(2015, 1, 1, 12, 0, 0, 0, time.UTC), LastTransitionTime: metav1.Date(2015, 1, 1, 12, 0, 0, 0, time.UTC), }, }, }, }, // Because of the logic that prevents NC from evicting anything when all Nodes are NotReady // we need second healthy node in tests. { ObjectMeta: metav1.ObjectMeta{ Name:              "node1", CreationTimestamp: metav1.Date(2012, 1, 1, 0, 0, 0, 0, time.UTC), Labels: map[string]string{ v1.LabelTopologyRegion:          "region1", v1.LabelTopologyZone:            "zone1", v1.LabelFailureDomainBetaRegion: "region1", v1.LabelFailureDomainBetaZone:   "zone1", }, }, Status: v1.NodeStatus{ Conditions: []v1.NodeCondition{ { Type:               v1.NodeReady, Status:             v1.ConditionTrue, LastHeartbeatTime:  metav1.Date(2017, 1, 1, 12, 0, 0, 0, time.UTC), LastTransitionTime: metav1.Date(2017, 1, 1, 12, 0, 0, 0, time.UTC), }, }, }, }, // NotReady Taint with NoExecute effect should be applied to this node. { ObjectMeta: metav1.ObjectMeta{ Name:              "node2", CreationTimestamp: metav1.Date(2012, 1, 1, 0, 0, 0, 0, time.UTC), Labels: map[string]string{ v1.LabelTopologyRegion:          "region1", v1.LabelTopologyZone:            "zone1", v1.LabelFailureDomainBetaRegion: "region1", v1.LabelFailureDomainBetaZone:   "zone1", }, }, Status: v1.NodeStatus{ Conditions: []v1.NodeCondition{ { Type:               v1.NodeReady, Status:             v1.ConditionFalse, LastHeartbeatTime:  metav1.Date(2015, 1, 1, 12, 0, 0, 0, time.UTC), LastTransitionTime: metav1.Date(2015, 1, 1, 12, 0, 0, 0, time.UTC), }, }, }, }, }, Clientset: fake.NewSimpleClientset(&v1.PodList{Items: []v1.Pod{*testutil.NewPod("pod0", "node0")}}), } healthyNodeNewStatus := v1.NodeStatus{ Conditions: []v1.NodeCondition{ { Type:               v1.NodeReady, Status:             v1.ConditionTrue, LastHeartbeatTime:  metav1.Date(2017, 1, 1, 12, 10, 0, 0, time.UTC), LastTransitionTime: metav1.Date(2017, 1, 1, 12, 0, 0, 0, time.UTC), }, }, } nodeController, _ := newNodeLifecycleControllerFromClient( fakeNodeHandler, evictionTimeout, testRateLimiterQPS, testRateLimiterQPS, testLargeClusterThreshold, testUnhealthyThreshold, testNodeMonitorGracePeriod, testNodeStartupGracePeriod, testNodeMonitorPeriod, true) nodeController.now = func() metav1.Time { return fakeNow } nodeController.recorder = testutil.NewFakeRecorder() nodeController.getPodsAssignedToNode = fakeGetPodsAssignedToNode(fakeNodeHandler.Clientset) if err := nodeController.syncNodeStore(fakeNodeHandler); err != nil { t.Errorf("unexpected error: %v", err) } // 1. monitor node health twice, add untainted node once if err := nodeController.monitorNodeHealth(); err != nil { t.Errorf("unexpected error: %v", err) } if err := nodeController.monitorNodeHealth(); err != nil { t.Errorf("unexpected error: %v", err) } // 2. mark node0 healthy node0, err := fakeNodeHandler.Get(context.TODO(), "node0", metav1.GetOptions{}) if err != nil { t.Errorf("Can't get current node0...") return } node0.Status = healthyNodeNewStatus _, err = fakeNodeHandler.UpdateStatus(context.TODO(), node0, metav1.UpdateOptions{}) if err != nil { t.Errorf(err.Error()) return } // add other notReady nodes fakeNodeHandler.Existing = append(fakeNodeHandler.Existing, []*v1.Node{ // Unreachable Taint with effect 'NoExecute' should be applied to this node. { ObjectMeta: metav1.ObjectMeta{ Name:              "node3", CreationTimestamp: metav1.Date(2012, 1, 1, 0, 0, 0, 0, time.UTC), Labels: map[string]string{ v1.LabelTopologyRegion:          "region1", v1.LabelTopologyZone:            "zone1", v1.LabelFailureDomainBetaRegion: "region1", v1.LabelFailureDomainBetaZone:   "zone1", }, }, Status: v1.NodeStatus{ Conditions: []v1.NodeCondition{ { Type:               v1.NodeReady, Status:             v1.ConditionUnknown, LastHeartbeatTime:  metav1.Date(2015, 1, 1, 12, 0, 0, 0, time.UTC), LastTransitionTime: metav1.Date(2015, 1, 1, 12, 0, 0, 0, time.UTC), }, }, }, }, // Because of the logic that prevents NC from evicting anything when all Nodes are NotReady // we need second healthy node in tests. { ObjectMeta: metav1.ObjectMeta{ Name:              "node4", CreationTimestamp: metav1.Date(2012, 1, 1, 0, 0, 0, 0, time.UTC), Labels: map[string]string{ v1.LabelTopologyRegion:          "region1", v1.LabelTopologyZone:            "zone1", v1.LabelFailureDomainBetaRegion: "region1", v1.LabelFailureDomainBetaZone:   "zone1", }, }, Status: v1.NodeStatus{ Conditions: []v1.NodeCondition{ { Type:               v1.NodeReady, Status:             v1.ConditionTrue, LastHeartbeatTime:  metav1.Date(2017, 1, 1, 12, 0, 0, 0, time.UTC), LastTransitionTime: metav1.Date(2017, 1, 1, 12, 0, 0, 0, time.UTC), }, }, }, }, // NotReady Taint with NoExecute effect should be applied to this node. { ObjectMeta: metav1.ObjectMeta{ Name:              "node5", CreationTimestamp: metav1.Date(2012, 1, 1, 0, 0, 0, 0, time.UTC), Labels: map[string]string{ v1.LabelTopologyRegion:          "region1", v1.LabelTopologyZone:            "zone1", v1.LabelFailureDomainBetaRegion: "region1", v1.LabelFailureDomainBetaZone:   "zone1", }, }, Status: v1.NodeStatus{ Conditions: []v1.NodeCondition{ { Type:               v1.NodeReady, Status:             v1.ConditionFalse, LastHeartbeatTime:  metav1.Date(2015, 1, 1, 12, 0, 0, 0, time.UTC), LastTransitionTime: metav1.Date(2015, 1, 1, 12, 0, 0, 0, time.UTC), }, }, }, }, }...) if err := nodeController.syncNodeStore(fakeNodeHandler); err != nil { t.Errorf("unexpected error: %v", err) } // 3. start monitor node health again, add untainted node twice, construct UniqueQueue with duplicated node cache if err := nodeController.monitorNodeHealth(); err != nil { t.Errorf("unexpected error: %v", err) } // 4. do NoExecute taint pass // when processing with node0, condition.Status is NodeReady, and return true with default case // then remove the set value and queue value both, the taint job never stuck nodeController.doNoExecuteTaintingPass() // 5. get node3 and node5, see if it has ready got NoExecute taint node3, err := fakeNodeHandler.Get(context.TODO(), "node3", metav1.GetOptions{}) if err != nil { t.Errorf("Can't get current node3...") return } if !taintutils.TaintExists(node3.Spec.Taints, UnreachableTaintTemplate) || len(node3.Spec.Taints) == 0 { t.Errorf("Not found taint %v in %v, which should be present in %s", UnreachableTaintTemplate, node3.Spec.Taints, node3.Name) } node5, err := fakeNodeHandler.Get(context.TODO(), "node5", metav1.GetOptions{}) if err != nil { t.Errorf("Can't get current node5...") return } if !taintutils.TaintExists(node5.Spec.Taints, NotReadyTaintTemplate) || len(node5.Spec.Taints) == 0 { t.Errorf("Not found taint %v in %v, which should be present in %s", NotReadyTaintTemplate, node5.Spec.Taints, node5.Name) } } </ins> func TestSwapUnreachableNotReadyTaints(t *testing.T) { fakeNow := metav1.Date(2017, 1, 1, 12, 0, 0, 0, time.UTC) evictionTimeout := 10 * time.Minute

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Nodelifecycle controller does not set NoExecute taint to NotReady node when kubelet has been stopped What you expected to happen: effect: NoExecute key: Nodelifecycle controller sets NoExecute taint to NotReady node when kubelet has been stopped How to reproduce it (as minimally and precisely as possible): systemctl stop kubelet Anything else we need to know?: I use default args to start kube-controller-manager. I see log from kube-controller-manager: Environment: Kubernetes version (use ): [root ~]# kubectl version Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.6", GitCommit:"", GitTreeState:"clean", BuildDate:"2020-07-15T16:58:53Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.6", GitCommit:"", GitTreeState:"clean", BuildDate:"2020-07-15T16:51:04Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: OS (e.g: ): [root ~]# cat /etc/os-release NAME="CentOS Linux" VERSION="8 (Core)" ID="centos" IDLIKE="rhel fedora" VERSIONID="8" PLATFORMID="platform:el8" PRETTYNAME="CentOS Linux 8 (Core)" ANSICOLOR="0;31" CPENAME="cpe:/o:centos:centos:8" HOMEURL="" BUGREPORTURL="" CENTOSMANTISBTPROJECT="CentOS-8" CENTOSMANTISBTPROJECTVERSION="8" REDHATSUPPORTPRODUCT="centos" REDHATSUPPORTPRODUCTVERSION="8" Kernel (e.g.


Question: } } <ins> // TestApplyNoExecuteTaintsToNodesEnqueueTwice ensures we taint every node with NoExecute even if enqueued twice func TestApplyNoExecuteTaintsToNodesEnqueueTwice(t *testing.T) { fakeNow := metav1.Date(2017, 1, 1, 12, 0, 0, 0, time.UTC) evictionTimeout := 10 * time.Minute fakeNodeHandler := &testutil.FakeNodeHandler{ Existing: []*v1.Node{ // Unreachable Taint with effect 'NoExecute' should be applied to this node. { ObjectMeta: metav1.ObjectMeta{ Name:              "node0", CreationTimestamp: metav1.Date(2012, 1, 1, 0, 0, 0, 0, time.UTC), Labels: map[string]string{ v1.LabelTopologyRegion:          "region1", v1.LabelTopologyZone:            "zone1", v1.LabelFailureDomainBetaRegion: "region1", v1.LabelFailureDomainBetaZone:   "zone1", }, }, Status: v1.NodeStatus{ Conditions: []v1.NodeCondition{ { Type:               v1.NodeReady, Status:             v1.ConditionUnknown, LastHeartbeatTime:  metav1.Date(2015, 1, 1, 12, 0, 0, 0, time.UTC), LastTransitionTime: metav1.Date(2015, 1, 1, 12, 0, 0, 0, time.UTC), }, }, }, }, // Because of the logic that prevents NC from evicting anything when all Nodes are NotReady // we need second healthy node in tests. { ObjectMeta: metav1.ObjectMeta{ Name:              "node1", CreationTimestamp: metav1.Date(2012, 1, 1, 0, 0, 0, 0, time.UTC), Labels: map[string]string{ v1.LabelTopologyRegion:          "region1", v1.LabelTopologyZone:            "zone1", v1.LabelFailureDomainBetaRegion: "region1", v1.LabelFailureDomainBetaZone:   "zone1", }, }, Status: v1.NodeStatus{ Conditions: []v1.NodeCondition{ { Type:               v1.NodeReady, Status:             v1.ConditionTrue, LastHeartbeatTime:  metav1.Date(2017, 1, 1, 12, 0, 0, 0, time.UTC), LastTransitionTime: metav1.Date(2017, 1, 1, 12, 0, 0, 0, time.UTC), }, }, }, }, // NotReady Taint with NoExecute effect should be applied to this node. { ObjectMeta: metav1.ObjectMeta{ Name:              "node2", CreationTimestamp: metav1.Date(2012, 1, 1, 0, 0, 0, 0, time.UTC), Labels: map[string]string{ v1.LabelTopologyRegion:          "region1", v1.LabelTopologyZone:            "zone1", v1.LabelFailureDomainBetaRegion: "region1", v1.LabelFailureDomainBetaZone:   "zone1", }, }, Status: v1.NodeStatus{ Conditions: []v1.NodeCondition{ { Type:               v1.NodeReady, Status:             v1.ConditionFalse, LastHeartbeatTime:  metav1.Date(2015, 1, 1, 12, 0, 0, 0, time.UTC), LastTransitionTime: metav1.Date(2015, 1, 1, 12, 0, 0, 0, time.UTC), }, }, }, }, }, Clientset: fake.NewSimpleClientset(&v1.PodList{Items: []v1.Pod{*testutil.NewPod("pod0", "node0")}}), } healthyNodeNewStatus := v1.NodeStatus{ Conditions: []v1.NodeCondition{ { Type:               v1.NodeReady, Status:             v1.ConditionTrue, LastHeartbeatTime:  metav1.Date(2017, 1, 1, 12, 10, 0, 0, time.UTC), LastTransitionTime: metav1.Date(2017, 1, 1, 12, 0, 0, 0, time.UTC), }, }, } nodeController, _ := newNodeLifecycleControllerFromClient( fakeNodeHandler, evictionTimeout, testRateLimiterQPS, testRateLimiterQPS, testLargeClusterThreshold, testUnhealthyThreshold, testNodeMonitorGracePeriod, testNodeStartupGracePeriod, testNodeMonitorPeriod, true) nodeController.now = func() metav1.Time { return fakeNow } nodeController.recorder = testutil.NewFakeRecorder() nodeController.getPodsAssignedToNode = fakeGetPodsAssignedToNode(fakeNodeHandler.Clientset) if err := nodeController.syncNodeStore(fakeNodeHandler); err != nil { t.Errorf("unexpected error: %v", err) } // 1. monitor node health twice, add untainted node once if err := nodeController.monitorNodeHealth(); err != nil { t.Errorf("unexpected error: %v", err) } if err := nodeController.monitorNodeHealth(); err != nil { t.Errorf("unexpected error: %v", err) } // 2. mark node0 healthy node0, err := fakeNodeHandler.Get(context.TODO(), "node0", metav1.GetOptions{}) if err != nil { t.Errorf("Can't get current node0...") return } node0.Status = healthyNodeNewStatus _, err = fakeNodeHandler.UpdateStatus(context.TODO(), node0, metav1.UpdateOptions{}) if err != nil { t.Errorf(err.Error()) return } // add other notReady nodes fakeNodeHandler.Existing = append(fakeNodeHandler.Existing, []*v1.Node{ // Unreachable Taint with effect 'NoExecute' should be applied to this node. { ObjectMeta: metav1.ObjectMeta{ Name:              "node3", CreationTimestamp: metav1.Date(2012, 1, 1, 0, 0, 0, 0, time.UTC), Labels: map[string]string{ v1.LabelTopologyRegion:          "region1", v1.LabelTopologyZone:            "zone1", v1.LabelFailureDomainBetaRegion: "region1", v1.LabelFailureDomainBetaZone:   "zone1", }, }, Status: v1.NodeStatus{ Conditions: []v1.NodeCondition{ { Type:               v1.NodeReady, Status:             v1.ConditionUnknown, LastHeartbeatTime:  metav1.Date(2015, 1, 1, 12, 0, 0, 0, time.UTC), LastTransitionTime: metav1.Date(2015, 1, 1, 12, 0, 0, 0, time.UTC), }, }, }, }, // Because of the logic that prevents NC from evicting anything when all Nodes are NotReady // we need second healthy node in tests. { ObjectMeta: metav1.ObjectMeta{ Name:              "node4", CreationTimestamp: metav1.Date(2012, 1, 1, 0, 0, 0, 0, time.UTC), Labels: map[string]string{ v1.LabelTopologyRegion:          "region1", v1.LabelTopologyZone:            "zone1", v1.LabelFailureDomainBetaRegion: "region1", v1.LabelFailureDomainBetaZone:   "zone1", }, }, Status: v1.NodeStatus{ Conditions: []v1.NodeCondition{ { Type:               v1.NodeReady, Status:             v1.ConditionTrue, LastHeartbeatTime:  metav1.Date(2017, 1, 1, 12, 0, 0, 0, time.UTC), LastTransitionTime: metav1.Date(2017, 1, 1, 12, 0, 0, 0, time.UTC), }, }, }, }, // NotReady Taint with NoExecute effect should be applied to this node. { ObjectMeta: metav1.ObjectMeta{ Name:              "node5", CreationTimestamp: metav1.Date(2012, 1, 1, 0, 0, 0, 0, time.UTC), Labels: map[string]string{ v1.LabelTopologyRegion:          "region1", v1.LabelTopologyZone:            "zone1", v1.LabelFailureDomainBetaRegion: "region1", v1.LabelFailureDomainBetaZone:   "zone1", }, }, Status: v1.NodeStatus{ Conditions: []v1.NodeCondition{ { Type:               v1.NodeReady, Status:             v1.ConditionFalse, LastHeartbeatTime:  metav1.Date(2015, 1, 1, 12, 0, 0, 0, time.UTC), LastTransitionTime: metav1.Date(2015, 1, 1, 12, 0, 0, 0, time.UTC), }, }, }, }, }...) if err := nodeController.syncNodeStore(fakeNodeHandler); err != nil { t.Errorf("unexpected error: %v", err) } // 3. start monitor node health again, add untainted node twice, construct UniqueQueue with duplicated node cache if err := nodeController.monitorNodeHealth(); err != nil { t.Errorf("unexpected error: %v", err) } // 4. do NoExecute taint pass // when processing with node0, condition.Status is NodeReady, and return true with default case // then remove the set value and queue value both, the taint job never stuck nodeController.doNoExecuteTaintingPass() // 5. get node3 and node5, see if it has ready got NoExecute taint node3, err := fakeNodeHandler.Get(context.TODO(), "node3", metav1.GetOptions{}) if err != nil { t.Errorf("Can't get current node3...") return } if !taintutils.TaintExists(node3.Spec.Taints, UnreachableTaintTemplate) || len(node3.Spec.Taints) == 0 { t.Errorf("Not found taint %v in %v, which should be present in %s", UnreachableTaintTemplate, node3.Spec.Taints, node3.Name) } node5, err := fakeNodeHandler.Get(context.TODO(), "node5", metav1.GetOptions{}) if err != nil { t.Errorf("Can't get current node5...") return } if !taintutils.TaintExists(node5.Spec.Taints, NotReadyTaintTemplate) || len(node5.Spec.Taints) == 0 { t.Errorf("Not found taint %v in %v, which should be present in %s", NotReadyTaintTemplate, node5.Spec.Taints, node5.Name) } } </ins> func TestSwapUnreachableNotReadyTaints(t *testing.T) { fakeNow := metav1.Date(2017, 1, 1, 12, 0, 0, 0, time.UTC) evictionTimeout := 10 * time.Minute

): [root ~]# uname -a Linux liabio 4.18.0-147.5.1.el81.x8664 SMP Wed Feb 5 02:00:39 UTC 2020 x8664 x8664 x8664 GNU/Linux Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig cluster-lifecycle /wg cluster-lifecycle
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig sig-cluster-lifecycle /wg sig-cluster-lifecycle /committee sig-cluster-lifecycle Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-sig cluster-lifecycle /sig node
Those labels are not set on the issue: <detailsIn response to : /remove-sig cluster-lifecycle /sig node </details
is NoExecute the expected taint for this condition?
- yes it is when running the taint-manager: evictions are usually safe eviction as we don't force delete pods
ref
I see the same, a node which has node condition set to Unknown, does not get an Unreachable taint with NoExecute effect. However I only see this on 1.19, works fine till 1.18.10 Till 1.18.10, such a node gets two Unreachable taints, one with effect NoExecute and one with effect NoSchedule. But on 1.19.3 it only gets the NoSchedule taint. I see that this is the only commit made in that area between 1.18 and 1.19: , could be related?
It totally breaks production readiness of all recent releases, yet doesn't seem to be regarded as a top priority issue. WTF?


Question: clientset-path cloud-config cloud-provider <ins> cloud-provider-gce-lb-src-cidrs </ins> cluster-cidr cluster-context cluster-dns

From GCE email today: Current behavior: You are not required to add firewall rules for Network LB health checks today Action required: We are introducing three new ranges for health checking: 209.85.152.0/22, 209.85.204.0/22, and 35.191.0.0/16. You need to ensure that your firewall allows traffic from 209.85.152.0/22, 209.85.204.0/22, and 35.191.0.0/16. The health check probes to your backend instances will come from one of these ranges. We have until Jan 31, 2018 - 1 year.
/assign this issue explains the confusion I had several weeks ago when speaking with and you about L4 health checks. I didn't realize a firewall rule isn't currently required.
/assign
Is this done now?
Not yet, there is a WIP on .
/close


Question: dc := &DisruptionController{ kubeClient:   kubeClient, queue:        workqueue.NewNamedRateLimitingQueue(workqueue.DefaultControllerRateLimiter(), "disruption"), <del> recheckQueue: workqueue.NewNamedDelayingQueue("disruption-recheck"), </del> <ins> recheckQueue: workqueue.NewNamedDelayingQueue("disruption_recheck"), </ins> broadcaster:  record.NewBroadcaster(), } dc.recorder = dc.broadcaster.NewRecorder(scheme.Scheme, v1.EventSource{Component: "controllermanager"})

What happened: From : What you expected to happen: Metrics should register successfully. How to reproduce it (as minimally and precisely as possible): Run kubernetes-integration test Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
There are no sig labels on this issue. Please add a sig label by either: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. See the . The in method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: By("Deleting provisioner daemonset") deleteProvisionerDaemonset(config) }) <ins> It("should discover dynamicly created local persistent volume mountpoint in discovery directory", func() { By("Starting a provisioner daemonset") createProvisionerDaemonset(config) By("Creating a volume in discovery directory") dynamicVolumePath := path.Join(config.discoveryDir, fmt.Sprintf("vol-%v", string(uuid.NewUUID()))) setupLocalVolumeProvisionerMountPoint(config, dynamicVolumePath, config.node0) By("Waiting for the PersistentVolume to be created") _, err := waitForLocalPersistentVolume(config.client, dynamicVolumePath) Expect(err).NotTo(HaveOccurred()) By("Deleting provisioner daemonset") deleteProvisionerDaemonset(config) By("Deleting volume in discovery directory") cleanupLocalVolumeProvisionerMountPoint(config, dynamicVolumePath, config.node0) }) </ins> }) Context("StatefulSet with pod anti-affinity", func() {

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: Now that the local provisioner uses mount propagation, the provisioner e2e test should also test adding mount points after the provisioner has started
would you have time to look into this?
/assign
FYI:


Question: endLine(tabWriter) for _, plState := range cfgCtlr.priorityLevelStates { if plState.queues == nil { <del> tabPrint(tabWriter, row( plState.pl.Name, // 1 "<none>",        // 2 "<none>",        // 3 "<none>",        // 4 "<none>",        // 5 "<none>",        // 6 )) if includeRequestDetails { continueLine(tabWriter) tabPrint(tabWriter, row( "<none>", // 7 "<none>", // 8 "<none>", // 9 "<none>", // 10 "<none>", // 11 "<none>", // 12 "<none>", // 13 "<none>", // 14 )) } endLine(tabWriter) </del> continue } queueSetDigest := plState.queues.Dump(includeRequestDetails)

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Testing API Priority and Fairness, I issued this command in a rather idle cluster: and I got this result: That is, a list of one request with no information of any sort. That comes from , which invents one phantom request for each priority level that is exempt. We do not want or need a phantom request; is for listing requests in queues. What you expected to happen: I expected to get only queued requests. How to reproduce it (as minimally and precisely as possible): As shown above, with API Priority and Fairness enabled. Anything else we need to know?: Environment: Kubernetes version (use ): this is in Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/cc /cc /cc


Question: } return feature.DefaultFeatureGate.Enabled(features.JobPodFailurePolicy) && job.Spec.PodFailurePolicy != nil } <ins> func (jm *Controller) backupRemovePodFinalizers(job *batch.Job) { // Listing pods shouldn't really fail, as we are just querying the informer cache. selector, err := metav1.LabelSelectorAsSelector(job.Spec.Selector) if err != nil { utilruntime.HandleError(fmt.Errorf("parsing deleted job selector: %v", err)) return } pods, _ := jm.podStore.Pods(job.Namespace).List(selector) for _, pod := range pods { if metav1.IsControlledBy(pod, job) && hasJobTrackingFinalizer(pod) { jm.enqueueOrphanPod(pod) } } } </ins>

When a Job is deleted (via Background cascade), we have a backup logic to remove finalizers from pods This logic could also be applied for any finished Job. While investigating a case similar to , we observed that the job controller didn't delete finalizers from Pods where the Job was deleted with Foreground cascading policy. Since the Job was already marked as Finished, it only makes sense to trigger the backup logic to remove finalizers, so that we can guard against any possible bug where a Job is marked as Finished but not all pod finalizers are removed. Note that I'm not aware of any other bug in tracking with finalizers.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig apps
/remove-kind feature /kind cleanup
/wg batch
I am happy to give this a go.
/assign
Just to check if I have the logic chain right: We are talking about adding this logic If so, I should probably turn the backup-logic code into a function that then apply that in both places? I also would be unsure of where in the unit tests this would be tested, are you able to point me to how this is tested in the deletePod function? I assume we add a pod that is never included in the "uncounted" lists.
I would put it in the event handler, to be more of a "catch all" Yes, that's a good idea. I think you would have to force it quite a bit, as we have fixed all the bugs that we have found. For example, from the test: the controller, but do not start the workers (this will setup the event handlers) a job the pods the job the workers some inspiration here
Alright, thank you :))
Should the issue title be: Remove finalizers from Pods belonging to finished Jobs ?
If someone has set a custom finalizer on a Pod, even one owned by a Job, (when) would we remove that finalizer?
/retitle Remove tracking finalizers from Pods belonging to finished Jobs No, only the finalizer specific to job tracking


Question: // Returns list of device Ids we need to allocate with Allocate rpc call. // Returns empty list in case we don't need to issue the Allocate rpc call. <del> func (m *ManagerImpl) devicesToAllocate(podUID, contName, resource string, required int) (sets.String, error) { </del> <ins> func (m *ManagerImpl) devicesToAllocate(podUID, contName, resource string, required int, reusableDevices sets.String) (sets.String, error) { </ins> m.mutex.Lock() defer m.mutex.Unlock() needed := required

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug /area hw-accelerators What happened: In the context of the device plugin, when using init containers in your pod spec, the current behavior will not reuse devices between init containers and regular containers. This is problematic as Pod scheduling is currently wrong: , the current predicate algorithm rightly assumes the maximum number of devices in use at one time is . However, the pod predicate ran on the node is the sum of devices used by regular and init containers. Meaning that your pod can bounce back between the scheduler and a node infinitely. Also given that many of the extended resources are typically more expensive that the built-in ones like CPU, Memory and Storage this bug has very real financial drawbacks in terms of resource usage (e.g: using 4 GPUs on a pod that only uses 2 GPUs at a time). Furthermore, this unexpected behavior was caught during implementation (but we did not catch the scheduling implications) and decision was taken to ignore it for the first implementation. We need to document it for 1.9 What you expected to happen: Devices can be reused by Init Containers Correct resource accounting In terms of the current behavior of the Allocate call to the device plugin, I'd expect the allocate call to be done before every Init containers start and once at the end of the last Init container (for regular containers). /assign
Yep, I realized the inconsistency between the predicate behavior of scheduler and that of node too. Having said that, considering we treat a pod as a whole when predicating currently, it's kind of difficult to reuse the runtime informations between init containers and regular containers. The real crux of the matter might be whether the scenario of requiring extended resources in init containers is common, if so, the issue should be addressed ASAP.
If I remembered correctly, the current design to improve DP includes a reconciliation in admit phase of Pod, it may be able to fix Allocate call for
can you point me to said design ? I'm not sure that the Allocate call for init containers is planned to be fixed as described.
Can you guys please point to the code where is happening. Looks to me that same logic is executed on both the scheduler and the node to calculate a pod's resource request and which is  On the node, function call tree looks like this: Is it possible to add few more words in explanation?


Question: // Returns list of device Ids we need to allocate with Allocate rpc call. // Returns empty list in case we don't need to issue the Allocate rpc call. <del> func (m *ManagerImpl) devicesToAllocate(podUID, contName, resource string, required int) (sets.String, error) { </del> <ins> func (m *ManagerImpl) devicesToAllocate(podUID, contName, resource string, required int, reusableDevices sets.String) (sets.String, error) { </ins> m.mutex.Lock() defer m.mutex.Unlock() needed := required

PR that the mentioned change: Code in question is here:
Above looks incorrect description. Predicate that run on the scheduler and on the node is same i.e max(init, regular). Pod admission will pass at the node. Problem might happen at allocation where allocated resources is not the max but the sum of (init, regular). And in that case, pod would hang there in fail state rather 'bounce back infinitely'.
Problem might happen at allocation where allocated resources is not the max but the sum of (init, regular). And in that case, pod would hang there in fail state rather 'bounce back infinitely'. Absolutely correct, sorry to make you confused, I think Renaud name it mainly bacause the allocate func is supposed to be called during the predicate phase, that is, passed in as a part of PredicateAdmitHandler.
The original idea was onced listed in as a sub-task: Now it's removed (I don't know why ...), while I believe the idea was: device resources allocation during pod admit phase. This comment is also related if needed during reconcile in 1 and cache these devices info cached devices later during container create phase
Handle device resources allocation during pod admit phase. This comment (comment) is related Call Allocate() if needed during reconcile in 1 and cache these devices info Use cached devices later during container create phase Current code behaviors exactly as you described, but to address the issue, the key point might be to decide which devices of init containers could be reused by which regular containers, and generate cache accordingly in step2; when creating containers, we simply fetch runtime information from cache.
Thanks a lot for filing this issue, I think for 1.9, we probably want to list this as a known issue and may advise people to not request gpu resources in init containers. We can target to fix this problem in 1.10.
Agreed We don't have a clear solution for this tough. It's pretty clear to me that device resources allocation during pod admit phase is not the right model when init containers comes into the picture. GPUs are pretty simple example. On of the things we plan to do is scrub the GPU memory during the Allocate phase. If GPUs are re-used between initContainers and regular containers the current model doesn't fit.


Question: // Returns list of device Ids we need to allocate with Allocate rpc call. // Returns empty list in case we don't need to issue the Allocate rpc call. <del> func (m *ManagerImpl) devicesToAllocate(podUID, contName, resource string, required int) (sets.String, error) { </del> <ins> func (m *ManagerImpl) devicesToAllocate(podUID, contName, resource string, required int, reusableDevices sets.String) (sets.String, error) { </ins> m.mutex.Lock() defer m.mutex.Unlock() needed := required

do you think if you have time to implement the solution you mentioned in the above comment We can try to get it into a 1.9 sub release. I think for now, lets just support device reuse among initContainers and normal containers. We can revisit this part when we need to guarantee an Allocate call per container.
Right what I am mentioning here is that it is not possible to do that. The point of the device plugin is so that we can execute vendor specific code before making the device available in the container (e.g: Scrub the GPU memory). I don't think the proposed solution addresses this problem.
Sure. But to clarify, I think if we dont't expect big shock on current implementation, we could only provide limited support on the point, that is, without re-generate runtime information for regular containers, only those whose requested resource is equal to or is a subset of that in init containers could make a resuse(e.g. if we have two init containers, each requests 2 devices, then a regular container that request 3 devices still could not reuse the devices.) Besides, if a initcontainer requests 2 devices, and we have two regular containers, each requests 1 device, then due to the lack of runtime info re-generation, the devices could only be reused by one of the containers, not both of them. In sum, since we don't re-generate runtime information, we could only reuse devices per container, not per device, like we do in scheduler. Is that what you mean?
Hmm, that is a good question. I forgot that we now return the combined runtime information in AllocResponse. Then for now, perhaps we should just return an error in device plugin manager Allocate() in case both init containers and containers request the same device plugin resource?
You mean we give up reuse and prohibit users to request same kind of device in init containers and regular containers? that seems a bit weird. And if we decide to do so, we might need to do the check at the first beginning, i.e. in API validation, not wait until it goes into DevicePlugin.
I think for 1.9, we just return an error when both init containers and normal containers of a pod request the same device resource, and document this in the device plugin user guide.


Question: // Returns list of device Ids we need to allocate with Allocate rpc call. // Returns empty list in case we don't need to issue the Allocate rpc call. <del> func (m *ManagerImpl) devicesToAllocate(podUID, contName, resource string, required int) (sets.String, error) { </del> <ins> func (m *ManagerImpl) devicesToAllocate(podUID, contName, resource string, required int, reusableDevices sets.String) (sets.String, error) { </ins> m.mutex.Lock() defer m.mutex.Unlock() needed := required

The reason is because in the current device plugin implementation, we don't have a good way to free devices used by init containers of an active pod. It is hard to do this kind of validation at API level or even scheduler because we can't know whether an extended resource is device resource or not. So for now, I think we should probably just return a clear error message during admission handler. I am not sure yet whether there will be strong use cases that need to share devices between init containers and normal containers. We can revisit this based on user feedback. If there is strong user demand to support this, something we could do is to differentiate init containers and normal containers in podDevices, and free up devices used by init containers when any normal container is about to run. That would request some code structure change, and we may wait to see whether there is strong demand to support this.
It is hard to do this at API level or even scheduler because we can't know whether an extended resource is device resource or not. Oh, you are right, I ignored the resource awareness issue. I also agree that it's hard to address the issue without big change to current implementation. But I still feel the inconsistency of behavior between API and kubelet might confusing users. However, its effect should be sustainable, since we haven't received feedback about init containers up to now.
Right. So for now, perhaps just return an error when this happens, to give clear error signals.
Oh, I'm afraid we miss something, the check may not work as expected. As I commented, because init-containers run sequentially, we did reuse between init containers in , that is, if we have two init containers, each request 1 device, then we expect 1 device in total in scheduler, but expect 2 in deviceplugin. The check could not cover the scenario.
Thinking about this problem a bit more, I wonder whether we may tackle this issue from the following approaches, ordered by their complexities: 1) Just return an error whenever an init container requests a device plugin resource, and document in the device plugin user guide that requiring device plugin resource in init containers are not supported. This may be the short-term change we can try to get into 1.9. 2) Maintain a list of devices allocated to init containers in ManagerImpl.Allocate().


Question: // Returns list of device Ids we need to allocate with Allocate rpc call. // Returns empty list in case we don't need to issue the Allocate rpc call. <del> func (m *ManagerImpl) devicesToAllocate(podUID, contName, resource string, required int) (sets.String, error) { </del> <ins> func (m *ManagerImpl) devicesToAllocate(podUID, contName, resource string, required int, reusableDevices sets.String) (sets.String, error) { </ins> m.mutex.Lock() defer m.mutex.Unlock() needed := required

Try to allocate from this list first for later init containers and normal containers. The required no. of devices we pass to ManagerImpl.devicesToAllocate() should subtract the no. of devices we have allocated to init containers. This may require some code structure change around ManagerImpl.Allocate(), ManagerImpl.allocateContainerResources(), and ManagerImpl.devicesToAllocate(). However, Allocate gRPC call is still issued during Admission all together, and it is possible that we issue an Allocate gRPC for [dev1] (say dev1 allocated to initContianer1), followed by another Allocate gRPC call for [dev1, dev2] (say dev1 and dev2 allocated to initContainer2), with none of containers started or finished. This assumes containers running in the same pod share the same security context, and device plugin doesn't need to perform per-container allocation operations. We just use Allocate gRPC call to learn container runtime settings. 3) Take a further step from 2), during Admission, we just determine what devices to allocate to containers without issuing the actual Allocate gRPC calls. I.e., we only fill deviceIds field in podDevices.deviceAllocateInfo but leaves allocResp field empty. During ManagerImpl.GetDeviceRunContainerOptions(), we check for this case and issue the actual Allocate gRPC call for the container. This ensures we can issue a gRPC Allocate call per container. However, if a pod is restarted multiple times, we don't guarantee Allocate gRPC call will be issued during subsequent pod runs. 4) Take a further step from 3), we can check for overlapping device allocations in ManagerImpl.GetDeviceRunContainerOptions() and invalidate their allocResp so that an Allocate gRPC call will be issued next time for containers using re-allocated devices. E.g., for containerA, if we find it is allocated with [dev1, dev2] and dev1 is also allocated to containerB, we set containerB's allocResp to nil so that next time before containerB is running, we will issue an Allocate gRPC call for dev1. Note that this approach does have the drawback that device plugin failure would cause previously running pods not be able to restart, so we need to carefully evaluate our use cases to see whether we need to implement this. My suggestion is to implement 1) now and gather some user feedbacks to determine whether we need to implement the following approaches.


Question: // Returns list of device Ids we need to allocate with Allocate rpc call. // Returns empty list in case we don't need to issue the Allocate rpc call. <del> func (m *ManagerImpl) devicesToAllocate(podUID, contName, resource string, required int) (sets.String, error) { </del> <ins> func (m *ManagerImpl) devicesToAllocate(podUID, contName, resource string, required int, reusableDevices sets.String) (sets.String, error) { </ins> m.mutex.Lock() defer m.mutex.Unlock() needed := required

I see this a very big limitation. Special devices/resources in init containers is a very common use case IMO. Therefore we should try to fix this. This approach perfectly makes sense to me. What is your concern in implementing this? Even today, IIUC, Allocate call is made per container. Is there any other value that this approach is adding over 1) ? I am not able to see any gains with 2) or 3) over 1). Comments please.
I feel we may not be able to implement 2) for 1.9 timeframe. Option 2) also doesn't address the case mentioned above: Option 3) covers that case for pods without restarts. Option 4) fully covers the requirement we we will scrub old container data before giving it to a new container, even if they both run in the same pod, bug it does have the mentioned drawback during device plugin failure.
Take a further step from 3), we can check for overlapping device allocations in ManagerImpl.GetDeviceRunContainerOptions() and invalidate their allocResp so that an Allocate gRPC call will be issued next time for containers using re-allocated devices. E.g., for containerA, if we find it is allocated with [dev1, dev2] and dev1 is also allocated to containerB, we set containerB's allocResp to nil so that next time before containerB is running, we will issue an Allocate gRPC call for dev1. Note that this approach does have the drawback that device plugin failure would cause previously running pods not be able to restart, so we need to carefully evaluate our use cases to see whether we need to implement this. How about making the allocation call optional? That is , if we find a contaienr using re-allocated devices, we'd try to issue an Allocate gRPC call first, and still use the cached setting if allocation failed. And, if we decide not to address the issue in v1.9, which means we have enough time to work on it later, I think approach 2) might not be a good choice. Because it has a premise that devices are stateless, recycle actions before reuse(e.g: Scrub the GPU memory mentioned above) could not be supported. And this way, as I commented above, we could only reuse devices per container, not per device, like we do in scheduler.


Question: // Returns list of device Ids we need to allocate with Allocate rpc call. // Returns empty list in case we don't need to issue the Allocate rpc call. <del> func (m *ManagerImpl) devicesToAllocate(podUID, contName, resource string, required int) (sets.String, error) { </del> <ins> func (m *ManagerImpl) devicesToAllocate(podUID, contName, resource string, required int, reusableDevices sets.String) (sets.String, error) { </ins> m.mutex.Lock() defer m.mutex.Unlock() needed := required

I noticed that you have already add the limitation into the , so I think it's already the decision that we add the limitation as a short-time solution in v1.9 So I posted a PR to add the check in code,, PTAL :)
Use Case: Let say there is a which node has two Devices of a resource type X and the device IDs are X1 and X2. User pod has one init container, lets call it cont-init and one regular container, lets call it cont-reg. Both cont-init and cont-reg requests 2 unit of resource X. cont-init will run some vendor specific code on allocated Devices and then cont-reg will use these 'initiated' devices. I think this is the use case that mentioned. And i feel 2) can handle this. 2) approach will allocate both the devices, X1 and X2, to both the containers. When the containers start, init container, cont-init, will be run first and will run to completion and here devices will get initialized. Strictly after this cont-reg will be started and now this regular container has initialized devices. Where i am wrong?
I guess in RenaudWasTaken 's case, the initialization task is considered to be managed in the plugin, not in init-container, and in init-container we only use the device, so then we might need to call plugin to do recycle action before the device reused by reguler containers. In practical use, both of the two scenarios may exist, it depends on the request of user. Besides, if we don't issue an Allocate gRPC call, we could not ensure a ”complete reuse“: say an init-container requests 2 devices, and we have two regular containers, each requests 1 device, then due to the lack of runtime info re-generation, the devices could only be reused by one of the containers, not both of them.
It is the init containers' job to initialize things and if in some special case device plugin is doing the initialization part, in that case, init containers will not be needed and serve no purpose in pod spec. Specific to example of memory scrubbing, i think this also makes more sense to be done in init container. Once Device's memory is scrubbed, it can be used by regular container. Let try to run your use case with approach 2).


Question: // Returns list of device Ids we need to allocate with Allocate rpc call. // Returns empty list in case we don't need to issue the Allocate rpc call. <del> func (m *ManagerImpl) devicesToAllocate(podUID, contName, resource string, required int) (sets.String, error) { </del> <ins> func (m *ManagerImpl) devicesToAllocate(podUID, contName, resource string, required int, reusableDevices sets.String) (sets.String, error) { </ins> m.mutex.Lock() defer m.mutex.Unlock() needed := required

There are three containers in pod, cont-init(requested 2 unit or resource X), cont-reg1(requested 1 unit of resource X) and cont-reg2(requested 1 unit of resource X). Now in approach 2), gRPC Allocate call is made for cont-init requesting X1 and X2. Next, gRPC Allocate call is made for cont-reg1. This time resource id will be send in Allocate from the previously allocated [X1, X2]. Lets say X1 is requested for cont-reg1 Next, gRPC Allocate for cont-reg2 and resource with ID X2. All this allocation info gets stored in internal cache. Now when container runs, runtimeOptions will be collected from internal cache. Where do you see a problem in this flow?
/cc
Yep, it could work if the plugin behavior the way you described, but I'm not sure if all kinds of devices could support "pre-allocation", that is, doing allocation on a same device multiple times at the same time, to prepare runtime informations for several containers that reuse the device later. Also I have no idea if all plugin developers consider their plugins "stateless", that is, managing initialization in init-containers. As I don't have enough knowledge on hardware, would like to hear opinions from other guys :)
This would defeat the purpose of a container. In the case of GPUs, when you run a container you expect the GPU to always be in the same state. Furthermore GPUs can't be shared (exclusive usage) between containers meaning that Init containers should behave in the same way as regular containers. With the only difference being that they are executed sequentially and before regular containers. Additionally, operations to setup a device before making them available in a container, whether functionally (load the driver, insert libraries, ....) or security wise (scrub the memory), should be handled by the infrastructure not the users pod spec.
I agree that it's a sane behavior to forbid the use of Device Plugin Resources in Init Containers for the 1.9 release
moving forward,
/cc I think what long-term solution we decide to implement will largely depend on use-case requirements from different device plugins. Would like to hear some thoughts from different device plugin developers on this. Can people cc-ed in this issue take a look at the proposed options mentioned in and provide some feedbacks on which one may better fit your use cases?


Question: // Returns list of device Ids we need to allocate with Allocate rpc call. // Returns empty list in case we don't need to issue the Allocate rpc call. <del> func (m *ManagerImpl) devicesToAllocate(podUID, contName, resource string, required int) (sets.String, error) { </del> <ins> func (m *ManagerImpl) devicesToAllocate(podUID, contName, resource string, required int, reusableDevices sets.String) (sets.String, error) { </ins> m.mutex.Lock() defer m.mutex.Unlock() needed := required

Answering on behalf of and Solution 1 makes sense for the moment Solution 2 defeats the purpose of the device plugin, which is setting up a device before making it available to a container. solution 4 seems to be a sane solution. As mentioned in my previous comment, as long as devices are exclusive, I expect the devices exposed in my container to always be in the same state at the start of my container. When / If we implement a mechanism for sharing a GPU / devices across containers then the scenario described by is something that we can have. It doesn't however fit the current situation.
Solution number 1 seems totally reasonable short-term. Long term, simplifying and documenting the relationship between plugin callbacks and container lifecycle would help plugin authors.
For reference, I implemented option 2 solution in PR The change itself is small. This is an alternative short-term fix we may consider to get into a 1.9 patch release. I think the main benefit of taking this fix compared to Option 1 solution is that we can change the underlying implementation to support init containers later with less user-facing impact.
No mater we take PR or PR for 1.9, I think we need to have this issue approved for 1.9 milestone. or or can any of you approve this issue for 1.9? Thanks!


Question: mountOptions  []string imageFormat   string imageFeatures []string <ins> accessModes   []v1.PersistentVolumeAccessMode </ins> } var _ volume.Mounter = &rbdMounter{}

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: When use ceph as storageclass, i set accessModes: ReadOnlyMany, and create pods using the claim, only one pod would be running and the others on different nodes would get stuck on ContainerCreating. But if the pods are all on same node, it works fine. What you expected to happen: Pods on different nodes can mount ceph rbd volume. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): v1.9.2 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): 3.10.0-327.18.2.el7.x86_64 Kernel (e.g. ): CentOS Linux release 7.2.1511 (Core) Install tools: kubeadm Others:
/sig storage
/assign
, how would this work? For example, how do you put the initial contents of the RBD volume on it if it is in the ReadOnlyMany access mode? How would you use this feature? As a cheap way of sharing files across a Kubernetes cluster? I'm just interested. -ben england
Do we really want to support ? I have the same questions as mentioned above. If we set the PVC AccessMode to , then the field in will need to be set true ? If so, for the first time, in , we will mount the device without formatting, is that intended ? We need to format the device manually before we use it ? /cc FYI: also there is an issue related to this:
We have this same problem for GCE PD, which supports ReadOnlyMany. You need to mount the volume first as RWO, in order to format the filesystem and populate it with data (mounting a read only volume with no data on it seems pointless). Then you have to fully unmount/detach it, and then you can deploy pods using it as ROX. Basically, you can dynamically provision it in RWO to format + populate data. But afterwards, you need to manually create a different PV/PVC for ROX
continuation


Question: } // WaitForPodCondition waits a pods to be matched to the given condition. <ins> // If the condition callback returns an error that matches FinalErr (checked with IsFinal), // then polling aborts early. </ins> func WaitForPodCondition(c clientset.Interface, ns, podName, conditionDesc string, timeout time.Duration, condition podCondition) error { e2elog.Logf("Waiting up to %v for pod %q in namespace %q to be %q", timeout, podName, ns, conditionDesc) var (

treats any error returned by the condition as terminal, but silently ignores any errors that aren't returned along with . We should change WaitForPodCondition to behave like . /sig testing /area e2e-test-framework
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
Was there any reason why you instead of doing this right away? I fully agree with this issue, we should abort early. I just found the TODO because I have a failing test that ran longer than expected after a final pod failure and that made me check out why it didn't abort quicker :grin:


Question: if err != nil { t.Fatalf("failed to create an http request: %v", err) } <ins> auditCtx := &audit.AuditContext{Event: &auditapi.Event{Level: auditapi.LevelMetadata}} req = req.WithContext(audit.WithAuditContext(req.Context(), auditCtx)) </ins> _, err = client.Transport.RoundTrip(req) if sanChecker.CheckRoundTripError(err) { sanChecker.IncreaseMetricsCounter(req) <ins> if len(auditCtx.Event.Annotations["missing-san.invalid-cert.kubernetes.io/"+req.URL.Hostname()]) == 0 { t.Errorf("expected audit annotations, got %#v", auditCtx.Event.Annotations) } </ins> } errorCounterVal := getSingleCounterValueFromRegistry(t, registry, "Test_checkForHostnameError")

go1.18 dropped support for certificates signed with the SHA-1 hash function - While this does improve security, and publicly trusted Certificate Authorities no longer issue such certificates, it is possible that user-provided certificates issued by private Certificate Authorities and used in webhooks or aggregated API servers do still use this. Similar to removal of support for certificates without subjectAltNames in go1.17, we need to: [x] add metrics when webhook/conversion/aggregated certificates with this hash are encountered - [x] document use of and the metric from 1 as a mitigation for clusters which still need to identify and replace use of such certificates - to release note and known issues [x] add audit annotation when SHA-1 cert is encountered - [x] update docs at and - /sig api-machinery instrumentation docs /priority important-soon /milestone v1.24 /kind bug documentation Upgraded 1.23 clusters to continue working as-is Configure an admission webhook, conversion webhook, or aggregated server with a serving certificate signed with the SHA-1 hash function. Upgrade to HEAD of 1.24. The certificate will no longer work by default No response 24 HEAD (go1.18-based) n/a n/a No response No response No response
it looks like NewMissingSANRoundTripperWrapperConstructor already has all the plumbing, and instead of missing SANs we could check the SignatureAlgorithm and output a distinct metric
had a not-yet-tested WIP branch at with a sketch of what it might look like
/assign who offered to pick this up
GitHub didn't allow me to assign the following users: stlaz. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign who offered to pick this up Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Indeed, please, you should trust Jordan :slightlysmilingface:
/assign
/triage accepted
These restrictions are on the GitHub side, this message is posted when it receives a specific error from the GitHub API while attempting to assign It is an annoying GitHub limitation, and one that seemingly does not exist in the web client vs the API.


Question: if err != nil { t.Fatalf("failed to create an http request: %v", err) } <ins> auditCtx := &audit.AuditContext{Event: &auditapi.Event{Level: auditapi.LevelMetadata}} req = req.WithContext(audit.WithAuditContext(req.Context(), auditCtx)) </ins> _, err = client.Transport.RoundTrip(req) if sanChecker.CheckRoundTripError(err) { sanChecker.IncreaseMetricsCounter(req) <ins> if len(auditCtx.Event.Annotations["missing-san.invalid-cert.kubernetes.io/"+req.URL.Hostname()]) == 0 { t.Errorf("expected audit annotations, got %#v", auditCtx.Event.Annotations) } </ins> } errorCounterVal := getSingleCounterValueFromRegistry(t, registry, "Test_checkForHostnameError")

adds the metrics. the release note on that issue describes the go change, links to go docs, describes use of the option, the new metrics, and the kube-apiserver logs what would you expect kubeadm to do with this? GODEBUG off by default and let users override GODEBUG on their own (is this possible)? GODEBUG by default on all clusters? GODEBUG by default only on upgrade? The risk of 1 is that a user upgrades without reading release notes and breaks a cluster that has SHA-1 certs in webhooks, etc The risk of 2 is that we propagate accepting insecure certificates longer, even in new clusters that we know don't have any pre-existing bad certs The risk of 3 is that it makes an upgraded 1.23 → 1.24 cluster and a new 1.24 cluster inconsistent
the majority of kubeadm users use kubeadm managed CA and certificates. there are less users that use external CA to sign their own certificates. sha1 attacks surfaced more than a before kubeadm's creation and Git hash collusion made the problem more known around 2017 - that was at a time when kubeadm did not support external CA. given these facts, an estimate on my side is that there are currently no kubeadm users that use sha1 certificates, but i would not exclude the possibility entirely. if such users exists, they are on they own and my vote goes for option 1, until they migrate their PKI to something secure. if the same users log tickets for us, our response would be "please use GODEBUG and migrate ASAP". all users should be able to see the notes in IMO, the PR needs an "action required" to appear on top of the release notes.
that's helpful, thanks agreed. weird, I set , but the label didn't get . the label manually.
and there's a way for them to set that envvar in kube-apiserver manifests when using kubeadm?
There's been a lot of intermittent github webhook delivery issues in the past few weeks or so, slash commands are going to be somewhat unreliable :/
they could use kubeadm patches to override the YAML manifest defaults - i.e. the contents of v1.Container -Env:


Question: expectedList: []HostPriority{{"machine1", 0}, {"machine2", 0}}, test:         "zero minion resources", pods: []api.Pod{ <del> {DesiredState: cpuOnly, CurrentState: machine1State}, {DesiredState: cpuAndMemory, CurrentState: machine2State}, </del> <ins> {DesiredState: cpuOnly}, {DesiredState: cpuAndMemory}, </ins> }, }, }

When creating replication controller (e.g. frontend php-redis from guestbook example), scheduler gets stuck and never recovers. This is a race, I have to run several times to see the error (A script to delete controller and pods, then create new controller, leave enough time in between for k8s to react). I've created PR for an attempted fix.
I saw this too while following the guest book and creating the php/frontend step. In my case, I had 2 minions while the replication controller configuration had configured 3 replicas, so 2 pods got scheduled -each on another minion, the third pod had empty Host (saw that when doing 'list pods'), and the 3 of them were stuck in "waiting" status. I deleted the controller and the pods, fixed the configuration in json file to be 2 replicas and recreated the controller and everything worked. Does your fix take care of this use case? (too many replicas, too few minions/one of the pods is not scheduled)
This is different from your use case. If you have more replicas than minions, then the scheduler will get stuck for sure (with the error msg probably being 'failed to find a fit for pod'). AFAICT, it's not support yet.
If you remove the HostPorts, it should work for you. thanks for report.
, thanks for your replies. I understand why it isn't working, but I'd expect one of the 2 options to happen instead of the current situation: the creation of replication controller if there are less minions than replicas, AND there is a hostPort property. In other words - validation of the content during creation. least schedule all replicas that can get a minion (in my case 2 out of 3) and leave only one at "waiting". Is there an issue open about the use case I describe?
should be true today. is less possible than it seems like-- minions can come and go, and become over or under loaded in the time between creation and when the pod shows up at the scheduler. So even if we check up front, we still have to check again later, and sometimes the first check will pass but not the second.


Question: for _, volumePlugin := range pluginWithLimits { attachLimits, err := volumePlugin.GetVolumeLimits() if err != nil { <del> klog.V(4).InfoS("Error getting volume limit for plugin", "plugin", volumePlugin.GetPluginName()) </del> <ins> klog.V(4).InfoS("Skipping volume limits for volume plugin", "plugin", volumePlugin.GetPluginName()) </ins> continue } for limitKey, value := range attachLimits {

kubelet logs misleading when volume limits not an
/sig storage /tirage accepted /assign
Note that it is set as info level in the log above: However, this is definitely confusing. /remove-kind bug /kind cleanup /triage accepted /sig node
I also have this problem Excuse me, how did you solve it
Which version do you use .. This was fixed here Check k8s version you use please


Question: By("Initializing local volume discovery base path") mkdirCmd := fmt.Sprintf("mkdir %v -m 777", path.Join(hostBase, discoveryDir)) <del> _, err := framework.NodeExec(config.node0.Name, mkdirCmd) </del> <ins> err := framework.IssueSSHCommand(mkdirCmd, framework.TestContext.Provider, config.node0) </ins> Expect(err).NotTo(HaveOccurred()) }

8 tests failed, 2 jobs failed, 88 builds failed. Failure stats cover 1 day time range '3 Aug 2017 03:31 UTC' to '4 Aug 2017 03:31 UTC'.
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig storage
This should have been fixed by /cc
The test panic is fixed, but now it's failing because it can't SSH into the node.
/assign


Question: # use JENKINS_PUBLISHED_VERSION, default to 'ci/latest', since that's # usually what we're testing. check_dirty_workspace <del> fetch_published_version_tars "${JENKINS_PUBLISHED_VERSION:-'ci/latest'}" </del> <ins> fetch_published_version_tars "${JENKINS_PUBLISHED_VERSION:-ci/latest}" </ins> fi # Copy GCE keys so we don't keep cycling them.

Getting this error... looks like we are quoting : Working on a hotfix...
Linking to
Fix is


Question: "//pkg/kubectl/cmd/templates:go_default_library", "//pkg/kubectl/cmd/util:go_default_library", "//pkg/kubectl/resource:go_default_library", <ins> "//pkg/util/i18n:go_default_library", </ins> "//vendor/github.com/golang/glog:go_default_library", "//vendor/github.com/spf13/cobra:go_default_library", "//vendor/github.com/spf13/pflag:go_default_library",

Referring and ; both have string in place of in section Example. This needs to be corrected in kubefed docs autogeneration. cc also see
/assign


Question: registry = "k8s.gcr.io/build-image", repository = "debian-iptables", # Ensure the digests above are updated to match a new tag <del> tag = "v12.1.2",  # ignored, but kept here for documentation </del> <ins> tag = "buster-v1.3.0",  # ignored, but kept here for documentation </ins> ) def etcd_tarballs():

What would you like to be : Have debian-iptables with iptables 1.8.5+ Why is this needed: To fix (nodelocaldns + iptables-nft) More informations Found a bug in iptables-nft, fixed in iptables 1.8.5, and now available in buster-backports https://alioth-
/sig api-machinery
ping as you did the last 2 updates
Yep, this is on my list. Will take of it next week. /sig release network /area release-eng /milestone v1.20 /assign
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
Opened to bump the debian-iptables images in kubernetes/kubernetes. /remove-triage unresolved
Here are the digests for the new image ():
Please reopen if you run into any issues with the new images. /close
Closing this issue. <detailsIn response to : Please reopen if you run into any issues with the new images. /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: volumeToAttach.VolumeSpec, volumeToAttach.NodeName) if attachErr != nil { <ins> uncertainNode := volumeToAttach.NodeName </ins> if derr, ok := attachErr.(*volerr.DanglingAttachError); ok { <del> addErr := actualStateOfWorld.MarkVolumeAsAttached( v1.UniqueVolumeName(""), originalSpec, derr.CurrentNode, derr.DevicePath) if addErr != nil { klog.Errorf("AttachVolume.MarkVolumeAsAttached failed to fix dangling volume error for volume %q with %s", volumeToAttach.VolumeName, addErr) } } else { addErr := actualStateOfWorld.MarkVolumeAsUncertain( v1.UniqueVolumeName(""), originalSpec, volumeToAttach.NodeName) if addErr != nil { klog.Errorf("AttachVolume.MarkVolumeAsUncertain fail to add the volume %q to actual state with %s", volumeToAttach.VolumeName, addErr) } </del> <ins> uncertainNode = derr.CurrentNode } addErr := actualStateOfWorld.MarkVolumeAsUncertain( v1.UniqueVolumeName(""), originalSpec, uncertainNode) if addErr != nil { klog.Errorf("AttachVolume.MarkVolumeAsUncertain fail to add the volume %q to actual state with %s", volumeToAttach.VolumeName, addErr) </ins> } // On failure, return error. Caller will log and retry. return volumeToAttach.GenerateError("AttachVolume.Attach failed", attachErr) }

We observed in an e2e that if attach is performed on a volume that was detached before from same node, sometimes the mount of the volume (for 2nd pod) could be stuck waiting on invalid devicePath in call. The problem is - when attach for second pod failed with this caused an dangling volume error from AWS cloudprovider(implemented by us). The dangling volume error is supposed to auto-correct any volumes which are attached to a node that it shouldn't be. But in this case - the dangling volume error was triggered by the fact that, volume was not actually attached to the node but because call for 2nd attachment returned stale data (because of AWS eventual consistency). So, dangling volume exception caused volume to be to actual state of world (so as it can be detached later), but this caused volume to be reported to node's Status field. And that caused second pod to assume volume as attached to invalid device path and hence 2nd pod was stuck for 10 minutes waiting for volume to attach on a device path (/dev/xvdbl) which does not exist.
/sig storage /sig aws cc
Here is k8s logs that confirm that, did return stale data for volume: See detach succeeds at 19.30.25: and yet: so second DescribeVolume API call is actually returning stale data.
There are couple of ways of fixing this: error is used by AWS and Openstack cloudproviders to fix incorrectly attached volumes. We could fix mechanism of dangling volumes, so as when it adds volumes to actual state of the world - it does so without reporting the volume to node's Status. can make a local fix inside AWS so that when we return error, before returning that error we also verify volume attachment by querying the instance. We already have instance information in that call. Verifying volume attachment by and should reduce likelyhood of returning incorrect errors. There is still a chance though that, both API calls would land on a AWS backend which does not yet have updated information.
expanding on fix - I am thinking when we add a volume to actual state of the world via DanglingVolume error, then we should add the volume as "uncertain". We will update the code to not report "uncertain" volumes in node's status and hence in this case volume state will be fixed correctly.


Question: volumeToAttach.VolumeSpec, volumeToAttach.NodeName) if attachErr != nil { <ins> uncertainNode := volumeToAttach.NodeName </ins> if derr, ok := attachErr.(*volerr.DanglingAttachError); ok { <del> addErr := actualStateOfWorld.MarkVolumeAsAttached( v1.UniqueVolumeName(""), originalSpec, derr.CurrentNode, derr.DevicePath) if addErr != nil { klog.Errorf("AttachVolume.MarkVolumeAsAttached failed to fix dangling volume error for volume %q with %s", volumeToAttach.VolumeName, addErr) } } else { addErr := actualStateOfWorld.MarkVolumeAsUncertain( v1.UniqueVolumeName(""), originalSpec, volumeToAttach.NodeName) if addErr != nil { klog.Errorf("AttachVolume.MarkVolumeAsUncertain fail to add the volume %q to actual state with %s", volumeToAttach.VolumeName, addErr) } </del> <ins> uncertainNode = derr.CurrentNode } addErr := actualStateOfWorld.MarkVolumeAsUncertain( v1.UniqueVolumeName(""), originalSpec, uncertainNode) if addErr != nil { klog.Errorf("AttachVolume.MarkVolumeAsUncertain fail to add the volume %q to actual state with %s", volumeToAttach.VolumeName, addErr) </ins> } // On failure, return error. Caller will log and retry. return volumeToAttach.GenerateError("AttachVolume.Attach failed", attachErr) }

If I understand correctly - did not intend to report "uncertain" volumes to node's status anyways.
This looks like another symptom of . I tried that, it does not always help. Both and may return the same stale data, see IMO, is the right way to go. Unless it break something else, all these AWS corner cases were complicated already and now anything can break.
/cc