Question: apiVersion: extensions/v1beta1 kind: DaemonSet metadata: <del> name: fluentd-es </del> <ins> name: fluentd-es-v2.0.1 </ins> namespace: kube-system labels: k8s-app: fluentd-es <del> version: v2.0.0 </del> <ins> version: v2.0.1 </ins> kubernetes.io/cluster-service: "true" addonmanager.kubernetes.io/mode: Reconcile spec:

: Environment: Kubernetes version (use ):Client Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-07-21T08:23:22Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-07-21T08:08:00Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration**: OS (e.g. from /etc/os-release):centos 7.3 Kernel (e.g. ):Linux node-21.46 3.10.0-514.26.2.el7.x8664 SMP Tue Jul 4 15:04:05 UTC 2017 x8664 x8664 x86_64 GNU/Linux Install tools: Others:
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
Yup, the situation with fluentd on the system with systemd is very sloppy. There's a hack to make it works sometimes, but it seems that it doesn't work. I'll look into it
/sig instrumentation /assign
Facing same issue.


Question: apiVersion: extensions/v1beta1 kind: DaemonSet metadata: <del> name: fluentd-es </del> <ins> name: fluentd-es-v2.0.1 </ins> namespace: kube-system labels: k8s-app: fluentd-es <del> version: v2.0.0 </del> <ins> version: v2.0.1 </ins> kubernetes.io/cluster-service: "true" addonmanager.kubernetes.io/mode: Reconcile spec:

Environment Details:kubectl version Client Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-07-03T15:31:10Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-07-03T15:31:10Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"} OS (e.g. from /etc/os-release):Centos 7 (Core) Kernel (e.g. uname -a): Linux xxxx 3.10.0-327.el7.x8664 Error for reference /var/lib/gems/2.1.0/gems/ffi- block in ffilib': Could not open library '': cannot open shared object file: No such file or directory. (LoadError) Could not open library '': cannot open shared object file: No such file or directory. Could not open library 'libsystemd-': cannot open shared object file: No such file or directory. Could not open library 'libsystemd-': libsystemd- cannot open shared object file: No such file or directory from /var/lib/gems/2.1.0/gems/ffi-map' from /var/lib/gems/2.1.0/gems/ffi- ffilib' from /var/lib/gems/2.1.0/gems/systemd-journal-<module:Native' from /var/lib/gems/2.1.0/gems/systemd-journal- <module:Id128' from /var/lib/gems/2.1.0/gems/systemd-journal-<module:Systemd' from /var/lib/gems/2.1.0/gems/systemd-journal- <top (required)' fromrequire' from require' from /var/lib/gems/2.1.0/gems/systemd-journal-<top (required)' from require' fromrequire' from /var/lib/gems/2.1.


Question: apiVersion: extensions/v1beta1 kind: DaemonSet metadata: <del> name: fluentd-es </del> <ins> name: fluentd-es-v2.0.1 </ins> namespace: kube-system labels: k8s-app: fluentd-es <del> version: v2.0.0 </del> <ins> version: v2.0.1 </ins> kubernetes.io/cluster-service: "true" addonmanager.kubernetes.io/mode: Reconcile spec:

0/gems/systemd-journal- <top (required)' fromrequire' from rescue in require' fromrequire' from /var/lib/gems/2.1.0/gems/fluent-plugin-systemd- <top (required)' fromrequire' from require' from /var/lib/gems/2.1.0/gems/fluentd-block in tryloadplugin' from /var/lib/gems/2.1.0/gems/fluentd- each' from /var/lib/gems/2.1.0/gems/fluentd-tryloadplugin' from /var/lib/gems/2.1.0/gems/fluentd- newimpl' from /var/lib/gems/2.1.0/gems/fluentd-newinput' from /var/lib/gems/2.1.0/gems/fluentd- addsource' from /var/lib/gems/2.1.0/gems/fluentd-block in configure' from /var/lib/gems/2.1.0/gems/fluentd- each' from /var/lib/gems/2.1.0/gems/fluentd-configure' from /var/lib/gems/2.1.0/gems/fluentd- configure' from /var/lib/gems/2.1.0/gems/fluentd-runconfigure' from /var/lib/gems/2.1.0/gems/fluentd- runconfigure' from /var/lib/gems/2.1.0/gems/fluentd-block in start' from /var/lib/gems/2.1.0/gems/fluentd- call' from /var/lib/gems/2.1.0/gems/fluentd-mainprocess' from /var/lib/gems/2.1.0/gems/fluentd- start' from /var/lib/gems/2.1.0/gems/fluentd-<top (required)' from require' fromrequire' from /var/lib/gems/2.1.0/gems/fluentd-0.12.39/bin/fluentd:5:in <top (required)' from /usr/local/bin/fluentd:23:inload' from /usr/local/bin/fluentd:23:in <


Question: apiVersion: extensions/v1beta1 kind: DaemonSet metadata: <del> name: fluentd-es </del> <ins> name: fluentd-es-v2.0.1 </ins> namespace: kube-system labels: k8s-app: fluentd-es <del> version: v2.0.0 </del> <ins> version: v2.0.1 </ins> kubernetes.io/cluster-service: "true" addonmanager.kubernetes.io/mode: Reconcile spec:

main'
You can solve it temporary by removing I'm working on the solution
A solution here might be to base fluentd-es off an image with journald supporting XZ and LZ4 compression. I'm looking :)
Thanks


Question: // in CLOSE_WAIT state will remain in the conntrack // table. (e.g. '60s'). Must be greater than 0 to set. TCPCloseWaitTimeout *metav1.Duration <ins> // tcpBeLiberal, if true, kube-proxy will configure conntrack // to run in liberal mode for TCP connections and packets with // out-of-window sequence numbers won't be marked INVALID. TCPBeLiberal bool </ins> // udpTimeout is how long an idle UDP conntrack entry in // UNREPLIED state will remain in the conntrack table // (e.g. '30s'). Must be greater than 0 to set.

What happened: When kube-proxy is configured with a cluster-cidr range, it drops packets with an INVALID conntrack state for all sources on the KUBE-FORWARD chain. This is meant to prevent spurious retransmits in long-lived TCP connections to a service IP, but it prevents clusters from having asymmetrical routing on private subnets with services outside of a cluster. What you expected to happen: Packets on the KUBE-FORWARD chain with an INVALID conntrack state should be dropped for only sources inside of the cluster-cidr if set. How to reproduce it (as minimally and precisely as possible): N/A Anything else we need to know?: Change to drop INVALID conntrack packets was in Backing out that commit and compiling is my current workaround. Environment: Kubernetes version (use ): 1.17.2 Cloud provider or hardware configuration: OS (e.g: ):Kernel (e.g. ):
/sig network
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
I am reading but I don't fully grok what caused the problem in the first place, or why we think "only drop if it was in-cluster" is a correct fix? Edit, OK I get the original problem, but still don't understand why packets from outside the pod range would not exhibit , thereby requiring yet another revisit of this problem. Is this a conntrack bug? Should it be applying the DNAT?
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
I can try to give a bit more context to our setup, but the knowledge I have around this all dates back to Feb, unfortuately, so I may have forgotten details. We were replacing our mesos cluster with a kubernetes cluster in Feb and using a custom ingress solution (HAProxy w/an outside process configuring it for pod ips / mesos ip-ports) for both. The ingress nodes lived in the private subnet, but outside of the kubernetes cluster cidr range. When we upgraded from kubernetes 1.14 to kubernetes 1.15 (the first version with PR ) our ingress could no longer talk to any pod ips.


Question: // in CLOSE_WAIT state will remain in the conntrack // table. (e.g. '60s'). Must be greater than 0 to set. TCPCloseWaitTimeout *metav1.Duration <ins> // tcpBeLiberal, if true, kube-proxy will configure conntrack // to run in liberal mode for TCP connections and packets with // out-of-window sequence numbers won't be marked INVALID. TCPBeLiberal bool </ins> // udpTimeout is how long an idle UDP conntrack entry in // UNREPLIED state will remain in the conntrack table // (e.g. '30s'). Must be greater than 0 to set.

I discovered that the KUBE-FORWARD chain had a rule to drop invalid packets, which was in Compiling kubernetes 1.15 without this commit fixed our problems, and we have been carrying that patch forward ever since. I would like to get rid of our forked kubernetes, hence this ticket. When I was writing this ticket up, I also found also mentions the same issue with the iptables dropping INVALID conntrack state packets near the end, regarding asymmetrical routing. Edit, OK I get the original problem, but still don't understand why packets from outside the pod range would not exhibit , thereby requiring yet another revisit of this problem. I think you are correct, I can imagine packets from outside of a cluster could exhibit the same behavior as packets inside the cluster and cause spurious resets. However, I am wondering if kube-proxy should be managing packets from outside of the cluster on a node... That seems outside of the scope of kube-proxy.
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
I took a stab at implementing what I think is the correct behavior for kube-proxy in .
/assign
I think that another option is to set
I don't understand how that will help me here. The packets with a ctstate INVALID are not generated due to conntrack saturation (), but due to asymmetric routing in our network. See for an example of a similar setup.
are the packets on your asymmetric routing setup marked as invalid if you set ? it should only mark the ones that are out of window per the description
/triage accepted
I'd love to pin this down, but I don't have the means to set up a repro. Is there a way I can set up a way to experiment?
We were bit by this too after revisiting an old service that had been unused since the heady days of pre-, and would also like to see the proposed patch merged (or a configuration option to prevent insertion of the INVALID state drop rule - either suits us). I unfortunately don't have any useful input as to setting up for reproduction. In our case we have a pretty odd anycast set up with a small pool of (public) addresses managed by calico.


Question: // in CLOSE_WAIT state will remain in the conntrack // table. (e.g. '60s'). Must be greater than 0 to set. TCPCloseWaitTimeout *metav1.Duration <ins> // tcpBeLiberal, if true, kube-proxy will configure conntrack // to run in liberal mode for TCP connections and packets with // out-of-window sequence numbers won't be marked INVALID. TCPBeLiberal bool </ins> // udpTimeout is how long an idle UDP conntrack entry in // UNREPLIED state will remain in the conntrack table // (e.g. '30s'). Must be greater than 0 to set.

However I do seem to have a dirty workaround for small clusters. By setting notrack (ctstate UNTRACKED?) in the raw table for packets in our specific ranges that would otherwise match ctstate INVALID, the packets aren't dropped. Which may be of help to someone else who finds this issue.
We are seeing something very similar on large file downloads and our issue exactly fits this one's subject. These RESET (corrected after are happening intermittently and I am seeing similar behavior mentioned in the blog mentioned below. I am unsure if someone has found a workaround for this issue or not, so yeah my +1 to this issue being looked up by the community.
We're using as a solution for the connection reset we were seeing and might fix this issue.
Let me try to avoid that this thread derails and see if I can get current status: -proxy install rules to drop INVALID cstate connections, this solves the ~REJECT~ RESET problems are some scenarios, with asymmetric routing per example, that this rules may drop unrelated traffic. The request is to remove the rule that drops those INVALID states or to try to restrict it, and that can cause that people start to have the ~REJECT~ RESET problems and drops again. If we have a reproducer we can be more precise in the final solution, but removing one rule without any strong evidence in kube-proxy is very risky As a workaround for the asymetric routing problem, I suggested to set the nfconntracktcpbeliberal on the kernel: I think that will solve the problems, it will be good if someone can test it and provide feedback
No sadly, its my bad, it is not REJECT, it is RESET that's happening and conntrack has nothing to do with it, mostly (I hope). Kubeproxy is doing what is intended but specifically when there are external agents in play, the packet doesn't sends an ACK back to the response of SYN-ACK. Our case, we hit A to initiate a download, A calls B to get the action completed to generate download data for the user finally via their browser. A and B both are on k8s, we saw A starts and a complete cycle goes for sometime until there multiple parallel downloads requested where few gets completed and few gets a RESET.


Question: // in CLOSE_WAIT state will remain in the conntrack // table. (e.g. '60s'). Must be greater than 0 to set. TCPCloseWaitTimeout *metav1.Duration <ins> // tcpBeLiberal, if true, kube-proxy will configure conntrack // to run in liberal mode for TCP connections and packets with // out-of-window sequence numbers won't be marked INVALID. TCPBeLiberal bool </ins> // udpTimeout is how long an idle UDP conntrack entry in // UNREPLIED state will remain in the conntrack table // (e.g. '30s'). Must be greater than 0 to set.

On primary debugging, it appears that RESET was getting invoked by the pod but pod is just sending a RST-ACK not RST itself. The missing packet is never found in the default route, unless we see that there is a chance where ACK is missed and we found some lines in tcpdump with missing sequence for ACKs. Thats why i doubt my issue is related here. One more weird coincidence is few downloads have spurious transmission.
Thankyou, I will take a look, but mostly I would like to wait before using it, debug more to understand the actual cause and then act on it.
sorry , I wanted to say RESET, thanks for catching it. If you think is different and you can provide information and how to reproduce it, I suggest you to open a new issue
I will sure do, when it is different. For now I would like to watch and submit my findings if it is related. PS: sysctl -w net.netfilter.nfconntracktcpbeliberal=1 is already implemented and we are testing if it does something helpful. so far it doesn't looks like it. thanks for your help.
/area kube-proxy
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle rotten
I have the feeling we are also affected by this with a standard calico installation: every node announces a /122 IPv6 network to the upstream router the upstream router uses ECMP to route the packet back packets that are sent from* the pod and returned via asymmetric path from outside seem to be dropped
and if you set ?
Same behaviour with set on all nodes, the outgoing pod traffic does not return. And I instantly see packets going back into the drop rule:
:thinking:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.


Question: // in CLOSE_WAIT state will remain in the conntrack // table. (e.g. '60s'). Must be greater than 0 to set. TCPCloseWaitTimeout *metav1.Duration <ins> // tcpBeLiberal, if true, kube-proxy will configure conntrack // to run in liberal mode for TCP connections and packets with // out-of-window sequence numbers won't be marked INVALID. TCPBeLiberal bool </ins> // udpTimeout is how long an idle UDP conntrack entry in // UNREPLIED state will remain in the conntrack table // (e.g. '30s'). Must be greater than 0 to set.

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hi, are you still working on this? Same problem happened in our clusters, just because of asymmetrical routing.
/reopen
Reopened this issue. <detailsIn response to : /reopen </details
Also the command suggested by is helping us, but I still think this should be a bug of kube-proxy.
The problem here is: what happens if the solutions are exclusive or we regress? The addition of this rule fixed a legit bug that was documented and there is also a nice blog explaining the details Removing this rule without a repro and understanding what is the root cause of this issue is not really the best way to proceed. I think that people affected should try to find the root cause and explain the problem or just provide a way to reproduce it, so we can fix it
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at .


Question: // in CLOSE_WAIT state will remain in the conntrack // table. (e.g. '60s'). Must be greater than 0 to set. TCPCloseWaitTimeout *metav1.Duration <ins> // tcpBeLiberal, if true, kube-proxy will configure conntrack // to run in liberal mode for TCP connections and packets with // out-of-window sequence numbers won't be marked INVALID. TCPBeLiberal bool </ins> // udpTimeout is how long an idle UDP conntrack entry in // UNREPLIED state will remain in the conntrack table // (e.g. '30s'). Must be greater than 0 to set.

/close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The same problem happened in our cluster, the same as because of asymmetrical routing. the command suggested by doesn't work for us, and I delete the iptables rule and add new rule(refer to ), it works I think that makes sense I'll put up a reproduction of the gist link later on /reopen
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The main problem is how to be completely sure we don't regress, this comment from Tim is spot on
I'm trying to check that non-local connections will still reset without the INVALID Rule
I think the patch in is wrong; it changes the rule from "if INVALID then drop" to "if INVALID and from a pod, then drop" so that it doesn't affect non-kubernetes traffic, but I think the rule we actually need is "if INVALID and to a service, then drop". In particular, I believe we need to make sure that the rule applies to traffic with a cluster-external source and an in-cluster destination. It's possible that changing it to "if INVALID and to a pod, then drop" would also work. I skimmed and some linked bugs elsewhere but I was unable to confirm whether that rule would cover the reporters' use cases.
I think that is only INVALID and to a pod
In my case, cluster-external source to service is both ok with the INVALID rule or without the INVALID rule. It seems the rule doesn't affect on access service.


Question: // in CLOSE_WAIT state will remain in the conntrack // table. (e.g. '60s'). Must be greater than 0 to set. TCPCloseWaitTimeout *metav1.Duration <ins> // tcpBeLiberal, if true, kube-proxy will configure conntrack // to run in liberal mode for TCP connections and packets with // out-of-window sequence numbers won't be marked INVALID. TCPBeLiberal bool </ins> // udpTimeout is how long an idle UDP conntrack entry in // UNREPLIED state will remain in the conntrack table // (e.g. '30s'). Must be greater than 0 to set.

I add this rule but it doesn't work
So, based on that, the bug happens when a reply packet fails to get un-DNAT'ed, causing the client to then send a RST to the server, which matches the source/destination IP that the server knows of. But this could happen if you had a service with external IPs too; in that case the reply packet would be from the external endpoint IP to the node IP (because the packet would have been masqueraded), and if conntrack didn't unmap it correctly then the node would send a RST to the external IP, breaking the connection in just the same way. In this case there is no good way kube-proxy can recognize the packet as being its own, other than by using conntrack, which presumably won't work if the packet is INVALID?
It seems that there is no better way to solve this for external sources accessing the cluster CIDR, is it right? If so, Can we have a flag to decide whether to create The invalid rule? It would be nice that It's up to the user to decide whether to create the rule.
There's related discussion in . I think we should create whatever iptables rules we can that will fix up our own traffic without interfering with other traffic, and also document the sysctl that admins can use to change conntrack behavior to work around the problem. So probably, we should change the "drop if INVALID" rule to "drop if INVALID and to a pod", and also address the problem in , but not try to solve the problem with external IPs I mentioned in because we don't have a good way of fixing that without potentially breaking other things. (And it's kind of way more of an edge case anyway.) Note that can currently output rules for "from a pod", but not for "to a pod", so that will need some tweaking. (FWIW will also eventually need the ability to output nftables rules rather than iptables rules so that's another thing to think about if someone is going to change its API.)
So in the absence of a way to fix the problem with external IPs, can we have a flag to decide whether to create this rule or not? If true (default), create the drop if INVALID and to a pod rule. Otherwise, We do not create the rule.


Question: // in CLOSE_WAIT state will remain in the conntrack // table. (e.g. '60s'). Must be greater than 0 to set. TCPCloseWaitTimeout *metav1.Duration <ins> // tcpBeLiberal, if true, kube-proxy will configure conntrack // to run in liberal mode for TCP connections and packets with // out-of-window sequence numbers won't be marked INVALID. TCPBeLiberal bool </ins> // udpTimeout is how long an idle UDP conntrack entry in // UNREPLIED state will remain in the conntrack table // (e.g. '30s'). Must be greater than 0 to set.

I believe that even without this rule we can solve this problem by way of sysctl, right? I think we should do the following things We need a flag to disable the rule We should change the "drop if INVALID" rule to "drop if INVALID and to a pod" We document sysctl as a way to solve the problem.
If the rule is "drop if INVALID and to a pod", we shouldn't need a flag to disable it; it won't affect non-pod traffic, and it won't interfere with users that want to set the sysctl instead. (Setting the sysctl will cause packets to not be marked as INVALID so the rule we add just won't get hit.)
OK, drop if INVALID and to a pod is help us identify cluster traffic, right? We only drop cluster traffic if it's invalid, but this will not solve the problem of external IP. So we need a flag to disable the rule is drop if INVALID?
The same problem happened in our cluster, the same as because of asymmetrical routing None of the above methods solve the problem in our ENV. Are there any other workarounds?
Can you execute the following command on each node? I think this is a workaround.
Thank you for your reply I also tried removing this rule. However, k8s will reconfigure the deleted rules to the system at some point in time.
I worded that badly: I meant "external IP" as in "the endpoints are cluster-external IPs rather than pod IPs", not as in . So that would apply only to people using kube-proxy to load-balance traffic from clients outside the cluster to servers outside the cluster, which is a pretty weird use case. And nobody ever reported the packet RST bug for that case anyway. So I was arguing that we can just ignore that case. So: change the existing "drop if invalid" rule to "drop if invalid and to a pod" document the sysctl We don't need an option to disable the drop rule, because it won't affect non-k8s traffic, and it will automatically become a no-op if someone decides to use the sysctl instead.
Thank you for your response. I'd love to take this one. But I have a question, How to define k8s traffic?


Question: // in CLOSE_WAIT state will remain in the conntrack // table. (e.g. '60s'). Must be greater than 0 to set. TCPCloseWaitTimeout *metav1.Duration <ins> // tcpBeLiberal, if true, kube-proxy will configure conntrack // to run in liberal mode for TCP connections and packets with // out-of-window sequence numbers won't be marked INVALID. TCPBeLiberal bool </ins> // udpTimeout is how long an idle UDP conntrack entry in // UNREPLIED state will remain in the conntrack table // (e.g. '30s'). Must be greater than 0 to set.

If a pod in a cluster, we call it an underlay pod, its IP is not allocated from cluster CIDR, this is a real underlay IP, when it accesses other underlay pods in the cluster or pods that allocate IP from cluster CIDR(We call it overlay pod), is this k8s traffic? In our cluster, multiple types of CNI pods are implemented through multus running in a cluster, because of the problem of asymmetrical routing and the rule that makes them inaccessible, it seems that the above method will not solve this problem.
For kube-proxy, this is done by the . If you are in a cluster where there is no simple way to discriminate pod from non-pod traffic, then you'd have to run kube-proxy with , and then kube-proxy will see that returns and so it won't try to distinguish local from non-local traffic, and will do some things (like internal/external traffic policy) slightly differently as a result. So in this case, if there was no functioning localDetector, it would also not create the "drop invalid" rule
Thanks for the details, this is probably the case we want.
/assign
I did not understand how you could get asymmetric routing in K8s, but reading up more carefully I saw: If I understand this correctly, this is a load-balancer outside K8s that load-balance directly to POD addresses. To get asymmetric routing some kind of Direct Server Return (DSR) must also be used. So, to get asymmetric routing: from an external source is directed to an external LB LB is configured with POD-IPs as targets and selects one packet is routed to a K8s node, e.g. with ECMP. But is may not be the node where the target POD is executing packet is handled by the POD, but the response packet may be directed to some other GW This will give asymmetric routing. But it's not the every-day K8s network setup for sure. Given the above it's fairly easy to reproduce the problem. You don't even have to setup a load-balancer. The biggest problem may be that you need two ways in to the cluster. Example: ! I setup direct routes that ensures asymmetric routing, which can be checked with on vm-201 and vm-202.


Question: // in CLOSE_WAIT state will remain in the conntrack // table. (e.g. '60s'). Must be greater than 0 to set. TCPCloseWaitTimeout *metav1.Duration <ins> // tcpBeLiberal, if true, kube-proxy will configure conntrack // to run in liberal mode for TCP connections and packets with // out-of-window sequence numbers won't be marked INVALID. TCPBeLiberal bool </ins> // udpTimeout is how long an idle UDP conntrack entry in // UNREPLIED state will remain in the conntrack table // (e.g. '30s'). Must be greater than 0 to set.

From I can now directly ping POD-IPs, but due to the problem in this issue, I can't make a TCP connection: The POD with address runs on . A similar setup as above can be used to verify a PR. Now, there is a simple work-around: use proxy-mode=ipvs The update with the rule never made it to the ipvs proxier (a rather common case :cry: ), but in this case it may save your day. The same as above with proxy-mode=ipvs; The server on 5001 responds with the POD name.
Yeah, Our production environment also solved this problem by switching to IPVS mode. In IPVS mode, there is no the DROP rule.
The sysctl solves the problem in , but not the problem in this issue. Those problems are kind of related, but rather distant.
The same affects outgoing traffic if packets are not masqueraded to the node-ip, as should be a common setup for ipv6, please see For Calico I think you can setup BGP for clusterCIDRs and route incoming packets to PODs to the correct K8s node, instead of using ECMP. This would solve the problem, and save the extra hop.
I met the same issue. KUBE-FORWARD drop my packages.
temptatively solution
The issue fixed by , I'll close it, you can feel free to open it if have any questions. /close
Closing this issue. <detailsIn response to : The issue fixed by , I'll close it, you can feel free to open it if have any questions. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: configFile := filepath.Join(kubeletDir, kubeadmconstants.KubeletConfigurationFileName) fmt.Printf("[kubelet] Writing kubelet configuration to file %qn", configFile) <ins> // creates target folder if not already exists if err := os.MkdirAll(kubeletDir, 0700); err != nil { return fmt.Errorf("failed to create directory %q: %v", kubeletDir, err) } </ins> if err := ioutil.WriteFile(configFile, b, 0644); err != nil { return fmt.Errorf("failed to write kubelet configuration to the file %q: %v", configFile, err) }

Error Logs /kind bug /priority failing-test /priority critical-urgent /milestone v1.11 /assign cc Possible breaking change as per offline discussions.
- Timeline of breakage aligns with merge of , nothing else was in flight to cause it.
could this be the problem?
Is that's the problem, may be this will help
/sig cluster-lifecycle
/status approved-for-milestone
[MILESTONENOTIFIER] Milestone Issue Needs Attention Action required: During code freeze, issues in the milestone should be in progress. If this issue is not being actively worked on, please remove it from the milestone. If it is being worked on, please add the label so it can be tracked with other in-flight issues. Note: This issue is marked as , and must be updated every 1 day during code freeze. Example update: <details<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
you can't re-open an issue/PR unless you authored it or you are assigned to it. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
https://k8s- https://k8s- both jobs are better now (past the failure where they keeled over before the test suite started). But both seem to have some test failures
we have one green run on each, so let's close this out. https://k8s- https://k8s- /close


Question: pluginBinDir: pluginBinDir, args:         provider.Args, envVars:      provider.Env, <ins> environ:      os.Environ, </ins> }, }, nil }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --I am attempting to pull an image from ECR using the new kubelet image credential provider feature which is an alpha features in Kubernetes v1.20. My CredentialProviderConfig file includes the environment variable AWSPROFILE which resides in the default () directory. The profile references a which allows me to execute a binary that returns a set of temporary AWS credentials. When I try running a pod with a container from ECR, I see the following errors in the kubelet logs: However, when I type the following into the command line on the kubelet: I get the following response: It doesn't appear the environment variables in my Config are being passed to the Kubelet when it executes the ecr-credential-provider binary. Below is Config file: I expect for the Kubelet to pass the environment variables in the Config file when it executes the ecr-credential-provider binary. I am using KinD to test this feature. Here is my config: No Kubernetes version (use ): 1.20 Cloud provider or hardware configuration: None OS (e.g: ): Ubuntu 20.10 Kernel (e.g. ): 5.10 Install tools: KinD Network plugin and version (if this is a network-related bug): Others: cc
/sig Node
/assign
from the code it appears that env variables are passed to the plugin from kubelet logs the plugin is only returning exit status 1, without any details about the error!!
Thanks Yes, I saw that. doesn't work. I wrote the following code to test this: If I uncomment , which imports the environment variables from the OS, it works.
Hello need some clarification. Is the intent of the above code to convey that isn't passing env vars correctly to the spawned process? Below code works as expected for me Also, could you tell if is a proprietary binary or a open source code something similar to because I want to use it in troubleshooting this issue.
Yes it is not passing the Envs to the spawned process.


Question: pluginBinDir: pluginBinDir, args:         provider.Args, envVars:      provider.Env, <ins> environ:      os.Environ, </ins> }, }, nil }

The code for ecr-credential-provider is in the AWS cloud controller repository,
Quick update, I had setup two test beds to reproduce the above issue, but both of them gave expected results and the issue was not reproducible. Below are my test setups: ECR Repository Test Bed 1 Execute from code and pass as environment variable Code Observed Result - Below credentials were generated Expected Result - Credentials should be generated Test Bed 2 Use as the credential provider within the kubelet, I had setup a Provider Config Expected result - Pods should be created based off custom image Observed result - Pods were created based off custom image Though, my test bed runs on kubernetes version , I could see that nothing significant has changed in since version Could you give me any other minor but important detail which you feel you could had missed out earlier which could help me in replicating the issue? Or could you double check if provider exists in your file?
i am using credential_process, in my profile. I am also using KinD for testing, but I don't think that should matter. I don't have a credentials file. I only have a config file.
This is the binary I'm calling with credential_process,
can you check kubelet logs for , It might be possible that is falling back to in-tree aws credential provider.
on my local setup credentials are being fetched. I'll setup another test bed with the Cognito provider which is using.
It is fetching credentials because it is falling back to other options available, It might be fetching from other credential provider if failed from external one. To confirm this you have to check kubelet logs.
In my env I just have one provider and there are no fallbacks. Though I was keeping an eye on kubelet logs while performing this activity and I saw no such logs. The creds were fetched in one shot via the profile which I had specified. And as I had mentioned earlier I don't see any issues in passing env vars to credential process. Also, I did try negative cases as well by passing invalid profile names and it did fail and pods weren't pulled from ECR
kubelet registers intree credential providers while startup, you need not to explicitly add any!! And I recently tried GCP credential provider, It does fall back to intree GCP credential provider after getting failed response from the one i set up.
Sure, makes sense.


Question: pluginBinDir: pluginBinDir, args:         provider.Args, envVars:      provider.Env, <ins> environ:      os.Environ, </ins> }, }, nil }

Though as I mentioned earlier I was keeping an eye on the kubelet logs and I didn't see the logs you had mentioned
i see in the logs when the kubelet starts, before attempting to run a pod that pulls an image from ECR.
Had a light bulb moment and I was able to reproduce this issue, it might not be applicable in your case but it could be worth trying out. In my local setup I ran the cluster as root and failed with error: E0614 18:19:27. ] Failed getting credential from external registry credential provider: error execing credential provider plugin ecr-credential-provider for image exit status 1 I have my aws credentials configure under my user () at path and since there is no similar configuration for the process failed. Since, I have access to code locally, I couple of log statements to print out current env vars and I could see that env var was set to :) Would it be possible for you to check is running as what user and is there any dissonance which can prohibit the user from successfully retrieving the credentials?
I am getting the error you mentioned, So far as I can tell, the kubelet is running as root and the file is in the directory. Again, I'm using KinD for testing. I'm not sure how KinD runs the kubelet. Below is the output from : The Go code I wrote to test this fails when using cmd.Env; it's only when I import the OS environment variables that it works. This may because I am using in my shared config file. credentialprocess is a mechanism to call another binary, e.g. , to get a set of temporary credentials from Cognito.
In order to simplify the sprawl and to pinpoint the offending component,could you remove the credential_process and Cognito and just try fetching creds via plain AWS credentials profile?
sure, however, supporting credential process is critical for my use case. I'm using Cognito as a gating mechanism to ECR. Only licensed users will be allowed to pull images from ECR. These users will not have IAM credentials (access key/secret key). They will authenticate to AWS using Cognito.
The following configuration works when I specify access key/secret key:
When I switch to a profile with credential process I get the following error:
isn't it the issue of ecr-credential-provider?


Question: pluginBinDir: pluginBinDir, args:         provider.Args, envVars:      provider.Env, <ins> environ:      os.Environ, </ins> }, }, nil }

This is the job of ecr-credential-provider to fetch credentials and return result in expected format to kubelet.
why does it work when I run it from the command line? Why does it work when I import envs from the OS?
you might have saved env on command line, same goes for OS.
That doesn't make sense to me. If I run it returns a properly formatted response. Same goes for the Go code I wrote. Does calling another binary, like credentialprocess, confuse the kubelet?
I am not familiar with aws credential process. Kubelet talks to with exec based plugin mechanism, is ecr-credential-provider calling the credential process you are talking about?
I have already validated that env vars are being passed to and there is no issue over there. If that wouldn't be the case then AWS credentials generation process via wouldn't work for you. I had debugged the code of kubelet myself and validated it + I wrote some logs in to print out the env vars and it was logging the passed vars. IMHO, now what remains to be seen here is why it fails with Cognito. I am going to create a test bed tomorrow replicating the exact setup which you have in the mean time if it's possible for you could you create a Linux VM and replicate your setup inside it? My intention is to see whether the Cognito based credential process goes through or not in a clean slate environment if it does then something on your local setup is interfering with passed env vars Thoughts?
Here's the execution flow ---- Does the above paints a fairly accurate picture? If not then feel free to correct :)
I included my KinD configuration in the beginning of this thread. Can you use that The binary has to be built from source, (or I can send it to you along with a temp username/password). There is an accompanying blog post for the binary at The flow is as describes above. The binary returns a json blob to the calling application that looks like this: During my testing (without the kubelet) I am able to get the proper response for the kubelet when referencing the AWS profile using or by setting to . However, when I try to pass the AWS profile in using exec.Env("AWS_PROFILE=default") I get and no further information. I have a feeling that doesn't properly chain or pass information back to the calling application.


Question: pluginBinDir: pluginBinDir, args:         provider.Args, envVars:      provider.Env, <ins> environ:      os.Environ, </ins> }, }, nil }

Let me replicate your setup in my local env and see if I can reproduce this issue. Also, could you tell what your AWS file look like? Is it something like this You can use placeholder values for sensitive information but correct the binary path if it's not
yes, that looks right. Ideally the cognito binary is located in a directory in the shell's PATH but it also works when you specify the full path in the config file like you've done. Are you planning to setup cognito in AWS? You'll need a cognito user pool and configure federation.
Yes, I am planning to setup Cognito Identity pool in my AWS Account.
So I replicated your setup on my local machine with slight change, instead of using AWS Cognito for generating credentials via I am calling a binary written by me which logs env vars to a specific path and prints out credentials in the format as prescribed by AWS. Below was my initial config file After I ran local cluster with this setup I noticed couple of log lines like below: I did some basic validation of perms + I even moved the binary to couple of well know locations but nothing worked Finally I modified my vars section of my deployment spec like below and it worked The logs which I had put in my custom credential provider printed following env vars validating the fact that, env vars are being passed In your logs do you see a log line something like above? If not then could you pass full contents of from start up until you create a Pod?
thanks for doing this. I never tried adding the PATH to the list of environment variables. I honestly didn't think it was necessary because I was referencing the full path to the binary in credential_process. I also never saw an error about not being able to find the executable in $PATH. I'll re-test after adding PATH as an ENV and let you know the results.
Adding the PATH did not work for me. I'm happy to send you the cognito binary with a username/password that you can use for testing. I have only tested this with KinD, but I don't think that is the issue here since it fails with exit code when when I try using .
^^ is my credential provider config
Thanks for the logs. I have cloned your code in my repo and I'll substitute my Cognito creds.


Question: pluginBinDir: pluginBinDir, args:         provider.Args, envVars:      provider.Env, <ins> environ:      os.Environ, </ins> }, }, nil }

If it doesn't work out then I'll reach out to you for creds
After several rounds of troubleshooting I was able to get working in my KinD environment. My credential loading setup is <-<-and below is my configuration of credential process Initially, I got the same error as yours, then I couple of custom log lines to print env vars in in directory and I could verify that the env vars specified in the credentials spec were indeed being passed. What I noticed in my KinD setup is that AWS SDK was logging an error because it couldn't find any "providers", though I could verify in the container that file and were indeed present. I altered my env vars section of the as below and it worked for me Apparently, what I noticed is is being passed ONLY the env vars which are present in the spec and everything else was missing! Or at least this is the behaviour in my local cluster setup + KinD cluster. Can you please add and other env vars necessary for the correct execution of AWS SDK and see if it works for you? If this doesn't work, then I'll send you the modified version of the which puts some additional logs to troubleshoot this issue further.
That did it! Thanks for helping me troubleshoot this. I wonder if this is unique to KinD or whether we would see this behavior in a "normal" cluster (i'll test it later). I will tell the developers who wrote the ecr-credential-provider to include an argument to increase the verbosity or to output logs to a file so this is easier to troubleshoot in the future.
What I have observed is, is only passed env vars which are in credential spec and the spec clearly says that the vars in spec are appended to existing env vars and this looks like an issue which needs deeper investigation. Could you please re-open this issue?
sure thing.
The label cannot be applied. Only GitHub organization members can add the label. <detailsIn response to : /triage accepted Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
To me this looks like an issue with the kubelet and I have observed this behaviour with as well as my local cluster. What are your thoughts on this? If this looks like a bug then can you accept the triage? I would fix this.


Question: pluginBinDir: pluginBinDir, args:         provider.Args, envVars:      provider.Env, <ins> environ:      os.Environ, </ins> }, }, nil }

Thanks for looking into the issue, you are right the spec does says that. Feel free to fix it /triage accepted
Thanks!!!
/area kubelet
: I am facing the similar issue on my bare metal cluster This is my config Can you help. It works when I try to execute the credentials binary from local but doesn't work in the pod. Can you pls guide me
I've been fighting this for days - did you ever figure it out?


Question: } if len(list.Items) != len(test.items) { <del> t.Errorf("unexpected length: %d vs %d", len(list.Items), len(test.items)) return </del> <ins> t.Fatalf("unexpected length: %d vs %d", len(list.Items), len(test.items)) </ins> } // The order of elements in LIST is not guaranteed. mapping := make(map[string]int)

A user report that it's broken in the slack channel:
Also and -- this probably broke with one of our recent refactors. The server should be accepting list options from any version, I thought?
FWIW, I think it's this line barf at the ListOptions: The parameter codec first decodes the query parameter to a versioned object and then convert it to the internal object. The code fails because it cannot create a "" ListOptions.
The server should accept list options in the API version it is sent. ListOptions should be registered in all third party groups.
The thirdPartyResourceDataCreator should handle "ListOptions", which surprises me that it doesn't then respect the group.
I think we should add a wrapper around the api.ParameterCodec here: Do something similar as what does.
Putting together a PR now and will let you guys review.
I guess my point is, the list options are just there from the URL query parameters. The user didn't send a group. Why do we assume that value for the group rather than "/v1" or something?
My understanding is we want to allow options to evolve differently in different groups. We do have v1.ListOptions and v1beta1.ListOptions. And if a client sends requeset to extensions/v1beta1 endpoint, then it's expected to encode the ListOptions to v1beta1 and then convert it to query parameters.
Mostly because the parameter codec is supposed to abstract that. "virtual" groups need to be implemented by the virtual "parameter codec". The statement on floating server objects (List, Delete etc) was that they have a locked API schema, and we have a mechanism to do that via the scheme which is registering them in a group. , Daniel Smith wrote:


Question: args         []string envVars      []kubeletconfig.ExecEnvVar pluginBinDir string <ins> environ      func() []string </ins> } // ExecPlugin executes the plugin binary with arguments and environment variables specified in CredentialProviderConfig:


Question: args         []string envVars      []kubeletconfig.ExecEnvVar pluginBinDir string <ins> environ      func() []string </ins> } // ExecPlugin executes the plugin binary with arguments and environment variables specified in CredentialProviderConfig:


Question: args         []string envVars      []kubeletconfig.ExecEnvVar pluginBinDir string <ins> environ      func() []string </ins> } // ExecPlugin executes the plugin binary with arguments and environment variables specified in CredentialProviderConfig:


Question: args         []string envVars      []kubeletconfig.ExecEnvVar pluginBinDir string <ins> environ      func() []string </ins> } // ExecPlugin executes the plugin binary with arguments and environment variables specified in CredentialProviderConfig:


Question: args         []string envVars      []kubeletconfig.ExecEnvVar pluginBinDir string <ins> environ      func() []string </ins> } // ExecPlugin executes the plugin binary with arguments and environment variables specified in CredentialProviderConfig:


Question: args         []string envVars      []kubeletconfig.ExecEnvVar pluginBinDir string <ins> environ      func() []string </ins> } // ExecPlugin executes the plugin binary with arguments and environment variables specified in CredentialProviderConfig:


Question: args         []string envVars      []kubeletconfig.ExecEnvVar pluginBinDir string <ins> environ      func() []string </ins> } // ExecPlugin executes the plugin binary with arguments and environment variables specified in CredentialProviderConfig:


Question: Method  string Request struct { VolumeContext map[string]string `json:"volume_context"` <del> Secret        map[string]string `json:"secret"` </del> <ins> Secrets       map[string]string `json:"secrets"` </ins> } FullError struct { Code    codes.Code `json:"code"`

Seen in pull-kubernetes-e2e-gce-cos-alpha-features Specific job failure
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
from the error message I don't know if it refers to the sig-storage error or to the sig-auth one
/sig storage
/kind failing-test
/assign From : ... I am not sure if it's related:
is the culprit. cannot be used inside a . This only gets caught when the test actually runs, otherwise Ginkgo never sees the incorrect .
I don’t understand why sometimes it can get the pv by namebut sometimes it failed
but then is a sig-auth test , not storage , sorry /sig auth this seems that was recently in
/remove sig-storage
/assign /priority important-soon
/unassign
is still failed. seems not fixing the whole problem. /reopen feel free to close if the ci is green.
Reopened this issue. <detailsIn response to : is still failed. /reopen feel free to close if the ci is green. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
the failure now looks like it is just a sig storage issue:
cc
/assign looks like the new test is still an issue
I think this line is the problem: When the test creates the object, it may take time for the PV to be provisioned and to be bound to the PVC. So you should do another before this call so you can get the latest PVC object with the bound PV.
I opened to get PVC again before using .
Only do PersistentVolumeClaims.Get() is not enough, whether we need to wait for sometimes to wait it bound? because sometimes the testcase is successful and sometime is failed.
If the Pod was able to run, then the PVC has to be bound.
so I can't understand why sometimes the testcase is successful and sometimes is failed.


Question: Method  string Request struct { VolumeContext map[string]string `json:"volume_context"` <del> Secret        map[string]string `json:"secret"` </del> <ins> Secrets       map[string]string `json:"secrets"` </ins> } FullError struct { Code    codes.Code `json:"code"`

By default, the DelayBinding is false, it will wait for PVC bound during creation. This is weird.
It waits for the claim to be bound, but the test case still needs to refresh its pvc object.
Because of the refresh is async, so after the pvc was bound, we need to get the pvc again, isn't it?


Question: "time" apps "k8s.io/api/apps/v1beta1" <del> "k8s.io/api/core/v1" </del> <ins> v1 "k8s.io/api/core/v1" </ins> "k8s.io/api/extensions/v1beta1" policy "k8s.io/api/policy/v1beta1" apiequality "k8s.io/apimachinery/pkg/api/equality"

From integration tests:
/assign
Nope - you need something like that:
I was looking at the GetPodFromDeleteAction when I noticed PR opened for this issue. Hence I didn't update my snippet.


Question: kubectl "k8s.io/kubernetes/pkg/kubectl/cmd" "k8s.io/kubernetes/pkg/kubectl/cmd/templates" cmdutil "k8s.io/kubernetes/pkg/kubectl/cmd/util" <ins> "k8s.io/kubernetes/pkg/util/i18n" </ins> "github.com/spf13/cobra" ) <ins> var ( kubefedVersionExample = templates.Examples(i18n.T(` # Print the client and server versions for the current context kubefed version`)) kubefedOptionsExample = templates.Examples(i18n.T(` # Print flags inherited by all commands kubefed options`)) ) </ins> // NewKubeFedCommand creates the `kubefed` command and its nested children. func NewKubeFedCommand(f cmdutil.Factory, in io.Reader, out, err io.Writer, defaultServerImage, defaultEtcdImage string) *cobra.Command { // Parent command to which all subcommands are added.

Referring and ; both have string in place of in section Example. This needs to be corrected in kubefed docs autogeneration. cc also see
/assign


Question: t.Errorf("RunDeleteToken() succeeded for an invalid token: %v", err) } } <del> var httpTestItr uint32 var httpSentResponse uint32 = 1 func TestRunListTokens(t *testing.T) { var err error var bufOut, bufErr bytes.Buffer tmpDir, err := ioutil.TempDir("", "kubeadm-token-test") if err != nil { t.Errorf("Unable to create temporary directory: %v", err) } defer os.RemoveAll(tmpDir) fullPath := filepath.Join(tmpDir, "test-config-file") f, err := os.Create(fullPath) if err != nil { t.Errorf("Unable to create test file %q: %v", fullPath, err) } defer f.Close() // test config without secrets; should fail if _, err = f.WriteString(testConfigToken); err != nil { t.Errorf("Unable to write test file %q: %v", fullPath, err) } client, err := getClientset(fullPath, true) if err != nil { t.Errorf("Unable to run getClientset() for test file %q: %v", fullPath, err) } if err = RunListTokens(&bufOut, &bufErr, client); err == nil { t.Errorf("RunListTokens() did not fail for a config without secrets: %v", err) } // test config without secrets but use a dummy API server that returns secrets portString := "9008" http.HandleFunc("/", httpHandler) httpServer := &http.Server{Addr: "localhost:" + portString} go func() { err := httpServer.ListenAndServe() if err != nil { t.Errorf("Failed to start dummy API server: localhost:%s", portString) } }() fmt.Printf("dummy API server listening on localhost:%sn", portString) testConfigTokenOpenPort := strings.Replace(testConfigToken, "server: localhost:8000", "server: localhost:"+portString, -1) if _, err = f.WriteString(testConfigTokenOpenPort); err != nil { t.Errorf("Unable to write test file %q: %v", fullPath, err) } client, err = getClientset(fullPath, true) if err != nil { t.Errorf("Unable to run getClientset() for test file %q: %v", fullPath, err) } // the order of these tests should match the case check // for httpTestItr in httpHandler testCases := []struct { name          string expectedError bool }{ { name:          "token-id not defined", expectedError: true, }, { name:          "secret name not formatted correctly", expectedError: true, }, { name:          "token-secret not defined", expectedError: true, }, { name:          "token expiration not formatted correctly", expectedError: true, }, { name:          "token expiration formatted correctly", expectedError: false, }, { name:          "token usage constant not true", expectedError: false, }, { name:          "token usage constant set to true", expectedError: false, }, } for _, tc := range testCases { bufErr.Reset() atomic.StoreUint32(&httpSentResponse, 0) fmt.Printf("Running HTTP test case (%d) %qn", atomic.LoadUint32(&httpTestItr), tc.name) // should always return nil here if a valid list of secrets if fetched err := RunListTokens(&bufOut, &bufErr, client) if err != nil { t.Errorf("HTTP test case %d: Was unable to fetch a list of secrets", atomic.LoadUint32(&httpTestItr)) } // wait for a response from the dummy HTTP server timeSpent := 0 * time.Millisecond timeToSleep := 50 * time.Millisecond timeMax := 2000 * time.Millisecond for { if atomic.LoadUint32(&httpSentResponse) == 1 { break } if timeSpent >= timeMax { t.Errorf("HTTP test case %d: The server did not respond within %d ms", atomic.LoadUint32(&httpTestItr), timeMax) } timeSpent += timeToSleep time.Sleep(timeToSleep) } // check if an error is written in the error buffer hasError := bufErr.Len() != 0 if hasError != tc.expectedError { t.Errorf("HTTP test case %d: RunListTokens expected error: %v, saw: %v; %v", atomic.LoadUint32(&httpTestItr), tc.expectedError, hasError, bufErr.String()) } } } // only one of these should run at a time in a goroutine func httpHandler(w http.ResponseWriter, r *http.Request) { tokenID := []byte("07401b") tokenSecret := []byte("f395accd246ae52d") tokenExpire := []byte("2012-11-01T22:08:41+00:00") badValue := "bad-value" name := bootstrapapi.BootstrapTokenSecretPrefix + string(tokenID) tokenUsageKey := bootstrapapi.BootstrapTokenUsagePrefix + "test" secret := v1.Secret{} secret.Type = bootstrapapi.SecretTypeBootstrapToken secret.TypeMeta = metav1.TypeMeta{APIVersion: "v1", Kind: "Secret"} secret.Data = map[string][]byte{} switch atomic.LoadUint32(&httpTestItr) { case 0: secret.Data[bootstrapapi.BootstrapTokenIDKey] = []byte("") case 1: secret.Data[bootstrapapi.BootstrapTokenIDKey] = tokenID secret.ObjectMeta = metav1.ObjectMeta{Name: badValue} case 2: secret.Data[bootstrapapi.BootstrapTokenIDKey] = tokenID secret.Data[bootstrapapi.BootstrapTokenSecretKey] = []byte("") secret.ObjectMeta = metav1.ObjectMeta{Name: name} case 3: secret.Data[bootstrapapi.BootstrapTokenIDKey] = tokenID secret.Data[bootstrapapi.BootstrapTokenSecretKey] = tokenSecret secret.Data[bootstrapapi.BootstrapTokenExpirationKey] = []byte(badValue) secret.ObjectMeta = metav1.ObjectMeta{Name: name} case 4: secret.Data[bootstrapapi.BootstrapTokenIDKey] = tokenID secret.Data[bootstrapapi.BootstrapTokenSecretKey] = tokenSecret secret.Data[bootstrapapi.BootstrapTokenExpirationKey] = tokenExpire secret.ObjectMeta = metav1.ObjectMeta{Name: name} case 5: secret.Data[bootstrapapi.BootstrapTokenIDKey] = tokenID secret.Data[bootstrapapi.BootstrapTokenSecretKey] = tokenSecret secret.Data[bootstrapapi.BootstrapTokenExpirationKey] = tokenExpire secret.Data[tokenUsageKey] = []byte("false") secret.ObjectMeta = metav1.ObjectMeta{Name: name} case 6: secret.Data[bootstrapapi.BootstrapTokenIDKey] = tokenID secret.Data[bootstrapapi.BootstrapTokenSecretKey] = tokenSecret secret.Data[bootstrapapi.BootstrapTokenExpirationKey] = tokenExpire secret.Data[tokenUsageKey] = []byte("true") secret.ObjectMeta = metav1.ObjectMeta{Name: name} } secretList := v1.SecretList{} secretList.Items = []v1.Secret{secret} secretList.TypeMeta = metav1.TypeMeta{APIVersion: "v1", Kind: "SecretList"} output, err := json.Marshal(secretList) if err == nil { w.Header().Set("Content-Type", "application/json") w.WriteHeader(200) w.Write([]byte(output)) } atomic.AddUint32(&httpTestItr, 1) atomic.StoreUint32(&httpSentResponse, 1) } </del>

Just got this while running the tests:
let's please move this to /close /sig cluster-lifecycle


Question: // Get returns the election record from a Endpoints Annotation func (el *endpointsLock) Get(ctx context.Context) (*LeaderElectionRecord, []byte, error) { var record LeaderElectionRecord <del> var err error el.e, err = el.Client.Endpoints(el.EndpointsMeta.Namespace).Get(ctx, el.EndpointsMeta.Name, metav1.GetOptions{}) </del> <ins> ep, err := el.Client.Endpoints(el.EndpointsMeta.Namespace).Get(ctx, el.EndpointsMeta.Name, metav1.GetOptions{}) </ins> if err != nil { return nil, nil, err } <ins> el.e = ep </ins> if el.e.Annotations == nil { el.e.Annotations = make(map[string]string) }

The lock acquired by tryAcquireOrRenew is released when the leader ends leadership. However, due to the cancellation of the context, the lock may be set as an empty lock, so the Update cannot be run normally, resulting in a failure to release the lock. The lock can be released every time the leader ends. run this: and then you can got a log like these: and you when you watch the leases at the same time: you can see that was failed to release because of No response <details</details<details</details<details</details<details</details<details</detailsrelated packages: v1.3.0 v0.25.0 v0.25.0 v2.70.1
/sig api-machinery
/triage accepted thanks!
/unassign
/assign
/unassign
/unassign me


Question: "k8s.io/kubernetes/pkg/cloudprovider/providers/gce" "k8s.io/kubernetes/pkg/controller" nodeutil "k8s.io/kubernetes/pkg/controller/util/node" <ins> "k8s.io/kubernetes/pkg/scheduler/algorithm" </ins> utilnode "k8s.io/kubernetes/pkg/util/node" <ins> utiltaints "k8s.io/kubernetes/pkg/util/taints" </ins> ) // cloudCIDRAllocator allocates node CIDRs according to IP address aliases

forked from As from the discussion with and we have to start checking taints wherever we check for node.Spec.Unschedulable. Also overlaps on some parts with /sig node /assign
cc
No sure the discussion history :); personally, prefer to deprecate and replace it with taint.
- same here, the goal would be to deprecate its usage, start checking taints everywhere unschedulable is used. Also, as long as we as we have v1, we might have to keep checking for both.
great, there's same subtask in my mind: deprecated description to the taint for command with taint others ??
- yup, but for 4 it is more something like check taints wherever we checn for unschedulable, since we cannot just replace until v1 is gone.
, my major point is to handle it task by task to make sure we did not miss anything :).
yes we definitely should
- should we define a specific taint for unschedulable (key and effect) ? (i.e what happens when a user uses cordon/uncordon)
I'm not sure about having specific Taint for Unschedulable is a good idea. Pretty much any Taint with NoSchedule effect will do (and this is what we should be checking).
- I agree, but in case of we can add a taint with key or something like that with a effect. This might help us especially if the node has no taints.
, not sure whether we should handle specially; AFAIK, set this flag for node maintenance, this taint should not be tolerated.
The whole point of moving from to Taints is to make things more configurable. as they're decision makers here.
If we are discussing about deprecate/re-implement by Unschedulable taint, +1 as it make sense. Current cordon is causing confusion in community since you can always taint it instead, and tainting is what most Kubernetes setup tool is doing today.
I agree with that we should use NoSchedule taint. I also agree that generally no pod should tolerate NoSchedule taints, but they may want to do so if the desire, knowing that anything could happen to the node they are scheduled on. For example, today DS pods tolerate "Unschedulable" bit.


Question: "k8s.io/kubernetes/pkg/cloudprovider/providers/gce" "k8s.io/kubernetes/pkg/controller" nodeutil "k8s.io/kubernetes/pkg/controller/util/node" <ins> "k8s.io/kubernetes/pkg/scheduler/algorithm" </ins> utilnode "k8s.io/kubernetes/pkg/util/node" <ins> utiltaints "k8s.io/kubernetes/pkg/util/taints" </ins> ) // cloudCIDRAllocator allocates node CIDRs according to IP address aliases

They may as well want to tolerate "NoSchedule" taints.
/sig scheduling
Agree; it's up to 'user' to decide whether tolerates the taint, we just need to taint it and doc what's happened in node for taints :)
We should have a mechanism that allows a cluster admin to limit types of tolerations that users can set. This is needed to prevent abuse in multi-tenant clusters. For example, cluster admin may want to taint nodes with rare resources to prevent pods from certain group of users to land on them. Users shouldn't be able to tolerate such taints. That said, this is a separate topic of its own which is related to policy and should be discussed separately.
- we’re already facing this in production for our spare nodes. I’ll bring this up in our next sig-meeting, I’ll also file an issue for this


Question: _, err := c.Fake.Invokes(action, eviction) return err } <ins> func (c *FakePods) ProxyGet(scheme, name, port, path string, params map[string]string) restclient.ResponseWrapper { return c.Fake.InvokesProxy(core.NewProxyGetAction(podsResource, c.ns, scheme, name, port, path, params)) } </ins>

Currently services/proxy is available in client-go, but pods/proxy resource is not. This requires manual usage via the rest client like This can be seen in Kubernetes e2e tests as well
/kind feature /sig api-machinery


Question: compute "google.golang.org/api/compute/v1" ) <ins> type cidrs struct { ipn   netsets.IPNet isSet bool } var lbSrcRngsFlag cidrs func init() { var err error lbSrcRngsFlag.ipn, err = netsets.ParseIPNets([]string{"130.211.0.0/22", "35.191.0.0/16", "209.85.152.0/22", "209.85.204.0/22", "35.191.0.0/16"}...) if err != nil { panic("Incorrect default GCE L7 source ranges") } flag.Var(&lbSrcRngsFlag, "cloud-provider-gce-lb-src-cidrs", "CIDRS opened in GCE firewall for LB traffic proxy & health checks") } // String is the method to format the flag's value, part of the flag.Value interface. func (c *cidrs) String() string { return strings.Join(c.ipn.StringSlice(), ",") } // Set supports a value of CSV or the flag repeated multiple times func (c *cidrs) Set(value string) error { // On first Set(), clear the original defaults if !c.isSet { c.isSet = true c.ipn = make(netsets.IPNet) } else { return fmt.Errorf("GCE LB CIDRS have already been set") } for _, cidr := range strings.Split(value, ",") { _, ipnet, err := net.ParseCIDR(cidr) if err != nil { return err } c.ipn.Insert(ipnet) } return nil } // LoadBalancerSrcRanges contains the ranges of ips used by the GCE load balancers (l4 & L7) // for proxying client requests and performing health checks. func LoadBalancerSrcRanges() []string { return lbSrcRngsFlag.ipn.StringSlice() } </ins> // GetLoadBalancer is an implementation of LoadBalancer.GetLoadBalancer func (gce *GCECloud) GetLoadBalancer(clusterName string, service *v1.Service) (*v1.LoadBalancerStatus, bool, error) { loadBalancerName := cloudprovider.GetLoadBalancerName(service)

From GCE email today: Current behavior: You are not required to add firewall rules for Network LB health checks today Action required: We are introducing three new ranges for health checking: 209.85.152.0/22, 209.85.204.0/22, and 35.191.0.0/16. You need to ensure that your firewall allows traffic from 209.85.152.0/22, 209.85.204.0/22, and 35.191.0.0/16. The health check probes to your backend instances will come from one of these ranges. We have until Jan 31, 2018 - 1 year.
/assign this issue explains the confusion I had several weeks ago when speaking with and you about L4 health checks. I didn't realize a firewall rule isn't currently required.
/assign
Is this done now?
Not yet, there is a WIP on .
/close


Question: "k8s.io/kubernetes/pkg/kubelet/config" "k8s.io/kubernetes/pkg/volume" volumeutil "k8s.io/kubernetes/pkg/volume/util" <ins> "k8s.io/kubernetes/pkg/volume/util/types" </ins> ) const (

What happened: An iSCSI PV with is mounted without these options. What you expected to happen: The volume is mounted with . Anything else we need to know?: iSCSI volume is mounted in and passes as : This can be fixed trivially, however, why is it mounted in at all? There is another mount attempt in , with correct mount options: This code is skipped, because the volume is already mounted. Environment: Kubernetes version (use ): today's master, v1.19.0-alpha.0.781+-dirty , do you remember why we mount stuff in ? iSCSI and FibreChannel does so, the others don't.
/sig storage


Question: <del> // Copyright 2017 The Kubernetes Authors. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // //     http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. #include <stdio.h> #include <unistd.h> #include <sys/types.h> int main(int argc, char *argv[]){ printf("Effective uid: %dn", geteuid()); return 0; } </del>

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind feature This issue is for creating image as multiarch
/assign
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig testing


Question: KubeProxyVersion:        version.Get().String(), }, Capacity: v1.ResourceList{ <del> v1.ResourceCPU:     *resource.NewMilliQuantity(2000, resource.DecimalSI), v1.ResourceMemory:  *resource.NewQuantity(10E9, resource.BinarySI), v1.ResourceStorage: *resource.NewQuantity(500*mb, resource.BinarySI), v1.ResourcePods:    *resource.NewQuantity(0, resource.DecimalSI), </del> <ins> v1.ResourceCPU:    *resource.NewMilliQuantity(2000, resource.DecimalSI), v1.ResourceMemory: *resource.NewQuantity(10E9, resource.BinarySI), v1.ResourcePods:   *resource.NewQuantity(0, resource.DecimalSI), </ins> }, Allocatable: v1.ResourceList{ <del> v1.ResourceCPU:     *resource.NewMilliQuantity(1800, resource.DecimalSI), v1.ResourceMemory:  *resource.NewQuantity(9900E6, resource.BinarySI), v1.ResourceStorage: *resource.NewQuantity(500*mb, resource.BinarySI), v1.ResourcePods:    *resource.NewQuantity(0, resource.DecimalSI), </del> <ins> v1.ResourceCPU:    *resource.NewMilliQuantity(1800, resource.DecimalSI), v1.ResourceMemory: *resource.NewQuantity(9900E6, resource.BinarySI), v1.ResourcePods:   *resource.NewQuantity(0, resource.DecimalSI), </ins> }, Addresses: []v1.NodeAddress{ {Type: v1.NodeInternalIP, Address: "127.0.0.1"},

Local storage is not expected to be available as a scheduling resource in v1.7. Kubelet is incorrectly exposing a resource named . This is incorrect because the feature is alpha and so it requires opt-in via feature flags and, the resource names are invalid. The expected resource names are listed - and . cc This is v1.7 blocker because we cannot expose a new resource in the API and rename it in a subsequent release.
CC I'll find someone on the storage team to address this.
/assign
GitHub didn't allow me to assign the following users: verult. Note that only can be assigned. <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: import ( "context" <ins> "encoding/json" </ins> "fmt" "strconv" "time"

[X] Test approval issue: [X] Test PR: 19 Oct 2022 [x] Two weeks soak end date: 2 Nov 2022 [x] Test promotion PR: According to this APIsnoop query, there are still three remaining ResourceQuota status endpoints which are untested. patchCoreV1NamespacedResourceQuotaStatusreplaceCoreV1NamespacedResourceQuotaStatusreadCoreV1NamespacedResourceQuotaStatus Using a number of existing e2e test practices a new has been created for 3 ResourceQuota Status endpoints. The e2e logs for this test are listed below. This query shows which Resource Status endpoints are hit within a short period of running this e2e test If a test with these calls gets merged, test coverage will go up by 3 point This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/triage accepted
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: # For systems without journald mkdir -p /var/log/journal <del> # Copy host libsystemd into image to avoid compatibility issues. if [ ! -z "$(ls /host/lib/libsystemd* 2>/dev/null)" ]; then rm /lib/x86_64-linux-gnu/libsystemd* cp -a /host/lib/libsystemd* /lib/x86_64-linux-gnu/ fi </del> /usr/local/bin/fluentd $@

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: [root EFK]# kubectl logs -f fluentd-es-jrsf7 -n kube-system /var/lib/gems/2.1.0/gems/ffi- block in ffilib': Could not open library '': cannot open shared object file: No such file or directory. (LoadError) Could not open library '': cannot open shared object file: No such file or directory. Could not open library 'libsystemd-': cannot open shared object file: No such file or directory. Could not open library 'libsystemd-': libsystemd- cannot open shared object file: No such file or directory from /var/lib/gems/2.1.0/gems/ffi-map' from /var/lib/gems/2.1.0/gems/ffi- ffilib' from /var/lib/gems/2.1.0/gems/systemd-journal-<module:Native' from /var/lib/gems/2.1.0/gems/systemd-journal- <module:Id128' from /var/lib/gems/2.1.0/gems/systemd-journal-<module:Systemd' from /var/lib/gems/2.1.0/gems/systemd-journal- <top (required)' fromrequire' from require' from /var/lib/gems/2.1.0/gems/systemd-journal-<top (required)' from require' fromrequire' from /var/lib/gems/2.1.0/gems/systemd-journal- <top (required)' fromrequire' from rescue in require' fromrequire' from /var/lib/gems/2.1.0/gems/fluent-plugin-systemd- <top (required)' fromrequire' from require' from /var/lib/gems/2.1.0/gems/fluentd-block in tryloadplugin' from /var/lib/gems/2.1.0/gems/fluentd- each' from /var/lib/gems/2.1.0/gems/fluentd-tryloadplugin' from /var/lib/gems/2.1.


Question: # For systems without journald mkdir -p /var/log/journal <del> # Copy host libsystemd into image to avoid compatibility issues. if [ ! -z "$(ls /host/lib/libsystemd* 2>/dev/null)" ]; then rm /lib/x86_64-linux-gnu/libsystemd* cp -a /host/lib/libsystemd* /lib/x86_64-linux-gnu/ fi </del> /usr/local/bin/fluentd $@

0/gems/fluentd- newimpl' from /var/lib/gems/2.1.0/gems/fluentd-newinput' from /var/lib/gems/2.1.0/gems/fluentd- addsource' from /var/lib/gems/2.1.0/gems/fluentd-block in configure' from /var/lib/gems/2.1.0/gems/fluentd- each' from /var/lib/gems/2.1.0/gems/fluentd-configure' from /var/lib/gems/2.1.0/gems/fluentd- configure' from /var/lib/gems/2.1.0/gems/fluentd-runconfigure' from /var/lib/gems/2.1.0/gems/fluentd- runconfigure' from /var/lib/gems/2.1.0/gems/fluentd-block in start' from /var/lib/gems/2.1.0/gems/fluentd- call' from /var/lib/gems/2.1.0/gems/fluentd-mainprocess' from /var/lib/gems/2.1.0/gems/fluentd- start' from /var/lib/gems/2.1.0/gems/fluentd-<top (required)' from require' fromrequire' from /var/lib/gems/2.1.0/gems/fluentd-0.12.39/bin/fluentd:5:in <top (required)' from /usr/local/bin/fluentd:23:inload' from /usr/local/bin/fluentd:23:in <main' What you expected to happen: fluentd-es run success How to reproduce it (as minimally and precisely as possible): kubectl create -f fluentd-es- the image is google-containers/fluentd-elasticsearch:v2.0.0 Anything else we need to know?


Question: # For systems without journald mkdir -p /var/log/journal <del> # Copy host libsystemd into image to avoid compatibility issues. if [ ! -z "$(ls /host/lib/libsystemd* 2>/dev/null)" ]; then rm /lib/x86_64-linux-gnu/libsystemd* cp -a /host/lib/libsystemd* /lib/x86_64-linux-gnu/ fi </del> /usr/local/bin/fluentd $@

: Environment: Kubernetes version (use ):Client Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-07-21T08:23:22Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-07-21T08:08:00Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration**: OS (e.g. from /etc/os-release):centos 7.3 Kernel (e.g. ):Linux node-21.46 3.10.0-514.26.2.el7.x8664 SMP Tue Jul 4 15:04:05 UTC 2017 x8664 x8664 x86_64 GNU/Linux Install tools: Others:
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
Yup, the situation with fluentd on the system with systemd is very sloppy. There's a hack to make it works sometimes, but it seems that it doesn't work. I'll look into it
/sig instrumentation /assign
Facing same issue.


Question: # For systems without journald mkdir -p /var/log/journal <del> # Copy host libsystemd into image to avoid compatibility issues. if [ ! -z "$(ls /host/lib/libsystemd* 2>/dev/null)" ]; then rm /lib/x86_64-linux-gnu/libsystemd* cp -a /host/lib/libsystemd* /lib/x86_64-linux-gnu/ fi </del> /usr/local/bin/fluentd $@

Environment Details:kubectl version Client Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-07-03T15:31:10Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-07-03T15:31:10Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"} OS (e.g. from /etc/os-release):Centos 7 (Core) Kernel (e.g. uname -a): Linux xxxx 3.10.0-327.el7.x8664 Error for reference /var/lib/gems/2.1.0/gems/ffi- block in ffilib': Could not open library '': cannot open shared object file: No such file or directory. (LoadError) Could not open library '': cannot open shared object file: No such file or directory. Could not open library 'libsystemd-': cannot open shared object file: No such file or directory. Could not open library 'libsystemd-': libsystemd- cannot open shared object file: No such file or directory from /var/lib/gems/2.1.0/gems/ffi-map' from /var/lib/gems/2.1.0/gems/ffi- ffilib' from /var/lib/gems/2.1.0/gems/systemd-journal-<module:Native' from /var/lib/gems/2.1.0/gems/systemd-journal- <module:Id128' from /var/lib/gems/2.1.0/gems/systemd-journal-<module:Systemd' from /var/lib/gems/2.1.0/gems/systemd-journal- <top (required)' fromrequire' from require' from /var/lib/gems/2.1.0/gems/systemd-journal-<top (required)' from require' fromrequire' from /var/lib/gems/2.1.


Question: # For systems without journald mkdir -p /var/log/journal <del> # Copy host libsystemd into image to avoid compatibility issues. if [ ! -z "$(ls /host/lib/libsystemd* 2>/dev/null)" ]; then rm /lib/x86_64-linux-gnu/libsystemd* cp -a /host/lib/libsystemd* /lib/x86_64-linux-gnu/ fi </del> /usr/local/bin/fluentd $@

0/gems/systemd-journal- <top (required)' fromrequire' from rescue in require' fromrequire' from /var/lib/gems/2.1.0/gems/fluent-plugin-systemd- <top (required)' fromrequire' from require' from /var/lib/gems/2.1.0/gems/fluentd-block in tryloadplugin' from /var/lib/gems/2.1.0/gems/fluentd- each' from /var/lib/gems/2.1.0/gems/fluentd-tryloadplugin' from /var/lib/gems/2.1.0/gems/fluentd- newimpl' from /var/lib/gems/2.1.0/gems/fluentd-newinput' from /var/lib/gems/2.1.0/gems/fluentd- addsource' from /var/lib/gems/2.1.0/gems/fluentd-block in configure' from /var/lib/gems/2.1.0/gems/fluentd- each' from /var/lib/gems/2.1.0/gems/fluentd-configure' from /var/lib/gems/2.1.0/gems/fluentd- configure' from /var/lib/gems/2.1.0/gems/fluentd-runconfigure' from /var/lib/gems/2.1.0/gems/fluentd- runconfigure' from /var/lib/gems/2.1.0/gems/fluentd-block in start' from /var/lib/gems/2.1.0/gems/fluentd- call' from /var/lib/gems/2.1.0/gems/fluentd-mainprocess' from /var/lib/gems/2.1.0/gems/fluentd- start' from /var/lib/gems/2.1.0/gems/fluentd-<top (required)' from require' fromrequire' from /var/lib/gems/2.1.0/gems/fluentd-0.12.39/bin/fluentd:5:in <top (required)' from /usr/local/bin/fluentd:23:inload' from /usr/local/bin/fluentd:23:in <


Question: # For systems without journald mkdir -p /var/log/journal <del> # Copy host libsystemd into image to avoid compatibility issues. if [ ! -z "$(ls /host/lib/libsystemd* 2>/dev/null)" ]; then rm /lib/x86_64-linux-gnu/libsystemd* cp -a /host/lib/libsystemd* /lib/x86_64-linux-gnu/ fi </del> /usr/local/bin/fluentd $@

main'
You can solve it temporary by removing I'm working on the solution
A solution here might be to base fluentd-es off an image with journald supporting XZ and LZ4 compression. I'm looking :)
Thanks


Question: if err != nil { return nil, fmt.Errorf("failed to fetch cadvisor stats: %v", err) } <del> caInfos := getCRICadvisorStats(allInfos) </del> <ins> caInfos, allInfos := getCRICadvisorStats(allInfos) </ins> for _, stats := range resp { containerID := stats.Attributes.Id

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --For single node with hundreds and thousands of pods and containers, the cpu usage of kubelet becomes high, one bottleneck is the cadvisor stats provider as below: the method iterate all containerInfos for every podStats cause high cpu the method should not iterate allcontainerInfos for every podStats, use map instead Kubernetes version (use ): v1.19.4 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig node
/triage accepted
/assign
I’ve done an analysis on the code and this is what I have found, I believe we have two issues to address. There are multiple range loops being performed over v2.ContainerInfo / PodStat maps within the ListPodStatus method which can be contributing to the high cpu utilisation. The allInfos argument which is supplied to the getCadvisorContainerInfo function is a map which uses a cgroup path as the key, in order to find matching container info to podUID, processing of the keys in the map needs to be performed until a match is found. Examples of keys are as follows: /kubepods/burstable/podf9e3a173-1470-451a-9ca2-/d8426b82582fbcfae16085bc794c49769d3a9648bd5463976aca4071f9342754 /docker/2dfa17fc81e781212bcc26e189ebb724721fe097394b4de3cbeb7e069bf23290 My recommendation would be to apply the following updates: Refactor the getCadvisorContainerInfo function to allow for a filter and key type parameter to be supplied by the caller, this enables reducing the number of range loops being performed over the container info map by removing elements that are not required and the ability to set the key being used in the returned container infos map to enable fast lookup of container info.


Question: if err != nil { return nil, fmt.Errorf("failed to fetch cadvisor stats: %v", err) } <del> caInfos := getCRICadvisorStats(allInfos) </del> <ins> caInfos, allInfos := getCRICadvisorStats(allInfos) </ins> for _, stats := range resp { containerID := stats.Attributes.Id

My initial thoughts are to use an enumeration which has two values KEYCGROUPPATH to provide backwards compatibility and KEYPODUID to enable fast lookups via podUID Refactor ListPodStats method to be more efficient in constructing the PodStats slice and taking into account the fast lookup via podUID I think this is the best approach instead of pushing the data structure implementation for the container info upstream to the cadviser clients, this will at least initially provide a better optimisation scoped within the kubelet/stats package and reduce other possible impacts to changing the cadvisor clients Please advise.
Just realized this looks like a duplicate issue and looking as fixed by PR Both this issue and were raised by Zac
/unassign
/close
You can't close an active issue/PR unless you authored it or you are a collaborator. <detailsIn response to : /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Didn't think I could close this, leaving this with
this will be automatically closed once the PR merges


Question: fs.StringVar(&s.EncryptionProviderConfigFilepath, "experimental-encryption-provider-config", s.EncryptionProviderConfigFilepath, "The file containing configuration for encryption providers to be used for storing secrets in etcd") <ins> fs.MarkDeprecated("experimental-encryption-provider-config", "use --encryption-provider-config.") fs.StringVar(&s.EncryptionProviderConfigFilepath, "encryption-provider-config", s.EncryptionProviderConfigFilepath, "The file containing configuration for encryption providers to be used for storing secrets in etcd") </ins> fs.DurationVar(&s.StorageConfig.CompactionInterval, "etcd-compaction-interval", s.StorageConfig.CompactionInterval, "The interval of compaction requests. If 0, the compaction request from apiserver is disabled.")

Ref Ref ref /sig auth /kind feature /milestone v1.11 /priority important-longterm /status approved-for-milestone cc
What's need to be done here? I can think only about renaming to and updating the docs. What else?
What's required to promote encryption at rest with static configs to beta?
an example of promoting flags is here: If there's a feature gate, we're going to want to enable it by default too.
/assign
submitted PRs for that: and
[MILESTONENOTIFIER] Milestone Removed From Issue Important: Code freeze is in effect and only issues with may remain in the v1.11 milestone. <details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
Will this make it to Beta in 1.12? From sig-auth on 7/11, it sounded like this was missing a release shepherd. Is anyone owning pushing this forward?
I'll do my best to push this. edit: I did not realize that 1.12 feature freeze is today (or so I heard), I don't think I'll be able to make it :slightlyfrowningface:
Ack - that's what I was wondering (with the feature freeze). Thanks
- Is this making it to the 1.12 ?
The current plan is 1.13


Question: metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/runtime" "k8s.io/apimachinery/pkg/util/intstr" <ins> "k8s.io/apimachinery/pkg/util/sets" </ins> "k8s.io/apimachinery/pkg/util/wait" "k8s.io/client-go/informers" clientset "k8s.io/client-go/kubernetes"

/kind bug What happened: On a large cluster with enough pods, endpoints take as long as 10 minutes to show up after their associated pods and service are created. We believe that this is because every pod change results in an update of the addresses within the endpoint since they refer to a pod via a whose changes every time a pod change is observed. All these non-update updates eventually overwhelm the endpoint controller's queue rate limit and the queue starts to grow. While an increase in the queue's rate limit can alleviate the problem, an admin can keep a much lower limit if the non-update updates are dropped rather than processed. What you expected to happen: When a pod related to an endpoint is updated, I would expect the endpoint to update only if there is a change in the pod that relates to the endpoint, such as a change in the pod's IP address. How to reproduce it (as minimally and precisely as possible): a pod and associated service: the endpoint for changes: the watch running, and in another session, edit a pod to add a label: Then add another label on the line after the label, such as that the watched endpoint changed despite there not being any meaningful change to the fields the endpoint cares about. Anything else we need to know?: The PR addresses this issue. Environment: Kubernetes version (use ):
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig scalability
what ever happened with the effort to move leases off of Endpoints?


Question: healthchecker.mutationRequestChannel <- req } <del> func updateServiceListener(serviceName types.NamespacedName, listenPort int, addOrDelete bool) bool { </del> <ins> func updateServiceListener(serviceName types.NamespacedName, listenPort int, add bool) bool { </ins> responseChannel := make(chan bool) req := &proxyListenerRequest{ serviceName:     serviceName, listenPort:      uint16(listenPort), <del> add:             addOrDelete, </del> <ins> add:             add, </ins> responseChannel: responseChannel, } healthchecker.listenerRequestChannel <- req

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? Currently, the 4th parameter name of updateServiceListener in kube-proxy is addOrDelete, However, from the func caller, we just pass for AddServiceListener, and for DeleteServiceListener, So, I think the name addOrDelete is unreasonable. I prefer to change the parameter name addOrDelete to add so that avoid making people confused. In addition, there is a typo, We should change addition to deletion! Also, there is a sentence breaking, we should change from No Service Endpoints Not Found =No Service Endpoints are found Thanks!
Fixed!


Question: import ( "fmt" <del> netutils "k8s.io/utils/net" </del> "reflect" "testing" <ins> netutils "k8s.io/utils/net" </ins> utilipvs "k8s.io/kubernetes/pkg/util/ipvs" utilipvstest "k8s.io/kubernetes/pkg/util/ipvs/testing" )

pull-kubernetes-unit ~1 day since merging of Test introduced in /sig network
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc
ahhhg let me fix it
oh, I will take a look.


Question: _, client, closeFn := setup(t) defer closeFn() <del> _, err := client.CoreV1().RESTClient().Patch(types.ApplyPatchType). Namespace("default"). Resource("pods"). Name("test-pod"). Body([]byte(`{ "apiVersion": "v1", "kind": "Pod", "metadata": { "name": "test-pod" }, "spec": { "containers": [{ "name":  "test-container", "image": "test-image" }] } }`)). Do(). Get() if err != nil { t.Fatalf("Failed to create object using Apply patch: %v", err) </del> <ins> testCases := []struct { resource string name     string body     string }{ { resource: "pods", name:     "test-pod", body: `{ "apiVersion": "v1", "kind": "Pod", "metadata": { "name": "test-pod" }, "spec": { "containers": [{ "name":  "test-container", "image": "test-image" }] } }`, }, { resource: "services", name:     "test-svc", body: `{ "apiVersion": "v1", "kind": "Service", "metadata": { "name": "test-svc" }, "spec": { "ports": [{ "port": 8080, "protocol": "UDP" }] } }`, }, </ins> } <del> _, err = client.CoreV1().RESTClient().Get().Namespace("default").Resource("pods").Name("test-pod").Do().Get() if err != nil { t.Fatalf("Failed to retrieve object: %v", err) </del> <ins> for _, tc := range testCases { _, err := client.CoreV1().RESTClient().Patch(types.ApplyPatchType). Namespace("default"). Resource(tc.resource). Name(tc.name). Body([]byte(tc.body)). Do(). Get() if err != nil { t.Fatalf("Failed to create object using Apply patch: %v", err) } _, err = client.CoreV1().RESTClient().Get().Namespace("default").Resource(tc.resource).Name(tc.name).Do().Get() if err != nil { t.Fatalf("Failed to retrieve object: %v", err) } </ins> } }

What happened: When running the apiserver returns an error: with being: What you expected to happen: The object being created if not found. How to reproduce it (as minimally and precisely as possible): get etcd up get apiserver up run kubectl apply (built from master as well) Anything else we need to know?: Everything built directly from master Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
I think this is caused by We do need to fix that before alpha is released. The issue is that services use a custom storage implementation which doesn't support create-on-update and the first line of the update function checks that the object exists.
Okay, thanks for looking into it . Anything I can do here?


Question: func findProbe(probes []network.Probe, probe network.Probe) bool { for _, existingProbe := range probes { <del> if strings.EqualFold(*existingProbe.Name, *probe.Name) { </del> <ins> if strings.EqualFold(*existingProbe.Name, *probe.Name) && *existingProbe.Port == *probe.Port { </ins> return true } }

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug /area cloudprovider /sig azure What happened: I've created a service of type in a Kubernetes cluster created via the ACS Engine. The load balancer has been created correctly after some time and a random NodePort (at this time ) has been allocated. However, in the Azure Portal I'm seeing that the health probe of that load balancer is pointing to port which does not exist in the whole Kubernetes system at all. Because of that wrongly configured health probe the load balancer is not reachable (i/o timeout). What you expected to happen: The health probe would have been configured to NodePort . How to reproduce it (as minimally and precisely as possible): Unfortunately, it does not behave always like this but only occasionally. In some cases the health probe gets configured correctly, but in others not. I've used the following manifest: Anything else we need to know?: When updating the health probe manually to the correct NodePort in the Azure portal, the load balancer starts to work properly. Environment: Kubernetes version (use ): 1.7.5 Cloud provider or hardware configuration: Azure Install tools: ACS Engine Others: -
Thanks for reporting the problem. Could you share the service spec after it is created? e.g. by running . And may I ask whether there is a service just removed with same name before this one?
Sure, thanks, please find the manifest below [1]. In fact I'll have some other services with the same name, but in other namespaces. In the namespace I didn't have a service named before. [1] Service manifest after creation
Filed PR , which should fix the problem. Need cherry-picked to v1.7 branch after it is merged.
Thank you very much that was really fast!
has been merged. created cherry pick PR for release 1.7.


Question: "runtime" "sort" "strings" <ins> "sync" </ins> "time" "github.com/golang/glog"

There is a defer calling in this method: defer close(resultCh) It may crash the goroutine created at line 777, which then will make the whole kubelet crash. A simple fix Another simple fix
cc do you think this make sense?
You could just drop the channel and go with a mutex to update the set.
indeed.