Question: <ins> // +build windows /* Copyright 2018 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package validation import ( "fmt" utilerrors "k8s.io/apimachinery/pkg/util/errors" "k8s.io/kubernetes/pkg/kubelet/apis/kubeletconfig" ) // validateKubeletOSConfiguration validates os specific kubelet configuration and returns an error if it is invalid. func validateKubeletOSConfiguration(kc *kubeletconfig.KubeletConfiguration) error { message := "invalid configuration: %v (%v) %v is not supported on Windows" allErrors := []error{} if kc.CgroupsPerQOS { allErrors = append(allErrors, fmt.Errorf(message, "CgroupsPerQOS", "--cgroups-per-qos", kc.CgroupsPerQOS)) } if len(kc.EnforceNodeAllocatable) > 0 { allErrors = append(allErrors, fmt.Errorf(message, "EnforceNodeAllocatable", "--enforce-node-allocatable", kc.EnforceNodeAllocatable)) } return utilerrors.NewAggregate(allErrors) } </ins>

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: Pods stuck in terminating state on Windows node after deleting it What you expected to happen: Pods get terminated and should disappear from kubectl How to reproduce it (as minimally and precisely as possible): Run a pod with windows container Delete it Pods are still terminating Anything else we need to know?: Could you pls help to resolve this issue. K8s pods stuck as Terminating after getting deleted on windows node with windows container(IIS). Docker ps -a not showing this container, so it is completely deleted, but in kubectl it is marked as Terminating. For networking we using OVN. kubectl logs: kubelet logs: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: Hyper-V VMs OS (e.g. from /etc/os-release): Windows Node: Master Node: NAME="Ubuntu" VERSION="16.04.4 LTS (Xenial Xerus)" ID=ubuntu IDLIKE=debian PRETTYNAME="Ubuntu 16.04.4 LTS" VERSIONID="16.04" HOMEURL="" SUPPORTURL="" BUGREPORTURL="" VERSIONCODENAME=xenial UBUNTUCODENAME=xenial Kernel (e.g. ): Linux esd-kbrnts-lnx-1 4.13.0-37-generic ~16.04.1-Ubuntu SMP Wed Mar 7 16:03:28 UTC 2018 x8664 x8664 x8664 GNU/Linux Install tools: OVN 2.7 on windows OVN 2.7.2 on linux Others: Docker Version on Windows node: Server Version: 17.06.2-ee-6
/sig windows
It needs sometime for kubelet to remove the pod. Could you wait a while and check whether the Pod removed?
Hi actually I have waited for 1 day, and it is not deleting. Can delete it only using --grace-period=0 --force. But I don't want to use this manual approach). Could you pls give some advice on this? Thanks in advance!
is a reference to this issue. Checking kubelet logs will always help to troubleshoot such problem.


Question: <ins> // +build windows /* Copyright 2018 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package validation import ( "fmt" utilerrors "k8s.io/apimachinery/pkg/util/errors" "k8s.io/kubernetes/pkg/kubelet/apis/kubeletconfig" ) // validateKubeletOSConfiguration validates os specific kubelet configuration and returns an error if it is invalid. func validateKubeletOSConfiguration(kc *kubeletconfig.KubeletConfiguration) error { message := "invalid configuration: %v (%v) %v is not supported on Windows" allErrors := []error{} if kc.CgroupsPerQOS { allErrors = append(allErrors, fmt.Errorf(message, "CgroupsPerQOS", "--cgroups-per-qos", kc.CgroupsPerQOS)) } if len(kc.EnforceNodeAllocatable) > 0 { allErrors = append(allErrors, fmt.Errorf(message, "EnforceNodeAllocatable", "--enforce-node-allocatable", kc.EnforceNodeAllocatable)) } return utilerrors.NewAggregate(allErrors) } </ins>

Actually, I have attached kubelet log in the ticket and think the main reason is Also as there any option for kubelet to automatically force delete Terminating pods after some period? And --containerized is also not solving my issue. Thanks!
Kubelet won't delete the pod from apiserver. Since it is already not exist on the node, you could force delete it by kubectl.
It is clear, so how we can resolve this error: ? I think this is the core item of our issue. Thanks.
I'm not using OVN. Have you met this problem when using OVN?
adding Bob to comment on this. i have seen this issue before, but it has been quite a while.
Hi, Guys will be very grateful if you will provide some solution for it. We have tried to add to deployment spec but it doesn't help. Thanks.
The error message of is actually a warning message. It shouldn't block the removal of pods. Could you recheck the following things: the node, check whether the pod is still existing, e.g. by the pod exists, check if there are other error or warning message in kubelet logs; or else, the pod could be force removed safely from apiserver
Hi, I had posted this info in the ticket description. not showing this container, so it is completely deleted, but in kubectl it is displayed as Terminating.
i think you may get more support and faster responses in our slack channel for sig-windows.
So it could be safely force removed from API. And agreed with slack is a more faster way to such troubleshooting issues.
The issue seems to be related to . The only platform specific implementation for PodContainerManager is the one for . There is a default implementation (), that always returns true in the Exists call. Disabling the CgroupsPerQOS option on the Windows nodes resolves the issue.
Windows doesn't support cgroups. You should always disable it.
it is clear enough that Windows does not support cgroups). Unfortunately, it's not specified in the documentation. So to disable CgroupsPerQOS you should need to add this flags to kubelet: Maybe it will help someone... Thank you all for discussion!


Question: // // In general, in order to preserve a well-defined interface between the // kubelet and the container runtime, annotations SHOULD NOT influence <del> // runtime behaviour. For legacy reasons, there are some annotations which // currently explicitly break this rule, listed below; in future versions // of the interface these will be promoted to typed features. </del> <ins> // runtime behaviour. </ins> // // Annotations can also be useful for runtime authors to experiment with // new features that are opaque to the Kubernetes APIs (both user-facing // and the CRI). Whenever possible, however, runtime authors SHOULD // consider proposing new typed fields for any new features instead. <del> // // 1. Seccomp // //      key: security.alpha.kubernetes.io/seccomp/pod //      description: the seccomp profile for the containers of an entire pod. //      value: see below. // //      key: security.alpha.kubernetes.io/seccomp/container/<container name> //      description: the seccomp profile for the container (overrides pod). //      value: see below // //      The value of seccomp is runtime agnostic: //      * runtime/default: the default profile for the container runtime //      * unconfined: unconfined profile, ie, no seccomp sandboxing //      * localhost/<profile-name>: the profile installed to the node's //        local seccomp profile root. Note that profile root is set in //        kubelet, and it is not passed in CRI yet, see https://issues.k8s.io/36997. // </del> Annotations map[string]string `protobuf:"bytes,7,rep,name=annotations" json:"annotations,omitempty" protobuf_key:"bytes,1,opt,name=key,proto3" protobuf_val:"bytes,2,opt,name=value,proto3"` // Optional configurations specific to Linux hosts. Linux *LinuxPodSandboxConfig `protobuf:"bytes,8,opt,name=linux" json:"linux,omitempty"`

CRI supports AppArmor, Seccomp and Sysctls via annotations , we should move them to typed and structured fields instead. (refer ) [x] Apparmor () [x] Sysctls () [x] Seccomp () cc
/cc
/sig node
Seccomp is addressed at .


Question: // // In general, in order to preserve a well-defined interface between the // kubelet and the container runtime, annotations SHOULD NOT influence <del> // runtime behaviour. For legacy reasons, there are some annotations which // currently explicitly break this rule, listed below; in future versions // of the interface these will be promoted to typed features. </del> <ins> // runtime behaviour. </ins> // // Annotations can also be useful for runtime authors to experiment with // new features that are opaque to the Kubernetes APIs (both user-facing // and the CRI). Whenever possible, however, runtime authors SHOULD // consider proposing new typed fields for any new features instead. <del> // // 1. Seccomp // //      key: security.alpha.kubernetes.io/seccomp/pod //      description: the seccomp profile for the containers of an entire pod. //      value: see below. // //      key: security.alpha.kubernetes.io/seccomp/container/<container name> //      description: the seccomp profile for the container (overrides pod). //      value: see below // //      The value of seccomp is runtime agnostic: //      * runtime/default: the default profile for the container runtime //      * unconfined: unconfined profile, ie, no seccomp sandboxing //      * localhost/<profile-name>: the profile installed to the node's //        local seccomp profile root. Note that profile root is set in //        kubelet, and it is not passed in CRI yet, see https://issues.k8s.io/36997. // </del> Annotations map[string]string `protobuf:"bytes,7,rep,name=annotations" json:"annotations,omitempty" protobuf_key:"bytes,1,opt,name=key,proto3" protobuf_val:"bytes,2,opt,name=value,proto3"` // Optional configurations specific to Linux hosts. Linux *LinuxPodSandboxConfig `protobuf:"bytes,8,opt,name=linux" json:"linux,omitempty"`

Seccomp profile is passed to runtime by in CRI: is relative to node's local seccomp profile root, it is defined in kubelet. But runtime doesn't know it. We should pass full profile path in CRI. cc/ Will file a PR to fix this later.
Add support for Seccomp sounds like a new feature. The milestone should be reset to , correct?
not really. The current implementation supports seccomp already. If it's missing critical information to function properly, this would be a bug.
are we blocking releases for CRI in 1.5? i was not sure, so marking this as non-release-blocker and leaving 1.5 milestone. please correct if i am mistaken.
I don't get why this is a 1.5 targetted thing myself. Yes we want CRI to be in a good state in 1.5, but afaik we have zero intention of anyone actually using (at the very least because of the lack of exec auth regressing kubelet security). why should this be part of 1.5?
First, this is a small enough change that allows non-docker integrations to properly support seccomp. Second, if "we have zero intention of anyone actually using" is a valid reason, we should never fix any bugs in CRI. We do want people to try using it (perhaps not in production yet). However, given that seccomp is only in alpha, I am ok with dropping this and just continue the docker support for now.
Perhaps I misunderstood the meaning of code freeze, but I thought it was only bug-fixes regardless of size of change (or exceptions that have gone through some process) What I meant is we don't intend this to be used generally as part of the 1.5 release. Because of that, I don't think we should consider bugs in CRI bugs for the purpose of the 1.5 release freeze. My statement is just that we shouldn't violate code-freeze for this. We should totally fix this right after the 1.5 freeze window ends, and saying that I'm arguing we should never fix anything is disingenuous and taking it to an absurd extreme.
the only thing we don't seem to agree on is whether seccomp support was considered to be supported for CRI in 1.5.


Question: // // In general, in order to preserve a well-defined interface between the // kubelet and the container runtime, annotations SHOULD NOT influence <del> // runtime behaviour. For legacy reasons, there are some annotations which // currently explicitly break this rule, listed below; in future versions // of the interface these will be promoted to typed features. </del> <ins> // runtime behaviour. </ins> // // Annotations can also be useful for runtime authors to experiment with // new features that are opaque to the Kubernetes APIs (both user-facing // and the CRI). Whenever possible, however, runtime authors SHOULD // consider proposing new typed fields for any new features instead. <del> // // 1. Seccomp // //      key: security.alpha.kubernetes.io/seccomp/pod //      description: the seccomp profile for the containers of an entire pod. //      value: see below. // //      key: security.alpha.kubernetes.io/seccomp/container/<container name> //      description: the seccomp profile for the container (overrides pod). //      value: see below // //      The value of seccomp is runtime agnostic: //      * runtime/default: the default profile for the container runtime //      * unconfined: unconfined profile, ie, no seccomp sandboxing //      * localhost/<profile-name>: the profile installed to the node's //        local seccomp profile root. Note that profile root is set in //        kubelet, and it is not passed in CRI yet, see https://issues.k8s.io/36997. // </del> Annotations map[string]string `protobuf:"bytes,7,rep,name=annotations" json:"annotations,omitempty" protobuf_key:"bytes,1,opt,name=key,proto3" protobuf_val:"bytes,2,opt,name=value,proto3"` // Optional configurations specific to Linux hosts. Linux *LinuxPodSandboxConfig `protobuf:"bytes,8,opt,name=linux" json:"linux,omitempty"`

My interpretation is that we assumed seccomp would be supported until we found out that it wouldn't for other runtimes due to the lack of profile path. That's the only reason why I considered this a bug. Of course we could say that this is not really an important feature in CRI, so we can drop it, but we can be more clear on that.
If there's a bug in a feature no one uses, does fixing it make a sound? I think what we're not agreeing on is the role of CRI in 1.5. I understood it as something that is not generally usable. We can't recommend a customer use it, even as an experimental feature, due to security aspects (if nothing else). Correct me if I'm wrong. It's a developer integration point, and developers integrating with it are doing so with the kubernetes master branch, not with some release. Those reasons together result in my opinion that CRI is still not really a customer/release feature, and thus bugs or features related to it should not be exceptions to the freeze window, but rather should targeted at master as soon as the freeze lifts. If this were already post-freeze, would we be cherry-picking this over to 1.5?
This is not true. cri-o is depending on this fix for apparmor support.
is it depending on the 1.5 release branch directly, or on master?
I think it's better to make this in 1.5, so cri-o could work with first release with experimental CRI. It helps to encourage people to use CRI and find features misses or bugs.
Modifying the user-specified annotation is a little bit unsettling. I think adding a new annotation explicitly for the path would be better. Or, we should just add an explicit field for this. I spoke to and separately. I still think this is a bug fix, but we may want to be more careful about the fix. Given the time constraint, let's just list this as a known issue and work on it for 1.6.
OK, let's move this to next release and think carefully about seccomp feature besides this: kubelet may just pass the profile as string via CRI; may assume the profile locate in a file in a known format and pass path to the runtime; or maybe some other solutions.


Question: // // In general, in order to preserve a well-defined interface between the // kubelet and the container runtime, annotations SHOULD NOT influence <del> // runtime behaviour. For legacy reasons, there are some annotations which // currently explicitly break this rule, listed below; in future versions // of the interface these will be promoted to typed features. </del> <ins> // runtime behaviour. </ins> // // Annotations can also be useful for runtime authors to experiment with // new features that are opaque to the Kubernetes APIs (both user-facing // and the CRI). Whenever possible, however, runtime authors SHOULD // consider proposing new typed fields for any new features instead. <del> // // 1. Seccomp // //      key: security.alpha.kubernetes.io/seccomp/pod //      description: the seccomp profile for the containers of an entire pod. //      value: see below. // //      key: security.alpha.kubernetes.io/seccomp/container/<container name> //      description: the seccomp profile for the container (overrides pod). //      value: see below // //      The value of seccomp is runtime agnostic: //      * runtime/default: the default profile for the container runtime //      * unconfined: unconfined profile, ie, no seccomp sandboxing //      * localhost/<profile-name>: the profile installed to the node's //        local seccomp profile root. Note that profile root is set in //        kubelet, and it is not passed in CRI yet, see https://issues.k8s.io/36997. // </del> Annotations map[string]string `protobuf:"bytes,7,rep,name=annotations" json:"annotations,omitempty" protobuf_key:"bytes,1,opt,name=key,proto3" protobuf_val:"bytes,2,opt,name=value,proto3"` // Optional configurations specific to Linux hosts. Linux *LinuxPodSandboxConfig `protobuf:"bytes,8,opt,name=linux" json:"linux,omitempty"`

We need to clearly define it. define profile format clearly
Please document known issues for 1.5 in
(copied from ) And let's move here to discuss how to pass profiles to runtime. Related with .
This is issue is related with two level APIs: CRI, since we may also want to support profiles from ConfigMaps, I think we should pass full profile contents instead of path to runtime. issues is also related with seccomp api in kube-apiserver, it is based on annotations today. We should define seccomp profile format clearly (see ) and decide whether to add struct types to pod spec or just continue the annotations way. I think it's more clear to add specific struct types to pod spec.
Agreed - the path stuff is really a crutch (and implementation-specific) - I think is the answer here.
I don't disagree, but it's not strictly a requirement for supporting ConfigMaps (kubelet could create a file with the contents). An open question is whether to pass it as a raw string, or wait for and pass the structured data. The annotations way is deprecated (). I think the bigger question is where/how to put the data. E.g. structured data in the API, unstructured (string type) data in the API, reference to another object (e.g. ConfigMap). If the Kubelet needs to parse the spec, I think that's a good argument for a structured data type (either a standalone object or embedded in the pod API).
Yes, already opened to replace annotations with typed structs. I personally prefer a structured data type in both kubernetes API and CRI. WDYT?
A counter argument is that as the spec grows or new features are , we need to keep updating the Kubernetes API to maintain parity, or update the code in each CRI implementation to translate the Kubernetes spec. OTOH, if we use unstructured data we can use 3rd party libraries to parse out the data we need (assuming the spec evolves in a backwards compatible way).
What spec?
Whichever seccomp profile spec is being implemented by runtimes. E.g. if we decide to base the Kubernetes seccomp profile API off the OCI seccomp profile spec, and OCI adds some new features, at some point we'd probably want to add those features to Kubernetes.


Question: // // In general, in order to preserve a well-defined interface between the // kubelet and the container runtime, annotations SHOULD NOT influence <del> // runtime behaviour. For legacy reasons, there are some annotations which // currently explicitly break this rule, listed below; in future versions // of the interface these will be promoted to typed features. </del> <ins> // runtime behaviour. </ins> // // Annotations can also be useful for runtime authors to experiment with // new features that are opaque to the Kubernetes APIs (both user-facing // and the CRI). Whenever possible, however, runtime authors SHOULD // consider proposing new typed fields for any new features instead. <del> // // 1. Seccomp // //      key: security.alpha.kubernetes.io/seccomp/pod //      description: the seccomp profile for the containers of an entire pod. //      value: see below. // //      key: security.alpha.kubernetes.io/seccomp/container/<container name> //      description: the seccomp profile for the container (overrides pod). //      value: see below // //      The value of seccomp is runtime agnostic: //      * runtime/default: the default profile for the container runtime //      * unconfined: unconfined profile, ie, no seccomp sandboxing //      * localhost/<profile-name>: the profile installed to the node's //        local seccomp profile root. Note that profile root is set in //        kubelet, and it is not passed in CRI yet, see https://issues.k8s.io/36997. // </del> Annotations map[string]string `protobuf:"bytes,7,rep,name=annotations" json:"annotations,omitempty" protobuf_key:"bytes,1,opt,name=key,proto3" protobuf_val:"bytes,2,opt,name=value,proto3"` // Optional configurations specific to Linux hosts. Linux *LinuxPodSandboxConfig `protobuf:"bytes,8,opt,name=linux" json:"linux,omitempty"`

It also means that if different runtimes support different feature sets, we'll be stuck either tracking only the common features, or specifying features that may be ignored by the runtime. This may be a non-issue if the seccomp feature set rarely changes.
Well, yes, but that applies to everything in the CRI, and is one of the tradeoffs we ma[d|k]e by introducing this abstraction in the first place.
My $0.02: Moving the existing logic/information for seccomp into a first-class field for CRI (as in ) is productive because of the known problems with the annotation approach. My initial proposal for seccomp support included a resource similar to what has proposed in kubernetes/community and we were unable to come to an agreement on the issue refers to here: I do not necessarily expect that we will be able to come to an agreement about the seccomp resource in time for 1.7, so I think we should proceed with and do backward compatible CRI changes later if we need to inject the content of the seccomp profile instead of the path. I think we would deeply regret embedding API surface to describe seccomp profiles in the Pod API :)


Question: assetfs "github.com/elazarl/go-bindata-assetfs" ) <ins> const dashboardPath = "/static/app/#/dashboard/" </ins> type MuxInterface interface { Handle(pattern string, handler http.Handler) <ins> HandleFunc(pattern string, handler func(http.ResponseWriter, *http.Request)) </ins> } func InstallSupport(mux MuxInterface, enableSwaggerSupport bool) {

I follow the instruction to start the kubernetes web UI. It's 404 when I open http://127.0.0.1:8080 . Finally I knew that the correct directory is http://127.0.0.1:8080/static/app/#/dashboard/ . We should start the web ui in root directory(I mean http://127.0.0.1:8080) so that anyone like me will not be confused.
/sub
I agree the UI isn't very discoverable. We should redirect from to the correct path. (Assuming that the UI is working well enough to expose it at all.)
I can do this if no one else is on it. Should we redirect and for all requests or just requests with specific user-agents or MIME types?
We return a list of resource paths on currently: We could either redirect for certain user-agents, or choose another, simpler, more obvious, top-level path, like or , which would at least show up in the list of valid paths.
That's great!
Great job :+1:


Question: // This flag sets a maximum number of images to report in the node status. nodeStatusMaxImages int32 <ins> //  This flag indicates that kubelet should start plugin watcher utility server for discovering Kubelet plugins enablePluginsWatcher bool </ins> } func allGlobalUnicastIPs() ([]net.IP, error) {


Question: if err != nil { return "", err } <del> // Simulate the global mount so that the fakeMounter returns the // expected number of mounts for the attached disk. b.mounter.Mount(globalPath, globalPath, b.fsType, nil) </del> fake.attachCalled = true return "", nil }

What happened: An iSCSI PV with is mounted without these options. What you expected to happen: The volume is mounted with . Anything else we need to know?: iSCSI volume is mounted in and passes as : This can be fixed trivially, however, why is it mounted in at all? There is another mount attempt in , with correct mount options: This code is skipped, because the volume is already mounted. Environment: Kubernetes version (use ): today's master, v1.19.0-alpha.0.781+-dirty , do you remember why we mount stuff in ? iSCSI and FibreChannel does so, the others don't.
/sig storage


Question: for format := range templateFormats { formats = append(formats, format) } <ins> sort.Strings(formats) </ins> return formats }

/kind bug What happened: When building kubernetes-1.7.0 packages for openSUSE, results differ for each build in and other man pages. coming from that uses and probably sets do not have any specific order, so it would need to be sorted on output. What you expected to happen: Every build should give identical results. See https://reproducible- for why this matters. How to reproduce it (as minimally and precisely as possible): See Anything else we need to know?: Environment: Kubernetes version: 1.7.0 OS (e.g. from /etc/os-release): openSUSE Tumbleweed
/sig release
somehow this bug seems to have been forgotten - can you help getting the right people to have a look?
- can you please confirm that PR works for you?
I tested it and it looks good.
/assign I enjoy reading my highlights to find the issue has already been addressed. :)
Sorry this wasn't addressed sooner. It does seem like it fell into a black hole, and pinging was the right way to resolve that. :)
/area build-release
related to /reopen
Reopening this issue. <detailsIn response to : related to Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: readOnly: true # BEGIN_PROMETHEUS_TO_SD - name: prom-to-sd <del> image: gcr.io/google-containers/prometheus-to-sd:v0.2.1 </del> <ins> image: gcr.io/google-containers/prometheus-to-sd:v0.2.2 </ins> command: - /monitor - --source=heapster:http://localhost:8082?whitelisted=stackdriver_requests_count,stackdriver_timeseries_count

Currently there are several, and some of them have problems with excessive logging could you please do this?
Any estimate on this fix? Asking because I'm seeing the issue referenced in
It was already fixed in head and in 1.8 long time ago. Fix was merged to 1.7 recently, so we are just waiting for new patch release.


Question: {kubeLoadBalancerSourceCIDRSet, string(KubeFireWallChain), "RETURN", "dst,dst,src", ""}, {kubeLoadBalancerSourceIPSet, string(KubeFireWallChain), "RETURN", "dst,dst,src", ""}, {kubeLoadBalancerLocalSet, string(KubeLoadBalancerChain), "RETURN", "dst,dst", ""}, <del> {kubeNodePortSetTCP, string(kubeServicesChain), string(KubeNodePortChain), "dst", "tcp"}, </del> {kubeNodePortLocalSetTCP, string(KubeNodePortChain), "RETURN", "dst", "tcp"}, <del> {kubeNodePortSetUDP, string(kubeServicesChain), string(KubeNodePortChain), "dst", "udp"}, </del> <ins> {kubeNodePortSetTCP, string(KubeNodePortChain), string(KubeMarkMasqChain), "dst", "tcp"}, </ins> {kubeNodePortLocalSetUDP, string(KubeNodePortChain), "RETURN", "dst", "udp"}, <ins> {kubeNodePortSetUDP, string(KubeNodePortChain), string(KubeMarkMasqChain), "dst", "udp"}, </ins> } var ipvsModules = []string{

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug /assign I'll make the pull request tomorrow. What happened: start kube-proxy in ipvs mode, create some NodePort services. Then look at the iptables rules. match-set of KUBE-NODE-PORT-TCP is inadequate to be qualified as NodePort traffic. "--dst-type LOCAL" is needed. What you expected to happen: since we have also KUBE-NODE-PORT-UDP, another way would be like this: How to reproduce it (as minimally and precisely as possible): start kube-proxy in ipvs mode, create some NodePort services. Then look at the iptables rules. Anything else we need to know?: Environment: Kubernetes version (use ): commit Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): CentOS Linux release 7.2.1511 (Core) Kernel (e.g. ): Linux VM34centos 3.10.0-514.26.2.el7.x8664 SMP Tue Jul 4 15:04:05 UTC 2017 x8664 x8664 x86_64 GNU/Linux Install tools: Others:
/sig network /area kube-proxy /area ipvs /assign
/cc i remember this bug fixed, that's right?
, Thanks, seems like necessary. But you have any use case that ipvs proxier will have bugs in current mach rules?
It's already fixed by ? Can we close this issue now?
I don't see how fixed this issue. How about this scenario: k8s cluster A uses node port for service A k8s cluster B uses node port for service B pod (or host) in cluster A tries to access node port on k8s cluster B local node port rule matches the traffic and NATs it to the local node port, pod (or host) cannot reach cluster B's node port. Also, the remote node port could be replaced by any non-k8s host that happens to use port .
Then I think can solve this problem.


Question: } } <del> func TestHTTP1Requests(t *testing.T) { </del> <ins> // TestGOAWAYHTTP1Requests tests GOAWAY filter will not affect HTTP1.1 requests. func TestGOAWAYHTTP1Requests(t *testing.T) { </ins> s := httptest.NewUnstartedServer(WithProbabilisticGoaway(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { w.WriteHeader(http.StatusOK) w.Write([]byte("hello"))

What happened: We enabled feature in our large-scale cluster, and we found client(Kubelet) may got error(500 http code response) like after received GOAWAY frame. I had already re-produce this bug and open an in golang repo. So far I think this bug only occur with a small probability when client request server in high concurrency use a same httpclient instance, and it's a golang sdk bug in http client. Before I solved this issue, I suggest do NOT enable GOAWAY() feature. As it's a golang sdk bug(currently I known and doubt) in http client, revert GOAWAY feature(codes) is not necessary.
cc
/sig api-machinery
Thanks for the report.
Based on not setting the field, I think it defaults that way, so that's good. We will leave the option and filter so that when golang updates its client, we will have a simple way to turn it on and execute it.
Sure. I am trying to fix the golang http client issue and will update progress at . After golang http client issue fixed, I will add more concurrent test cases to see if there are any other bugs may occur by GOAWAY feature. BTW, we still enable GOAWAY feature in our production cluster, because there is no any evidence told us this error will influence SLO, as all most components(kubelet, controller-manager, scheduler) is controller work mode and will re-try after got error. /assign
Golang version or required, which will release at 07/01/2020.
Kubernetes v1.19(or master branch) now is using golang 1.15, and golang 1.15 has fixed the issue. And is on going. Let me add more test cases to ensure the error will not happen again.
I think we should re-open this issue because of . /re-open


Question: import ( "context" <ins> "errors" </ins> "fmt" <ins> kubeletconfig "k8s.io/kubernetes/pkg/kubelet/apis/config" </ins> "reflect" "sync" "testing"

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --I am attempting to pull an image from ECR using the new kubelet image credential provider feature which is an alpha features in Kubernetes v1.20. My CredentialProviderConfig file includes the environment variable AWSPROFILE which resides in the default () directory. The profile references a which allows me to execute a binary that returns a set of temporary AWS credentials. When I try running a pod with a container from ECR, I see the following errors in the kubelet logs: However, when I type the following into the command line on the kubelet: I get the following response: It doesn't appear the environment variables in my Config are being passed to the Kubelet when it executes the ecr-credential-provider binary. Below is Config file: I expect for the Kubelet to pass the environment variables in the Config file when it executes the ecr-credential-provider binary. I am using KinD to test this feature. Here is my config: No Kubernetes version (use ): 1.20 Cloud provider or hardware configuration: None OS (e.g: ): Ubuntu 20.10 Kernel (e.g. ): 5.10 Install tools: KinD Network plugin and version (if this is a network-related bug): Others: cc
/sig Node
/assign
from the code it appears that env variables are passed to the plugin from kubelet logs the plugin is only returning exit status 1, without any details about the error!!
Thanks Yes, I saw that. doesn't work. I wrote the following code to test this: If I uncomment , which imports the environment variables from the OS, it works.
Hello need some clarification. Is the intent of the above code to convey that isn't passing env vars correctly to the spawned process? Below code works as expected for me Also, could you tell if is a proprietary binary or a open source code something similar to because I want to use it in troubleshooting this issue.
Yes it is not passing the Envs to the spawned process.


Question: import ( "context" <ins> "errors" </ins> "fmt" <ins> kubeletconfig "k8s.io/kubernetes/pkg/kubelet/apis/config" </ins> "reflect" "sync" "testing"

The code for ecr-credential-provider is in the AWS cloud controller repository,
Quick update, I had setup two test beds to reproduce the above issue, but both of them gave expected results and the issue was not reproducible. Below are my test setups: ECR Repository Test Bed 1 Execute from code and pass as environment variable Code Observed Result - Below credentials were generated Expected Result - Credentials should be generated Test Bed 2 Use as the credential provider within the kubelet, I had setup a Provider Config Expected result - Pods should be created based off custom image Observed result - Pods were created based off custom image Though, my test bed runs on kubernetes version , I could see that nothing significant has changed in since version Could you give me any other minor but important detail which you feel you could had missed out earlier which could help me in replicating the issue? Or could you double check if provider exists in your file?
i am using credential_process, in my profile. I am also using KinD for testing, but I don't think that should matter. I don't have a credentials file. I only have a config file.
This is the binary I'm calling with credential_process,
can you check kubelet logs for , It might be possible that is falling back to in-tree aws credential provider.
on my local setup credentials are being fetched. I'll setup another test bed with the Cognito provider which is using.
It is fetching credentials because it is falling back to other options available, It might be fetching from other credential provider if failed from external one. To confirm this you have to check kubelet logs.
In my env I just have one provider and there are no fallbacks. Though I was keeping an eye on kubelet logs while performing this activity and I saw no such logs. The creds were fetched in one shot via the profile which I had specified. And as I had mentioned earlier I don't see any issues in passing env vars to credential process. Also, I did try negative cases as well by passing invalid profile names and it did fail and pods weren't pulled from ECR
kubelet registers intree credential providers while startup, you need not to explicitly add any!! And I recently tried GCP credential provider, It does fall back to intree GCP credential provider after getting failed response from the one i set up.
Sure, makes sense.


Question: import ( "context" <ins> "errors" </ins> "fmt" <ins> kubeletconfig "k8s.io/kubernetes/pkg/kubelet/apis/config" </ins> "reflect" "sync" "testing"

Though as I mentioned earlier I was keeping an eye on the kubelet logs and I didn't see the logs you had mentioned
i see in the logs when the kubelet starts, before attempting to run a pod that pulls an image from ECR.
Had a light bulb moment and I was able to reproduce this issue, it might not be applicable in your case but it could be worth trying out. In my local setup I ran the cluster as root and failed with error: E0614 18:19:27. ] Failed getting credential from external registry credential provider: error execing credential provider plugin ecr-credential-provider for image exit status 1 I have my aws credentials configure under my user () at path and since there is no similar configuration for the process failed. Since, I have access to code locally, I couple of log statements to print out current env vars and I could see that env var was set to :) Would it be possible for you to check is running as what user and is there any dissonance which can prohibit the user from successfully retrieving the credentials?
I am getting the error you mentioned, So far as I can tell, the kubelet is running as root and the file is in the directory. Again, I'm using KinD for testing. I'm not sure how KinD runs the kubelet. Below is the output from : The Go code I wrote to test this fails when using cmd.Env; it's only when I import the OS environment variables that it works. This may because I am using in my shared config file. credentialprocess is a mechanism to call another binary, e.g. , to get a set of temporary credentials from Cognito.
In order to simplify the sprawl and to pinpoint the offending component,could you remove the credential_process and Cognito and just try fetching creds via plain AWS credentials profile?
sure, however, supporting credential process is critical for my use case. I'm using Cognito as a gating mechanism to ECR. Only licensed users will be allowed to pull images from ECR. These users will not have IAM credentials (access key/secret key). They will authenticate to AWS using Cognito.
The following configuration works when I specify access key/secret key:
When I switch to a profile with credential process I get the following error:
isn't it the issue of ecr-credential-provider?


Question: import ( "context" <ins> "errors" </ins> "fmt" <ins> kubeletconfig "k8s.io/kubernetes/pkg/kubelet/apis/config" </ins> "reflect" "sync" "testing"

This is the job of ecr-credential-provider to fetch credentials and return result in expected format to kubelet.
why does it work when I run it from the command line? Why does it work when I import envs from the OS?
you might have saved env on command line, same goes for OS.
That doesn't make sense to me. If I run it returns a properly formatted response. Same goes for the Go code I wrote. Does calling another binary, like credentialprocess, confuse the kubelet?
I am not familiar with aws credential process. Kubelet talks to with exec based plugin mechanism, is ecr-credential-provider calling the credential process you are talking about?
I have already validated that env vars are being passed to and there is no issue over there. If that wouldn't be the case then AWS credentials generation process via wouldn't work for you. I had debugged the code of kubelet myself and validated it + I wrote some logs in to print out the env vars and it was logging the passed vars. IMHO, now what remains to be seen here is why it fails with Cognito. I am going to create a test bed tomorrow replicating the exact setup which you have in the mean time if it's possible for you could you create a Linux VM and replicate your setup inside it? My intention is to see whether the Cognito based credential process goes through or not in a clean slate environment if it does then something on your local setup is interfering with passed env vars Thoughts?
Here's the execution flow ---- Does the above paints a fairly accurate picture? If not then feel free to correct :)
I included my KinD configuration in the beginning of this thread. Can you use that The binary has to be built from source, (or I can send it to you along with a temp username/password). There is an accompanying blog post for the binary at The flow is as describes above. The binary returns a json blob to the calling application that looks like this: During my testing (without the kubelet) I am able to get the proper response for the kubelet when referencing the AWS profile using or by setting to . However, when I try to pass the AWS profile in using exec.Env("AWS_PROFILE=default") I get and no further information. I have a feeling that doesn't properly chain or pass information back to the calling application.


Question: import ( "context" <ins> "errors" </ins> "fmt" <ins> kubeletconfig "k8s.io/kubernetes/pkg/kubelet/apis/config" </ins> "reflect" "sync" "testing"

Let me replicate your setup in my local env and see if I can reproduce this issue. Also, could you tell what your AWS file look like? Is it something like this You can use placeholder values for sensitive information but correct the binary path if it's not
yes, that looks right. Ideally the cognito binary is located in a directory in the shell's PATH but it also works when you specify the full path in the config file like you've done. Are you planning to setup cognito in AWS? You'll need a cognito user pool and configure federation.
Yes, I am planning to setup Cognito Identity pool in my AWS Account.
So I replicated your setup on my local machine with slight change, instead of using AWS Cognito for generating credentials via I am calling a binary written by me which logs env vars to a specific path and prints out credentials in the format as prescribed by AWS. Below was my initial config file After I ran local cluster with this setup I noticed couple of log lines like below: I did some basic validation of perms + I even moved the binary to couple of well know locations but nothing worked Finally I modified my vars section of my deployment spec like below and it worked The logs which I had put in my custom credential provider printed following env vars validating the fact that, env vars are being passed In your logs do you see a log line something like above? If not then could you pass full contents of from start up until you create a Pod?
thanks for doing this. I never tried adding the PATH to the list of environment variables. I honestly didn't think it was necessary because I was referencing the full path to the binary in credential_process. I also never saw an error about not being able to find the executable in $PATH. I'll re-test after adding PATH as an ENV and let you know the results.
Adding the PATH did not work for me. I'm happy to send you the cognito binary with a username/password that you can use for testing. I have only tested this with KinD, but I don't think that is the issue here since it fails with exit code when when I try using .
^^ is my credential provider config
Thanks for the logs. I have cloned your code in my repo and I'll substitute my Cognito creds.


Question: import ( "context" <ins> "errors" </ins> "fmt" <ins> kubeletconfig "k8s.io/kubernetes/pkg/kubelet/apis/config" </ins> "reflect" "sync" "testing"

If it doesn't work out then I'll reach out to you for creds
After several rounds of troubleshooting I was able to get working in my KinD environment. My credential loading setup is <-<-and below is my configuration of credential process Initially, I got the same error as yours, then I couple of custom log lines to print env vars in in directory and I could verify that the env vars specified in the credentials spec were indeed being passed. What I noticed in my KinD setup is that AWS SDK was logging an error because it couldn't find any "providers", though I could verify in the container that file and were indeed present. I altered my env vars section of the as below and it worked for me Apparently, what I noticed is is being passed ONLY the env vars which are present in the spec and everything else was missing! Or at least this is the behaviour in my local cluster setup + KinD cluster. Can you please add and other env vars necessary for the correct execution of AWS SDK and see if it works for you? If this doesn't work, then I'll send you the modified version of the which puts some additional logs to troubleshoot this issue further.
That did it! Thanks for helping me troubleshoot this. I wonder if this is unique to KinD or whether we would see this behavior in a "normal" cluster (i'll test it later). I will tell the developers who wrote the ecr-credential-provider to include an argument to increase the verbosity or to output logs to a file so this is easier to troubleshoot in the future.
What I have observed is, is only passed env vars which are in credential spec and the spec clearly says that the vars in spec are appended to existing env vars and this looks like an issue which needs deeper investigation. Could you please re-open this issue?
sure thing.
The label cannot be applied. Only GitHub organization members can add the label. <detailsIn response to : /triage accepted Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
To me this looks like an issue with the kubelet and I have observed this behaviour with as well as my local cluster. What are your thoughts on this? If this looks like a bug then can you accept the triage? I would fix this.


Question: import ( "context" <ins> "errors" </ins> "fmt" <ins> kubeletconfig "k8s.io/kubernetes/pkg/kubelet/apis/config" </ins> "reflect" "sync" "testing"

Thanks for looking into the issue, you are right the spec does says that. Feel free to fix it /triage accepted
Thanks!!!
/area kubelet
: I am facing the similar issue on my bare metal cluster This is my config Can you help. It works when I try to execute the credentials binary from local but doesn't work in the pod. Can you pls guide me
I've been fighting this for days - did you ever figure it out?


Question: } } <ins> func TestAtomicUpdateNoChange(t *testing.T) { fakeClient := MakeFakeEtcdClient(t) fakeClient.TestIndex = true helper := EtcdHelper{fakeClient, scheme, api.NewJSONBaseResourceVersioner()} // Create a new node. fakeClient.ExpectNotFoundGet("/some/key") obj := &TestResource{JSONBase: api.JSONBase{ID: "foo"}, Value: 1} err := helper.AtomicUpdate("/some/key", &TestResource{}, func(in interface{}) (interface{}, error) { return obj, nil }) if err != nil { t.Errorf("Unexpected error %#v", err) } // Update an existing node with the same data callbackCalled := false objUpdate := &TestResource{JSONBase: api.JSONBase{ID: "foo"}, Value: 1} fakeClient.Err = errors.New("should not be called") err = helper.AtomicUpdate("/some/key", &TestResource{}, func(in interface{}) (interface{}, error) { callbackCalled = true return objUpdate, nil }) if err != nil { t.Errorf("Unexpected error %#v", err) } if !callbackCalled { t.Errorf("tryUpdate callback should have been called.") } } </ins> func TestAtomicUpdate_CreateCollision(t *testing.T) { fakeClient := MakeFakeEtcdClient(t) fakeClient.TestIndex = true

When you setup TLS assets, you need to make sure each master API server is using the same ones, unless you were running a single master. But this is a super common mistake, and it often breaks things. Because the masters validating tokens using that key, the controller-manager also has this service-account key, and it’s responsible for creating tokens for service accounts. So, depending on which apiserver the pod is talking to, that service-account token may, or may not work. This most commonly surfaces in kube2sky (the dns addon). Result: Tokens are invalidated regularly (~10 minutes) Expected: Warning about system having different certs (or some alert). Would this ever be a valid configuration?
Is this specifically for an HA setup (since you mention multiple apiservers)? Or do you mean that each master component should have the same cert?
Correct - it's the HA deployment.
I don't think that the apiservers should necessarily cross-check each other (and all other system components like the controller manager) and alert. If you are running an HA deployment, you should have something else making sure that your deployment is correct. On cluster creation, the startup scripts should do it.
I guess what I'm saying is that we know this is going to be a problem. There's almost no way to make this NOT a problem, yet it is a very easy bug to get into. So... with that in mind, what's the path for doing this? When you add a second master to your HA config, how does it establish trust with the first HA? Could we check at that point?
Yup, we do this on provision time with Ansible.


Question: mv "${src_dir}/kubelet" "${KUBE_BIN}" mv "${src_dir}/kubectl" "${KUBE_BIN}" <ins> # Some older images have LICENSES baked-in as a file. Presumably they will # have the directory baked-in eventually. rm -rf "${KUBE_HOME}"/LICENSES </ins> mv "${KUBE_HOME}/kubernetes/LICENSES" "${KUBE_HOME}" mv "${KUBE_HOME}/kubernetes/kubernetes-src.tar.gz" "${KUBE_HOME}" fi

Which jobs are failing: Which test(s) are failing: BeforeSuite Overall Test Since when has it been failing: Testgrid link: Reason for failure: Anything else we need to know: /cc /milestone v1.19 /priority critical-urgent /sig testing
Taking an example from 1 of the runs, these failures seem to show up across all failures. These pods failed to reach a "Running" state after 10 minutes Then, while dumping events from the kube-system namespace these couple look interesting That kube proxy image seems to not exist (?), I tried pulling it from another machine and got this The commit that is part of the kube proxy tag, , does coincide with the k/k commit that went in right as this test broke and that commit range only introduces this PR The following CI tests are listed fro that specific commit (when looking at the commit history of k/k) ci-kubernetes-bazel-build ci-kubernetes-bazel-test post-kubernetes-bazel-build post-kubernetes-bazel-test These jobs dont show anything particularly interesting though.
looks like failed in /cc
I think it there may be a bad manifests used for this job. The kube proxy images are being built. If you look for in the logs of other ci jobs, i.e., you will see that an alpha 3 with a k/k commit is being used. For example, The thing is that, I think, there should be a mention of the ARCH in the container image, e.g., amd64 in the above example, where as teh GPU job doesnt have any arch.
Mmm though im not 100 sold on the image name being the problem. Checking out both kube-proxy and kube-proxy-amd64 both exist and both seem to be updated frequently
/assign I had similar issues trying to use the debian-iptables image, let me check if I'm able to solve it here too /cc FYI
this may help:
Seems the job is failing to install the nodes with an error related to the LICENSES folder LICENSES has a recent history of changes: I'm not able to correlate between the changes and job failures, you have better insights in the LICENSES folder, what's the best way to proceed here?
I think changed LICENSES in the root of the built artifacts from a file to a folder /assign
ARAGAGRAGAHHHHH !!!


Question: mv "${src_dir}/kubelet" "${KUBE_BIN}" mv "${src_dir}/kubectl" "${KUBE_BIN}" <ins> # Some older images have LICENSES baked-in as a file. Presumably they will # have the directory baked-in eventually. rm -rf "${KUBE_HOME}"/LICENSES </ins> mv "${KUBE_HOME}/kubernetes/LICENSES" "${KUBE_HOME}" mv "${KUBE_HOME}/kubernetes/kubernetes-src.tar.gz" "${KUBE_HOME}" fi

FTR, I am looking at this, please don't revert my revert revert yet
Failed to repro, but I also get:
seems that this is only affecting the jobs
Yeah, I think I messed up the invocation. It should have run tests.
OK, I got it to run a test but it doesn't fail this way. I have another umask fix, let me back that out.
I can't get it to fail like this. On my run I see 27/27 pods running, but this run seems stuck with: that varies from 24 - 26. My personal kubetest invocation: And my run failed, but with:
Further more, I can't think of ANY way the old non-directory LICENSES file would be present unless there's some cache that is not invalidated when it needs to be and is leaking across tests
-- is this greenhouse/bazel?
I sincerely doubt greenhouse. I don't understand enough about the bazel build in this repo to say one way or the other. Why would it be consistently affecting this and no other job that runs in the same cluster, e.g.
and why started to fail this morning, if the patch was merged a few days ago?
Jordan figured it out. This node image has /home/kubernetes baked in.
2nd time I get to use this today:


Question: _, err := client.CoreV1().Nodes().Get(nodeName, metav1.GetOptions{}) if err != nil && errors.IsNotFound(err) { node := v1.Node{ObjectMeta: metav1.ObjectMeta{Name: nodeName}} <del> schedulerCache.RemoveNode(&node) </del> <ins> if err := schedulerCache.RemoveNode(&node); err != nil { klog.V(4).Infof("Node %q is not found; failed to remove it from the cache.", node.Name) } </ins> } } } else {

<!-- Please only use this template for submitting enhancement requests --What would you like to be : 1) Increase the log level of info logs to "v=2" when the scheduler or a node from its cache. 2) Increase the log level of info logs to "v=2" when the scheduler can't schedule a pod , , and add a log message for 2) report memory/cpu status of . Why is this needed: With cluster autoscaler now being used in production by some vendors, we want to have more visibility to events that could be related to the auto scaler to help debug customer issues. /sig scheduling /priority important-soon /assign
GitHub didn't allow me to assign the following users: congliu01. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : <!-- Please only use this template for submitting enhancement requests --1) Increase the log level of info logs to "v=2" when the scheduler or a node from its cache. 2) report memory/cpu status of . Why is this needed: With cluster autoscaler now being used in production by some vendors, we want to have more visibility to events that could be related to the auto scaler to help debug customer issues. /priority important-soon /assign </details
will consider this assigned to you.
/assign


Question: return threadsPerCore > 1 } <ins> func isMultiNUMA() bool { outData, err := exec.Command("/bin/sh", "-c", "lscpu | grep "NUMA node(s):" | cut -d ":" -f 2").Output() framework.ExpectNoError(err) numaNodes, err := strconv.Atoi(strings.TrimSpace(string(outData))) framework.ExpectNoError(err) return numaNodes > 1 } </ins> func getCPUSiblingList(cpuRes int64) string { out, err := exec.Command("/bin/sh", "-c", fmt.Sprintf("cat /sys/devices/system/cpu/cpu%d/topology/thread_siblings_list | tr -d "nr"", cpuRes)).Output() framework.ExpectNoError(err) return string(out) } <ins> func getCoreSiblingList(cpuRes int64) string { out, err := exec.Command("/bin/sh", "-c", fmt.Sprintf("cat /sys/devices/system/cpu/cpu%d/topology/core_siblings_list | tr -d "nr"", cpuRes)).Output() framework.ExpectNoError(err) return string(out) } </ins> func deleteStateFile() { err := exec.Command("/bin/sh", "-c", "rm -f /var/lib/kubelet/cpu_manager_state").Run() framework.ExpectNoError(err, "error deleting state file")

Which jobs are failing: N/A - AFAICT CI jobs works fine Which test(s) are failing: Since when has it been failing: no historical record on CI Testgrid link: N/A Reason for failure: I believe the tests make too many assumptions around the hardware, and the code is actually OK. to be confirmed. Anything else we need to know:
/sig node /sig testing
also affects the recently topology manager e2e tests. My PR uses a different approach to check resource alignment so is not affected. /cc
I'll work on a PR to fix this issue ASAP
Thanks for opening this issue. We knew about this, just had not opened an issue to track it, so thanks for doing that! Deepthi has been looing at CPU Manager tests, not sure if she has some PR in progress or not. I'll make sure to forward this to her (can find her user id here ....) in case she has something in progress


Question: }) } <del> // Wait for a job to not exist by listing jobs explicitly. func waitForJobNotExist(c clientset.Interface, ns string, targetJob *batchv1.Job) error { </del> <ins> // Wait for a job to disappear by listing them explicitly. func waitForJobToDisappear(c clientset.Interface, ns string, targetJob *batchv1.Job) error { </ins> return wait.Poll(framework.Poll, cronJobTimeout, func() (bool, error) { jobs, err := c.BatchV1().Jobs(ns).List(metav1.ListOptions{}) if err != nil {

What happened: There seems to be a regression after 1.13 release. A Cronjob with default settings does not clean up completed pods. The job themselves are getting deleted as expected, but the pods are left behind. This seems to be related to What you expected to happen: Expected the pods for now non-existent child jobs of a cronjob to be cleaned up. This is the way it used to work prior to 1.13/ How to reproduce it (as minimally and precisely as possible): run on this manifest: Wait, more than 3 minutes, observe, that while only 3 successful jobs are present, the pods from now deleted jobs are left behind and never get deleted. Anything else we need to know?: pointed out the following relevant places in the code base: deleting a job orphans pods by default and the cronjob controller doesn't do anything different than the default when deleting Environment: Kubernetes version (use ): 1.13.0 Cloud provider or hardware configuration: ubuntu VM in vSphere OS (e.g. from /etc/os-release): 16.04.5 LTS (Xenial Xerus) Kernel (e.g. ): 4.4.0-135-generic Install tools: kubeadm <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
/sig apps
Should include a known issues in 1.13.0 as well /priority critical-urgent
Lemme see if I can easily reproduce this.
/milestone v1.13
Fixes are in (master) and (1.13).


Question: } // If any adoptions are attempted, we should first recheck for deletion with // an uncached quorum read sometime after listing Pods (see #42639). <del> canAdoptFunc := controller.RecheckDeletionTimestamp(func() (metav1.Object, error) { </del> <ins> dsNotDeleted := controller.RecheckDeletionTimestamp(func() (metav1.Object, error) { </ins> fresh, err := dsc.kubeClient.ExtensionsV1beta1().DaemonSets(ds.Namespace).Get(ds.Name, metav1.GetOptions{}) if err != nil { return nil, err

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: The DaemonSet controller will schedule a pod on a node when the old pod is still in Terminating state. This breaks the expectation that only one pod will be running at a time per node. What you expected to happen: The DaemonSet controller should not let two pods running at the same time on the same node. How to reproduce it (as minimally and precisely as possible): Create a DaemonSet. Set terminationGracePeriodSeconds to be relatively long. Kill a pod. Watch a new pod start on the same node when the old pod is still terminating. Anything else we need to know?: Seems this behavior was introduced in Environment: Kubernetes version (use ): 1.7.2 Cloud provider or hardware configuration**: GKE OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig apps
/cc
cc
cc: I think the expected behavior is correct, the terminating pod could be performing an important step such as 'removing a lock' or 'cleanup' etc., we should allow that to gracefully die before creating a new one. IMHO we should revoke to old behavior, but why did we need this new behavior in the first place? That is not clear to me from the PR and Issue.
We don't currently guarantee at least or at most one scheduling for DaemonSet. Both behaviors are under consideration via issue and in the future work section of the . The reason for is to "Make sure that controllers are not adopting children that are marked for deletion" (). This is necessary for GC to work properly.
Well the behavior may be technically undefined, but can I request that in the future changes like this be mentioned in the release notes?
Yes, certainly.


Question: } // If any adoptions are attempted, we should first recheck for deletion with // an uncached quorum read sometime after listing Pods (see #42639). <del> canAdoptFunc := controller.RecheckDeletionTimestamp(func() (metav1.Object, error) { </del> <ins> dsNotDeleted := controller.RecheckDeletionTimestamp(func() (metav1.Object, error) { </ins> fresh, err := dsc.kubeClient.ExtensionsV1beta1().DaemonSets(ds.Namespace).Get(ds.Name, metav1.GetOptions{}) if err != nil { return nil, err

Also "The DaemonSet controller should not let two pods running at the same time on the same node." is not necessarily the same thing as respecting termination prior to launching a new Pod. The former implies a more strict requirement than the latter. I think we should consider at least attempting to wait for a terminating Pod to be deleted prior to creating another Pod on the Node for same DaemonSet as default behavior.
will something like work then?


Question: // DeleteNominatedPodIfExists does nothing in FIFO. func (f *FIFO) DeleteNominatedPodIfExists(pod *v1.Pod) {} <ins> // NumUnschedulablePods returns the number of unschedulable pods exist in the SchedulingQueue. func (f *FIFO) NumUnschedulablePods() int { return 0 } </ins> // NewFIFO creates a FIFO object. func NewFIFO() *FIFO { return &FIFO{FIFO: cache.NewFIFO(cache.MetaNamespaceKeyFunc)}

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Today, the scheduler re-processes unschedulable nodes . Node updates are sent by Kubelet frequently (every 10 seconds) and often times they don't change any of the node properties that could potentially make them feasible. We should be smarter about these updates and move pods to the schedulable queue only when certain properties of nodes change that could make them feasible. These properties include node allocatable, node conditions, taints, labels, .... Why is this needed: Improves performance of the scheduler and avoids doing unnecessary attempts to schedule pods. This is particularly helpful when a cluster has a lot of unschedulable pods. <!-- DO NOT EDIT BELOW THIS LINE --/kind feature This is not a feature really. It is performance optimization, but in the absence of such a label, I need to use "feature" here. /sig scheduling
I can help to fix it. /assign mlmhl
The main reason for this problem is / update their status too frequently and most of these updates are heartbeats without any meaningful change. Recently sig-node introduces a new to alleviate this problem. With this mechanism, kubelet will split heartbeat and node status updates into two independent operations, and reduce the frequency of status updates to once per 1 minute(or more). In the long run this mechanism can indeed alleviate our problems, but at present it does not help us much as this is still an alpha feature and the node status updates frequency is still a bit high(per 1 minute right now).
Maybe we can try comparing old and new nodes after stripping their heartbeat info, like this:
I think that had sent earlier is more efficient. However, that PR does not filter out . In Wei's code, the heartbeat time is removed correctly. I think we should fix this issue and resubmit that PR.
could you resubmit a PR to: disregard in conditions comparison: and (optionally) disregard timestamp in taints comparison:
Thanks for all your advice, a new PR is submitted: This PR does not include your second suggestion(disregard timestamp in taints comparison) as I'm not sure if can be changed when node updating.


Question: // DeleteNominatedPodIfExists does nothing in FIFO. func (f *FIFO) DeleteNominatedPodIfExists(pod *v1.Pod) {} <ins> // NumUnschedulablePods returns the number of unschedulable pods exist in the SchedulingQueue. func (f *FIFO) NumUnschedulablePods() int { return 0 } </ins> // NewFIFO creates a FIFO object. func NewFIFO() *FIFO { return &FIFO{FIFO: cache.NewFIFO(cache.MetaNamespaceKeyFunc)}

If it can be changed, I will add this filter later, otherwise I think we can keep the current implementation to make the compare operation simpler and faster. What's more, just as I said in the PR's description, I'm not sure how much side effects this change has on scheduler's resource consumption, as it always copy node's conditions before comparison.
I am thinking whether we could move this logic(whether node became more schedulable) to kubelet, kubelet reports it through heartbeats.


Question: ExpectedObject: &unstructured.Unstructured{ Object: map[string]interface{}{ "apiVersion": "example.com/v2", <ins> "metadata":   map[string]interface{}{}, </ins> "other":      "data", "kind":       "foo", },

If a CR converter is wired in and it modifies the input list, the delegating CR converter does not protect against this. A converter cannot effect the input list Not really possible externally - it's a coding issue that the current converters don't hit. No response <details</details<detailsN/A </details<details</details<details</details<details</details<details</details
/sig api-machinery /assign
/triage accepted


Question: } } } <ins> // TestProcessHostnameOverrideFlag tests processing hostname-override arg func TestProcessHostnameOverrideFlag(t *testing.T) { testCases := []struct { name                 string hostnameOverrideFlag string expectedHostname     string }{ { name:                 "Hostname from config file", hostnameOverrideFlag: "", expectedHostname:     "foo", }, { name:                 "Hostname from flag", hostnameOverrideFlag: "  bar ", expectedHostname:     "bar", }, } for _, tc := range testCases { t.Run(tc.name, func(t *testing.T) { options := NewOptions() options.config = &kubeproxyconfig.KubeProxyConfiguration{ HostnameOverride: "foo", } options.hostnameOverride = tc.hostnameOverrideFlag err := options.processHostnameOverrideFlag() assert.NoError(t, err, "unexpected error %v", err) if tc.expectedHostname != options.config.HostnameOverride { t.Fatalf("expected hostname: %s, but got: %s", tc.expectedHostname, options.config.HostnameOverride) } }) } } </ins>

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: kube-proxy came up cleanly but mis-configured iptable rules due to a mis-match in the hostname and node name within the cluster triggering a default behavior. In this case the hostname returned the fqdn of the host and the cluster only had the hostname. This resulted in the following warning messages when kube-proxy started. Which then presents itself as pods not being able to access clusterip ip addresses with no route errors. What you expected to happen: Would it be better to fail to in this scenario rather than try to fall back to some default? I spent a good amount of time trying to track down the Error messages that were a result of this before finally working back to taking a closer at the Warning message. How to reproduce it (as minimally and precisely as possible): Start the kube-proxy with set to an invalid hostname should recreate this issue. Anything else we need to know?: While I think we can chalk up this issue to mis-configuration the purpose of opening this issue is really to have a conversation about the default behavior. I've seen several other issues filed that appear to be related to this behavior: , , . Would throwing an error and exiting non-zero be a better default action when the nodeIP is nil? What use case(s) does a default of support? If there must be a default behavior here maybe the IP address on the interface associated with the default gateway is a better default than ? Maybe the issue isn't in the proxier code but the kube-proxy itself should error when it fails to get node info rather than a warning? Environment: I'm not sure how much of this is important to this but here's what my environment was when I ran into this. Kubernetes version: 1.8.5 Cloud provider or hardware configuration: Bare metal OS (e.g. from /etc/os-release): Centos 7 and Ubuntu 16.04
This is similar (though not the same symptom) to for which was supposed to fix it but was closed. One of the objections was up til now, kube-proxy has had no requirement that the hostname = nodename or even that the hostname be defined.


Question: } } } <ins> // TestProcessHostnameOverrideFlag tests processing hostname-override arg func TestProcessHostnameOverrideFlag(t *testing.T) { testCases := []struct { name                 string hostnameOverrideFlag string expectedHostname     string }{ { name:                 "Hostname from config file", hostnameOverrideFlag: "", expectedHostname:     "foo", }, { name:                 "Hostname from flag", hostnameOverrideFlag: "  bar ", expectedHostname:     "bar", }, } for _, tc := range testCases { t.Run(tc.name, func(t *testing.T) { options := NewOptions() options.config = &kubeproxyconfig.KubeProxyConfiguration{ HostnameOverride: "foo", } options.hostnameOverride = tc.hostnameOverrideFlag err := options.processHostnameOverrideFlag() assert.NoError(t, err, "unexpected error %v", err) if tc.expectedHostname != options.config.HostnameOverride { t.Fatalf("expected hostname: %s, but got: %s", tc.expectedHostname, options.config.HostnameOverride) } }) } } </ins>

Making kube-proxy exit on failure to retrieve the node object might cause regressions in some legitimate use-cases. But I have no idea what those are. ?
I have a hard time imagining what the legit cases might be. I am confused, though - I only see this field being used in one very specific place, that would not have the effect you are claiming.
I can't see the error message from log? If kube-proxy failed to create kube-ipvs0 dummy interface, it will log something like "Failed to create dummy interface: kube-ipvs0, error:...".
This breaks Services of type and with . This is problematic for AWS environments where the kubelet is configured to override the hostname (AWS instances default to the short name and for the cloud provider integrations to work this needs to be overridden to the private fqdn of the host).
Any chance we could get this prioritized as a bug fix for v1.11?
cc
After putting a bit more thought into this, would it be possible to standardize on an environment variable that could be set through the downward api to override the default behavior of treating the hostname as the node name in the proxy? That would simplify the configuration on the kubeadm side (no need to override the command with the downward api, only setting the appropriate env variable), and would also provide a more accurate way to retrieve the actual node name for kube-proxy when deployed as a daemonset without the need to interact with the kubelet directly. I would suggest that we could use for this purpose. I'd also be happy to take a stab at implementing the changes needed for the v1.12 cycle.


Question: } } } <ins> // TestProcessHostnameOverrideFlag tests processing hostname-override arg func TestProcessHostnameOverrideFlag(t *testing.T) { testCases := []struct { name                 string hostnameOverrideFlag string expectedHostname     string }{ { name:                 "Hostname from config file", hostnameOverrideFlag: "", expectedHostname:     "foo", }, { name:                 "Hostname from flag", hostnameOverrideFlag: "  bar ", expectedHostname:     "bar", }, } for _, tc := range testCases { t.Run(tc.name, func(t *testing.T) { options := NewOptions() options.config = &kubeproxyconfig.KubeProxyConfiguration{ HostnameOverride: "foo", } options.hostnameOverride = tc.hostnameOverrideFlag err := options.processHostnameOverrideFlag() assert.NoError(t, err, "unexpected error %v", err) if tc.expectedHostname != options.config.HostnameOverride { t.Fatalf("expected hostname: %s, but got: %s", tc.expectedHostname, options.config.HostnameOverride) } }) } } </ins>

Pods not getting scheduled on the new node joined to the cluster using kubespray The node joined to the cluster but new pods schedueld on the nodes are failing with error "get https://10.233.0.1 : dial tcp 10.233.0.1: 443 connection refused" Network plugin - calico kube-proxy running in ipvs mode new worker node, can connect to master server ip address over 6443 new worker node, cannot access cluster ip 10.233.0.1 over port 443 , nodelocaldns and kube-proxy pods are in running state on new node -proxy logs show error "invalid nodeIP, initializing kube-proxy with 127.0.0.1 as nodeIP" Unable to write event: 'Post https://127.0.0.1:6443/api/v1/namespaces/default/events dial tcp 127.0.0.1:6443 connection refused" -ln output do not show any services addr show output do not show kube-ipvs0 and calico interfaces Can someone please help. Thanks


Question: kubelet.containerManager = &localCM{ ContainerManager: cm.NewStubContainerManager(), allocatable: v1.ResourceList{ <del> v1.ResourceCPU:     *resource.NewMilliQuantity(200, resource.DecimalSI), v1.ResourceMemory:  *resource.NewQuantity(100E6, resource.BinarySI), v1.ResourceStorage: *resource.NewQuantity(200*mb, resource.BinarySI), </del> <ins> v1.ResourceCPU:    *resource.NewMilliQuantity(200, resource.DecimalSI), v1.ResourceMemory: *resource.NewQuantity(100E6, resource.BinarySI), </ins> }, }

Local storage is not expected to be available as a scheduling resource in v1.7. Kubelet is incorrectly exposing a resource named . This is incorrect because the feature is alpha and so it requires opt-in via feature flags and, the resource names are invalid. The expected resource names are listed - and . cc This is v1.7 blocker because we cannot expose a new resource in the API and rename it in a subsequent release.
CC I'll find someone on the storage team to address this.
/assign
GitHub didn't allow me to assign the following users: verult. Note that only can be assigned. <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: func (c *pods) GetLogs(name string, opts *v1.PodLogOptions) *restclient.Request { return c.client.Get().Namespace(c.ns).Name(name).Resource("pods").SubResource("log").VersionedParams(opts, scheme.ParameterCodec) } <ins> // ProxyGet returns a response of the pod by calling it through the proxy. func (c *pods) ProxyGet(scheme, name, port, path string, params map[string]string) restclient.ResponseWrapper { request := c.client.Get(). Namespace(c.ns). Resource("pods"). SubResource("proxy"). Name(net.JoinSchemeNamePort(scheme, name, port)). Suffix(path) for k, v := range params { request = request.Param(k, v) } return request } </ins>

Currently services/proxy is available in client-go, but pods/proxy resource is not. This requires manual usage via the rest client like This can be seen in Kubernetes e2e tests as well
/kind feature /sig api-machinery


Question: <ins> # Replication controller tools ## resize.sh Resizes a replication controller to the specified number of pods. ``` $ resize.sh usage: resize.sh <replication controller name> <size> $ resize.sh redisslave 4 ``` ## stop.sh Resizes a replication controller to 0 pods and waits until the pods are deleted. ``` $ stop.sh usage: stop.sh <replication controller name> $ stop.sh redisslave ``` </ins>

kubectl describe emits 1024 NULL characters prior to actual text. This prevents all text-based processing tools from working on its output. /cc
This has got to be due to:
(IRL discussion: there's no reason to pass 1024 0 bytes to NewBuffer.)
Fixing in PR .


Question: return } }() <del> defer errorStream.Reset() </del> if e.stdin != nil { <del> headers.Set(api.StreamType, api.StreamTypeStdin) remoteStdin, err := conn.CreateStream(headers) if err != nil { return err } defer remoteStdin.Reset() </del> // TODO this goroutine will never exit cleanly (the io.Copy never unblocks) // because stdin is not closed until the process exits. If we try to call // stdin.Close(), it returns no error but doesn't unblock the copy. It will

We create the stdin stream and immediately start copying local stdin to the container. We do the same for stdout and stderr (create stream, start copying immediately). In the kubelet, we wait to receive all the streams before we start the exec and copy between the streams and the container. The spdy connection frame handling code has 5 frame workers, each with a queue of up to 50 frames. In the event that we try to push more than 50 frames to a single queue before all streams are ready, the 51st attempt to push a frame blocks until there is space available in the queue. But there won't be any space until the server starts reading from the streams, which won't happen until all the streams have been received, so we're stuck at this point. The fix is to create all the streams first, and only start copying after all the streams have been created and the server is ready to copy between the streams and the container. Originally reported here:
FYI
This seems pretty high priority.
It's an easy fix. has already tested the patch for OpenShift and it works. Are we willing to pull it in before is resolved, as that will take longer to put in place?
Here's the patch, for reference:
5 * 50 = 250 frames before you get stuck? Or did I misunderstand?
Oh, I guess all the frames go to the same worker for some reason?
anyway, I'm fine with taking this and cherry picking it. Order doesn't matter; someone has to rebase either way, right?
frames are assigned to each queue based on stream id, so if you sent 50 frames to the stdin stream, that queue fills up and the loop that processes incoming frames is blocked until something starts reading from the stream. Working on a PR now.
Gotcha, thanks. , Andy Goldstein wrote:
an update: I spent some time trying to create an e2e that would fail more often than not without this fix, but I haven't had any luck so far (I have it stream a copy of the kubectl binary from the local system to the pod). The issue that originally uncovered the problem was from running exec on a Windows system, where it reproduces consistently, but I haven't seen it fail much on Linux. I will keep trying...


Question: // isPrivileged will return true a pod has any privileged containers func isPrivileged(pod *api.Pod) bool { for _, c := range pod.Spec.InitContainers { <del> if c.SecurityContext == nil { </del> <ins> if c.SecurityContext == nil || c.SecurityContext.Privileged == nil { </ins> continue } if *c.SecurityContext.Privileged {

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): No What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): panic apiserver invalid memory address Is this a BUG REPORT or FEATURE REQUEST? (choose one): Bug Report <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Environment: Cloud provider or hardware configuration: aws OS (e.g. from /etc/os-release): Ubuntu 16.04 Kernel (e.g. ): Linux ip-10-50-193-54 4.4.0-59-generic -Ubuntu SMP Fri Jan 6 17:47:47 UTC 2017 x8664 x8664 x86_64 GNU/Linux Install tools: Others: What happened: Tried to exec: . Got back: apiserver logs: It only seems to happen on this one pod - other pods are fine. What you expected to happen: should succeed or at least give a sensible error message. How to reproduce it (as minimally and precisely as possible): Anything else do we need to know:
There are no errors on the node with the pod. I also tried kubectl v1.4.8 and same problem.
It looks like your pod that's having the issue has a container with a security context, but privileged appears to be nil. Can you confirm that is the case?
Yep that's it. It seems when securityContext is specified without setting , it isn't set correctly to its default of false. e.g. the pod yaml has: Explicitly adding allows exec to succeed.
Ah, this was caused by the kubelet using versioned APIs - for his awareness. Basically any code that depends on nil pointers being set is vulnerable.


Question: <ins> /* Copyright 2016 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package cmd import ( "io" "strings" "github.com/spf13/cobra" cmdutil "k8s.io/kubernetes/pkg/kubectl/cmd/util" ) const help_long = `Help provides help for any command in the application. Simply type kubectl help [path to command] for full details.` func NewCmdHelp(f *cmdutil.Factory, out io.Writer) *cobra.Command { cmd := &cobra.Command{ Use:   "help [command] | STRING_TO_SEARCH", Short: "Help about any command", Long:  help_long, Run: RunHelp, } return cmd } func RunHelp(cmd *cobra.Command, args []string) { foundCmd, a, err := cmd.Root().Find(args) // NOTE(andreykurilin): actually, I did not find any cases when foundCmd can be nil, //   but let's make this check since it is included in original code of initHelpCmd //   from github.com/spf13/cobra if foundCmd == nil { cmd.Printf("Unknown help topic %#q.n", args) cmd.Root().Usage() } else if err != nil { // print error message at first, since it can contain suggestions cmd.Println(err) argsString := strings.Join(args, " ") var matchedMsgIsPrinted bool = false for _, foundCmd := range foundCmd.Commands() { if strings.Contains(foundCmd.Short, argsString) { if !matchedMsgIsPrinted { cmd.Printf("Matchers of string '%s' in short descriptions of commands: n", argsString) matchedMsgIsPrinted = true } cmd.Printf("  %-14s %sn", foundCmd.Name(), foundCmd.Short) } } if !matchedMsgIsPrinted { // if nothing is found, just print usage cmd.Root().Usage() } } else if len(a) == 0 { // help message for help command :) cmd.Root().Usage() } else { helpFunc := foundCmd.HelpFunc() helpFunc(foundCmd, args) } } </ins>

It'd be similar to Linux , a command to search the man page files
Why not just make do a search if it doesn't have an exact match?
sounds good.
SGTM
- Can be useful in this issue
I will work on this issue, if nobody is already working on it ( is not working on it).
thanks!
I focused currently on a bit different stuff, sorry. So I will start/finish it later. If anyone want to take over this one, feel free for now.


Question: minTolerationTime := getMinTolerationTime(usedTolerations) // getMinTolerationTime returns negative value to denote infinite toleration. if minTolerationTime < 0 { <del> klog.V(4).Infof("New tolerations for %v tolerate forever. Scheduled deletion won't be cancelled if already scheduled.", podNamespacedName.String()) </del> <ins> klog.V(4).Infof("Current tolerations for %v tolerate forever, cancelling any scheduled deletion.", podNamespacedName.String()) tc.cancelWorkWithEvent(podNamespacedName) </ins> return }

What happened: A Pod was scheduled for deletion because a taint w/ effect was observed in the Node the Pod was assigned to, and after the taint is removed from the Node, the Pod scheduled deletion isn't canceled. After scheduling pod deletion, the only cancels pod deletion if all taints are removed from the node, including any taints that the user relies on for their use-cases (example below). What you expected to happen: After the taint that triggered scheduling pod deletion is removed, cancels the scheduled pod deletion. How to reproduce it (as minimally and precisely as possible): A node is tainted w/ , so any workloads already assigned to it which don't tolerate this taint are evicted. A pod is created w/ tolerations: w/ unspecified and w/ . is assigned to . At some point in time after the previous steps took place, is tainted as . gets an update event for and observes two taints. It proceeds to calculate the minimum time (in seconds) that tolerates for the two taints which at this point is 300 seconds, and marks for deletion in ~300 seconds - this happens in-memory, in a timed-queue. At some point in time after the previous steps took place, and before the 300 seconds trigger happen, taint is removed from . gets an update event for and observes only one taint, . It proceeds to calculate the minimum time (in seconds) that tolerates for the this taint which is infinity, and returns, not canceling the previous deletion. It completely ignores the fact that the taint that triggered deletion is no longer observed. Anything else we need to know?: Worked w/ to figure this out after all pods in a node got evicted without apparent reason. Environment: Kubernetes version (use ): 1.17.3 (but looking at the code seems to affect all of 1.17.x and 1.18.x and master. Cloud provider or hardware configuration: N/A OS (e.g: ): N/A Kernel (e.g. ): 5.x Install tools: N/A Network plugin and version (if this is a network-related bug): N/A Others: N/A cc (owners of this code)
There seems to be at least .


Question: minTolerationTime := getMinTolerationTime(usedTolerations) // getMinTolerationTime returns negative value to denote infinite toleration. if minTolerationTime < 0 { <del> klog.V(4).Infof("New tolerations for %v tolerate forever. Scheduled deletion won't be cancelled if already scheduled.", podNamespacedName.String()) </del> <ins> klog.V(4).Infof("Current tolerations for %v tolerate forever, cancelling any scheduled deletion.", podNamespacedName.String()) tc.cancelWorkWithEvent(podNamespacedName) </ins> return }

You can sort of get a feel for what's going on here by looking at these two code snippets: This is what happens in when the node does not have any NoExecute taints: If you have any NoExecute taints on the node processPodOnNode gets called and this routine is really, really eager to evict pods. So : We are currently working around this issue by changing over our NoExecute taints to NoSchedule.
/sig scheduling node
/cc
/remove-sig scheduling scheduler isn't involved in this case.
Which SIG owns the controller-manager or is the node lifecycle manager (and its NoExecuteTaintManager) owned by sig-node alone?
I think node lifecycle is jointly owned by sig-node and sig-cloud-provider
API machinery owns the mechanics of the controller manager (controller loop setup and management), but the individual controllers are SIG-specific
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Looks like we lost some momentum here and I'd like to avoid this falling into the abyss of bugs that never get fixed. When this bug is triggered the outcome is pretty terrible and once it does happen it is very difficult to dig into and understand what actually happened. Did we ever get this routed to the right owner?
adding sig-node leads for triage/routing. this seems like a clearly reproducible bug with significant correctness issues. /assign
thanks for the detailed report and reproducer. to help accelerate this, do you have a form of your reproducer we could add in the e2e suite (even if its failing initially)?
custom taints are considered disruptive, so it couldn't run in the main suite
It's been a while since I wrote tests for the e2e suite but if you are available over slack in case guidance is needed, I'm willing to put the effort.
Do all suites need to pass for a fix to be merged? I'd assume it is the case, otherwise how would a regression be prevented after the fix lands?


Question: minTolerationTime := getMinTolerationTime(usedTolerations) // getMinTolerationTime returns negative value to denote infinite toleration. if minTolerationTime < 0 { <del> klog.V(4).Infof("New tolerations for %v tolerate forever. Scheduled deletion won't be cancelled if already scheduled.", podNamespacedName.String()) </del> <ins> klog.V(4).Infof("Current tolerations for %v tolerate forever, cancelling any scheduled deletion.", podNamespacedName.String()) tc.cancelWorkWithEvent(podNamespacedName) </ins> return }

i was more inquiring if there was a test that could automate the reproduction, so even just a linked commit may be helpful for someone that wants to pick this up. i am happy to help if you want to reach out over slack later this week.
looks like a unit test could reproduce the issue, opened a WIP fix in
our code is not publicly accessible, I'm sorry. PR linked by seems to cover both the fix and the test that proves it.
/remove-lifecycle frozen
frozen keeps the bot from auto-closing
Ah, sorry then.


Question: for idx, watchNamespace := range watchNamespaces { i := idx w, err := o.Watch(testResource, watchNamespace) <ins> if err != nil { t.Fatalf("test resource watch failed in %s: %v", watchNamespace, err) } </ins> go func() { assert.NoError(t, err, "watch invocation failed") for _, c := range cases {

What happened: What you expected to happen: I am not sure if is it a bug, I think we should catch err when Watch testResource failed in func TestWatchCallNonNamespace. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
Absolutely we should check err gracefully.
/sig cli
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/fixes it


Question: CudaVectorAdd // CudaVectorAdd2 image CudaVectorAdd2 <del> // DebianIptables Image DebianIptables </del> <ins> // DistrolessIptables Image DistrolessIptables </ins> // Etcd image Etcd // GlusterDynamicProvisioner image

Istio folks have a iptables image here that is quite small In the past, we did talk about this, for example wrote it well what we could try: We could even do for running the hack script we have for switching the symlinks when vs : kube-proxy image is definitely one of the main causes of the cost we incur in our GCR repository, so we should definitely give this a shot again. /sig network
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I don't think this is a sig/network-only issue. A gives; The looks like this (narroved): If I don't misinterpret this, K8s uses a unified server-image build and it should not be different for . The size of the binary for amd64 in is; It is stripped and can't easily be "made smaller" I guess.
We (Istio) would definitely be interested in sharing ownership (hopefully by upsteaming to distroless mainline) in the image we have if there is interest. Some other projects have expressed similar needs as well
!
kube-proxy make up 9% of current image data traffic
I threw together this little repo (based on our next-gen distroless infrastructure): It pushed which on x64 is roughly another 5x smaller (musl vs. glibc): If folks are interested in this, I'd be happy to create a repo under which is where we've been standing up new images, or you can create your own from pretty easily.
debian based please!! we have KEPs and stuff written that lay down that direction:
For a possible approach + above in that issue and for "distroless" debian based images with arbitrary C/C++ binaries + deps. Note that we need things besides just the distro binaries (see comment linked). and is pretty straightforward for one way to handle the debian binaries part.
I don't even know how we build the kube-proxy image. Where does that happen and form which dockerfile?


Question: CudaVectorAdd // CudaVectorAdd2 image CudaVectorAdd2 <del> // DebianIptables Image DebianIptables </del> <ins> // DistrolessIptables Image DistrolessIptables </ins> // Etcd image Etcd // GlusterDynamicProvisioner image

Note we need iptables, iptables-save and iptables-restore but also ipvsadm, shell, conntrack, ipset, and maybe other tools
It used to be in kubernetes/kubernetes under build/images, it's now in (Dockerfile etc there)
That's the base - where is the logic to turn that into a kube-proxy image?
Turning that image into a kube-proxy image is just adding the kube-proxy binary with a dockerfile and entirely too much bash which ultimately just does: (except api-server, where we do a setcap in )
/triage accepted Will take a look between this and next week
It doesn't even need bash; the version of the script in intentionally avoids bash-specific features (and gets tested against Alpine where /bin/sh is busybox (which I guess is "ash"?)). And the version in kubernetes/release appears to be just a stripped-down version of that script. (But actually, you should update the script to be based on the latest version anyway, to get the new logic in , in preparation for .)
Probably per one of the things we don't want to do right now is jump into a musl based image like Alpine, right? So I would take the case that we want to use distroless, and copy all the binaries we need to inside that container (conntrack, ipvs, iptables-*, etc etc). per iptables-wrapper, you said it's not bash specific so maybe we can use distroless + the smaller available shell? I will check some approaches here before jumping into changing the release/dockerfiles/whatever scripts, and check if I can run some comformance tests in a distroless k/proxy. Maybe starting with a good binary list is a way to cross check the binaries + libraries required for this distroless container. I did a list with the installed packages per kproxy needs, and some ldd on it: Will start working on something to assemble an image just with the binaries and required libraries, and check size + libraries
I agree no musl/alpine On Sun, Apr 17, 2022, 11:38 AM Ricardo Katz
If you identify the list of binaries, has logic to handle ldd for a binary (e.g. ignore vdso) + keep the copyright and package version info via dpkg.


Question: CudaVectorAdd // CudaVectorAdd2 image CudaVectorAdd2 <del> // DebianIptables Image DebianIptables </del> <ins> // DistrolessIptables Image DistrolessIptables </ins> // Etcd image Etcd // GlusterDynamicProvisioner image

We probably need a few more not in the list above for the other proxy modes?
Thanks this is gonna be very helpful! I will work on it using the script you provided during the week!
/cc
From script and some changes: There's a pretty good reduction of size (50%). But this is without kproxy binary, that is almost 40Mb. I will open a Draft PR in k/release so we can discuss over there the approach. BTW, anyone knows why kube-proxy binary is that big? Never digged into that, but curious about it.
https://dr- !
(updated with conntrack-tools) Why? The binary is statically linked so it does not need or use , it only needs some binaries to be executable and it doesn't matter if they are built with musl. They just have to work. I have sucessfully built and executed my own tests, both with proxy-mode and , using an image built with this : The image size on cluster including the binary (~40MB) is; (I use a private local registry) Compare to;
what about the different iptables-nft vs iptables-legacy binaries?
I have assumed that takes care of that (executed by the Dockerfile).
Re size: Are DWARF2 symbols stripped? Ideally k8s should produce artifacts without DWARF2 symbols and distribute them separatelly. Or produce the optional -debug artifacts. This can cut GCR/GCS costs, but introduce some debug/profiling complexity. The Go symbol table should be always included. Related
IIRC wrapper in prod needs both binaries existing in the container, and will use this in a way to check if the host machine is using iptables-legacy or iptables-nft.
Both exist in Alpine; The first example in is Alpine. can you please confirm that it works with Alpine?
I have altered my testenv so I can switch between legacy and nft and verified that the tests work with both and that shows the same in the container and the K8s node, e.g;
Side remark; to my suprise is not needed in the kube-proxy image
Tested the with K8s e2e;
There are / we're legal reasons to avoid Alpine. Have those been resolved? I haven't heard anything.