Question:
I have a cronjob that runs a Python script every 5 minutes. After the cronjob finishes running, I start seeing a bunch of log messages in my systemd journal like the following: Those directories in /var/log don't exist as they belong to old pods that have finished running. These messages keep increasing in volume as time passes (almost received a 1000 of those messages at one time, after which I had to reboot the node to fix the issue). It almost looks like the kubelet is not cleaning up the old pods for some reason. I can't see these old pods when I run , so I'm assuming their is some issue with the kubelet's cleanup process. kubelet (or cadvisor) shouldn't try to read logs for pods that don't exist. Setup a short lived cronjob running every 5 minutes and observe the logs (Kubernetes v1.22.4, CRI-O v1.22.1, Ubuntu 20.04) No response <details</detailsBaremetal cluster <details</details<detailskubeadm (baremetal cluster) </details<detailsCRI-O v1.22.1 </details<details</details
/sig node
/assign for triage
I have something similar on a fresh cluster I use for educational purposes using kubernetes 22.5 with cri-o. Here's an excerpt: This is from the master node. Nodes run Debian bulsseye in KVM provisioned with Vagrant. I have not yet tested it out with different versions. If I can assist in testing, I'd be happy to do so!
/triage accepted /priority important-longterm
I've produced the same issue on my clusters. I'm running Debian 11 on bare metal (homelab) and in VM (test). My clusters are installed using kubeadm, and I'm using multus and calico as CNI. The issue is triggered as soon as a pod is removed. I've triggered it by deleting a simple Pod, scaling down a Deployment, or running a CronJob. On the node where the pod was removed, the logs are then flooded by messages, at the rate of 1 line per pod removed every 10 seconds. I've seen it both on aregular worker nodes, and untainted master nodes. I've tested under Vagrant with various K8S / CRI-O versions. With CRI-O , I never triggered the issue with Kubernetes : , and . However, with CRI-O = , all my tests triggered the issue : Kubernetes , CRI-O Kubernetes , CRI-O Kubernetes , CRI-O Let me know if I can help by running other tests or capturing specific logs.
Is this only issue with CRI-O? What about containerd? The reports above seem to all be on CRI-O? /cc
openshift seems to have hit this case a couple of times and digging into some bugs leads me to think it will be fixed with and kube 1.24. can you try one of the kube alpha releases? cri-o 1.23.1 should do, or you can try the main branch
it's possible it's unrelated, but is why I am making this guess
I'm used to installing a cluster using Debian packages, so it took me a few tries to have a working v1.24 cluster. I followed the "Without a package manager" to install from the binaries. I used Kubernetes and CRI-O . I'm not certain everything is working fine, but I can schedule a Pod / Deployment / Job, so I assume it's good enough to test this issue. And I still reproduce the issue as soon as I remove a Pod.
I'm also hitting the issue in my lab environment(Vagrant). Running Rocky Linux 8.5 Kubernetes 1.22.6 and CRI-O 1.22.2
ah hah! I found a fix (and found out why it's only being hit in cri-o and not cadvisor)
We're also having this issue on a cluster hosting Gitlab CI jobs, this is quite a trouble since many jobs are spawned and removed daily. Kubernetes version: v1.22.7 CRI-O version: 1.22.2 Is there any news about the fix ? I looked into 's PR, it seemed ok to me but I'm not really experienced in Go... Nevertheless I'd be happy to help
up
I can confirm on a bare metal cluster that I'm running on containerd that I am also seeing these log messages.
Observing similar issues on (k8s and crio ) and (k8s and crio ). The strange thing is that one cluster just some nodes affected while the other, all of them. Below an overview of 30m occurrences of Cluster running version : Cluster running version : No idea where the discrepancy is coming from.
Tested with and on a fresh cluster bootstraped with kubeadm, all machines run on Debian 11. Ran a job 3 times and there's a log line for every execution, so it spams 3 log lines per kubelet loop.
It looks like cgroups are left behind. Removing them fixes the logspam (temporarily). Can use something like this to find all of them, depending on actual error in journal: Then iterate over the and remove them, while checking that running pods are not considered: It's quick and dirty, but does the job. I've deployed a garbage collector daemonset with the above and until this is properly addressed, I'm quite happy with the fix. The problem is, for a medium to large size cluster with pods coming and going, it's very easy to bust your elasticsearch storage and performance with this log spam.
Yesterday we tested with the newly released version of and the problem was still here. After reading answer I digged a little bit because we use systemd to manage cgroups and it seemed the error was on our side. I updated the kernel boot command in the grub file because in order to have cgroupsv2 unified behavior in systemd you have to set both options: and and we only had : Used the following command for the changes to take effect: And then rebooted the node. At first it seemed like this did the trick as I popped a job on the node I changed and the kubelet didn't spam the logfile anymore. As soon as I deleted the pod from the cluster the kubelet started spamming the logs again with: So this did not resolve the issue. This was tested on a node running , , and .
Looks like the cgroup slices not being cleaned up by systemd is an old issue:
would you mind giving a try to see if it helps?
while your fix does make sense, the root cause (which atm is somewhat unknown) is not really addressed. Something's leaking cgroup slices... and have both been closed off.
I am not sure if they're leaking as much as have yet to be cleaned up. Have those pods been deleted by the kubelet yet? and by deleted, I mean fully removed (no traces if searched by ). The kubelet will clean up the pod scope when GC kicks in, but will keep them around until then to allow for forensics and inspection to be performed. Since your script uses to find the scopes, I am guessing the pod themselves have yet to be GC'd, and should still exist. Which is why I think that is the correct solution: we expect those scopes to be there, but shouldn't unnecessarily log an error if a pod that's stopped but not yet removed is found
to verify my suspicion, I would be interested in seeing for all the pods that are showing up in your script, as well as to cross reference pods to containers, if you wouldn't mind uploading that somewhere.
I'm sure they're not in crio anymore. I'm only using to ensure I'm never going to touch a scope that's reported under a pod still seen by crio. I'll scale down the DaemonSet on my test cluster, run some workload to generate the leaking pods and then provide you some log snippets and requested output from crictl.
I took one of my clusters that's suffering heavily on this issue (hosting gitlab CI jobs), reactivated the logs on it for a second, doesn't take long for the kubelet to log things like this: Trying to inspect the pod with returns: I tested with a few pods and the result is always the same. This seems logic as gives me the following: But the slices are still here tho: So in my opinion crictl return the right things, it's either the kubelet that doesn't update correctly the list of pods to watch or the systemd problem with the slices.
I've attached the logs and requested outputs here (sanitized hostnames). Let me know if you'd need anything else; for now I'll leave them around and not scale up the DS on the test cluster.
I've posted a quick gist with all the manifests I'm using to deploy a garbage collector daemon set. Nothing fancy, just a bit of patchwork to address the issues I was having.
Here's another really interesting bit of information. In my case, this issue seems to be only affecting VMs and not bare-metal nodes. It's completely puzzling, but I have 12 bare-metal nodes and 1 VM node in one cluster (the VM is just for testing) and it manifests only on the VM node. As well, my test cluster is made up fully of VMs (ie no bare-metal) and there it happens across the board. The bare metal nodes where the VMs are running are the same are same hardware as the bare-metal nodes of my main cluster (just more NVMe drives). Edit: amended to clarify this is observed in my case running OKD.
Sorry but I have the same issue on bare-metal nodes (x86 ArchLinux kernel 5.17.5 with K8S 1.23.6/Crio 1.23.2)
The issue in title happens on bare metal nodes too.
I've got the same issue on bare-metal nodes(k8s v1.23.0/cri-o v1.22.3) and I'm not sure if it's for the same reason, but I have this problem as well.
I figured out the reason for the discrepancy. The VM node was using cgroups v2, while the bare-metal, older nodes, had the kernel parameter . Removing that and rebooting, triggers the problem.
Did some testing and I can confirm that switching from CRI-O to containerd has stopped these log messages from popping up all the time. For now, this seems like a reasonable solution, at least for me. Using Cgroups v2 Kubernetes Version: v1.23.7 OS: Ubuntu 22.04 Containerd Version: v1.6.4 Runc Version: 1.1.1 EDIT: Kubernetes Version was v1.23, not v1.24
Another option is one could switch to the CRI stats provider by setting kubelet's containerruntimeendpoint to instead of
Is this also an option for OKD/OCP? Is there any progress on a fix or should we look at the above workaround instead?
Out of curiosity, why does this work? /cc
This would work for OKD, but wouldn't be supported for OCP OOTB. should fix, but it needs a reviewer/approver... Fun kubernetes fact! The CRI-O team found performance issues with the CRI stats provider, so there's a hardcoded check to see if the socket is . However, since is symlinked to , and the check is a string comparison, one can use and not trigger that toggle, turning on CRI stats provider. We're working on fixing the perf troubles with CRI stats KEP 2371
Using doesn't solve the problem on my environment, we're still seeing the same error logs. We're running: Flatcar 3033.2.4 Kubelet 1.22.4 CRI-O 1.22.5 cgroup v2 enabled.
I had this error on Kubernetes version 1.24, with CRI-O 1.24. Today I did a fresh install with Kubernetes 1.25 + CRI-O 1.25 (with the same machine types and OS versions) and the error went away. I tried deployment scale-down, pod deletion, but I no longer see the mentioned "Unable to fetch log stats" messages in the log. VirtualBox machines OS: RHEL 9 with kernel 5.14.0-70.22.1.el90.x8664 Kubernetes version: 1.25.0 CRI-O version: 1.25.0 Runc version 1.1.3 Using cgroups v2
We are running kubernetes 1.24.9 with cri-o 1.24.4 and runc 1.1.4 on Ubuntu 22.04 with cgroups v2 and experienced the same issue. For some time I was running a deployment with a dozen of pods and rotated them every 10 seconds to create a lot of garbage in cgroups. This caused rather slow increase in both mem & cpu usage and a lot of error messages like : Kubelet mem & cpu usage: ! ! After I stopped generating those pods the resource usage stopped. After running script from the CPU&mem usage dropped.

} } <del> func (p *criStatsProvider) makePodStorageStats(s *statsapi.PodStats, rootFsInfo *cadvisorapiv2.FsInfo) { podNs := s.PodRef.Namespace podName := s.PodRef.Name podUID := types.UID(s.PodRef.UID) vstats, found := p.resourceAnalyzer.GetPodVolumeStats(podUID) if !found { return } logStats, err := p.hostStatsProvider.getPodLogStats(podNs, podName, podUID, rootFsInfo) if err != nil { klog.ErrorS(err, "Unable to fetch pod log stats", "pod", klog.KRef(podNs, podName)) // If people do in-place upgrade, there might be pods still using // the old log path. For those pods, no pod log stats is returned. // We should continue generating other stats in that case. // calcEphemeralStorage tolerants logStats == nil. } etcHostsStats, err := p.hostStatsProvider.getPodEtcHostsStats(podUID, rootFsInfo) if err != nil { klog.ErrorS(err, "Unable to fetch pod etc hosts stats", "pod", klog.KRef(podNs, podName)) } ephemeralStats := make([]statsapi.VolumeStats, len(vstats.EphemeralVolumes)) copy(ephemeralStats, vstats.EphemeralVolumes) s.VolumeStats = append(append([]statsapi.VolumeStats{}, vstats.EphemeralVolumes...), vstats.PersistentVolumes...) s.EphemeralStorage = calcEphemeralStorage(s.Containers, ephemeralStats, rootFsInfo, logStats, etcHostsStats, true) } </del> func (p *criStatsProvider) addPodNetworkStats( ps *statsapi.PodStats, podSandboxID string,


Question:
I have a cronjob that runs a Python script every 5 minutes. After the cronjob finishes running, I start seeing a bunch of log messages in my systemd journal like the following: Those directories in /var/log don't exist as they belong to old pods that have finished running. These messages keep increasing in volume as time passes (almost received a 1000 of those messages at one time, after which I had to reboot the node to fix the issue). It almost looks like the kubelet is not cleaning up the old pods for some reason. I can't see these old pods when I run , so I'm assuming their is some issue with the kubelet's cleanup process. kubelet (or cadvisor) shouldn't try to read logs for pods that don't exist. Setup a short lived cronjob running every 5 minutes and observe the logs (Kubernetes v1.22.4, CRI-O v1.22.1, Ubuntu 20.04) No response <details</detailsBaremetal cluster <details</details<detailskubeadm (baremetal cluster) </details<detailsCRI-O v1.22.1 </details<details</details
/sig node
/assign for triage
I have something similar on a fresh cluster I use for educational purposes using kubernetes 22.5 with cri-o. Here's an excerpt: This is from the master node. Nodes run Debian bulsseye in KVM provisioned with Vagrant. I have not yet tested it out with different versions. If I can assist in testing, I'd be happy to do so!
/triage accepted /priority important-longterm
I've produced the same issue on my clusters. I'm running Debian 11 on bare metal (homelab) and in VM (test). My clusters are installed using kubeadm, and I'm using multus and calico as CNI. The issue is triggered as soon as a pod is removed. I've triggered it by deleting a simple Pod, scaling down a Deployment, or running a CronJob. On the node where the pod was removed, the logs are then flooded by messages, at the rate of 1 line per pod removed every 10 seconds. I've seen it both on aregular worker nodes, and untainted master nodes. I've tested under Vagrant with various K8S / CRI-O versions. With CRI-O , I never triggered the issue with Kubernetes : , and . However, with CRI-O = , all my tests triggered the issue : Kubernetes , CRI-O Kubernetes , CRI-O Kubernetes , CRI-O Let me know if I can help by running other tests or capturing specific logs.
Is this only issue with CRI-O? What about containerd? The reports above seem to all be on CRI-O? /cc
openshift seems to have hit this case a couple of times and digging into some bugs leads me to think it will be fixed with and kube 1.24. can you try one of the kube alpha releases? cri-o 1.23.1 should do, or you can try the main branch
it's possible it's unrelated, but is why I am making this guess
I'm used to installing a cluster using Debian packages, so it took me a few tries to have a working v1.24 cluster. I followed the "Without a package manager" to install from the binaries. I used Kubernetes and CRI-O . I'm not certain everything is working fine, but I can schedule a Pod / Deployment / Job, so I assume it's good enough to test this issue. And I still reproduce the issue as soon as I remove a Pod.
I'm also hitting the issue in my lab environment(Vagrant). Running Rocky Linux 8.5 Kubernetes 1.22.6 and CRI-O 1.22.2
ah hah! I found a fix (and found out why it's only being hit in cri-o and not cadvisor)
We're also having this issue on a cluster hosting Gitlab CI jobs, this is quite a trouble since many jobs are spawned and removed daily. Kubernetes version: v1.22.7 CRI-O version: 1.22.2 Is there any news about the fix ? I looked into 's PR, it seemed ok to me but I'm not really experienced in Go... Nevertheless I'd be happy to help
up
I can confirm on a bare metal cluster that I'm running on containerd that I am also seeing these log messages.
Observing similar issues on (k8s and crio ) and (k8s and crio ). The strange thing is that one cluster just some nodes affected while the other, all of them. Below an overview of 30m occurrences of Cluster running version : Cluster running version : No idea where the discrepancy is coming from.
Tested with and on a fresh cluster bootstraped with kubeadm, all machines run on Debian 11. Ran a job 3 times and there's a log line for every execution, so it spams 3 log lines per kubelet loop.
It looks like cgroups are left behind. Removing them fixes the logspam (temporarily). Can use something like this to find all of them, depending on actual error in journal: Then iterate over the and remove them, while checking that running pods are not considered: It's quick and dirty, but does the job. I've deployed a garbage collector daemonset with the above and until this is properly addressed, I'm quite happy with the fix. The problem is, for a medium to large size cluster with pods coming and going, it's very easy to bust your elasticsearch storage and performance with this log spam.
Yesterday we tested with the newly released version of and the problem was still here. After reading answer I digged a little bit because we use systemd to manage cgroups and it seemed the error was on our side. I updated the kernel boot command in the grub file because in order to have cgroupsv2 unified behavior in systemd you have to set both options: and and we only had : Used the following command for the changes to take effect: And then rebooted the node. At first it seemed like this did the trick as I popped a job on the node I changed and the kubelet didn't spam the logfile anymore. As soon as I deleted the pod from the cluster the kubelet started spamming the logs again with: So this did not resolve the issue. This was tested on a node running , , and .
Looks like the cgroup slices not being cleaned up by systemd is an old issue:
would you mind giving a try to see if it helps?
while your fix does make sense, the root cause (which atm is somewhat unknown) is not really addressed. Something's leaking cgroup slices... and have both been closed off.
I am not sure if they're leaking as much as have yet to be cleaned up. Have those pods been deleted by the kubelet yet? and by deleted, I mean fully removed (no traces if searched by ). The kubelet will clean up the pod scope when GC kicks in, but will keep them around until then to allow for forensics and inspection to be performed. Since your script uses to find the scopes, I am guessing the pod themselves have yet to be GC'd, and should still exist. Which is why I think that is the correct solution: we expect those scopes to be there, but shouldn't unnecessarily log an error if a pod that's stopped but not yet removed is found
to verify my suspicion, I would be interested in seeing for all the pods that are showing up in your script, as well as to cross reference pods to containers, if you wouldn't mind uploading that somewhere.
I'm sure they're not in crio anymore. I'm only using to ensure I'm never going to touch a scope that's reported under a pod still seen by crio. I'll scale down the DaemonSet on my test cluster, run some workload to generate the leaking pods and then provide you some log snippets and requested output from crictl.
I took one of my clusters that's suffering heavily on this issue (hosting gitlab CI jobs), reactivated the logs on it for a second, doesn't take long for the kubelet to log things like this: Trying to inspect the pod with returns: I tested with a few pods and the result is always the same. This seems logic as gives me the following: But the slices are still here tho: So in my opinion crictl return the right things, it's either the kubelet that doesn't update correctly the list of pods to watch or the systemd problem with the slices.
I've attached the logs and requested outputs here (sanitized hostnames). Let me know if you'd need anything else; for now I'll leave them around and not scale up the DS on the test cluster.
I've posted a quick gist with all the manifests I'm using to deploy a garbage collector daemon set. Nothing fancy, just a bit of patchwork to address the issues I was having.
Here's another really interesting bit of information. In my case, this issue seems to be only affecting VMs and not bare-metal nodes. It's completely puzzling, but I have 12 bare-metal nodes and 1 VM node in one cluster (the VM is just for testing) and it manifests only on the VM node. As well, my test cluster is made up fully of VMs (ie no bare-metal) and there it happens across the board. The bare metal nodes where the VMs are running are the same are same hardware as the bare-metal nodes of my main cluster (just more NVMe drives). Edit: amended to clarify this is observed in my case running OKD.
Sorry but I have the same issue on bare-metal nodes (x86 ArchLinux kernel 5.17.5 with K8S 1.23.6/Crio 1.23.2)
The issue in title happens on bare metal nodes too.
I've got the same issue on bare-metal nodes(k8s v1.23.0/cri-o v1.22.3) and I'm not sure if it's for the same reason, but I have this problem as well.
I figured out the reason for the discrepancy. The VM node was using cgroups v2, while the bare-metal, older nodes, had the kernel parameter . Removing that and rebooting, triggers the problem.
Did some testing and I can confirm that switching from CRI-O to containerd has stopped these log messages from popping up all the time. For now, this seems like a reasonable solution, at least for me. Using Cgroups v2 Kubernetes Version: v1.23.7 OS: Ubuntu 22.04 Containerd Version: v1.6.4 Runc Version: 1.1.1 EDIT: Kubernetes Version was v1.23, not v1.24
Another option is one could switch to the CRI stats provider by setting kubelet's containerruntimeendpoint to instead of
Is this also an option for OKD/OCP? Is there any progress on a fix or should we look at the above workaround instead?
Out of curiosity, why does this work? /cc
This would work for OKD, but wouldn't be supported for OCP OOTB. should fix, but it needs a reviewer/approver... Fun kubernetes fact! The CRI-O team found performance issues with the CRI stats provider, so there's a hardcoded check to see if the socket is . However, since is symlinked to , and the check is a string comparison, one can use and not trigger that toggle, turning on CRI stats provider. We're working on fixing the perf troubles with CRI stats KEP 2371
Using doesn't solve the problem on my environment, we're still seeing the same error logs. We're running: Flatcar 3033.2.4 Kubelet 1.22.4 CRI-O 1.22.5 cgroup v2 enabled.
I had this error on Kubernetes version 1.24, with CRI-O 1.24. Today I did a fresh install with Kubernetes 1.25 + CRI-O 1.25 (with the same machine types and OS versions) and the error went away. I tried deployment scale-down, pod deletion, but I no longer see the mentioned "Unable to fetch log stats" messages in the log. VirtualBox machines OS: RHEL 9 with kernel 5.14.0-70.22.1.el90.x8664 Kubernetes version: 1.25.0 CRI-O version: 1.25.0 Runc version 1.1.3 Using cgroups v2
We are running kubernetes 1.24.9 with cri-o 1.24.4 and runc 1.1.4 on Ubuntu 22.04 with cgroups v2 and experienced the same issue. For some time I was running a deployment with a dozen of pods and rotated them every 10 seconds to create a lot of garbage in cgroups. This caused rather slow increase in both mem & cpu usage and a lot of error messages like : Kubelet mem & cpu usage: ! ! After I stopped generating those pods the resource usage stopped. After running script from the CPU&mem usage dropped.

cadvisorapiv1 "github.com/google/cadvisor/info/v1" cadvisorapiv2 "github.com/google/cadvisor/info/v2" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" <ins> "k8s.io/apimachinery/pkg/types" </ins> "k8s.io/klog/v2" statsapi "k8s.io/kubelet/pkg/apis/stats/v1alpha1" "k8s.io/kubernetes/pkg/kubelet/cadvisor" <ins> "k8s.io/kubernetes/pkg/kubelet/server/stats" </ins> ) // defaultNetworkInterfaceName is used for collectng network stats.


Question:
I have a cronjob that runs a Python script every 5 minutes. After the cronjob finishes running, I start seeing a bunch of log messages in my systemd journal like the following: Those directories in /var/log don't exist as they belong to old pods that have finished running. These messages keep increasing in volume as time passes (almost received a 1000 of those messages at one time, after which I had to reboot the node to fix the issue). It almost looks like the kubelet is not cleaning up the old pods for some reason. I can't see these old pods when I run , so I'm assuming their is some issue with the kubelet's cleanup process. kubelet (or cadvisor) shouldn't try to read logs for pods that don't exist. Setup a short lived cronjob running every 5 minutes and observe the logs (Kubernetes v1.22.4, CRI-O v1.22.1, Ubuntu 20.04) No response <details</detailsBaremetal cluster <details</details<detailskubeadm (baremetal cluster) </details<detailsCRI-O v1.22.1 </details<details</details
/sig node
/assign for triage
I have something similar on a fresh cluster I use for educational purposes using kubernetes 22.5 with cri-o. Here's an excerpt: This is from the master node. Nodes run Debian bulsseye in KVM provisioned with Vagrant. I have not yet tested it out with different versions. If I can assist in testing, I'd be happy to do so!
/triage accepted /priority important-longterm
I've produced the same issue on my clusters. I'm running Debian 11 on bare metal (homelab) and in VM (test). My clusters are installed using kubeadm, and I'm using multus and calico as CNI. The issue is triggered as soon as a pod is removed. I've triggered it by deleting a simple Pod, scaling down a Deployment, or running a CronJob. On the node where the pod was removed, the logs are then flooded by messages, at the rate of 1 line per pod removed every 10 seconds. I've seen it both on aregular worker nodes, and untainted master nodes. I've tested under Vagrant with various K8S / CRI-O versions. With CRI-O , I never triggered the issue with Kubernetes : , and . However, with CRI-O = , all my tests triggered the issue : Kubernetes , CRI-O Kubernetes , CRI-O Kubernetes , CRI-O Let me know if I can help by running other tests or capturing specific logs.
Is this only issue with CRI-O? What about containerd? The reports above seem to all be on CRI-O? /cc
openshift seems to have hit this case a couple of times and digging into some bugs leads me to think it will be fixed with and kube 1.24. can you try one of the kube alpha releases? cri-o 1.23.1 should do, or you can try the main branch
it's possible it's unrelated, but is why I am making this guess
I'm used to installing a cluster using Debian packages, so it took me a few tries to have a working v1.24 cluster. I followed the "Without a package manager" to install from the binaries. I used Kubernetes and CRI-O . I'm not certain everything is working fine, but I can schedule a Pod / Deployment / Job, so I assume it's good enough to test this issue. And I still reproduce the issue as soon as I remove a Pod.
I'm also hitting the issue in my lab environment(Vagrant). Running Rocky Linux 8.5 Kubernetes 1.22.6 and CRI-O 1.22.2
ah hah! I found a fix (and found out why it's only being hit in cri-o and not cadvisor)
We're also having this issue on a cluster hosting Gitlab CI jobs, this is quite a trouble since many jobs are spawned and removed daily. Kubernetes version: v1.22.7 CRI-O version: 1.22.2 Is there any news about the fix ? I looked into 's PR, it seemed ok to me but I'm not really experienced in Go... Nevertheless I'd be happy to help
up
I can confirm on a bare metal cluster that I'm running on containerd that I am also seeing these log messages.
Observing similar issues on (k8s and crio ) and (k8s and crio ). The strange thing is that one cluster just some nodes affected while the other, all of them. Below an overview of 30m occurrences of Cluster running version : Cluster running version : No idea where the discrepancy is coming from.
Tested with and on a fresh cluster bootstraped with kubeadm, all machines run on Debian 11. Ran a job 3 times and there's a log line for every execution, so it spams 3 log lines per kubelet loop.
It looks like cgroups are left behind. Removing them fixes the logspam (temporarily). Can use something like this to find all of them, depending on actual error in journal: Then iterate over the and remove them, while checking that running pods are not considered: It's quick and dirty, but does the job. I've deployed a garbage collector daemonset with the above and until this is properly addressed, I'm quite happy with the fix. The problem is, for a medium to large size cluster with pods coming and going, it's very easy to bust your elasticsearch storage and performance with this log spam.
Yesterday we tested with the newly released version of and the problem was still here. After reading answer I digged a little bit because we use systemd to manage cgroups and it seemed the error was on our side. I updated the kernel boot command in the grub file because in order to have cgroupsv2 unified behavior in systemd you have to set both options: and and we only had : Used the following command for the changes to take effect: And then rebooted the node. At first it seemed like this did the trick as I popped a job on the node I changed and the kubelet didn't spam the logfile anymore. As soon as I deleted the pod from the cluster the kubelet started spamming the logs again with: So this did not resolve the issue. This was tested on a node running , , and .
Looks like the cgroup slices not being cleaned up by systemd is an old issue:
would you mind giving a try to see if it helps?
while your fix does make sense, the root cause (which atm is somewhat unknown) is not really addressed. Something's leaking cgroup slices... and have both been closed off.
I am not sure if they're leaking as much as have yet to be cleaned up. Have those pods been deleted by the kubelet yet? and by deleted, I mean fully removed (no traces if searched by ). The kubelet will clean up the pod scope when GC kicks in, but will keep them around until then to allow for forensics and inspection to be performed. Since your script uses to find the scopes, I am guessing the pod themselves have yet to be GC'd, and should still exist. Which is why I think that is the correct solution: we expect those scopes to be there, but shouldn't unnecessarily log an error if a pod that's stopped but not yet removed is found
to verify my suspicion, I would be interested in seeing for all the pods that are showing up in your script, as well as to cross reference pods to containers, if you wouldn't mind uploading that somewhere.
I'm sure they're not in crio anymore. I'm only using to ensure I'm never going to touch a scope that's reported under a pod still seen by crio. I'll scale down the DaemonSet on my test cluster, run some workload to generate the leaking pods and then provide you some log snippets and requested output from crictl.
I took one of my clusters that's suffering heavily on this issue (hosting gitlab CI jobs), reactivated the logs on it for a second, doesn't take long for the kubelet to log things like this: Trying to inspect the pod with returns: I tested with a few pods and the result is always the same. This seems logic as gives me the following: But the slices are still here tho: So in my opinion crictl return the right things, it's either the kubelet that doesn't update correctly the list of pods to watch or the systemd problem with the slices.
I've attached the logs and requested outputs here (sanitized hostnames). Let me know if you'd need anything else; for now I'll leave them around and not scale up the DS on the test cluster.
I've posted a quick gist with all the manifests I'm using to deploy a garbage collector daemon set. Nothing fancy, just a bit of patchwork to address the issues I was having.
Here's another really interesting bit of information. In my case, this issue seems to be only affecting VMs and not bare-metal nodes. It's completely puzzling, but I have 12 bare-metal nodes and 1 VM node in one cluster (the VM is just for testing) and it manifests only on the VM node. As well, my test cluster is made up fully of VMs (ie no bare-metal) and there it happens across the board. The bare metal nodes where the VMs are running are the same are same hardware as the bare-metal nodes of my main cluster (just more NVMe drives). Edit: amended to clarify this is observed in my case running OKD.
Sorry but I have the same issue on bare-metal nodes (x86 ArchLinux kernel 5.17.5 with K8S 1.23.6/Crio 1.23.2)
The issue in title happens on bare metal nodes too.
I've got the same issue on bare-metal nodes(k8s v1.23.0/cri-o v1.22.3) and I'm not sure if it's for the same reason, but I have this problem as well.
I figured out the reason for the discrepancy. The VM node was using cgroups v2, while the bare-metal, older nodes, had the kernel parameter . Removing that and rebooting, triggers the problem.
Did some testing and I can confirm that switching from CRI-O to containerd has stopped these log messages from popping up all the time. For now, this seems like a reasonable solution, at least for me. Using Cgroups v2 Kubernetes Version: v1.23.7 OS: Ubuntu 22.04 Containerd Version: v1.6.4 Runc Version: 1.1.1 EDIT: Kubernetes Version was v1.23, not v1.24
Another option is one could switch to the CRI stats provider by setting kubelet's containerruntimeendpoint to instead of
Is this also an option for OKD/OCP? Is there any progress on a fix or should we look at the above workaround instead?
Out of curiosity, why does this work? /cc
This would work for OKD, but wouldn't be supported for OCP OOTB. should fix, but it needs a reviewer/approver... Fun kubernetes fact! The CRI-O team found performance issues with the CRI stats provider, so there's a hardcoded check to see if the socket is . However, since is symlinked to , and the check is a string comparison, one can use and not trigger that toggle, turning on CRI stats provider. We're working on fixing the perf troubles with CRI stats KEP 2371
Using doesn't solve the problem on my environment, we're still seeing the same error logs. We're running: Flatcar 3033.2.4 Kubelet 1.22.4 CRI-O 1.22.5 cgroup v2 enabled.
I had this error on Kubernetes version 1.24, with CRI-O 1.24. Today I did a fresh install with Kubernetes 1.25 + CRI-O 1.25 (with the same machine types and OS versions) and the error went away. I tried deployment scale-down, pod deletion, but I no longer see the mentioned "Unable to fetch log stats" messages in the log. VirtualBox machines OS: RHEL 9 with kernel 5.14.0-70.22.1.el90.x8664 Kubernetes version: 1.25.0 CRI-O version: 1.25.0 Runc version 1.1.3 Using cgroups v2
We are running kubernetes 1.24.9 with cri-o 1.24.4 and runc 1.1.4 on Ubuntu 22.04 with cgroups v2 and experienced the same issue. For some time I was running a deployment with a dozen of pods and rotated them every 10 seconds to create a lot of garbage in cgroups. This caused rather slow increase in both mem & cpu usage and a lot of error messages like : Kubelet mem & cpu usage: ! ! After I stopped generating those pods the resource usage stopped. After running script from the CPU&mem usage dropped.

total := *first + *second return &total } <ins> func makePodStorageStats(s *statsapi.PodStats, rootFsInfo *cadvisorapiv2.FsInfo, resourceAnalyzer stats.ResourceAnalyzer, hostStatsProvider HostStatsProvider, isCRIStatsProvider bool) { podNs := s.PodRef.Namespace podName := s.PodRef.Name podUID := types.UID(s.PodRef.UID) var ephemeralStats []statsapi.VolumeStats if vstats, found := resourceAnalyzer.GetPodVolumeStats(podUID); found { ephemeralStats = make([]statsapi.VolumeStats, len(vstats.EphemeralVolumes)) copy(ephemeralStats, vstats.EphemeralVolumes) s.VolumeStats = append(append([]statsapi.VolumeStats{}, vstats.EphemeralVolumes...), vstats.PersistentVolumes...) } logStats, err := hostStatsProvider.getPodLogStats(podNs, podName, podUID, rootFsInfo) if err != nil { klog.V(6).ErrorS(err, "Unable to fetch pod log stats", "pod", klog.KRef(podNs, podName)) // If people do in-place upgrade, there might be pods still using // the old log path. For those pods, no pod log stats is returned. // We should continue generating other stats in that case. // calcEphemeralStorage tolerants logStats == nil. } etcHostsStats, err := hostStatsProvider.getPodEtcHostsStats(podUID, rootFsInfo) if err != nil { klog.V(6).ErrorS(err, "Unable to fetch pod etc hosts stats", "pod", klog.KRef(podNs, podName)) } s.EphemeralStorage = calcEphemeralStorage(s.Containers, ephemeralStats, rootFsInfo, logStats, etcHostsStats, isCRIStatsProvider) } </ins>


Question:
PIDPressure is not set to true when the system has insufficient pid resource. PIDPressure is set to true correctly when the system is insufficient in pid resource. See cause below The max pid is not solely controlled by which is referred by . But it is also controlled by (whichever is smaller). However, they have different semantics resulting in the same constraint to how many processes/threads can be running on the system. <details</details<details</details<details</details<details</details<details</details<details</details
/sig node
I asked the same issue before,
My opinion: the minimum of and is the right number to count, for Linux. That will provide better instrumentation. If you're reserving a number of task IDs (PIDs) for system use, that minimum is the number that you're reserving from.
/triage accepted /priority important-longterm
/retitle System maximum PID inconsiderate of sysctl 'threads-max'
BTW I made a thing that might help folks test a fix: Originally I used it to create PIDs, to test whether apps cope with long PIDs. This version keeps the threads around so that you see them in .

"syscall" "time" <del> "k8s.io/apimachinery/pkg/apis/meta/v1" </del> <ins> v1 "k8s.io/apimachinery/pkg/apis/meta/v1" </ins> statsapi "k8s.io/kubelet/pkg/apis/stats/v1alpha1" )


Question:
PIDPressure is not set to true when the system has insufficient pid resource. PIDPressure is set to true correctly when the system is insufficient in pid resource. See cause below The max pid is not solely controlled by which is referred by . But it is also controlled by (whichever is smaller). However, they have different semantics resulting in the same constraint to how many processes/threads can be running on the system. <details</details<details</details<details</details<details</details<details</details<details</details
/sig node
I asked the same issue before,
My opinion: the minimum of and is the right number to count, for Linux. That will provide better instrumentation. If you're reserving a number of task IDs (PIDs) for system use, that minimum is the number that you're reserving from.
/triage accepted /priority important-longterm
/retitle System maximum PID inconsiderate of sysctl 'threads-max'
BTW I made a thing that might help folks test a fix: Originally I used it to create PIDs, to test whether apps cope with long PIDs. This version keeps the threads around so that you see them in .

func Stats() (*statsapi.RlimitStats, error) { rlimit := &statsapi.RlimitStats{} <del> if content, err := ioutil.ReadFile("/proc/sys/kernel/pid_max"); err == nil { if maxPid, err := strconv.ParseInt(string(content[:len(content)-1]), 10, 64); err == nil { rlimit.MaxPID = &maxPid </del> <ins> taskMax := int64(-1) // Calculate the mininum of kernel.pid_max and kernel.threads-max as they both specify the // system-wide limit on the number of tasks. for _, file := range []string{"/proc/sys/kernel/pid_max", "/proc/sys/kernel/threads-max"} { if content, err := ioutil.ReadFile(file); err == nil { if limit, err := strconv.ParseInt(string(content[:len(content)-1]), 10, 64); err == nil { if taskMax == -1 || taskMax > limit { taskMax = limit } } </ins> } } <ins> // Both reads did not fail. if taskMax >= 0 { rlimit.MaxPID = &taskMax } </ins> var info syscall.Sysinfo_t syscall.Sysinfo(&info)


Question:
PIDPressure is not set to true when the system has insufficient pid resource. PIDPressure is set to true correctly when the system is insufficient in pid resource. See cause below The max pid is not solely controlled by which is referred by . But it is also controlled by (whichever is smaller). However, they have different semantics resulting in the same constraint to how many processes/threads can be running on the system. <details</details<details</details<details</details<details</details<details</details<details</details
/sig node
I asked the same issue before,
My opinion: the minimum of and is the right number to count, for Linux. That will provide better instrumentation. If you're reserving a number of task IDs (PIDs) for system use, that minimum is the number that you're reserving from.
/triage accepted /priority important-longterm
/retitle System maximum PID inconsiderate of sysctl 'threads-max'
BTW I made a thing that might help folks test a fix: Originally I used it to create PIDs, to test whether apps cope with long PIDs. This version keeps the threads around so that you see them in .

type RlimitStats struct { Time metav1.Time `json:"time"` <del> // The max PID of OS. </del> <ins> // The max number of extant process (threads, precisely on Linux) of OS. See RLIMIT_NPROC in getrlimit(2). // The operating system ceiling on the number of process IDs that can be assigned. // On Linux, tasks (either processes or threads) consume 1 PID each. </ins> MaxPID *int64 `json:"maxpid,omitempty"` <del> // The number of running process in the OS. </del> <ins> // The number of running process (threads, precisely on Linux) in the OS. </ins> NumOfRunningProcesses *int64 `json:"curproc,omitempty"` }


Question:
Add more message for no PodSandbox container when kubelet start. When error return, We need more message to solve it, contains container name and container id.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
/assign

containerList, _ := runtimeService.ListContainers(nil) for _, c := range containerList { if _, exists := podSandboxMap[c.PodSandboxId]; !exists { <del> return nil, fmt.Errorf("no PodsandBox found with Id '%s'", c.PodSandboxId) </del> <ins> return nil, fmt.Errorf("no PodsandBox found with Id '%s' for container with ID '%s' and Name '%s'", c.PodSandboxId, c.Id, c.Metadata.Name) </ins> } containerMap.Add(podSandboxMap[c.PodSandboxId], c.Metadata.Name, c.Id) }


Question:
I've been following the official docs for kubeadm and how to set up a new cluster. The cluster is up and running but I noticed warning events when creating node port services: However, the node port does work (I see the output when accessing node ip:nodeport). Might it be related to something else than Kubernetes? There is no such warning event. Set up a fresh cluster with kubeadm and then create a service of nodeport type. No response <details</details<detailsHetzner cloud </details<details</details<detailscloud-init </details<detailscontainerd </details<detailscni=cilium csi=hetzner </details
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
check with netstat or ss if there is a process listening on that port
no, I specified exact random port e.g. and checked it before with ss - it was not being used. Then I created the service, the warning occured, and after that the port was taken:
kube-proxy binds the NodePort to warn users that they may be shadowing a process, but it logs the errors if it can not bind the port
I met the same error and I confirmed the node port was not used by any other process. After looking into the code, I think this is a bug in kube-proxy that it would try to open a node port twice with same (empty) IP and port when is empty. contains and in below code.: I have created to fix it.
Awesome find. Did you figure out a work around? I am encountering this issue with this version of the image:
The error looked scaring but the nodePort should be working, as it was just caused by a duplicate port opening attempt. v1.23.5 and later releases have the fix.

if err != nil { klog.ErrorS(err, "Failed to get node ip address matching nodeport cidrs, services with nodeport may not work as intended", "CIDRs", proxier.nodePortAddresses) } <ins> // nodeAddresses may contain dual-stack zero-CIDRs if proxier.nodePortAddresses is empty. // Ensure nodeAddresses only contains the addresses for this proxier's IP family. isIPv6 := proxier.iptables.IsIPv6() for addr := range nodeAddresses { if utilproxy.IsZeroCIDR(addr) && isIPv6 == netutils.IsIPv6CIDRString(addr) { // if any of the addresses is zero cidr of this IP family, non-zero IPs can be excluded. nodeAddresses = sets.NewString(addr) break } } </ins> // Build rules for each service. for svcName, svc := range proxier.serviceMap {


Question:
I've been following the official docs for kubeadm and how to set up a new cluster. The cluster is up and running but I noticed warning events when creating node port services: However, the node port does work (I see the output when accessing node ip:nodeport). Might it be related to something else than Kubernetes? There is no such warning event. Set up a fresh cluster with kubeadm and then create a service of nodeport type. No response <details</details<detailsHetzner cloud </details<details</details<detailscloud-init </details<detailscontainerd </details<detailscni=cilium csi=hetzner </details
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
check with netstat or ss if there is a process listening on that port
no, I specified exact random port e.g. and checked it before with ss - it was not being used. Then I created the service, the warning occured, and after that the port was taken:
kube-proxy binds the NodePort to warn users that they may be shadowing a process, but it logs the errors if it can not bind the port
I met the same error and I confirmed the node port was not used by any other process. After looking into the code, I think this is a bug in kube-proxy that it would try to open a node port twice with same (empty) IP and port when is empty. contains and in below code.: I have created to fix it.
Awesome find. Did you figure out a work around? I am encountering this issue with this version of the image:
The error looked scaring but the nodePort should be working, as it was just caused by a duplicate port opening attempt. v1.23.5 and later releases have the fix.

// Finally, tail-call to the nodeports chain.  This needs to be after all // other service portal rules. <del> isIPv6 := proxier.iptables.IsIPv6() </del> for address := range nodeAddresses { // TODO(thockin, m1093782566): If/when we have dual-stack support we will want to distinguish v4 from v6 zero-CIDRs. if utilproxy.IsZeroCIDR(address) {


Question:
I've been following the official docs for kubeadm and how to set up a new cluster. The cluster is up and running but I noticed warning events when creating node port services: However, the node port does work (I see the output when accessing node ip:nodeport). Might it be related to something else than Kubernetes? There is no such warning event. Set up a fresh cluster with kubeadm and then create a service of nodeport type. No response <details</details<detailsHetzner cloud </details<details</details<detailscloud-init </details<detailscontainerd </details<detailscni=cilium csi=hetzner </details
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
check with netstat or ss if there is a process listening on that port
no, I specified exact random port e.g. and checked it before with ss - it was not being used. Then I created the service, the warning occured, and after that the port was taken:
kube-proxy binds the NodePort to warn users that they may be shadowing a process, but it logs the errors if it can not bind the port
I met the same error and I confirmed the node port was not used by any other process. After looking into the code, I think this is a bug in kube-proxy that it would try to open a node port twice with same (empty) IP and port when is empty. contains and in below code.: I have created to fix it.
Awesome find. Did you figure out a work around? I am encountering this issue with this version of the image:
The error looked scaring but the nodePort should be working, as it was just caused by a duplicate port opening attempt. v1.23.5 and later releases have the fix.

-A KUBE-SERVICES -m comment --comment "kubernetes service nodeports; NOTE: this must be the last rule in this chain" -m addrtype --dst-type LOCAL -j KUBE-NODEPORTS COMMIT ` <ins> assert.Equal(t, []*netutils.LocalPort{ { Description: "nodePort for ns1/svc1:p80", IP:          "", IPFamily:    netutils.IPv4, Port:        svcNodePort, Protocol:    netutils.TCP, }, }, fp.portMapper.(*fakePortOpener).openPorts) </ins> assertIPTablesRulesEqual(t, expected, fp.iptablesData.String()) }


Question:
kube-controller-manager yields whole stack trace when it loses leaderelection. kube-controller-manager to only log and to exit (without logging 17K lines of stack trace that is not useful for anyone). kube-controller-manager yields around 17K lines of stack trace (omitted in the above example). No response <detailsv1.20.12 </details<details</details<details</details<details</details<details</details<details</details
/sig api-machinery
/assign /triage accepted
has Fatalf always triggered a stack dump of all goroutines?
/cc
Seems like does a dump of all goroutines Could we and then maybe? /cc in case you have thoughts about this
If I read the right, it only printed the current goroutine to stderr. Dumping everything seems to have been later. make sense to me.
Dumping of all goroutines was in
I have similar doubts. I want to fix it. /assign
kube-scheduler seems to have a similar issue (but a shorter stack trace)
When replacing (useful because it doesn't do structured logging, not just because of the stack trace problem), remember to use + the new . Updated migration instructions are pending in There's also - reverting the klog change might be a faster way to address this.

run(ctx, startSATokenController, initializersFunc) }, OnStoppedLeading: func() { <del> klog.Fatalf("leaderelection lost") </del> <ins> klog.ErrorS(nil, "leaderelection lost") klog.FlushAndExit(klog.ExitFlushTimeout, 1) </ins> }, })


Question:
kube-controller-manager yields whole stack trace when it loses leaderelection. kube-controller-manager to only log and to exit (without logging 17K lines of stack trace that is not useful for anyone). kube-controller-manager yields around 17K lines of stack trace (omitted in the above example). No response <detailsv1.20.12 </details<details</details<details</details<details</details<details</details<details</details
/sig api-machinery
/assign /triage accepted
has Fatalf always triggered a stack dump of all goroutines?
/cc
Seems like does a dump of all goroutines Could we and then maybe? /cc in case you have thoughts about this
If I read the right, it only printed the current goroutine to stderr. Dumping everything seems to have been later. make sense to me.
Dumping of all goroutines was in
I have similar doubts. I want to fix it. /assign
kube-scheduler seems to have a similar issue (but a shorter stack trace)
When replacing (useful because it doesn't do structured logging, not just because of the stack trace problem), remember to use + the new . Updated migration instructions are pending in There's also - reverting the klog change might be a faster way to address this.

run(ctx, nil, createInitializersFunc(leaderMigrator.FilterFunc, leadermigration.ControllerMigrated)) }, OnStoppedLeading: func() { <del> klog.Fatalf("migration leaderelection lost") </del> <ins> klog.ErrorS(nil, "migration leaderelection lost") klog.FlushAndExit(klog.ExitFlushTimeout, 1) </ins> }, }) }


Question:
kube-controller-manager yields whole stack trace when it loses leaderelection. kube-controller-manager to only log and to exit (without logging 17K lines of stack trace that is not useful for anyone). kube-controller-manager yields around 17K lines of stack trace (omitted in the above example). No response <detailsv1.20.12 </details<details</details<details</details<details</details<details</details<details</details
/sig api-machinery
/assign /triage accepted
has Fatalf always triggered a stack dump of all goroutines?
/cc
Seems like does a dump of all goroutines Could we and then maybe? /cc in case you have thoughts about this
If I read the right, it only printed the current goroutine to stderr. Dumping everything seems to have been later. make sense to me.
Dumping of all goroutines was in
I have similar doubts. I want to fix it. /assign
kube-scheduler seems to have a similar issue (but a shorter stack trace)
When replacing (useful because it doesn't do structured logging, not just because of the stack trace problem), remember to use + the new . Updated migration instructions are pending in There's also - reverting the klog change might be a faster way to address this.

default: // We lost the lock. klog.ErrorS(nil, "Leaderelection lost") <del> os.Exit(1) </del> <ins> klog.FlushAndExit(klog.ExitFlushTimeout, 1) </ins> } }, }


Question:
kube-controller-manager yields whole stack trace when it loses leaderelection. kube-controller-manager to only log and to exit (without logging 17K lines of stack trace that is not useful for anyone). kube-controller-manager yields around 17K lines of stack trace (omitted in the above example). No response <detailsv1.20.12 </details<details</details<details</details<details</details<details</details<details</details
/sig api-machinery
/assign /triage accepted
has Fatalf always triggered a stack dump of all goroutines?
/cc
Seems like does a dump of all goroutines Could we and then maybe? /cc in case you have thoughts about this
If I read the right, it only printed the current goroutine to stderr. Dumping everything seems to have been later. make sense to me.
Dumping of all goroutines was in
I have similar doubts. I want to fix it. /assign
kube-scheduler seems to have a similar issue (but a shorter stack trace)
When replacing (useful because it doesn't do structured logging, not just because of the stack trace problem), remember to use + the new . Updated migration instructions are pending in There's also - reverting the klog change might be a faster way to address this.

run(ctx, initializers) }, OnStoppedLeading: func() { <del> klog.Fatalf("leaderelection lost") </del> <ins> klog.ErrorS(nil, "leaderelection lost") klog.FlushAndExit(klog.ExitFlushTimeout, 1) </ins> }, })


Question:
kube-controller-manager yields whole stack trace when it loses leaderelection. kube-controller-manager to only log and to exit (without logging 17K lines of stack trace that is not useful for anyone). kube-controller-manager yields around 17K lines of stack trace (omitted in the above example). No response <detailsv1.20.12 </details<details</details<details</details<details</details<details</details<details</details
/sig api-machinery
/assign /triage accepted
has Fatalf always triggered a stack dump of all goroutines?
/cc
Seems like does a dump of all goroutines Could we and then maybe? /cc in case you have thoughts about this
If I read the right, it only printed the current goroutine to stderr. Dumping everything seems to have been later. make sense to me.
Dumping of all goroutines was in
I have similar doubts. I want to fix it. /assign
kube-scheduler seems to have a similar issue (but a shorter stack trace)
When replacing (useful because it doesn't do structured logging, not just because of the stack trace problem), remember to use + the new . Updated migration instructions are pending in There's also - reverting the klog change might be a faster way to address this.

run(ctx, filterInitializers(controllerInitializers, leaderMigrator.FilterFunc, leadermigration.ControllerMigrated)) }, OnStoppedLeading: func() { <del> klog.Fatalf("migration leaderelection lost") </del> <ins> klog.ErrorS(nil, "migration leaderelection lost") klog.FlushAndExit(klog.ExitFlushTimeout, 1) </ins> }, }) }


Question:
/sig api-machinery /kind bug /priority important-soon
It seems something is racing writhing the logs ? cc:
/cc /triage accepted
Failed with this: pull-kubernetes-unit
this looks like a test-only race where the test overwriting the global os.Stderr in order to capture error output races with an async attempt to log to it from another test in the same package
which still needs to be fixed
It seems to me that this bug can not be fixed with as it is, namely used without a logger object or even a . The smallest tweak that occurs to me would be to add an option to to add an output destination, and later remove that addition. Thus, a test could add logging to its own buffer for the duration of running the code under test.
This may be similar to . How can I reproduce this issue?

"crypto/x509" "encoding/json" "fmt" <del> "io" </del> "io/ioutil" "net" "net/http" "net/http/httptest" "net/http/httptrace" <del> "os" </del> "reflect" "strings" "sync"


Question:
/sig api-machinery /kind bug /priority important-soon
It seems something is racing writhing the logs ? cc:
/cc /triage accepted
Failed with this: pull-kubernetes-unit
this looks like a test-only race where the test overwriting the global os.Stderr in order to capture error output races with an async attempt to log to it from another test in the same package
which still needs to be fixed
It seems to me that this bug can not be fixed with as it is, namely used without a logger object or even a . The smallest tweak that occurs to me would be to add an option to to add an output destination, and later remove that addition. Thus, a test could add logging to its own buffer for the duration of running the code under test.
This may be similar to . How can I reproduce this issue?

res.Body.Close() } <del> func captureStdErr() (func() string, func(), error) { </del> <ins> func TestErrConnKilled(t *testing.T) { </ins> var buf bytes.Buffer <del> reader, writer, err := os.Pipe() if err != nil { return nil, nil, err } stderr := os.Stderr readerClosedCh := make(chan struct{}) stopReadingStdErr := func() string { writer.Close() <-readerClosedCh return buf.String() } klog.LogToStderr(true) cleanUp := func() { os.Stderr = stderr klog.LogToStderr(false) stopReadingStdErr() } os.Stderr = writer go func() { io.Copy(&buf, reader) readerClosedCh <- struct{}{} close(readerClosedCh) }() klog.LogToStderr(true) </del> <ins> klog.SetOutput(&buf) klog.LogToStderr(false) defer klog.LogToStderr(true) </ins> <del> return stopReadingStdErr, cleanUp, nil } func TestErrConnKilled(t *testing.T) { readStdErr, cleanUp, err := captureStdErr() if err != nil { t.Fatalf("unable to setup the test, err %v", err) } defer cleanUp() </del> handler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { // this error must be ignored by the WithPanicRecovery handler // it is thrown by WithTimeoutForNonLongRunningRequests handler when a response has been already sent to the client and the handler timed out


Question:
/sig api-machinery /kind bug /priority important-soon
It seems something is racing writhing the logs ? cc:
/cc /triage accepted
Failed with this: pull-kubernetes-unit
this looks like a test-only race where the test overwriting the global os.Stderr in order to capture error output races with an async attempt to log to it from another test in the same package
which still needs to be fixed
It seems to me that this bug can not be fixed with as it is, namely used without a logger object or even a . The smallest tweak that occurs to me would be to add an option to to add an output destination, and later remove that addition. Thus, a test could add logging to its own buffer for the duration of running the code under test.
This may be similar to . How can I reproduce this issue?

ts := httptest.NewServer(WithPanicRecovery(handler, resolver)) defer ts.Close() <del> _, err = http.Get(ts.URL) </del> <ins> _, err := http.Get(ts.URL) </ins> if err == nil { t.Fatal("expected to receive an error") } <del> capturedOutput := readStdErr() </del> <ins> klog.Flush() klog.SetOutput(&bytes.Buffer{}) // prevent further writes into buf capturedOutput := buf.String() </ins> // We don't expect stack trace from the panic to be included in the log. if isStackTraceLoggedByRuntime(capturedOutput) {


Question:
/sig api-machinery /kind bug /priority important-soon
It seems something is racing writhing the logs ? cc:
/cc /triage accepted
Failed with this: pull-kubernetes-unit
this looks like a test-only race where the test overwriting the global os.Stderr in order to capture error output races with an async attempt to log to it from another test in the same package
which still needs to be fixed
It seems to me that this bug can not be fixed with as it is, namely used without a logger object or even a . The smallest tweak that occurs to me would be to add an option to to add an output destination, and later remove that addition. Thus, a test could add logging to its own buffer for the duration of running the code under test.
This may be similar to . How can I reproduce this issue?

// TestErrConnKilledHTTP2 check if HTTP/2 connection is not closed when an HTTP handler panics // The net/http library recovers the panic and sends an HTTP/2 RST_STREAM. func TestErrConnKilledHTTP2(t *testing.T) { <del> readStdErr, cleanUp, err := captureStdErr() if err != nil { t.Fatalf("unable to setup the test, err %v", err) } defer cleanUp() </del> <ins> var buf bytes.Buffer klog.SetOutput(&buf) klog.LogToStderr(false) defer klog.LogToStderr(true) </ins> handler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { // this error must be ignored by the WithPanicRecovery handler // it is thrown by WithTimeoutForNonLongRunningRequests handler when a response has been already sent to the client and the handler timed out


Question:
/sig api-machinery /kind bug /priority important-soon
It seems something is racing writhing the logs ? cc:
/cc /triage accepted
Failed with this: pull-kubernetes-unit
this looks like a test-only race where the test overwriting the global os.Stderr in order to capture error output races with an async attempt to log to it from another test in the same package
which still needs to be fixed
It seems to me that this bug can not be fixed with as it is, namely used without a logger object or even a . The smallest tweak that occurs to me would be to add an option to to add an output destination, and later remove that addition. Thus, a test could add logging to its own buffer for the duration of running the code under test.
This may be similar to . How can I reproduce this issue?

t.Fatal("expected to receive an error") } <del> capturedOutput := readStdErr() </del> <ins> klog.Flush() klog.SetOutput(&bytes.Buffer{}) // prevent further writes into buf capturedOutput := buf.String() </ins> // We don't expect stack trace from the panic to be included in the log. if isStackTraceLoggedByRuntime(capturedOutput) {


Question:
As suggested in , some portions of the code contains the map checking for nil values in map, and others don't have. Even though we're never assigning nil values to this map, we should standardize the portions that does not have this check in module. Example: Should be like this: I can take care of this issue.
/sig network
There have been discussions of deprecating the user-space proxier. Maybe transfer it to . A question is if anybody still uses it. Seems like you do, so if possible please explain if you have a use-case where the user-space proxier is wanted/needed.
Hey Actually, I was casually reading the source code and I saw a over there regarding the nil checks. Regarding your question, we should check if anyone actually uses it though, or maybe discuss its usage or deprecation a little further.
Quick question though: I saw in the news that the user-space proxy porting was near completion, maybe it's worth talking to them in order to check upon the status.
I don't object to making it clearer or more consistent, if you want to spend time on it, but it's not a high priority. That said, I think a nil value in that map would be really bad (and should be impossible) and I don't know how we would raise a signal about it.
let us know if you want to try to fix it, otherwise we can just close this.
I already opened a PR with the clearer nil checks for this specific module in , so you can review it. I think whether this would have nil values into the service map we should discuss this apart and try to reproduce this case (if any would happen).
PR submitted, so let's review and discuss further there. /assign dcbw /triage accepted

ttlSeconds = int(v1.DefaultClientIPServiceAffinitySeconds) //default to 3 hours if not specified.  Should 0 be unlimited instead???? } <del> if _, exists := lb.services[svcPort]; !exists { </del> <ins> if state, exists := lb.services[svcPort]; !exists || state == nil { </ins> lb.services[svcPort] = &balancerState{affinity: *newAffinityPolicy(affinityType, ttlSeconds)} klog.V(4).InfoS("LoadBalancerRR service does not exist, created", "servicePortName", svcPort) } else if affinityType != "" {


Question:
As suggested in , some portions of the code contains the map checking for nil values in map, and others don't have. Even though we're never assigning nil values to this map, we should standardize the portions that does not have this check in module. Example: Should be like this: I can take care of this issue.
/sig network
There have been discussions of deprecating the user-space proxier. Maybe transfer it to . A question is if anybody still uses it. Seems like you do, so if possible please explain if you have a use-case where the user-space proxier is wanted/needed.
Hey Actually, I was casually reading the source code and I saw a over there regarding the nil checks. Regarding your question, we should check if anyone actually uses it though, or maybe discuss its usage or deprecation a little further.
Quick question though: I saw in the news that the user-space proxy porting was near completion, maybe it's worth talking to them in order to check upon the status.
I don't object to making it clearer or more consistent, if you want to spend time on it, but it's not a high priority. That said, I think a nil value in that map would be really bad (and should be impossible) and I don't know how we would raise a signal about it.
let us know if you want to try to fix it, otherwise we can just close this.
I already opened a PR with the clearer nil checks for this specific module in , so you can review it. I think whether this would have nil values into the service map we should discuss this apart and try to reproduce this case (if any would happen).
PR submitted, so let's review and discuss further there. /assign dcbw /triage accepted

lb.lock.RLock() defer lb.lock.RUnlock() state, exists := lb.services[svcPort] <del> // TODO: while nothing ever assigns nil to the map, *some* of the code using the map // checks for it.  The code should all follow the same convention. return exists && state != nil && len(state.endpoints) > 0 </del> <ins> if !exists || state == nil { return false } return len(state.endpoints) > 0 </ins> } // NextEndpoint returns a service endpoint.


Question:
As suggested in , some portions of the code contains the map checking for nil values in map, and others don't have. Even though we're never assigning nil values to this map, we should standardize the portions that does not have this check in module. Example: Should be like this: I can take care of this issue.
/sig network
There have been discussions of deprecating the user-space proxier. Maybe transfer it to . A question is if anybody still uses it. Seems like you do, so if possible please explain if you have a use-case where the user-space proxier is wanted/needed.
Hey Actually, I was casually reading the source code and I saw a over there regarding the nil checks. Regarding your question, we should check if anyone actually uses it though, or maybe discuss its usage or deprecation a little further.
Quick question though: I saw in the news that the user-space proxy porting was near completion, maybe it's worth talking to them in order to check upon the status.
I don't object to making it clearer or more consistent, if you want to spend time on it, but it's not a high priority. That said, I think a nil value in that map would be really bad (and should be impossible) and I don't know how we would raise a signal about it.
let us know if you want to try to fix it, otherwise we can just close this.
I already opened a PR with the clearer nil checks for this specific module in , so you can review it. I think whether this would have nil values into the service map we should discuss this apart and try to reproduce this case (if any would happen).
PR submitted, so let's review and discuss further there. /assign dcbw /triage accepted

} state, exists := lb.services[svcPort] <del> if !exists { </del> <ins> if !exists || state == nil { </ins> return } for _, existingEndpoint := range state.endpoints {


Question:
As suggested in , some portions of the code contains the map checking for nil values in map, and others don't have. Even though we're never assigning nil values to this map, we should standardize the portions that does not have this check in module. Example: Should be like this: I can take care of this issue.
/sig network
There have been discussions of deprecating the user-space proxier. Maybe transfer it to . A question is if anybody still uses it. Seems like you do, so if possible please explain if you have a use-case where the user-space proxier is wanted/needed.
Hey Actually, I was casually reading the source code and I saw a over there regarding the nil checks. Regarding your question, we should check if anyone actually uses it though, or maybe discuss its usage or deprecation a little further.
Quick question though: I saw in the news that the user-space proxy porting was near completion, maybe it's worth talking to them in order to check upon the status.
I don't object to making it clearer or more consistent, if you want to spend time on it, but it's not a high priority. That said, I think a nil value in that map would be really bad (and should be impossible) and I don't know how we would raise a signal about it.
let us know if you want to try to fix it, otherwise we can just close this.
I already opened a PR with the clearer nil checks for this specific module in , so you can review it. I think whether this would have nil values into the service map we should discuss this apart and try to reproduce this case (if any would happen).
PR submitted, so let's review and discuss further there. /assign dcbw /triage accepted

func (lb *LoadBalancerRR) resetService(svcPort proxy.ServicePortName) { // If the service is still around, reset but don't delete. <del> if state, ok := lb.services[svcPort]; ok { </del> <ins> if state, ok := lb.services[svcPort]; ok && state != nil { </ins> if len(state.endpoints) > 0 { klog.V(2).InfoS("LoadBalancerRR: Removing endpoints service", "servicePortName", svcPort) state.endpoints = []string{}


Question:
As suggested in , some portions of the code contains the map checking for nil values in map, and others don't have. Even though we're never assigning nil values to this map, we should standardize the portions that does not have this check in module. Example: Should be like this: I can take care of this issue.
/sig network
There have been discussions of deprecating the user-space proxier. Maybe transfer it to . A question is if anybody still uses it. Seems like you do, so if possible please explain if you have a use-case where the user-space proxier is wanted/needed.
Hey Actually, I was casually reading the source code and I saw a over there regarding the nil checks. Regarding your question, we should check if anyone actually uses it though, or maybe discuss its usage or deprecation a little further.
Quick question though: I saw in the news that the user-space proxy porting was near completion, maybe it's worth talking to them in order to check upon the status.
I don't object to making it clearer or more consistent, if you want to spend time on it, but it's not a high priority. That said, I think a nil value in that map would be really bad (and should be impossible) and I don't know how we would raise a signal about it.
let us know if you want to try to fix it, otherwise we can just close this.
I already opened a PR with the clearer nil checks for this specific module in , so you can review it. I think whether this would have nil values into the service map we should discuss this apart and try to reproduce this case (if any would happen).
PR submitted, so let's review and discuss further there. /assign dcbw /triage accepted

defer lb.lock.Unlock() state, exists := lb.services[svcPort] <del> if !exists { </del> <ins> if !exists || state == nil { </ins> return } for ip, affinity := range state.affinity.affinityMap {


Question:
As discussed in , Endpoint(Slice) controllers can get stuck if an Endpoints or EndpointSlice resource is stuck in a terminating state. I don't think there's much we can do for Endpoints here since the same name is reused, but EndpointSlice controllers can ignore EndpointSlices in a terminating state. See No response All supported versions Any Any Any Any Any
/sig network
/triage accepted
deleting a service with a finalizer doesn't affect the endpointslice, it doesn't change its status to termination or unready Repro: finalizer to the service and delete it, it keeps waiting until the finalizer is removed with the deletion timestamp set. the endpoint slice associated Do you mean that when the endpoint slice controller detects that the Service is being deleted should switch the endpoint slice status to Terminating? cc:
I don't think we need to do anything special if the Service is waiting to be deleted. There are many legitimate reasons for this to happen, including waiting for infrastructure to be removed. My main concern with a finalizer was that one on an EndpointSlice could cause the controller to get stuck on a Service. We don't have the same risk with a finalizer on a Service - I don't think that would cause the controller to get stuck in any way.
i was unable to repro the issue on my machine: I was monitoring the logs of , and the endpointslice controller never complained about , when I deleted the service. Though, I was able to get the controller to log this error under a different scenario. If we don't explicitly remove the finalizer from the endpointslice, and instead re-create the pod and service like: From what I can understand, because the endpointslice with the finalizer is owned by a service with a different UID than the new endpointslice, it is marked as expected for deletion . The endpointslice controllers calls , which returns true for the endpointslice with the finalizer, because of the above reason. I might be wrong anywhere here, so please let me know if that is the case. As for further action, I'm not sure what can be done here? We could check the endpointslice for finalizers and then skip them . Keen to hear your thoughts about this.

return err } <ins> // Drop EndpointSlices that have been marked for deletion to prevent the controller from getting stuck. endpointSlices = dropEndpointSlicesPendingDeletion(endpointSlices) </ins> if c.endpointSliceTracker.StaleSlices(service, endpointSlices) { return endpointsliceutil.NewStaleInformerCache("EndpointSlice informer cache is out of date") }


Question:
As discussed in , Endpoint(Slice) controllers can get stuck if an Endpoints or EndpointSlice resource is stuck in a terminating state. I don't think there's much we can do for Endpoints here since the same name is reused, but EndpointSlice controllers can ignore EndpointSlices in a terminating state. See No response All supported versions Any Any Any Any Any
/sig network
/triage accepted
deleting a service with a finalizer doesn't affect the endpointslice, it doesn't change its status to termination or unready Repro: finalizer to the service and delete it, it keeps waiting until the finalizer is removed with the deletion timestamp set. the endpoint slice associated Do you mean that when the endpoint slice controller detects that the Service is being deleted should switch the endpoint slice status to Terminating? cc:
I don't think we need to do anything special if the Service is waiting to be deleted. There are many legitimate reasons for this to happen, including waiting for infrastructure to be removed. My main concern with a finalizer was that one on an EndpointSlice could cause the controller to get stuck on a Service. We don't have the same risk with a finalizer on a Service - I don't think that would cause the controller to get stuck in any way.
i was unable to repro the issue on my machine: I was monitoring the logs of , and the endpointslice controller never complained about , when I deleted the service. Though, I was able to get the controller to log this error under a different scenario. If we don't explicitly remove the finalizer from the endpointslice, and instead re-create the pod and service like: From what I can understand, because the endpointslice with the finalizer is owned by a service with a different UID than the new endpointslice, it is marked as expected for deletion . The endpointslice controllers calls , which returns true for the endpointslice with the finalizer, because of the above reason. I might be wrong anywhere here, so please let me know if that is the case. As for further action, I'm not sure what can be done here? We could check the endpointslice for finalizers and then skip them . Keen to hear your thoughts about this.

} endpointslicemetrics.EndpointSliceSyncs.WithLabelValues(metricLabel).Inc() } <ins> func dropEndpointSlicesPendingDeletion(endpointSlices []*discovery.EndpointSlice) []*discovery.EndpointSlice { n := 0 for _, endpointSlice := range endpointSlices { if endpointSlice.DeletionTimestamp == nil { endpointSlices[n] = endpointSlice n++ } } return endpointSlices[:n] } </ins>


Question:
As discussed in , Endpoint(Slice) controllers can get stuck if an Endpoints or EndpointSlice resource is stuck in a terminating state. I don't think there's much we can do for Endpoints here since the same name is reused, but EndpointSlice controllers can ignore EndpointSlices in a terminating state. See No response All supported versions Any Any Any Any Any
/sig network
/triage accepted
deleting a service with a finalizer doesn't affect the endpointslice, it doesn't change its status to termination or unready Repro: finalizer to the service and delete it, it keeps waiting until the finalizer is removed with the deletion timestamp set. the endpoint slice associated Do you mean that when the endpoint slice controller detects that the Service is being deleted should switch the endpoint slice status to Terminating? cc:
I don't think we need to do anything special if the Service is waiting to be deleted. There are many legitimate reasons for this to happen, including waiting for infrastructure to be removed. My main concern with a finalizer was that one on an EndpointSlice could cause the controller to get stuck on a Service. We don't have the same risk with a finalizer on a Service - I don't think that would cause the controller to get stuck in any way.
i was unable to repro the issue on my machine: I was monitoring the logs of , and the endpointslice controller never complained about , when I deleted the service. Though, I was able to get the controller to log this error under a different scenario. If we don't explicitly remove the finalizer from the endpointslice, and instead re-create the pod and service like: From what I can understand, because the endpointslice with the finalizer is owned by a service with a different UID than the new endpointslice, it is marked as expected for deletion . The endpointslice controllers calls , which returns true for the endpointslice with the finalizer, because of the above reason. I might be wrong anywhere here, so please let me know if that is the case. As for further action, I'm not sure what can be done here? We could check the endpointslice for finalizers and then skip them . Keen to hear your thoughts about this.

assert.EqualValues(t, endpoint.TargetRef, &v1.ObjectReference{Kind: "Pod", Namespace: ns, Name: pod1.Name}) } <ins> func TestSyncServiceEndpointSlicePendingDeletion(t *testing.T) { client, esController := newController([]string{"node-1"}, time.Duration(0)) ns := metav1.NamespaceDefault serviceName := "testing-1" service := createService(t, esController, ns, serviceName) err := esController.syncService(fmt.Sprintf("%s/%s", ns, serviceName)) assert.Nil(t, err, "Expected no error syncing service") gvk := schema.GroupVersionKind{Version: "v1", Kind: "Service"} ownerRef := metav1.NewControllerRef(service, gvk) deletedTs := metav1.Now() endpointSlice := &discovery.EndpointSlice{ ObjectMeta: metav1.ObjectMeta{ Name:            "epSlice-1", Namespace:       ns, OwnerReferences: []metav1.OwnerReference{*ownerRef}, Labels: map[string]string{ discovery.LabelServiceName: serviceName, discovery.LabelManagedBy:   controllerName, }, DeletionTimestamp: &deletedTs, }, AddressType: discovery.AddressTypeIPv4, } err = esController.endpointSliceStore.Add(endpointSlice) if err != nil { t.Fatalf("Expected no error adding EndpointSlice: %v", err) } _, err = client.DiscoveryV1().EndpointSlices(ns).Create(context.TODO(), endpointSlice, metav1.CreateOptions{}) if err != nil { t.Fatalf("Expected no error creating EndpointSlice: %v", err) } numActionsBefore := len(client.Actions()) err = esController.syncService(fmt.Sprintf("%s/%s", ns, serviceName)) assert.Nil(t, err, "Expected no error syncing service") // The EndpointSlice marked for deletion should be ignored by the controller, and thus // should not result in more than one action from the client (an update to the non-terminating // EndpointSlice removing the trigger time annotation.) if len(client.Actions()) != numActionsBefore+1 { t.Errorf("Expected 1 more action, got %d", len(client.Actions())-numActionsBefore) } } </ins> // Ensure SyncService correctly selects and labels EndpointSlices. func TestSyncServiceEndpointSliceLabelSelection(t *testing.T) { client, esController := newController([]string{"node-1"}, time.Duration(0))


Question:
As discussed in , Endpoint(Slice) controllers can get stuck if an Endpoints or EndpointSlice resource is stuck in a terminating state. I don't think there's much we can do for Endpoints here since the same name is reused, but EndpointSlice controllers can ignore EndpointSlices in a terminating state. See No response All supported versions Any Any Any Any Any
/sig network
/triage accepted
deleting a service with a finalizer doesn't affect the endpointslice, it doesn't change its status to termination or unready Repro: finalizer to the service and delete it, it keeps waiting until the finalizer is removed with the deletion timestamp set. the endpoint slice associated Do you mean that when the endpoint slice controller detects that the Service is being deleted should switch the endpoint slice status to Terminating? cc:
I don't think we need to do anything special if the Service is waiting to be deleted. There are many legitimate reasons for this to happen, including waiting for infrastructure to be removed. My main concern with a finalizer was that one on an EndpointSlice could cause the controller to get stuck on a Service. We don't have the same risk with a finalizer on a Service - I don't think that would cause the controller to get stuck in any way.
i was unable to repro the issue on my machine: I was monitoring the logs of , and the endpointslice controller never complained about , when I deleted the service. Though, I was able to get the controller to log this error under a different scenario. If we don't explicitly remove the finalizer from the endpointslice, and instead re-create the pod and service like: From what I can understand, because the endpointslice with the finalizer is owned by a service with a different UID than the new endpointslice, it is marked as expected for deletion . The endpointslice controllers calls , which returns true for the endpointslice with the finalizer, because of the above reason. I might be wrong anywhere here, so please let me know if that is the case. As for further action, I'm not sure what can be done here? We could check the endpointslice for finalizers and then skip them . Keen to hear your thoughts about this.

} } } <ins> func Test_dropEndpointSlicesPendingDeletion(t *testing.T) { now := metav1.Now() endpointSlices := []*discovery.EndpointSlice{ { ObjectMeta: metav1.ObjectMeta{ Name:              "epSlice1", DeletionTimestamp: &now, }, }, { ObjectMeta: metav1.ObjectMeta{ Name: "epSlice2", }, AddressType: discovery.AddressTypeIPv4, Endpoints: []discovery.Endpoint{ { Addresses: []string{"172.18.0.2"}, }, }, }, { ObjectMeta: metav1.ObjectMeta{ Name: "epSlice3", }, AddressType: discovery.AddressTypeIPv6, Endpoints: []discovery.Endpoint{ { Addresses: []string{"3001:0da8:75a3:0000:0000:8a2e:0370:7334"}, }, }, }, } epSlice2 := endpointSlices[1] epSlice3 := endpointSlices[2] result := dropEndpointSlicesPendingDeletion(endpointSlices) assert.Len(t, result, 2) for _, epSlice := range result { if epSlice.Name == "epSlice1" { t.Errorf("Expected EndpointSlice marked for deletion to be dropped.") } } // We don't use endpointSlices and instead check manually for equality, because // `dropEndpointSlicesPendingDeletion` mutates the slice it receives, so it's easy // to break this test later. This way, we can be absolutely sure that the result // has exactly what we expect it to. if !reflect.DeepEqual(epSlice2, result[0]) { t.Errorf("EndpointSlice was unexpectedly mutated. Expected: %+v, Mutated: %+v", epSlice2, result[0]) } if !reflect.DeepEqual(epSlice3, result[1]) { t.Errorf("EndpointSlice was unexpectedly mutated. Expected: %+v, Mutated: %+v", epSlice3, result[1]) } } </ins>


Question:
As discussed in , Endpoint(Slice) controllers can get stuck if an Endpoints or EndpointSlice resource is stuck in a terminating state. I don't think there's much we can do for Endpoints here since the same name is reused, but EndpointSlice controllers can ignore EndpointSlices in a terminating state. See No response All supported versions Any Any Any Any Any
/sig network
/triage accepted
deleting a service with a finalizer doesn't affect the endpointslice, it doesn't change its status to termination or unready Repro: finalizer to the service and delete it, it keeps waiting until the finalizer is removed with the deletion timestamp set. the endpoint slice associated Do you mean that when the endpoint slice controller detects that the Service is being deleted should switch the endpoint slice status to Terminating? cc:
I don't think we need to do anything special if the Service is waiting to be deleted. There are many legitimate reasons for this to happen, including waiting for infrastructure to be removed. My main concern with a finalizer was that one on an EndpointSlice could cause the controller to get stuck on a Service. We don't have the same risk with a finalizer on a Service - I don't think that would cause the controller to get stuck in any way.
i was unable to repro the issue on my machine: I was monitoring the logs of , and the endpointslice controller never complained about , when I deleted the service. Though, I was able to get the controller to log this error under a different scenario. If we don't explicitly remove the finalizer from the endpointslice, and instead re-create the pod and service like: From what I can understand, because the endpointslice with the finalizer is owned by a service with a different UID than the new endpointslice, it is marked as expected for deletion . The endpointslice controllers calls , which returns true for the endpointslice with the finalizer, because of the above reason. I might be wrong anywhere here, so please let me know if that is the case. As for further action, I'm not sure what can be done here? We could check the endpointslice for finalizers and then skip them . Keen to hear your thoughts about this.

// 1. One or more of the provided EndpointSlices have older generations than the //    corresponding tracked ones. // 2. The tracker is expecting one or more of the provided EndpointSlices to be <del> //    deleted. </del> <ins> //    deleted. (EndpointSlices that have already been marked for deletion are ignored here.) </ins> // 3. The tracker is tracking EndpointSlices that have not been provided. func (est *EndpointSliceTracker) StaleSlices(service *v1.Service, endpointSlices []*discovery.EndpointSlice) bool { est.lock.Lock()


Question:
As discussed in , Endpoint(Slice) controllers can get stuck if an Endpoints or EndpointSlice resource is stuck in a terminating state. I don't think there's much we can do for Endpoints here since the same name is reused, but EndpointSlice controllers can ignore EndpointSlices in a terminating state. See No response All supported versions Any Any Any Any Any
/sig network
/triage accepted
deleting a service with a finalizer doesn't affect the endpointslice, it doesn't change its status to termination or unready Repro: finalizer to the service and delete it, it keeps waiting until the finalizer is removed with the deletion timestamp set. the endpoint slice associated Do you mean that when the endpoint slice controller detects that the Service is being deleted should switch the endpoint slice status to Terminating? cc:
I don't think we need to do anything special if the Service is waiting to be deleted. There are many legitimate reasons for this to happen, including waiting for infrastructure to be removed. My main concern with a finalizer was that one on an EndpointSlice could cause the controller to get stuck on a Service. We don't have the same risk with a finalizer on a Service - I don't think that would cause the controller to get stuck in any way.
i was unable to repro the issue on my machine: I was monitoring the logs of , and the endpointslice controller never complained about , when I deleted the service. Though, I was able to get the controller to log this error under a different scenario. If we don't explicitly remove the finalizer from the endpointslice, and instead re-create the pod and service like: From what I can understand, because the endpointslice with the finalizer is owned by a service with a different UID than the new endpointslice, it is marked as expected for deletion . The endpointslice controllers calls , which returns true for the endpointslice with the finalizer, because of the above reason. I might be wrong anywhere here, so please let me know if that is the case. As for further action, I'm not sure what can be done here? We could check the endpointslice for finalizers and then skip them . Keen to hear your thoughts about this.

epSlice1NewerGen := epSlice1.DeepCopy() epSlice1NewerGen.Generation = 2 <ins> epTerminatingSlice := epSlice1.DeepCopy() now := metav1.Now() epTerminatingSlice.DeletionTimestamp = &now </ins> testCases := []struct { name         string tracker      *EndpointSliceTracker


Question:
Cluster api provides a framework to run the e2e tests which uses the conformance image: The gmsa tests use bash to install the webhook: The test will fail when run in the conformance image because of the bash: we should install the component another way run this e2e conformance image with /sig windows fyi <details</details<details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Conformance image:
Hi does the "work e2e" test case pass? Deploying GMSA webhook is the same for both "work e2e" and this test case.
no, they both fail due to bash not being present in the conformance image. The gmsa tests themselves pass if I run locally
/sig testing

"context" "fmt" "os" <del> "os/exec" "path" </del> "regexp" "strings" "time" <ins> "github.com/onsi/ginkgo" "github.com/onsi/gomega" appsv1 "k8s.io/api/apps/v1" </ins> v1 "k8s.io/api/core/v1" rbacv1 "k8s.io/api/rbac/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/util/uuid" clientset "k8s.io/client-go/kubernetes" "k8s.io/kubernetes/test/e2e/framework" <ins> e2epod "k8s.io/kubernetes/test/e2e/framework/pod" </ins> e2eskipper "k8s.io/kubernetes/test/e2e/framework/skipper" imageutils "k8s.io/kubernetes/test/utils/image" <del> "github.com/onsi/ginkgo" "github.com/onsi/gomega" </del> ) const (


Question:
Cluster api provides a framework to run the e2e tests which uses the conformance image: The gmsa tests use bash to install the webhook: The test will fail when run in the conformance image because of the bash: we should install the component another way run this e2e conformance image with /sig windows fyi <details</details<details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Conformance image:
Hi does the "work e2e" test case pass? Deploying GMSA webhook is the same for both "work e2e" and this test case.
no, they both fail due to bash not being present in the conformance image. The gmsa tests themselves pass if I run locally
/sig testing

gmsaCustomResourceName = "gmsa-e2e" // gmsaWebhookDeployScriptURL is the URL of the deploy script for the GMSA webook <del> // TODO(wk8): we should pin versions. </del> gmsaWebhookDeployScriptURL = "https://raw.githubusercontent.com/kubernetes-sigs/windows-gmsa/master/admission-webhook/deploy/deploy-gmsa-webhook.sh" // output from the nltest /query command should have this in it


Question:
Cluster api provides a framework to run the e2e tests which uses the conformance image: The gmsa tests use bash to install the webhook: The test will fail when run in the conformance image because of the bash: we should install the component another way run this e2e conformance image with /sig windows fyi <details</details<details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Conformance image:
Hi does the "work e2e" test case pass? Deploying GMSA webhook is the same for both "work e2e" and this test case.
no, they both fail due to bash not being present in the conformance image. The gmsa tests themselves pass if I run locally
/sig testing

ginkgo.By("retrieving the contents of the GMSACredentialSpec custom resource manifest from the node") crdManifestContents := retrieveCRDManifestFileContents(f, node) <del> ginkgo.By("downloading the GMSA webhook deploy script") deployScriptPath, err := downloadFile(gmsaWebhookDeployScriptURL) defer func() { os.Remove(deployScriptPath) }() if err != nil { framework.Failf(err.Error()) } </del> ginkgo.By("deploying the GMSA webhook") <del> webhookCleanUp, err := deployGmsaWebhook(f, deployScriptPath) </del> <ins> webhookCleanUp, err := deployGmsaWebhook(f) </ins> defer webhookCleanUp() if err != nil { framework.Failf(err.Error())