Question: CudaVectorAdd // CudaVectorAdd2 image CudaVectorAdd2 <del> // DebianIptables Image DebianIptables </del> <ins> // DistrolessIptables Image DistrolessIptables </ins> // Etcd image Etcd // GlusterDynamicProvisioner image

On Tue, Apr 19, 2022, 11:31 AM Lars Ekman
i don't think so. it's status quo on Alpine which is to avoid it totally.
Can I read about those somewhere? Are the legal reasons about Alpine? Or about musl?
Not my personal decision, and I'm not a lawyer let alone an OSS license expert but ... sharing what I've heard on the subject: IIRC for example Alpine does not make available the source code (AFAICT Alpine publishes source code for the packaging scripts, but not the package sources), or even the LICENSE of binary packages, which is problematic for compliance with licenses ... (Various common license require including the license text, and / or making source code available). Normal alpine images don't appear to install these doc packages: So those images don't even ship with license / copyright info files for packages which is ... not a great start. I'm not sure where a good public reference is for this, Google OSPO has some relevant docs internally about following license requirements when producing container images. So far the best public doc on the topic I've found is
Thanks!
TL:DR;
I wasn't suggesting switching to Alpine, I was just saying that I unit test the iptables-wrappers scripts against Alpine, which doesn't use bash, and so I know the scripts don't require bash. Yeah, I don't know the details of how distroless works, but if we're talking about debian packages then it looks like is 1/10 the size of (212k vs 2285k)
In Alpine (and Debian, and probably most distros other than Fedora), the "iptables" package includes both the legacy and nft binaries, so that's the only dependency.
Made a comment on bandwidth that may be better in this issue rahter than the PR;
Want to give a quick update on this (EDIT: I wrote this was going to be quick update, but ended up not being! sorry!): The image generation is working "fine", see: This process generates a temporary that can be used later with and/or changed in Kubernetes code to trigger the tests (see ) Due to the way iptables-wrapper-installer works, it calls iptables-legacy and iptables-nft to check if they are working fine.


Question: CudaVectorAdd // CudaVectorAdd2 image CudaVectorAdd2 <del> // DebianIptables Image DebianIptables </del> <ins> // DistrolessIptables Image DistrolessIptables </ins> // Etcd image Etcd // GlusterDynamicProvisioner image

We need to skip it (via --skip-integrity-check) in arch not amd64, as those are really specific kernel features and when called from arm64, as an example the command ends in an error. We need /bin/sleep to be present in distroless as well, due specifically to this image being used in this test: cloud-build timeout needs to be increased, it takes a long time for arm/s390/etc to build this The image can be optimized a bit more, removing some steps like linking the iptables binaries (as pointed by and removing some binaries like wc (as pointed by ) but right now I didn't wanted to change the image build AND the iptables-wrapper script. My take here would be: let's see now with sig-release if the build process is fine, get this image promoted and used before v1.25 All the tests but verify ran fine (verify due to dependency check, etc etc) - Here are the main tests with the distroless image: kind - kind ipv6 - gce ipvs - performance 100 nodes - fyi. I can bring this to next sig-net meeting, wanted to check: from sig/release perspective, is the PR fine? I will make some more changes here, but wanted even to check if this should be called debian-iptables:distroless or something like from sig/net if we want to move forward migrating the iptables-wrapper to shell script this release or not. I tend of agree with Rob that maybe we are getting too many changes in one release, and we can plan this for v1.26 (the PR is already there, I need to check Dan's comments only) Thanks
/assign
/reopen let's close it once we switch to the new image
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hi, Could you suggest if it's ok to use already? That would be super cool to use the image as a base for NodeLocalDNS (as it uses iptables). I already did manual smoke test in cluster - all is working correctly. Thanks!


Question: CudaVectorAdd // CudaVectorAdd2 image CudaVectorAdd2 <del> // DebianIptables Image DebianIptables </del> <ins> // DistrolessIptables Image DistrolessIptables </ins> // Etcd image Etcd // GlusterDynamicProvisioner image

Let's introduce it gradually, I suggest to wait until the image is promoted for kube-proxy
I've lost track of this, what do we need to do to promote it for kube-proxy?
I think Ricardo was on it But if someone is busy and others have time, let's sync next monday on slack and organize the work, so someone can get this to the finish line, there is no need to overload one person if there are other wanting to help
I didn’t saw there was a comment there. I can fix that, and wanted to plan with you folks next steps. I agree with Antonio and don’t want to block this on my availability :) I will check suggestion tomorrow and as we get this merged I’m more than happy to handle to someone else! On Sat, 16 Jul 2022 at 07:21 Antonio Ojea
I've opened a new PR to do the proper promotion. As soon as this is promoted I think we can start testing and migrating to distroless :) From here I think I can handle to someone else as I wont have much time and attention to this specific effort and I know this is important :) I'm still willing to migrate iptables-wrapper to go code (we started it in kubernetes-sigs/iptables-wrapper) so I will probably finish fixing Dan's reviews and we can discuss the distroless v0.2.0 being based entirely in Go :)
we should probably nudge a few of the subprojects to migrate, we just migrated KIND but I'm sure we have at least a few more out there like ip-masq-agent
I'm migrating nodelocaldns. I've got a couple of questions: nodelocaldns uses registry . Should I switch it to ? Is considered sort of a prod image registry? a multi-arch image? Because currently we use is it better to keep up with the new tags for ? So that we could quickly upgrade base image version as well? Thanks for the work!
YES! Make happy! This registry is actually a "proxy" that sig-k8s-infra is maintaining so people get directed to the closest Image registry, based on their cloud provider. As an example, with this users in AWS wont need to go to GCR to pull their image Yes, it is. It follows the same build pattern of debian-iptables. See Well, this is a good question.


Question: CudaVectorAdd // CudaVectorAdd2 image CudaVectorAdd2 <del> // DebianIptables Image DebianIptables </del> <ins> // DistrolessIptables Image DistrolessIptables </ins> // Etcd image Etcd // GlusterDynamicProvisioner image

AFAIK we don't have an event / notification system that sends a message (in Slack, as an example) whenever a base image gets promoted. It would be cool to have at least for debian-base, debian-iptables and distroless-iptables, probably it's worth to question sig-release on this :) ( maybe know)
Thanks for the answer! Regarding the - we use it as a compile image for dnsmasq. Will this image still be supported or should I find something other to replace this compile image?
Short answer: I imagine/suppose there is no deprecation plan for this image right now, as it is used as the compilation base for a bunch of other things (and it is base of etcd image) Long answer: In case of distroless-iptables the rationale was: we need to reduce costs (and by reducing costs, reducing image size) we need to use something with a smaller footprint and reduce the amount of packages, and as a result the amount of CVEs in image package. kube-proxy exists in all nodes (unless you use a CNI that implements its features) so it is a good candidate I guess right now debian-base will still be used (I don't remember on the top of my head if other kubernetes components are already running using distroless or compiled based on debian-base, as far as I can see at least etcd image depends on debian-base () so probably any effort on this image should be announced.


Question: } } } <del> return result </del> <ins> return result, cinfosByPodCgroupKey </ins> } // ByCreationTime implements sort.Interface for []containerInfoWithCgroup based

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --For single node with hundreds and thousands of pods and containers, the cpu usage of kubelet becomes high, one bottleneck is the cadvisor stats provider as below: the method iterate all containerInfos for every podStats cause high cpu the method should not iterate allcontainerInfos for every podStats, use map instead Kubernetes version (use ): v1.19.4 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig node
/triage accepted
/assign
I’ve done an analysis on the code and this is what I have found, I believe we have two issues to address. There are multiple range loops being performed over v2.ContainerInfo / PodStat maps within the ListPodStatus method which can be contributing to the high cpu utilisation. The allInfos argument which is supplied to the getCadvisorContainerInfo function is a map which uses a cgroup path as the key, in order to find matching container info to podUID, processing of the keys in the map needs to be performed until a match is found. Examples of keys are as follows: /kubepods/burstable/podf9e3a173-1470-451a-9ca2-/d8426b82582fbcfae16085bc794c49769d3a9648bd5463976aca4071f9342754 /docker/2dfa17fc81e781212bcc26e189ebb724721fe097394b4de3cbeb7e069bf23290 My recommendation would be to apply the following updates: Refactor the getCadvisorContainerInfo function to allow for a filter and key type parameter to be supplied by the caller, this enables reducing the number of range loops being performed over the container info map by removing elements that are not required and the ability to set the key being used in the returned container infos map to enable fast lookup of container info.


Question: } } } <del> return result </del> <ins> return result, cinfosByPodCgroupKey </ins> } // ByCreationTime implements sort.Interface for []containerInfoWithCgroup based

My initial thoughts are to use an enumeration which has two values KEYCGROUPPATH to provide backwards compatibility and KEYPODUID to enable fast lookups via podUID Refactor ListPodStats method to be more efficient in constructing the PodStats slice and taking into account the fast lookup via podUID I think this is the best approach instead of pushing the data structure implementation for the container info upstream to the cadviser clients, this will at least initially provide a better optimisation scoped within the kubelet/stats package and reduce other possible impacts to changing the cadvisor clients Please advise.
Just realized this looks like a duplicate issue and looking as fixed by PR Both this issue and were raised by Zac
/unassign
/close
You can't close an active issue/PR unless you authored it or you are a collaborator. <detailsIn response to : /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Didn't think I could close this, leaving this with
this will be automatically closed once the PR merges


Question: package filters import ( <ins> "bytes" "context" </ins> "crypto/tls" <ins> "fmt" </ins> "io" <ins> "io/ioutil" </ins> "math/rand" "net" "net/http" "net/http/httptest" <ins> "reflect" "sync" </ins> "testing" "time"

What happened: We enabled feature in our large-scale cluster, and we found client(Kubelet) may got error(500 http code response) like after received GOAWAY frame. I had already re-produce this bug and open an in golang repo. So far I think this bug only occur with a small probability when client request server in high concurrency use a same httpclient instance, and it's a golang sdk bug in http client. Before I solved this issue, I suggest do NOT enable GOAWAY() feature. As it's a golang sdk bug(currently I known and doubt) in http client, revert GOAWAY feature(codes) is not necessary.
cc
/sig api-machinery
Thanks for the report.
Based on not setting the field, I think it defaults that way, so that's good. We will leave the option and filter so that when golang updates its client, we will have a simple way to turn it on and execute it.
Sure. I am trying to fix the golang http client issue and will update progress at . After golang http client issue fixed, I will add more concurrent test cases to see if there are any other bugs may occur by GOAWAY feature. BTW, we still enable GOAWAY feature in our production cluster, because there is no any evidence told us this error will influence SLO, as all most components(kubelet, controller-manager, scheduler) is controller work mode and will re-try after got error. /assign
Golang version or required, which will release at 07/01/2020.
Kubernetes v1.19(or master branch) now is using golang 1.15, and golang 1.15 has fixed the issue. And is on going. Let me add more test cases to ensure the error will not happen again.
I think we should re-open this issue because of . /re-open


Question: // tolerations are checked before the deep copy, so munge those too mungedPodSpec.Tolerations = oldPod.Spec.Tolerations // +k8s:verify-mutation:reason=clone <ins> // Relax validation of immutable fields to allow it to be set to 1 if it was previously negative. if oldPod.Spec.TerminationGracePeriodSeconds != nil && *oldPod.Spec.TerminationGracePeriodSeconds < 0 && mungedPodSpec.TerminationGracePeriodSeconds != nil && *mungedPodSpec.TerminationGracePeriodSeconds == 1 { mungedPodSpec.TerminationGracePeriodSeconds = oldPod.Spec.TerminationGracePeriodSeconds // +k8s:verify-mutation:reason=clone } </ins> if !apiequality.Semantic.DeepEqual(mungedPodSpec, oldPod.Spec) { // This diff isn't perfect, but it's a helluva lot better an "I'm not going to tell you what the difference is". //TODO: Pinpoint the specific field that causes the invalid error after we have strategic merge diff specDiff := diff.ObjectDiff(mungedPodSpec, oldPod.Spec) <del> allErrs = append(allErrs, field.Forbidden(specPath, fmt.Sprintf("pod updates may not change fields other than `spec.containers[*].image`, `spec.initContainers[*].image`, `spec.activeDeadlineSeconds` or `spec.tolerations` (only additions to existing tolerations)n%v", specDiff))) </del> <ins> allErrs = append(allErrs, field.Forbidden(specPath, fmt.Sprintf("pod updates may not change fields other than `spec.containers[*].image`, `spec.initContainers[*].image`, `spec.activeDeadlineSeconds`, `spec.tolerations` (only additions to existing tolerations) or `spec.terminationGracePeriodSeconds` (allow it to be set to 1 if it was previously negative)n%v", specDiff))) </ins> } return allErrs

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Some k8s client tried to evict a pod by setting negative value for deletionGracePeriodSeconds. But all eventual delete calls (and force-delete calls) made by kubelet, pod-garbage-collector, etc were not able to delete that pod, because apiserver doesnt allow deletionGracePeriodSeconds to be increased once it is set. So force-delete (which set it to 0) doesn't take effect when it's already negative. Not even kubectl is able to call delete with negative grace-period. But curl with an even more negative grace-period value worked. What you expected to happen: force-delete works. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: I think the issue is in the code below As the comment suggest, in phase 1, the was set to a negative value (meaning was in the past), and in phase 2, has to be set even lower, otherwise, it will be considered as pending graceful deletion and not performing the immediate delete. I think there're two possible fixes. not allowing setting in the past ( should be non-negative). In that case, we could convert to 0 if negative. that could be set in the past. In that case, in phase 2, we treat negative value of the same as 0, and allow immediate delete. Thoughts? Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig api-machinery
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig api-machinery Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/cc /triage accepted
No component in the system should ever treat the timestamp of DeletionTimestamp as authoritative.


Question: // tolerations are checked before the deep copy, so munge those too mungedPodSpec.Tolerations = oldPod.Spec.Tolerations // +k8s:verify-mutation:reason=clone <ins> // Relax validation of immutable fields to allow it to be set to 1 if it was previously negative. if oldPod.Spec.TerminationGracePeriodSeconds != nil && *oldPod.Spec.TerminationGracePeriodSeconds < 0 && mungedPodSpec.TerminationGracePeriodSeconds != nil && *mungedPodSpec.TerminationGracePeriodSeconds == 1 { mungedPodSpec.TerminationGracePeriodSeconds = oldPod.Spec.TerminationGracePeriodSeconds // +k8s:verify-mutation:reason=clone } </ins> if !apiequality.Semantic.DeepEqual(mungedPodSpec, oldPod.Spec) { // This diff isn't perfect, but it's a helluva lot better an "I'm not going to tell you what the difference is". //TODO: Pinpoint the specific field that causes the invalid error after we have strategic merge diff specDiff := diff.ObjectDiff(mungedPodSpec, oldPod.Spec) <del> allErrs = append(allErrs, field.Forbidden(specPath, fmt.Sprintf("pod updates may not change fields other than `spec.containers[*].image`, `spec.initContainers[*].image`, `spec.activeDeadlineSeconds` or `spec.tolerations` (only additions to existing tolerations)n%v", specDiff))) </del> <ins> allErrs = append(allErrs, field.Forbidden(specPath, fmt.Sprintf("pod updates may not change fields other than `spec.containers[*].image`, `spec.initContainers[*].image`, `spec.activeDeadlineSeconds`, `spec.tolerations` (only additions to existing tolerations) or `spec.terminationGracePeriodSeconds` (allow it to be set to 1 if it was previously negative)n%v", specDiff))) </ins> } return allErrs

In Kube, components are required to calculate the time period from when they observe the event in their local clock (i.e. 30s from now), not use the deletion timestamp. A negative value of deletion grace period should be rejected by the apiserver as an invalid value (bad request / invalid field).
attempts to take a missing or positive deletion grace period negative should be rejected, I agree. if a negative value is already persisted, I'd be inclined to allow a negative value in an update and clip to 0
As I mentioned in the PR, eviction is passing negatives through to the store: In kubectl, we specify the default value for GracePeriod is -1. In the case of delete, we simply don't set the value in the deleteOptions we send to the API. In the case of eviction, we send the -1 value, and that gets passed along to the store.
I don't see the -1 getting sent to the eviction API. It looks like values < 0 result in a nil value being sent to the API: I would not expect the API to accept a negative value as-is.
Looks like you are correct. Do the deletionOptions go through their own validation or through some kind of validation for the evict API? The original bug report mentions 'evict' though it's not clear if the eviction API was used or not.
ValidateDeleteOptions is called inside pod deletion, but did not check that the field was non-negative. I don't see any eviction-specific validation.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle rotten
/lifecycle rotten
Im investigating the and . Surprisingly see you here...


Question: defer tc.Unlock() obj := &v1.PodList{} <del> for i := 0; i < len(tc.reportedCPURequests); i++ { </del> <ins> specifiedCPURequests := tc.reportedCPURequests != nil numPodsToCreate := int(tc.initialReplicas) if specifiedCPURequests { numPodsToCreate = len(tc.reportedCPURequests) } for i := 0; i < numPodsToCreate; i++ { </ins> podReadiness := v1.ConditionTrue if tc.reportedPodReadiness != nil { podReadiness = tc.reportedPodReadiness[i] } podPhase := v1.PodRunning if tc.reportedPodPhase != nil { podPhase = tc.reportedPodPhase[i] } podName := fmt.Sprintf("%s-%d", podNamePrefix, i) <ins> reportedCPURequest := resource.MustParse("1.0") if specifiedCPURequests { reportedCPURequest = tc.reportedCPURequests[i] } </ins> pod := v1.Pod{ Status: v1.PodStatus{ Phase: podPhase,

Is this a BUG REPORT or FEATURE REQUEST?: /kind feature What happened: We are starting to use the HPA (horizontal pod autoscaling) for one of our components. What is special about this particular component is that the warm up period is very long (sometimes even more than 30 minutes). During that time, the pod is in Running state but its containers are in Not Ready state. This pod is of course not able to receive any traffic so the metric we use in the HPA is ignoring those pods that are not Ready. The equation used by HPA to calculate the new number of replicas is as follows Our problem is that the equation used by HPA to calculate the new number of replicas is including in the "current replica count" the pods that are "Not Ready" so the value for "new replica count" is not correct. Since this warm up is so long we end up by having a lot of more instances of our services of what it should be necessary. They eventually get deleted once gradually the metrics start going down but that will take a lot of time and a big waste of resources without mentioning that with a changing traffic this HPA will never stabilize What you expected to happen: I would like to have a flag somewhere where I tell HPA to don't include in the "current replica count" the pods that are in "Not Ready" state How to reproduce it (as minimally and precisely as possible): Have a component that takes a lot of time to start and being able to receive traffic. Then define a metric that will ignore the pods that are Not Ready. Finally create a metric that will scale that component based on the mentioned metric Anything else we need to know?: Environment: Kubernetes version 1.7 Cloud provider or hardware configuration: Azure
/sig autoscaling
Hi - thanks for reaching out about this! It looks like this section of the , handles when pods aren't ready. I think you this behavior in , so I'm interested in your thoughts on whats best here. If we do end up adding a flag for this, I'd be happy to write the code :) Also, purely for my own curiosity, I'm curious what type of pod is taking 30 minutes to become ready. If your traffic pattern varies at a rate faster than 30 minutes, it may be difficult for auto-scaling to be helpful.
can you also share a bit more about your HPA?


Question: defer tc.Unlock() obj := &v1.PodList{} <del> for i := 0; i < len(tc.reportedCPURequests); i++ { </del> <ins> specifiedCPURequests := tc.reportedCPURequests != nil numPodsToCreate := int(tc.initialReplicas) if specifiedCPURequests { numPodsToCreate = len(tc.reportedCPURequests) } for i := 0; i < numPodsToCreate; i++ { </ins> podReadiness := v1.ConditionTrue if tc.reportedPodReadiness != nil { podReadiness = tc.reportedPodReadiness[i] } podPhase := v1.PodRunning if tc.reportedPodPhase != nil { podPhase = tc.reportedPodPhase[i] } podName := fmt.Sprintf("%s-%d", podNamePrefix, i) <ins> reportedCPURequest := resource.MustParse("1.0") if specifiedCPURequests { reportedCPURequest = tc.reportedCPURequests[i] } </ins> pod := v1.Pod{ Status: v1.PodStatus{ Phase: podPhase,

What kind of metric type is it? Based on the description I'm suspecting that it is an Object metric, am I right?
So answer about why our process takes that much time to start. Basically this is a server that in order to do what he needs to do, he needs to load a lot of data in memory (several GBs) and then perform some warm up that are pretty intensive. The idea is that he doesn't need to grab (almost) anything from outside to attend a request so it's latency is extremely low (in the order of millisecond) By the way. Because of this we have another problem which is that there is not an specific flag per HPA object that we could tell k8s to don't try to scale up again passed some time so the new pods have enough time to get ready before the next scale happens. I know that there is a cluster wide flag but we will need to have a different one by resource that we scale. Now to answer You pointed another interesting topic. The metric we use is % Utilization. We are able to calculate the level of utilization of our pods and we want to keep this metric in an specific value. About the flag. I suggested the flag so it doesn't change the existing behavior by default but I honestly think that the default behavior should be to don't include the NonReady pods as part of the calculation
do you think we can add this change to 1.10? it's critical bug/feature for us and we are happy to help with code/testing.
It's really late for 1.10, especially since we don't have any code here, and I'd like to know more about what's going on here, and especially see if it reproduces on Kubernetes 1.9 (we've had HPA controller bugs in the past). FWIW, currently unready pods are treated as consuming 0% of their utilization on scale up, and it was intended that this should prevent such issues. That is, the formula is actually: If the pods are marked as not ready, I find it strange that they would be affecting the scaling that much, unless your other pods are under real heavy load (i.e. I would assume that the unready pods would skew the scaling back down so that the number of pods just stayed constant).
I believe there is different code branch here.


Question: defer tc.Unlock() obj := &v1.PodList{} <del> for i := 0; i < len(tc.reportedCPURequests); i++ { </del> <ins> specifiedCPURequests := tc.reportedCPURequests != nil numPodsToCreate := int(tc.initialReplicas) if specifiedCPURequests { numPodsToCreate = len(tc.reportedCPURequests) } for i := 0; i < numPodsToCreate; i++ { </ins> podReadiness := v1.ConditionTrue if tc.reportedPodReadiness != nil { podReadiness = tc.reportedPodReadiness[i] } podPhase := v1.PodRunning if tc.reportedPodPhase != nil { podPhase = tc.reportedPodPhase[i] } podName := fmt.Sprintf("%s-%d", podNamePrefix, i) <ins> reportedCPURequest := resource.MustParse("1.0") if specifiedCPURequests { reportedCPURequest = tc.reportedCPURequests[i] } </ins> pod := v1.Pod{ Status: v1.PodStatus{ Phase: podPhase,

metric definition { "type": "Object", "object": { "target": { "kind": "Service", "name": "prometheus-exporter" }, "metricName": "PercentUtilization", "targetValue": "20" } } which should handle as and doesn't check Pod state.
I think proper handling of unready pods is a difficult problem that will require significant work. I suspect we may find out we need to start distinguishing pods that are booting up from pods that are 'broken' and it took us quite a while to solve analogous problem with nodes in Cluster Autoscaler (to be clear: I don't know if we will have to do that, that's just my intuition for now). That being said I agree the formula in GetObjectMetricReplicas will generally lead to scaling-up too much if the pod takes longer to initialize then backoff period in HPA (3 minutes). Assume it takes pod 4 minutes to boot up and we're currently running 2 pods. Metric value is 20 and target is 10. In this case: scales-up to 4 pods (20/10 utilization 2 pods). are starting up, they're not doing anything yet. 3 minutes later HPA scales-up to 8 pods (20/10 utilization 4 pods). If we only counted ready pods the scale-up in 3 would not occur, because the desired replicas would still be 4 (20/10 * 2). My intuition would be that this is the expected behavior of HPA. Also if there is no space in the cluster we will just keep scaling-up until we hit max replicas. The only case where current behavior would be more correct than filtering out unready pods in this formula is if someone uses a metric that is calculated based on total number of replicas. Do we advertise that as the right way to use HPA? If we add to Object metrics in 1.11 this usage pattern will stop making sense. To sum up - I think we will need to change the formula to only count ready pods at some point. I can see how current behavior could be a problem in any use-case except the one described in paragraph above. I'm inclined to agree with and provide some way to change the behavior in 1.10 (a flag or an annotation?), treating it as a bug fix.


Question: defer tc.Unlock() obj := &v1.PodList{} <del> for i := 0; i < len(tc.reportedCPURequests); i++ { </del> <ins> specifiedCPURequests := tc.reportedCPURequests != nil numPodsToCreate := int(tc.initialReplicas) if specifiedCPURequests { numPodsToCreate = len(tc.reportedCPURequests) } for i := 0; i < numPodsToCreate; i++ { </ins> podReadiness := v1.ConditionTrue if tc.reportedPodReadiness != nil { podReadiness = tc.reportedPodReadiness[i] } podPhase := v1.PodRunning if tc.reportedPodPhase != nil { podPhase = tc.reportedPodPhase[i] } podName := fmt.Sprintf("%s-%d", podNamePrefix, i) <ins> reportedCPURequest := resource.MustParse("1.0") if specifiedCPURequests { reportedCPURequest = tc.reportedCPURequests[i] } </ins> pod := v1.Pod{ Status: v1.PodStatus{ Phase: podPhase,

WDYT
Ah, yeah, I missed that it was object metrics before. I'm not entirely convinced we need a flag here -- it might just make sense to bring Object metrics in line with all the other metrics types. On the other hand, there's a subtlety here -- the other metric types don't quite ignore unready pods -- they just ignore the metrics from unready pods, which causes a heavy bias in the direction of not scaling, but an overwhelming spike can still cause scaling. That behavior is not as achievable using object metrics. I'm inclined to just ignore that subtlety, though, and say that object metrics should ignore unready pods entirely.
I think this is what we should do long term. I was worried about breaking someone who relies on targetValue + averaged metric to get effect similar to targetAverageValue and though about providing something for transitional period, but if you're ok with it I'd rather just change it as you say.
I think it's safe to change as-is -- that feature's still in beta, and it's not an API change, so it shouldn't just hard-break anyone. Plus, we've change stuff like this before (when we did the initial resource metrics change).
As said, we are more than ready to help with the fix/test if required. This since this is a major blocking for us in the implementation of the HPA in our side
is anyone actively working on implementing this? If not, I'm happy to. I think I have a fairly clear picture of the change we need to make and will likely have time early this week to get it done.
(Also, from the conversation above, to we want to tag this as a release blocker for 1.10)?
I probably won't be able to work on this before the end of this week. Happy to do a review if you jump in and implement it.
I don't think this qualifies as a release blocking bug (but I'm willing to have someone try and convince me otherwise ;-)). There doesn't seem to be any behavior here that's explicitly broken -- it's just sub-optimal in some use cases. I'd definitely like to address this in 1.11, though.
it's not release blocker, but to share our experience ;).


Question: defer tc.Unlock() obj := &v1.PodList{} <del> for i := 0; i < len(tc.reportedCPURequests); i++ { </del> <ins> specifiedCPURequests := tc.reportedCPURequests != nil numPodsToCreate := int(tc.initialReplicas) if specifiedCPURequests { numPodsToCreate = len(tc.reportedCPURequests) } for i := 0; i < numPodsToCreate; i++ { </ins> podReadiness := v1.ConditionTrue if tc.reportedPodReadiness != nil { podReadiness = tc.reportedPodReadiness[i] } podPhase := v1.PodRunning if tc.reportedPodPhase != nil { podPhase = tc.reportedPodPhase[i] } podName := fmt.Sprintf("%s-%d", podNamePrefix, i) <ins> reportedCPURequest := resource.MustParse("1.0") if specifiedCPURequests { reportedCPURequest = tc.reportedCPURequests[i] } </ins> pod := v1.Pod{ Status: v1.PodStatus{ Phase: podPhase,

without this feature we cannot use HPA as is and we need to figure out how to build/run fork (which can be challenging because everything inside controller-manager). also we are planning to upgrade to 1.10 quickly, but not sure for 1.11 (e.g. we are two versions behind now).
Sounds like a plan :) Will try and get this done tomorrow morning!
- I submitted a pr for this - would love your eyes on it so you can see if you think it meets your use case :)
/assign
GitHub didn't allow me to assign the following users: mattjmcnaughton. Note that only and repo collaborators can be assigned. <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I checked the code and it seems good to me. About the comment about scaling down if the readyPods<CurrentPods, we should be fine since our metric is already not including the pods that are not ready.
Is there any advance in this ticket? I'm wondering if this made it to k8s 1.10 or if we need to wait for k8s 1.11?
The PR is lgtm-ed and it will merge once the code freeze is lifted. Meaning it will go to 1.11. To get it into 1.10 you would need to get approval for milestone from (sig-autoscaling lead).


Question: } } } <ins> func TestInt64AmountMul(t *testing.T) { for _, test := range []struct { a  int64Amount b  int64 c  int64Amount ok bool }{ {int64Amount{value: 100, scale: 1}, 1000, int64Amount{value: 100000, scale: 1}, true}, {int64Amount{value: 100, scale: -1}, 1000, int64Amount{value: 100000, scale: -1}, true}, {int64Amount{value: 1, scale: 100}, 10, int64Amount{value: 1, scale: 100}, false}, {int64Amount{value: 1, scale: -100}, 10, int64Amount{value: 1, scale: -100}, false}, {int64Amount{value: -5, scale: 2}, 500, int64Amount{value: -2500, scale: 2}, true}, {int64Amount{value: -5, scale: -2}, 500, int64Amount{value: -2500, scale: -2}, true}, {int64Amount{value: 0, scale: 1}, 0, int64Amount{value: 0, scale: 1}, true}, {int64Amount{value: mostPositive, scale: -1}, 10, int64Amount{value: mostPositive, scale: -1}, false}, {int64Amount{value: mostPositive, scale: -1}, 0, int64Amount{value: 0, scale: 0}, true}, {int64Amount{value: mostPositive, scale: 0}, 1, int64Amount{value: mostPositive, scale: 0}, true}, {int64Amount{value: mostPositive / 10, scale: 1}, 10, int64Amount{value: mostPositive / 10, scale: 1}, false}, {int64Amount{value: mostPositive, scale: 0}, -1, int64Amount{value: -mostPositive, scale: 0}, true}, {int64Amount{value: mostNegative, scale: 0}, 1, int64Amount{value: mostNegative, scale: 0}, true}, {int64Amount{value: mostNegative, scale: 1}, 0, int64Amount{value: 0, scale: 0}, true}, {int64Amount{value: mostNegative, scale: 1}, 1, int64Amount{value: mostNegative, scale: 1}, false}, } { c := test.a ok := c.Mul(test.b) if ok && !test.ok { t.Errorf("unextected success: %v", c) } else if !ok && test.ok { t.Errorf("unexpeted failure: %v", c) } else if ok { if c != test.c { t.Errorf("%v: unexpected result: %d", test, c) } } else { if c != test.a { t.Errorf("%v: overflow multiplication mutated source: %d", test, c) } } } } </ins> func TestInt64AsCanonicalString(t *testing.T) { for _, test := range []struct { value    int64

I'd like to add a multiplication method for the resource.Quantity. Currently, the resource package provides helper methods to calculate Quantity, such as and : Batch workloads often replicate the same PodTemplate, and controllers calculate the resources needed for all containers. In that case, It is inconvenient to call the function as many times as podTemplate, so I would like to create a helper function for multiplication.
/cc /sig api-machinary /wg batch
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /cc /sig api-machinary /wg batch </details
/sig api-machinery /wg batch
Do you have an example of usage for these utilities? Just curious?
/triage accepted
Sorry for the late response. Taking k-sigs/kueue as an example, I assumed the below: It helps the controllers for batch workloads such as k-sigs/kueue, k-sigs/jobset, kubeflow/mpi-operator, and kubeflow/training-operator. I'm thinking of implementing this function once the code freeze term is finished.
/assign
Hi, may i take this up?
I took this one, and working on this now.


Question: []*scheduling.PriorityClass{systemClusterCritical}, *pods[7], scheduling.SystemCriticalPriority, <del> true, </del> <ins> false, </ins> nil, }, {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Following the documentation at the , I tried enabling quota for a namespace other than but I was still not able to create a pod with priorityClass in my namespace with following error: Error from server (Forbidden): error when creating "": pods "memory-demo" is forbidden: pods with system-cluster-critical priorityClass is not permitted in my-namespace namespace What you expected to happen: Expected pod to be created with namespace where I allowed quota. I can see that both ResourceQuota and Priority Plugins are enabled from logs. I0408 17:15:08. ] Loaded 8 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,MutatingAdmissionWebhook. I0408 17:15:08. ] Loaded 6 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,ValidatingAdmissionWebhook,ResourceQuota. How to reproduce it (as minimally and precisely as possible): Follow the steps at Anything else we need to know?: I think, the issue is with priorityPlugin where we are rejecting instead of letting resourceQuota plugin to handle the admission when is enabled. Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Others:
/sig scheduling /cc
Same happens to me. Correction, it does not allow to use the default permissions outside of kube-system
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
See also:
/remove-lifecycle stale
/assign Per offline discussion with
Same issue here.


Question: []*scheduling.PriorityClass{systemClusterCritical}, *pods[7], scheduling.SystemCriticalPriority, <del> true, </del> <ins> false, </ins> nil, }, {

We are using namespaces other than kube-system for system pods as it is getting too crowded, therefore we would like to run cluster-node-critical pods in other namespaces.
Waiting for this to be available myself. No reason my Node Exporter dataset has to run in the namespace instead of my namespace.


Question: <del> { "id": "heapster", "kind": "Pod", "apiVersion": "v1beta1", "desiredState": { "manifest": { "version": "v1beta1", "id": "heapster", "containers": [ { "name": "heapster", "image": "kubernetes/heapster" } ] } }, "labels": { "name": "heapster" } } </del>

examples/monitoring/influx-grafana- and friends NEED to be replaced by a replication controller ASAP. As it stands they are bad examples. Also, they should probably move to contrib/monitoring to match contrib/logging and contrib/dns


Question: the Job MUST execute to completion. release: v1.16 file: test/e2e/apps/job.go <ins> - testname: ReplicaSet, completes the scaling of a ReplicaSet subresource codename: '[sig-apps] ReplicaSet Replicaset should have a working scale subresource [Conformance]' description: Create a ReplicaSet (RS) with a single Pod. The Pod MUST be verified that it is running. The RS MUST get and verify the scale subresource count. The RS MUST update and verify the scale subresource. The RS MUST patch and verify a scale subresource. release: v1.21 file: test/e2e/apps/replica_set.go </ins> - testname: Replica Set, adopt matching pods and release non matching pods codename: '[sig-apps] ReplicaSet should adopt matching pods on creation and release no longer matching pods [Conformance]'

[x] two weeks soak end date : 26 Feb 2021 According to this APIsnoop query, there are still some remaining RESOURCENAME endpoints which are untested. with this query you can filter untested endpoints by their category and eligiblity for conformance. e.g below shows a query to find all conformance eligible untested,stable,core endpoints Create a ReplicaSet with a static label Read the ReplicaSetScale Update the ReplicaSetScale Get the ReplicaSetScale to ensure it is updated Patch the ReplicaSetScale Read the ReplicaSetScale to ensure it is Patch Delete the Namespace and ReplicaSet Due to the complexity of setting up the resources for APIService we have used the current e2e test as template. It has been extended in a for review. example endpoint | useragent ----------------------------------------|
patchAppsV1NamespacedReplicaSetScale | (linux/amd64) kubernetes/$Format -- [sig-apps] ReplicaSet Replicaset should have a working scale subresource readAppsV1NamespacedReplicaSetScale | (linux/amd64) kubernetes/$Format -- [sig-apps] ReplicaSet Replicaset should have a working scale subresource replaceAppsV1NamespacedReplicaSetScale | (linux/amd64) kubernetes/$Format -- [sig-apps] ReplicaSet Replicaset should have a working scale subresource (3 rows) If a test with these calls gets merged, test coverage will go up by 3 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture
/triage accepted
Presented in Conformance meeting of 9 Feb 2021 and it was agreed that that method see to achieve the purpose and a PR can be created. A similar approach would be taken for the DepoymentScale endpoints.
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: // getCadvisorPodInfoFromPodUID returns a pod cgroup information by matching the podUID with its CgroupName identifier base name func getCadvisorPodInfoFromPodUID(podUID types.UID, infos map[string]cadvisorapiv2.ContainerInfo) *cadvisorapiv2.ContainerInfo { <del> for key, info := range infos { if cm.IsSystemdStyleName(key) { // Convert to internal cgroup name and take the last component only. internalCgroupName := cm.ParseSystemdToCgroupName(key) key = internalCgroupName[len(internalCgroupName)-1] } else { // Take last component only. key = path.Base(key) } if cm.GetPodCgroupNameSuffix(podUID) == key { return &info } </del> <ins> if info, found := infos[cm.GetPodCgroupNameSuffix(podUID)]; found { return &info </ins> } return nil } <del> // removeTerminatedContainerInfo returns the specified containerInfo but with // the stats of the terminated containers removed. // </del> <ins> // filterTerminatedContainerInfoAndAssembleByPodCgroupKey returns the specified containerInfo but with // the stats of the terminated containers removed and all containerInfos assembled by pod cgroup key. // the first return map is container cgroup name <-> ContainerInfo and // the second return map is pod cgroup key <-> ContainerInfo. </ins> // A ContainerInfo is considered to be of a terminated container if it has an // older CreationTime and zero CPU instantaneous and memory RSS usage. <del> func removeTerminatedContainerInfo(containerInfo map[string]cadvisorapiv2.ContainerInfo) map[string]cadvisorapiv2.ContainerInfo { </del> <ins> func filterTerminatedContainerInfoAndAssembleByPodCgroupKey(containerInfo map[string]cadvisorapiv2.ContainerInfo) (map[string]cadvisorapiv2.ContainerInfo, map[string]cadvisorapiv2.ContainerInfo) { </ins> cinfoMap := make(map[containerID][]containerInfoWithCgroup) <ins> cinfosByPodCgroupKey := make(map[string]cadvisorapiv2.ContainerInfo) </ins> for key, cinfo := range containerInfo { <ins> var podCgroupKey string if cm.IsSystemdStyleName(key) { // Convert to internal cgroup name and take the last component only. internalCgroupName := cm.ParseSystemdToCgroupName(key) podCgroupKey = internalCgroupName[len(internalCgroupName)-1] } else { // Take last component only. podCgroupKey = path.Base(key) } cinfosByPodCgroupKey[podCgroupKey] = cinfo </ins> if !isPodManagedContainer(&cinfo) { continue }


Question: // getCadvisorPodInfoFromPodUID returns a pod cgroup information by matching the podUID with its CgroupName identifier base name func getCadvisorPodInfoFromPodUID(podUID types.UID, infos map[string]cadvisorapiv2.ContainerInfo) *cadvisorapiv2.ContainerInfo { <del> for key, info := range infos { if cm.IsSystemdStyleName(key) { // Convert to internal cgroup name and take the last component only. internalCgroupName := cm.ParseSystemdToCgroupName(key) key = internalCgroupName[len(internalCgroupName)-1] } else { // Take last component only. key = path.Base(key) } if cm.GetPodCgroupNameSuffix(podUID) == key { return &info } </del> <ins> if info, found := infos[cm.GetPodCgroupNameSuffix(podUID)]; found { return &info </ins> } return nil } <del> // removeTerminatedContainerInfo returns the specified containerInfo but with // the stats of the terminated containers removed. // </del> <ins> // filterTerminatedContainerInfoAndAssembleByPodCgroupKey returns the specified containerInfo but with // the stats of the terminated containers removed and all containerInfos assembled by pod cgroup key. // the first return map is container cgroup name <-> ContainerInfo and // the second return map is pod cgroup key <-> ContainerInfo. </ins> // A ContainerInfo is considered to be of a terminated container if it has an // older CreationTime and zero CPU instantaneous and memory RSS usage. <del> func removeTerminatedContainerInfo(containerInfo map[string]cadvisorapiv2.ContainerInfo) map[string]cadvisorapiv2.ContainerInfo { </del> <ins> func filterTerminatedContainerInfoAndAssembleByPodCgroupKey(containerInfo map[string]cadvisorapiv2.ContainerInfo) (map[string]cadvisorapiv2.ContainerInfo, map[string]cadvisorapiv2.ContainerInfo) { </ins> cinfoMap := make(map[containerID][]containerInfoWithCgroup) <ins> cinfosByPodCgroupKey := make(map[string]cadvisorapiv2.ContainerInfo) </ins> for key, cinfo := range containerInfo { <ins> var podCgroupKey string if cm.IsSystemdStyleName(key) { // Convert to internal cgroup name and take the last component only. internalCgroupName := cm.ParseSystemdToCgroupName(key) podCgroupKey = internalCgroupName[len(internalCgroupName)-1] } else { // Take last component only. podCgroupKey = path.Base(key) } cinfosByPodCgroupKey[podCgroupKey] = cinfo </ins> if !isPodManagedContainer(&cinfo) { continue }


Question: // according to the corresponding EndpointSlice addressType field. Consumers // must handle different types of addresses in the context of their own // capabilities. This must contain at least one address but no more than <del> // 100. </del> <ins> // 100. These are all assumed to be fungible and clients may choose to only // use the first element. Refer to: https://issue.k8s.io/106267 </ins> // +listType=set Addresses []string `json:"addresses" protobuf:"bytes,1,rep,name=addresses"` // conditions contains information about the current status of the endpoint.

An can contain up to 1000 , which can each contain up to 100 . Or at least, that's what the docs say, but actually, kube-proxy (or more specifically, ) only looks at the first Address in each Endpoint: (This is compatible with how works, since it would only generate multi-Address Endpoints if a pod had multiple of the same address family, which is not allowed. But it's not compatible with manually-generated EndpointSlices.) Either all are used, or else the docs state that only is used and the field is an array solely for historical reasons.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig network /priority important-longterm /cc
All addresses in the slice must be of the same type, do you mean creating an Slice with different address types in the Addresses field?
I'm not sure if/how kube-proxy should handle an endpoint with more than 1 address. I think the goal of kube-proxy is to give each endpoint an equivalent probability of being selected. To extend that logic here, I think we'd need to take the probability assigned to an endpoint and divide it by the number of addresses attached to that endpoint. So if an endpoint has a 50% probability of being selected, and it provides 50 addresses, each should end up with a 1% probability of being selected. Of course with the way we need to do descending probabilities, the logic may get a bit complicated. This isn't as clear as it should be, but I don't think kube-proxy or any consumer of the EndpointSlice API needs to commit to supporting every value specified by that API. For example, many may filter by address type, Service, or some other means. I think it's also valid for a consumer to say they only support a single address per family per endpoint. So all of that to say, I'd support either documenting that kube-proxy only selects the first address in an endpoint, or updating kube-proxy to support more than one address per endpoint.


Question: // according to the corresponding EndpointSlice addressType field. Consumers // must handle different types of addresses in the context of their own // capabilities. This must contain at least one address but no more than <del> // 100. </del> <ins> // 100. These are all assumed to be fungible and clients may choose to only // use the first element. Refer to: https://issue.k8s.io/106267 </ins> // +listType=set Addresses []string `json:"addresses" protobuf:"bytes,1,rep,name=addresses"` // conditions contains information about the current status of the endpoint.

Given the number of changes currently in flight for kube-proxy, documentation seems like a safer initial approach to me.
/assign
can I have this editing only the first address(Address[0]) in Endpoint.Address to be used. The use of an array in this field is based on historical reasons. Isn't that right
thanks for volunteering to help with this one! I know has some big kube-proxy changes/fixes in flight right now, so it would be good to be sure you don't conflict with him. Otherwise, as long as isn't already working on this, I think you're free to have it.
I don't think we agreed that the array is for historical reasons only. We aren't currently using it, but we aren't ready to say that we won't use it in the future at this point. So, the docs should warn that kube-proxy currently ignores additional addresses, but it shouldn't imply that it's actually wrong to have additional addresses I'm not
thanks i will work on this
It was for a reason, but I think that reason was mixed-family slices (before we decided to break it up into single-family slices). What would it mean to have multiple addresses here? Are they assumed to be equivalent? Or do they have different context (like a split-horizon)? If the latter, there isn't any metadata here (i.e. it's not a struct like to allow new fields). I am inclined to believe it was meant as equivalent, and as such it's probably OK to document that assumption clearly. That said, I don't see how anyone could use multiple addresses in a meaningful way - they are either assumed equivalent or they lack enough metadata to distinguish their differences. ?? If we do document the equivalence, it should be generically worded. E.g. "These addresses are assumed to be equivalent and some consumers may choose to ignore all but the first value."
if it was meant as equivalent, why put multiple same addresses into Addresses? I don't understand what the purpose of this is.
The first version of the API had both IP families in a single slice, so it made sense to have multiple addresses. We changed that, though we didn't immediately remove the old support. So my best guess is that we should have converted this to a singular value when we did that, but we missed it.


Question: // according to the corresponding EndpointSlice addressType field. Consumers // must handle different types of addresses in the context of their own // capabilities. This must contain at least one address but no more than <del> // 100. </del> <ins> // 100. These are all assumed to be fungible and clients may choose to only // use the first element. Refer to: https://issue.k8s.io/106267 </ins> // +listType=set Addresses []string `json:"addresses" protobuf:"bytes,1,rep,name=addresses"` // conditions contains information about the current status of the endpoint.

On Wed, Nov 24, 2021, 5:45 PM Cyclinder
so we should change the API doc? Or It's just that documenting it will only use the first address in the , like you said above
I'd like some of the others to comment, but it's this a US holiday, so it might be a few days..
So maybe we should just declare that the array is essentially a bug, and only the first address is used? Maybe something should warn if you ever set additional addresses? (FTR, the context that I noticed this in was that I was adding unit tests, and was creating EndpointSlices with a single Endpoint and multiple Addresses, which doesn't work; you need multiple Endpoints with a single Address each.)
I think the simplest and most compat approach is to say "these are all assumed to be fungible and clients may choose to only use the first element". Thoughts?
I think the simplest and most compat approach is to say "these are all assumed to be fungible and clients may choose to only use the first element". Big +1 to that approach. I don't think it's necessarily invalid to set multiple addresses here, there may actually be some use cases for it, but adding this kind of statement should help clear things up significantly.
hmm as we push for multi homed pod this will make a lot of sense. There might also be a chance for community to provide a drop-in replacement for proxy that does much more than we do now. I think we should update the docs and don't modify anything for now.
Note that this API doesn't include per-IP metadata, so a multi-homed pod would be challenging with this mechanic. I think it's maybe time to reconsider multi-net more holistically. On Mon, Dec 13, 2021, 10:24 AM Khaled Henidak (Kal) <
:thinking:
If they're fungible, then "clients may choose to use only one of the elements" (or "only a subset"?). No reason to call out the first one specifically.
That's fair On Tue, Dec 14, 2021, 6:09 AM Dan Winship
I would like to add to this issue that it would makes sense to have kube-proxy reduce the subset of endpoints in the endpointslice based on the label of the node and the & attributes of the endpoint.


Question: // according to the corresponding EndpointSlice addressType field. Consumers // must handle different types of addresses in the context of their own // capabilities. This must contain at least one address but no more than <del> // 100. </del> <ins> // 100. These are all assumed to be fungible and clients may choose to only // use the first element. Refer to: https://issue.k8s.io/106267 </ins> // +listType=set Addresses []string `json:"addresses" protobuf:"bytes,1,rep,name=addresses"` // conditions contains information about the current status of the endpoint.

Taking only into consideration the first endpoint which is in the same zone of the node or has no zone dependency and is ready for serving. & Would it make sense to reopen this issue?
I don't follow entirely what you are suggesting, can you elaborate?
kube-proxy should understand which zone a node is in and prefer endpoints in the same zone.
but that is unrelated to this issue, topology is considered See KEPs related, topology-aware-routing was replaced by topology-aware-hints
thank you for clarifying. It seems ambiguous to me to have a zone attribute and a hint attribute. I fail to implement the hinting on a service in v1.23.8, but this is urelated to this issue.
heh, don't worry, this topic is complex and we went through different solutions, you have more historical context on the KEPs I linked above and this thread in the mailing list


Question: numPodsToSchedule: 2000 - template: desc: MixedSchedulingBasePod <ins> nodes: nodeTemplatePath: config/node-default.yaml labelNodePrepareStrategy: labelKey: "topology.kubernetes.io/zone" labelValues: ["zone1"] </ins> initPods: - podTemplatePath: config/pod-default.yaml - podTemplatePath: config/pod-with-pod-affinity.yaml

What happened: The performance benchmark does not finish even after 10 minutes of waiting. This is because the nodes do not have a label set but the pod affinity rules in the benchmark's pod spec expects one: What you expected to happen: The test should finish. How to reproduce it (as minimally and precisely as possible):
/sig scheduling /sig testing /kind bug


Question: } } <ins> func TestExternalIPsReject(t *testing.T) { ipt := iptablestest.NewFake() fp := NewFakeProxier(ipt) svcIP := "10.20.30.41" svcPort := 80 svcExternalIPs := "50.60.70.81" svcPortName := proxy.ServicePortName{ NamespacedName: makeNSN("ns1", "svc1"), Port:           "p80", } makeServiceMap(fp, makeTestService(svcPortName.Namespace, svcPortName.Name, func(svc *api.Service) { svc.Spec.Type = "ClusterIP" svc.Spec.ClusterIP = svcIP svc.Spec.ExternalIPs = []string{svcExternalIPs} svc.Spec.Ports = []api.ServicePort{{ Name:       svcPortName.Port, Port:       int32(svcPort), Protocol:   api.ProtocolTCP, TargetPort: intstr.FromInt(svcPort), }} }), ) makeEndpointsMap(fp) fp.syncProxyRules(syncReasonForce) kubeSvcRules := ipt.GetRules(string(kubeServicesChain)) if !hasJump(kubeSvcRules, iptablestest.Reject, svcExternalIPs, svcPort) { errorf(fmt.Sprintf("Failed to a %v rule for externalIP %v with no endpoints", iptablestest.Reject, svcPortName), kubeSvcRules, t) } } </ins> func TestNodePortReject(t *testing.T) { ipt := iptablestest.NewFake() fp := NewFakeProxier(ipt)

Is this a request for help? No What keywords did you search in Kubernetes issues before filing this one? externalIPs kube-proxy connection accepted no pods running. icmp reject rules. Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT Kubernetes version (use ): kubectl version Environment: Cloud provider or hardware configuration: Bare Metal. OS (e.g. from /etc/os-release):Kernel (e.g. ):Install tools: Others: What happened: is exposed using externalIP construct. pods are backing the service (using scale deployment 0). The pod is expected to run a TCP based server (non-http) client connects to this service using externalIP as dest-ip; connection gets accepted by kube-proxy. What you expected to happen: When no pods are backing service; the connection should have been rejected and definitely not accepted by kube-proxy. How to reproduce it (as minimally and precisely as possible): a service and expose it using externalIP construct pods are present; bring them down using"kubectl scale deployment my_svc --replicas 0 --namespace xyz initiates the connection to this service using externalIP and connection goes fine but to kube-proxy. Anything else we need to know: Info on iptable rules - When service and pods are present the rules like these - pods are not present; iptables rules look like this - that the reject rule exist only for the destination=ServiceIP=10.100.200.6/32 no pods are present; client gets connected to kube-proxy On VM - Based on and ; I manually installed reject rule on INPUT chain. I do not see the issue happening again - meaning client then cannot connect to the server when no pods are present.
I have a suggestion for installing icmp reject rules. Why cant icmp reject rule always present in INPUT chain? kube-proxy do not really need to check if pods are present or not for installing these rules. For the svcip/externaip to pod-ip translation; kube-proxy anyways install the rule to do DNAT in which case the icmp reject rule from INPUT wont be hit. In case when DNAT fails to happen - e.g. in cases like pods not present - icmp reject rule will anyways kick in to reject such packet.


Question: } } <ins> func TestExternalIPsReject(t *testing.T) { ipt := iptablestest.NewFake() fp := NewFakeProxier(ipt) svcIP := "10.20.30.41" svcPort := 80 svcExternalIPs := "50.60.70.81" svcPortName := proxy.ServicePortName{ NamespacedName: makeNSN("ns1", "svc1"), Port:           "p80", } makeServiceMap(fp, makeTestService(svcPortName.Namespace, svcPortName.Name, func(svc *api.Service) { svc.Spec.Type = "ClusterIP" svc.Spec.ClusterIP = svcIP svc.Spec.ExternalIPs = []string{svcExternalIPs} svc.Spec.Ports = []api.ServicePort{{ Name:       svcPortName.Port, Port:       int32(svcPort), Protocol:   api.ProtocolTCP, TargetPort: intstr.FromInt(svcPort), }} }), ) makeEndpointsMap(fp) fp.syncProxyRules(syncReasonForce) kubeSvcRules := ipt.GetRules(string(kubeServicesChain)) if !hasJump(kubeSvcRules, iptablestest.Reject, svcExternalIPs, svcPort) { errorf(fmt.Sprintf("Failed to a %v rule for externalIP %v with no endpoints", iptablestest.Reject, svcPortName), kubeSvcRules, t) } } </ins> func TestNodePortReject(t *testing.T) { ipt := iptablestest.NewFake() fp := NewFakeProxier(ipt)

This would be better than accidentally giving the packet to kube-proxy which is always listening on the port and accepting this accidental connection even when it does not intend to. Actually, for now that's how I am putting workaround for this problem in our setup.
That might be a clever answer - I hadn't thought of it that way. Would need good tests and comments.
I can give a shot at testing it. I didnt find a good reference document / todo for testing such changes. If you can help me locate it; I can give it a try.


Question: if err != nil { t.Errorf("Attach() failed: %s", err) } <del> if expectAttach && attachID == "" { t.Errorf("Expected attachID, got nothing") } if !expectAttach && attachID != "" { </del> <ins> if attachID != "" { </ins> t.Errorf("Expected empty attachID, got %q", attachID) } }(spec, test.expectVolumeAttachment)

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: Followup to In 1.15, we can merge this commit: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
/assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


Question: if kc.RegistryPullQPS < 0 { allErrors = append(allErrors, fmt.Errorf("invalid configuration: RegistryPullQPS (--registry-qps) %v must not be a negative number", kc.RegistryPullQPS)) } <ins> if kc.RotateCertificates && !localFeatureGate.Enabled(features.RotateKubeletClientCertificate) { allErrors = append(allErrors, fmt.Errorf("invalid configuration: RotateCertificates %v requires feature gate RotateKubeletClientCertificate", kc.RotateCertificates)) } </ins> if kc.ServerTLSBootstrap && !localFeatureGate.Enabled(features.RotateKubeletServerCertificate) { allErrors = append(allErrors, fmt.Errorf("invalid configuration: ServerTLSBootstrap %v requires feature gate RotateKubeletServerCertificate", kc.ServerTLSBootstrap)) }

/kind cleanup What happened: and both require opt-in configuration knobs to the kubelet. They are kinda global settings, you probably either opt into dynamic cert rotation or you don't, so they qualify for being in the struct. The knob for enabling dynamic serving certs was (I fully support that :+1:). What you expected to happen: I'd like the same to happen to  to be consistent. (Also, we're planning to use the config file for the kubelet by default in kubeadm soon, and it'd be nice to be able to set this value then in a structured manner) /assign


Question: if kc.RegistryPullQPS < 0 { allErrors = append(allErrors, fmt.Errorf("invalid configuration: RegistryPullQPS (--registry-qps) %v must not be a negative number", kc.RegistryPullQPS)) } <ins> if kc.RotateCertificates && !localFeatureGate.Enabled(features.RotateKubeletClientCertificate) { allErrors = append(allErrors, fmt.Errorf("invalid configuration: RotateCertificates %v requires feature gate RotateKubeletClientCertificate", kc.RotateCertificates)) } </ins> if kc.ServerTLSBootstrap && !localFeatureGate.Enabled(features.RotateKubeletServerCertificate) { allErrors = append(allErrors, fmt.Errorf("invalid configuration: ServerTLSBootstrap %v requires feature gate RotateKubeletServerCertificate", kc.ServerTLSBootstrap)) }

Flag name: Help text: <Warning: Beta featureAuto rotate the kubelet client certificates by requesting new certificates from the kube-apiserver when the certificate expiration approaches. This is part of migrating the Kubelet command-line to a Kubernetes-style API. The flag should either be migrated to the Kubelet's API group, or simply removed from the Kubelet. If this could be considered an instance-specific flag, or a descriptor of local topology managed by the Kubelet, see: If this flag is only registered in os-specific builds, see: /assign /sig node /kind feature /priority important-soon /milestone v1.11 /status approved-for-milestone
[MILESTONENOTIFIER] Milestone Issue: Up-to-date for process <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. : New functionality. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details


Question: # This seems to matter to some tools export GO15VENDOREXPERIMENT=1 <ins> } kube::golang::setup_gomaxprocs() { </ins> # GOMAXPROCS by default does not reflect the number of cpu(s) available # when running in a container, please see https://github.com/golang/go/issues/33803 <del> if ! command -v ncpu >/dev/null 2>&1; then # shellcheck disable=SC2164 pushd "${KUBE_ROOT}/hack/tools" >/dev/null GO111MODULE=on go install ./ncpu # shellcheck disable=SC2164 popd >/dev/null </del> <ins> if [[ -z "${GOMAXPROCS:-}" ]]; then if ! command -v ncpu >/dev/null 2>&1; then # shellcheck disable=SC2164 pushd "${KUBE_ROOT}/hack/tools" >/dev/null GO111MODULE=on go install ./ncpu || echo "Will not automatically set GOMAXPROCS" # shellcheck disable=SC2164 popd >/dev/null fi if command -v ncpu >/dev/null 2>&1; then GOMAXPROCS=$(ncpu) export GOMAXPROCS kube::log::status "Set GOMAXPROCS automatically to ${GOMAXPROCS}" fi </ins> fi <del> GOMAXPROCS=${GOMAXPROCS:-$(ncpu)} export GOMAXPROCS kube::log::status "Setting GOMAXPROCS: ${GOMAXPROCS}" </del> } # This will take binaries from $GOPATH/bin and copy them to the appropriate

fails in build environments without network connection due to missing in-tree vendored dependency . Build to succeed with without needing to download any go dependencies. In a network-less build environment, run . No response Building off of the v1.28.0-rc.0 tag N/A <details</detailsN/A N/A N/A
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig release
This is a hack/tools dependency here by , he may have more context
hack/tools aren't vendored. if you want to ensure those are available offline, will do that
My hack for building with OBS (), see
Is this actually a bug?
in I agree we probably don't want to calls in the build path... in code generation or verify tools, that's more ok, but the build target of a particular SHA should be able to execute hermetically
opened to scope down when we do this to test targets, and avoid the install if the env already explicitly set GOMAXPROCS
fix merged to master and head of release-1.28 branch


Question: export SECONDARY_RANGE_NAME="pods-default" export STORAGE_BACKEND="etcd3" export STORAGE_MEDIA_TYPE="application/vnd.kubernetes.protobuf" <del> export ETCD_IMAGE=3.5.0-rc.0-0 export ETCD_VERSION=3.5.0-rc.0 </del> <ins> export ETCD_IMAGE=3.5.0-0 export ETCD_VERSION=3.5.0 </ins> # Upgrade master with updated kube envs "${KUBE_ROOT}/cluster/gce/upgrade.sh" -M -l

/sig scalability This issue is tracking process of upgrading etcd to 3.5.0 planned for 1.22 K8s release. Etcd has recently released Beta versions and is starting the process of correctness and performance testing. To reduce change of finding performance regression late, we will start qualifying Beta Etcd releases on GKE master branch and will upgrade to 3.5.0 as soon as it is available. Steps: [x] Do preliminary scale testing of etcd v3.5.0-beta.3 release [x] Release etcd v3.5.0-beta.3 image (, ) [x] Release etcd v3.5.0-rc.0 image () [x] Upgrade etcd server to v3.5.0-rc.0 () [x] Wait for 3.5.0 release and upgrade etcd client () [x] Release etcd v3.5.0 image (, ) [x] Upgrade etcd server to v3.5.0 /milestone v1.22
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/area etcd
Does this meant that the 3.5.0 should be promoted like
Yes, created
/area release-eng release-team /sig release For tracking: cc:
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /area release-eng release-team /sig release cc: Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I do think this can be pointed in the Major Themes for the 1.22 release. I would like your opinions on this. If you do think it has a place in it, I will add it to the .
How can I be a contributor to this issue? what are the technical checklist that needs to be achieved to close out on the issue?


Question: export SECONDARY_RANGE_NAME="pods-default" export STORAGE_BACKEND="etcd3" export STORAGE_MEDIA_TYPE="application/vnd.kubernetes.protobuf" <del> export ETCD_IMAGE=3.5.0-rc.0-0 export ETCD_VERSION=3.5.0-rc.0 </del> <ins> export ETCD_IMAGE=3.5.0-0 export ETCD_VERSION=3.5.0 </ins> # Upgrade master with updated kube envs "${KUBE_ROOT}/cluster/gce/upgrade.sh" -M -l

PR to update Etcd is ready, but we found a blocker for merging . There is one test failing I will take a look into it on Monday, still more eyes on the problem could speed up resolution.
We'll also need a new kube-cross image w/ etcd v3.5.0. I'm working on that here:
We just need to upgrade the server to use new image like in
ok, looks like is busy. Let me take a look /assign
Thanks.


Question: tags = append(tags, tag) } <del> tagRequest := &ec2.CreateTagsInput{} tagRequest.Resources = []*string{&groupID} tagRequest.Tags = tags if _, err := s.createTags(tagRequest); err != nil { // Not clear how to recover fully from this; we're OK because we don't match on tags, but that is a little odd return "", fmt.Errorf("error tagging security group: %v", err) </del> <ins> if len(tags) > 0 { tagRequest := &ec2.CreateTagsInput{} tagRequest.Resources = []*string{&groupID} tagRequest.Tags = tags if _, err := s.createTags(tagRequest); err != nil { // Not clear how to recover fully from this; we're OK because we don't match on tags, but that is a little odd return "", fmt.Errorf("error tagging security group: %v", err) } </ins> } return groupID, nil }

Getting the following error when creating a service with type when kubernetes has been deployed without tagging the vpc/node. This prevents from using ELBs with the AWS cloud provider when deploying kubernetes in an existing vpc (which cannot be tagged because they already have).
I also experienced this failure when trying to provision Kubernetes on AWS with kube-up and specifying an explicit, existing VPC_ID — which wasn't tagged, but then I didn't know that was required :)
I see we have a pr, can we also update the docs in time for 1.2.0?
sure, what do you want me to update in the docs exactly?


Question: name = "go_default_test", srcs = [ "aws_ebs_test.go", <ins> "azure_disk_test.go", </ins> "gce_pd_test.go", ], embed = [":go_default_library"],

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Azure is not listed as "migratable" in could refer to Why is this needed:
/sig azure /assign


Question: "source":                   source, }.AsSelector().String() eventListOptions = metav1.ListOptions{FieldSelector: selector} <del> ginkgo.By("Create the test log file") framework.ExpectNoError(err) </del> ginkgo.By("Create config map for the node problem detector") _, err = c.CoreV1().ConfigMaps(ns).Create(context.TODO(), &v1.ConfigMap{ ObjectMeta: metav1.ObjectMeta{Name: configName}, Data:       map[string]string{path.Base(configFile): config}, }, metav1.CreateOptions{}) framework.ExpectNoError(err) <ins> ginkgo.By("Create the service account for node problem detector") _, err = f.ClientSet.CoreV1().ServiceAccounts(f.Namespace.Name).Create(context.TODO(), &v1.ServiceAccount{ ObjectMeta: metav1.ObjectMeta{ Name:      name, Namespace: ns, }, }, metav1.CreateOptions{}) framework.ExpectNoError(err) ginkgo.By("Bind the cluster role system:node-problem-detector with the service account for node problem detector") _, err = f.ClientSet.RbacV1().ClusterRoleBindings().Create(context.TODO(), &rbacv1.ClusterRoleBinding{ ObjectMeta: metav1.ObjectMeta{ Name: name, }, RoleRef: rbacv1.RoleRef{ APIGroup: rbacv1.GroupName, Kind:     "ClusterRole", Name:     "system:node-problem-detector", }, Subjects: []rbacv1.Subject{ { Kind:      rbacv1.ServiceAccountKind, Name:      name, Namespace: ns, }, }, }, metav1.CreateOptions{}) framework.ExpectNoError(err) </ins> ginkgo.By("Create the node problem detector") hostPathType := new(v1.HostPathType) <del> *hostPathType = v1.HostPathType(string(v1.HostPathFileOrCreate)) </del> <ins> *hostPathType = v1.HostPathFileOrCreate </ins> pod := f.PodClient().CreateSync(&v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name: name, }, Spec: v1.PodSpec{ <del> HostNetwork:     true, SecurityContext: &v1.PodSecurityContext{}, </del> <ins> HostNetwork:        true, SecurityContext:    &v1.PodSecurityContext{}, ServiceAccountName: name, </ins> Volumes: []v1.Volume{ { Name: configVolume,

Which jobs are failing: Which test(s) are failing: Since when has it been failing: Last successful run (23 Oct 2020, 10:07 PDT): First unsuccessful run (23 Oct 2020, 12:08 PDT): Testgrid link: Reason for failure: Anything else we need to know: presubmit job in repository also started to fail in the same fashion. Attempted to debug it myself, found out the following stuff: last successful run was accessing the apiserver via HTTP: all unsuccessful runs attempted to access the apiserver via HTTPS but to no avail: perused the history of to find out that the PR is almost certainly the reason of the failures (or rather uncovered some problem with the NPD or its tests): first, it coincides with the start of the failures of , second, we changed the default hostname of the apiserver in node tests in that PR. Judging by the code and the changes in the aforementioned PR I think that a quick mitigation would be to add a flag to the of these jobs. /assign /sig node /cc
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Opened for a mitigation that would make those jobs stop failing.
This won't work, unfortunately. Copying response:
I have filed to address this issue, but it seems I could not run the test in advance to verify my patch, did you know how to manually trigger the presubmit job?
Unfortunately I don't :/. My PR was also a wild guess. could you take a look at the proposed fix? Thanks once again for your help - really appreciated :).
the submission of hasn't fixed the problems, unfortunately :/. Sample failing run: We still have some peculiar problem related to health checking: and now NPD panics, too:
Thanks for your report, I believe the reason is admission controller has been disabled in node e2e tests: We need to figure out a new approach now. /reopen
Reopened this issue.


Question: "source":                   source, }.AsSelector().String() eventListOptions = metav1.ListOptions{FieldSelector: selector} <del> ginkgo.By("Create the test log file") framework.ExpectNoError(err) </del> ginkgo.By("Create config map for the node problem detector") _, err = c.CoreV1().ConfigMaps(ns).Create(context.TODO(), &v1.ConfigMap{ ObjectMeta: metav1.ObjectMeta{Name: configName}, Data:       map[string]string{path.Base(configFile): config}, }, metav1.CreateOptions{}) framework.ExpectNoError(err) <ins> ginkgo.By("Create the service account for node problem detector") _, err = f.ClientSet.CoreV1().ServiceAccounts(f.Namespace.Name).Create(context.TODO(), &v1.ServiceAccount{ ObjectMeta: metav1.ObjectMeta{ Name:      name, Namespace: ns, }, }, metav1.CreateOptions{}) framework.ExpectNoError(err) ginkgo.By("Bind the cluster role system:node-problem-detector with the service account for node problem detector") _, err = f.ClientSet.RbacV1().ClusterRoleBindings().Create(context.TODO(), &rbacv1.ClusterRoleBinding{ ObjectMeta: metav1.ObjectMeta{ Name: name, }, RoleRef: rbacv1.RoleRef{ APIGroup: rbacv1.GroupName, Kind:     "ClusterRole", Name:     "system:node-problem-detector", }, Subjects: []rbacv1.Subject{ { Kind:      rbacv1.ServiceAccountKind, Name:      name, Namespace: ns, }, }, }, metav1.CreateOptions{}) framework.ExpectNoError(err) </ins> ginkgo.By("Create the node problem detector") hostPathType := new(v1.HostPathType) <del> *hostPathType = v1.HostPathType(string(v1.HostPathFileOrCreate)) </del> <ins> *hostPathType = v1.HostPathFileOrCreate </ins> pod := f.PodClient().CreateSync(&v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name: name, }, Spec: v1.PodSpec{ <del> HostNetwork:     true, SecurityContext: &v1.PodSecurityContext{}, </del> <ins> HostNetwork:        true, SecurityContext:    &v1.PodSecurityContext{}, ServiceAccountName: name, </ins> Volumes: []v1.Volume{ { Name: configVolume,

<detailsIn response to : Thanks for your report, I believe the reason is admission controller has been disabled in node e2e tests: We need to figure out a new approach now. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
filed . I hope we could completely address this issue this time.
/milestone v1.20


Question: if err != nil { return nil, err } <ins> // Avoid collector collects it as a timestamped metric // See PR #95210 and #97006 for more details. machineInfo.Timestamp = time.Time{} </ins> klet.setCachedMachineInfo(machineInfo) imageBackOff := flowcontrol.NewBackOff(backOffPeriod, MaxContainerBackOff)

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: prometheus cadvisor machine metrics from kubelet are missing What you expected to happen: How to reproduce it (as minimally and precisely as possible): install k8s 1.19, compare behavior to k8s <1.19 Anything else we need to know?: Looks like missing call to in , machine metrics collection was split out in cadvisor 0.37 Environment: Kubernetes version (use ):
/sig node (hope that's the correct sig :)
Suggest from as an alternative (declared as "stable").
Related PR Their output format changed too:
Multiple queries is invalide now. Is there any workaroud for this? Please help. Thanks.
if you run see for an alternative metric.
+1 on the issue. This issue has caused the node metric collection for Azure Monitor for Containers to be broken which also impacts alerting.
Exposed metrics are a kind of an API and should not be removed without warning.
any news, when this will be fixed? Thanks in advance!
It is not possible to have a metric available without exposing it. To provide stability information on metrics, similar to APIs, you might want to read up on the . There currently are no metrics in Kubernetes itself that are considered stable, but SIG Instrumentation hopes to be selecting some for the 1.21 release. This particular issue seems like an unintended bug as opposed to an intentional deprecation. Suspect this issue got lost in SIG Node's backlog. I'll put it on the agenda for the next SIG Instrumentation meeting. /sig instrumentation
Backports in and


Question: conntrack: maxPerCore: 32768 min: 131072 <ins> tcpBeLiberal: false </ins> tcpCloseWaitTimeout: 1h0m0s tcpEstablishedTimeout: 24h0m0s udpStreamTimeout: 0s

What happened: When kube-proxy is configured with a cluster-cidr range, it drops packets with an INVALID conntrack state for all sources on the KUBE-FORWARD chain. This is meant to prevent spurious retransmits in long-lived TCP connections to a service IP, but it prevents clusters from having asymmetrical routing on private subnets with services outside of a cluster. What you expected to happen: Packets on the KUBE-FORWARD chain with an INVALID conntrack state should be dropped for only sources inside of the cluster-cidr if set. How to reproduce it (as minimally and precisely as possible): N/A Anything else we need to know?: Change to drop INVALID conntrack packets was in Backing out that commit and compiling is my current workaround. Environment: Kubernetes version (use ): 1.17.2 Cloud provider or hardware configuration: OS (e.g: ):Kernel (e.g. ):
/sig network
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
I am reading but I don't fully grok what caused the problem in the first place, or why we think "only drop if it was in-cluster" is a correct fix? Edit, OK I get the original problem, but still don't understand why packets from outside the pod range would not exhibit , thereby requiring yet another revisit of this problem. Is this a conntrack bug? Should it be applying the DNAT?
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
I can try to give a bit more context to our setup, but the knowledge I have around this all dates back to Feb, unfortuately, so I may have forgotten details. We were replacing our mesos cluster with a kubernetes cluster in Feb and using a custom ingress solution (HAProxy w/an outside process configuring it for pod ips / mesos ip-ports) for both. The ingress nodes lived in the private subnet, but outside of the kubernetes cluster cidr range. When we upgraded from kubernetes 1.14 to kubernetes 1.15 (the first version with PR ) our ingress could no longer talk to any pod ips.


Question: conntrack: maxPerCore: 32768 min: 131072 <ins> tcpBeLiberal: false </ins> tcpCloseWaitTimeout: 1h0m0s tcpEstablishedTimeout: 24h0m0s udpStreamTimeout: 0s

I discovered that the KUBE-FORWARD chain had a rule to drop invalid packets, which was in Compiling kubernetes 1.15 without this commit fixed our problems, and we have been carrying that patch forward ever since. I would like to get rid of our forked kubernetes, hence this ticket. When I was writing this ticket up, I also found also mentions the same issue with the iptables dropping INVALID conntrack state packets near the end, regarding asymmetrical routing. Edit, OK I get the original problem, but still don't understand why packets from outside the pod range would not exhibit , thereby requiring yet another revisit of this problem. I think you are correct, I can imagine packets from outside of a cluster could exhibit the same behavior as packets inside the cluster and cause spurious resets. However, I am wondering if kube-proxy should be managing packets from outside of the cluster on a node... That seems outside of the scope of kube-proxy.
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
I took a stab at implementing what I think is the correct behavior for kube-proxy in .
/assign
I think that another option is to set
I don't understand how that will help me here. The packets with a ctstate INVALID are not generated due to conntrack saturation (), but due to asymmetric routing in our network. See for an example of a similar setup.
are the packets on your asymmetric routing setup marked as invalid if you set ? it should only mark the ones that are out of window per the description
/triage accepted
I'd love to pin this down, but I don't have the means to set up a repro. Is there a way I can set up a way to experiment?
We were bit by this too after revisiting an old service that had been unused since the heady days of pre-, and would also like to see the proposed patch merged (or a configuration option to prevent insertion of the INVALID state drop rule - either suits us). I unfortunately don't have any useful input as to setting up for reproduction. In our case we have a pretty odd anycast set up with a small pool of (public) addresses managed by calico.


Question: conntrack: maxPerCore: 32768 min: 131072 <ins> tcpBeLiberal: false </ins> tcpCloseWaitTimeout: 1h0m0s tcpEstablishedTimeout: 24h0m0s udpStreamTimeout: 0s

However I do seem to have a dirty workaround for small clusters. By setting notrack (ctstate UNTRACKED?) in the raw table for packets in our specific ranges that would otherwise match ctstate INVALID, the packets aren't dropped. Which may be of help to someone else who finds this issue.
We are seeing something very similar on large file downloads and our issue exactly fits this one's subject. These RESET (corrected after are happening intermittently and I am seeing similar behavior mentioned in the blog mentioned below. I am unsure if someone has found a workaround for this issue or not, so yeah my +1 to this issue being looked up by the community.
We're using as a solution for the connection reset we were seeing and might fix this issue.
Let me try to avoid that this thread derails and see if I can get current status: -proxy install rules to drop INVALID cstate connections, this solves the ~REJECT~ RESET problems are some scenarios, with asymmetric routing per example, that this rules may drop unrelated traffic. The request is to remove the rule that drops those INVALID states or to try to restrict it, and that can cause that people start to have the ~REJECT~ RESET problems and drops again. If we have a reproducer we can be more precise in the final solution, but removing one rule without any strong evidence in kube-proxy is very risky As a workaround for the asymetric routing problem, I suggested to set the nfconntracktcpbeliberal on the kernel: I think that will solve the problems, it will be good if someone can test it and provide feedback
No sadly, its my bad, it is not REJECT, it is RESET that's happening and conntrack has nothing to do with it, mostly (I hope). Kubeproxy is doing what is intended but specifically when there are external agents in play, the packet doesn't sends an ACK back to the response of SYN-ACK. Our case, we hit A to initiate a download, A calls B to get the action completed to generate download data for the user finally via their browser. A and B both are on k8s, we saw A starts and a complete cycle goes for sometime until there multiple parallel downloads requested where few gets completed and few gets a RESET.


Question: conntrack: maxPerCore: 32768 min: 131072 <ins> tcpBeLiberal: false </ins> tcpCloseWaitTimeout: 1h0m0s tcpEstablishedTimeout: 24h0m0s udpStreamTimeout: 0s

On primary debugging, it appears that RESET was getting invoked by the pod but pod is just sending a RST-ACK not RST itself. The missing packet is never found in the default route, unless we see that there is a chance where ACK is missed and we found some lines in tcpdump with missing sequence for ACKs. Thats why i doubt my issue is related here. One more weird coincidence is few downloads have spurious transmission.
Thankyou, I will take a look, but mostly I would like to wait before using it, debug more to understand the actual cause and then act on it.
sorry , I wanted to say RESET, thanks for catching it. If you think is different and you can provide information and how to reproduce it, I suggest you to open a new issue
I will sure do, when it is different. For now I would like to watch and submit my findings if it is related. PS: sysctl -w net.netfilter.nfconntracktcpbeliberal=1 is already implemented and we are testing if it does something helpful. so far it doesn't looks like it. thanks for your help.
/area kube-proxy
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle rotten
I have the feeling we are also affected by this with a standard calico installation: every node announces a /122 IPv6 network to the upstream router the upstream router uses ECMP to route the packet back packets that are sent from* the pod and returned via asymmetric path from outside seem to be dropped
and if you set ?
Same behaviour with set on all nodes, the outgoing pod traffic does not return. And I instantly see packets going back into the drop rule:
:thinking:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.


Question: conntrack: maxPerCore: 32768 min: 131072 <ins> tcpBeLiberal: false </ins> tcpCloseWaitTimeout: 1h0m0s tcpEstablishedTimeout: 24h0m0s udpStreamTimeout: 0s

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hi, are you still working on this? Same problem happened in our clusters, just because of asymmetrical routing.
/reopen
Reopened this issue. <detailsIn response to : /reopen </details
Also the command suggested by is helping us, but I still think this should be a bug of kube-proxy.
The problem here is: what happens if the solutions are exclusive or we regress? The addition of this rule fixed a legit bug that was documented and there is also a nice blog explaining the details Removing this rule without a repro and understanding what is the root cause of this issue is not really the best way to proceed. I think that people affected should try to find the root cause and explain the problem or just provide a way to reproduce it, so we can fix it
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at .


Question: conntrack: maxPerCore: 32768 min: 131072 <ins> tcpBeLiberal: false </ins> tcpCloseWaitTimeout: 1h0m0s tcpEstablishedTimeout: 24h0m0s udpStreamTimeout: 0s

/close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The same problem happened in our cluster, the same as because of asymmetrical routing. the command suggested by doesn't work for us, and I delete the iptables rule and add new rule(refer to ), it works I think that makes sense I'll put up a reproduction of the gist link later on /reopen
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The main problem is how to be completely sure we don't regress, this comment from Tim is spot on
I'm trying to check that non-local connections will still reset without the INVALID Rule
I think the patch in is wrong; it changes the rule from "if INVALID then drop" to "if INVALID and from a pod, then drop" so that it doesn't affect non-kubernetes traffic, but I think the rule we actually need is "if INVALID and to a service, then drop". In particular, I believe we need to make sure that the rule applies to traffic with a cluster-external source and an in-cluster destination. It's possible that changing it to "if INVALID and to a pod, then drop" would also work. I skimmed and some linked bugs elsewhere but I was unable to confirm whether that rule would cover the reporters' use cases.
I think that is only INVALID and to a pod
In my case, cluster-external source to service is both ok with the INVALID rule or without the INVALID rule. It seems the rule doesn't affect on access service.