Hello, we have been using client-go to measure the latency for all client-go traffic from Cilium to kube-apiserver. We have that since v1.24.0 the URL path is no longer aggregated in a way that this cardinality would no longer happen as presented in the following image: <img width="330" alt="Screen Shot 2022-08-08 at 10 18 24 AM" src="https://user-"It seems this functionality was removed as a part of as it seems that PR only took into account that users of the would only be using it to measure latency to a particular host and not for a particular URL path as we are doing in . Would it be reasonable to revert commit in order to fix this regression? Thanks cc
/sig api-machinery /sig instrumentation
But reverting will bring back the cardinality problem, will not? The latency of each URL path is available in the apiserver metrics ...
We have never any cardinality problem with the previous code since we were not using the "host" field which is what I assume the original issue about cardinality was trying to fix. We only used the "path" of an URL, which was aggregated in a way that would not be facing any cardinality, like the figure attached in this GH description. Now, starting with = 1.24.0, each and is actually a different value which does cause the cardinality problem to appear.
ah, I see the disconnection, I wasn't understanding the issue sorry. The Interface uses the url as parameter to keep backwards compatibility, but the method that gathers the metrics only uses the host to avoid the cardinality problem. You have to update the method that gathers the metrics to only use the host part of the url
Filed
should fix the code in kubernetes ? however, that will break consumer of the Interface that still wants to process the path, but we know using the path will cause issues with cardinality :shrug:
I would be fine with adding a new label to mitigate that issue since originally the cardinality explosion that was reported was caused by the query string and not the path: When trimming down the whole URL to only the host we even had a discussion to keep the path, but we decided not to since we didn't have any use case for this data. If you have a use case where you need the path in order to get more granularity on the latency of your network then I would be fine with adding the label to Kubernetes, but another path forward could be to also remove the path information from cilium if deemed unnecessary. In the most common scenarios, you will only need the host to get the network latency based on different proxy routes.
yeah we can't really do much about it, it is either we leak the unsanitized paths or we prevent the users from using the path completely. Since we want to not break the interface and keep the URL parameter, we might just have to keep sanitizing the paths for our consumers even though we are not using the path anymore.
IIUIC the RESTClient is used in the dynamic client-go too, for these users with 1900 CRDs, the path is really going to explode for them
Then it seems that the best option to not break users would be to revert , but at the same time I don't really like the idea of spending CPU time sanitizing something that will never be used in the RESTClient
A suggestion would be to create a new interface similar to the that would have the following method: Thus, those users with 1900 CRDs would use the instead of the .
after digging a little bit the underlying issue it looks that the problem was initially caused by a kubelet bug as pointed out . The dependents of client-go don't have access to all the fields of the struct which uses heavily to generate the aggregated URL.
well, it is clear now that current status is wrong, we pass the whole URL to the interface and rely on the to use only the Host part of the URL :smile: I'm fine with whatever you decide, to restore or just completely strip the Path Sorry for the unfortunate consequence, at that time I didn't consider that the Interface/Method drift and thought that function was orphan
/triage accepted /assign


Add additional logging at level that associates UID with the metadata from the request. xref: /sig auth
/triage accepted
/milestone v1.26
Mind if I take a look at this? /assign
Hello I am wondering how's this coming along. Is there anything I can help with?
Sorry. I got a bit occupied with a different project.. Will have a PR opened for this for review tomorrow.. I am trying to find a way to inject the Verbs into log. But haven't found the right way to include that into the context.. Any suggestion on what can be done?
any suggestion on what is the best way to get the additional metadata for verbs into this log? Move that info across using ?
Hello , bug triage team here, just wanted to touch base with you for updates on this ticket. Thanks!
Thanks! This is being worked on and is still on track for .
There is a Pull request opened on this already as well. But Waiting on some clarification to see if that needs additional changes.
/reopen /assign /unassign
Reopened this issue. <detailsIn response to : /reopen /assign /unassign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/milestone v1.27
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
Hi Bug Triage Team for v1.27 here o/ Just wanted to touch base with you for updates on this ticket. Thanks!
I am not working on this anymore Carlos. This was re-assigned
This is tracked for . I've linked the PR.


4 Oct 2022 [x] Two weeks soak end date: 18 Oct 2022 [x] Test promotion PR: According to this APIsnoop query, there are still three remaining LimitRange endpoints which are untested. listCoreV1LimitRangeForAllNamespacespatchCoreV1NamespacedLimitRangedeleteCoreV1CollectionNamespacedLimitRange Using a number of existing e2e test practices a new has been created for three LimitRange endpoints. The e2e logs for this test are listed below. This query shows which LimitRange endpoints are hit within a short period of running this e2e test example endpoint | useragent --------------------------------------------+
createCoreV1NamespacedLimitRange | should ensure that a limitRange can be listed, patched and deleted by collection deleteCoreV1CollectionNamespacedLimitRange | should ensure that a limitRange can be listed, patched and deleted by collection listCoreV1LimitRangeForAllNamespaces | should ensure that a limitRange can be listed, patched and deleted by collection listCoreV1NamespacedLimitRange | should ensure that a limitRange can be listed, patched and deleted by collection patchCoreV1NamespacedLimitRange | should ensure that a limitRange can be listed, patched and deleted by collection (5 rows) If a test with these calls gets merged, test coverage will go up by 3 point This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/triage accepted
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


/kind failing-test <!-- If this is a flake, please add: /kind flake --/kind flake <!-- Please assign a SIG using: /sig SIG-NAME --/sig testing apps
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
can you PTAL, it seems this test was in It doesn't seem to have a high rate of failure , but seems flakes are legit
/assign
side note: it's really hard to make it fail in a fast machine :) I had to temporarily add a sleep in the reconciler.
have you tried Jordan's instructions compiling and running with ? for integration you should have an external etcd instance
I haven't. Good to know for next time :)


There is a regression in v1.24.0 (and above) that causes stale HNS load balancer proxy rules anytime a backend pod is deleted. Each subsequent deletion will leave behind an additional external VIP load balancing rule that references endpoints which no longer exist. This can cause occasional connectivity issues and timeouts, if a stale load balancing rule is matched and it redirects traffic to an endpoint which no longer exists. We are under the impression this may have been introduced (creds to for identifying this.) I expect all load balancing rules created by winkernel proxier (aka HNS load balancer) to be referencing valid backends (aka HNS endpoints) a Kubernetes cluster v1.24 or higher with Windows nodes. a Kubernetes Service with type "LoadBalancer" on Windows. one or more pods of the Service. stale load balancing rule left behind referencing endpoint that was already deleted. This issue can be discovered by establishing multiple connections to the service, after pods were deleted. Some of the requests will fail. This issue can be monitored using the following script: Execute as follows: It will print something along the lines of: Another way to inspect this manually is using the following PowerShell: In the $refs, you would see multiple entries (indicating there are duplicate HNS load balancer proxy rules with the same VIP) and that some of the references are invalid (ie not showing up in the get-hnsendpoint output) can force all pods of a given service onto a single Node. This will still cause duplicated load balancer entries, but the connections should all succeed as the right rule should be matched. K8s version <v1.24.0 . This will remove all the rules and re-create them. 24.0 and above Azure Kubernetes Service, but likely impacting others. Windows Server 2019, Windows Server 2022 n.A. n.A. n.A.
/sig windows
/triage accepted
/sig network
/milestone v1.26 /priority critical-urgent
/reopen Let's keep this open until backport PRs merge and the fix is available in v1.24 and v1.25 builds.
Reopened this issue. <detailsIn response to : /reopen Let's keep this open until backport PRs merge and the fix is available in v1.24 and v1.25 builds. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
backports are merged and patches were released /close
Closing this issue. <detailsIn response to : backports are merged and patches were released /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Occasionally job completions are counted twice (or potentially multiple times?) in the metric. This happens for about 20% of jobs, but may depend on the Job specificity, which results in an overestimation of the number of finished jobs. The metric reflects the actual number of completed jobs. a kind cluster with prometheus monitoring stack (for example follow this: ). the number of completed jobs by inspecting the metric and delete a job multiple times in a loop Example job: Example repro script: . Check the number of completed jobs by inspecting the metric and the job Yes, in the kube-controller-logs we find the following log lines which are related and probably indicate repeated updated which results in double counting. <details</details<detailsReproducible with kind </details<details</details<details</details<details</details<details</details
/assign
FYI
/sig apps
/triage accepted


As per, , wire up metrics/slis for kube-proxy
/assign
/sig network /sig instrumentation
This sounds OK, but we need to translate it into real work-to-do - volunteers? /triage accepted
is tackling this one :)


As per, , wire up metrics/slis for kubelet
/assign
/sig instrumentation
/sig node /triage accepted


The code at: assumes that the transformation does not change the size of the data. Since we know it most cases that it does, we should try to account for that during the allocation.
/sig auth
/triage accepted
/assign
I think there is no problem with the memory allocation of , because needs to be executed afterwards, in order to avoid executing new memory allocation during append, so is no problem. IMO
After reading your code, I understand.
/assign


I started missing error checks. When adding error checking to this call here, the test started to fail: The test should have continued to work when checking the error. No response <detailsmaster </details<details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig apps
/priority important-soon can you or someone in apps PTAL?
The Kubernetes project currently lacks enough contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
/help
This request has been marked as needing help from a contributor. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/assign


pull-kubernetes-unit /cc /sig scalability
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
In local tests, he always passes, so I'll see if the problem is caused by the environment
If the lru cache exceeds 30 seconds, the cache will be invalidated, which will lead to unit test errors, and the problem will run normally again after retesting.
After I manually reduce an unhold <- struct{}{}, the same problem occurs, so it seems that the problem is caused by the cache being invalidated for more than 30 seconds. If the test doesn't complete in 30 seconds, then this is probably a machine performance issue
stress will hang after about a minute in my local test.
/cc
The issue is being fixed


The of a stale pod disruption condition remains after two minutes as the disruption controller fails to change it to to disable the condition. The of a stale pod disruption condition should be changed to after two minutes. with Alpha feature gate enabled is without associated delete operation (it may happen if preemption delete call fails, but it can be simulated by API) 2min as after this time the condition is recognized as stale by disruption controller, as the pod is not terminating The patch of disruption controller fails with the following message in the logs: <details25.x <!-- --</details<detailsReproducible on KIND. </details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig apps /assign


Currently we have inconsistency in most of the API where the field name should be used in description so the generated docs match the API fields people actually set. A linter could be for checking To avoid confusing users
/good-first-issue
/triage accepted
/assign
Hey can you provide some more information for this issue
are you still working on this?
Hey feel free to take it. I am unassigning myself
/triage needs-information /remove-good-first-issue
/sig api-machinery


The node selectors gets parsed by in for each node. That work can be done once in . Better performance.
/sig node /sig scheduling /triage accepted /priority backlog
I can try this one. /assign


During kubelet start up if the system reserved or kube reserved values failed to parse, the kubelet won't start up. The error logged looks like, However, it doesn't not inform the user which system reserved or kube reserved quantity has failed and what was the corresponding value of that quantity. The error log should also container quantity type and the value that failed to parse, so that user can fix them easily. Try to start kubelet with unparsable values of system reserved, something like, The error logged would look like, As you can see above, it doesn't say was not able to parse with the value No response <details</details<details</details<details</details<details</details<details</details<details</details
/kind feature /sig node
/assign
Thanks for Should I close in favour of yours?
Oh sorry, I didn't see your PR. Either way works for me. You can either incorporate ideas from my PR and I'll drop mine, or drop your PR. I don't feel strongly either way.
/triage accepted /priority backlog /assign
Let's go with yours!
Thanks /close (in favor of )
Closing this issue. <detailsIn response to : Thanks (in favor of ) Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


When submitting a windows pod using resource limits on a node with more than one processor group, the specified count is incorrect due to the usage of which does not take processor groups into account. For example: node with 96 cores cpu limit 94 returns This results in a CPU maximum of since the range is from [1, ]: Our initial goal for this was to limit a process to not use all available CPU and therefore render the node uncontrollable. A CPU maximum of instead of for that specific case. For the calculation, it should use the CPU count for all available processor groups instead of just the current one. Launch a node with a lot of CPUs, for example and schedule the following pod on it: Then SSH into the node and inspect the running container using . The value will be set to . (which just uses ) is able to correctly find out the number of CPUs available to a system. Using for calculation or incorporating that code might solve this issue. <details</details<detailsAzure in-tree </details<details</details<details</details<detailscontainerd://1.6.10 </details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig windows
There is already some code which uses GetActiveProcessorCount:
/assign
/triage accepted


an E2E test that assumes that a service named "kube-dns" exists in the "kube-system" namespace. This is a kubeadm-ism, and the test fails if the cluster DNS is not configured the way kubeadm configures it. Although this E2E test isn't a conformance test, making it functional on non-kubeadm clusters would increase its utility for distributors that may wish to run this E2E test as a conformance test. [sig-network] DNS HostNetwork should resolve DNS of partial qualified names for services on hostNetwork pods with dnsPolicy: ClusterFirstWithHostNet Since we ran it on an OpenShift cluster. No response The test makes assumptions about how the cluster DNS is configured. The test uses (not ), so it does not need to specify , as far as I can tell. Omitting obviates the need to look up the service. I believe that the service lookup was left in as an oversight during the development of the test when it was changed from specifying "None" to specifying "ClusterFirst": Cc: /sig network
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
responded on the pr.
It seems kubeadm inherited the status-quo, I can't find any place that mandates the Service name for DNS Asking for double checking, but the docs seems to mention that this is more a recommendation than a preescription If it seems that the test can be done in a more portable way we should do it


In , we discovered that our plan for eventually removing the scheduling plugin when CSI migration is GA is not possible because the plugin also applies to non-migratable volume types. However, if we leave the plugin as is, then even migratable PVs will be broken if we remove beta topology labels from Nodes in the future. This is because the plugin does not have any handling for migrated volumes. I suggest we add CSI migration handling to the VolumeZone plugin. I think we just need to detect if the volume is migrated, and if so, skip the VolumeZone processing. For migrated volumes, we translate into so they end up getting processed by the plugin. In a cluster with csi migration for a specific intree volume type enabled, an intree PV with beta topology labels the beta topology label from a node to schedule a pod using that pv No response <details</details<detailsAny that supports csi migration </details<details</details<details</details<details</details<details</details
/sig storage /triage accepted /priority important-soon For e2e testing, this needs an environment that is configured with csi migration
Hi,I'd like to work on this issue. I previously wrote for plugin, so I'm familiar with it's logic. However I'm not familiar with the whole thing, and I may ask you some questions during the work :( Can I try this one? Here are some questions ahead: I think what you want to do is to schedule a pv with old labels to a node with only new labels. And you think we can translate into in volumezone during (maybe), and update this pv so that in stage, can , so we don't have compare Topology labels in . My question is: Can't we just translate this for all pvs? Why do we need to detect if the volume is migrated? What's the difference between a migrated pv and a non-migratable pv in this scenario?
I think this is also another alternative solution that could also work, but from , it may not scalable to do an automatic migration for all the different APIs and controllers that use the node labels. The main difference with CSI migration, is we have a good idea of all the different components that would be impacted and also have reasonable control over those components.
A few more questions Will the volumes be migrated before we take a pod off the scheduler queue? If so, how can I detect this. Currently I find a function in , but nothing like . Is this translation done in plugin or before scheduling? I think currently in , we consider topology label and beta label as different labels. That is ,if we have a pv with and a node with , even if they have same values, will let the pod pass filter. If we're going to remove beta labels on node,maybe we should try to consider two labels as the same in ? Here's my understanding: For migrated volumes, we translate Topology labels to pv.NodeAffinity and update the pv, then we skip volume_zone(that is,don't compare labels with nodes). What about non-migratable volumes, I think we should continue to compare with node labels for non-migratable volumes?
For detecting if a PV is migrated, you can use the same method that's used in the VolumeBinding plugin: The only difference is you don't actually need to translate the PV in the VolumeZone plugin since it's already handled by the VolumeBinding plugin. For non-migratable volumes, I think we can continue with the same VolumeZone logic. For now, we'll rely on users to update their PVs with the GA labels.
Thanks ,for your detailed explanation. This is the implementation plan I prepared: In , we check if a pv is migrtable by using ,if so we skip volumezone since will translate this pv and handle it later, otherwise we save the label of this pv in state,and compare it with node's label in .Is that feasible to you?
Yup that sounds good to me!
Thanks! I'll try to work on this one. I'll file a pr a few days later and we can talk about the details in there. In the meantime, I think I previously wrote a bug in plugin, I wonder if you have time to confirm it /assign
This issue is labeled with but has not been updated in over 90 days, and should be re-triaged. Important-soon issues must be staffed and worked on either currently, or very soon, ideally in time for the next release. You can: Confirm that this issue is still relevant with (org members only) Deprioritize it with or Close this issue with For more details on the triage process, see /remove-triage accepted
/triage accepted


In a cluster with large enough numbers of objects, it seems to be impossible to clean them up with . Because the deletecollection is failing while listing the objects from etcd. The failure happens after exactly 34 seconds, due to the 34-second max timeout introduced in PR (cc for insight on that PR). The deletecollection should be able to run for longer, and succeed. I recognise that there are still risks of the API server using too much RAM, if deleting a very large collection due to issue . But, until that issue is fixed, it seems like allowing to run for longer might be a good idea. As far as I can tell, the 34 second timeout does not apply to normal LIST calls, so on that basis I feel it should not apply to the giant unchunked LIST call that happens inside . a few hundred thousand objects of some particular type. (Argo workflowtaskresults happen to be the type I'm looking at while I write this) deletecollection. (Optionally passing some kind of fieldSelector, but I don't think that's essential) that a 504 failure code is logged in kube-audit logs, and that a log entry similar to this is logged by the API Server (In practice, the total time may be a little over ms). far as I can tell, the delete collection really has failed at this point. I notice that PR says that, in general, "The go-routine can keep running, while the timeout logic will return a timeout to the client" but I don't think that's what's happening here. I think that, because the 34 sec timeout is used in the context of the etcd call, the etcd list call really does fail, and so the code never proceeds to actually deleting anything. No response 22 AKS <details</details<details</details<details</details<details</details
/sig scalability
/cc
the deadline has been in effect before PR was introduced If we remove this deadline from here, then the deletecollection request will have at most to complete (default). The list call that is failing in your case, is a direct call to the storage, right? on the contrary, we don't enforce the timeout for LIST thinking out loud here: I don't know if it makes sense to remove the from should we allow timeout for DELETECOLLECTION to exceed ? thoughts?
The list call that is failing in your case, is a direct call to the storage, right? It reads from etcd, not from API Server Cache. Is that what you were asking Looking at the bigger picture, I think your question about 60s timeout is a good one. Especially in the context of this: In particular, changing the timeouts won't solve all of the problems with DELETECOLLECTION. My assumption, when logging this issue, is that it might be a quick way to solve just some of them.
I think we should remove this - it's effectively a regression that we introduced with the PR. That's a good question, and I'm not entirely sure. Maybe we should, but I would do that separately. In the meantime, I would much more prefer to switch the DeleteCollection implementation to internally use pagination, as it would allow us to 1) not block on full list (and as a result make some progress) 2) would be some preparation towards potentially exposing a Limit option in the API... /kind regression
/sig api-machinery
/triage accepted
I suggest the following: - can you please take care of reverting the 34s timeout for DeleteCollection calls? this is done, let's close this issue and continue operating on
sounds good, thanks! /assign


handles cases when csiNode is nil. This is condition is used by cluster-autoscaler internal working. However, the log line refers to causing seg fault
/sig scheduling /sig storage
/sig storage /priority important-soon /triage accepted
/sig storage /sig scheduling Note that the lister returns an error for the case when the doesn't exist. But it's ignored here: It looks like the log line that causes the panic was introduced in in 1.25+
cc
/assign
As far as I can tell this issue is only fixed in version 1.27 (by ). Is there a plan to port it back to releases 1.25 and 1.26 as well?
it has already been picked to 1.25 and 1.26 () and will be in the next patch releases for those versions
Great. Thank you for the pointer.


[X] Test approval issue: [X] Test PR: [x] Two weeks soak end date: [X] Test promotion PR: According to this APIsnoop query, there are still some remaining APIService endpoints which are untested. Using a number of existing e2e test practices the has to cover the outstanding APIService endpoints. There has been some around how the current Conformance test is using RBAC, which is optional and not part of Conformance. Those calls will now only happen if is true. The e2e logs for this test are listed below. This query shows the following APIService endpoints are hit within a short period of running this e2e test example endpoint | useragent ---------------------------------------------+
createApiregistrationV1APIService | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] deleteApiregistrationV1APIService | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] deleteApiregistrationV1CollectionAPIService | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] listApiregistrationV1APIService | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] patchApiregistrationV1APIService | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] patchApiregistrationV1APIServiceStatus | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] readApiregistrationV1APIService | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] readApiregistrationV1APIServiceStatus | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] replaceApiregistrationV1APIService | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] replaceApiregistrationV1APIServiceStatus | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] (10 rows) If a test with these calls gets merged, test coverage will go up by 4 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/triage accepted


After the cluster was updated to enable PD CSI Driver, there was a race condition where AD Controller (kube-controller-manager) started to BEFORE the PD CSI Driver was up, thus AD Controller gave us the error Then AD Controller triggered the disk detach because the disk/volume was not populated in the desired state of world. Unfortunately the disk was in use during this detach and got corrupted. Since cluster upgrades can cause the controller manager and CSI drivers to start at different times, the AD controller needs to be more conservative when detaching We have not been able to reproduce this, it came up from a customer. /sig storage /assign <detailsk8s 1.23, CSI migration enabled </details<detailsGKE </details<details</details<details</details<details</details<details</details
/triage accepted
I think there's potentially two changes we should do: 1) for the initial DSW population, we should skip the check that if the csi driver is not installed. 2) in general, we should not skip volumes during dsw initial population if an error is returned. If we had some temporary api server error then that would cause volumes to not get to dsw and similar behavior happen. cc
for a repro, maybe you can try something like: a cluster with csi migration enabled (control plane + nodes) a pod with intree-volume and get it running the csi driver node daemonset. verify the driver is removed from CSINode kube-controller-manager
cc
/cc was trying to repro this, but on GKE with the managed driver which made it complicated.
Yeah maybe gce kube-up would be easier for a repro
If CSINode says that it has migrated GCP PD volume plugin to CSI, then IMO A/D controller should always attach GCP PD volumes to it using CSI, regardless if the CSI driver is present or not. In-tree volume plugin in the kubelet is disabled and can't mount any in-tree volumes anyway. To flip the migration feature gate / upgrade / downgrade kubelet, the cluster admin must drain the node first to make sure everything is detached. A/D controller should not see change of CSINode migration annotations with any volume attached to it.
/priority important-soon
Another option could be to process these volumes anyways but add them to DSOW with migration status as "uncertain", so as they can be re-processed later. But surely it looks like might be easiest unless it has any downsides.
There wasn't a node update in this case---there was a control plane update which restarted the PD CSI driver, and it looks like the CSIDriver resource was updated or changed in some way which raced with the kube-controller-manager restart.
Is there anyway we can know if the csi driver is up and running(like a health check ping) before populating DSW?
We use the CSINode registration to determine that. The discussion here I think is concluding that we should not use that as a criteria for adding the volume to DSW. We only need to check that CSI migration is enabled on the node.
+1 we need to be very resilient against races, I think we have to assume a driver is in process of being installed at any point in the AD controller reconcilliation.
/assign sunnylovestiramisu
Reproduced with the steps: In /var/log/kube-controller-
Tested with the and doing the reproduce steps above. The error of does not show up anymore. The error message still get logged in the kube-controller- for debugging purpose: Ran Ran On the GCP console, the disk is used by node: Ran I redeploy the pd-csi-driver and then make sure it is up and running. Ran


At (originally ) component-base metrics exposed Registerer API for raw prometheus registration, and I took dependency in The resulting does not seem to emit the new metrics (example: ). The metrics handler should emit the registered metrics. Falling back to does work. So there is some material difference with . 24, 1.25, 1.26 GKE No response No response No response No response
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/assign jkh52


I am tracking a changed behavior in networking with kubernetes 1.26. If I create a service of type externalTrafficPolicy: Local, and there are no local endpoints in a node, kube-proxy does not create hns loadbalancers for the clusterIP of that service. With kubernetes 1.25, it creates a loadbalancer for clusterIP (but not externalIP). This is the relevant log message: Looks like this comes with a result from a new function isAllEndpointsTerminating introduced in kubernetes 1.26. In Linux, we can still access clusterIP even if there are no local endpoints for the service. Thoughts? The end result is that in Windows, clients cannot access the clusterIP of such a service with there are no local endpoints.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig windows
cc
Does this issue occur when there are no local endpoints for a given service? If there is a local endpoint, then you can reach the service again, right?
, Also let us know, do you have any InternalTrafficPolicy configured ?
Does this issue occur when there are no local endpoints for a given service? If there is a local endpoint, then you can reach the service again, right? That is correct , Also let us know, do you have any InternalTrafficPolicy configured ? We do not have that configured


I am a new guy to kubernetes world, and today I am looking at the source code k8s, I found out a line of code can optimize in this line, `, cus the logic here should be like impl the startController logic and then we put it into the register to find out if the controller was registered twice, if so the program will panic. I think it's better that we change to the one that I provide, which can make code clean as well, but as I'm a new guy, I need some advice on it, can someone help me out? /help wanted /area controller-manager
/sig api-machinery
/sig node Would you submit a PR to fix it if you think this is a problem? /cc as the owner of DynamicResourceAllocation was as alpha level feature in v1.26.
you're say me or the pohly? this is my first time contributing on open source, might have some issues on communicate
You. Feel free to submit PR if this is an optimization. I as he is the feature owner.
thanks, I will submit PR! so the following step should be: me fork the k8s repo to my github and git pull to my local, optimize code, then git push and submit PR, right? the editing should be written on branch master
/triage accepted


Serial e2e node jobs, e.g. Since 03-01 No response Seems related to No response /sig node
Failing run -
I think there is a typo there - should fix it
cc
/triage accepted
/cc


The test is flaking: /assign /sig api-machinery /priority important-soon
Ok - it's fairly easy to reproduce by just adding into the test I will validate the fix.
/triage accepted


During OPNFV test execution we noticed the folder selinux context changed. e2e test log How the folder changed after the test executed. I expected the TC won't change the selinux context. The question here, should this sigstorageserial TC set work on an selinux enabled K8S cluster or not? This question valid for the another sigstorage TC set with several test case, where selinux block a lot of TC too. See above. No response_ <details</details<detailsOpenStack </details<details</details<details</details<details</details<details</details
/sig testing
/sig storage
The Kubernetes project currently lacks enough contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
/assign
Reproducer with simple kubernetes/kubernetes artifacts:
/triage accepted


Hello, Upon visiting the . There is a link for interactive tutorial which is being inactive. Error from (k8 page)[] is ! We have to remove the page link.
label:sig/docs
/sig docs /kind documentation
Per let's delete this reference for now and we can add something later potentially following
/triage accepted /assign


In , when timeout or other error occurs, is called. This will make current routine terminated and miss the , which will lead to some leak routines. For example, this routine will block at waiting for No leak make any of these errors happen No response <details</details<details</details<details</details<details</details<details</details<details</details
sig Testing
I would like to take this up!
Actually, I have a fix in , but its fine if you have a better solution.
/sig api-machinery
/assign /triage accepted


Data as of June 3rd, 9:00 PM Eastern. Latest data can be found here: Prow link to the 2 jobs: * Additional Context:
/sig storage cc
I'm working on , it's hard to get a job yaml right at the first time. I'll check iscsi. /assign
/triage accepted /kind failing-test
ci-kubernetes-e2e-storage-kind-disruptive fixed: has 1 green run pull-kubernetes-e2e-storage-kind-disruptive fixed: , it fails when it should fail, it's green otherwise, I'll continue with iscsi.
/reopen it's still red, but at least it installs now:
Reopened this issue. <detailsIn response to : /reopen it's still red, but at least it installs now: Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
e2e-gce-iscsi-serial job passes now: e2e-gce-iscsi is getting greener: I'll let it run for few more days to see if today's failures are flakes. /reopen
Reopened this issue. <detailsIn response to : It's getting greener: /reopen </details
iscsi-serial is green:


When a ResourceClaim: uses delayed allocation does not get created for a specific pod (i.e. not from a template) then it currently gets allocated and remains allocated until: scheduling of a pod fails because the node originally chosen for the allocation isn't suitable, in which case the scheduler triggers deallocation via the user deletes the claim What we want instead is that deallocation gets triggered when the last user of such an allocated claim gets removed from . To avoid races, deallocation must be done through . This can be implemented either as a controller or directly in the API server.
/assign
/triage accepted /sig node /priority important-longterm /kind feature


We ran into an issue with an old version of k8s 1.18 that we were running where subnet for cluster IP is given as . The control plane comes up just fine but then when you try to create a k8s service, it fails due to a broken behavior with the IP Allocator Since this is not really a supported version of the k8s, I went around digging up a bit more to see what is the regular behavior and found this to be very inconsistent across a few release branches. bash go test -timeout 30m -run ^TestAllocateNextFC$ -v === RUN TestAllocateNextFC ? --- PASS: TestAllocateNextFC (0.00s) PASS ok 0.566s go func TestAllocateNextFC(t *testing.T) { , cidr, err := netutils.ParseCIDRSloppy("fc::/112") if err != nil { t.Fatal(err) } // Need to be replaced with the suitable handler for that version of the branch r, err := newTestAllocator(cidr) if err != nil { t.Fatal(err) } defer r.Destroy() ip, err := r.AllocateNext() if err != nil { t.Fatalf("%s", err.Error()) } fmt.Println(ip.String()) } Add the above Test case under the IP allocator test case No response_ <detailsNA </details<detailsNA </details<detailsNA </details<detailsNA </details<detailsNA </details<detailsNA </details
/sig network /assign /kind bug
Thanks for helping me with this issue.
/triage accepted /assign


During implementation of KEP 3393 (PodReplacementPolicy for Jobs) we found that the unit tests of do not follow common patterns established in k/k. 1) The tests are not written using table driven design (). 2) The tests use rather than . This probably should be done as two separate PRs but wanted to highlight some areas that could use some cleanup. Not using means that we don't see diffs in the unit tests that makes this difficult. And not using table driven tests makes it difficult to add new test cases as it usually is brought up during review that we should use a table-driven test but the entire file does not obey this format.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig apps /kind cleanup /help /good-first-issue
This request has been marked as suitable for new contributors. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /sig apps /kind cleanup /help /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I'd like to work on it /assign
I'd also like to make some contributions. /assign
Looks like this need two PRs. tests are not written using table driven design (). tests use Differences rather than Cmp.Diff. I think I can work on the 2nd one and leave the 1st one to you. Sounds good?
By looking through the , I found the main function to comapre the results have already used cmp.Diff. Can you elaborate more on which part you expect to use cmp.Diff? func verifyDeletedAndPatchedPods(t testing.T, client fake.Clientset, wantDeletedPodNames, wantPatchedPodNames sets.String) { t.Helper() deletedPodNames := getDeletedPodNames(client) if diff := cmp.Diff(wantDeletedPodNames, deletedPodNames); diff != "" { t.Errorf("Deleted pod names (-want,+got):n%s", diff) } patchedPodNames := getPatchedPodNames(client) if diff := cmp.Diff(wantPatchedPodNames, patchedPodNames); diff != "" { t.Errorf("Patched pod names (-want,+got):n%s", diff) } }
I am really sorry about this. Too much multitasking!! I meant not
Hey ! Sorry, I didn't see this before. No problem at all, but consider that one change depends on the other, and since my task implies a big refactor, there might be conflicts when merging your PR... That's why I see two possible scenarios happening: Your PR gets merged before I raise mine: Then fine, I'll rebase your changes and adapt my code Mine is ready before yours is merged: In this case, I propose you commit your changes to my PR. Both of us will appear as coauthors of the change Getting a PR merged in the Kubernetes project takes quite some time (trust me, I talk by experience and can confirm ) WDYT?
Let's see how it goes. We can always rebase the code change to adapt new.
btw, did you already begin to work on task 1? If not, I may just go ahead and work on it right away since I have already dig into it.
yes, I already started And If you ask me, better to keep it this way so we both have an opportunity to have a contribution
cool. Sounds good.
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hi if this issue needs to be worked on any further, please assign me for the same.
has a PR up that will solve the second part of this issue.


When I put a Pod yaml file without in , started the kubelet and hoped it would create the corresponding staticPod for me, I got the following error, which also caused the staticPod to not be created. I then read the kubelet code and quickly found the problem. When kubelet tries to decode the pod yaml file, it will apply default values and validate the pod. The function to apply the defaults is in . In this function, the is set. This causes an error when is not set, because it causes the '-' character to be at the front. I think kubelet should circumvent this error, because is optional, It is possible for this field not to be set. When Pod's is not set, the '-' character at the start should be avoided, as it will cause an error. Prepare a Pod yaml file without and set it into . If the maintainers think this circumvention change is necessary, I'm happy to contribute code to the kubelet! <details</details<details</details<detailsconsole $ cat /etc/os-release NAME="Alibaba Cloud Linux" VERSION="3 (Soaring Falcon)" ID="alinux" IDLIKE="rhel fedora centos anolis" VERSIONID="3" PLATFORMID="platform:al8" PRETTYNAME="Alibaba Cloud Linux 3 (Soaring Falcon)" ANSICOLOR="0;31" HOMEURL="" $ uname -a Linux cloud 5.10.134-14.al8.x8664 SMP Thu Apr 27 16:46:29 CST 2023 x8664 x8664 x8664 GNU/Linux </details<detailsusing kubeadm </details<details</details<details</details
/sig node
I would like to open a PR to fix this issue, can I be assigned?
If the maintainer thinks this should be fixed, I want it fixed by myself:)
OK
I do not know if a pod without name should be considered valid Please discuss with sig-node before doing anything to understand what is a valid solution for this problem /cc
names are needed for the Pod. I think the correct fix for this issue will be to add validation for static Pods to have a name. This validation is OK to add since Pods without names are not working anyways because of this bug.
Okay, I will submit a PR to add validation for static Pods to have a name.
Hey, I found out that kubelet will apply default value first and then validate the pod. Do you mean to let me add a single validation for static Pods to have a name before applying default value?
hi and If modified according to this logic, it seems that the current code is fine. Current logic: it will use the old pod.Name to generate the new pod.Name , in ValidatePodCreate, it will validate pod name. The pod.Name is already to new pod.Name this case, the original pod does not have a name and will also be recognized. It's just that it's identified with a validation prefixed with "-". The logic of the current validation will recognize the pod without a name. In this way, is it necessary to improve the error information?
/triage accepted
/priority backlog


What happened: Shortly after was merged, the "multiple nodes reallocation works" E2E DRA test started to fail randomly. Please provide links to example occurrences, if any: /sig node /kind bug needs to stay green. No response master n/a No response No response No response No response
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Kubernetes does not respect the of a probe. I found this while fixing the e2e tests. Test result: /kind regression /kind bug /sig node /priority important-soon The of a container's probe should override the pod's . a pod with long pod-level terminationGracePeriodSeconds and short probe-level terminationGracePeriodSeconds. for the container not respecting the probe-level terminationGracePeriodSeconds No response <details</details<details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I guess the probe's terminationGracePeriodSeconds field is dropped for some reason.
/assign


A client is allowed to add custom finalizers to Custom Resources with a finalizer name that's not fully qualified (e.g. or ). However they're prevented from doing the same on builtin resource types, returning the following error: Invalid value: "": name is neither a standard finalizer name nor is it fully qualified Per , custom finalizer names are expected to be in format "consist of a domain name, a forward slash and the name of the finalizer" (e.g. ). Therefore, I expected the enforcement to be the same for CRs and builtin types. a custom resource on the cluster it to add the edit does not fail the same on a builtin resource (e.g. Namespace or Pod), observe it fails No response Reproducible at least starting from 1.21 and still on latest versions. <details</details<detailsN/A kind cluster </details<detailsN/A - kind cluster accessed via macOS </details<details</details<details</details<details</details
/area api /cc /sig api-machinery
/assign /triage accepted
There is no validation of finalizer in and . Please unassign me if you guys have started to work on it. /assign
it is called from here on custom resources:
lacks validation for finalizer name. I have raised a pr for this. PTAL.


/kind failing-test <!-- If this is a flake, please add: /kind flake --/sig network seems to be related.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/triage accepted


When PreFilter returns UnschedulableAndUnresolvable, we don't need to run the preemption, but currently we run. This unexpected preemption could cause errors from the preemption by trying to read non-exist PreFilter data. As pointed out (), actually we didn't run the preemption in such cases in the past versions, but introduced this bug. /kind bug /kind regression /sig scheduling /triage accepted /priority important-soon
/assign
will fix this.


introduced configuration of additional logging aspects (e.g. format) through command line flags and config file. This was merged for Kubernetes 1.28. Now the following invocation doesn't work as expected when the config file doesn't configure logging: It should run with verbosity 4. Instead it runs with verbosity 0, from the config. Run with this config file: No response 28 rc n/a No response No response No response No response
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The problem is caused by this line: At that point, is 4 (from the command line flag), and then it gets overwritten with == 0. The normal approach would be: set up config with defaults modify config through flags modify config further by parsing a config file into it. Can kube-proxy do this? It has the downside that a config file can only overwrite fields if they are in the file. A config field doesn't get reset to defaults just by parsing the file. /cc
it looks like kube-proxy special-cases the to let a flag override config... can/should it do the same thing for verbosity? are there any other flags like this?
I don't think that's the normal approach... it's more normal to make flags and config file mutually exclusive. That's difficult for things which started as flags and moved into config
/sig network /sig instrumentation /kind bug /kind regression /milestone v1.28
let's go with and warn it and comment as much as possible, I was going with something like but Patrick is working on it so I assign to him the issue
/assign
Hey folks, v1.28 Bug Triage Lead here Just wanted to know if this is already tracked by SIG leads and someone already working actively fixing it and set a priority?
Perhaps I should have said "this is how component-base/logs was meant to be used". Somehow it worked for kubelet. Here's my preliminary solution: It needs more testing, but I wanted to share it early because I am running out of time today.
it is , I can take over Patrick's work in case is needed
It would be nice if the logs API could distinguish between and , how about make to a pointer? But this change is supposed to be a breaking change, I'm not sure of the scope of the impact. For side, if set and at the same time, should the command flag override the config file or warn it and be discarded? I think it should be overridden.
xref:


Drop the and dependencies from to keep the import lighter. /sig auth /triage accepted
/assign Hi, , I created a PR to fix this. I am not sure If I am correct due to the repo seems generated. Would you like to provides some hints for me?
This will likely require a new go module in to move the client-go dep and then mark it as internal, ex: . is the only API machinery dep so we can probably come up with some way to re-implement the wait logic locally? Then we drop these lines so that these imports do not come back again:
Whoa, yeah, I'd take a for loop and a time.Sleep in a test to drop those deps
What is kms using client-go for? Tests only?
These were all needed by the KMS repo back when we had a much more complicated reference implementation which required a key hierarchy. Now they are just used in some tests and mock implementations, so we should be able to simplify things.
I found a place for rate limit. #liggitt
that package just delegates to "", switch that mock provider to use that directly?


There are some cases when kube-proxy has to exit and restart, in order to do that we panic, but this has the problem that the log output shows a literal "panic" message and engineers, support, customers and automation gets confused, thinking that is a problem when is the expected behavior. As suggested by in , we can replace by /sig network /help /kind cleanup
This request has been marked as needing help from a contributor. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : There are some cases when kube-proxy has to exit and restart, in order to do that we panic, but this has the problem that the log output shows a literal "panic" message and engineers, support, customers and automation gets confused, thinking that is a problem when is the expected behavior. /help /kind cleanup </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign I would like to raise a PR to fix this.


found this by running tests on this PR: it's problematic for a couple of reasons: prevents kubeadm from moving out (eventually) there are no API guarantees in the given code/structs/functions in pkiutils the logic in cmd/kube- must be reworked to generate cert/key pair without using kubeadm. originally in pending fix PR
/sig api-machinery cluster-lifecycle architecture /area code-organization
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /sig api-machinery sig-cluster-lifecycle architecture /area code-organization </details
/assign
cc
Please make sure to add an import restriction to prevent a regression in the future.
/triage accepted


master-informing: ! 2023-09-07 03:13:59 +0000 UTC No response /sig network
Checking this failure This is running ingress-gce version, logs in failure is It seems the healthceck interval is not updated correctly, by reading the test tagging the people with more expertise in this area /cc
/cc
it seems the test restarts the kube-controller-manager but we landed recently the switch from intree to outtree cloud providers, so this logic now runs in the cloud-controller-manager, can this be the problem, that we have to restart the cloud-controller-manager now instead of the kcm?
/triage accepted This flake started after and kept failing after 09-07.
yeah, the logic that handles the loadbalancer shifted to that cloud controller manager with that PR


With the migration to podutils and adding some presubmits (from periodic jobs) we are seeing some failures in presubmits. The error seems to have issues finding the right node to create.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node /sig testing
any leads on solving this one?
see notes on slack -
/assign
this looks good now!


Two serial tests from sig-node are keep failing: E2eNode Suite. E2eNode Suite. pls see: [1] [2] <fmt.wrapError | 0x40005e7840: failed to update pod "pod--65da-42b7-a6a5-": Pod "pod--65da-42b7-a6a5-" is invalid: spec: Forbidden: pod updates may not change fields other than ,,, (only additions to existing tolerations), (allow it to be set to 1 if it was previously negative) core.PodSpec{ Volumes: []core.Volume{ { Name: "cm", Name: "secret", VolumeSource: core.VolumeSource{ ... // 3 identical fields AWSElasticBlockStore: nil, GitRepo: nil, Secret: nil, Secret: &core.SecretVolumeSource{SecretName: "secret", DefaultMode: &420}, NFS: nil, ISCSI: nil, ... // 9 identical fields FC: nil, AzureFile: nil, ConfigMap: &core.ConfigMapVolumeSource{ LocalObjectReference: core.LocalObjectReference{Name: "does-not-exist"}, DefaultMode: &420, }, ConfigMap: nil, VsphereVolume: nil, AzureDisk: nil, ... // 7 identical fields }, }, }, InitContainers: {{Name: "init1", Image: "", Command: {"sh", "-c", "sleep 5s"}, TerminationMessagePath: "/dev/termination-log", ...}}, Containers: []core.Container{ { ... // 9 identical fields ResizePolicy: nil, RestartPolicy: nil, VolumeMounts: []core.VolumeMount{ { Name: "cm", Name: "secret", ReadOnly: false, MountPath: "/config", ... // 3 identical fields }, }, VolumeDevices: nil, LivenessProbe: nil, ... // 10 identical fields }, }, EphemeralContainers: nil, RestartPolicy: "Always", ... // 28 identical fields } happened since 18/10/23 run the e2e testcases manually No response <details</details<detailsImpacts all </details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign /cc
cc
failed on x86 as well:
/sig node
/unassign
is merged today. /cc
will take a look at this ASAP /assign
and I are sorry for the issue this caused. We decided to open to fix this issue. I think this test was more complicated than we thought. Sorry for this.


CI always has error when using to replace in function in file . No error occurred Use to replace in function in file . No response <details</details<details</details<details</details<details</details<details</details<details</details
/sig api-machinery /area test
/triage accepted /assign /assign


The controller-manager node-lifecycle-controller watches node lease and update probeTimestamp , and it check node probeTimestamp is before . The node-monitor-grace-period default value is 40s. When the client connection lost, the lease watcher cannot receive new event. With the http2 health check, HTTP2PINGTIMEOUTSECONDS default is 30s, HTTP2READIDLETIMEOUTSECONDS default is 15s, it will delay 45s to kill http2 connection. During this period, node-lifecycle-controller will mark all nodes NotReady. Here are some logs When the http2 connection lost, the watch should stop as soon as possible. The reflector should relist resource, and node-lifecycle-controller should not mark all node NotReady. HTTP2PINGTIMEOUTSECONDS+HTTP2READIDLETIMEOUTSECONDS should be less than 40s. the controller-manager node-lifecycle-controller watch client srcIP:srcPort. Drop all tcp package send into srcIP:srcPort. for more than 40 seconds, all node will be marked NotReady node will be Ready after some seconds. No response <details</details<details</details<details</details<details</details<details</details<details</details
How about double-check node lease before marking node NotReady? The node-lifecycle-controller can get reference lease from apiserver with
/sig node
/sig api-machinery
/cc We have discussed this timeouts values before and myself, I also think got some metrics with lower timeouts for the HTTP2 connections. In this situation is clear that the defaults of the node watcher and the timeouts are related and having good defaults will be ideal, but the question is which values do we change ?
I think in internal network, the ping response almost return in 1 second, the HTTP2PINGTIMEOUT_SECONDS default value should be 5 seconds. And the total timeout will 35 seconds. WDYT?
/triage accepted any pointers in other places where those timeouts were changed for http2? Any notes from those discussions?
/priority important-longterm since it is not a regression I will keep it longterm priority and not higher. But we can reevaluate if we hear more reports about it.
/assign
I'm investigating problems similar to this, but I'm looking for a way to confirm this is what is happening - and if there are tips for how to recreate such a scenario, that would be very helpful. We get infrequent reports of "worker goes NotReady for a short time" incidents that appear to be triggered by events like failing over a LB sitting front of the kube-apiservers. We've done some tests with a simple client-go app and have recreated a scenario in which the app logs messages after 45 secs of "context exceeded" errors. The failures reported to us from customers don't have http2 errors in the kubelet logs, and our attempts to recreate with kubelet have never shown more more than a single error which works on subsequent retry.
Hi folks, we have a simulated recreate of the problem. This recreate involves injecting a 40 second delay in "connectivity" between the Kubernetes API server and etcd (we a 40 second delay to etcd defrag and defragged all etcd members at the same time). The connectivity between these two components is obviously critical to nodes successfully updating their leases in order to stay ready. We use 40 seconds because it matches . The test will usually cause one or more of the cluster nodes will flap their readiness state. Setting on the Kubernetes controller manager prevents the flap. Given this, I think that one or more of the Kubernetes default configurations needs to change in order to tolerate such "connectivity" problems in order to maintain healthy node state.
What's the proposal then? Do we update --node-monitor-grace-period to 50s? It seems the less risky change as the client-go defaults are more sensitive and impact the whole ecosystem, beyond kubernetes itself are you open to increase this default? It seems worth it to improve the reliability of cluster
I think we should update .
can someone send a PR to see how it will look like?
I have been following this discussion as we had encountered the bug as well. I have created a PR - Please let me know if it works
I'll add another data point from our testing. Namely, the simulated recreate noted depends on the Kubernetes controller manager (KCM) leader election configuration. Using the following leader election configuration for the KCM yields a KCM that has no leader after the delayed etcd defrag. As a result, nodes won't be marked not ready unless the KCM leadership is restored before the nodes renew their lease. Extending the KCM leader election configuration to the following, makes the problem easier to recreate.
/assign Reassigning the issue to me


The issue is to follow up discussions in : running conformance test with the commonly used tool, sonobuoy, failed because the tool's test runner is lack of permission towards . The original issue was resolved by changing the e2e (conformance) test of to an integration test. Several problems remained: privilege requirement in is outdated. discovery is one of the least-privileged permissions, and is expected to be granted to every authenticated user but is not actually. the result returned by should be part of conformance? 1st and 2nd are straightforward to fix, not sure about 3rd, should we add a separate test or add a verification step in existing one like With the fix of 2nd, e2e for should no longer fail with any test runner. cc - should be reflect the accurate permission requirement discovery permission should be granted to all authenticated users. test should check the result returned by . Inapplicable No response <details</details<details</details<details</details<details</details<details</details<details</details
/sig api-machinery /sig testing
I agree with this. We have conformance tests around things like webhooks and custom resource definitions. I'd recommend rephrasing that as "limits itself to capabilities exposed via APIs (e.g., does not require root on nodes, access to raw network interfaces) and does not require write access to system namespaces (like kube-system)" I'm not sure I agree with this. No binaries Kubernetes provides (that I'm aware of) and nothing in client-go needs to make use of root discovery information. I'd be very slow to expand endpoints included in . Unless we have compelling reasons for people to use and rely on this data, I would lean towards no.
can we fix 1 , that seems that everybody agree and looks like the most clear one? the other sound more contentious
thanks for your comments. Created for 1 and closed the other one.
Thanks all. Agree on this outcome.
/triage accepted


When installing the metrics-server to implement the API and the prometheus-adapter to implement the API, it is possible to make the OpenAPI handler fail to initialize with this error log in apiserver: This will result in kubectl discovery calls to /openapi/v2 to fail with a 503: The OpenAPI handler should not fail to initialize if both and are serving the same path for different API groups/versions. I can provide exact installation steps if needed, however, a high-level repro is: install metrics-server to serve install prometheus-adapter to serve ensure resourceRules is populated in the prometheus-adapter config so that resource metrics are also served by restarting apiserver will result in the OpenAPI failure and 503 responses to /openapi/v2 path No response <details</details<detailsAWS </details<details</details<details</details<details</details<details</details
Please let me know if I should reopen the issue in
/sig api-machinery
/sig instrumentation [metrics apis]
Would you mind taking a look when have time? Thank you^^ /triage accepted
I have a question. should be serving under . Why does an apiservice serving another gvk register paths under ? cc we've made the assumption that an APIService (such as metrics-server or prometheus-adapter) serves only gvks under its own registered group version. I'll do some more investigation but just wondering if prometheus doesn't follow that assumption?
the prometheus-adapter can also be configured to serve and replace the metrics-server by enabling these metrics, However, if the prometheus-adapter fronts and metrics-server fronts , it does seem like the correct behavior would be for prometheus-adapter to serve until an operator updates the APIService to reference prometheus-adapter rather than metrics-server. We're seeing this issue occur for upgrades between 1.27 and 1.28 so it does look this path duplication did not cause the OpenAPI handler to fail on 1.27.
I just upgraded from 1.27 to 1.28 and running into this problem. Some kubectl apply operations fail, and ArgoCD cannot load its custom resources. Any suggested workaround? Would removing the from the prometheus configuration resolve it? Or perhaps disable metrics-server?
Simply looking at the code diff between 1.27 and 1.28 I can't pinpoint the cause. Can I get your exact repro steps? suggests that the conflicting paths should be removed before the merge. We always call this function with
One other question is if this is reproducible by just applying a set of yaml/installations or does it require a cluster upgrade to trigger? Thanks!
hit same problem and we cannot install any other new helm plugins. maybe we need to move forward
Here are verbose repro instructions. An apiserver restart on 1.28 will induce the issue.
Hello. I'm still having some trouble reproducing although I'm not 100% sure if the configuration is correct. I'm trying to repro via kind. afaik installing via helm has prometheus adapter serving . seems to suggest that prometheus adapter is serving under . Applying the update to the configmap per the instructions didn't change the apiservice on my end. Not sure if this is my misconfiguration or intended. I've also tested via integration test that having two openapis serving the same path doesn't cause errors and one will get filtered out. A simple workaround is probably to delete the metrics-server if it is already superseded by the prometheus-adapter.
The is not set up in an resource for us, we only created an APIService with . When I go fetch the openAPI schema served up by the endpoint, it has schema information for in it if that is enabled using . If I remove then the schema fetched from the server doesn't have stuff in it, and the error is resolved. We have been keeping around because I think something uses it directly, or used to. Maybe ? That could have changed But in any case, this problem is quite severe when it happens so it would be nice to fix it as there may be others who do the same update and have to do some deep diving into the logs and all kinds of crazy diagnostics and guesswork to figure it out.
How does the get reached without an APIService? The resources would be unavailable for the apiserver to query and it'll be through something else? I see, adding resources for me doesn't populate the OpenAPI. I wonder if this could be cloud provider specific. I'm totally supportive of fixing, it's just quite difficult to reproduce on my part .
Hmm good question. In our staging cluster I tried bringing back the in the custom metrics server configuration and restarting all the masters, and the problem did not come back, everything seems OK now. So it maybe isn't that easy to reproduce
After updating the configmap, restarting the prometheus-adapter pod to pickup the changes and restarting the apiserver are both required to induce the issue. I have been able to deterministically reproduce this following those steps, please follow up here if you are still having issues.
That's good to hear. I'll need some more help with reproducing it. I've been able to verify that restarting the prometheus adapter pod picks up the configmap changes but restarting the apiserver hasn't caused any crashes for me. I've tested this both on the latest master branch and 1.28.
While I'm yet to fully understand these changes, the 2 PRs below (both of which went into 1.28) seem likely candidates for a regression here: - introduced lazy loading in openAPI v2 handler to load/compute the aggregator spec when requested by client - switched the spec merging function from to any clue if one of the above changes could be causing mischief?
Also from example request at the top, the impact seems to be to the v2 API: Nathan - when you get a chance, could you also confirm if v1 API doesn't run into this?
OpenAPI only supports v2 and v3.
hi, any clue about this issue?
It's possible but I haven't been able to reproduce it. Can I ask if this is happening on cloud providers outside AWS?
Here are the apiservices and relevant pod objects with a working repro from these steps If it's useful, I can provide yamls for these objects in addition to the settings applied in the prometheus-adapter configmap. Can you confirm that the pods backing your prometheus-adapter and metrics-server are running?
I meet this problem
We the problem in GCP as well.
ACK in Alibaba Cloud also meets this problem
We were able to fix the v2 availability by and re-installing the kube-prometheus helm chart. Unfortunately, after restarting the k8s api server, v2 availability is broken again.
yes, I'm running on kind. I'll do some more digging, assuming it's related to those PRs linked by but still haven't pinpointed anything.
I tried to reproduce on my local setup with based on steps + , but I could still get a successful response when I executed . Then I reverted one of the step mentioned in to and waited until the new pod came alive and then did a apiserver restart and I was able to reproduce the same thing on as well i.e; I get the 503 response when I execute following command after the fact. After this point, even if i change it back to and restart the apiserver , the error is persistent. Let me know if this works for you to reproduce ? PS: Also, i want to call out that there is no issue with , its .
Thanks, alright I found a way to reproduce it consistently without restarting the apiserver. Still investigating the root cause. Thanks for the detailed reproduction and
Quick update: This issue was a combination of bugs both in k/k and prometheus adapter. I opened to mitigate on k/k side which should fix the issue. The issue was difficult to reproduce because it requires a very specific ordering of OpenAPI merging to trigger, which is why I suspect many people were able to reproduce after some restarts to kube-apiserver. Prometheus and metrics server serve differing OpenAPIs for the endpoint. Prometheus also doesn't seem to be registering discovery's OpenAPI properly (missing OpenAPI for but having the OpenAPI for whereas it is all present in metrics server. This inconsistency in how prometheus and metrics server handle the OpenAPI surfaced the merging issue.
a little wired question: why do we need to restart kube-apiserver to reproduce this problem?
Tried to reproduce the issue after cherry picking the fix and didn't run into below error. Fix is working as expected.
It can look like restarting prometheus-adapter "fixes" the issue? I am also curious about suggestion of removing the section from the config map works as a temporary workaround until released upstream (we use and cannot delete the apiregistration)? As I understand the fix for this issue is currently in


We should check the images configured in test/e2e to ensure we're not adding new images: In almost all cases, should be extended to support any missing functionality and used as the sole image for new tests. This has been the policy in SIG Testing for at least a few years now. To prevent regressing on this, we should check the list of images and permit tag updates but NOT adding any more names to the list. We can do this with an allowlist against /sig testing /kind test Others are welcome to work on this, but I'll submit something soon if nobody beats me to it. /assign Follow up to , we should aim to ship something this release (1.30).
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : We should check the images configured in test/e2e to ensure we're not adding new images: /kind test /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hello! Release Signal shadow here. This issue has not been updated for a long time, so I'd like to check what's the status. The code freeze is starting (less than a month from now), and while there is still plenty of time, we want to ensure that each PR has a chance to be merged on time. As this issue is tagged for 1.30, is it still planned for this release?
We still intend to do this and it's a test-only change so we're aiming for test freeze.
Hello! Release Signal shadow here. I'd like to check what's the status. The code freeze is starting in a week, and we want to ensure that each PR has a chance to be merged on time. Is it still planned for this release?
As has no update recently, /milestone clear we may make it in v1.31.
Yeah I think we have to revisit this later.


Part of From For now posted as a "help wanted" issue, however if we cannot hit the timeline for 1.30 code freeze () I would take it over so we don't miss. Still issue is simple enough that it can be quickly implemented. Implementation tips: Add a metric around waitUnitlFreshAndBlock method Metric should be a histogram with latency buckets similar to api latency Metric should have label for resource version. A way to monitor watch cache impact on request latency.
/sig api-machinery /triage accepted /help
This request has been marked as needing help from a contributor. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /sig api-machinery /triage accepted /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


07-02 and 13-02 No response No response /sig
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scalability
I might be related to the npd.
The same test failures happening again on 25.02:
/sig node
/sig node /cc The restart of caused the flake. I also opened to track it in NPD repo.
I missed this log. It is caused by oom. Adding some memory may help. I submitted to fix it. Does mean killing metrics-server?
/remove-sig node Sorry, I misread the log. The NPD found the metric server is OOM. It is about the metrics server, not the NPD. Sorry for disturb. I may add some memory for metric-server and check why metric server memory changed. I heard that metrics server recently released a new version. I will check if it is related.
/sig node
OOM metrics shows the metric server is OOMed. This is odd, that only metrics-server-nanny is with a memory limit. It uses whose command is . metrics-server container has no memory limit, but it got killed by OOM. Does this indicate that the node is OOM? (edited)
The last run failed in node-problem-detector ContainerStateTerminated. I0227 05:03:31. ] PodStartupLatency: perc50: 14m29.018886827s, perc90: 28m44.625518807s, perc99: 32m2.19303496s; threshold 1h0m0s I0227 05:03:32. ] Loaded restart count threshold overrides: map[coredns:2 fluentd-gcp:999 konnectivity-agent:2 kube-controller-manager:1 kube-proxy:2 kube-scheduler:1 l7-lb-controller:1 metadata-proxy:2 prometheus-to-sd-exporter:2] I0227 05:03:32. ] ClusterOOMsTracker: gathering cluster OOMs tracking measurement E0227 05:03:32. ] TestMetrics: [action gather failed for SystemPodMetrics measurement: restart counts violation: RestartCount(node-problem-detector-tgl8h, node-problem-detector)=1, want <= 0; RestartCount(node-problem-detector-vpsmz, node-problem-detector)=1, want <= 0] I0227 05:03:32. ] Step "[step: 29] gathering measurements" ended W0227 05:03:32. ] Got errors during step execution: [measurement call TestMetrics - TestMetrics error: [action gather failed for SystemPodMetrics measurement: restart counts violation: RestartCount(node-problem-detector-tgl8h, node-problem-detector)=1, want <= 0; RestartCount(node-problem-detector-vpsmz, node-problem-detector)=1, want <= 0]] I0227 05:03:32. ] Waiting for the chaos monkey subroutine to end... I0227 05:03:32. ] Chaos monkey ended. I0227 05:03:32. ] I0227 05:03:32. ] Stopping DnsLookupLatency probe... I0227 05:03:47. ] Stopping InClusterNetworkLatency probe... I0227 05:04:22. ] Resources cleanup time: 50.539700215s E0227 05:04:22. ]
E0227 05:04:22. ] Test Finished E0227 05:04:22. ] Test: E0227 05:04:22. ] Status: Fail E0227 05:04:22. ] Errors: [measurement call TestMetrics - TestMetrics error: [action gather failed for SystemPodMetrics measurement: restart counts violation: RestartCount(node-problem-detector-tgl8h, node-problem-detector)=1, want <= 0; RestartCount(node-problem-detector-vpsmz, node-problem-detector)=1, want <= 0]] E0227 05:04:22. ]
I0227 05:04:22. ]
I0227 05:04:22. ] Running testing/huge- I0227 05:04:22. ]


Kubernetes 1.27 with feature gate enabled and disabled, a rebooted node is not able to re-start pods that were running there before the reboot. The node reports in Pod events: The Pod should start just fine. On Kubernetes 1.27: NewVolumeManagerReconstruction feature gate. a StatefulSet that uses a volume. -reboot the node which runs the StatefulSet Pod. The reason is that kubelet's VolumeManager reconstructs the volume as uncertain during startup and it uses a provisional outer volume name for it, because it has not seen the API server and Pods in it yet. This provisional volume ID is then used in ActualStateOfWorld forever. While kubelet does mount the volume after reboot, it's not able to match it with the Pod that needs it. Kubelet reports , while the volume is mounted with ID in my case (= the provisional name, we use for it) and not (=). It is not reproducible with enabled. <details</details<detailsAWS + AWS EBS CSI driver, but it should not be relevant. </details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage /priority important-soon This is reproducible when SELinuxMountReadWriteOncePod is disabled and NewVolumeManagerReconstruction enabled. They both got enabled by default as beta in 1.28. So someone has to explicitly disable SELinuxMountReadWriteOncePod to hit this bug.


Extension API servers do not currently receive the header when requests are proxied through kube-aggregator, so spans are not linked properly. Traces from the extension API sever should be linked to the parent in kube-aggregator. tracing for kube-apiserver tracing for sample-apiserver and aggregate it to kube-apiserver a request to kube-apiserver for one of the resources from sample-apiserver sample-apiserver traces will not be linked to the kube-apiserver traces No response <detailsThis problem currently exists on . </details<details</details<details</details<details</details<details</details<details</details
/sig api-machinery
/cc /triage accepted


We use a to nil out the : However on a re-sync, this results in a data race that was observed when running unit tests: The read had occurred previously during a callback. On re-sync, the retrieves every object from the cache, invokes the transform function, and re-queues them. However, the object instances are the same as those stored in the cache so mutating them outside the cache without lock protection is unsafe. It is documented that the "TransformFunc sees the object before any other actor, and it is now safe to mutate the object in place instead of making a copy" however this is not the case on a re-sync. The should make a copy of the objects retrieved from the cache store on a re-sync or perhaps elide the on a re-sync since the objects would already have been transformed. As a workaround, I changed our to only nil the if not already nil but this is a latent bug that others will likely run into.
/assign That means that and introduced possible panic regressions into kube-controller-manager and kube-scheduler /priority critical-urgent /kind regression
/transfer kubernetes
/sig api-machinery
kube-controller-manager and kube-scheduler do the same in 1.29+ and likely have the same issue
I think delete has the same problem, and replace may have the same problem: Passing the same object to the transformer after it is already in the cache is unsafe and we should not do that at all.
/assign /cc
Looking into this. I may take time to come up with a proper fix. But would like to evaluate the risk for the current situation. Not entirely sure how delete and replace would be triggered in normal settings, but resync is fortunately low frequency as the default setting for resync period is between 12h and 24h for (--min-resync-period duration Default: 12h0m0s, see ) . As for the resync function is Just think out aloud, I read a thread discussion somewhere that resync was a mechanism in the past to prevent data inconsistency due to bugs but client-go has been mature these days, resync is no-longer needed in theory but for a safe guard. So maybe disabling resync is a reasonable fix if delete/replace can be fixed.
You're mixing resync with relist - I believe relist is no longer needed. Resync is actually used. FWIW - I opened an initial version of the fix:
I think it's resync. The thread I mentioned is
/triage accepted


The test is failing on Windows [0]: The idea of the test is to check that kubelet still follows and reads the Container Logs after a Log Rotation occured. However, because the followed file is open, it cannot be renamed, and thus, the container logs cannot be rotated. This issue could be prevented if the file would be opened with the [0] [1] Kubelet should not block container log rotation. Run the test ` is failing on Windows. /sig windows /kind bug /kind failing-test latest, master. This unit test has been failing for a while. N/A Windows nodes </detailsN/A N/A N/A
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/kind failing-test /sig windows


in This flaking test case was newly as part of the above PR Flake was reproducible with stress tool: [root kubernetes]# go version go version go1.22.2 linux/ppc64le [root kubernetes]# stress - ^TestLog /sig testing
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
Looks like in this test case is a concatenation of logs from Pod0 and Pod1, hence it is failing in some cases. I will add a PR to split it.


Currently, the scheduler honors Unschedulable(AndUnresolvable) or Pending only from PreFilter, Filter, Reserve, and Permit (WaitOnPermit). It doesn't expect other extension points to return those statuses and just regard all non-success status as errors, that is, immediately retrying scheduling of Pods. Actually returns Pending at PreBind, which is currently being ignored as described. So, we should expand UnschedulablePlugins/PendingPlugins to include PreBind plugins. (or maybe all extension points too?) /cc /sig scheduling /kind feature
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Both seems reasonable to me.
Because is not handled for , the following error is logged:
This patch gets rid of the error. I am uncertain whether it has other implications? Performance was the same with and without it.
What I imagined is like that. It shouldn't have any other impact because volumebinding, which is the only PreBind plugin apart from DRA in in-tree, only returns success or error status. We have to mention the change in the release note for custom plugin developers though.
I've turned this into a PR:


/kind failing-test <!-- If this is a flake, please add: /kind flake --/sig node
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Failures look like started when landed