A common pattern in Kubernetes (both internally and in third party usage) is something like: (this is from the ). This is commonly called at a fairly high rate; in our case we would actually end up seeing this account for significant portions of CPU/memory to compute the and run on it. In our controller, as an optimization, we replaced usage of and just used our own simpler label matching code, leading to a 70% speed up in our function (which does more than just match) and 96% allocation size improvement; see I believe similar could be done in Kubernetes directly by having a specialized Selector implementation for simple labels. This would transparently given substantial performance benefits to anyone using the library
/sig api-machinery
is the change here a new kind of selector matching or a tweak to something existing? It's hard to tell if this is proposing a new design to partition types of selectors that needs a kep versus a perf tweak to existing code that is straight-forward.
The proposal is an internal performance improvement to make a specialized implementation of the selector matching internally. So two concrete implementations for 1 shared interface. is a PoC of this. I think it can be extended (by me, or others if someone wants to) to complete it
We discussed this one during our bug triage. There seemed to be the feeling that this is large enough to need a KEP? Would you be up for that? Additionally you could bring it up in our SIG meeting to socialize / discuss it more if you wanted.
The patch shared in helps to explain, it's a lot smaller than the description made it sound. This is a special case for equality-only selectors. Probably still worth bringing up at a sig-meeting, but if if the patch is a good representation, it is more bounded than expected from the issue description.
/triage accepted


Created limit range object with this parameters then tried to create this pod got message: Error from server (Invalid): error when creating "pod-to-be-": Pod "nginx-to-be-removed-4" is invalid: spec.containers[0].resources.requests: Invalid value: "700m": must be less than or equal to cpu limit Pod should be created with Requests & Limits of cpu equal to 700m (since it is in range of min/max cpu limitrange object) create limitrange object and pod as in description No response <details</details<detailson premise/AKS </details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
You have a default CPU limit of 10m, and no explicit CPU limit on the container, but a request larger than the limit. The request must always be less than or equal to the limit. Your pod needs to also set a higher limit, or the limit range needs to not set a default limit.
/remove-kind bug /kind support
/cc
but this is not obvious behavior since there is min & max set in limit range. This is limits (not min/max) that should be overwritten by pod. limits should be set to requests in this case imho
limit range limits what the user can set, it doesn't perform any mutations other than the defaulting and doesn't claim to. Min/max enforce that pods cannot set higher / lower. The default ensures that if the pod doesn't set a limit it will have one. personally I would remove the default limit, especially at 10m, this will aggressively limit pods. it would be, if the pod set a limit that would be a substantial breaking change, I don't think we could do that short of a v2 API and I'm not sure it would be correct behavior even then. we could make it opt-in, but then, why not just set them to match on the pod to begin with?


As per, , wire up metrics/slis for ccm
/assign /sig instrumentation /sig cloud-provider
GitHub didn't allow me to assign the following users: Richabanker. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign /sig instrumentation /sig cloud-provider Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/triage accepted
/remove-triage accepted
/triage accepted


After merged, build logs for e2e tests have started showing stack traces that look like: It seems like the used in this is nil I have a suspicion this is happening for storage tests that skip No new stack traces in build logs Run e2e tests against the in-development version of kubernetes I identified when this started happening by looking for an increase in build-log size is the first build where this started happening, which happens to be the first build after merged, which is the PR that introduced this new way of invoking <details</details<details</details<details</details<details</details<details</details<details</details
/sig testing /triage accepted /assign To put this on your radar
/cc
Correct. This happens because of the ordering in Tests gets skipped before is called. This avoids the expensive namespace creation/deletion for tests that don't need to run. The for then gets executed although the framework didn't get initialized. My preferred solution is to document that the framework instance might not be initialized in an AfterEach callback and add a nil check to .
See Only was affected. The other framework extensions use DeferCleanup, which only runs when needed.


The replicas is 3 but just create 1 pod 'web-0' and one pvc, then kube-controller-manager panic happend. Should create 3 pod and 3 pvc add '--feature-gates=StatefulSetAutoDeletePVC=true' arg for kube-controller-manager, then create statefulset with 2 replicas and volumeClaimTemplates. No response <details</details<details</details<details</details<details</details<details</details<details</details
/sig apps
I can reproduce the issue and want to fix it. It seems like variables cause . /assign
/triage accepted
Great find! Does this only happen on an upgraded cluster? I'll spin up a cluster and see if the e2e test catches this problem.
The e2e tests are still passing on a 1.24 cluster, so I suspect this is an upgrade problem. I wonder if there are other holes here.
I installed the Kubernetes 1.25 cluster directly using and still reproduced this issue, so it was probably not caused by upgrading the cluster.
Interesting. I thought that would set the policy to be non-nil. Let's keep this bug open after you submit your PR as I think something else is going on here---I don't think we should be seeing the null policy in the first place.
I suspect that somewhere accidentally set it to nil, and I'll keep looking at this code.
I tried the following (and also with parallel pod management), and it didn't seem to cause the panic. This was a 1.24 cluster, maybe this is only happening on head versions?
I just tried this on a kind cluster running on k8s head, and the problem didn't seem to reproduce. But this was setting the feature flag default in I wonder----does the api server need to have the feature gate as well? That would explain why defaulting wasn't happening.
As far as I'm concerned,I just set the in kube-controller-manager manifests to reproduce this issue. I repeat the yaml used as follows: I tested that in statefulset yaml, setting or not would cause the component to panic. The panic logs are the same as above, so ignore them here. statefulset with : The environment
I believe you---this doesn't seem to be well-documented, but it's my understanding that you need to set the feature gate on all of the system components. Obviously failing to do so shouldn't cause a panic, so thank you again for finding & fixing this bug! Let me ask around to see what the expected behavior is. I know there may be skew between kubelets and master versions, but I didn't expect to have to deal with skew between master components.
I read the code, and now I understand what you mean. This feature gate must be enabled in both the and components. We need to describe it in the to prevent someone from ignoring the component, which also needs to enable the feature gate.
I don't think we need all the system components, like the component, right? I read the original KEP documentation. This feature gate only involves components and :
I had a bit of discussion about this with others. Because upgrades don't occur atomically, any feature gate has to work regardless of how the other system components are configured (things shouldn't panic!). I suppose if the cluster administrator wants to track what needs to be enabled on each feature they can try to sequence which component has which feature gate. If it were me I'd try to make all the components consistent though.
This is certainly a method, but it is a bit more cumbersome for the administrator. It is best to have a place for the document to be queried by the administrator and avoid confusion. I don't think everyone is very familiar with the k8s mechanism. I suggest adding a partial description to this section of the : Before: After: I can submit this PR on .
This is going to be true of any api change. When you tried to use it with only adding the controller manager feature gate, I think you would be able to set the retain policy, but there wouldn't be any defaulting or validation (?).
I've seen other feature gates that say what components are needed to enable it to use the new feature. For example: If it's not necessary, then I close .
only adding the controller manager feature gate, I think you would be able to set the retain policy, but there wouldn't be any defaulting or validation (?). I remember just enabling it in kube-controller-manager. it using the retain policy, but the statefulset does not have the policy from output of (It should be the apiserver authentication forbids it)
Yeah I think that makes sense. Thanks for finding this issue! After your PR with the panic fix is submitted, I think we can close this?
Yes. I agree. /close
Closing this issue. <detailsIn response to : Yes. I agree. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


release-master-informing: capz-windows-containerd-masterKubernetes e2e suite: [It] [sig-node] Container Lifecycle Hook when create a pod with lifecycle hook should execute prestop https hook properly [MinimumKubeletVersion:1.23] [NodeConformance] Kubernetes e2e suite: [It] [sig-node] Container Lifecycle Hook when create a pod with lifecycle hook should execute poststart https hook properly [MinimumKubeletVersion:1.23] [NodeConformance] Both the tests have been failing during most of the runs since 20th October Kubernetes e2e suite: [It] [sig-node] Container Lifecycle Hook when create a pod with lifecycle hook should execute prestop https hook properly [MinimumKubeletVersion:1.23] [NodeConformance] | 1m6s -- | -- {Oct 31 18:42:02.425: Timed out after 30.001s. Expected <errors.errorString | 0xc0015283b0: { s: "failed to match regexp "GET /echo?msg=prestop" in output "I1031 18:41:06. 8988 ] Started HTTP server on port 9090nI1031 18:41:06. 8988 ] Started UDP server on port 9091n"", } to be nil failed (0xc000f1d000) +0x469 () +0x335} Kubernetes e2e suite: [It] [sig-node] Container Lifecycle Hook when create a pod with lifecycle hook should execute poststart https hook properly [MinimumKubeletVersion:1.23] [NodeConformance] | 5m11s -- | -- {Oct 31 19:05:50.205: timed out while waiting for pod container-lifecycle-hook-6892/pod-with-poststart-https-hook to be running and ready failed (PodClient).CreateSync(0xc000eb5920, 0x22?) +0x94 (0xc0045e8c00) +0x73 () +0x325} No response_ /sig windows cc
/triage accepted /cc
These failures happen also with Kubernetes master + containerd master with Flannel CNI. Testgrid link:
I figured out why these new failures appear. I've submitted the fix via PR
This issue is also affecting 1.25 tests: If possible, we should cherry-pick to the release-1.25 branch once it's merged on master.


Why is this needed? The functionality that the package provides is already present in golang's builtin errors (using in ) and the github repo is now read-only. /kind cleanup /priority backlog /good-first-issue Notes to people that want to collaborate Avoid creating a single PR for the entire repo. Create multiple PRs, each targeting a specific SIG. Do not submit PRs for
/area code-organization
/kind cleanup
this is only partially true, e.g. %w does not provide stack traces
/assign
I see. Should we close this issue then?
I think it depends on how it's being used. If the traces are not the goal then we can do a simple substitution. It's probably best to drop read only dependencies ASAP but it's worth noting that if we depended on the full functionality in any way we'll need a more complex replacement (KIND did/does, not sure about k/k)
k/k git log suggests that some cleanup has already been done in the past (maybe in other commits too): This issue triggered 3 PRs and 2 of them are merged. there does not seem to be calls to so most cases it looks it's just wrapping errors.
for cmd/kubeadm in kubernetes/kubernetes we decided to not drop usage of pkg/errors even if it's a deprecated project. prior ticket and comments are here
for cmd/kubeadm in kubernetes/kubernetes we decided to not drop usage of pkg/errors even if it's a deprecated project. prior ticket and comments are here
I see. Added a note.
is it still open ?
should be replaced with the standard library for the following reasons: archived library with no maintenance or security updates the go community used pkg/errors to introduce eventually into the standard library hence the approach that was selected is more idiomatic stack traces are hardly used in error wrapping and are not a reason not to refactor
On place left in Except the k/k repo, as is archived, some other repos may need an update.
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/triage accepted
The label cannot be applied. Only GitHub organization members can add the label. <detailsIn response to : /triage accepted Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
is it necessary to remove "" from every location? I noticed it in many places. Also, do I need to create separate PRs for each file where I remove it?
/assign
From kuberentes/kubernetes repo view, only two locations of kubectl code should be updated. (Do not submit PRs for ) see For other repos, we should follow up until the dependencies' repo is updated. I will track them in
/assign
/assign
to remove dependency you can use the standard Go error package or creating your own handling package, this could contain custom functions for generating and handling errors.
let me work on this issue now please /assign
We are in code freeze. I suggest waiting for a couple of weeks.
Hello, is this still up?
Let's get this going
/assign
Just opened a PR for this issue
/assign
Update the remaining todo items.
/sig architecture umbrella / code org
/assign
Hey Are you still working on this one? If not- would you mind if I take a stab at it? Thanks!
Is this issue really still open? When I do I only get hits for cmd/kubeadm. Am I missing something?
/close
Closing this issue. <detailsIn response to : /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


It was found during a review that stateful set status might not be correctly updated when new revision is being rolled out and API error occurs when deleting a pod. The status should be updated correctly Not easy to reproduce, but should be possible with this unit test No response_ <details</details<details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig apps


It's at line 111 When the server stopped before timeout, the first goroutine will be block on because the receiver in the second goroutine is not exist. Changing to a buffered one may fix it. no goroutine leaks run related test with leak checker No response <detailsdev </details<detailsunknown </details<details</details<details</details<details</details<details</details
/sig api-machinery
/assign /triage accepted
/cc
It should close the channel rather than write to it, that's the easiest fix


Events from node taint manager doesn't report involved object apiVersion: ! apiVersion is reported as other events. For example: ! a pod on node A (not master) off node A is reported, but without apiVersion in involvedObject. The apiVersion is not reported in The issue is still not fixed in master: ! <details</details<detailsIrrelevant </details<details</details<details</details<details</details<details</details
/sig node /sig apps
/assign In the latest version, the issue is confirmed
one suggestion you should wait for the corresponding SIG to triage the issue and accept it before sending PRs, the chances of your PR getting reviewed and approver are much higher
You are right sorry about that. It won’t happen again.
no no worries, it was a suggestion, is not that you did something wrong, but it is possible that your PR doesn't get reviews and you get frustrated
/remove-sig node please re-add if this is node-related issue
I think it is a node related issue. The root cause in taint manager.
/sig node
/assign /triage accepted


For 1.27 we should graduate the ReadWriteOncePod feature to beta. This is needed to rollout the feature to GA. In beta this feature will support preemption of pods using PVCs. See:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage /sig scheduling


A status of Pod mounted NFS PV remain as terminating when a node shutdown and restart. A Pod is completely terminated. Deploy a Pod mounted NFS PV Shutdown the node where the Pod is running Wait for 5 min and the Pod status become Restart the node The Pod status is still and the Pod is never terminated completely <detailsPrepare a NFS Server.(I used ) Deploy a Pod mounted NFS PV like this. Shutdown the node where the Pod is running.(In this case ) And the status of node02 becomes . After 5 min later(by default), the Pod status becomes . (The time a Pod becomes status depends on .) Then, restart the node and the status of node02 becomes . But the Pod status is still and the Pod is never terminated completely. It seems that a faild to delete Pod because of trying to unmount a not mounted directory. Force delete Pod. Delete Pod's directory on the node. </detailsNo response <details</details<detailsnone(on premise) </details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage
/sig node
It dose not happen using , so it seems to depend on NFS volume plugin.
/assign
I try to build and run kubelet with comment out this logic. Then it seems to get well. related: This change checks the result of instead of the mount point check. However, check logic is not implemented in and (these functions are used in NFS volume plugin), despite it is done in .
/remove-sig node /sig storage
Reopening, fix had to be reverted due to unit tests requiring root permissions ()
I am terribly sorry... Please wait a moment. I'll try to fix it.
no worries, just get to review the rework, making sure the units tests don't require root permissions
Thank you! Fixed in .
Are there any plans to backport this to older Kubernetes versions? This seems like a critical fix.


It "connects" to dockershim There is no Run crictl v1.26.0, without setting up (or using -r) Worked in crictl v1.25.0, that still had CRI v1alpha2 This patch fixed the issue, to make the fallbacks work: <detailsv1.26.0 </details<detailsLima </details<detailsPRETTY_NAME="Ubuntu 22.04.1 LTS" </details<details</details<detailsv1.6.12 </details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
I agree that we can check for unavailable connection because it will only be done on kubelet startup.


If a CR converter is wired in and it modifies the input list, the delegating CR converter does not protect against this. A converter cannot effect the input list Not really possible externally - it's a coding issue that the current converters don't hit. No response <details</details<detailsN/A </details<details</details<details</details<details</details<details</details
/sig api-machinery /assign
/triage accepted


It seems that it was recently and it was always failing ! /sig cli /kind failing-test /assign
/triage accepted


master-blocking build-master Overall build is failing ! 2023-02-09 23:09:58 +0000 UTC make: [Makefile:487: cross] Error 1 !!! [0210 00:35:55] Call tree: !!! [0210 00:35:55] 1: kube::build::runbuildcommandex(...) !!! [0210 00:35:55] 2: kube::build::runbuildcommand(...) make: [Makefile:374: release] Error 1 level=fatal msg="failed to run: running make release: command /usr/bin/make release did not succeed: !!! [0210 00:35:54] Call tree:n!!! [0210 00:35:54] 1: hack/make- kube::golang::buildbinaries(...)nmake[1]: [Makefile:92: all] Error 1nmake: [Makefile:487: cross] Error 1n!!! [0210 00:35:55] Call tree:n!!! [0210 00:35:55] 1: kube::build::runbuildcommandex(...)n!!! [0210 00:35:55] 2: kube::build::runbuildcommand(...)nmake: * [Makefile:374: release] Error 1n" No response_ /sig testing
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/priority critical-urgent
looks like the same issue again: /cc
This would fall under sig-release or sig-testing? couldn't really gather much from logs about this though failure is so I guess sig-release then.
I'm going to reopen for that, which should work around the issue.
that's correct :) /remove-sig testing /sig release
The PR does not seem to fix the problem without disabling test cases. Opening a slack discussion and closing the loop:
Failing again. /reopen
Reopened this issue. <detailsIn response to : Failing again. /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
this is a different error: can we spawn a new issue to track rather than reopening the same one and mixing investigations for different issues?
Oh just the stack trace for make was the same, my bad, opened /close
Closing this issue. <detailsIn response to : Oh just the stack trace for make was the same, my bad, opened Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Currently, when the Label Selector in the Pod Affinity is empty, no objects will be match. That means if it's in a required PodAffinity, the Pod will never be schedulable. And if it's in a required PodAntiAffinity, that PodAntiAffinity will be no-op. So, I'm thinking PodAffinity with empty Label Selector is very likely mistake. It'd be good if we could notice it for users. To prevent such a misconfiguration. Probably, we cannot filter out such a Pod creation in the validation for a backward compatibility. /assign /sig scheduling
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I'm wondering if it has already discussed somewhere or not. Please just let me know if we have some decisions on it.
I am not sure, where do you propose to put the warning? If the pod is unschedulable, that will show up in the events and pod status.
Do you mean the place to implement this warning? I suspect ↓ is a suitable place:
A warning sounds useful, even if the pod is unschedulable, as it gives you a heads up at the time when you create the Pod. Whereas the Pod scheduling failure might be a bit harder to interpret.
ok, I thought you were referring to something that the scheduler will produce after attempting to schedule the pod. Yeah, if there is already a setup to print warnings on pod creation, it is certainly useful for this case.
Can you please also add it in topology spread?
Sure


The EndpointSlice API says and it works as expected in normal circumstances I delete the pod and the endpoints conditions are the expected However, if during this time I set we run into an incoherence with the API doc, but it seems that we already documented this edge case in the code I think that at this point is just to amend this information to the API, no? /kind bug /kind documentation
/sig network /assign
I would like to work on this. I just want a bit more clarification on what needs to be done!
I proposed a solution, that is clarify it in the doc ... once we reach an agreement in this issue we implement it ...
+1 to docs. I'd make it a little more vague, like: """ when the normal readiness behavior is being explicitly overridden, for example when the associated Service has set the publishNotReadyAddresses flag. """
so, the thing is to update the docs here with Tim's suggestion and here and do to update the generated code and submit a PR
/assign
I am not able to run command in windows, I am checking it. This error message is coming: *
/triage accepted
Is there a PR for this? Happy to review
I will raise the PR if my above problem gets solved. I am trying to make an update but the error mentioned above is coming.
What platform are you on? If the Makefile were really broken EVERYTHING would be failing.
he's on windows ... you will need a linux/macos. see
Let me know if you want someone to take it over.
I think in that case, you can assign this to someone else.


From the Deprecated API Migration Guide: , since v1.19, the has been using the new reportingController and reportingInstance to replace the decrecated source.component and But when I checked the Kube events in a cluster with version 1.26.0, those new fields are empty and the supposed to be deprecated fields have not been renamed. Seeing the values in the new fields like reportingController and reportingInstance an Kubernetes cluster with version 1.26.0 some Kube events the Kube events details for the field of reportingController, reportingInstance, source.component, No response <detailsServer Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2022-12-11T00:15:47Z", GoVersion:"go1.19.4", Compiler:"gc", Platform:"linux/amd64"} </details<detailsAzure </details<details</details<details</details<details</details<details</details
/sig instrumentation
I have also encountered this issue in my local 1.26.0 Kubernetes cluster, which does not align with the deprecation guide described in I believe I can investigate this further
/assign
Thank you for making the PR to fix this! Also a comment in the PR for the doc update.
/triage accepted


tc.shutdownActive is captured by 'func' literals in 'go' statements(Line177), it may cause a data race. I use t.Run(..., ){...}() or t.Run() {...}(tc) to avoid it . I don't know whether it is accurate, so I 'm looking forward to your reply. Data Race Nil No response <detailsNil </details<detailsNil </details<details</details<details</details<details</details<details</details
/sig node
Target file:
/area test /priority backlog
have you observed any test failures or this is found by code analysis?
Yes, it is found by code analysis. In terms of the source code, the scenario can happen.
/assign
/remove-kind bug /kind failing-test since it is not a product bug, but the test issue
Although each test case calls a goroutine, tests run serially and block on the goroutine return, so it doesn't look like it could cause an issue. However, to be safe we can create a local copy of inside the loop. I'll send a PR briefly.
Created to address this.
/triage accepted /priority important-longterm


Example: this PR seems to be flaking: comparing age of a node from the output of two commands: Kubernetes e2e suite: [It] [sig-cli] Kubectl client kubectl subresource flag GET on status subresource of built-in type (node) returns identical info as GET on the built-in type expandless 5 hours back No response No response No response_ /sig cli
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/priority important-soon
it seems that is comparing subresources and there is something there that changes over time As Sergey correctly points out is the Age The test can not assume the executions will be done in the same second
/assign


The following log line is often (~30times) logged during our tests in : . This suggests a possible bug as it was assumed to do not happen. No log line saying . Look at artifacts of all (or almost all) builds of: No response <details</details<detailsReproducible in tests </details<details</details<details</details<details</details<details</details
/sig node FYI
/triage accepted since it's a new error, it is a regression. So adding a corresponding priority /priority important-soon
are you working on this? Should I assign bug to you?
Yes, I'm taking a look at this. /assign
/cc
This is a metrics only error (we don’t report a fraction of metrics) and has no functional impact.
That makes sense, but is it expected that this is a valid state in first place? . My assumption was that if static is false, (i.e. API pod) then we should have a config?
I took a closer look at the logs, it seems that this is happening when we enter while a static pod is being synced which has UID reuse and another pod which has the same full name did not terminate yet. In this case, we block the static pod from starting i.e. () returns false: Because the pod can't start, we don't set : which means that when we log metrics there is no set and thus or are both defaulted to false:
Based on the above, it looks like this case is expected. As a result, maybe the simplest option is to simply drop the log statement since by design we only want to into account a subset of the states for the metrics?


If a pod is created and then deleted an entry in processedPods map remains forever (sometimes, it's racy). The problem comes from the fact that calls volume_manager asynchronously. If the last call happens after populator cleaned up resources, an entry is to processedPods and will never be seen. An entry in processedPods map must be deleted if pod doesn't exist and pod runtime should removed according to . To verify that the issue really exists I applaied a simple printing resources in map. Here is a reproducer: brand-new fresh k8s cluster with entries in : a pod, check its uid: that this uid is accurately reported: the pod, wait for a few mins, check for again: The entry will be kept forever because kubelet doesn't know anything about this pod anymore: so no more events will ever come to populater for this uid, and the loop in won't find it because it was already deleted from . n/a <details</details<detailsany </details<details</details<details</details<details</details<details</details
/sig storage
/triage accepted
/priority impoerant-soon
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /priority impoerant-soon Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/priority important-soon
/assign
PR () fixed this issue. It is ready to merge since Mar 21.
/assign


Not , but removed: Looking at this comment: this should be removed once the migration to lease in leader-election was done. Can this be removed already? I mean, the following 2 lines: Security folks internally were asking if we could get rid of this upstream if it is not needed anymore (if this "migration to lease in leader-election" was fully done)
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig sig-scheduling
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /sig sig-scheduling Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scheduling
/cc
/assign
Support for endpoints and configmap based leader election has been removed with this PR: Validation to error out leader elections configuration for resource lock type other than "lease" has been implemented in With this, it looks safe to remove the two lines mentioned in this issue.


Followup feedback from In a few e2e node tests (e.g. tests of deletion grace period) we have some busybox pods that wait for SIGTERM and perform some action like a small sleep to simulate SIGTERM taking some time. These tests follow the pattern of -- see The will block the SIGTERM handler from being run until after the time is elapsed. We should consider to remove the and instead use to avoid blocking the SIGTERM handler, see pattern followed in n/a n/a No response n/a n/a <details</details<details</details<details</details<details</details
/sig node /cc
/assign
/triage accepted


No response No response /sig windows /kind failing-test
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/sig windows
i think this problem can be fixed when it imports os/net in Am i right?
I also found some other unit test bugs in CI, such as running linux-related tests on windows. I will try to fix them. /assign
/cc
Thanks for opening the issue! I didn't notice the failure through the testgrid dashboard. I've sent a PR here: . While fixing this issue, I noticed there's one unit test that was failing, so that PR also addresses that as well.
Yep, the imports you mentioned are needed. Keep in mind that imports are in alphabetical order. But in addition to that, there was another failing unit test which was masked by this import errors.


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Following the example here: Everything here works when you use , but when using 1.27.1, I see the following panic: What you expected to happen: When downgrading, this works as expected: How to reproduce it (as minimally and precisely as possible): <!-- Please make sure you are able to reproduce the bug using a supported version and version skew of Kubernetes. See for which versions and are currently supported. --You will need a Kubernetes cluster with Agones installed (I'm testing on 1.25.x, as you can see above), then follow this quickstart: Anything else we need to know?: The is implemented as an API Extension, not a CRD -- which is probably where the issue lies. Looking at the line: There seems to be an assuming that there will be a under and , in - there should be some nil checks in there. Environment: Kubernetes client and server versions (use ): See above Cloud provider or hardware configuration: GKE OS (e.g: ): Linux
seems like an another edge case; Due to the server version is old and openapiv3 is returning empty?
Thanks for opening this issue /triage accepted
I'm digging into the Agones code, but I'm 99.4% sure we don't even implement openapiv3 at all for our api endpoint
Yep, confirmed -- I see a bunch of 404's in my API Extension logs for requests for
/transfer kubernetes /kind regression
/sig cli /assign
I can also work on tomorrow if Sean can not get time to look at it
If nothing else, this does also force me to publish an openapi spec for our API extension
not sure if needed to handle nils inside the schema better, or if didn't fall back to v2 correctly in the absence of v3 data
It appears to be the former, where an OpenAPI V3 document is returned without , while the code assumes this field is non-nil.
seems like that should still fall back to v2
Can we get a kubectl -v8 to see what actually happens with downloading the OpenAPI?
As requested!
Thanks for the quick response. It looks like the OpenAPI V3 document is almost completely empty The code is expecting an Open API document with . I will have a fix out shortly.
I was digging through that - I found that really interesting. We only expose an basically empty spec for the api extension. It's always "just worked", so never needed to update it. I am curious where K8s is even getting the something that it is outputting for , but that's a side note.


is recording the time that a Pod is gated. We use the timestamp when the scheduler inserts the pod into the queue: The period of time when a Pod fails PreEnque (like being gated) shouldn't be accounted in the podschedulingdurationseconds. Create a Pod with scheduling gates. Wait some time before removing the gate. Observe the metric No response <details</details<detailsAny </details<details</details<details</details<details</details<details</details
/sig scheduling
/help /triage accepted
This request has been marked as needing help from a contributor. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help /triage accepted Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
would you want to take a stab on this issue?
Yup, I can give it a shot. /assign helayoty
, as we discussed, we can address this issue with different solutions: the pod timestamp to the current time , so the gated time is substituted. timestamp . Use a pointer to the InitialAttemptTimestamp and interpret as the newest timestamp so the pod will go to the back of the queue. I'll go ahead and implement solution no. 3, wdyt?
sgtm. Any concerns ?
I think it should be option 1 + 3, and pInfo.Timestamp should be pInfo.InitialAttemptTimestamp, right? as records the time based on :
I think we should revisit this issue, in terms of its assumption: So is it reasonable to always exclude time spent in PreEnqueue from ? Note that PreEnqueue != SchedulingGates, is one of the PreEnqueue's usecases. In other words, some user of PreEnqueue may want to include the time spent in PreEnqueue.
is a signal that cluster providers can use to build SLOs. The time spent before passing PreEnqueue is not under control of kube-scheduler, so considering it would not allow to build an SLO. Are you aware of other users of PreEnqueue?
True, but comparing to state that metric X can or cannot be used to build SLO (which is hard IMO), I'm in favor of giving suggestion on SLI/SLO's semantic definition, for example: the official SLI definition of has a footnote clearly explaining its definition: So if a pod is gated, it's not "immediately schedulable", and hence doesn't fall into the definition of podstartup_latency.
A viable solution might be providing an extra label like to explicitly exclude the time spent on preenqueue.
That is true, but there is no way today to exclude those pods from the metric, or separate the time. Maybe we need an extra metric, but I think the new metric could be the one that includes the time guarded in PreEnqueue. It should probably be part of the KEP. The old metric is already stable, so we can't add labels. And it might already be in use for SLOs.
cc
I think we should be working on that. And treat it as part of GA requirement for PodSchedulingReadiness.
An edge(maybe not edge) case is a pod's scheduling attempts is 0 but the is a long time, so prefer not to account for the waiting time when gated. For users not using PreEnqueue, they will not be impacted, but for users using PreEnqueue, it's a break change for it increases the duration time. And we can add a new metric like to measure how long a pod stays in a scheduler.
I thought of "attempts", it's just I'm not sure it's a good idea to rely on "attempts" to define a SLI/SLO, b/c "attempts" is obtained from the in-memory object and in a HA config env, it's only updated by the primary scheduler instance. In other words, the value is inaccurate when the lease owner switches.
attempt=0 is good enough for most scenarios
do we have a conclusion? Should we move on with the solution or remove this issue and the linked PR from the current release?
I would like to move forward with this proposal. ?
To move it forward, I think we should add new metric(s) to explicitly tell it's including or excluding the time spent on PreEnqueue. Given is in stable level: We will need a new metric like "podschedulingexcludingpreenqueuedurationseconds".
28 Bug Triage Shadow here. Just checking-in, if this is still on track for the 1.28 release?
We don't have a conclusion yet, but as a bug, it should qualify for changes after the freezes.


Data as of June 3rd, 9:00 PM Eastern. Latest data can be found here: Additional Context:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig release
cc Do we want to keep this job and fix it, or do something different considering the debs/rpms efforts?
I think we should keep this job as long as we build packages for Google infra (those are using different specs then packages for OBS).
fixed it:


This pertains more for clusters running in GCP but not necessarily in a GKE environment. Clusters that using a CNI supporting overlay such as Calico, do not need explicit routes to be created on the infrastructure level, meaning that the route controller should not be necessary. The following snippet sets the condition for new GCE nodes when a new node is picked up by the node controller. This condition is supposed to be removed later by the route controller in However, as stated above there are cases where the route controller is not needed and should not be running. When Calico is deployed in such a way that the calico-node pods are ignoring any and all taints including the cloudprovider's taint (which we consider reasonable for a system pod such as calico), the above check creates the following race condition: a Calico pod may start and set the node network conditions ( to false) before the node controller reconciles the node. After that the node controller picks up the node and sets the condition to an otherwise healthy node. Leaving aside the race condition itself, I have to ask whether the check is meaningful in the first place. AFAIK, the kubelet in Kubelet only sets the condition for provider, but clusters should already be using kubelet with cloudprovider= For the node controller to not set the condition unnecessarily. An easy way to do it with a custom CCM image is to introduce some delay in the node sync loop, enough for pods such as calico-node to start on that node. No response <details</details<detailsGCP </details<details</details<details</details<details</details<detailsCalico </details
This should be fixed by /cc
/sig cloud-provider /sig network /assign
/triage accepted


Add the following to be safe sysctls: net.ipv4.tcpfintimeout net.ipv4.tcpkeepaliveintvl net.ipv4.tcpkeepaliveprobes These are related settings to net.ipv4.tcpkeepalivetime that was enabled in These are related ipv4 TCP settings for tuning network connections between a client and a pod.
/sig network
May you help provide some tests to ensure that setting this sysctl can't affect any other pods or the node.
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : May you help provide some tests to ensure that setting this sysctl can't affect any other pods or the node. /sig nodes Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
I adjusted it with the plugin.
Would this change apply at the node level or the pod level? I assumed it would be at the pod level, and should not affect any other pods--however Kubernetes does not natively have any limits on network connections that I am aware of--but only the pod with the changed net settings would be affected.
To verify that a sysctl has an effect on other pods, we need to verify that it's a namespace sysctl.
Here's a simple test Use net.ipv4.tcpfintimeout sysctl to create a container check if it set on the host =It is the default value on host run another container attach to the network namespace of the one where the sysctl is set =results in sysctl being set Verify in another container under a different network namespace that it is not set =It is the default value According to the test results, net.ipv4.tcpfintimeout is a namespaced sysctl
I likewise verified and , which are also namespaced sysctl, and I believe we can add them to the safe sysctl /assign
Probably need to wait for to finish before start working on it
/triage accepted


N/A 07/18 02:38 PDT No response /sig release
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Slack thread:
/sig cli


/sig scheduling /kind regression caused a performance degradation in the 5k nodes scalability tests ! The PR changed how to split the load among parallel threads. Previously, we would parallelize over the number of nodes. The PR does it over the number of plugins, which is much smaller. Performance to improve or remain the same https://perf- No response 28 N/A <details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign The quickest fix is to revert, but I'm open to other suggestions.
/priority critical-urgent
Can we just use another Parallelizer to handle the score process for each plugin? I ran the UT locally and it seems fine, but I'm not familar with
Depending on how many cores you have available, you might just overload the routines. I think it's better to revert the parallizer over plugins while keeping the change from map to slice. Although I don't expect a major impact for the latter. You might want to run a profiler for the benchmark with a higher number of nodes.
I'll create a revert version first :)
Let's revert this first since we reached the code freeze. We can do the refactor later.
Sorry for causing this trouble and thanks everyone for a prompt action


This flakiness is reproducible on x86 and ppc64le architectures. https://prow.ppc64le- in /sig
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig testing
/sig node /assign


/sig scheduling Include the Pod UID in the event message for preemption by kube-scheduler Minimizes the need for an end user to consult the ops team to learn about what caused their pods to be preempted, without compromising PII. From
/triage accepted
Can I pick this up?
Yes, just type . Note that we are in code freeze, so this won't be merged until about a month from now.
/assign
PII? I know that as personally identifying information (ie, data about humans).
BTW, ideally we'd also document the change.
In a multi-tenant environment, pod names could be sensitive information.


Calculate sidecar containers resource in pod autoscaler. From 1.28 user could run Sidecar containers in , those sidecar containers remain running during the entire life of the Pod, it should be calculated in . If missing the sidecar resources, we will get a wrong pod resource utilization. We should also correct the related Cadvisor metrics.
/assign /sig autoscaling
/triage accepted /priority important-soon


We have a pod fake function in , but this is not a base util anymore, we have a lot of arguments, we can replace this with here /sig scheduling /kind cleanup /help This is an easy one to get start. Cleaner code for maintenance.
This request has been marked as needing help from a contributor. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : ### What would you like to be ? /kind cleanup /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-kind feature
I want to try this one, the problem we solve here is to replace the *v1.Pod with type PodWrapper {}? /assign
Replace with in
you mean replace ` in I check the code and didn't found out that we have makeBasePod in
I have tried it. But there will be error about cycle import, because package import package, where locate.
yes, I've tried it as well. guess I need to wait for kerthcet's response <img width="805" alt="image" src=""
Sorry, forgot to reply on this, I introduced another PR to help solve this import cycle, but waits for other maintainers' response. So generally, if that pr accepted, we can work on this one then. Thanks for the volunteer.
the import cycle is resolved, you can start your work now, thanks!
thanks for the ping!


remove and was introduced in to fix This is to overcome the situation that volume is to DSW at first and then deleted before it is to ASW, which causes mounted volume neither in DSW nor ASW and won't execute unmount/detach: The reason the volume cannot be to ASW in time is , which is . However, there is another PR also solved the issue. What it did is to check whether ASW already has the volume before deleting it from DSW, this makes sure that volume can be detached/unmounted after it is deleted from DSW. I suggest that we can keep the second solution and remove related code to make the reconstruction process and reconcile loop cleaner?
/sig storage
cc
Should I proceed with this issue? Your guidance is appreciated.
I think we'd better wait for the feedback from
cc
Sorry for a late reply. This is how behind we are with reviews :-(. Reading Now that is on by default, it reconstructs volumes first and always puts them to ASW, it should never happen that a reconstructed volume is in DSW and not in ASW, right? So we can remove easily. I would not be so sure without , because the code is 5 years old and who knows all the implications. You could be right though, it may not be necessary there too.
/triage accepted
/assign


We need to explore how DRA interacts with One likely outcome is that the DRA helper code watches for nodes with the special label and helps DRA drivers clean up. /sig node
/assign
/triage important-soon as this is a DRA Beta criteria and DRA is going to be graduated to Beta in 1.29
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /triage important-soon as this is a DRA Beta criteria and DRA is going to be graduated to Beta in 1.29 Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/triage accepted
Here's a test that was written for volumes:
Thank you. I’ve found it last week. Unfortunately kind image doesn’t have ssh, so stopping kubelet is not possible the way this test does. I’ve found another solution.
I was able to simulate non-graceful node shutdown by running . It turned out that after adding tent, all node pods are deleted by the GC controller and the resource claim controller successfully deallocated the resource. So, it looks like a non-graceful shutdown is handled correctly and no code changes are required. However, the fact that can potentially be called without calling should be explained in the DRA KEP, so that DRA plugin authors will be aware of this. I'll add an explanation of handling non-graceful node shoutdown to the and add a correspondent e2e test case to the DRA e2e test suite.
KEP update PR: E2E PR:


transformation expandless | 14m1s Failed; === RUN TestEncryptionConfigHotReload TestEncryptionConfigHotReload When I see this flake, 10-09 is the first one.(But the job history in the testgrid only started from 10-03) some PR that may be related: /sig auth /cc
/cc
/triage accepted
/cc failed two times in recent 3 runs:
/reopen as of this is still .
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


4 other Test Failures in ci-crio-cgroupv1-node-e2e-features. Test has failed 61 times (no passing result found). Earliest failure found at commit/ (Thu Oct 19 21:41 PDT). /cc { failed [FAILED] Error creating Pod: pods "pod-dualstack-host-ips9ddb9810-134c-4d35-a0c6-" is forbidden: violates PodSecurity "restricted:latest": host namespaces (hostNetwork=true) In [It] at: @ 10/23/23 01:38:06.024 } No response_ /sig network node
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
is also failing on these tests
the test has a bug, it checks the contrary
, let's start over, 1 commit for the e2e test, other for the promotion, and let's validate everything is alright before merging


we just log that the user should install etcd, which is impractical in an ephemeral container we actually already call installing etcd in the verify script, but we don't export it into the current PATH for the process so it doesn't work ... ref: /sig api-machinery /assign /kind bug
/triage accepted
for one possible fix
/sig etcd


Runs k8s with master golang: test Failure was observed on Aug 25th, it is passing after that. However it can be reproduced with stress tool. No response No response It is failing on both ppc and x86 architectures. Steps to reproduce: up golang tip version. clone kubernetes master /sig testing
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
it is a sig node test, sig testing charter is not about doing or maintain tests, we need to remove that old mindset of having a QA department, that is not how software development works or should work :) Besides, thanks for spotting it and submittn a patch /sig node /remove-sig testing
Flaky in
merging this PR will fix the issue


The swap fedora job is failing consistently for OOM. For a while. It looks like OOM is not actually killing these pods when swap is enabled. I did not see any examples of this failing without swap enabled so I think this is a real issue. There is a failure in OOMKilled as part of the serial fedora job but that is not related to this. This is part of and should be required for promotion to Beta2. /sig node
/triage accepted /priority important-longterm


pkg/registry/core/persistentvolume/storage TestUpdateStatus unknown No response /sig storage
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/priority important-soon as it failed in master blocking


When unit tests in are invoked with 1, it fails Unit tests should work for any . This is important for troubleshooting flaking tests. Discovered when I tried to add parallel execution as described in master N/A N/A N/A N/A N/A
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig network /area kube-proxy
/cc


Created a RWX volume and mounted it on a pod having readOnly flag set to true. While the volume provisioning happened successfully, we did observe some warning message on describing the pod: Warning FileSystemResizeFailed 27m kubelet MountVolume.NodeExpandVolume failed for volume "pvc--a6ba-49f4-a560-" requested read-only file system It seems like this message is being triggered from here: But we never did an expand volume call, so we should not be seeing warning message. It can be confusing to see this warning message. The warning message should not be displayed when expand volume was not even called. Create a RWX volume and mount it on a pod with readOnly flag set to true. Wait for a few minutes and describe the pod. No response_ 28 vSphere <details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage
/assign


When running subcommand like this: a warning message will display: which is incorrect in mode. It is only correct in mode. Warning message to only display when flag is passed in. Run No response <details</details<details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cli


! Please also see /kind failing-test <!-- If this is a flake, please add: /kind flake --<!-- Please assign a SIG using: /sig SIG-NAME --/sig testing
cc
slack thread:
/triage accepted
got one good run!


/kind failing-test /sig node
/assign
/triage accepted
Seems to me that problem is related with test-infra config ? I noticed Checking kubelet logs there are complains about kubelet not finding configuration file. Even if this job run in AWS it is defined under ( ) and not under aws subfolder. Performing vimdiff periodic- ./aws/periodic-eks- i notice some variables are missing, but i don't have permissions to re-run the job to be able to debug more on this. Test code for InPlacePodVerticalScaling doesn't seem to be the root cause here, if we could increase the kubelet log level and perform a rerun might be helpful to debug more. Checking containerd logs , it seems that a container failed to stop, this might explain why pod resize failed due to lack of resources)
These two tests are permanent failing on this job
True, but looking deeper it seems for the first test it fails ( unable to resize there are no available resources ) because a previous pod has not been deleted according to containerd logs . For second error it seems to be a config error , it complains that it could not login. I believe something is wrong with the setup ( repeated kubelet config errors in kubelet logs )
/sig storage
More information for the container delete error, it belongs to mainpod-with-poststart-http-hookrestartable-init-container-lifecycle-hook. I will try to check if InPlacePodVerticalScaling test can be improved to cover this corner case. Please note, Garbage collect error happens before the InPlacePodVerticalScaling test failure, perhaps this is the root cause of the failure in InPlacePodVerticalScaling error. Perhaps we should update Issue title and include the garbage collect error for this issue? ! Source: / . Please search for


,%20google-gce all I can't say , but at least a few weeks ,%20google-gce No response No response /sig network
/assign
/triage accepted


The following api endpoints are currently in - As each group has no stable endpoints they are reported in as promoted to stable without tests. The current logic for endpoints will address this issue.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig architecture /area test


After restart Kubelet, node will become notReady in first kubelet update period. node condition is : in kubelet's logs: when go in to next period (default 10s), node status resume to Ready. Restart Kubelet should not make node status change to notReady. Restart Kubelet, and watch node status for second The root case maybe here: When kubelet start, it will start two go routine, one detect runtime status, one check node status and update to APIServer; detect runtime status: in it will update : in it will check runtimeStatus include : is one of the notReady setter func: So if nodestatus execute before , kubelet will update a node status with notReady condition: In kubelet starting, both go routine and will be called before , so the first node status is notReady. In our cluster is v1.22.17, but I belive it can reproduce in latest version. <details</details<details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /sig node Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I can reproduce this issue in v1.30 When the container runtime is healthy, the kubelet should not report with the reason We called in the function to update the status of the container runtime, but we did not do this in , We only need to execute function once before and , just like we did in the function, than this problem can be avoided. But in , seems to have some different opinions...
Yeah, in our env, we modify kubelet the same way as you said, to let it execute once before , it worked. I can submit a PR later.
I may be missing something but what happens if something has changed that caused the kubelet to not be ready ... imagine, the Network plugin is not working ... during the initial period until the check changes the state pods will be scheduled on the node and will fail .... I think that we are operating the assumption that this is a restart and during that time nothing changed, but is this a safe assumption or can we guarantee somehow that nothing has changed that could impact the node readiness state?
In this issue, what I found is that nothing has changed, every time I restart kubelet, node will become notReady only in first sync period, in next period it will be ready; It's not an assumption. or other problem of course will cause node notReady, but they are not in this issue's scope, in this scope, notReady reason is message, have not relationship with other problem;
ok, I misread this issue sorry, so it is not to blindly set the node to ready, is just to perform the runtime check before the other checks ... I think you both are right ... actually it seems that if wins the race then there is no problem, righ ?
Yes, but and run in different goroutines, so we cannot guarantee that will complete faster. So it seems like a good choice to execute before running , or do you have better suggestions?
We need to put the goroutines in the right order to make the startup more predictable
Yes, if execute very fast, may there is no problem, because in it will execute . I have test in v1.28 (which code in fastStatusUpdateOnce almost same as 1.30 ) , it will not always reproduce, but still occurr some times. In 1.22, will sleep for about 100ms before everything, it can be reproduced everytimes.
Arranging the goroutines in order can solve our problems in most cases, but in the method, we need to call the containerRuntime api to get the status. I am worried that its response will not be timely enough in special scenarios, there may be a risk that will complete faster.
moving will execute first
I think I must not have expressed clearly. What I'm worried about is that under special circumstances, calls the API of the container runtime and cannot return immediately (not sure if this situation really exists). Even if the method is executed first, it will still complete later than :)
Add some detail: in it call runtime api to check runtime status, then set variable . in , it will check variable, if is nil, this issue's problem will occurr. Even if goroutine call firstly, call container runtime check will still need some time, it can't guarantee that when firstly call, is setted.
are you suggesting to lock syncNodeStatus on updateRuntimeUp? I'm afraid of some corner cases we can hit ... if the container runtime does not return do we block forever?
Container runtime API call have timeout.
Can you try this on a newer version? 1.22 is out of support and we have made changes to node readiness since. /triage needs-information
I have tried in 1.28, it can reproduce, can reproduce in 1.30.
/remove-triage needs-information


What happened: If you use to test whether a user can approve certificate signing requests, you get a response indicating that is not a valid verb An example of the verb being used can be seen What you expected to happen: should be recognized as a valid verb on objects. How to reproduce it (as minimally and precisely as possible): Run Anything else we need to know?: I think that needs to be to the kubectl auth can-i code Environment: Kubernetes client and server versions (use ): Cloud provider or hardware configuration: OS (e.g: ):
This issue is currently awaiting triage. SIG CLI takes a lead on issue triage for this repo, but any Kubernetes member can accept issues by applying the label. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I don't think is a valid verb for kubectl auth command. Valid verbs are in
so in that line of code there are a number of verbs which are only usuable on specific objects for example which works on users. AFAIK the point of that line is to inform kubectl what the valid verbs are on any object in Kubernetes, so that it knows what's possible when users use the command. the verb is valid on certificate objects as shown in the docs example I listed. Without being present in that line of code, will incorrectly state that there is no verb in Kubernetes.
I tested it, seems working when we add in that list. But the question is should we add it or not?
There have been cases where Kubectl has had additions to handle non-standard resources/verbs in the past (e.g. I raised a couple of years back). AFAIK the explicit list of verbs here is designed to address this point, it's just not got approve in it. You could ask SIG-Auth about it, I'm sure they'd have more info.
/transfer kubenertes /sig auth Decision can be made by sig-auth
Something went wrong or the destination repo kubernetes/kubenertes does not exist. <detailsIn response to : /transfer kubenertes /sig auth Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/transfer kubernetes/kubernetes
Something went wrong or the destination repo kubernetes/kubernetes/kubernetes does not exist. <detailsIn response to : /transfer kubernetes/kubernetes Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/transfer kubernetes
/triage accepted
This will work even without adding verb there, adding verb just removes the warning. Submitted a PR to add that. /assign


Everything 2024-05-24 No response /sig testing
/sig node
kubelet logs dmesg:
Seems a regression in
/cc
/triage accepted /priority important-longterm


master-blocking: ci-kubernetes-unit It starts flaking today 30-05-2024 The test fail is This failure may be caused by the error message's order sensitivity. The error message's order of Resource Classes versus is different. N/A /sig node /sig scheduling
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-sig testing /sig scheduling (sig testing doesn't own all of the unit tests, just test frameworks, tooling, CI infra, etc) This appears to be a flake in scheduler unit tests which would be sig scheduling ( )
.. and it appears to be DRA specific subset of scheduling so /sig node as well
FYI - looks like this is happening every 3-6 hours give or take. Yesterday it was every six hours and today it seems to be every 3-4 hours. Looking into it a bit more, it looks like it may just be an issue with a connection via the test while it's trying to hit the server. I don't think the test itself is failing, I think the connection to the server is failing. ! CC
Hey All - hope you're well. Michael here from the Release Signal team. Just wanted to touch base and make sure we're good to go for the cut on the 4th. CC /sig-node


Kubelet soft admission handlers were created as a way for the Kubelet to block pods that cannot be run for reasons that can be resolved. In practice, this mechanism was only ever used by AppArmor (). Problems with soft-admission rejection: is only used by AppArmor, and only for situations that cannot realistically be resolved (host or build does not support apparmor) held in the blocked state must be manually cleaned up. While there they hold reserved capacity from the scheduler and continually retry syncpod. other features work this way - usually Kubelet hard rejects pods that it knows it cannot run, or the runtime fails to start the pod/container. This was originally to prevent controllers from continuously recreating pods that cannot run, but that is a separate problem that should be approached globally, rather than with this 1-off AppArmor solution. I propose we remove Kubelet soft admission entirely, and move the AppArmor admission handler to the hard-admission list. /sig node /milestone v1.31 /kind bug
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
We have a different, alpha mechanism: . We could change AppArmor, seccomp and other mechanisms to work either with PodSchedulingContext or its eventual replacement. IMO we only need to sketch out an approach and see that it looks viable; we don't have to block graduation for AppArmor support on a fleshed-out design.
From my admittedly limited PoV I'm not against removing soft admission, and I got no opinions about AppArmor specifically. However this seems to connected to the recent conversation we're having in WG serving and SIG-node about retriable vs unretriable errors so I think we would need some coordination cc
+1
My understanding is that after this PR: soft admission just doesn't make sense any longer.
Hello ! Just a reminder, the code freeze is starting (about from now), and while there is still plenty of time, we want to ensure that each PR has a chance to be merged on time. As the issue is tagged for , is it still planned for this release?
Discussed in sig-node, decided to move forward with this (move AppArmor admission to the hard admission set, remove soft-admission code)
/assign
- do you want me to incorporate this change into


pull-kubernetes-unit TestStoreListResourceVersion in seems like July 7th 2024: ! Don't know, here is the log: No response /sig
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig api-machinery
See also
/sig etcd
/assign /cc
Bisected this to cc
I'm not sure if this is the proper way to fix it, but setting a limit in the ListOptions deflakes the test. The flakiness comes from the fact that the cache is sometimes not initialized in time which causes it to return a "too many requests". Perhaps in tests we could wait till the cache is ready in order to proceed?
Right, setting a limit when a resource version is set would delegate the list to etcd when cache is not initialized based on the KEP description. It is due to this type of request is not that costly to etcd. Looks like it is exactly how the wait cache ready is in See below So I think we should implement this same thing in when initializing the .
Agree - waiting for cacher to be initialized when creating it here is a better fix for it:
I thought about this as well, but waiting on is not possible outside of the cacher module, since it is a private field. I'm thinking to either add an option to cacherstorage.Config and make wait or add a new public method to . Do you guys have a better idea? Thanks.
Exposing Wait(context) method sounds fine
I have pushed the changes we discussed. Thank you all for your guidance.


The PR above moved everything related to into because that was where most of the code already was, but didn't renamed the files to avoid a larger diff and merge conflicts. Once is merged, we can do the renaming. Code clarity. /sig node
/triage accepted /priority important-soon
/assign
a PR can be created for this now.
will do


/kind failing-test <!-- If this is a flake, please add: /kind flake --<!-- Please assign a SIG using: /sig SIG-NAME --
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
looks like started a few days ago ! /sig cluster-lifecycle
cc
fix was sent earlier
/sig cluster-lifecycle
thanks :) i saw it yesterday and forgot to raise the issue. i should have checked the PR queue :)


ci-cgroupv1-containerd-node-arm64-e2e-serial-ec2-eks ci-cgroupv1-containerd-node-e2e-serial-ec2-eks It has been failing consistently for a long time. No response /sig node
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc
There's another bug that was filed a long time ago. But there's no update on it and the same test in gce jobs seem to be green now.
cc


ci-kubernetes-unit TestPickBestStrategy shows the jobs are success. but in test-grid and ci report, it shows a failure. After This failed the {Failed;Failed;Failed;Failed;Failed;Failed;Failed; === RUN TestPickBestStrategy === RUN TestPickBestStrategy/singlecandidate,singlepreferredstrategy === RUN TestPickBestStrategy/multiplecandidates,differentpreferredstrategiesshouldfail === RUN TestPickBestStrategy/multiplecandidates,multipleresolvedpreferredstrategy No response_ /sig testing
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc
Thanks for opening this Paco, just noticed.
cc
! It is weird as the test passed indeed and I cannot reproduce it.
Fixed in


This issue was supposed to have fixed the issue where endpoint stay out of sync I also wrote a small script which would get me the out of sync endpoints compared to the endpointslices I expect the endpoints to eventually sync and reflect the most upto date information. I have just deployed the newer patch to our cluster and that has resulted in endpoints never ending up being updated if the status goes out of sync. No response Client Version: v1.29.7 Kustomize Version: v5.0.4-0.- Server Version: v1.29.7 <details</detailsalmalinux-9 <details</detailscri-o No response
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig network
from the logs, endpoint controller was retrying test1/test-qa in the expected way. It has retried 5 times and the next retry should be after 320ms, which should long enough for the informer cache to get the updated endpoints. If there is no more logs related to the failure, I think it may have synced it successfully. The python script used to detect the failure seems capturing diff between endpoints and endpointslice at a moment, but that could happen if there is any Pod change in the cluster, and technically the script is comparing endpoints of one moment with endpointslice of another moment. So did you see the endpoints not being synced after endpoint controller stopped retrying it?
Yeah i still see it and see similar messages continuously. Let me post some additional log messages after the one i posted above.
Observed the same issue in 1.29 and 1.30. I keep seeing the same error then new pods were behind a service. And the endpoint never catch up after all pods are settled.
could you provide complete log of kube-controller-manager and the yamls of the related endpoints and endpointslice?
/assign Are you able to triage this since you are already looking at this?
Yes i can get that over by EOD today
Sure, I'm waiting for more logs and the content of the endpoints to understand how it happened. Currently there are only retry logs which is expected in some cases.
We found a customer that triggered an edge case in your change of , which will cause Endpoints never updated. They have a mutatingWebhook on Endpoints object that automatically injects a label, let's say . syncServices, the endpoints controller , and they won't match since Endpoints have the extra "dummy-k: value". endpoints controller will by removing the label. mutatingWebhook will automatically adds back the label as part of the update request. From the APIServer's POV, the Endpoints object don't have any changes, thus the update request will succeed but returns the same resourceVersion as before. change in will mark that resourceVersion as stale. future pods events won't be handled correctly due to the step didn't result in a new resourceVersion. I think we should fix this by check the resulting resourceVersion from the update request, and only mark the old resourceVersion as stale when a new resourceVersion is generated. Not sure whether this is the same case others in this thread encountered :D
thanks for explaining the use case and the analysis. I did analyze the code after the issue was opened and suspected the only possibility is the returned resource version somehow equals to the one in the request but didn't know how it could happen after it has checked the desired endpoints and the current endpoints are different: Now I understand this could happen when the endpoints appear different on client side but are mutated to be same on server side. Do you plan to create a PR since you have proposed a fix? could you confirm if this is also your case, i.e. there is a mutatingWebhook for endpoints objects that could make the endpoints different from what endpoints controller expects?
Yeah, i can do a fix for this
- can we assign to you?
Sure, I'm already assigned. Given that we've had three user reports on this issue, I think it's important to address it as soon as possible. Could you please let me know when you plan to create the PR? If you're currently occupied, I'm happy to handle it and submit the PR.
Sorry, were occupied by some internal incidents last week. Just created a WIP PR for it. Will test it in kops for this specific webhook scenario in the following two days and and remove the WIP flag
/assign


seen in this inside a like loop we execute a function that return a bool and an error. If we want to fail fast we return the error directly and assert on the output of the wait function. If we want to retry we log the error and return false nil /kind flake /kind bug /sig testing /sig network
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc since I think you this test but most probable the code is inherited and is a copy paste thing, but besides that, the change is to remove those there
/assign


The code says this (): While the comment says this (): So..., what happened? The feature is still Alpha or it is a Beta? Be consistent about the features we add or change. Read the code. No response 30.0 <details</details<details</details<details</details<details</details<details</details
According to the changelog: This feature has been demoted from Beta to Alpha. That is fine. We need to revise the code comment then.
/sig api-machinery (not sure who owns it per se)
not worry, i think this can be resolved with
I found same like this inconsistency between the comment and the actual feature gate definition for APIServingWithRoutine in due to this
I want to raise PR on this if need another PR separately
Looks like this comment should be updated to alpha based on . Thanks for catching. if you'd like to raise a PR to fix the comment that'd be great!
/triage accepted
sure I will raise


k8s component developers are using FQDNs as finalizers (e.g. or which triggers an error message that isn't particularly easy for an end user to understand: To a naive end user, a fully qualified domain name is a pretty safe thing that is unlikely to conflict with anything else (as long as it's only used by someone from within the domain that owns it, which is a reasonable assumption). If the finalizer has s inside it but no , the message should suggest that finalizers need a . Something like: Do whatever it is that triggered the warning in (this is probably simpler than installing cert-manager and asking for a certificate) introduced: <details</details<detailsGKE </details<details</details<details</details<details</details<details</details
/sig api-machinery
If thats ok I would like to work on that.
/assign
/triage accepted


[ ] two weeks soak end date : xxxx-xx-xx According to following APIsnoop query, there are six CSINode endpoints that are untested. Using a number of existing e2e test practices a new has been created to provide future Conformance coverage for the 7 endpoints. The e2e logs for this test are listed below. This query shows the following CSINode endpoints are hit within a short period of running this e2e test. If a test with these calls gets merged, test coverage will go up by 6 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/triage accepted