<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --When I ran , it failed with the following error: I expected the build to successfully complete, or to be warned that I am missing a required build dependency. If you don't have docker buildx installed, you will hit this error. However, that error is masked when you pass the full command line: Appears to have been introduced in Kubernetes version (use ): HEAD (1.22.x) Cloud provider or hardware configuration: N/A, local build OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig release
That's not the same as in the script, which should be setting which ungates this command. (we depended on this env for much longer in the project to use ) Stable docker versions should have buildx, at least behind this env. unfortunately the project has never validated a minimum docker version for building.
buildx as a build requirement was actually introduced at least as back as 1.21 as well, for some of the core images. we've just standardized on it more since then (including the linked commit), so if we do add some validation we should probably backport it. I think docker buildx has been available since 19.03?, but it appears your install is too old to have the flag in buildx. we could also try switching to , I'm not sure which version introduced .
No difference with the flag. But I am not running upstream Docker, I'm running Docker out of the Debian repos.
discussed on slack: seems debian packages don't have the buildx plugin , while docker-ce / docker desktop do. we should at least be checking for the command and throwing a clear error like we do with e.g. tar.
Filed to discuss the expectation users might have re: the docker PPAs vs the Debian repos and buildx availability.
/
also
Note: reverting fixes the issue, so I have been building locally with that as a carry patch as a workaround.
cc apparently there are docker packages that don't ship this, which maybe changes the equations a bit, see above.
/assign
tl;dr from isn't actually part of moby, but Docker Inc. ships it as part of their Docker packages in their PPA. It's a separate tool with a separate code repo.
would it be eligible for another package? otherwise we probably need to provide support for installing it at least to some extent, e.g. if you are on mac with ancient bash we point to installing with homebrew in the failure message that detects this (and we should also be detecting that the command is missing).
You can build it from source and install it as a docker plugin by following the instructions at I doubt it'll be part of any of the FOSS distro repos soon (although I'm happy to be proven wrong on that).
we don't have the yard stick that something we use should be part of FOSS distro repos. Just that it needs to be easily installable.
Seems like you can also use the release artifacts from github as well, you don't need their ppa and you won't need to build it either.
Here is an image I created to solve not having buildx in the official image
/assign
has been updated to use the official binary buildx image
AL2 also does not come with buildx built in and I landed in with the same issue.


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --pod with nodeSeletor edit pod, delete the nodeSeletor field report error with: ! The errors message should displayed as: Because the nodeSeletor is been deleted with this edit. Kubernetes version (use ):Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig api
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /sig api Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig api-machinery
/remove-sig api-machinery /sig apps
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
/remove-lifecycle stale
/remove-lifecycle rotten


/kind failing-test <!-- If this is a flake, please add: /kind flake --/sig node /cc
The failure should be caused by
/assign
/triage accepted /assign


I am getting a crash when trying to sort pod with kubectl on Not crashing and at least an error message <details<summaryCrash log</summary</details- Kubernetes version (use ): Cloud provider or hardware configuration: tried on kind latest and openshift 4.7 OS (e.g: ): debian sid Kernel (e.g. ): 5.11.x Install tools: NA Network plugin and version (if this is a network-related bug): NA Others: kubectl compiled from tip branch (as of this bug report) with GO version 1.16.5
/sig cli
Hey I'm not able to reproduce the crash, although I do get an error on running However, the following works: Could you try this and check if this crashes as well?
It definitively crashes it, did you test it in a repo where there is actual pod in there? that error would not show if there is nothing.
Yep Hmm, interesting. Will try to reproduce it again
Sorry, not sure how to remove the label but this is expected. When sorting if the type is not expected it will output errors using over which will logs to the FATAL, ERROR, WARNING, and INFO logs, including a stack trace of all running goroutines, then calls os.Exit(255). Arguments are handled in the manner of fmt.Printf; a newline is appended if missing.
Those labels are not set on the issue: <detailsIn response to : /remove-kind bug Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
You need to have multiple pods to reproduce this error. It won't happen with just one pod. For example: The error says the type can't be sorted, so that is the problem: As said, this error is generated intentionally. Here is the code: The problem stems from the fact that  has no way to return an error, so it logs a fatal error which os.Exits. It seems like the problem is that the error message is too verbose. One solution could be to change the code to only print the error message and exit, instead of using which is what is outputting the stack trace. It probably doesn't make sense to dump a stack trace for something that is a user error.


<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI -- (containerd) (crio) [sig-node] ResourceMetricsAPI [NodeFeature:ResourceMetrics] when querying /resource/metrics should report resource usage through the resource metrics api Ever since this commit was merged (containerd) (crio) We need to see what the commit is breaking this particular NodeFeatures test on both crio and containerd.
/sig node
/cc
/cc
/triage accepted
ah, let's use the RT's issue /close
Closing this issue. <detailsIn response to : ah, let's use the RT's issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --kubelet failed to start when specifying ,with following error: Try to start kubelet with : Kubernetes version (use ): master branch Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig node
Submitted PR to fix this issue.
/triage accepted /priority important-soon


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --I found that for with returns a very small value. The value should be almost the same as the one of . My minimal : Result: While the top three lines () are good, the float values in the bottom three lines () might be broken. Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig api-machinery
/assign
/cc /triage accepted


While investigating some slowness in mounting/unmounting operations of PVs (in this case EFS volumes) for pods, we observed errors like these: fortunately it does recover quickly on Kubelet’s next retry: Looking through the Kubelet code I found out that the issue is not related to the umount calls themselves, but rather a race condition inside where we call umount. Because of the way golang’s function (called inside ) is implemented, it for the child process to finish after it is . Since there isn't any locking on the child process across those function calls, it's possible that the child process exits event before the Wait() call starts. In such case the wait call returns with a error (). This error is more likely to happen especially if the gap between the Start() and Wait() calls is more (which could for e,g happen if the goroutine had CPU cycles snatched away from it for some time in between those calls). A similar issue also exists in the . While this is a self-recovering issue (because of Kubelet retries), it does seem to increase the pod startup/cleanup latency upto a 1s. Open to suggestions for a fix. The simplest one occurring to me is to treat the error as a special one on which we don't actually return an error for the mount/unmount calls. /sig storage /sig scalability /assign
Another thing is when analyzing a distribution of such errors across different nodes, we observed a skew. The top 10% of the nodes were seeing more than 90% of these errors. And these are nodes that are running hotter (w.r.t the pod churn) compared to others: ! Given this, I'm wondering if those errors are higher on those nodes just because of statistical significance or other causes like more GC pauses happening on those nodes.
/triage accepted
^ I've opened this PR to fix the Mount/Unmount functions in the file. In general we might want to make a similar fix to other invocations of exec.Run() and exec.CombinedOutput() within our codebase. But that'll be a bigger change. Happy to open another issue for it.
Sorry for opening the PR against wrong repo, here's the right one -


This shipped in 1.22 as deprecated in 1.22. As a hacky fix to unbreak tests on 1.23 dev, I bumped the deprecation policy one release: We will need to revert that before 1.23 ships and figure out a better solution. /kind cleanup /milestone v1.23 /sig node
/triage accepted
I want to try it. Can I take this issue?
I have explained how to fix this on PR:
/reopen reverted in as fix did not work
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
Hi, Im the bug triage shadow. Since we're in 1.23 milestone code freeze, could we move this issue to milestone 1.24?
this is critical for 1.23 and I think it just got forgotten... we need to get this fixed before test freeze
/priority important-soon


When calling GetAllocatableDevices, the API returns all the registered devices, regardless of their health status. Thing is, the - and in general doesn't make much sense to report as allocatable Unhealthy devices, as is also documented in the (see last paragraph) The API should not report unhealthy device. This means the reported device set can potentially change across multiple invocations of the API, as . a kube = 1.21 environment, enable the feature gate any device plugin - including sample device plugin sure 1+ devices are reported unhealthy the podresources GetAllocatableResources API, observe the unhealthy device is reported in the return value. N/A Kubernetes version (use ): 1.21 and beyond Cloud provider or hardware configuration: not relevant OS (e.g: ): any Kernel (e.g. ): not relevant Install tools: not relevant Network plugin and version (if this is a network-related bug): not relevant Others: not relevant
/sig node
/triage accepted


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --Following link on sig-network page () is broken Link should be working Kubernetes version (use ):1.21 Cloud provider or hardware configuration:None OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig-network
/sig network
/triage accepted


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --As described in issue , we found kubernetes have been running normally, but the control-plane nodes's metrics value of startTime is similar to the timestamp. Get the correct metrics of node As described in issue anything else we need to provide? As described in issue Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
/remove-sig node /sig instrumentation /assign
I think the issue is about Summary API having start time for node metric, but this timestamp is incorrect. This is node issue, not instrumentation. /sig node /remove-sig instrumentation /unassign
When discussing adding node start time metric to /metrics/resource endpoint I remember that SIG Node response to this was that there is no good way to define node start time. I don't think we will be able to provide correct value here. As summary API is technically deprecated, it makes no sense to fix it. We should warn users against using it and wait for removal of Summary API.
Okay, let's wait for the release of metrics-server v0.6.0
Do you know if this is fixed in metrics server?
/triage needs-information Can you please add info on name of timeserias and other steps to repro ti understand exactly what is needed here. Thank you!
in metrics server v0.6.0, we will use /metrics/resource Kubelet endpoint instead summary API
as issue described , the result is: for example : the startup time of the node test-us-west-1-control-plane-577lp is "startTime": "2021-08-03T19:33:50Z",, and timestamp reported by metrics is "2021-08-03T19:34:36Z", but as described, output from control plane node


I'm seeing a panic in kube apiserver (see stack trace below). It appears to be linked to the migration away from Hashicorp's LRU implementation to in The Hashicorp implementation is thread-safe, however the forked copy in utils is not. See the comment here Looking at the utils PR, I see there's actually a wrapper around the thread unsafe LRU cache, and that's being used here: The LRU isn't a lot of code and I think I've convinced myself the locking works fine, so not sure why the element Value is coming back as nil instead of *entry. Stack trace: Not to panic Not sure, but I see it in our CI fairly often Kubernetes version (use ): 1.22Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/cc
/sig api-machinery
if Add(key, nil) was called, that could set the value to nil in an existing object... does that happen?
The nil value we're hitting is on the linked list element, which is managed by the LRU code, not the user-facing value given in -- that gets saved The panic we hit is , where it tries to re-cast the LL element value back to the lru entry type. I don't see how that can happen.
The actual cache is perfectly happy with nil values:
From is mutating since it moves an element to the front when it's accessed. Thanks !
would you consider sending a PR to fix it? /assign /triage accepted
is the PR to fix


SSH'ed into a GKE node running on GCP and was welcomed by following MOTD message: a) Noticed download link is not working, I suspect this storage bucket is no longer used (?) b) Link to get Kuberntes source version does not work: This should be following (without ) Working links :) This is defined in the "configure-" file bellow: As this seems GCP specific (IIUC), I think filtering "-gke.501" will fix part a) For b) I am not sure what the correct download link should be Kubernetes version: Kubernetes v1.18.20-gke.501 Cloud provider: GCP OS: COS
Pull request to remove "-gke." from link to Github to make it valid. Regarding the wrong download link to GKE source code release, I noticed that: When running on GKE the link has (GCS storage bucket name) while in OOS Kubernetes it is . (So GKE is patched here) I think is depreciated and replaced by but in either case link don' work. It's unclear how one can download the source code.
/sig cloud-provider release for triagingnot sure if the link is maintained by release team /area provider/gcp
Was able to confirm the right download location ( GCS bucket) to download GKE source release. Submitted separate PR to fix the URL.
/assign /triage accepted
This is an related to the GKE product. I've filed this to the public :
/close
Closing this issue. <detailsIn response to : /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


N/A [sig-node] NodeProblemDetector: N/A https://prow- Not cgroupv2 compatible when access cpu and memory data
/sig node
Hi, can i work on this ?
Sure. Thanks
/assign
/triage accepted
/priority important-soon
/cc
are you still working on this? Were you able to setup environment to repro the test failure? Let us know if you need any help. Thanks!
Sorry for the delay, I followed the wrong guide and ended up getting stuck. But now I have managed to set it up. Please give me few more days to work on it.
Thanks Feel free to reach out here or on slack if you need any help, thanks for working on this!
/retitle Make nodeproblemdetector test cgroupv2 compatible
Failing job:
Testgrid:


<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --ci-cos-cgroupv2-containerd-node-e2e <!-- You can identify the SIG from the "prowjobconfigurl" on the testgrid dashboard for a test --<!-- Please see the deflaking doc () for more guidance! --/sig node
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : #### Which jobs are flaking: links to appreciated links to specific failures in spyglass appreciated </details
On cgroupv2, it looks like like the number of for cgroup is sometimes higher than is currently expected in the test. We currently expect this however, in a few runs of the test we've seen We may need to adjust this value similar as was done in for
/cc
yes, good idea. I think we need to adjust the PageFaults test as well. I've opened a PR:
/assign /triage accepted /area test /priority important-soon


pull-kubernetes-unit TestUpdatePodWithTerminatedPod 10/8 (when it was introduced in ) No response No response Running with shows immediate failures: /sig node
/cc
/assign I will try to fix this. At present, I found that adding before the status check will definitely trigger the error
Looks like it stopped flaking? Was this fixed? /triage accepted
If the test runs fast enough, it won't failed but it's not guaranteed. fixed in
It's still flaking
/assign
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
this started flaking more on 3/21 - adding to the milestone for investigation to make sure something didn't regress here
cc
/unassign /assign


As per the PRs kubernetes/kubernetes kubernetes/kubernetes, the defaulting behaviour of the pod exec parameters had been changed however in the code there is a comment where it is mentioned it defaults to True The comment about the default value must be removed and the openapi spec must be fixed Present in the file as comment and also in the OpenApiSpec swagger If this issue seems appropriate can I take it up please <details</details<details</details<details</details<details</details<details</details<details</details
/sig docs
/sig node
/assign
/assign
/help /triage accepted
This request has been marked as needing help from a contributor. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help /triage accepted </details
/kind documentation


runs in a container. Talos sets OOM score to -450 for the kubelet container and passes in the kubelet config. End result is that OOM score adj is -999 though. should have configured OOM score . Source code analysis. First, OOM score is set correctly from the config (flags): But then, in container manager, OOM score is reset to -999 via the following: This function should skip action if the PID is not running on the host: But the function looks to be confusing: When kubelet runs in a container with PID namespace shared with the host, kubelet's PID namespace will be equal to PID 1 namespace. If kubelet runs in a container with unshared PID namespace, kubelet's PID is 1 and its PID namespace is identical to PID 1 namespace. So how this function can return ever? Moreover, function is always setting OOM score to a hardcoded constant -999 ignoring configuation (). No response <details, reproduced with 1.22. </details<details</details<details</details<details</details<details</details<details</details
/sig node
/triage accepted
/assign
I don't think this method needs to change... I have make a draft PR for the default value to use the flag if present, but I would like more time to add some testing. Maybe you can have a look: ?
Thanks, this PR looks good to me, as it's going to set the OOM score to the configured value. And in our case kubelet runs in a container (not on the host, with proper OOM score via OCI spec), so this code should be even skipped, not sure how this can be addressed.
neither do I...
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale


This is an umbrella issue to track changes in the . It is required to update the guide based on the suggestion we've received so far after officially migrating , , and (ongoing migration). The guide can be improved by adding more relevant details about structured logging migration, examples, common mistakes, and easy navigation in the guide using appropriate referencing. We've observed common mistakes made by contributors while doing the migration. To avoid those "common mistakes", more examples are needed in the structured logging migration guide: approach could be manually looking at various review comments that we've received so far while performing migration of , , and and figure out common mistakes done by contributors. Add examples to avoid those mistakes, and enhance the guidelines present in addition to this, people involved in the previous migration process, either as approver/reviewer or as contributors, can suggest changes in this umbrella issue or open a pull request towards the necessary changes referencing this issue. As we advance, we need to make the migration guide suitable for both "contributors" and "users" of Kubernetes. Since slowly we are progressing and covering more components of Kubernetes, it is required to improve the guide so that both contributors and Kubernetes users can effectively use it.
Suggestions in form of review comments from previous migrations are here:
/sig instrumentation /wg structured-logging /area logging /priority important-longterm /kind documentation /cc
Do you intend to update the documentation yourself, or are you looking for others to help here?
Yes, 1) I'm looking for contributors and reviewers to suggest more changes (I may not be able to cover all the required changes) and I'll add those changes in the table above. 2) Contributors can submit PRs for these changes in the migration guide, I can definitely submit PRs if we see that the issue is not progressing enough
Thanks. I'm going to assign this to you for now then. /assign shivanshu1333
/assign
/triage accepted
/sig docs
/assign
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale


I was trying a scenario where liveness/readiness probe command run inside the container for long time('sleep 10 && exit 10') with shorter timeoutSecod value say 2 Sec. As expected container restarted because of timeout but when i check the events in pod describe it stated only unhealthy and in events actual reason for probe failure is not printed "liveness probe failed:<'no reason printed'". Without clear logs or pod events description it is very difficult to debug and understand what is the failure. Below is the example manifest file to reproduce the issue. apiVersion: v1 kind: Pod metadata: labels: test: liveness name: liveness-exec spec: containers: name: liveness image: args: /bin/sh -c touch /tmp/healthy; sleep 30; rm -rf /tmp/healthy; sleep 600 readinessProbe: exec: command: [/bin/sh, -c , 'sleep 10 && exit10'] initialDelaySeconds: 5 periodSeconds: 5 timeoutSeconds: 2 livenessProbe: exec: command: [/bin/sh, -c ,'sleep 5 && exit10'] initialDelaySeconds: 5 periodSeconds: 5 timeoutSeconds: 2 It would be better if actual reason or something for probe failure printed in pod describe events incase of liveness/readiness probe gets timeout. Example like below in pod describe events: Warning Unhealthy 3s (x2 over 8s) kubelet Liveness probe failed: "probe failed due to timeout " Warning Unhealthy 3s (x2 over 8s) kubelet Readiness probe failed: "probe failed due to timeout " ! It would be better if actual reason or something for probe failure printed in pod describe events incase of liveness/readiness probe gets timeout. Example like below in pod describe events: Warning Unhealthy 3s (x2 over 8s) kubelet Liveness probe failed: "probe failed due to timeout " Warning Unhealthy 3s (x2 over 8s) kubelet Readiness probe failed: "probe failed due to timeout " Below is the example manifest file to reproduce the issue. apiVersion: v1 kind: Pod metadata: labels: test: liveness name: liveness-exec spec: containers: name: liveness image: args: /bin/sh -c touch /tmp/healthy; sleep 30; rm -rf /tmp/healthy; sleep 600 readinessProbe: exec: command: [/bin/sh, -c , 'sleep 10 && exit10'] initialDelaySeconds: 5 periodSeconds: 5 timeoutSeconds: 2 livenessProbe: exec: command: [/bin/sh, -c ,'sleep 5 && exit10'] initialDelaySeconds: 5 periodSeconds: 5 timeoutSeconds: 2 Create a pod using above manifest file and do pod describe (kubectl describe pod <pod-name ) and check for the events. ! No response kubectl version Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2021-08-31T06:59:53Z", GoVersion:"go1.16.4", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2021-08-31T06:52:44Z", GoVersion:"go1.16.4", Compiler:"gc", Platform:"linux/amd64"} openstack <details</details<details</detailscontainerd No response
I tried to fix one similar issue in
/sig node
/triage accepted
can I work on this issue? Thanks very much. /assign
We've experienced this issue with clusters on v1.20. The for that version mention the following change: Since v1.20 is seemingly when exec probe timeouts started being enforced, would it be possible to have this fix for this issue backported to all minor versions since then? Thanks!
bump on that question
What do you think of? Thanks very much.


kubeadm-kinder-latest ci-kubernetes-e2e-kubeadm-kinder-latest.Overall E2EKubeadm suite.[sig-cluster-lifecycle] [area-kubeadm] proxy addon kube-proxy ServiceAccount should be bound to the system:node-proxier cluster role 2021-11-09 23:32 CET () E2EKubeadm suite: [sig-cluster-lifecycle] [area-kubeadm] proxy addon kube-proxy ServiceAccount should be bound to the system:node-proxier cluster role This issue seems to be linked to FYI: /cc /sig cluster-lifecycle /area kubeadm
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
fix PR is here, needs LGTM:
Also failing on Thanks !


ci-kubernetes-e2e-aks-engine-azure-master-windows-containerd-serial-slow Kubernetes e2e suite: [sig-windows] [Feature:Windows] Memory Limits [Serial] [Slow] Allocatable node memory should be equal to a calculated allocatable memory value expandmore this month No response_ /sig windows
/assign
/triage accepted /priority important-soon


integration-master ci-kubernetes-integration-master.OverallChanges 11/19/2021 23:54 CET see this as example to get information about the failure No response No response_
great catch /sig api-machinery /priority critical-urgent it seems all tests fail with how can this fail consistently only in periodics and not in presubmits? is an external dependency?
I would say this is the origin Server Side Strict Field Validation
presubmits only run short tests, periodics run all long ones... looking if there's a long test that needs fixup
fix in


ci-kubernetes-node-kubelet-serial-containerd: ci-kubernetes-node-kubelet-serial: (This is just an example, most of the tests appear to be failing) /sig node /kind failing-test /milestone v1.23 since test is failing in a release-blocking job /priority critical-urgent /assign /cc I believe this stems from
/triage accepted These failures look weird - I don't understand why dockershim is sometimes failing after restarting the kubelet. I guess we should double check that timings match the old GPU test (I think I might've tightened them a bit, bc they were passing when I ran the device suite on the same nodes). What's weirder is that they passed on my PR and all of a sudden broke after merging
Discussed at today - the easiest solution now that 1.23 has been branched is to punt these tests out of the serial lane so we can get it green again for 1.23, since any changes at this point must be backported and we would prefer that this is not release-blocking.
/milestone clear should address this; we won't get a fresh serial run for another ~6 hours so let's confirm next business day that this is temporarily fixed.
/remove-priority critical-urgent /priority important-longterm
can we keep it in the milestone until we get a verified green run?
CI is green today after skipping DevicePluginProbe tests .
/assign Maybe related to
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
Looks like serial has been stable for a while, closing: /close
Closing this issue. <detailsIn response to : Looks like serial has been stable for a while, closing: Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen it is referred to here: Tests are skipped today
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
did you have a chance to look at this? I remember you were close to understanding on what is going wrong here
oh welp, missed that
I was (past tense) but now I need to go back and look at my notes. If someone else wants to take over, be my guest, I've been quite busy with other tasks lately. Otherwise I'll get back to this!
So! I'm able to resume working in this area. So I'm testing last k/k master branch tip and running with : and the tests seems to be running and passing quite reliably! (5 times in a row seems like a promising start :) ) To move forward I suggest to: a explicit s in . This will make more evident we have a pending issue in our camp, that's the most egregious exception we have and it's confusing the skip in the config: - this will enable us to do focused work in this area a PR to remove the skip and iterate over it running CI until it's good enough I can take care of all the above. If CI is still failing, we need to explore possible missing/wrong configuration, but then other e2enode tests should be failing, not just this one.
I agree, this plan sounds good. Thank you!
xref:
started to fail also locally - we have baremetal lanes in RH - in the project CI they skip the relevant test. Investigating a possible common root cause
after a very long troubleshooting and quite few attempts, is ready for review


Flaked a couple of times on serial lane with the same error. Test is relatively new. This run only has the single test flake: /sig node /kind flake /cc
/assign
New recent failures: /triage accepted


/kind failing-test <!-- If this is a flake, please add: /kind flake --<!-- Please assign a SIG using: /sig SIG-NAME --
/sig cli
There may have two issues to fix: doesn't support run with assertion must check whether schema is nil first
/assign
/triage accepted


gce-master-scale-correctness Kubernetes e2e suite.[sig-network] Networking Granular Checks: Services should update nodePort: udp [Slow] ci-kubernetes-e2e-gce-scale-correctness.Overall kubetest.Test Kubernetes e2e suite.[sig-storage] CSI Volumes [Driver: csi-hostpath] [Testpattern: Dynamic PV (filesystem volmode)] volumeMode should not mount / map unused volumes in a pod [LinuxOnly] Kubernetes e2e suite.[sig-storage] CSI Volumes [Driver: csi-hostpath] [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral should support two pods which have the same volume definition Kubernetes e2e suite.[sig-network] Services should be able to update service type to NodePort listening on same port number but different protocols 12/1/2021 17:36:39 ET No response No response /sig storage /sig network
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/milestone v1.23 /priority important-soon
/assign aojea
/assign bowei
heh, what are the chances, two different test services took the same port The test was expecting to not receive any answer but the second service replies and makes the test fail link


During a Kubernetes deployment the connectivity to web services from Windows nodes is flapping. There are intermittent traffic delays and some errors observed on web services during the deployment. Post deployment the flapping is no longer observed. No flapping in connectivity during the deployment. Create a deployment with multiple services and monitor connectivity. No response <details20 </details<detailsAKS </details<details</details<details</details<details</details<details</details
/sig windows
/triage accepted /priority important-soon


We should make the change in PR API reference links to should use HTTPS, not HTTP Visit h and look at the IANA hyperlinks Once this fix is made, a PR to is needed to update the docs (by autogenerating them). This typically would be deferred until the next Kubernetes release. v1.23 not applicable not applicable not applicable not applicable not applicable
/sig docs
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I opened this issue so that someone else can open a new PR that is equivalent to .
/retitle Update links to IANA for Service Name and Transport Protocol Port Number Registry
I would like to work on this issue. May I assign it to myself?
Anyone - including you - is welcome to work on this issue! So long as you've you can open a pull request to propose a fix.
Thanks
/assign
I have made the PR. I hope this works. If not then will be happy to do the necessary changes.
Had to close the previous PR because of issues, here is the new one I made.


Condition PIDPressure is not set to true when the system thread number exceeds even it's reaching the limit. Condition PIDPressure is set to true correctly See cause below The bug is due to , which truncates any process number greater than because of type conversion. See kernel code . Reading it from should be working. v1.23.1 <detailsNA </details<details</details<details</details<details</details<details</details
/sig node
/triage accepted /priority important-longterm


There is no receive or close operation on channel of struct () . Thus, a goroutine leak happens when is executed (), such as when running . no goroutine leak run TestRunPositiveRegister() No response <details97d40890d00acf721ecabb8c9a6fec3b3234b74b and following versions </details<details</details<details</details<details</details<details</details<details</details
/area kubelet /sig node /assign I'd try to fix it
/sig node
The author has submitted a PR for this issue:
/triage accepted /assign /priority important-longterm
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
The issue has been marked as an important bug and triaged. Such issues are automatically marked as frozen when hitting the rotten state to avoid missing important bugs. Please send feedback to sig-contributor-experience at . /lifecycle frozen


I deployed a pod with and on a namesapace with a set with and elements and I get the error: The pod to be deployed Create a new namespace Create (using a manifest file named resource-quota-) and verify a named compute-resources Content of the resource-quota- file Create (using a manifest file named ) a pod named high-priority with the and for both and . Content of the file The pod is deployed correctly if I delete the <details</details<detailsaws </details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig Scheduling
/wg API Expression
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /wg API Expression Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/wg <API Expression
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /wg <API ExpressionInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/wg "API Expression"
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /wg "API Expression" Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/wg APIExpression
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /wg APIExpression Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/wg wg-api-expression
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /wg wg-api-expression Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
hey, v1.18.20 is out of support now, you should try with a newer version. Besides, I tried with an old version v1.18.19, but worked as well.
Are u using some mutating webhook which inject some sidecar into the pod without setting resources fields?
It's probably that, but IDK which WebHook is doing that. The feature you suggested would really help.
hi the fix pr has been merged, maybe u could use it to locate which container is causing the problem.
Thanks


In the implement of GCE mode WaitForAttach: when the runtime.GOOS = "windows" and it meets err , it should instead of . the code should be like this : None The modification is in order to match the code cosistant pattern. Any GCE Any Any Any Any
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage
/sig cloud-provider
/assign
/assign


/kind failing-test <!-- If this is a flake, please add: /kind flake --/kind flake <!-- Please assign a SIG using: /sig SIG-NAME --/sig api-machinery
/assign I want to take a try
2 but actual 0 till timeout resource services, expected 2, actual 0 error waiting or ResourceQuota status: timed out waiting for the condition time later, actual state became 2 Seems polls a ResourceQuota status with timeout 10s may be not enough, we should enlarge it into 300s, that is poll for 300s timeout instead of 10s.
/triage accepted


There are pre-defined tests patterns that specify "CSI Ephemeral" in combination with a non-default fstype, for example: This code then should set but currently doesn't: Not sure whether these patterns are used in Kubernetes. I ran into this with PMEM-CSI, which uses them. The effect is that all test cases using those patterns use the default fstype instead of ext4 or XFS. /sig storage Kubernetes 1.24 and several older versions. n/a No response No response No response No response_
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/assign /remove-lifecycle stale


Integration test timed out due to the following error The error appeared 748 times in the log. Looks like lots of churning happens during the apf configuration bootstrapping phase. Is there a race condition between when the type is being registered and the API calls? Stack trace: Integration job link: PR: I searched for this error in CI and this PR, looks like we only see this error in this PR so far. So I am hoping it has not introduced any flake. (maybe we need to let more time to pass to start seeing flakes) We should not see this error I saw it only in the PR mentioned.
/sig api-machinery cc
I see
The fact that it happens to a non-mandatory object implies that the object was received from the server, so the type must be in the server. The APF config-consuming controller does this:
Look for the output from this line:
Another PR with this failure:
the panic means the request scope FieldManager is nil that is initialized here: OpenAPIModels is set here: called from here: and openAPIModels is constructed here: it looks like it is possible for openapiconfig to be unset, which causes the openapi models to be nil, which causes apply to panic. I don't know if it is only the integration tests that are subject to that configuration. cc /priority important-soon (since this is now being exercised in bring-up of integration test servers)
if APF is creating/bootstrapping the objects without using apply, and then updating status using apply, that would explain how the object exists but apply fails
I suspect that APF is not successfully updating status at all in the integration tests with this server configuration
do you want me to revert , while we work on a solution?
I don't know how bad the test impact is. Work with / to determine if we should revert or fix forward
: I do not follow why you talked about server-side stuff then suggested the problem is in the client (a controller in the kube-apiserver) not supplying a fieldManager. The client does supply the fieldManager, for both creates and updates. See the following lines:
Different field manager. is set server-side in the endpoint handler based on the server configuration.
So if the problem is server-side, why did you suggest the problem is in the client?
Regarding the client, I looked in and found this line: actually several of the same form, all with the same fieldManager.
I don't think I did suggest that :) all of the setup and calls in are server-side
: please help me understand . That is about the client side. Are you saying that a client that creates an object using and later updates the object using --- with a non-empty fieldManager in both --- will run into this server-side scenario? If so, is that because this client behavior is wrong or because there is a bug server side?
That was in response to , trying to figure out how the object was created but could not be updated. Yes, when the server is not configured to enable openapi and therefore does not have apply enabled. Because the server is misconfigured (which, from what I can tell, is limited to some of the server instances started for integration tests)
Looking at and it is not obvious to me whether or how openapi could be disabled. Would that be by turning off the feature flag? FYI, the inclusion of "V3" in the flag makes me unsure what the server does about other versions. : I think you are saying that it is possible to disable openapi and that this in turn disables server-side apply. That seems kind of surprising to me. Does this mean that it is simply incorrect to run a client that does server-side apply with a server that has openapi disabled? Is this intended? Is it documented anywhere?
It's not intended as an externalized configuration option. integration tests have the ability to fiddle with many of the config struct options in the server that a kube-apiserver binary invocation could not. They intentionally use this to do things like inject specialized authorizers / authenticators for testing purposes, but also can unintentionally result in a test server that doesn't match a real API server invocation. I think this is an integration test server configuration problem, which should be fixed.
is correct that we have a misconfiguration in the integration tests and SSA is only enabled on a subset of integration tests. I'd like to understand the test failure conditions a bit better though. The can be consistently reproduced on a non SSA targeted test. (Eg: selected from one of the test failures). However, I'm seeing the panic message while the test is still green and passes locally. it's a bit bizarre what contributes to the flakiness of the test because the FieldManager panic can be reproduced 100% of the time.
The panic occurs when APF is updating status (which it just switched to do using SSA). The APF status update failure does not block the API requests the test cares about, so the test passes. I could see extra CPU usage contributing to flakiness.
Thanks for the explanation. Created to fix the integration test configurations.
/triage accepted
The integration test job started timing out dramatically last night (50% of runs) once merged. Since that was the only change made to integration tests recently, it seems related. I think we should revert (and therefore ) until the test impact is understood
revert of -
did this get fully reverted? we're still seeing panics in integration tests over this:
bumping priority and milestoning until we know we're not currently triggering panics
looks like it got reintroduced in , revert open in
For integration tests, maybe we can cheat and cache the OpenAPI data, so that it's not recomputed for each test?
:-/ maybe... but I really don't want to have non-real init paths in integration if at all possible
I wish we could go a little further in lazily initializing openapi (especially v2), so let's see what we can do here (after code-freeze probably)
if SSA is actively using the openapi, it being lazy won't help, right?
does the revert mean that we can remove milestone v1.24 from this issue?
Yes, cleared
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
/remove-lifecycle rotten
Taking a fresh look at this PR and wondering if it has been mostly fixed already? From comments my understanding is that this panic occurred due to two conditions: running as part of the integration test suite is asked to service an SSA request while are nil. The apiservice has logic to panic in this case. From the code path commented is nil when the server is created with is nil. APF worker (running alongside an integration test) issues an SSA request to the test apiserver. Since the apiserver is misconfigured it panics. This panic could happen sporadically because sometimes the tests finish before the APF worker is able to issue the request. As a brief check to see if this might be fixed, I ran the test mentioned was failing from SSA. I logged out the 's created and found that they all had managed fields. I saw this as an indication SSA was now functioning properly. Indeed, to start the test server the function calls . This function has a code path which unconditionally leads to the apiserver's being set. calls before running the returned apiserver: calls before using the returned config to create the returned apiserver: uses which sets : Thus, any test using should no longer be affected by this panic. The question then becomes: do all integration tests start their servers in this way? It's hard to say. I found a PR (, ) which refactors many occurrences of the old stanza: Into something using the new method: ` But it appears is still being used in a few integration tests: Should these occurrences be refactored to use too?
There was a ton of changes I did around that. But the last PR (already removing the old is already out for review): - FYI Thinking about it now, I guess I agree that the problem might have been 10 different way kube-apiserver was set up through different tests. I just don't know why it was causing flakes, not failing consistently...
There were definitely test setups that were not enabling openapi in the recent past. Two possibilities for flakes: Enabling openapi in all integration test setups was expensive and caused spurious timeouts The excessive numbers of background panics handled and logged in integration tests when the API server tried to handle SSA requests bogged down the CI machine (I think something like 40,000 panics with full stack traces each were getting logged)
i don't think that's the case. With the recent changes that I was doing as part of I think OpenAPI is already enabled for all tests having kube-apiserver (almost all) and that seems to work. But you remainded me that back then we also had an issue that we were running many more tests simulanuously, which could have been contributing to this problem significantly. The second reason is definitely possible contributor too. I guess my concern was that if some requests were panicing due to lack of OpenAPI, this means that these aren't really needed. But that actually might make sense - the fact that we weren't able to upgrade status of PriorityLevels/FlowSchemas shouldn't prevent non P&F-tests from succeeding, and that worked in P&F tests...
I think the panics were happening in async controllers, and no functionality blocked by the controllers intersected with the surface being tested, it just (maybe) caused log/perf issues
I believe has resolved the panic issue, I was able to verify with which changes the priority & fairness controller to use server side apply. Before merged, I analyzed the integration log: Integration test log file from an unrelated pr does not contain any trace of the panic as expected: On the other hand, the log file from integration job for (before merged) had plenty of these panics even though the integration job turned green: I wanted to check which tests produced these panics, so I did a breakdown by package Then I narrowed it down to the individual test(s): Before merged, the above tests used , and either is or is which results in to be nil. But a quick code search revealed that many more integration tests used , so why were the other tests not affected by the panic? I inspected the log of a test that was not affected by the panic, and grepped for , and found the following: In the above case, the apf controller had not advanced far enough, the controller was active for about , may be these tests finish quickly and signals the server to shutdown before the apf controller can send an APPLY command. On the other hand, log from a test that was affected by the panic shows that the apf controller was active for more than , and has lot more activities: I am assuming that integration tests have at least log level 4. Also, the size of the log file differed significantly, without panics around , and with panics over and changed all these tests to use and now integration log for shows no panic.
Great! (This also proves why eliminating gazilion different things of initializing stuff in favor of one makes sense). - FYI ^^ - let's merge the PR then - I just one question there, so let's wait for Jordan to respond it.
Can we just force the openapi/fieldmanager to be set at start-time now that SSA is GA, it should basically always be present, rather than fail when SSA is unavailable (that's probably wrong by the way). If that triggers any test flakiness, then we would solve these. We can wait for to be solved since it will almost certainly have a huge impact on this.
Has this been done Jeff?


Unit test in package flakes there is a race condition between the go routine that executes the request handler and the timeout go routine that waits for the handler to finish /sig api-machinery
/assign
I have a PR in the works
should fix it, I can easily reproduce it with the PR, the flake is not reproducible any more cc
/triage accepted


In </span /kind failing-test <!-- If this is a flake, please add: /kind flake --/sig node I believe the issue here is that the test is trying to get 5 restarts, but if we don't start the loop until after a few failures have occurred we start getting caught up in exponential backoff. The 6th backoff iteration is a 2m30s wait (i believe), so we exceed the default timeout of 5m for the test. We may need to change the test so that we are watching for pod status updates and catch the first one. HOWEVER - there's also some head of line blocking in pod status manager updates that I suspect might result in updates getting delayed, leading to timeout. Either way, the test has too little margin with exponential failure backoff to be reliable, and needs to be improved.
/triage accepted /assign


Since scheduler Policy API was total removed, we don't support any more. Remove unsupported configuration.
/sig scheduling /assign
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scheduling /cc
/sig scheduling (trying to tell the bots to remove )
The label(s) cannot be applied. These labels are supported: <detailsIn response to : /remove-needs sig /remove-label needs-sig Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
It's true scheduler doesn't quite use this field anymore, but I'm wondering if some other project like ClusterAutoscaler or simulator still consumes this via .
There's no place we can config this field now by configure file(before we can apply the change by policy api, but now it's removed). So it seems like unless we support this feature with CC, or this field is useless. To other projects, it's the same. When they are using legacy policy api, they can configure this field, but once they synced with the latest kubernetes, with the policy api removed, they can not configure this anymore. In this situation, we should add support in CC. I don't quite understand the relationship between k/k with k/autoscaling or others, I think they sync with k/k periodic.
As I mentioned above, it's still accessible via a framework runtime programmatically, and thus may be still useful. For example, upon a scheduling failure, a ClusterAutoscaler's implementation may choose to dryrun through all filter plugins to ensure a new provisioned node can fit the pod, instead of resolving the failure while failing at another latter filter plugin.
Oh, I haven't thought of configuration via programmatically, that saved my concern. But a little strange.
That's exactly what I meant. Given it's a public option in framework, and there is no alternative if we disabled it, it'd be good to check with potential consumers. For now I can think of CA ( and k-sigs/scheduler-simulator ( If they can confirm it's not needed in their usecases, I'm fine with removing .
To k/autoscaler, at least these codes are still there together with the legacy policy api. , looking forward to the response.
is not used in sigs/kube-scheduler-simulator.
this isn't used by cluster autoscaler, we can remove it.


In the K8s Conformance meeting of 6 April the testing of Node endpoints and their eligibility for conformance testing was discussed. This issue is created to get more community input on the topic. Should node endpoints be tested? Can we accualy test these endpoints with the current test framework /sig testing /sig architecture /area conformance
/triage accepted /cc
I have lost track of this issue where it have already been discussed. Let's continue to discuss further here. Apologies for the confusion /cc
Clayton We have historically focused conformance on the perceived end user experience deploying applications to a conformant Kubernetes cluster as being consistently successful. From that perspective, while these are actions that are indeed key to conformant behavior on the majority of distributions, the first two are not invoked by end users, only by kubelets and potentially administrators in rare cases. I would expect to be able to create a node object, delete it, mutate its status, and delete a portion of them on a conformant Kubernetes cluster as an admin - but I would not expect a similar end user to do so. If we can make a case that a conformant Kubernetes distribution would reasonably not allow any of these actions via the API, then we could make a case to omit them from conformance. I can see a small path to adding these to conformance under a "they may be necessary as administrative action on the vast majority of kubernetes clusters", but I can't necessarily bump that to 100% of Kube clusters, especially if we put the application deployer as the center of our story (an application deployer does not need to delete node collections in order to succeed at running an application on conformant kubernetes)
Addressed the question in SIG Node
Created to make the remaining node endpoints ineligible. readCoreV1NodeStatus replaceCoreV1NodeStatus deleteCoreV1CollectionNode


ci-kubernetes-e2e-gce-master-new-gci-kubectl-skew kubetest.SkewTest 2022-04-14 15:13:42 +0000 UTC The error message is It's complaining about "file contents" and not "url contents". And because this is a skew test I am assuming this is due to The fix might be to cherry pick this to 1.23. No response /sig cli
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I've opened the cherry pick at if we decide that's the right route. The other options I can think of are to revert this change on master or to add the old environment variable back in as a file path. /assign /priority critical-urgent
Tracked the root of this to This should start passing once that merges. The cherry pick will also fix. We really need to get skew tests running on PRs.


Please publish 3.5.4-0 image and update to to benefit from fix for , while still keeping v3.5.3 critical correctness fixes (, ). For recent PRs for update to etcd v3.5.3 see and
/sig release /sig api-machinery /kind bug
/cc /triage accepted
Hey, Can I work on this ? I'm a beginner, but all the good first issues are mostly assigned. Can you guide me on doing this ?
I am working on this. /assign
Hi I would like to work on Make sense to backport in v1.22, v1.23 & v1.24? Thanks!!
hi, it makes sense to me, but requires approval from dependency maintainers. i can only approve the kubeadm changes.
was this backported ?


/kind failing-test <!-- If this is a flake, please add: /kind flake --/kind flake the test case was add in /cc smarterclayton <!-- Please assign a SIG using: /sig SIG-NAME --/sig node
/assign
bumping priority, seeing this in PR presubmit runs
/triage accepted


Follow-up cleanup work for: Most of the information about this issue can be found in , where the work merged with a to change the function signature of to return a func() as well as an error. This issue will help to track that. All TODOs introduced in are resolved. N/A N/A <detailsN/A </details<detailsN/A </details<details</details<details</details<details</details<details</details
/sig api-machinery
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /priority low Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc /triage accepted
/priority important-longterm
/assign
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale


Un-deperecate the kubelet flag, as this option is typically set per-node similar to (which is not deprecated). Or more generally, consider not deprecating fields available in config that make sense to set per-node. I noticed this while trying to ensure KIND moves towards not setting any deprecated flags, this one is tricky because it's not a fixed value. For a lot of kubernetes users, kubeadm is used for bootstrapping, and kubeadm supports supplying additional kubelet flags when configuring each node, but the kubelet config is cluster-wide, created during setting up the first node and uploaded to a configmap then used for all nodes. In the future kubeadm may have the ability to do per-node kubelet config, but that effort will probably be involved and isn't staffed at the moment. technically you could run kubeadm in phases and then modify the config, but this is more complex to implement. It's possible this won't be necessary if kubeadm patches can be extended to kubelet which may be a smaller feature change for kubeadm, but personally I think being able to configure simple per-node strings with a flag override is desirable for implementing clusters, and we've had the flag for many years with no trouble ... The actual deprecation seems to be due to a blanket deprecation on things possible to set in config, and not targeted at this flag in particular. See discussion in
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
technically KIND doesn't strictly need this flag, FWIW, but we were asked by users to set it to make detection easier and have values more similar to "real" clusters. other projects in Kubernetes are also doing this, e.g. some of of the cluster api-provider implementations:
I think for 1.25 in kubeadm we can enable patching of kubeletconfuguration. This will allow us to use the current global kubletconfiguration for all nodes, but patches can act as the node instance specific config. Flags would still take precedence over patches and are the simpler UX. But if one day all flags are removed the above solution will still support the instance specific use case. I don't know whether some flags should be de-deprecated, but IIRC the plan to migrate all flags to kubeletconfiguration and not add more flags was still desirable by sig node.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
/remove-lifecycle rotten
/assign It seems to be reasonable. I opened to undeprecate it.


release-master-informing capg-conformance-main-ci-artifacts Kubernetes e2e suite.BeforeSuite 05-25 06:37 IST Failure Error waiting for all pods to be running and ready: 3 / 13 pods in namespace kube-system are NOT in RUNNING and READY state in 10m0s No response /sig network /sig node cc
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove sig-network /remove sig-node /unassign This seems a deployment problem ... on calico to be precise first point of contact for initial triage should be the job/sig owners ... this job also seems to be failing for a long time /sig cluster-lifecycle sig-cluster-lifecycle-cluster-api-alerts
/remove-sig network /remove-sig node
cc
will take a look /assign
Check the Calico version, there was a similar report that was fixed bumping the version. As a side not, I'm seeing a lot of problems related to Lots of projects/test/code were assuming the secret was created and is causing a lot of noise in different CIs
Hey thanks for the PR! The test still shows in the - will it take some time for the changes you made in the PR to reflect?
hello! those are not failures, they are flakys :/ afaik
Hey folks, this test has now started failing again on the dashboard. Should we open a new issue for it or is it okay to track it here itself? !
checking that
the capg conformance failed but now have flakys only
Thanks a ton


[X] test approval issue : [X] test pr : [x] two weeks soak end date : 15 Jul 2022 [x] test promotion pr : According to this APIsnoop query, there are still a single remaining Event endpoint which is untested. replaceCoreV1NamespacedEvent Using a number of existing e2e test practices a new has been created for this pod status endpoint. The e2e logs for this test are listed below. This query shows the event endpoints hit within a short period of running the e2e test. If a test with these calls gets merged, test coverage will go up by 1 point. This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/triage accepted
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Creating a destroying services test is a required test case in , but it's missing in the upstream e2e test. Since creating a destroying services test should work in both windows and linux cluster, it's worthy adding it to create a service and corresponding pod delete the service re-create the service load balancing rules for pods are recovered e2e tests for sig-network would have coverage for recover load balancing rules for pods after service is deleted and re-created. Reproducer is above No response <detailsmain </details<detailsany </details<details</details<details</details<details</details<details</details
/sig network testing
/triage accepted /assign


in this test the rootfs is limited to 5M and the pod is evicted by creating a 10M file with; But this fails if the fs doesn't support the fallocate syscall. The K8s e2e must not itself require support for features that K8s does not require. I am unsure why my overlay fs in containers doesn't support fallocate(2) while the fs on the node (ext4) does. I have queried on the slack. I build my Linux kernel myself but I can't find anything related. If the command is changed to; the test works on an fs that doesn't support fallocate(2). It may seem more "primitive" but it doesn't require a feature that K8s doesn't. <detailsNone </detailsNo response No response
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig testing
Interesting /Triage accepted


The project uses a lot of CRDs - in some cases almost 2,000. I do mean here, not custom resources. We've noticed that API server memory consumption balloons when many CRDs are installed - for example with ~1,900 CRDs we see the API server using up to ~8Gi RSS. A little profiling shows that around 1.5Gi of that memory is being used by the etcd3 client's logger. With Kubernetes built from with the CRDs at loaded I'm seeing the API server using up to ~7.5GiB RSS. I see it dip to ~5.5GiB, presumably when garbage collection runs. You can see taking up a ton of memory below. Zap is the logger used by the etcd client. Note that this is a sample, so it's actually using more than the indicated ~800MiB. ! I'm seeing the API server using up to ~6GiB RSS. I see it dip to ~4.5GiB, presumably due to GC runs. You can see no longer appears below. ! I would not expect a logger to be responsible for such a significant amount of the API server's memory consumption. You can get the above profile by: out . to build a kind node. to start a control plane for the above build. to allow anonymous pprof access the CRDs from I used to see the RSS but I imagine would show it too. See for details on how I disabled the logger to get the "without logger" numbers. No response <details</details<detailsN/A </details<details</details<details</details<details</details<details</details
/sig api-machinery
Since you looked at the code, and I'd rather not spend time doing that. Any chance you can tell if it's because it's being mis-used?
This is a scale dimension where the apiserver has historically . With the OpenAPI discovery performance being improved it looks like this might be a hint at what the next bottleneck might be? Any idea how many etcd clients were instantiated? I suspect it's quite a lot.
Sadly I didn't yet look deeply past turning it off. I was slightly optimistic that we might just be able to disable it without any meaningful impact. What I do know is that the offending code seems to be on the etcd3 client end - i.e. it defaults to creating its own logger when one isn't supplied. You can see that code at
I don't - will try to find out.
The zap logs might tell you if you have the data. If my memory of the code is right, you should get 1 etcd client per CRD right now (plus some for builtin types).
/triage accepted
Based on my read of the code, you're right. I see roughly: All of /apis is handled by a crdHandler When a request comes in it gets an instance of the crd (from an informer) Gets serving info from a map of CRDs - if there is none it... Calls customresource.NewStorage which creates a genericregistry.Store Calls its CompleteWithOptions method That calls its “Decorator”, which is a genericregistry.StorageWithCacher That calls generic.NewRawStorage which just wraps factory.Create factory.Create calls newETCD3Storage which calls newETCD3Client I'm experiencing technical difficulties getting lsof running in my kind container but /proc/pid/net confirms we have a connection to etcd per CRD: Note that 1,937 - 60 = 1,877 - the number of CRDs we have loaded (the 1,878 above includes the header line).


The gets very long testing mulitple functions in such as , and . These checks should be performed in dedicated Test functions using table for different scenarios. Target functions and are being as part of The remaining checks should be migrated to these functions. Also, a test function should be to migrate the remaining tests.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig apps
/assign
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
Any progress on this? I'm working on and I noticed that has relatively low coverage so I was going to work on getting the coverage up.
I created that is somewhat related to this.
no progess. go ahead please
/assign


When enable apiserver tracing, the webhook request spans only show spanName as /, mostly because use http request operations as default span name, which is hard to understand. ! Following the open-telemetry standard of spanName, it's recommended to use as spanName. ! SpanName of webhook request should be , like ! APIServerTracing feature gates of apiserver a pod tracing dashboard No response <details18 </details<detailsNone </details<details</details<details</details<details</details<details</details
I make a fix for this.
/sig api-machinery instrumentation
Do you know if this is fixed in later versions of opentelemetry?
/triage accepted /assign
Looks like might be related
not yet, I think we can fix this in otel upstream, but it's hard to backport to otel , which is the version of otel used in Kubernetes
Good point. It think fixing it here is a good idea in that case.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
could I take on this issue?
yes. Feel free to work on this.
! I test locally by 1.29 kubernetes and start a sample webhook, looks like the client span has already conforms to semantic conventions , which should be in the form of "HTTP {METHODNAME}"
! For apiserver as a server, now the spanName is still KubernetesAPI, which I think we could enhance it into "HTTP get_account/{accountId}" format
After digging for a while, I found one solution to implement this, I draft this idea in , could you help review about whether this is a correct way to implement it?


This one involves systemd, DNS, musl, and Kubelet v1.25 Starting in v1.25.0, pods running on Kubelets on certain nodes with a FQDN hostname (more on that later) will not be able to resolve DNS queries if they're using musl (e.g. alpine:3.16.2, busybox, etc). Our story starts with systemd. Back in 2020, systemd a "search ." in stub resolvers if it detected the hostname to be a FQDN (to solve a separate problem). Check for a static hostname with . In , a change was made to Kubelet to propagate that "search ." into pods (to solve a separate DNS problem). That change landed in and appears in . Unfortunatey, musl based DNS resolvers struggle with this "search ." EDIT: details in musl-based DNS resolvers (depite their other quirks) should continue to function To see if you have a suitable repro node, check your (likely a symlink to . Next, start a container (any CNI) that makes some use of musl resolvers. Confirm that starting in Kubelet v1.25.0, the container now includes the "search ." if it was present on the host. Try using musl-based DNS resolution. Note that uses its own DNS resolver, its not a reproducer. Note also that pod-to-pod traffic and outbound masquerade are fine too. This is not a masquerade problem. Edit the in the container to remove the "." and your musl problems instantly disappear. Add the dot and they're back. I've been spelunking for a while. I can confirm this is not related to a specific CNI - flannel, Calico, and Cilium all reproduce the issue. Not networking related OpenShift is not affected. In 2021, RedHat folks noticed DNS "breaks" and reverted the systemd change on their OS cc <details</details<detailsThis isn't cloud provider specific. However, the most likely cases where folks use FQDN's are on-prem imo. So you're likely to see reports originating from there. </details<details</details<detailsThis was found while preparing Typhoon's v1.25.0 release of Kubernetes v1.25.0 </details<detailscontainerd 1.6.6 </details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
Could either revert (which propagates "search ." into containers) or find a different solution to the empty "search " problem it was trying to solve (does OpenShift even need that fix if they already ? cc A workaournd to problem is to configure hosts to strip "search ." from ala
has a great packet capture and this has been reported to musl
/sig network /cc
impressive investigation in the linked issue , that seems to conclude that is a muslc bug, since it happens outside of Kubernetes too. Just reverting the PR in Kubernetes doesn't looks right, it fixed a known bug ... and this is really a muslc bug .... maybe always removing the single dot from ? I think that for Kubernetes we should understand first what does and means and its implications, maybe some developer can help? you know someone?
Its discussed by the musl folks what means. It does not sound trivial to implement on the musl side, its a new case they did not anticipate supporting. Unfortunately this is subtle, and folks using alpine or related ecosystem things will get bitten and eventually end up here. I agree its a musl issue, but Kubelet now propagating triggers this I think its worth noting that did not set out to propagate , it was merely trying to prevent an empty . There may be better ways to achive that.
IIUC means "try the input name without any suffix". Resolvers will do this ANYWAY, but it's implementation-defined whether they do it first or last or in parallel, so allowing lets node owners be explicit (though, curiously, it seems at least glibc may try the same name AGAIN at the end of the search list). This is pretty clearly a bug in MUSL, but I wonder if, instead of , we should have treated as rather than or the errant ? It nullifies the control that provides, but it wouldn't tickle musl like this.
/cc I foresee a new wave of bug reports caused because of this muslc bug , better to keep CoreDNS friends in the loop
The patch for this got merged for the upcoming 1.25.1 and 1.26.


adds new version for in 1.26. For 1.26, we are setting the storage version to because vN-1 understands that level. In 1.27, we should change the storage version from to .
/sig api-machinery
/cc
/cc
/triage accepted
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:


Upgraded from Kubernetes v1.25.0 to v1.25.1. Standard metrics-server deployment now causes the following message to appear in the kube-apiserver logs on a recurring basis: Adding a quick hack debug Infof print shows: It appears that the apiserver is sending an If-None-Match cache control header when refreshing the openapi spec, and the metrics-server service is correctly responding with a 304 Not Modified. No errors logged; proper refresh of OpenAPI specs Deploy metrics-server, or any other service that uses apiserver aggregation and honors caching headers such as , to a Kubernetes 1.25.1 cluster. made bad assumptions; not all HTTP 3XX responses are redirects. It should probably only reject the response if it is 300-399 AND the header is set. <details</detailsnone <details</detailsk3s containerd flannel
Taking a stab at some tags here... /cc /cc /cc /area apiserver /sig api-machinery
thanks for the report, looks like this should be /priority critical-urgent /kind regression cc since this requires a respin of a final 1.22 patch release :-/
Fix open in , will get cherry-picks open to start greening up CI
/triage accepted
I need to make a redirect to the external URL but it's not allowed :(


When a Pod admission fails due to node-pressure condition then the Pod's looks like this: "Pod The node had condition: [DiskPressure]." The "Pod The node..." reads awkward. A message like: "The node had condition: [DiskPressure].". a job with a pod on a single node cluster, when the node is under disk pressure (then it is tainted with ). The Pod gets stuck in the phase. the node by kubectl command line: Pod is scheduled but admission by Kubelet fails as the taint is re- shortly after its manual removal. Then, the status of the Pod contains the awkward message. No response <details</details<detailsReproducible on kind </details<details</details<details</details<details</details<details</details
/sig node /triage accepted /priority backlog
The first part () is returned here: . I wonder where the is being appended to the beginning. It's my first try to contribute, so I am a little bit lost.
I think this is a candidate place which might be responsible for the prefix (but haven't checked): I would suggest you to try to reproduce the issue locally first then experiment with modifying the code.
/assign
thanks, I've managed to reproduce and used Delve to make sure who's responsible. I opened the changing the message. If it's accepted it will change to .