Question: return nil } <ins> // InspectFeatureGates allows setting bools without taking a dep on a global variable func (p *Plugin) InspectFeatureGates(featureGates featuregate.FeatureGate) { p.nonPreemptingPriority = featureGates.Enabled(features.NonPreemptingPriority) p.resourceQuotaFeatureGateEnabled = featureGates.Enabled(features.ResourceQuotaScopeSelectors) } </ins> // SetExternalKubeClientSet implements the WantsInternalKubeClientSet interface. func (p *Plugin) SetExternalKubeClientSet(client kubernetes.Interface) { p.client = client

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Following the documentation at the , I tried enabling quota for a namespace other than but I was still not able to create a pod with priorityClass in my namespace with following error: Error from server (Forbidden): error when creating "": pods "memory-demo" is forbidden: pods with system-cluster-critical priorityClass is not permitted in my-namespace namespace What you expected to happen: Expected pod to be created with namespace where I allowed quota. I can see that both ResourceQuota and Priority Plugins are enabled from logs. I0408 17:15:08. ] Loaded 8 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,MutatingAdmissionWebhook. I0408 17:15:08. ] Loaded 6 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,ValidatingAdmissionWebhook,ResourceQuota. How to reproduce it (as minimally and precisely as possible): Follow the steps at Anything else we need to know?: I think, the issue is with priorityPlugin where we are rejecting instead of letting resourceQuota plugin to handle the admission when is enabled. Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Others:
/sig scheduling /cc
Same happens to me. Correction, it does not allow to use the default permissions outside of kube-system
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
See also:
/remove-lifecycle stale
/assign Per offline discussion with
Same issue here.


Question: return nil } <ins> // InspectFeatureGates allows setting bools without taking a dep on a global variable func (p *Plugin) InspectFeatureGates(featureGates featuregate.FeatureGate) { p.nonPreemptingPriority = featureGates.Enabled(features.NonPreemptingPriority) p.resourceQuotaFeatureGateEnabled = featureGates.Enabled(features.ResourceQuotaScopeSelectors) } </ins> // SetExternalKubeClientSet implements the WantsInternalKubeClientSet interface. func (p *Plugin) SetExternalKubeClientSet(client kubernetes.Interface) { p.client = client

We are using namespaces other than kube-system for system pods as it is getting too crowded, therefore we would like to run cluster-node-critical pods in other namespaces.
Waiting for this to be available myself. No reason my Node Exporter dataset has to run in the namespace instead of my namespace.


Question: var wg sync.WaitGroup var once sync.Once <del> // set up stdin stream </del> if e.stdin != nil { <del> headers.Set(api.StreamType, api.StreamTypeStdin) remoteStdin, err := conn.CreateStream(headers) if err != nil { return err } </del> // copy from client's stdin to container's stdin go func() { // if e.stdin is noninteractive, e.g. `echo abc | kubectl exec -i <pod> -- cat`, make sure

We create the stdin stream and immediately start copying local stdin to the container. We do the same for stdout and stderr (create stream, start copying immediately). In the kubelet, we wait to receive all the streams before we start the exec and copy between the streams and the container. The spdy connection frame handling code has 5 frame workers, each with a queue of up to 50 frames. In the event that we try to push more than 50 frames to a single queue before all streams are ready, the 51st attempt to push a frame blocks until there is space available in the queue. But there won't be any space until the server starts reading from the streams, which won't happen until all the streams have been received, so we're stuck at this point. The fix is to create all the streams first, and only start copying after all the streams have been created and the server is ready to copy between the streams and the container. Originally reported here:
FYI
This seems pretty high priority.
It's an easy fix. has already tested the patch for OpenShift and it works. Are we willing to pull it in before is resolved, as that will take longer to put in place?
Here's the patch, for reference:
5 * 50 = 250 frames before you get stuck? Or did I misunderstand?
Oh, I guess all the frames go to the same worker for some reason?
anyway, I'm fine with taking this and cherry picking it. Order doesn't matter; someone has to rebase either way, right?
frames are assigned to each queue based on stream id, so if you sent 50 frames to the stdin stream, that queue fills up and the loop that processes incoming frames is blocked until something starts reading from the stream. Working on a PR now.
Gotcha, thanks. , Andy Goldstein wrote:
an update: I spent some time trying to create an e2e that would fail more often than not without this fix, but I haven't had any luck so far (I have it stream a copy of the kubectl binary from the local system to the pod). The issue that originally uncovered the problem was from running exec on a Windows system, where it reproduces consistently, but I haven't seen it fail much on Linux. I will keep trying...


Question: })) } <ins> // AddEphemeralContainerSync adds an EphemeralContainer to a pod and waits for it to be running. func (c *PodClient) AddEphemeralContainerSync(pod *v1.Pod, ec *v1.EphemeralContainer, timeout time.Duration) { namespace := c.f.Namespace.Name podJS, err := json.Marshal(pod) ExpectNoError(err, "error creating JSON for pod %q", format.Pod(pod)) ecPod := pod.DeepCopy() ecPod.Spec.EphemeralContainers = append(ecPod.Spec.EphemeralContainers, *ec) ecJS, err := json.Marshal(ecPod) ExpectNoError(err, "error creating JSON for pod with ephemeral container %q", format.Pod(pod)) patch, err := strategicpatch.CreateTwoWayMergePatch(podJS, ecJS, pod) ExpectNoError(err, "error creating patch to add ephemeral container %q", format.Pod(pod)) _, err = c.Patch(context.TODO(), pod.Name, types.StrategicMergePatchType, patch, metav1.PatchOptions{}, "ephemeralcontainers") ExpectNoError(err, "Failed to patch ephemeral containers in pod %q", format.Pod(pod)) ExpectNoError(e2epod.WaitForContainerRunning(c.f.ClientSet, namespace, pod.Name, ec.Name, timeout)) } </ins> // DeleteSync deletes the pod and wait for the pod to disappear for `timeout`. If the pod doesn't // disappear before the timeout, it will fail the test. func (c *PodClient) DeleteSync(name string, options metav1.DeleteOptions, timeout time.Duration) {

This issue is to discuss an e2e testing plan for Ephemeral Containers. /sig node
As a first step towards this issue, I am planning to create node-e2e tests (under ) to verify the following: With a running pod client adds a valid EphemeralContainer resource, and the ephemeral container starts from the ephemeral container, list processes of itself and other containers, and then signal other processes (if is ) from the ephemeral container, ping & curl other containers try to read/write to stdin/stdout of the ephemeral container verify resource request/limit values against calculated expectations check oom scores of processes in the ephemeral container verify volume mounts restarts according to policy, when process exits client tries to add EphemeralContainer resource with ports (should be rejected) client tries to add EphemeralContainer resource with probes (should be rejected) client tries to add EphemeralContainer resource with resources (should be rejected) client tries to add EphemeralContainer resource with securityContext (should be rejected) removes the ephemeral container, and the ephemeral container is stopped and deleted adds multiple ephemeral containers, should all startWith pod of non-running status (TBD)
Awesome, thanks! This is a great start, here are some things to look out for: Visibility should work, but I expect signaling to fail without which is not currently allowed. iirc was interested in how this would be tested for ephemeral containers. Can you point us towards any examples that would help us get started? Ephemeral containers will never restart (and the policy should always be never) These will be denied by the api server without ever making it to the kubelet, so they are better tested in the apiserver integration tests (and I think they are already). Removing ephemeral containers is not currently allowed. /assign
Validation of ephemeralContainers is well tested in Testing of removing ephemeral containers is in ,
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with .


Question: })) } <ins> // AddEphemeralContainerSync adds an EphemeralContainer to a pod and waits for it to be running. func (c *PodClient) AddEphemeralContainerSync(pod *v1.Pod, ec *v1.EphemeralContainer, timeout time.Duration) { namespace := c.f.Namespace.Name podJS, err := json.Marshal(pod) ExpectNoError(err, "error creating JSON for pod %q", format.Pod(pod)) ecPod := pod.DeepCopy() ecPod.Spec.EphemeralContainers = append(ecPod.Spec.EphemeralContainers, *ec) ecJS, err := json.Marshal(ecPod) ExpectNoError(err, "error creating JSON for pod with ephemeral container %q", format.Pod(pod)) patch, err := strategicpatch.CreateTwoWayMergePatch(podJS, ecJS, pod) ExpectNoError(err, "error creating patch to add ephemeral container %q", format.Pod(pod)) _, err = c.Patch(context.TODO(), pod.Name, types.StrategicMergePatchType, patch, metav1.PatchOptions{}, "ephemeralcontainers") ExpectNoError(err, "Failed to patch ephemeral containers in pod %q", format.Pod(pod)) ExpectNoError(e2epod.WaitForContainerRunning(c.f.ClientSet, namespace, pod.Name, ec.Name, timeout)) } </ins> // DeleteSync deletes the pod and wait for the pod to disappear for `timeout`. If the pod doesn't // disappear before the timeout, it will fail the test. func (c *PodClient) DeleteSync(name string, options metav1.DeleteOptions, timeout time.Duration) {

Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen /assign /priority important-soon
Reopened this issue. <detailsIn response to : /reopen /assign /priority important-soon Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-contributor-experience at . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Rotten issues close after 30d of inactivity.


Question: })) } <ins> // AddEphemeralContainerSync adds an EphemeralContainer to a pod and waits for it to be running. func (c *PodClient) AddEphemeralContainerSync(pod *v1.Pod, ec *v1.EphemeralContainer, timeout time.Duration) { namespace := c.f.Namespace.Name podJS, err := json.Marshal(pod) ExpectNoError(err, "error creating JSON for pod %q", format.Pod(pod)) ecPod := pod.DeepCopy() ecPod.Spec.EphemeralContainers = append(ecPod.Spec.EphemeralContainers, *ec) ecJS, err := json.Marshal(ecPod) ExpectNoError(err, "error creating JSON for pod with ephemeral container %q", format.Pod(pod)) patch, err := strategicpatch.CreateTwoWayMergePatch(podJS, ecJS, pod) ExpectNoError(err, "error creating patch to add ephemeral container %q", format.Pod(pod)) _, err = c.Patch(context.TODO(), pod.Name, types.StrategicMergePatchType, patch, metav1.PatchOptions{}, "ephemeralcontainers") ExpectNoError(err, "Failed to patch ephemeral containers in pod %q", format.Pod(pod)) ExpectNoError(e2epod.WaitForContainerRunning(c.f.ClientSet, namespace, pod.Name, ec.Name, timeout)) } </ins> // DeleteSync deletes the pod and wait for the pod to disappear for `timeout`. If the pod doesn't // disappear before the timeout, it will fail the test. func (c *PodClient) DeleteSync(name string, options metav1.DeleteOptions, timeout time.Duration) {

Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-contributor-experience at . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen /remove-lifecycle rotten
Reopened this issue. <detailsIn response to : /reopen /remove-lifecycle rotten Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: ginkgo.Context("when querying /metrics", func() { var oldCfg *kubeletconfig.KubeletConfiguration var testPod *v1.Pod <del> var cpusNumPerNUMA, numaNodes int </del> <ins> var cpusNumPerNUMA, coresNumPerNUMA, numaNodes, threadsPerCore int </ins> ginkgo.BeforeEach(func(ctx context.Context) { var err error

/kind failing-test <!-- If this is a flake, please add: /kind flake --/sig node started to fail since 09-13.
/cc
/assign
/cc
/triage accepted /priority important-soon it's indeed failing too often
/reopen The test is still failing: Now timing out at 120s i.e. 2 minutes as we increased the default timeout . We need to understand the root cause and get to the bottom of this.
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Snippet from the build log () corresponding from this failure We see from the above logs that: node has two numa nodes, 8 cpus per NUMA so the root cause of the failure is not because the tests are running on a node that doesn't have multi-numa support. is started and the health check fails a couple of times but eventually succeeds. pod is created with 9 cpus (cpusperNUMA +1 i.e. 8+1) which is expected to fail with Topology Affinity Error as we cannot allocate resources for such a pod from a single NUMAnode. We fail to get matching metrics. It is not clear if the check fails because we don't get kubelet metrics at all or the expected metrics but based on point 6 below it seems like it is because of latter. We can add additional logs for more clarity here. each is executed and Kubelet is restarted and health check is done. are collected from the namespace corresponding to the test pod which shows that the pod started even though it was expected to fail with Topology Affinity Error. We need to understand why the pod got successfully created even though it requested CPUs that couldn't be allocated from a single NUMAnode.


Question: "fmt" "strings" <del> "k8s.io/api/core/v1" </del> <ins> v1 "k8s.io/api/core/v1" </ins> "k8s.io/apimachinery/pkg/api/resource" "k8s.io/apimachinery/pkg/util/sets" "k8s.io/apimachinery/pkg/util/validation/field"

Created limit range object with this parameters then tried to create this pod got message: Error from server (Invalid): error when creating "pod-to-be-": Pod "nginx-to-be-removed-4" is invalid: spec.containers[0].resources.requests: Invalid value: "700m": must be less than or equal to cpu limit Pod should be created with Requests & Limits of cpu equal to 700m (since it is in range of min/max cpu limitrange object) create limitrange object and pod as in description No response <details</details<detailson premise/AKS </details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
You have a default CPU limit of 10m, and no explicit CPU limit on the container, but a request larger than the limit. The request must always be less than or equal to the limit. Your pod needs to also set a higher limit, or the limit range needs to not set a default limit.
/remove-kind bug /kind support
/cc
but this is not obvious behavior since there is min & max set in limit range. This is limits (not min/max) that should be overwritten by pod. limits should be set to requests in this case imho
limit range limits what the user can set, it doesn't perform any mutations other than the defaulting and doesn't claim to. Min/max enforce that pods cannot set higher / lower. The default ensures that if the pod doesn't set a limit it will have one. personally I would remove the default limit, especially at 10m, this will aggressively limit pods. it would be, if the pod set a limit that would be a substantial breaking change, I don't think we could do that short of a v2 API and I'm not sure it would be correct behavior even then. we could make it opt-in, but then, why not just set them to match on the pod to begin with?


Question: build_args=( ${goflags:+"${goflags[@]}"} -gcflags="${gogcflags}" <del> -asmflags="${goasmflags}" </del> -ldflags="${goldflags}" -tags="${gotags:-}" )

While working on for , I found that our kubernetes1.24 package stopped to produce bit-identical results on 2 fresh builds. The first chunk is likely just the hash of the remainder, so can be ignored. The other chunks indicate that a random build temporary path is used for Sources should be able to reproduce bit-identical binaries e.g. for SLSA level4 On Debian or openSUSE do No response <detailsWARNING: This version information is deprecated and will be replaced with the output from kubectl version --short. Use --output=yaml|json to get the full version. Client Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2022-05-05T00:00:00Z", GoVersion:"go1.18.3", Compiler:"gc", Platform:"linux/amd64"} Kustomize Version: v4.5.4 console $ cat /etc/os-release $ uname -a Linux bmwiedemann-factory 5.18.6-1-default SMP PREEMPTDYNAMIC Thu Jun 23 05:46:18 UTC 2022 () x8664 x8664 x86_64 GNU/Linux </details<detailsgo1.18 </details<details</details<details</details
/sig release /area build-release
can you please post a link to the full logs from the build?
any news on this? Haven't heard from (context: we worked to fix this a couple of times in )
Unfortunately not, is this something we can fix on the OBS side? If the go build timestamp () is the main issue here then we can approach a fix in our build pipeline.
It would be good if would not be embedded in the first place. It might be possible to replace it somehow in our builds, but all other people wanting reproducible builds, would need to replicate that.
full logs from the build: from kubernetes-1.24.3 built 2022-08-18
can you please try setting GOCACHE=off when you run the build? I am suspecting that the temp directory is being generated here: (or add an export in for that env var)
I to but still got these diffs.


Question: build_args=( ${goflags:+"${goflags[@]}"} -gcflags="${gogcflags}" <del> -asmflags="${goasmflags}" </del> -ldflags="${goldflags}" -tags="${gotags:-}" )

So i built golang from source, peppered it with changes to track down where this is getting created and found that it is coming from: I don't know yet how to fix it. will dig more when i can (travelling for the next few days)
Would it work to override the , I see that they use the : Ref: https://reproducible- Edit: Probably not, we may make it configurable in golang :thinking:
this is so tricky :( will dig more when i get back from my event this week
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue with Mark this issue as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close not-planned [1]:
Closing this issue, marking it as "Not Planned".


Question: build_args=( ${goflags:+"${goflags[@]}"} -gcflags="${gogcflags}" <del> -asmflags="${goasmflags}" </del> -ldflags="${goldflags}" -tags="${gotags:-}" )

<detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue with Mark this issue as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This is still relevant even with 1.27. It might be worth revisiting this issue again. /reopen /triage accepted
Reopened this issue. <detailsIn response to : This is still relevant even with 1.27. It might be worth revisiting this issue again. /reopen /triage accepted Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-lifecycle rotten
latest run -
I found that our package has the same issue, so it probably it is a toolchain issue in golang - and much faster to debug outside of the slow kubernetes build. =filed
AIUI, the conclusion by the golang people seems to be that commit is not the right approach and one should use with the go call instead. Not sure how that interacts with other parts of your build.
cc (please see referenced commit above!)
I see. I inherited the gcflags before so I plead innocence. It seems plausible to change it from to - I don't know when I will have time to look at that, though.


Question: build_args=( ${goflags:+"${goflags[@]}"} -gcflags="${gogcflags}" <del> -asmflags="${goasmflags}" </del> -ldflags="${goldflags}" -tags="${gotags:-}" )

My patch doesn't seem to fix the whole problem: thockin-glaptop4 kubernetes buildtrimpath /$ make kubelet +++ [1114 19:10:12] Building go targets for linux/amd64 (non-static) thockin-glaptop4 kubernetes buildtrimpath /$ md5sum output/bin/kubelet output/bin/kubelet thockin-glaptop4 kubernetes buildtrimpath /$ make kubelet +++ [1114 19:12:40] Building go targets for linux/amd64 (non-static) thockin-glaptop4 kubernetes buildtrimpath /$ md5sum output/bin/kubelet output/bin/kubelet thockin-glaptop4 kubernetes build_trimpath /$
diff --- a 2023-11-14 19:13:45. -0800 +++ b 2023-11-14 19:13:56. -0800 -1,4 +1,4 -38rZHdQckYF9EEzmndR/BAXOwzCcq0Mk1UHT62ha/QLiRgXP4oojWh3oWP9/TWG7KW90IL8vKt1UOEPR +LeLArvl7sgWAd55ROKN2/BAXOwzCcq0Mk1UHT62ha/QLiRgXP4oojWh3oWP9/cYb5ZGQwa6js077YiAvK /lib64/ld-linux-x86- AUATUSH []AA] -,8 +,8 go1.21.3 runtime/cgo: call frame too large -2023-11-15T03:12:40Z -2023-11-15T03:12:40Z +2023-11-15T03:13:47Z +2023-11-15T03:13:47Z pthreadcreate failed: %s bad stack bounds: lo=%p hi=%p expand 32-byte kexpand 32-byte k
That seems unrelated. I tested with and it made our openSUSE package fully reproducible. Your diff contains a timestamp - maybe would get rid of that. The top part of the diff is just the go build-id hash of the remainder and can be ignored.


Question: build_args=( ${goflags:+"${goflags[@]}"} -gcflags="${gogcflags}" <del> -asmflags="${goasmflags}" </del> -ldflags="${goldflags}" -tags="${gotags:-}" )

Thanks for confirming
Will there be backports to 1.24++ ? I found it applies cleanly to 1.26++ and works with some small tweaks in 1.24 and 1.25. If not, I'd submit the patch for our openSUSE packages.
1.24 and 1.25 reached EOL and we don't support these branches any longer. We can cherry-pick this to 1.26+ if folks ( agree
Cherry-picks: 1.29 1.28 1.27 1.26


Question: if [[ ! ${KUBE_COVER} =~ ^[yY]$ ]]; then kube::log::status "Running unit tests without code coverage" go test "${goflags[@]:+${goflags[@]}}"  <del> ${KUBE_RACE} ${KUBE_TIMEOUT} "${@+${@/#/${KUBE_GO_PACKAGE}/}}" || true </del> <ins> ${KUBE_RACE} ${KUBE_TIMEOUT} "${@+${@/#/${KUBE_GO_PACKAGE}/}}" </ins> return 0 fi

Take great care merging! Example PR run (look at the "succeeded" builds): https://travis-
Somewhat related, (bug) and (fix, as yet unmerged) mean that "guestbook" and "shell" e2e tests are being silently skipped on all platforms since Wed last week (4/15). should merge shortly and resolve the issue.
From the Travis run, So it looks like either the shell script is eating the exit code, or the test is exiting 0 even when tests fail. Similarly for Shippable, it seems.
I think broke this. It appears that will always exit 0, even when tests fail or timeout. We should revert this change ASAP.
Fixed by


Question: } func newResourcePod(usage ...framework.Resource) *v1.Pod { <del> containers := []v1.Container{} </del> <ins> var containers []v1.Container </ins> for _, req := range usage { containers = append(containers, v1.Container{ Resources: v1.ResourceRequirements{Requests: req.ResourceList()},

What happened: In a v1.18 cluster with the LocalStorageCapacityIsolation feature gate disabled setting an ephemeral-storage request or limit causes the pod to be unschedulable: If the ephemeral-storage request is removed the pod is scheduled. This behavior exists in v1.19 and a recent v1.20 build. What you expected to happen: In previous releases such pods are scheduled. I expect that any ephemeral-storage limit is not enforced. How to reproduce it (as minimally and precisely as possible): Disable the LocalStorageCapacityIsolation feature guide by adding to the kube-scheduler. Create a pod like this: Anything else we need to know?: The problem was originally seen after moving from IBM Cloud Red Hat OpenShift v4.4 to v4.5. Pods that worked in 4.4 (Kubernetes v1.17) remained in a Pending state in OpenShift v4.5 (Kubernetes 1.18). We discovered that the IBM Cloud OpenShift service has the LocalStorageCapacityIsolation feature-gate disabled while Red Hat has enabled the feature. We then replicated the behavior with a plain Kubernetes cluster. Environment: Kubernetes version (use ): v1.18.10, v1.19.3, v1.20.0 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig sig-scheduling
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig sig-scheduling Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scheduling
I can confirm it's a bug. Will work on it.


Question: } func newResourcePod(usage ...framework.Resource) *v1.Pod { <del> containers := []v1.Container{} </del> <ins> var containers []v1.Container </ins> for _, req := range usage { containers = append(containers, v1.Container{ Resources: v1.ResourceRequirements{Requests: req.ResourceList()},

/assign
Clarifying the recreate steps: I disabled the LocalStorageCapacityIsolation feature guide by adding to the kube-scheduler since this behavior seems to be within the scheduler.
Thanks for confirming!
This bug has been fixed in master branch, and will be backported to [1.18] and [1.19] soon. [1.18]: [1.19]:


Question: createConfig.Config.ExposedPorts = exposedPorts hc.PortBindings = portBindings <del> // Set DNS options. if dnsConfig := c.GetDnsConfig(); dnsConfig != nil { hc.DNS = dnsConfig.Servers hc.DNSSearch = dnsConfig.Searches hc.DNSOptions = dnsConfig.Options } </del> // Apply resource options. setSandboxResources(hc)

Kubernetes version (use ): HEAD (~v1.6.0-beta.4) Environment: any What happened: I want to use ClusterFirstWithHostNet which was implemented in , but only dockertools (legacy/non-CRI) was updated to support this feature. The CRI dockershim should be modified to support this rewriting the file of Pods with HostNetwork=true as well. This is a bug in v1.6, I hope it can be prioritized to be fixed in v1.6.1, since it's blocking me from using CRI. Since CRI is enabled by default, it's expected that it should have feature parity with the earlier implementation.
Related
- I guess I didn't realize these crossed paths. Do you have time to take a look?
Will take a look
Thanks! Ping me directly on this.
Filed for supporting it in CRI.


Question: // setupLocalVolume setups a directory to user for local PV func setupLocalVolume(config *localTestConfig) *localTestVolume { testDirName := "local-volume-test-" + string(uuid.NewUUID()) <del> testDir := filepath.Join(containerBase, testDirName) </del> hostDir := filepath.Join(hostBase, testDirName) // populate volume with testFile containing testFileContent <del> writeCmd, _ := createWriteAndReadCmds(testDir, testFile, testFileContent) </del> <ins> writeCmd, _ := createWriteAndReadCmds(hostDir, testFile, testFileContent) </ins> By(fmt.Sprintf("Creating local volume on node %q at path %q", config.node0.Name, hostDir)) <del> _, err := framework.NodeExec(config.node0.Name, writeCmd) </del> <ins> err := framework.IssueSSHCommand(writeCmd, framework.TestContext.Provider, config.node0) </ins> Expect(err).NotTo(HaveOccurred()) return &localTestVolume{ <del> node:         config.node0, hostDir:      hostDir, containerDir: testDir, </del> <ins> node:    config.node0, hostDir: hostDir, </ins> } }

8 tests failed, 2 jobs failed, 88 builds failed. Failure stats cover 1 day time range '3 Aug 2017 03:31 UTC' to '4 Aug 2017 03:31 UTC'.
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig storage
This should have been fixed by /cc
The test panic is fixed, but now it's failing because it can't SSH into the node.
/assign


Question: } } <ins> func verifyStringArrayEqualsAnyOrder(t *testing.T, actual, expected []string) { invalid := len(actual) != len(expected) if !invalid { for _, exp := range expected { found := false for _, act := range actual { if exp == act { found = true break } } if !found { t.Errorf("Expected element %s not found in %#v", exp, actual) } } } if invalid { t.Errorf("Expected: %#v, Actual: %#v", expected, actual) } } </ins> func verifyBoolean(t *testing.T, expected, value bool) { if expected != value { t.Errorf("Unexpected boolean.  Expected %t.  Found %t", expected, value)

Successfully ran containers with "onFailure" restartPolicy are restarting after purging oldest docker containers. Kubelet purges/removes oldest containers periodically from docker. After purging, we loose the exit code and history of previously ran containers and restart containers as if they were never run.
Thanks & I can help fix this. Let me know. Thanks
Thanks for reporting Just to make sure I understand the issue: You ran some containers with an "onFailure" restartPolicy. When the Kubelet purged old containers the container was restarted (unexpectedly).
Ah, after reading again I think the issue is that after the containers are purged, when the container exits successfully it get restarted although it shouldn't. Is that correct?
After reading through the code I found a bug in the implementation that is most likely causing this error. Should have a PR soon.
Yes. Correct. After initial container run, it should never be run again, since it already ran to completion. Will wait for your PR.
PR is out. This should fix the issue. Feel free to test if you can :)


Question: github.com/urfave/negroni => github.com/urfave/negroni v1.0.0 github.com/vishvananda/netlink => github.com/vishvananda/netlink v0.0.0-20171020171820-b2de5d10e38e github.com/vishvananda/netns => github.com/vishvananda/netns v0.0.0-20171111001504-be1fbeda1936 <del> github.com/vmware/govmomi => github.com/vmware/govmomi v0.20.0 </del> <ins> github.com/vmware/govmomi => github.com/vmware/govmomi v0.20.1 </ins> github.com/vmware/photon-controller-go-sdk => github.com/vmware/photon-controller-go-sdk v0.0.0-20170310013346-4a435daef6cc github.com/xanzy/go-cloudstack => github.com/xanzy/go-cloudstack v0.0.0-20160728180336-1e2cbf647e57 github.com/xiang90/probing => github.com/xiang90/probing v0.0.0-20160813154853-07dd2e8dfe18

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: The vSphere cloud provider is unable to fetch the node failure domain labels when using VCP with SAML auth because of an authentication failure. For context, the SAML token auth is described here - SAML auth support for the zones api was introduced in this PR - , which was cherry-picked to 1.14.1 here - I've tested the credentials I'm using with govc to ensure that I'm able to get a token: The credentials also work fine when using VCP without the node labels feature. Relevant kubelet logs: What you expected to happen: VCP should add failure domain labels to the node object. How to reproduce it (as minimally and precisely as possible): Setup SAML auth for VCP with zone labeling enabled. Anything else we need to know?: There was a previous issue that was raised () but that was before SAML auth support for zones was implemented. This could potentially be an issue with govc handling of SAML tokens since even with the Administrator role assigned to the user, I'm still unable to list tags. The role and keys were generated with the following: Environment: Kubernetes version (use ): 14.1 Cloud provider or hardware configuration: vSphere OS (e.g: ): Linux & Windows
/sig vmware
Would you mind taking a look at this? I can provide more info if needed.
govc's command only works with the SOAP endpoint currently. The REST endpoint has its own session manager and requires a separate login. At the moment, govc's tags commands login/logout when each command is run and don't use a persisted session (which are saved to ~/.govmomi/sessions). However, you can use a token returned by to test SAML auth with the tags commands. For example: Can you give that try? There are some TODOs around this btw:
Those commands work for me. So that would imply the solution user, role, and permissions are set up correctly? Would you expect the to also work?


Question: github.com/urfave/negroni => github.com/urfave/negroni v1.0.0 github.com/vishvananda/netlink => github.com/vishvananda/netlink v0.0.0-20171020171820-b2de5d10e38e github.com/vishvananda/netns => github.com/vishvananda/netns v0.0.0-20171111001504-be1fbeda1936 <del> github.com/vmware/govmomi => github.com/vmware/govmomi v0.20.0 </del> <ins> github.com/vmware/govmomi => github.com/vmware/govmomi v0.20.1 </ins> github.com/vmware/photon-controller-go-sdk => github.com/vmware/photon-controller-go-sdk v0.0.0-20170310013346-4a435daef6cc github.com/xanzy/go-cloudstack => github.com/xanzy/go-cloudstack v0.0.0-20160728180336-1e2cbf647e57 github.com/xiang90/probing => github.com/xiang90/probing v0.0.0-20160813154853-07dd2e8dfe18

I seem to be getting a 404 when using with the flag - Does seem to work when specifying a tag name: Are there any other govc commands I should check / verify?
yes, the solution user setup looks right. Not sure what the issue is with . Can you share your vCenter build number ()? I'll try to reproduce the issue.
$ govc about -u vcdo Name: VMware vCenter Server Vendor: VMware, Inc. Version: 6.7.0 Build: OS type: linux-x64 API type: VirtualCenter API version: 6.7.2 Product ID: vpx UUID: -55ed-4737-9cc4-
I can reproduce this, but don't know what the issue is yet. The version of Go and vSphere have changed since , but if it was one of those was the cause, I'd expect SAML auth to fail with the govc tags commands too. All I have to go on for now is the vCenter log message below. I'll update when I have more info.
I have it working with the change below, but need to figure out a few things first: Why this is the case when run inside kubelet and not govc (note that the port change to 443 breaks govc) A good way to determine which port to use (depends on the answer to the above) If this is behavior is specific to 6.7U2 or something triggered by a change in K8s I suppose a quick fix would change to try port 443 if 80 fails, but would prefer to avoid that as the failure generates stack traces in the
/assign
This is obvious in retrospect, but the issue is the message signature needs to use the same port as the original request URL. However, in the govc case there was no port set in the request URL. The client defaults to port when connecting (based on the scheme), but does not include any port with the header. In the case, the default is on the server side. This can be reproduced with if includes the port , rather than just as we were using. The VCP tests that use token auth for tags lookup just don't validate the signature, so this wasn't a change related to 6.7U2.


Question: github.com/urfave/negroni => github.com/urfave/negroni v1.0.0 github.com/vishvananda/netlink => github.com/vishvananda/netlink v0.0.0-20171020171820-b2de5d10e38e github.com/vishvananda/netns => github.com/vishvananda/netns v0.0.0-20171111001504-be1fbeda1936 <del> github.com/vmware/govmomi => github.com/vmware/govmomi v0.20.0 </del> <ins> github.com/vmware/govmomi => github.com/vmware/govmomi v0.20.1 </ins> github.com/vmware/photon-controller-go-sdk => github.com/vmware/photon-controller-go-sdk v0.0.0-20170310013346-4a435daef6cc github.com/xanzy/go-cloudstack => github.com/xanzy/go-cloudstack v0.0.0-20160728180336-1e2cbf647e57 github.com/xiang90/probing => github.com/xiang90/probing v0.0.0-20160813154853-07dd2e8dfe18

I'll open a K8s PR to pull in the fix:
Updating k/vendor to govmomi master adds almost 6K lines for a 3 line fix. I may just create a govmomi branch off of v0.20 to reduce churn in the K8s release branches for this fix, may have to wait until tomorrow.
Really appreciate the insight into this issue. Can confirm that specifying the port (443) in the GOVC_URL reproduces the issue. Thank you!
Unfortunately, I'm seeing the same issue with v1.14.3: cloud_config Works fine when I use the username and password of a vsphere admin, but using cert and key still has authentication issues.
any related errors in vCenter's ?
Seeing a lot of this exception:
whoa! that stack trace is like punch in the gut (brings back memories!) :)
indeed!! apologies, I must have had the change for PR locally, but failed to include it as I had focused on the signature issue and updating vendor/


Question: fwk.RunUnreservePlugins(pluginContext, assumedPod, scheduleResult.SuggestedHost) sched.recordSchedulingFailure(assumedPod, err, SchedulerError, fmt.Sprintf("Binding rejected: %v", err)) } else { <del> klog.V(2).Infof("pod %v/%v is bound successfully on node %v, %d nodes evaluated, %d nodes were found feasible", assumedPod.Namespace, assumedPod.Name, scheduleResult.SuggestedHost, scheduleResult.EvaluatedNodes, scheduleResult.FeasibleNodes) </del> <ins> // Calculating nodeResourceString can be heavy. Avoid it if klog verbosity is below 2. if klog.V(2) { node, _ := sched.Cache().GetNodeInfo(scheduleResult.SuggestedHost) klog.Infof("pod %v/%v is bound successfully on node %q, %d nodes evaluated, %d nodes were found feasible. Bound node resource: %q.", assumedPod.Namespace, assumedPod.Name, scheduleResult.SuggestedHost, scheduleResult.EvaluatedNodes, scheduleResult.FeasibleNodes, nodeResourceString(node)) } </ins> metrics.PodScheduleSuccesses.Inc() // Run "postbind" plugins.

<!-- Please only use this template for submitting enhancement requests --What would you like to be : 1) Increase the log level of info logs to "v=2" when the scheduler or a node from its cache. 2) Increase the log level of info logs to "v=2" when the scheduler can't schedule a pod , , and add a log message for 2) report memory/cpu status of . Why is this needed: With cluster autoscaler now being used in production by some vendors, we want to have more visibility to events that could be related to the auto scaler to help debug customer issues. /sig scheduling /priority important-soon /assign
GitHub didn't allow me to assign the following users: congliu01. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : <!-- Please only use this template for submitting enhancement requests --1) Increase the log level of info logs to "v=2" when the scheduler or a node from its cache. 2) report memory/cpu status of . Why is this needed: With cluster autoscaler now being used in production by some vendors, we want to have more visibility to events that could be related to the auto scaler to help debug customer issues. /priority important-soon /assign </details
will consider this assigned to you.
/assign


Question: if err != nil { t.Fatalf("failed to create an http request: %v", err) } <ins> auditCtx := &audit.AuditContext{Event: &auditapi.Event{Level: auditapi.LevelMetadata}} req = req.WithContext(audit.WithAuditContext(req.Context(), auditCtx)) </ins> // can't use tlsServer.Client() as it contains the server certificate // in tls.Config.Certificates. The signatures are, however, only checked

go1.18 dropped support for certificates signed with the SHA-1 hash function - While this does improve security, and publicly trusted Certificate Authorities no longer issue such certificates, it is possible that user-provided certificates issued by private Certificate Authorities and used in webhooks or aggregated API servers do still use this. Similar to removal of support for certificates without subjectAltNames in go1.17, we need to: [x] add metrics when webhook/conversion/aggregated certificates with this hash are encountered - [x] document use of and the metric from 1 as a mitigation for clusters which still need to identify and replace use of such certificates - to release note and known issues [x] add audit annotation when SHA-1 cert is encountered - [x] update docs at and - /sig api-machinery instrumentation docs /priority important-soon /milestone v1.24 /kind bug documentation Upgraded 1.23 clusters to continue working as-is Configure an admission webhook, conversion webhook, or aggregated server with a serving certificate signed with the SHA-1 hash function. Upgrade to HEAD of 1.24. The certificate will no longer work by default No response 24 HEAD (go1.18-based) n/a n/a No response No response No response
it looks like NewMissingSANRoundTripperWrapperConstructor already has all the plumbing, and instead of missing SANs we could check the SignatureAlgorithm and output a distinct metric
had a not-yet-tested WIP branch at with a sketch of what it might look like
/assign who offered to pick this up
GitHub didn't allow me to assign the following users: stlaz. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign who offered to pick this up Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Indeed, please, you should trust Jordan :slightlysmilingface:
/assign
/triage accepted
These restrictions are on the GitHub side, this message is posted when it receives a specific error from the GitHub API while attempting to assign It is an annoying GitHub limitation, and one that seemingly does not exist in the web client vs the API.


Question: if err != nil { t.Fatalf("failed to create an http request: %v", err) } <ins> auditCtx := &audit.AuditContext{Event: &auditapi.Event{Level: auditapi.LevelMetadata}} req = req.WithContext(audit.WithAuditContext(req.Context(), auditCtx)) </ins> // can't use tlsServer.Client() as it contains the server certificate // in tls.Config.Certificates. The signatures are, however, only checked

adds the metrics. the release note on that issue describes the go change, links to go docs, describes use of the option, the new metrics, and the kube-apiserver logs what would you expect kubeadm to do with this? GODEBUG off by default and let users override GODEBUG on their own (is this possible)? GODEBUG by default on all clusters? GODEBUG by default only on upgrade? The risk of 1 is that a user upgrades without reading release notes and breaks a cluster that has SHA-1 certs in webhooks, etc The risk of 2 is that we propagate accepting insecure certificates longer, even in new clusters that we know don't have any pre-existing bad certs The risk of 3 is that it makes an upgraded 1.23 → 1.24 cluster and a new 1.24 cluster inconsistent
the majority of kubeadm users use kubeadm managed CA and certificates. there are less users that use external CA to sign their own certificates. sha1 attacks surfaced more than a before kubeadm's creation and Git hash collusion made the problem more known around 2017 - that was at a time when kubeadm did not support external CA. given these facts, an estimate on my side is that there are currently no kubeadm users that use sha1 certificates, but i would not exclude the possibility entirely. if such users exists, they are on they own and my vote goes for option 1, until they migrate their PKI to something secure. if the same users log tickets for us, our response would be "please use GODEBUG and migrate ASAP". all users should be able to see the notes in IMO, the PR needs an "action required" to appear on top of the release notes.
that's helpful, thanks agreed. weird, I set , but the label didn't get . the label manually.
and there's a way for them to set that envvar in kube-apiserver manifests when using kubeadm?
There's been a lot of intermittent github webhook delivery issues in the past few weeks or so, slash commands are going to be somewhat unreliable :/
they could use kubeadm patches to override the YAML manifest defaults - i.e. the contents of v1.Container -Env:


Question: package apiserver import ( <ins> "fmt" </ins> "io/ioutil" "net/http" "net/http/httptest"

Have only seen this once, personally.
This flaked in Jenkins as well. [job/kubernetes-test-go/5351]
can you take a look at this? I think has a lot already :)


Question: By("Waiting for GPUs to become available on the local node") Eventually(func() bool { return framework.NumberOfNVIDIAGPUs(getLocalNode(f)) > 0 <del> }, 10*time.Second, time.Second).Should(BeTrue()) </del> <ins> }, 10*time.Second, framework.Poll).Should(BeTrue()) </ins> if framework.NumberOfNVIDIAGPUs(getLocalNode(f)) < 2 { Skip("Not enough GPUs to execute this test (at least two needed)")


Question: - To generate definition for a specific type or package add "+k8s:openapi-gen=true" tag to the type/package comment lines. - To exclude a type or a member from a tagged package/type, add "+k8s:openapi-gen=false" tag to the comment lines. <ins> # OpenAPI Extensions OpenAPI spec can have extensions on types. To define one or more extensions on a type or its member add "+k8s:openapi-gen=x-kubernetes-$NAME:$VALUE" to the comment lines before type/member. A type/member can have multiple extensions. The rest of the line in the comment will be used as $VALUE so there is no need to escape or quote the value string. Extensions can be use to pass more information to client generators or documentation generators. For example a type my have a friendly name to be displayed in documentation or being used in a client's fluent interface. </ins>

Background: kubectl and the Kubernetes reference docs both depend on the spec to operate on Kubernetes resources. The lacks important information required by kubectl and the docs. Usecase: kubectl can identify the patchStrategy and mergeKey for resource fields kubectl can identify the columns to print for on resources docs can parse and display the default value for fields docs can identify top-level objects Solution: Add items to the comment tag: Example: The long term plan is to remove the and struct tags once all code has been updated to use the swagger spec instead. Until then, comments should be for the struct tags and . Some verification process will need to be written to make sure the struct tags and comments remain in sync.
I think this wouldn't fall under sig-apps. cc
I would not do the columns stuff - we're moving get to the server side. Top level objects is already available in some form by the client - specifically the list that we use (should use) to get the swagger doc tells us what the top level type is (or is that for something like explain). Merge key, patch strategy, default values all belong as "real" attributes, although we had not necessarily intended those to be arguments to k8s:openapi-gen, but actual tags in their own right (like "+optional")
Thanks Is there a design proposal / eta for this? This is for generating the docs from swagger. Right now the list of top level resources for the docs to display is hardcoded in a . I would like to move the majority of the metadata out of the config file so that: same tool can generate reference docs for federated apiservers without writing a custom config. don't need to manually maintain and update the config for each release - e.g. adding new types I have no strong opinion about this as long as they appear in the swagger so it is possible right tools the can use this data. Can you comment on why openapi-gen vs specific tags?
Describe in the server makes sense to me. IIUC, just prints out a set of columns. Was the idea that the server would just print out the columns or return a list of column / values or something more complicated? No need to answer if it is in the issue / proposal already.


Question: - To generate definition for a specific type or package add "+k8s:openapi-gen=true" tag to the type/package comment lines. - To exclude a type or a member from a tagged package/type, add "+k8s:openapi-gen=false" tag to the comment lines. <ins> # OpenAPI Extensions OpenAPI spec can have extensions on types. To define one or more extensions on a type or its member add "+k8s:openapi-gen=x-kubernetes-$NAME:$VALUE" to the comment lines before type/member. A type/member can have multiple extensions. The rest of the line in the comment will be used as $VALUE so there is no need to escape or quote the value string. Extensions can be use to pass more information to client generators or documentation generators. For example a type my have a friendly name to be displayed in documentation or being used in a client's fluent interface. </ins>

is the bigger issue that touches on a rough overview, I can't find my rambling brain dump of the problem, but get would be rough tabular json, you could also get that as csv, and describe could be a richer but not quite full structured key/value structure.
regarding tags, they do not need to be in the same k8s:openapi-gen tag, it can be repeated: they can also be separate tags but still general like this: The idea is to have a general way to add arbitrary extension to a type so we don't need to change code generator for each specific tag. I have no strong feeling about the format, but I would like it to be general vs tags like +optional. The difference is +optional is a defined tag in OpenAPI () but extensions does not have any meaning for code generator. They better to be general so generate can pass them without change.
No, I mean we should define conventions in our API code that match the intent (optional) rather than have tons and tons of repeated code that is setting x-kubernete-describe-columns (or similar)
I spent some time thinking about your comments last night. I don't think this is mutually exclusive with , and could be complementary. IMHO this is general metadata associated with the type, and including it in the doesn't preclude supporting an api endpoint that pre-parses and formats the data. However, including it in the swagger does allow clients using the swagger to take advantage of it. Following is an example of where the swagger would be useful even if get was in the sever. A generated Java client could use the display columns for the "toString" method of the resource. Another use case that had mentioned to me was to allow some kubectl commands to operate on directly on local files. e.g. (strawman) could print the columns. I don't think we want to do this now, but it is an example of why keeping this object metadata private to the server limits what we can do. My hope is that this swagger tags solution will also oprove portable to easily work with ThirdPartyResources as well by writing the display-columns as an annotation on the TPR and keeping the same format. is a can of worms, and I would agree that the solution is capable of providing a richer experience for aggregating information across multiple resource types - e.g.


Question: - To generate definition for a specific type or package add "+k8s:openapi-gen=true" tag to the type/package comment lines. - To exclude a type or a member from a tagged package/type, add "+k8s:openapi-gen=false" tag to the comment lines. <ins> # OpenAPI Extensions OpenAPI spec can have extensions on types. To define one or more extensions on a type or its member add "+k8s:openapi-gen=x-kubernetes-$NAME:$VALUE" to the comment lines before type/member. A type/member can have multiple extensions. The rest of the line in the comment will be used as $VALUE so there is no need to escape or quote the value string. Extensions can be use to pass more information to client generators or documentation generators. For example a type my have a friendly name to be displayed in documentation or being used in a client's fluent interface. </ins>

how describing Nodes fetches Pod information. I still think that the solution is compatible with the richer solution you proposed and there is not much additional cost in doing it. Howv What would you think of supporting a set of optional ALPHA extensions for "display-columns" and "describe-columns" allowing any consumers of the swagger to print the objects. In theory it could also be used by the UI to provide a summary view without having to do separate lookups. As an ALPHA feature we could try it for supporting sig-service-catalog and revisit it in 1.7. If we consider it successful, the solution outlined in could become another consumer of the display-columns tag? I like this approach. My biggest concern here is being able to get this into 1.6. Since kubectl supports +/-1 version skew, we need will need to maintain the old logic for at least 1 full release after we add support for the new swagger based logic. That puts us at 6 months from now to fully migrate if we don't have any swagger extensions in 1.6.
I think that's a nice to have in the long run - but then people will say "why are the columns different when the server prints them from the local print"? And most of the pod columns in no way can represent the full complexity of what they print with swagger, so the best effort local printing is going to be really, really, really best effort. Also, local print is a really specialized case - if we just did the generic metadata print that would probably be ok. So while I agree it's valid, it seems like a lot of work for very little practical value. Again, not opposed, just don't want to do two things halfway and then have nothing. So just biased towards not adding more client complexity in the near term. So you want to hack this in for service catalog in 1.6 on TPR? I feel like that's a lot of hacks - the generic printer stuff at least gives you name and namespace, and I'd rather do nothing in 1.7 for generic client printer rather than jump ahead. I won't stop you if you want to spend time and effort on it, but it's a lot of specialized work that will then be neglected while we do the server side stuff.


Question: - To generate definition for a specific type or package add "+k8s:openapi-gen=true" tag to the type/package comment lines. - To exclude a type or a member from a tagged package/type, add "+k8s:openapi-gen=false" tag to the comment lines. <ins> # OpenAPI Extensions OpenAPI spec can have extensions on types. To define one or more extensions on a type or its member add "+k8s:openapi-gen=x-kubernetes-$NAME:$VALUE" to the comment lines before type/member. A type/member can have multiple extensions. The rest of the line in the comment will be used as $VALUE so there is no need to escape or quote the value string. Extensions can be use to pass more information to client generators or documentation generators. For example a type my have a friendly name to be displayed in documentation or being used in a client's fluent interface. </ins>

My intend here was avoiding repeated code for adding each of these tags somewhere (in openapi-gen code or some root level place). Let me distinguish these tags in two categories with respect to OpenAPI code generator: that need some sort of processing that affects OpenAPI spec (e.g. +optional removes the field from OpenAPI field) that need no more processing and will be directly put into OpenAPI spec (e.g. +gvk that we don't have but we can define as a way to define Group/Version/Kind of an object vs register method) My argument is about the second one. I think patch strategy here falls into this category. The question is only about naming it. Naming Option 1: +patch-strategy=... This looks cleaner but we need to define an alias somewhere to tell generator this is the same as x-kubernetes-patch-strategy. The alias definition and the tags are in two places. Putting them in the generator code make the code non-generic and we don't want to do that. Putting these aliases somewhere as an argument to the generator begs for bugs as people that look at the type and add this tag to their types need to know (maybe by reading a doc, but who do that) that it needs to pass the alias to the generator. I, in 6 month, will look at a type and see a tag and copy the tag to my type and expect it to work that will not with this option. Naming Option 2: use something generic for these type of tags. after all generator does not need to do any processing. I suggest +openapi-tag:tag=value that will result in "x-kubernetes-tag: value" in OpenAPI spec. There is no alias defined anywhere and without any code change or alias copying or any knowledge about passing any argument, a tag can be to OpenAPI spec. The value of these tags and their use cases are outside of the scope of what I am arguing about. Anyway a solution is better than no solution, I would suggest we continue with Option 2 as Alpha and change it later if we see fit. Adding an aliasing mechanism to this option later should be easy.
On patch strategy, I had assumed we'd read the value from what's in the struct tag, rather than requiring a net new tag?


Question: - To generate definition for a specific type or package add "+k8s:openapi-gen=true" tag to the type/package comment lines. - To exclude a type or a member from a tagged package/type, add "+k8s:openapi-gen=false" tag to the comment lines. <ins> # OpenAPI Extensions OpenAPI spec can have extensions on types. To define one or more extensions on a type or its member add "+k8s:openapi-gen=x-kubernetes-$NAME:$VALUE" to the comment lines before type/member. A type/member can have multiple extensions. The rest of the line in the comment will be used as $VALUE so there is no need to escape or quote the value string. Extensions can be use to pass more information to client generators or documentation generators. For example a type my have a friendly name to be displayed in documentation or being used in a client's fluent interface. </ins>

, Mehdy Bohlool <:
The get stuff is less urgent for me as long as we aren't doing even more hacky things to work around the current limitations, and can make a commitment to delivering it in a specific release (sounds like 1.7). For service catalog there was a plan to fork kubectl, compile the service catalog types into the binary, and provide the fork as a separate binary. I have some concerns with this approach. I would rather provide a temporary hack in the code that can provide a consistent user experience from day one and re-implement later vs having users get used to downloading separate binaries for each aggregated apiserver. I have no plans to do any work for on this for the core Kubernetes types, and can punt on it until the scope of the work is understood to be relatively small and easily upgraded to the server-side get.


Question: This is an example of a pod that consumes its name and namespace via the downward API: <del> ```json { "apiVersion":"v1beta3", "name": "downward-api-pod", "kind": "Pod", "spec": { "manifest": { "containers": [{ "name": "example-container", "image": "gcr.io/google_containers/busybox", "command": [ "sh", "-c", "env" ] "env": [{ "name": "POD_NAMESPACE", "valueFrom": { "fieldPath": { "apiVersion": "v1beta3", "fieldPath": "metadata.namespace" } } }, { "name": "POD_NAME", "valueFrom": { "fieldPath": { "apiVersion": "v1beta3", "fieldPath": "metadata.name" } } }] }] } } }] </del> <ins> ```yaml apiVersion: v1beta3 kind: Pod metadata: name: dapi-test-pod spec: containers: - name: test-container image: gcr.io/google_containers/busybox command: [ "/bin/sh", "-c", "env" ] env: - name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name - name: POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace restartPolicy: Never </ins> ```

I tried the json file in docs/downward API. After the pod is created, the environment variables are created but have empty value: The json file in the doc has a few grammar issues, so I modified it a bit:


Question: return writeCmd, readCmd } <ins> // Create command to verify that the file doesn't exist // to be executed via SSH on the node with the local PV func createFileDoesntExistCmd(testFileDir string, testFile string) string { testFilePath := filepath.Join(testFileDir, testFile) return fmt.Sprintf("[ ! -e %s ]", testFilePath) } </ins> // Execute a read or write command in a pod. // Fail on error func podRWCmdExec(pod *v1.Pod, cmd string) string {


Question: if service.Spec.ClusterIP != "" { allErrs = append(allErrs, field.Forbidden(specPath.Child("clusterIP"), "must be empty for ExternalName services")) } <del> if len(service.Spec.ExternalName) > 0 { allErrs = append(allErrs, ValidateDNS1123Subdomain(service.Spec.ExternalName, specPath.Child("externalName"))...) </del> <ins> // The value (a CNAME) may have a trailing dot to denote it as fully qualified cname := strings.TrimSuffix(service.Spec.ExternalName, ".") if len(cname) > 0 { allErrs = append(allErrs, ValidateDNS1123Subdomain(cname, specPath.Child("externalName"))...) </ins> } else { allErrs = append(allErrs, field.Required(specPath.Child("externalName"), "")) }

What happened: The externalName of a Service is not allowed to end with a trailing dot. What you expected to happen: The externalName of a Service should allow a trailing dot. This is very useful to denote a fully qualified value of the CNAME record created by the service and prevent DNS-trial-and-error by appending different (cluster-local) search domains. How to reproduce it (as minimally and precisely as possible): Try to set as a externalName for a Service of type ExternalName. More details The reference reads: RFC-1123 section 6.1.4.3 also states: Also in the errata of RFC-1123 a reference to RFC-1034 is to reiterate on what a legal hostname is. Section 3.1 of RFC-1034 explains the use and validity of a trailing dot. So while one could argue that a valid RFC-1123 subdomain may have a trailing dot, it might still be best to not change the which is used in a lot of places, but instead change the validation locally. Anything else we need to know?: I'll send a PR. Environment: Not much applicable.
/sig network
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.


Question: // TODO: Find a uniform way to deal with systemctl/initctl/service operations. #34494 func restartKubelet(f *framework.Framework) { <ins> beforeSocks, err := filepath.Glob("/var/lib/kubelet/device-plugins/nvidiaGPU*.sock") framework.ExpectNoError(err) Expect(len(beforeSocks)).NotTo(BeZero()) </ins> stdout, err := exec.Command("sudo", "systemctl", "list-units", "kubelet*", "--state=running").CombinedOutput() framework.ExpectNoError(err) regex := regexp.MustCompile("(kubelet-[0-9]+)")


Question: // whether to disable use of standard error when streaming output from tty DisableStderr bool <del> CommandName       string SuggestedCmdUsage string </del> <ins> CommandName             string ParentCommandName       string EnableSuggestedCmdUsage bool </ins> Pod *corev1.Pod

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: The pod name is missing in suggested cmd usage when executing : What you expected to happen: The suggested cmd usage should be a complete cmd like the one got from How to reproduce it (as minimally and precisely as possible): Use to attach any pod/deployment/service, the PodName will be missing in the suggested cmd usage like above. Anything else we need to know?: It works fine for kubectl < v1.12 Environment: Kubernetes version (use ): v1.13.1 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig cli /assign


Question: if exists { // For GPUs, not only requests can't exceed limits, they also can't be lower, i.e. must be equal. if quantity.Cmp(limitQuantity) != 0 && !v1helper.IsOvercommitAllowed(resourceName) { <del> allErrs = append(allErrs, field.Invalid(reqPath, quantity.String(), fmt.Sprintf("must be equal to %s limit", resourceName))) </del> <ins> allErrs = append(allErrs, field.Invalid(reqPath, quantity.String(), fmt.Sprintf("must be equal to %s limit of %s", resourceName, limitQuantity.String()))) </ins> } else if quantity.Cmp(limitQuantity) > 0 { <del> allErrs = append(allErrs, field.Invalid(reqPath, quantity.String(), fmt.Sprintf("must be less than or equal to %s limit", resourceName))) </del> <ins> allErrs = append(allErrs, field.Invalid(reqPath, quantity.String(), fmt.Sprintf("must be less than or equal to %s limit of %s", resourceName, limitQuantity.String()))) </ins> } } }

Created limit range object with this parameters then tried to create this pod got message: Error from server (Invalid): error when creating "pod-to-be-": Pod "nginx-to-be-removed-4" is invalid: spec.containers[0].resources.requests: Invalid value: "700m": must be less than or equal to cpu limit Pod should be created with Requests & Limits of cpu equal to 700m (since it is in range of min/max cpu limitrange object) create limitrange object and pod as in description No response <details</details<detailson premise/AKS </details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
You have a default CPU limit of 10m, and no explicit CPU limit on the container, but a request larger than the limit. The request must always be less than or equal to the limit. Your pod needs to also set a higher limit, or the limit range needs to not set a default limit.
/remove-kind bug /kind support
/cc
but this is not obvious behavior since there is min & max set in limit range. This is limits (not min/max) that should be overwritten by pod. limits should be set to requests in this case imho
limit range limits what the user can set, it doesn't perform any mutations other than the defaulting and doesn't claim to. Min/max enforce that pods cannot set higher / lower. The default ensures that if the pod doesn't set a limit it will have one. personally I would remove the default limit, especially at 10m, this will aggressively limit pods. it would be, if the pod set a limit that would be a substantial breaking change, I don't think we could do that short of a v2 API and I'm not sure it would be correct behavior even then. we could make it opt-in, but then, why not just set them to match on the pod to begin with?


Question: "//test/e2e/framework:go_default_library", "//test/e2e/framework/auth:go_default_library", "//test/e2e/framework/deployment:go_default_library", <ins> "//test/e2e/framework/events:go_default_library", </ins> "//test/e2e/framework/kubectl:go_default_library", "//test/e2e/framework/kubesystem:go_default_library", "//test/e2e/framework/metrics:go_default_library",

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: This feature is lacking coverage, especially for containerized kubelet environments.
/assign since I'm the owner of this feature.
And just fixes a bug for containerized kubelet when the mount file is an empty file.
Are we going to enable these tests in v1.11? We're getting code freeze soon.
There is a local containerized cluster pull and CI job that we can probably leverage. Right now it only runs conformance tests, we may be able to add new suites to enable other tests. Cc who can point us to the owner for the containerized testing effort
yes, seems to be holding up pretty well - https://k8s-,%20master%20(dev)
In I made pull-kubernetes-local-e2e-containerized to run also all test with string. I'm enabling some tests (subpath!) in , feel free to suggest more tests.
Ref
IIUC, the default host path type is . And when host path type is , we expect container runtime to create the host path if it doesn't exist (docker is doing that today). Is that true? If that is, we should revert part of , because container runtime should still create host path if it does not exist. /cc I tried to enforce , and got a lot of errors , e.g.:
Correct. This is mainly for backwards compatibility.
I see. If so, I'll remove the CRI definition "If the hostPath doesn't exist (e.g. hostPath volume), runtimes should report errors". Thanks!
SGTM. Refer to .
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale will help fix this.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close.


Question: "//test/e2e/framework:go_default_library", "//test/e2e/framework/auth:go_default_library", "//test/e2e/framework/deployment:go_default_library", <ins> "//test/e2e/framework/events:go_default_library", </ins> "//test/e2e/framework/kubectl:go_default_library", "//test/e2e/framework/kubesystem:go_default_library", "//test/e2e/framework/metrics:go_default_library",

If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with .


Question: "//test/e2e/framework:go_default_library", "//test/e2e/framework/auth:go_default_library", "//test/e2e/framework/deployment:go_default_library", <ins> "//test/e2e/framework/events:go_default_library", </ins> "//test/e2e/framework/kubectl:go_default_library", "//test/e2e/framework/kubesystem:go_default_library", "//test/e2e/framework/metrics:go_default_library",

Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: codecs := serializer.NewCodecFactory(scheme) o := NewObjectTracker(scheme, codecs.UniversalDecoder()) watch, err := o.Watch(testResource, ns) <ins> if err != nil { t.Fatalf("test resource watch failed in %s: %v ", ns, err) } </ins> go func() { err := o.Create(testResource, testObj, ns) if err != nil {

What happened: What you expected to happen: I am not sure if is it a bug, I think we should catch err when Watch testResource failed in func TestWatchCallNonNamespace. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
Absolutely we should check err gracefully.
/sig cli
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/fixes it


Question: nodeFn             func() *v1.Node existingPods, pods []*v1.Pod want               []bool <ins> qHintEnabled       bool </ins> }{ { name: "regular node, pods with a single constraint",

In our custom plugin, we want to move pods that is unscheduled by the plugin to activeQ/backoffQ when a new Pod is created. So, we define like this: But, even when a new Pod is created, the Pod that was unscheduled by our custom plugin isn't moved to backoffQ/activeQ. when a new Pod is created, the Pod unscheduled by our custom plugin is moved to backoffQ/activeQ. a custom plugin that has pod events in EventsToRegister. a new Pod that the custom plugin will reject. another new Pod that makes the Pod created in (1) schedulable. can see the Pod created in (1) isn't scheduled immediately (it should've been scheduled immediately after (3) because (3) should move the Pod created in (1) to activeQ/backoffQ.) If I understand correctly and if it's not expected behavior, this bug has a non-small impact, especially in v1.24 because we change to 5 min. v1.23.4 but I think latest master has the same issue. <details</details<details</details<details</details<details</details<details</details
/assign /sig scheduling
Maybe it's only on a newly created unscheduled Pod. We've moved correctly for events related to assigned Pod.
And deleting Node events has the same issue...? I guess we need to run when some Nodes are deleted. In in-tree plugins, only pod topology spread uses the deleting Node event and I guess this bug affects its performance. (Even when some Nodes are deleted, the Pods, that were unscheduled by Pod topology spread, aren't moved to activeQ/backoffQ by that deleting events.)
Hm, it may be a problem that events related to Node and Pod are not always called when an event occurs. (Even if it's intended behavior, at least we need to write this on the comment of .) /retitle scheduler: events in related to Pod and Node don't always work
To summarize: The problem is "not all events related to Pod or Node don't always make move from unschedulableQ to active/backoffQ and it may not be intended for custom-plugin developers because this behavior isn't even documented." I think we should move all unschedulable Pods with all Pod and Node events.


Question: nodeFn             func() *v1.Node existingPods, pods []*v1.Pod want               []bool <ins> qHintEnabled       bool </ins> }{ { name: "regular node, pods with a single constraint",

If concerned about performance degradation, we can extend or , for example, to have the function like ? This problem has a non-small impact, especially in v1.24+ because we change PodMaxUnschedulableQDuration to 5 min. Let's see the details... ↓ We move the unscheduled Pods (rejected by the plugin that has Pod add/update events in ) in or But, the target Pods against this moving are only unschedulable pods which have any affinity term that matches "pod". So,,,, users cannot move from unschedulableQ to activeQ/backoffQ with all assignedPod events? For example, for Pod Topology Spread (it uses Pod All event in ), the new assigned Pod creation may make some Pods, that are rejected by Pod Topology Spread, schedulable. Regardless of whether that unschedulable Pods have any affinity term that matches a newly created assigned pod or not. Of course this is not just a problem for Pod Topology Spread, it could be a problem for user's custom-plugins. I guess we should pass all unscheduled Pods to . Or at least we should document this behavior. No issue on this. We don't move the unschedulable Pods with those events at all. That's a problem... Even when a plugin uses Pod add/update/delete event in and wants to move the rejected Pods with non-scheduled Pod's add/update/delete events, scheduling queue doesn't move the unschedulable Pods with those events. (In our company, we have the custom plugin for and the plugin wants the scheduler queye to move the Pods rejected by the plugin from unschedulableQ to active/backoffQ when a new non-scheduled Pod is created.) We move only unscheduled Pods that pass . checks the Node can run that unscheduled Pod from the points of view of some in-tree filter plugins like noderesources/nodeport/nodeAffinity/nodename. The problem that arises here is similar to "Assigned Pod's add/update events". For example, for Pod Topology Spread (it uses Node Add event in ), the new Node creation may make some Pods, that are rejected by Pod Topology Spread, schedulable. Regardless of the results of against that Node. (In , we change the skew calculation of Pod Topology Spread to exclude Nodes that don't match the node affinity.


Question: nodeFn             func() *v1.Node existingPods, pods []*v1.Pod want               []bool <ins> qHintEnabled       bool </ins> }{ { name: "regular node, pods with a single constraint",

But, after is implemented, the calculation can be configured to include those Nodes.) We don't move the unschedulable Pods with Node delete events... That's a problem. Even when the plugin use Node delete event in , scheduling queue doesn't move the unschedulable Pods with any Node delete events.)
/cc Could you please take a look
These are all intended optimizations. These basic checks prevent the scheduler from retrying unschedulable pods over and and over. So far we haven't had the need to make these checks configurable as they check well established scheduling features. +1 on documenting the behavior, but I'm not sure about changing it or making it configurable (that adds complexity that might not be justified). What is it that you are trying to do?
What is it that you are trying to do? We are trying to remove flushing in queue in , right? Without improving like I suggest, some Pods in unschedulableQ cannot move back to activeQ forever in some cases if we remove flushing. (Actually, our company's custom plugin needs to wait flush because non-assigned Pod's events don't let scheduling queue move unschedulable Pods rejected by that plugin.) So, I think we should simply inform all events to queue and let queue move unschedulable Pods with that events. This can happen in some custom plugins like our company's one and also even in in-tree plugins. For example, Pod Topology Spread (after is implemented). When a user uses and , the creation of Node that doesn't match nodeAffinity/nodeSelector may change the scheduling result.
cc who implemented NodeInclusionPolicy. Can you provide a concrete example? The policy just refers to the topology spread calculations. It doesn't mean that the node doesn't have to match the nodeaffinity to be able to accept the pod. I'm asking what is your custom plugin trying to do. Depending on whether it makes sense, we need to evaluate if it's worth the complexity.
For example, for Pod Topology Spread (it uses Pod All event in EventsToRegister), the new assigned Pod creation may make some Pods, that are rejected by Pod Topology Spread, schedulable This is actually a problem here, 5 mins would be too long for background flush. But it doesn't mean that we should simply shorten the interval time.


Question: nodeFn             func() *v1.Node existingPods, pods []*v1.Pod want               []bool <ins> qHintEnabled       bool </ins> }{ { name: "regular node, pods with a single constraint",

For example, Pod Topology Spread (after is implemented). When a user uses NodeAffinityPolicy: Ignore and WhenUnsatisfiable: DoNotSchedule, the creation of Node that doesn't match nodeAffinity/nodeSelector may change the scheduling result. The skew of a new created node is always 0, so it's still unfit I think.
Our company's custom plugin is like plugin. And like coscheduling's prefilter, our plugin rejects Pods in PreFilter when the number of Pods is less than the defined number. Yes, so, maybe coscheduling plugin has the same issue as our plugin. The coscheduling plugin also uses Pod Add event and I guess it's for non-scheduled Pod's creation. (I don't know much about the coscheduling plugin though.) Oh, that's true Please forget about that example. But, how about like this scenario: user uses and Pod is rejected by Pod Topology Spread because of the skew. e.g. a user uses and TopologyKey is hostname, NodeA/NodeB has 10 Pods each, and NodeC has 8 Pods. adds un-tolerated taint to NodeC skew calculation result should be changed and the Pod rejected in (2) will be allowed by Pod Topology Spread. But the Pod isn't moved from unschedulableQ without flushing because of this issue.
Maybe we can find some pointcuts by the in , e.g. if pod was rejected by PodTopologySpread, we'll when Assigned Pod's add/update events arrived. For customize plugins, we may have to do some refactoring to register the filter func together with the event.
So now NodeC is excluded from calculations. Hence the pod fits in either NodeA or NodeB, which were unchanged. Tricky... I'll have to think about it. We really need those quick checks to pass for performance in most scenarios. /triage accepted
Thanks for bringing this up. Some comments below: You're right. Pods failed by PodTopologySpread may not get re-queued immediately. However, an assigned Pod's add/update event would only impact unschedulable Pods with cross-node scheduling directives. As of today, it's InterPodAffinity and PodTopologySpread.


Question: nodeFn             func() *v1.Node existingPods, pods []*v1.Pod want               []bool <ins> qHintEnabled       bool </ins> }{ { name: "regular node, pods with a single constraint",

So instead of "pass all unscheduled Pods to movePodsToActiveOrBackoffQueue", I'd like to come up with a solution like: It's pretty rare to move pods depending on Unschedulable Pods' events. In your particular case, maybe a proactive call would behave more efficiently. It doesn't sound correlated, is it? As the check will just pre-verify the basic checks which are orthogonal with PodTopologySpread. In other words, it's impossible that a Pod failed can become schedulable eventually. Updated: The example is a valid point... Can you share some use-cases to move Pods upon node's deletion? In the before, I don't think we have any, until the last release the introduction of feature (as in that case deleting a node would impact the number of domains).
to come up with a solution like: If we do so, also, we need to provide the way to pass some functions to decide which Pods should pass to like for out-of-tree plugins? call would behave more efficiently. Ah~, that's a good idea :) Maybe like this (similar to ) user uses WhenUnsatisfiable: DoNotSchedule. Pod is rejected by Pod Topology Spread because of the skew. e.g. a user uses maxSkew: 1 and TopologyKey is hostname, NodeA/NodeB has 10 Pods each, and NodeC has 8 Pods. deletes NodeC skew calculation result should be changed and the Pod rejected in (2) will be allowed by Pod Topology Spread. But the Pod isn't moved from unschedulableQ without flushing because of this issue. So,,, ignoring some events for performance in scheduler side (without providing a way to change this) could be inconvenient for some plugins. What I thought in the above previous comment () is to define "which kind of resource's events should be ignored" in plugin side. (similar to 's idea? ) We can add pre filter functions in like: Then we can set any pre-filter functions for each plugin. And, we can define some default behavior for PreFilterFn and use them when is empty. So, for example, if most of the plugins are considered to prefer to ignore the event for Node that doesn't match nodeAffinity/nodeSelector, we can define this behavior as the default behavior.


Question: nodeFn             func() *v1.Node existingPods, pods []*v1.Pod want               []bool <ins> qHintEnabled       bool </ins> }{ { name: "regular node, pods with a single constraint",

What do you think about this idea?
Thanks. It makes sense to me. To summarize, the following cases doesn't properly re-queue relevant pods: Pod's add/update events -scheduled Pod's add/update/delete events 's add/update/delete events For 1 & 2, I've commented above with solutions. (1 needs core changes similar like podAffinity, 2 may leverage the Activate hook) For 3, with 's examples, I'm convinced this is some area we need to improve. One simple and stratightforward solution is to plumb a hook to let the plugin claim "alright, upon Node events, move the pod failed by me anyways". Then once this knob is registered, we move pods failed by that plugin back unconditionally; instead of proceeding with . This hook should be available to in-tree and out-of-tree plugins. In terms of in-tree plugins, it's just PodTopologySpread (we discussed this in today's sig meeting, even InterPodAffinity doesn't fall into this case) So, as a first step, let's ensure we're on the same page and agree on the ^^ summary. Next, we can better evaluate trade-offs and come up with an optimal solution.
Thanks for discussing this topic in the sig meeting. But, I am still believing that all logic related to a plugin should be controllable from the plugin side and we should not create something that can be achieved in in-tree plugins but not in out-of-tree plugins. So... the benefit of the "customizable PreCheck approach" () isn't only to solve this issue. We can use this feature for other optimization: like improving the whole performance of scheduler by moving only Pods that are more likely to be able to be scheduled by defining more detailed events on with customizable PreCheck. Many plugins have a particular scenario that makes rejected Pods schedulable and the current doesn't have an abundant enough ability of expression for this. (So, today many plugins need to say "register this hoge event on foo resource. This event may make rejected Pods schedulable" on comments in . ) Yes, I watched the sig-meeting recording as well and the opinion that the "customizable PreCheck approach" increases some implementation complexity surely makes sense. But,,, isn't this complexity increasing acceptable even if the above is taken into account?


Question: nodeFn             func() *v1.Node existingPods, pods []*v1.Pod want               []bool <ins> qHintEnabled       bool </ins> }{ { name: "regular node, pods with a single constraint",

EDIT: It is a bad habit of mine to always write long sentences even though I am not very good at English. Summarizing the above: I think it's not a good idea to create something that can be achieved in in-tree plugins but not in out-of-tree plugins. If we need a solid advantage to extend like this, then it has the advantage of optimizing the queue per plugin like described above.
I didn't jump in the thread since it's actively on the discussion. But, the issue is the similar one. (This time, we can solve that issue by changing scheduling queue logic since It's the k/k feature though.)
I'm not sure I fully understand your intention. The slack thread was discussing the AssignedPod update case, so it fallls into case 1 in , which we already agreed to improve in upstream.
Hi Are you working on this? I think this is quite important and if you're not working on this, I can follow.
Sorry, leave here for a while. I wanted to wait for (and then forgot to come back here. ) That proposal is quite similar to what I want to see in the conclusion of this issue; every plugin can judge if each Pod in unschedQ can be moved to schedQ/backoffQ. Do you think it makes sense to add "By which event the pod will be moved back to schedQ/backoffQ" on 's argument? I know and isn't needed in DefaultEnqueue plugin for API, but that's the last missing puzzle piece : If has , enqueue plugin can judge if the event makes it possible to schedule the Pod. For example, I believe we can move ↓ this logic to Pod Affinity plugin by implementing in Pod Affinity plugin. Pod Affinity plugin will remember Pods rejected by itself in . And in , if the given Pod was the one rejected by Pod Affinity, is AssignedPod/Created|Updated and is Pod affinity with the rejected Pod, then move the Pod back to activeQ/backoffQ. Otherwise reject the Pod moving. We can resolve (3) by implementing as well. (by exactly the way you said.↓) My original concern in this issue is that having logic for specific plugins in the scheduler core may prevent out-of-tree plugins' extendability. The scheduler itself should be pure, and shouldn't do anything special for any plugins implementation ideally.


Question: nodeFn             func() *v1.Node existingPods, pods []*v1.Pod want               []bool <ins> qHintEnabled       bool </ins> }{ { name: "regular node, pods with a single constraint",

If we can move all such logic to each plugin, that'd be perfect.
^ is more like a new feature than just a solution to this problem. I created another issue and describe my proposal in detail. Let's have a discussion there.
It can be resolved by moving all preCheck into each plugin's QueueingHint. Let me retitle this issue so that the current status is easy to understand. /retitle scheduler: move all preCheck to QueueingHint
This issue has not been updated in over 1 year, and should be re-triaged. You can: Confirm that this issue is still relevant with (org members only) Close this issue with For more details on the triage process, see /remove-triage accepted
/triage accepted


Question: if err != nil { return nil, fmt.Errorf("failed to fetch cadvisor stats: %v", err) } <del> caInfos := getCRICadvisorStats(allInfos) </del> <ins> caInfos, allInfos := getCRICadvisorStats(allInfos) </ins> // get network stats for containers. // This is only used on Windows. For other platforms, (nil, nil) should be returned.

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --For single node with hundreds and thousands of pods and containers, the cpu usage of kubelet becomes high, one bottleneck is the cadvisor stats provider as below: the method iterate all containerInfos for every podStats cause high cpu the method should not iterate allcontainerInfos for every podStats, use map instead Kubernetes version (use ): v1.19.4 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig node
/triage accepted
/assign
I’ve done an analysis on the code and this is what I have found, I believe we have two issues to address. There are multiple range loops being performed over v2.ContainerInfo / PodStat maps within the ListPodStatus method which can be contributing to the high cpu utilisation. The allInfos argument which is supplied to the getCadvisorContainerInfo function is a map which uses a cgroup path as the key, in order to find matching container info to podUID, processing of the keys in the map needs to be performed until a match is found. Examples of keys are as follows: /kubepods/burstable/podf9e3a173-1470-451a-9ca2-/d8426b82582fbcfae16085bc794c49769d3a9648bd5463976aca4071f9342754 /docker/2dfa17fc81e781212bcc26e189ebb724721fe097394b4de3cbeb7e069bf23290 My recommendation would be to apply the following updates: Refactor the getCadvisorContainerInfo function to allow for a filter and key type parameter to be supplied by the caller, this enables reducing the number of range loops being performed over the container info map by removing elements that are not required and the ability to set the key being used in the returned container infos map to enable fast lookup of container info.