Question:
[x] two weeks soak end date : 26 Feb 2021 According to this APIsnoop query, there are still some remaining RESOURCENAME endpoints which are untested. with this query you can filter untested endpoints by their category and eligiblity for conformance. e.g below shows a query to find all conformance eligible untested,stable,core endpoints Create a ReplicaSet with a static label Read the ReplicaSetScale Update the ReplicaSetScale Get the ReplicaSetScale to ensure it is updated Patch the ReplicaSetScale Read the ReplicaSetScale to ensure it is Patch Delete the Namespace and ReplicaSet Due to the complexity of setting up the resources for APIService we have used the current e2e test as template. It has been extended in a for review. example endpoint | useragent ----------------------------------------|
patchAppsV1NamespacedReplicaSetScale | (linux/amd64) kubernetes/$Format -- [sig-apps] ReplicaSet Replicaset should have a working scale subresource readAppsV1NamespacedReplicaSetScale | (linux/amd64) kubernetes/$Format -- [sig-apps] ReplicaSet Replicaset should have a working scale subresource replaceAppsV1NamespacedReplicaSetScale | (linux/amd64) kubernetes/$Format -- [sig-apps] ReplicaSet Replicaset should have a working scale subresource (3 rows) If a test with these calls gets merged, test coverage will go up by 3 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture
/triage accepted
Presented in Conformance meeting of 9 Feb 2021 and it was agreed that that method see to achieve the purpose and a PR can be created. A similar approach would be taken for the DepoymentScale endpoints.
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

testRSAdoptMatchingAndReleaseNotMatching(f) }) <del> ginkgo.It("Replicaset should have a working scale subresource", func() { </del> <ins> /* Release: v1.21 Testname: ReplicaSet, completes the scaling of a ReplicaSet subresource Description: Create a ReplicaSet (RS) with a single Pod. The Pod MUST be verified that it is running. The RS MUST get and verify the scale subresource count. The RS MUST update and verify the scale subresource. The RS MUST patch and verify a scale subresource. */ framework.ConformanceIt("Replicaset should have a working scale subresource", func() { </ins> testRSScaleSubresources(f) }) })


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: What you expected to happen: The test should pass How to reproduce it (as minimally and precisely as possible): I think the key here is being on a linux host with cgroupsv2 enabled. Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /sig node
/assign
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

package oom import ( <ins> "os" </ins> "github.com/opencontainers/runc/libcontainer/cgroups" "testing"


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: What you expected to happen: The test should pass How to reproduce it (as minimally and precisely as possible): I think the key here is being on a linux host with cgroupsv2 enabled. Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /sig node
/assign
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

func TestPidListerFailure(t *testing.T) { _, err := getPids("/does/not/exist") <del> assert.True(t, cgroups.IsNotFound(err), "expected getPids to return not exists error. Got %v", err) </del> <ins> assert.True(t, cgroups.IsNotFound(err) || os.IsNotExist(err), "expected getPids to return not exists error. Got %v", err) </ins> }


Question:
file was found 'empty'. It blocked running etcd in a fail loop with Before this issue: Data disk (etcd) run out of space due to . .txt be not corrupted. It should be updated: only when needed (infrequently) safely (rename swap or writing on pre-allocated space) migrator script that runs etcd in case of the file not being found. Use whole space of etcd data disk and restart migrator+etcd when concurrent process is disk hungry. Requires changes of this logic: This logic (file write) should not trigger if the minor version matches: The write itself: should be defensive: Please notice that truncates file before writingKubernetes version: 1.18 Cloud provider or hardware configuration: GCE
/sig api-machinery
/assign /triage accepted
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale /lifecycle frozen

path string } <ins> func (v *VersionFile) nextPath() string { return fmt.Sprintf("%s-next", v.path) } </ins> // Exists returns true if a version.txt file exists on the file system. func (v *VersionFile) Exists() (bool, error) { return exists(v.path)


Question:
file was found 'empty'. It blocked running etcd in a fail loop with Before this issue: Data disk (etcd) run out of space due to . .txt be not corrupted. It should be updated: only when needed (infrequently) safely (rename swap or writing on pre-allocated space) migrator script that runs etcd in case of the file not being found. Use whole space of etcd data disk and restart migrator+etcd when concurrent process is disk hungry. Requires changes of this logic: This logic (file write) should not trigger if the minor version matches: The write itself: should be defensive: Please notice that truncates file before writingKubernetes version: 1.18 Cloud provider or hardware configuration: GCE
/sig api-machinery
/assign /triage accepted
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale /lifecycle frozen

// Write creates or overwrites the contents of the version.txt file with the given EtcdVersionPair. func (v *VersionFile) Write(vp *EtcdVersionPair) error { <del> data := []byte(fmt.Sprintf("%s/%s", vp.version, vp.storageVersion)) return ioutil.WriteFile(v.path, data, 0666) </del> <ins> // We do write + rename instead of just write to protect from version.txt // corruption under full disk condition. // See https://github.com/kubernetes/kubernetes/issues/98989. err := ioutil.WriteFile(v.nextPath(), []byte(vp.String()), 0666) if err != nil { return fmt.Errorf("failed to write new version file %s: %v", v.nextPath(), err) } return os.Rename(v.nextPath(), v.path) </ins> } func exists(path string) (bool, error) {


Question:
was removed in 1.20 in favor of a plugin. However, this is not included in release artifacts. The binary is included in release artifacts, and ideally easily installable via kubectl plugin managers like . /sig cli /assign
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This question came up in a docs review of a page referencing () Once this is resolved, please ensure the website is updated to document installation/use of this tool.
/sig release

name = "client-targets", conditioned_srcs = for_platforms(for_client = [ "//cmd/kubectl", <ins> "//cmd/kubectl-convert", </ins> ]), )


Question:
was removed in 1.20 in favor of a plugin. However, this is not included in release artifacts. The binary is included in release artifacts, and ideally easily installable via kubectl plugin managers like . /sig cli /assign
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This question came up in a docs review of a page referencing () Once this is resolved, please ensure the website is updated to document installation/use of this tool.
/sig release

load("//staging/src/k8s.io/component-base/version:def.bzl", "version_x_defs") go_binary( <del> name = "kubectl", </del> <ins> name = "kubectl-convert", </ins> embed = [":go_default_library"], pure = "on", visibility = ["//visibility:public"],


Question:
was removed in 1.20 in favor of a plugin. However, this is not included in release artifacts. The binary is included in release artifacts, and ideally easily installable via kubectl plugin managers like . /sig cli /assign
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This question came up in a docs review of a page referencing () Once this is resolved, please ensure the website is updated to document installation/use of this tool.
/sig release

# If you update this list, please also update build/BUILD. readonly KUBE_CLIENT_TARGETS=( cmd/kubectl <ins> cmd/kubectl-convert </ins> ) readonly KUBE_CLIENT_BINARIES=("${KUBE_CLIENT_TARGETS[@]##*/}") readonly KUBE_CLIENT_BINARIES_WIN=("${KUBE_CLIENT_BINARIES[@]/%/.exe}")


Question:
was removed in 1.20 in favor of a plugin. However, this is not included in release artifacts. The binary is included in release artifacts, and ideally easily installable via kubectl plugin managers like . /sig cli /assign
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This question came up in a docs review of a page referencing () Once this is resolved, please ensure the website is updated to document installation/use of this tool.
/sig release

<ins> #!/usr/bin/env bash # Copyright 2021 The Kubernetes Authors. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # #     http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. set -o errexit set -o nounset set -o pipefail run_convert_tests() { set -o nounset set -o errexit ### Convert deployment YAML file locally without affecting the live deployment # Pre-condition: no deployments exist kube::test::get_object_assert deployment "{{range.items}}{{${id_field:?}}}:{{end}}" '' # Command # Create a deployment (revision 1) kubectl create -f hack/testdata/deployment-revision1.yaml "${kube_flags[@]:?}" kube::test::get_object_assert deployment "{{range.items}}{{${id_field:?}}}:{{end}}" 'nginx:' kube::test::get_object_assert deployment "{{range.items}}{{${image_field0:?}}}:{{end}}" "${IMAGE_DEPLOYMENT_R1}:" # Command output_message=$(kubectl convert --local -f hack/testdata/deployment-revision1.yaml --output-version=apps/v1beta1 -o yaml "${kube_flags[@]:?}") # Post-condition: apiVersion is still apps/v1 in the live deployment, but command output is the new value kube::test::get_object_assert 'deployment nginx' "{{ .apiVersion }}" 'apps/v1' kube::test::if_has_string "${output_message}" "apps/v1beta1" # Clean up kubectl delete deployment nginx "${kube_flags[@]:?}" ## Convert multiple busybox PODs recursively from directory of YAML files # Command output_message=$(! kubectl-convert -f hack/testdata/recursive/pod --recursive 2>&1 "${kube_flags[@]:?}") # Post-condition: busybox0 & busybox1 PODs are converted, and since busybox2 is malformed, it should error kube::test::if_has_string "${output_message}" "Object 'Kind' is missing" # check that convert command supports --template output output_message=$(kubectl-convert "${kube_flags[@]:?}" -f hack/testdata/deployment-revision1.yaml --output-version=apps/v1beta2 --template="{{ .metadata.name }}:") kube::test::if_has_string "${output_message}" 'nginx:' set +o nounset set +o errexit } </ins>


Question:
was removed in 1.20 in favor of a plugin. However, this is not included in release artifacts. The binary is included in release artifacts, and ideally easily installable via kubectl plugin managers like . /sig cli /assign
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This question came up in a docs review of a page referencing () Once this is resolved, please ensure the website is updated to document installation/use of this tool.
/sig release

source "${KUBE_ROOT}/test/cmd/authorization.sh" source "${KUBE_ROOT}/test/cmd/batch.sh" source "${KUBE_ROOT}/test/cmd/certificate.sh" <ins> source "${KUBE_ROOT}/test/cmd/convert.sh" </ins> source "${KUBE_ROOT}/test/cmd/core.sh" source "${KUBE_ROOT}/test/cmd/crd.sh" source "${KUBE_ROOT}/test/cmd/create.sh"


Question:
was removed in 1.20 in favor of a plugin. However, this is not included in release artifacts. The binary is included in release artifacts, and ideally easily installable via kubectl plugin managers like . /sig cli /assign
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This question came up in a docs review of a page referencing () Once this is resolved, please ensure the website is updated to document installation/use of this tool.
/sig release

kube::util::ensure-gnu-sed kube::log::status "Building kubectl" <del> make -C "${KUBE_ROOT}" WHAT="cmd/kubectl" </del> <ins> make -C "${KUBE_ROOT}" WHAT="cmd/kubectl cmd/kubectl-convert" </ins> # Check kubectl kube::log::status "Running kubectl with no options"


Question:
was removed in 1.20 in favor of a plugin. However, this is not included in release artifacts. The binary is included in release artifacts, and ideally easily installable via kubectl plugin managers like . /sig cli /assign
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This question came up in a docs review of a page referencing () Once this is resolved, please ensure the website is updated to document installation/use of this tool.
/sig release

fi ###################### <ins> # Convert            # ###################### if kube::test::if_supports_resource "${deployments}"; then record_command run_convert_tests fi ###################### </ins> # Delete             # ###################### if kube::test::if_supports_resource "${configmaps}" ; then


Question:
on the new node fails sometimes. The following error appears in the log: on the new node succeeds every time Join a control plane node The node has over 40 GB of free space on its root partition. I some more logging to the following functions: , , , , , , It appears that the ephemeral storage is detected a bit too late: /sig node storage Kubernetes version (use ): 1.20.4, commit Kernel (e.g. ): 4.9.252-1.ph2
/assign I will try to reproduce it today.
LocalStorageCapacityIsolation is a feature gate for kubelet. Can you check if you disable it? ReadyCondition will check ephemeral-storage as required capacities if this feature gate is enabled in kubelet configuration. Would you check it first?
[x] 1.8 alpha ephemeral-storage [x] 1.10 Promote LocalStorageCapacityIsolation feature to beta [x] 1.15 KEPs Quotas for ephemeral-storage for LocalStorageCapacityIsolationFSQuotaMonitoring default to false is Alpha since 1.15 [x] Failed to promote LocalStorageCapacityIsolationFSQuotaMonitoring feature to beta [x] 1.20 Honor disabled LocalStorageCapacityIsolation in scheduling I am no sure whether there is a plan to GA LocalStorageCapacityIsolation?
I tried to fix it in . Need some testing later.
It appears that the pod fails before sees the storage capacity:
Would you try this ? I build it with on PR branch.
I tried to apply the PR to 1.20.4. After four successful s, one failed. I got the following error:
! It means that during the node resource updated the first time, the cadvisor is not available yet. If we wait here, it would be a problem. I would think more of how to fix it.
Hi, Same problem here. I'm using K8s v1.20.4 with CentOS7 (2 cores, 4GB and 20GB storage) Cluster creation with kubeadm init always works, but when I try to add several master nodes with kubeadm join it takes a lot of time waiting for etcd pod to be created. In most of cases, etcd pod never runs. The rest of components (APIserver, controller, scheduler and proxy) exists and works (except APIServer which reboots trying to connect the missing etcd) Checking the kubelet logs in the missing etcd hosts I have found this: Mar 10 14:23:27 kmaster-02 kubelet[9448]: W0310 14:23:27. 9448 ] Failed to admit pod etcd-kmaster-02_kube-system() - Unexpected error while attempting to recover from admission failure: preemption: error finding a set of pods to preempt: no set of running pods found to reclaim resources: [(res: ephemeral-storage, q: ), ] That node have 17GB storage free. If kubelet is restarted, the etcd (pod and container in docker ps -a) appears. If I could help doing something please tell me. I would like to help
For anyone who is experiencing this issue and still wants to use kubeadm until it is fixed: You can remove the ephemeral-storage request from the etcd static pod and the join will work. Just make sure that your nodes actually have more than 100MB free storage left. You can reinsert the storage request by patching the etcd manifests after the nodes are up. To patch etcd during a create the following file (the filename matters, the file location doesn't) and use to join additional control plane nodes.
Thanks for the workaround I'm encountering this same issue and the workaround worked for me. Edit: the patch I tried doesn't work though (not sure if this is really a problem). I issued the command: and the response shows that the resource can't be patched this way: There may be another way but I'm still fairly new to Kubernetes so it is not obvious to me at the moment. Anyhow, thanks for the suggestion!
I updated Add a start-delay of 10 seconds for a condition that the capacity of node ephemeral-storage is 0. Not sure if this will fix it.
we are starting to see reports about this every week.
cc
cc
From it sounds like the issue is kubeadm etcd ephemeral-storage requests are too big for some environments? So far this is sounding like a kubeadm issue, not a kubernetes issue.
it seems to be something else. i doubt that users are trying machines that don't satisfy 100Mi. for example, the report in the OP points out that the machine has 40GB.
I can confirm that the VMs on which I installed k8s control plane components had sufficient disk space when I encountered the problem. I'm running Kubernetes v1.20.4 and kubeadm v1.20.4.
I think the root cause here sounds right. There is kind of race here. adding delay will help, but it does not have 100% guarantee. Not sure whether it is easy to add logic to make sure capacity is reported correctly before starting pods..
Did something change in 1.20 regarding the ordering of cadvisor and node startup?
If the reason would be a disk problem(or something we can figure out) and the delay can solve it, I think we should add a warning besides the delay.
We need to figure out what triggers the race condition. This only happens on some machines. I have not seen this myself.
We're able to repro this nearly 100% of the time with Cluster API Provider Azure + k8s 1.20.x + Ubuntu 18.04 + default template, which makes k8s 1.20 pretty much unusable at this point. Trying to see if this repros with other non-Azure cluster-api providers with kubeadm 1.20.
Does anyone know why this only repros on and not ? The ephemeral storage resource requirement here is used in common by both and : In practice I can workaround this issue in cluster-api + Azure by adding a sleep command to my spec, but it requires a significant number to make work ~100% of the time (like 90 seconds). Does the flow take significantly longer before it gets to the part where it submits an etcd pod to the apiserver? Is there any other explanation for this being a -only symptom?
i can see this failing for any static pod with ephemeral storage requests. that is, unless there is proof that this is somehow specific to the kubeadm static pod for etcd that kubeadm deploys.
I agree that the specific kubeadm-delivered etcd pod is not interesting. (In fact it's very permissive, it's only asking for 100Mi.) What definitely seems to be happening (in Azure) is that kubelet takes a long time to know about its underlying node's ephemeral-storage availability. Kubelet starts: ~2 mins later, it complains that it can't schedule etcd: Again, why does this not repro on nodes created by ?
it also seems to be an even more scoped problem because it's not reproducible for me locally (i create a number of clusters every day, usually), the kubeadm CI or in other high level projects that use kubeadm (e.g. kind, minikube, etc). we should try to find a common reason / condition from all the reporting users that are seeing this.
Experiencing same problem on openSuSE Kubic/Tumbletweed snapshot with kubeadm/kubelet 1.20.2, kernel 5.11.15 running on Xen, 2 CPU, 4GB ram 30GB disk (BRTFS)
Problem consistently appear/disappear when switching between snapshots (Working) and , so a package update in that period has probably triggered this. Following files contains changes between those two snapshots.
are you suggesting a root cause in underlying Linux storage driver surface area that is adding delay/uncertainty to kubelet's ability to introspect available ephemeral-storage on the node host OS?
A bit of luck combined with 6th sense i got it narrowed down to systemd update here Looking at the changelog, one item sticks out: Anyone else enabled cgroupsv2 on i.e. the CentOS cases (it needs to be enabled explicitly for that AFAIK)?
ACK on your latest respond regarding unified cgroup hierarchy. Is there a quick, cross-distro way to verify that any folks who are repro'ing have a similar systemd config?
The precence of file/dir /sys/fs/cgroup/cgroup.controllers should mean you are using cgroupsv2 ( Ref: ) I can confirm that the file is present on the kubic snapshot which is not working. It is not present on the one that does work. ( I'm using cri-o as runtime which should support cgroupsv2 ) cgroup2 support was introduced for K8S in 1.19 ( ).
Thanks. So tl;dr your working theory is that cgroupsv2 + kubelet ephemeral-storage admission has some pathologies. I'll post quickly with what my repro containerd + Ubuntu 18.04-LTS OS surface area looks like.
I ran into this issue consistently on Ubuntu 20.04.2 LTS using cri-o but is not present on my systems.
I can also confirm that cgroupsv2 is not a factor in my repro'ing environment:
Also, I'm not using cri-o, containerd here:
Ok - so much for that theory. But still: What I did to narrow it to systemd-update was taking the working kubic snapshot () and run zypper update systemd which updates systemd (along with systemd-sysvinit and udev packages to same version). That consistently made it fail. So I think a pointer to the root-cause might still be hiding somewhere around this changelog - even if it is not cgroupsv2. That part was just a theory (seemed like a "big" thing), but the update of systemd package is fairly hard evidence.
I have identified where these particular symptoms (probably) originate, and cursory testing suggests something like this eliminates the issue: I'll be advocating for a few mins in today's sig-node discussion if any folks following this thread want to chime in. cc
/sig cluster-lifecycle /triage accepted
I've also observed that on some occasions the node allocatable storage is in a continual state of reporting "0". Restarting kubelet immediately fixed that. More debugging to do....
did you manage to reproduce this with an arbitrary static pod that is not the etcd one?
I was not able to reproduce in a non-kubeadm test, but I suspect that the way kubelet bootstrap is implemented there (and possibly environmental reasons) the static pod handling was always happening after this code executed: (the above is in the containerManager method) kubeadm w/ ephemeral-storage requirements in etcd is the most effective repro at this time
the writing of the etcd pod manifest in during is after the kubelet has already started and i wonder if that is contributing to the observed failures. probably not given this is a cadvisor related problem. to verify that i would write an arbitrary static pod manifest with the same request in : before (which also starts the kubelet) right after the kubelet has started if is causing such racy problems i don't think it should be graduated to GA. though, it has been Beta for ~12 releases (on by default), so it's already used everywhere.
Based on your assessment do you think we should remove the ephemeral-storage requirement from default kubeadm etcd static pod? I can make that change.
based on the investigations here (thanks all), it feels like is a feature that may or may not work (for static pods at least). it doesn't feel right to mitigate a kubelet bug on the side of the deployer, but given how many reports there are about this i think it should be done at this point. happy to LGTM approve your PRs. note it must also be backported to all versions in support (back to 1.19). suggesting the following release note: we should keep this issue open, too.
Sounds good. I'll submit a PR soon with the kubeadm change, and I'll continue to drive toward a root cause solution on the kubelet side. I'll bring this up at sig-node tomorrow. Thanks!
is a possible duplicate of this issue - seeing a CrashLoopBackOff failure for kube-apiserver under a similar scenario to those mentioned here, e.g. failure to join a third control plane node with mentions of in the kubelet logs.
An ugly as sin workaroud: (please don't hate me :D )
It seems like this could be closed with and related cherry picks now being part of the newest releases? Bumping to 1.21.2 allowed me to remove the hack for etcd and several test deployments have steered clear entirely of the original issue. Thanks
great news. I'd prefer to pin this issue to this PR: That PR will deliver more race condition resilience for all static pod + ephemeral-storage requirement scenarios.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:

return err != nil && os.IsNotExist(err) } <ins> // GetCapacity returns node capacity data for "cpu", "memory", "ephemeral-storage", and "huge-pages*" // At present this method is only invoked when introspecting ephemeral storage </ins> func (cm *containerManagerImpl) GetCapacity() v1.ResourceList { <ins> if utilfeature.DefaultFeatureGate.Enabled(kubefeatures.LocalStorageCapacityIsolation) { // We store allocatable ephemeral-storage in the capacity property once we Start() the container manager if _, ok := cm.capacity[v1.ResourceEphemeralStorage]; !ok { // If we haven't yet stored the capacity for ephemeral-storage, we can try to fetch it directly from cAdvisor, if cm.cadvisorInterface != nil { rootfs, err := cm.cadvisorInterface.RootFsInfo() if err != nil { klog.ErrorS(err, "Unable to get rootfs data from cAdvisor interface") // If the rootfsinfo retrieval from cAdvisor fails for any reason, fallback to returning the capacity property with no ephemeral storage data return cm.capacity } // We don't want to mutate cm.capacity here so we'll manually construct a v1.ResourceList from it, // and add ephemeral-storage capacityWithEphemeralStorage := v1.ResourceList{} for rName, rQuant := range cm.capacity { capacityWithEphemeralStorage[rName] = rQuant } capacityWithEphemeralStorage[v1.ResourceEphemeralStorage] = cadvisor.EphemeralStorageCapacityFromFsInfo(rootfs)[v1.ResourceEphemeralStorage] return capacityWithEphemeralStorage } } } </ins> return cm.capacity }


Question:
on the new node fails sometimes. The following error appears in the log: on the new node succeeds every time Join a control plane node The node has over 40 GB of free space on its root partition. I some more logging to the following functions: , , , , , , It appears that the ephemeral storage is detected a bit too late: /sig node storage Kubernetes version (use ): 1.20.4, commit Kernel (e.g. ): 4.9.252-1.ph2
/assign I will try to reproduce it today.
LocalStorageCapacityIsolation is a feature gate for kubelet. Can you check if you disable it? ReadyCondition will check ephemeral-storage as required capacities if this feature gate is enabled in kubelet configuration. Would you check it first?
[x] 1.8 alpha ephemeral-storage [x] 1.10 Promote LocalStorageCapacityIsolation feature to beta [x] 1.15 KEPs Quotas for ephemeral-storage for LocalStorageCapacityIsolationFSQuotaMonitoring default to false is Alpha since 1.15 [x] Failed to promote LocalStorageCapacityIsolationFSQuotaMonitoring feature to beta [x] 1.20 Honor disabled LocalStorageCapacityIsolation in scheduling I am no sure whether there is a plan to GA LocalStorageCapacityIsolation?
I tried to fix it in . Need some testing later.
It appears that the pod fails before sees the storage capacity:
Would you try this ? I build it with on PR branch.
I tried to apply the PR to 1.20.4. After four successful s, one failed. I got the following error:
! It means that during the node resource updated the first time, the cadvisor is not available yet. If we wait here, it would be a problem. I would think more of how to fix it.
Hi, Same problem here. I'm using K8s v1.20.4 with CentOS7 (2 cores, 4GB and 20GB storage) Cluster creation with kubeadm init always works, but when I try to add several master nodes with kubeadm join it takes a lot of time waiting for etcd pod to be created. In most of cases, etcd pod never runs. The rest of components (APIserver, controller, scheduler and proxy) exists and works (except APIServer which reboots trying to connect the missing etcd) Checking the kubelet logs in the missing etcd hosts I have found this: Mar 10 14:23:27 kmaster-02 kubelet[9448]: W0310 14:23:27. 9448 ] Failed to admit pod etcd-kmaster-02_kube-system() - Unexpected error while attempting to recover from admission failure: preemption: error finding a set of pods to preempt: no set of running pods found to reclaim resources: [(res: ephemeral-storage, q: ), ] That node have 17GB storage free. If kubelet is restarted, the etcd (pod and container in docker ps -a) appears. If I could help doing something please tell me. I would like to help
For anyone who is experiencing this issue and still wants to use kubeadm until it is fixed: You can remove the ephemeral-storage request from the etcd static pod and the join will work. Just make sure that your nodes actually have more than 100MB free storage left. You can reinsert the storage request by patching the etcd manifests after the nodes are up. To patch etcd during a create the following file (the filename matters, the file location doesn't) and use to join additional control plane nodes.
Thanks for the workaround I'm encountering this same issue and the workaround worked for me. Edit: the patch I tried doesn't work though (not sure if this is really a problem). I issued the command: and the response shows that the resource can't be patched this way: There may be another way but I'm still fairly new to Kubernetes so it is not obvious to me at the moment. Anyhow, thanks for the suggestion!
I updated Add a start-delay of 10 seconds for a condition that the capacity of node ephemeral-storage is 0. Not sure if this will fix it.
we are starting to see reports about this every week.
cc
cc
From it sounds like the issue is kubeadm etcd ephemeral-storage requests are too big for some environments? So far this is sounding like a kubeadm issue, not a kubernetes issue.
it seems to be something else. i doubt that users are trying machines that don't satisfy 100Mi. for example, the report in the OP points out that the machine has 40GB.
I can confirm that the VMs on which I installed k8s control plane components had sufficient disk space when I encountered the problem. I'm running Kubernetes v1.20.4 and kubeadm v1.20.4.
I think the root cause here sounds right. There is kind of race here. adding delay will help, but it does not have 100% guarantee. Not sure whether it is easy to add logic to make sure capacity is reported correctly before starting pods..
Did something change in 1.20 regarding the ordering of cadvisor and node startup?
If the reason would be a disk problem(or something we can figure out) and the delay can solve it, I think we should add a warning besides the delay.
We need to figure out what triggers the race condition. This only happens on some machines. I have not seen this myself.
We're able to repro this nearly 100% of the time with Cluster API Provider Azure + k8s 1.20.x + Ubuntu 18.04 + default template, which makes k8s 1.20 pretty much unusable at this point. Trying to see if this repros with other non-Azure cluster-api providers with kubeadm 1.20.
Does anyone know why this only repros on and not ? The ephemeral storage resource requirement here is used in common by both and : In practice I can workaround this issue in cluster-api + Azure by adding a sleep command to my spec, but it requires a significant number to make work ~100% of the time (like 90 seconds). Does the flow take significantly longer before it gets to the part where it submits an etcd pod to the apiserver? Is there any other explanation for this being a -only symptom?
i can see this failing for any static pod with ephemeral storage requests. that is, unless there is proof that this is somehow specific to the kubeadm static pod for etcd that kubeadm deploys.
I agree that the specific kubeadm-delivered etcd pod is not interesting. (In fact it's very permissive, it's only asking for 100Mi.) What definitely seems to be happening (in Azure) is that kubelet takes a long time to know about its underlying node's ephemeral-storage availability. Kubelet starts: ~2 mins later, it complains that it can't schedule etcd: Again, why does this not repro on nodes created by ?
it also seems to be an even more scoped problem because it's not reproducible for me locally (i create a number of clusters every day, usually), the kubeadm CI or in other high level projects that use kubeadm (e.g. kind, minikube, etc). we should try to find a common reason / condition from all the reporting users that are seeing this.
Experiencing same problem on openSuSE Kubic/Tumbletweed snapshot with kubeadm/kubelet 1.20.2, kernel 5.11.15 running on Xen, 2 CPU, 4GB ram 30GB disk (BRTFS)
Problem consistently appear/disappear when switching between snapshots (Working) and , so a package update in that period has probably triggered this. Following files contains changes between those two snapshots.
are you suggesting a root cause in underlying Linux storage driver surface area that is adding delay/uncertainty to kubelet's ability to introspect available ephemeral-storage on the node host OS?
A bit of luck combined with 6th sense i got it narrowed down to systemd update here Looking at the changelog, one item sticks out: Anyone else enabled cgroupsv2 on i.e. the CentOS cases (it needs to be enabled explicitly for that AFAIK)?
ACK on your latest respond regarding unified cgroup hierarchy. Is there a quick, cross-distro way to verify that any folks who are repro'ing have a similar systemd config?
The precence of file/dir /sys/fs/cgroup/cgroup.controllers should mean you are using cgroupsv2 ( Ref: ) I can confirm that the file is present on the kubic snapshot which is not working. It is not present on the one that does work. ( I'm using cri-o as runtime which should support cgroupsv2 ) cgroup2 support was introduced for K8S in 1.19 ( ).
Thanks. So tl;dr your working theory is that cgroupsv2 + kubelet ephemeral-storage admission has some pathologies. I'll post quickly with what my repro containerd + Ubuntu 18.04-LTS OS surface area looks like.
I ran into this issue consistently on Ubuntu 20.04.2 LTS using cri-o but is not present on my systems.
I can also confirm that cgroupsv2 is not a factor in my repro'ing environment:
Also, I'm not using cri-o, containerd here:
Ok - so much for that theory. But still: What I did to narrow it to systemd-update was taking the working kubic snapshot () and run zypper update systemd which updates systemd (along with systemd-sysvinit and udev packages to same version). That consistently made it fail. So I think a pointer to the root-cause might still be hiding somewhere around this changelog - even if it is not cgroupsv2. That part was just a theory (seemed like a "big" thing), but the update of systemd package is fairly hard evidence.
I have identified where these particular symptoms (probably) originate, and cursory testing suggests something like this eliminates the issue: I'll be advocating for a few mins in today's sig-node discussion if any folks following this thread want to chime in. cc
/sig cluster-lifecycle /triage accepted
I've also observed that on some occasions the node allocatable storage is in a continual state of reporting "0". Restarting kubelet immediately fixed that. More debugging to do....
did you manage to reproduce this with an arbitrary static pod that is not the etcd one?
I was not able to reproduce in a non-kubeadm test, but I suspect that the way kubelet bootstrap is implemented there (and possibly environmental reasons) the static pod handling was always happening after this code executed: (the above is in the containerManager method) kubeadm w/ ephemeral-storage requirements in etcd is the most effective repro at this time
the writing of the etcd pod manifest in during is after the kubelet has already started and i wonder if that is contributing to the observed failures. probably not given this is a cadvisor related problem. to verify that i would write an arbitrary static pod manifest with the same request in : before (which also starts the kubelet) right after the kubelet has started if is causing such racy problems i don't think it should be graduated to GA. though, it has been Beta for ~12 releases (on by default), so it's already used everywhere.
Based on your assessment do you think we should remove the ephemeral-storage requirement from default kubeadm etcd static pod? I can make that change.
based on the investigations here (thanks all), it feels like is a feature that may or may not work (for static pods at least). it doesn't feel right to mitigate a kubelet bug on the side of the deployer, but given how many reports there are about this i think it should be done at this point. happy to LGTM approve your PRs. note it must also be backported to all versions in support (back to 1.19). suggesting the following release note: we should keep this issue open, too.
Sounds good. I'll submit a PR soon with the kubeadm change, and I'll continue to drive toward a root cause solution on the kubelet side. I'll bring this up at sig-node tomorrow. Thanks!
is a possible duplicate of this issue - seeing a CrashLoopBackOff failure for kube-apiserver under a similar scenario to those mentioned here, e.g. failure to join a third control plane node with mentions of in the kubelet logs.
An ugly as sin workaroud: (please don't hate me :D )
It seems like this could be closed with and related cherry picks now being part of the newest releases? Bumping to 1.21.2 allowed me to remove the hack for etcd and several test deployments have steered clear entirely of the original issue. Thanks
great news. I'd prefer to pin this issue to this PR: That PR will deliver more race condition resilience for all static pod + ephemeral-storage requirement scenarios.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:

package cm import ( <ins> "errors" </ins> "io/ioutil" "os" "path" "testing" <ins> gomock "github.com/golang/mock/gomock" cadvisorapiv2 "github.com/google/cadvisor/info/v2" utilfeature "k8s.io/apiserver/pkg/util/feature" featuregatetesting "k8s.io/component-base/featuregate/testing" kubefeatures "k8s.io/kubernetes/pkg/features" </ins> "github.com/opencontainers/runc/libcontainer/cgroups" "github.com/stretchr/testify/assert" "github.com/stretchr/testify/require" <ins> v1 "k8s.io/api/core/v1" "k8s.io/apimachinery/pkg/api/resource" cadvisortest "k8s.io/kubernetes/pkg/kubelet/cadvisor/testing" </ins> "k8s.io/mount-utils" )


Question:
on the new node fails sometimes. The following error appears in the log: on the new node succeeds every time Join a control plane node The node has over 40 GB of free space on its root partition. I some more logging to the following functions: , , , , , , It appears that the ephemeral storage is detected a bit too late: /sig node storage Kubernetes version (use ): 1.20.4, commit Kernel (e.g. ): 4.9.252-1.ph2
/assign I will try to reproduce it today.
LocalStorageCapacityIsolation is a feature gate for kubelet. Can you check if you disable it? ReadyCondition will check ephemeral-storage as required capacities if this feature gate is enabled in kubelet configuration. Would you check it first?
[x] 1.8 alpha ephemeral-storage [x] 1.10 Promote LocalStorageCapacityIsolation feature to beta [x] 1.15 KEPs Quotas for ephemeral-storage for LocalStorageCapacityIsolationFSQuotaMonitoring default to false is Alpha since 1.15 [x] Failed to promote LocalStorageCapacityIsolationFSQuotaMonitoring feature to beta [x] 1.20 Honor disabled LocalStorageCapacityIsolation in scheduling I am no sure whether there is a plan to GA LocalStorageCapacityIsolation?
I tried to fix it in . Need some testing later.
It appears that the pod fails before sees the storage capacity:
Would you try this ? I build it with on PR branch.
I tried to apply the PR to 1.20.4. After four successful s, one failed. I got the following error:
! It means that during the node resource updated the first time, the cadvisor is not available yet. If we wait here, it would be a problem. I would think more of how to fix it.
Hi, Same problem here. I'm using K8s v1.20.4 with CentOS7 (2 cores, 4GB and 20GB storage) Cluster creation with kubeadm init always works, but when I try to add several master nodes with kubeadm join it takes a lot of time waiting for etcd pod to be created. In most of cases, etcd pod never runs. The rest of components (APIserver, controller, scheduler and proxy) exists and works (except APIServer which reboots trying to connect the missing etcd) Checking the kubelet logs in the missing etcd hosts I have found this: Mar 10 14:23:27 kmaster-02 kubelet[9448]: W0310 14:23:27. 9448 ] Failed to admit pod etcd-kmaster-02_kube-system() - Unexpected error while attempting to recover from admission failure: preemption: error finding a set of pods to preempt: no set of running pods found to reclaim resources: [(res: ephemeral-storage, q: ), ] That node have 17GB storage free. If kubelet is restarted, the etcd (pod and container in docker ps -a) appears. If I could help doing something please tell me. I would like to help
For anyone who is experiencing this issue and still wants to use kubeadm until it is fixed: You can remove the ephemeral-storage request from the etcd static pod and the join will work. Just make sure that your nodes actually have more than 100MB free storage left. You can reinsert the storage request by patching the etcd manifests after the nodes are up. To patch etcd during a create the following file (the filename matters, the file location doesn't) and use to join additional control plane nodes.
Thanks for the workaround I'm encountering this same issue and the workaround worked for me. Edit: the patch I tried doesn't work though (not sure if this is really a problem). I issued the command: and the response shows that the resource can't be patched this way: There may be another way but I'm still fairly new to Kubernetes so it is not obvious to me at the moment. Anyhow, thanks for the suggestion!
I updated Add a start-delay of 10 seconds for a condition that the capacity of node ephemeral-storage is 0. Not sure if this will fix it.
we are starting to see reports about this every week.
cc
cc
From it sounds like the issue is kubeadm etcd ephemeral-storage requests are too big for some environments? So far this is sounding like a kubeadm issue, not a kubernetes issue.
it seems to be something else. i doubt that users are trying machines that don't satisfy 100Mi. for example, the report in the OP points out that the machine has 40GB.
I can confirm that the VMs on which I installed k8s control plane components had sufficient disk space when I encountered the problem. I'm running Kubernetes v1.20.4 and kubeadm v1.20.4.
I think the root cause here sounds right. There is kind of race here. adding delay will help, but it does not have 100% guarantee. Not sure whether it is easy to add logic to make sure capacity is reported correctly before starting pods..
Did something change in 1.20 regarding the ordering of cadvisor and node startup?
If the reason would be a disk problem(or something we can figure out) and the delay can solve it, I think we should add a warning besides the delay.
We need to figure out what triggers the race condition. This only happens on some machines. I have not seen this myself.
We're able to repro this nearly 100% of the time with Cluster API Provider Azure + k8s 1.20.x + Ubuntu 18.04 + default template, which makes k8s 1.20 pretty much unusable at this point. Trying to see if this repros with other non-Azure cluster-api providers with kubeadm 1.20.
Does anyone know why this only repros on and not ? The ephemeral storage resource requirement here is used in common by both and : In practice I can workaround this issue in cluster-api + Azure by adding a sleep command to my spec, but it requires a significant number to make work ~100% of the time (like 90 seconds). Does the flow take significantly longer before it gets to the part where it submits an etcd pod to the apiserver? Is there any other explanation for this being a -only symptom?
i can see this failing for any static pod with ephemeral storage requests. that is, unless there is proof that this is somehow specific to the kubeadm static pod for etcd that kubeadm deploys.
I agree that the specific kubeadm-delivered etcd pod is not interesting. (In fact it's very permissive, it's only asking for 100Mi.) What definitely seems to be happening (in Azure) is that kubelet takes a long time to know about its underlying node's ephemeral-storage availability. Kubelet starts: ~2 mins later, it complains that it can't schedule etcd: Again, why does this not repro on nodes created by ?
it also seems to be an even more scoped problem because it's not reproducible for me locally (i create a number of clusters every day, usually), the kubeadm CI or in other high level projects that use kubeadm (e.g. kind, minikube, etc). we should try to find a common reason / condition from all the reporting users that are seeing this.
Experiencing same problem on openSuSE Kubic/Tumbletweed snapshot with kubeadm/kubelet 1.20.2, kernel 5.11.15 running on Xen, 2 CPU, 4GB ram 30GB disk (BRTFS)
Problem consistently appear/disappear when switching between snapshots (Working) and , so a package update in that period has probably triggered this. Following files contains changes between those two snapshots.
are you suggesting a root cause in underlying Linux storage driver surface area that is adding delay/uncertainty to kubelet's ability to introspect available ephemeral-storage on the node host OS?
A bit of luck combined with 6th sense i got it narrowed down to systemd update here Looking at the changelog, one item sticks out: Anyone else enabled cgroupsv2 on i.e. the CentOS cases (it needs to be enabled explicitly for that AFAIK)?
ACK on your latest respond regarding unified cgroup hierarchy. Is there a quick, cross-distro way to verify that any folks who are repro'ing have a similar systemd config?
The precence of file/dir /sys/fs/cgroup/cgroup.controllers should mean you are using cgroupsv2 ( Ref: ) I can confirm that the file is present on the kubic snapshot which is not working. It is not present on the one that does work. ( I'm using cri-o as runtime which should support cgroupsv2 ) cgroup2 support was introduced for K8S in 1.19 ( ).
Thanks. So tl;dr your working theory is that cgroupsv2 + kubelet ephemeral-storage admission has some pathologies. I'll post quickly with what my repro containerd + Ubuntu 18.04-LTS OS surface area looks like.
I ran into this issue consistently on Ubuntu 20.04.2 LTS using cri-o but is not present on my systems.
I can also confirm that cgroupsv2 is not a factor in my repro'ing environment:
Also, I'm not using cri-o, containerd here:
Ok - so much for that theory. But still: What I did to narrow it to systemd-update was taking the working kubic snapshot () and run zypper update systemd which updates systemd (along with systemd-sysvinit and udev packages to same version). That consistently made it fail. So I think a pointer to the root-cause might still be hiding somewhere around this changelog - even if it is not cgroupsv2. That part was just a theory (seemed like a "big" thing), but the update of systemd package is fairly hard evidence.
I have identified where these particular symptoms (probably) originate, and cursory testing suggests something like this eliminates the issue: I'll be advocating for a few mins in today's sig-node discussion if any folks following this thread want to chime in. cc
/sig cluster-lifecycle /triage accepted
I've also observed that on some occasions the node allocatable storage is in a continual state of reporting "0". Restarting kubelet immediately fixed that. More debugging to do....
did you manage to reproduce this with an arbitrary static pod that is not the etcd one?
I was not able to reproduce in a non-kubeadm test, but I suspect that the way kubelet bootstrap is implemented there (and possibly environmental reasons) the static pod handling was always happening after this code executed: (the above is in the containerManager method) kubeadm w/ ephemeral-storage requirements in etcd is the most effective repro at this time
the writing of the etcd pod manifest in during is after the kubelet has already started and i wonder if that is contributing to the observed failures. probably not given this is a cadvisor related problem. to verify that i would write an arbitrary static pod manifest with the same request in : before (which also starts the kubelet) right after the kubelet has started if is causing such racy problems i don't think it should be graduated to GA. though, it has been Beta for ~12 releases (on by default), so it's already used everywhere.
Based on your assessment do you think we should remove the ephemeral-storage requirement from default kubeadm etcd static pod? I can make that change.
based on the investigations here (thanks all), it feels like is a feature that may or may not work (for static pods at least). it doesn't feel right to mitigate a kubelet bug on the side of the deployer, but given how many reports there are about this i think it should be done at this point. happy to LGTM approve your PRs. note it must also be backported to all versions in support (back to 1.19). suggesting the following release note: we should keep this issue open, too.
Sounds good. I'll submit a PR soon with the kubeadm change, and I'll continue to drive toward a root cause solution on the kubelet side. I'll bring this up at sig-node tomorrow. Thanks!
is a possible duplicate of this issue - seeing a CrashLoopBackOff failure for kube-apiserver under a similar scenario to those mentioned here, e.g. failure to join a third control plane node with mentions of in the kubelet logs.
An ugly as sin workaroud: (please don't hate me :D )
It seems like this could be closed with and related cherry picks now being part of the newest releases? Bumping to 1.21.2 allowed me to remove the hack for etcd and several test deployments have steered clear entirely of the original issue. Thanks
great news. I'd prefer to pin this issue to this PR: That PR will deliver more race condition resilience for all static pod + ephemeral-storage requirement scenarios.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:

assert.NoError(t, err) assert.True(t, f.cpuHardcapping, "cpu hardcapping is expected to be enabled") } <ins> func TestGetCapacity(t *testing.T) { ephemeralStorageFromCapacity := int64(2000) ephemeralStorageFromCadvisor := int64(8000) mockCtrl := gomock.NewController(t) defer mockCtrl.Finish() mockCtrlError := gomock.NewController(t) defer mockCtrlError.Finish() mockCadvisor := cadvisortest.NewMockInterface(mockCtrl) rootfs := cadvisorapiv2.FsInfo{ Capacity: 8000, } mockCadvisor.EXPECT().RootFsInfo().Return(rootfs, nil) mockCadvisorError := cadvisortest.NewMockInterface(mockCtrlError) mockCadvisorError.EXPECT().RootFsInfo().Return(cadvisorapiv2.FsInfo{}, errors.New("Unable to get rootfs data from cAdvisor interface")) cases := []struct { name                                string cm                                  *containerManagerImpl expectedResourceQuantity            *resource.Quantity expectedNoEphemeralStorage          bool enableLocalStorageCapacityIsolation bool }{ { name: "capacity property has ephemeral-storage", cm: &containerManagerImpl{ cadvisorInterface: mockCadvisor, capacity: v1.ResourceList{ v1.ResourceEphemeralStorage: *resource.NewQuantity(ephemeralStorageFromCapacity, resource.BinarySI), }, }, expectedResourceQuantity:            resource.NewQuantity(ephemeralStorageFromCapacity, resource.BinarySI), expectedNoEphemeralStorage:          false, enableLocalStorageCapacityIsolation: true, }, { name: "capacity property does not have ephemeral-storage", cm: &containerManagerImpl{ cadvisorInterface: mockCadvisor, capacity:          v1.ResourceList{}, }, expectedResourceQuantity:            resource.NewQuantity(ephemeralStorageFromCadvisor, resource.BinarySI), expectedNoEphemeralStorage:          false, enableLocalStorageCapacityIsolation: true, }, { name: "capacity property does not have ephemeral-storage, error from rootfs", cm: &containerManagerImpl{ cadvisorInterface: mockCadvisorError, capacity:          v1.ResourceList{}, }, expectedNoEphemeralStorage:          true, enableLocalStorageCapacityIsolation: true, }, { name: "capacity property does not have ephemeral-storage, cadvisor interface is nil", cm: &containerManagerImpl{ cadvisorInterface: nil, capacity:          v1.ResourceList{}, }, expectedNoEphemeralStorage:          true, enableLocalStorageCapacityIsolation: true, }, { name: "LocalStorageCapacityIsolation feature flag is disabled", cm: &containerManagerImpl{ cadvisorInterface: mockCadvisor, capacity: v1.ResourceList{ v1.ResourceCPU:    resource.MustParse("4"), v1.ResourceMemory: resource.MustParse("16G"), }, }, expectedNoEphemeralStorage:          true, enableLocalStorageCapacityIsolation: false, }, } for _, c := range cases { t.Run(c.name, func(t *testing.T) { defer featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, kubefeatures.LocalStorageCapacityIsolation, c.enableLocalStorageCapacityIsolation)() ret := c.cm.GetCapacity() if v, exists := ret[v1.ResourceEphemeralStorage]; !exists { if !c.expectedNoEphemeralStorage { t.Errorf("did not get any ephemeral storage data") } } else { if v.Value() != c.expectedResourceQuantity.Value() { t.Errorf("got unexpected %s value, expected %d, got %d", v1.ResourceEphemeralStorage, c.expectedResourceQuantity.Value(), v.Value()) } } }) } } </ins>


Question:
Currently whenever a vSphere volume is provisioned, we are looking at all nodes in the cluster and then fetching their hosts and then their datastores and then ensuring that selected datastore in present on all the nodes. This is done every time a new PVC is provisioned. It can be pretty expensive in large clusters and number of involved API calls can be big. /sig storage /area vsphere cc
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : Currently whenever a vSphere volume is provisioned, we are looking at all nodes in the cluster and then fetching their hosts and then their datastores and then ensuring that selected datastore in present on all the nodes. This is done every time a new PVC is provisioned. /area vsphere Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/triage accepted
/assign

<ins> // +build !providerless /* Copyright 2021 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package vsphere import ( "context" "fmt" "github.com/vmware/govmomi/property" "github.com/vmware/govmomi/vim25/mo" "github.com/vmware/govmomi/vim25/types" "k8s.io/klog/v2" "k8s.io/legacy-cloud-providers/vsphere/vclib" ) type sharedDatastore struct { nodeManager         *NodeManager candidateDatastores []*vclib.DatastoreInfo } type hostInfo struct { hostUUID   string hostMOID   string datacenter string } const ( summary       = "summary" runtimeHost   = "summary.runtime.host" hostsProperty = "host" nameProperty  = "name" ) func (shared *sharedDatastore) getSharedDatastore(ctcx context.Context) (*vclib.DatastoreInfo, error) { nodes := shared.nodeManager.getNodes() // Segregate nodes according to VC-DC dcNodes := make(map[string][]NodeInfo) nodeHosts := make(map[string]hostInfo) for nodeName, node := range nodes { nodeInfo, err := shared.nodeManager.GetNodeInfoWithNodeObject(node) if err != nil { return nil, fmt.Errorf("unable to find node %s: %v", nodeName, err) } vcDC := nodeInfo.vcServer + nodeInfo.dataCenter.String() dcNodes[vcDC] = append(dcNodes[vcDC], nodeInfo) } for vcDC, nodes := range dcNodes { var hostInfos []hostInfo var err error hostInfos, err = shared.getNodeHosts(ctcx, nodes, vcDC) if err != nil { if vclib.IsManagedObjectNotFoundError(err) { klog.Warningf("SharedHost.getSharedDatastore: batch fetching of hosts failed - switching to fetching them individually.") hostInfos, err = shared.getEachNodeHost(ctcx, nodes, vcDC) if err != nil { klog.Errorf("SharedHost.getSharedDatastore: error fetching node hosts individually: %v", err) return nil, err } } else { return nil, err } } for _, host := range hostInfos { hostDCName := fmt.Sprintf("%s/%s", host.datacenter, host.hostMOID) nodeHosts[hostDCName] = host } } if len(nodeHosts) < 1 { msg := fmt.Sprintf("SharedHost.getSharedDatastore unable to find hosts associated with nodes") klog.Error(msg) return nil, fmt.Errorf("") } for _, datastoreInfo := range shared.candidateDatastores { dataStoreHosts, err := shared.getAttachedHosts(ctcx, datastoreInfo.Datastore) if err != nil { msg := fmt.Sprintf("error finding attached hosts to datastore %s: %v", datastoreInfo.Name(), err) klog.Error(msg) return nil, fmt.Errorf(msg) } if shared.isIncluded(dataStoreHosts, nodeHosts) { return datastoreInfo, nil } } return nil, fmt.Errorf("SharedHost.getSharedDatastore: unable to find any shared datastores") } // check if all of the nodeHosts are included in the dataStoreHosts func (shared *sharedDatastore) isIncluded(dataStoreHosts []hostInfo, nodeHosts map[string]hostInfo) bool { result := true for _, host := range nodeHosts { hostFound := false for _, targetHost := range dataStoreHosts { if host.hostUUID == targetHost.hostUUID && host.hostMOID == targetHost.hostMOID { hostFound = true } } if !hostFound { result = false } } return result } func (shared *sharedDatastore) getEachNodeHost(ctx context.Context, nodes []NodeInfo, dcVC string) ([]hostInfo, error) { var hosts []hostInfo for _, node := range nodes { host, err := node.vm.GetHost(ctx) if err != nil { klog.Errorf("SharedHost.getEachNodeHost: unable to find host for vm %s: %v", node.vm.InventoryPath, err) return nil, err } hosts = append(hosts, hostInfo{ hostUUID:   host.Summary.Hardware.Uuid, hostMOID:   host.Summary.Host.String(), datacenter: node.dataCenter.String(), }) } return hosts, nil } func (shared *sharedDatastore) getNodeHosts(ctx context.Context, nodes []NodeInfo, dcVC string) ([]hostInfo, error) { var vmRefs []types.ManagedObjectReference if len(nodes) < 1 { return nil, fmt.Errorf("no nodes found for dc-vc: %s", dcVC) } var nodeInfo NodeInfo for _, n := range nodes { nodeInfo = n vmRefs = append(vmRefs, n.vm.Reference()) } pc := property.DefaultCollector(nodeInfo.dataCenter.Client()) var vmoList []mo.VirtualMachine err := pc.Retrieve(ctx, vmRefs, []string{nameProperty, runtimeHost}, &vmoList) if err != nil { klog.Errorf("SharedHost.getNodeHosts: unable to fetch vms from datacenter %s: %w", nodeInfo.dataCenter.String(), err) return nil, err } var hostMoList []mo.HostSystem var hostRefs []types.ManagedObjectReference for _, vmo := range vmoList { if vmo.Summary.Runtime.Host == nil { msg := fmt.Sprintf("SharedHost.getNodeHosts: no host associated with vm %s", vmo.Name) klog.Error(msg) return nil, fmt.Errorf(msg) } hostRefs = append(hostRefs, vmo.Summary.Runtime.Host.Reference()) } pc = property.DefaultCollector(nodeInfo.dataCenter.Client()) err = pc.Retrieve(ctx, hostRefs, []string{summary}, &hostMoList) if err != nil { klog.Errorf("SharedHost.getNodeHosts: unable to fetch hosts from datacenter %s: %w", nodeInfo.dataCenter.String(), err) return nil, err } var hosts []hostInfo for _, host := range hostMoList { hosts = append(hosts, hostInfo{hostMOID: host.Summary.Host.String(), hostUUID: host.Summary.Hardware.Uuid, datacenter: nodeInfo.dataCenter.String()}) } return hosts, nil } func (shared *sharedDatastore) getAttachedHosts(ctx context.Context, datastore *vclib.Datastore) ([]hostInfo, error) { var ds mo.Datastore pc := property.DefaultCollector(datastore.Client()) err := pc.RetrieveOne(ctx, datastore.Reference(), []string{hostsProperty}, &ds) if err != nil { return nil, err } mounts := make(map[types.ManagedObjectReference]types.DatastoreHostMount) var refs []types.ManagedObjectReference for _, host := range ds.Host { refs = append(refs, host.Key) mounts[host.Key] = host } var hs []mo.HostSystem err = pc.Retrieve(ctx, refs, []string{summary}, &hs) if err != nil { return nil, err } var hosts []hostInfo for _, h := range hs { hosts = append(hosts, hostInfo{hostUUID: h.Summary.Hardware.Uuid, hostMOID: h.Summary.Host.String()}) } return hosts, nil } </ins>


Question:
Currently whenever a vSphere volume is provisioned, we are looking at all nodes in the cluster and then fetching their hosts and then their datastores and then ensuring that selected datastore in present on all the nodes. This is done every time a new PVC is provisioned. It can be pretty expensive in large clusters and number of involved API calls can be big. /sig storage /area vsphere cc
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : Currently whenever a vSphere volume is provisioned, we are looking at all nodes in the cluster and then fetching their hosts and then their datastores and then ensuring that selected datastore in present on all the nodes. This is done every time a new PVC is provisioned. /area vsphere Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/triage accepted
/assign

return diskUUID, nil } <ins> // GetHost returns host of the virtual machine func (vm *VirtualMachine) GetHost(ctx context.Context) (mo.HostSystem, error) { host, err := vm.HostSystem(ctx) var hostSystemMo mo.HostSystem if err != nil { klog.Errorf("Failed to get host system for VM: %q. err: %+v", vm.InventoryPath, err) return hostSystemMo, err } s := object.NewSearchIndex(vm.Client()) err = s.Properties(ctx, host.Reference(), []string{"summary"}, &hostSystemMo) if err != nil { klog.Errorf("Failed to retrieve datastores for host: %+v. err: %+v", host, err) return hostSystemMo, err } return hostSystemMo, nil } </ins> // DetachDisk detaches the disk specified by vmDiskPath func (vm *VirtualMachine) DetachDisk(ctx context.Context, vmDiskPath string) error { device, err := vm.getVirtualDeviceByPath(ctx, vmDiskPath)


Question:
Currently whenever a vSphere volume is provisioned, we are looking at all nodes in the cluster and then fetching their hosts and then their datastores and then ensuring that selected datastore in present on all the nodes. This is done every time a new PVC is provisioned. It can be pretty expensive in large clusters and number of involved API calls can be big. /sig storage /area vsphere cc
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : Currently whenever a vSphere volume is provisioned, we are looking at all nodes in the cluster and then fetching their hosts and then their datastores and then ensuring that selected datastore in present on all the nodes. This is done every time a new PVC is provisioned. /area vsphere Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/triage accepted
/assign

if len(zonesToSearch) == 0 { // If zone is not provided, get the shared datastore across all node VMs. klog.V(4).Infof("Validating if datastore %s is shared across all node VMs", datastoreName) <del> sharedDsList, err = getSharedDatastoresInK8SCluster(ctx, vs.nodeManager) </del> <ins> sharedDSFinder := &sharedDatastore{ nodeManager:         vs.nodeManager, candidateDatastores: candidateDatastoreInfos, } datastoreInfo, err = sharedDSFinder.getSharedDatastore(ctx) </ins> if err != nil { klog.Errorf("Failed to get shared datastore: %+v", err) return "", err } <del> // Prepare error msg to be used later, if required. err = fmt.Errorf("The specified datastore %s is not a shared datastore across node VMs", datastoreName) </del> <ins> if datastoreInfo == nil { err = fmt.Errorf("The specified datastore %s is not a shared datastore across node VMs", datastoreName) klog.Error(err) return "", err } </ins> } else { // If zone is provided, get the shared datastores in that zone. klog.V(4).Infof("Validating if datastore %s is in zone %s ", datastoreName, zonesToSearch)


Question:
Currently whenever a vSphere volume is provisioned, we are looking at all nodes in the cluster and then fetching their hosts and then their datastores and then ensuring that selected datastore in present on all the nodes. This is done every time a new PVC is provisioned. It can be pretty expensive in large clusters and number of involved API calls can be big. /sig storage /area vsphere cc
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : Currently whenever a vSphere volume is provisioned, we are looking at all nodes in the cluster and then fetching their hosts and then their datastores and then ensuring that selected datastore in present on all the nodes. This is done every time a new PVC is provisioned. /area vsphere Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/triage accepted
/assign

klog.Errorf("Failed to find a shared datastore matching zone %s. err: %+v", zonesToSearch, err) return "", err } <del> // Prepare error msg to be used later, if required. err = fmt.Errorf("The specified datastore %s does not match the provided zones : %s", datastoreName, zonesToSearch) } found := false // Check if the selected datastore belongs to the list of shared datastores computed. for _, sharedDs := range sharedDsList { if datastoreInfo, found = candidateDatastores[sharedDs.Info.Url]; found { klog.V(4).Infof("Datastore validation succeeded") found = true break </del> <ins> found := false for _, sharedDs := range sharedDsList { if datastoreInfo, found = candidateDatastores[sharedDs.Info.Url]; found { klog.V(4).Infof("Datastore validation succeeded") found = true break } } if !found { err = fmt.Errorf("The specified datastore %s does not match the provided zones : %s", datastoreName, zonesToSearch) klog.Error(err) return "", err </ins> } <del> } if !found { klog.Error(err) return "", err </del> } } }


Question:
Seen while running on If hack/update- fails, the output should make it clear Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /area code-organization /sig architecture /cc /assign
Maybe we can add a to check the exit code.
/cc
something like that seems like a good idea
usually when we want to do stuff like that we instead record the exit code for specific steps with something like so we can check after and inform the user of the specific failure (and suggest what to do), then
/assign

LOG_FILE="${LOG_FILE:-${TMP_DIR}/update-vendor.log}" kube::log::status "logfile at ${LOG_FILE}" <ins> function finish { ret=$? if [[ ${ret} != 0 ]]; then echo "An error has occurred. Please see more details in ${LOG_FILE}" fi exit ${ret} } trap finish EXIT </ins> if [ -z "${BASH_XTRACEFD:-}" ]; then exec 19> "${LOG_FILE}" export BASH_XTRACEFD="19"


Question:
Scheduler throughput drops significantly in http://perf- ! After , seems to be the curlspirit.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/priority critical-urgent /sig scheduling /sig scalability

"context" "fmt" "math/rand" <del> "sort" </del> "sync" "sync/atomic" "time"


Question:
Scheduler throughput drops significantly in http://perf- ! After , seems to be the curlspirit.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/priority critical-urgent /sig scheduling /sig scalability

return feasibleNodes, nil } <del> type pluginScores struct { Plugin           string Score            int64 AverageNodeScore float64 } </del> // prioritizeNodes prioritizes the nodes by running the score plugins, // which return a score for each node from the call to RunScorePlugins(). // The scores from each plugin are added together to make the score for that node, then


Question:
Scheduler throughput drops significantly in http://perf- ! After , seems to be the curlspirit.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/priority critical-urgent /sig scheduling /sig scalability

return nil, scoreStatus.AsError() } <ins> if klog.V(10).Enabled() { for plugin, nodeScoreList := range scoresMap { for _, nodeScore := range nodeScoreList { klog.InfoS("Plugin scored node for pod", "pod", klog.KObj(pod), "plugin", plugin, "node", nodeScore.Name, "score", nodeScore.Score) } } } </ins> // Summarize all scores. result := make(framework.NodeScoreList, 0, len(nodes)) for i := range nodes { result = append(result, framework.NodeScore{Name: nodes[i].Name, Score: 0}) for j := range scoresMap {


Question:
Scheduler throughput drops significantly in http://perf- ! After , seems to be the curlspirit.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/priority critical-urgent /sig scheduling /sig scalability

} } <del> if klog.V(4).Enabled() { logPluginScores(nodes, scoresMap, pod) } </del> if len(g.extenders) != 0 && nodes != nil { var mu sync.Mutex var wg sync.WaitGroup


Question:
Scheduler throughput drops significantly in http://perf- ! After , seems to be the curlspirit.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/priority critical-urgent /sig scheduling /sig scalability

} } <del> if klog.V(4).Enabled() { </del> <ins> if klog.V(10).Enabled() { </ins> for i := range result { klog.InfoS("Calculated node's final score for pod", "pod", klog.KObj(pod), "node", result[i].Name, "score", result[i].Score) }


Question:
Scheduler throughput drops significantly in http://perf- ! After , seems to be the curlspirit.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/priority critical-urgent /sig scheduling /sig scalability

return result, nil } <del> // logPluginScores adds summarized plugin score logging. The goal of this block is to show the highest scoring plugins on // each node, and the average score for those plugins across all nodes. func logPluginScores(nodes []*v1.Node, scoresMap framework.PluginToNodeScores, pod *v1.Pod) { totalPluginScores := make(map[string]int64) for j := range scoresMap { for i := range nodes { totalPluginScores[j] += scoresMap[j][i].Score } } // Build a map of Nodes->PluginScores on that node nodeToPluginScores := make(framework.PluginToNodeScores, len(nodes)) for _, node := range nodes { nodeToPluginScores[node.Name] = make(framework.NodeScoreList, len(scoresMap)) } // Convert the scoresMap (which contains Plugins->NodeScores) to the Nodes->PluginScores map for plugin, nodeScoreList := range scoresMap { for _, nodeScore := range nodeScoreList { klog.V(10).InfoS("Plugin scored node for pod", "pod", klog.KObj(pod), "plugin", plugin, "node", nodeScore.Name, "score", nodeScore.Score) nodeToPluginScores[nodeScore.Name] = append(nodeToPluginScores[nodeScore.Name], framework.NodeScore{Name: plugin, Score: nodeScore.Score}) } } for node, scores := range nodeToPluginScores { // Get the top 3 scoring plugins for each node sort.Slice(scores, func(i, j int) bool { return scores[i].Score > scores[j].Score }) var topScores []pluginScores for _, score := range scores { pluginScore := pluginScores{ Plugin:           score.Name, Score:            score.Score, AverageNodeScore: float64(totalPluginScores[score.Name]) / float64(len(nodes)), } topScores = append(topScores, pluginScore) if len(topScores) == 3 { break } } klog.InfoS("Top 3 plugins for pod on node", "pod", klog.KObj(pod), "node", node, "scores", topScores, ) } } </del> // NewGenericScheduler creates a genericScheduler object. func NewGenericScheduler( cache internalcache.Cache,


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --Serializing a v2beta2 HorizontalPodAutoscaler with (and ) produces yaml that cannot be applied due to a ValidationError on the field which is under the optional field . A valid HPA serialized with should be able to be applied to the cluster with no validation errors, without filling out the optional field. Other valid objects (such as Namespaces and Deployments) can be applied directly, without filling out their optional fields. Run this to generate the serialized yaml (same outcome with both and functions) Running the go code should yield this yaml: Writing the above yaml to a file and applying it (with ) will result in: Kubernetes version (use ): Server: v1.20.2, Client: v1.20.4 go version: go1.16 darwin/amd64 has: cc cc
/sig api-machinery /sig autoscaling /sig cli
/assign /triage accepted
marking conditions as +optional in the API type would resolve the client-side apply error
It would remove the validation error, but I'd be concerned about any side effects or unwanted outcomes when moving that field from required to optional. A somewhat analogous field is required. But if you anticipate this won't change much else, that will work.
Client-side validation is non-authoritative, and problematic for the reasons described in this issue ... the client is disallowing a request the server would actually have no problems with. No field the server would permit to be empty (e.g. succeeds with ) should be marked as required for this reason. fwiw, openapi schema generation marks fields tagged as as optional:
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
/remove-lifecycle rotten

}, "required": [ "currentReplicas", <del> "desiredReplicas", "conditions" </del> <ins> "desiredReplicas" </ins> ], "type": "object" },


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --Serializing a v2beta2 HorizontalPodAutoscaler with (and ) produces yaml that cannot be applied due to a ValidationError on the field which is under the optional field . A valid HPA serialized with should be able to be applied to the cluster with no validation errors, without filling out the optional field. Other valid objects (such as Namespaces and Deployments) can be applied directly, without filling out their optional fields. Run this to generate the serialized yaml (same outcome with both and functions) Running the go code should yield this yaml: Writing the above yaml to a file and applying it (with ) will result in: Kubernetes version (use ): Server: v1.20.2, Client: v1.20.4 go version: go1.16 darwin/amd64 has: cc cc
/sig api-machinery /sig autoscaling /sig cli
/assign /triage accepted
marking conditions as +optional in the API type would resolve the client-side apply error
It would remove the validation error, but I'd be concerned about any side effects or unwanted outcomes when moving that field from required to optional. A somewhat analogous field is required. But if you anticipate this won't change much else, that will work.
Client-side validation is non-authoritative, and problematic for the reasons described in this issue ... the client is disallowing a request the server would actually have no problems with. No field the server would permit to be empty (e.g. succeeds with ) should be marked as required for this reason. fwiw, openapi schema generation marks fields tagged as as optional:
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
/remove-lifecycle rotten

// conditions is the set of conditions required for this autoscaler to scale its target, // and indicates whether or not those conditions are met. <ins> // +optional </ins> repeated HorizontalPodAutoscalerCondition conditions = 6; }


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --Serializing a v2beta2 HorizontalPodAutoscaler with (and ) produces yaml that cannot be applied due to a ValidationError on the field which is under the optional field . A valid HPA serialized with should be able to be applied to the cluster with no validation errors, without filling out the optional field. Other valid objects (such as Namespaces and Deployments) can be applied directly, without filling out their optional fields. Run this to generate the serialized yaml (same outcome with both and functions) Running the go code should yield this yaml: Writing the above yaml to a file and applying it (with ) will result in: Kubernetes version (use ): Server: v1.20.2, Client: v1.20.4 go version: go1.16 darwin/amd64 has: cc cc
/sig api-machinery /sig autoscaling /sig cli
/assign /triage accepted
marking conditions as +optional in the API type would resolve the client-side apply error
It would remove the validation error, but I'd be concerned about any side effects or unwanted outcomes when moving that field from required to optional. A somewhat analogous field is required. But if you anticipate this won't change much else, that will work.
Client-side validation is non-authoritative, and problematic for the reasons described in this issue ... the client is disallowing a request the server would actually have no problems with. No field the server would permit to be empty (e.g. succeeds with ) should be marked as required for this reason. fwiw, openapi schema generation marks fields tagged as as optional:
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
/remove-lifecycle rotten

// conditions is the set of conditions required for this autoscaler to scale its target, // and indicates whether or not those conditions are met. <ins> // +optional </ins> Conditions []HorizontalPodAutoscalerCondition `json:"conditions" protobuf:"bytes,6,rep,name=conditions"` }