Question: exit 1 fi <del> APIROOT="${KUBE_ROOT}/pkg/api" TMP_APIROOT="${KUBE_ROOT}/_tmp/api" </del> <ins> DIFFROOT="${KUBE_ROOT}/pkg" TMP_DIFFROOT="${KUBE_ROOT}/_tmp/pkg" </ins> _tmp="${KUBE_ROOT}/_tmp" mkdir -p "${_tmp}" <del> cp -a "${APIROOT}" "${TMP_APIROOT}" </del> <ins> cp -a "${DIFFROOT}" "${TMP_DIFFROOT}" </ins> "${KUBE_ROOT}/hack/update-generated-swagger-docs.sh" <del> echo "diffing ${APIROOT} against freshly generated swagger type documentation" </del> <ins> echo "diffing ${DIFFROOT} against freshly generated swagger type documentation" </ins> ret=0 <del> diff -Naupr -I 'Auto generated by' "${APIROOT}" "${TMP_APIROOT}" || ret=$? cp -a "${TMP_APIROOT}" "${KUBE_ROOT}/pkg" </del> <ins> diff -Naupr -I 'Auto generated by' "${DIFFROOT}" "${TMP_DIFFROOT}" || ret=$? cp -a "${TMP_DIFFROOT}/" "${KUBE_ROOT}/" </ins> rm -rf "${_tmp}" if [[ $ret -eq 0 ]] then <del> echo "${APIROOT} up to date." </del> <ins> echo "${DIFFROOT} up to date." </ins> else <del> echo "${APIROOT} is out of date. Please run hack/update-generated-swagger-docs.sh" </del> <ins> echo "${DIFFROOT} is out of date. Please run hack/update-generated-swagger-docs.sh" </ins> exit 1 fi

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Following the example in , I created a pod with 3 sysctls specified in the pod's security context. Setting the sysctl resulted in the pod failing to start. What you expected to happen: I expected the pod to start with no issues. If there is something missing in the docs, I appreciate updating There is also a discussion at that suggests that kernel does not namespace certain net.* sysctls. If that's the case, can we document which sysctls are supported? I assumed is a supported sysctl because it's in the official k8s docs. How to reproduce it (as minimally and precisely as possible): Following the pod creation, the pod doesn't start: Once I delete the following two lines from the pod securityContext, the pod starts fine: Anything else we need to know?: You can see that I've enabled the unsafe sysctls with kubelet: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: VMware Fusion OS (e.g: ):Kernel (e.g. ):Install tools: A single-node k8s cluster created with kubeadm Network plugin and version (if this is a network-related bug): Calico Others:
/sig node
Thank you for the incredibly detailed issue report :) I did some poking around, and if is accurate and still current, then I believe the issue is that is not namespaced (i.e. cannot be set on a per pod/container level). If I'm understanding correctly, we'd need to make changes to the Linux kernel in order for it to be namespaced. As a disclaimer, I'm not positive if this parameter became namespaced in new kernels... it could be interesting to investigate if this still happens on a machine with the latest kernel. W.r.t. fixes... I would agree with you that short term, we need to update this documentation so that it doesn't include . do you have any interest in doing so? If not, I'm happy to.


Question: exit 1 fi <del> APIROOT="${KUBE_ROOT}/pkg/api" TMP_APIROOT="${KUBE_ROOT}/_tmp/api" </del> <ins> DIFFROOT="${KUBE_ROOT}/pkg" TMP_DIFFROOT="${KUBE_ROOT}/_tmp/pkg" </ins> _tmp="${KUBE_ROOT}/_tmp" mkdir -p "${_tmp}" <del> cp -a "${APIROOT}" "${TMP_APIROOT}" </del> <ins> cp -a "${DIFFROOT}" "${TMP_DIFFROOT}" </ins> "${KUBE_ROOT}/hack/update-generated-swagger-docs.sh" <del> echo "diffing ${APIROOT} against freshly generated swagger type documentation" </del> <ins> echo "diffing ${DIFFROOT} against freshly generated swagger type documentation" </ins> ret=0 <del> diff -Naupr -I 'Auto generated by' "${APIROOT}" "${TMP_APIROOT}" || ret=$? cp -a "${TMP_APIROOT}" "${KUBE_ROOT}/pkg" </del> <ins> diff -Naupr -I 'Auto generated by' "${DIFFROOT}" "${TMP_DIFFROOT}" || ret=$? cp -a "${TMP_DIFFROOT}/" "${KUBE_ROOT}/" </ins> rm -rf "${_tmp}" if [[ $ret -eq 0 ]] then <del> echo "${APIROOT} up to date." </del> <ins> echo "${DIFFROOT} up to date." </ins> else <del> echo "${APIROOT} is out of date. Please run hack/update-generated-swagger-docs.sh" </del> <ins> echo "${DIFFROOT} is out of date. Please run hack/update-generated-swagger-docs.sh" </ins> exit 1 fi

Long term, I think it would be great to have better documentation re which sysctl parameters are namespaced (and thus supported) and which ones are not, and also a more descriptive error message if you try and utilize a non-supported parameter.
This historic issue also seems relevant: Perhaps we were premature to close it.
Matt, thanks for confirming the problem! I found the following comments very helpful: Looking at , there doesn't seem to be any difference between Linux 4.15 and 5.1. I'm happy to remove the references to in the docs, but before doing so, should we confirm sysctl worked at some point in the past? As you said, we should enhance the error message and the docs with the supported sysctls. I understand documenting the namespaced sysctls can be a moving target, but not everyone likes to check the Linux source code to see what's namespaced or not. :)
It's probably better to change the docs to set a namespaced sysctl like or . A good way to figure out what network sysctls are namespaced is to pass to kubelet and see all the sysctls listed under inside the pod.
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
Agree with all your statements here :) I'm also surprised that ever worked, but perhaps we could sync with the original author of the docs? +1 to in the meantime changing the docs to a namespaced sysctl.
I don't think ever worked as it's hardly to believe a sysctl got unnamespacified in the kernel (it's always the other way around). is a good example of namespaced networking sysctl. However, the above suggested is not in some cases - it's unnamespaced in kernel 4.14 and became namespaced in 4.15. In general, this is hard to document because there are hundreds of parameters and some of them is kernel version dependent and you always need to lookup the kernel code for answers. I think what we should do now is to with in the example in the doc. in the doc to something like . I think this should be very close ( is an exception) - I couldn't verify all of them as there are so many sysctls and they can also be by loading kernel modules.
SGTM, can you drive that?


Question: "${DOCKER[@]}" rmi "${release_docker_image_tag}" 2>/dev/null || true "${DOCKER[@]}" tag "${docker_image_tag}" "${release_docker_image_tag}" 2>/dev/null fi <ins> else kube::log::status "Not a release, so deleting docker image ${docker_image_tag}" "${DOCKER[@]}" rmi ${docker_image_tag} 2>/dev/null || true </ins> fi <del> kube::log::status "Deleting docker image ${docker_image_tag}" "${DOCKER[@]}" rmi ${docker_image_tag} 2>/dev/null || true </del> ) & done

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT Kubernetes version (use ): HEAD Since v1.7.0-alpha.3, control plane images are only pushed for amd64. Compare: to This is a release-blocker and should be fixed ASAP. I guess the actual regression code-wise happened in test infra. cc
/assign
also cc
/remove-sig testing
Still looking, but "Only tag release containers if the name is different." makes me suspicious.
oh, this is a bug introduced by . fix should be easy.
I can open a new issue as well to track what I sig-testing for but I want us to set up the required testing to make sure this never happens again
the removal was under the guise of "sig testing doesn't control the cutting of bits", if there are tests that need to be put in place to ensure all the right things are cut/released, I would expect sig-release to track that eg: if there's code that needs to be changed in kubernetes/release, it's a sig-release issue; if there's code that needs to be changed in kubernetes/test-infra, it might be a sig-testing issue (or we can triage to the right group)
Sounds good
Details about the exact bug and the fix are in . The reason we didn't notice this sooner is that anago only if the required image doesn't exist - it does so because we don't support all arches for all release versions, and we didn't want to hard-code a list of expected versions into the release repo. (Also, anago automatically , so we were still pushing the legacy versions - we were only dropping the non-amd64 images with this bug.)
Can we add a after the warns? Otherwise even is the user of anago sees the warning, there's nothing he can do about it.
Anago is moving toward being less interactive. If any checks need to be done, they should be done at build time. 1) Determine what should be built 2) Stop right there if the build fails for whatever reason. Adding redundant checks comes with the cost of maintaining hardcoded lists in multiple places. The push step (and release infra in general) should be agnostic and just push what was built.
But in this issue, the build was successful, but some images were removed due to the bug.


Question: "${DOCKER[@]}" rmi "${release_docker_image_tag}" 2>/dev/null || true "${DOCKER[@]}" tag "${docker_image_tag}" "${release_docker_image_tag}" 2>/dev/null fi <ins> else kube::log::status "Not a release, so deleting docker image ${docker_image_tag}" "${DOCKER[@]}" rmi ${docker_image_tag} 2>/dev/null || true </ins> fi <del> kube::log::status "Deleting docker image ${docker_image_tag}" "${DOCKER[@]}" rmi ${docker_image_tag} 2>/dev/null || true </del> ) & done

I'm worried that when building beta.2 this Wednesday, if a similar problem happens, i won't be able to stop anago from pushing the incomplete image sets.
I'm reopening this unless we are really sure that it actually works...
How can we create the testing required to stop this from happening again?
I'm doing a dry run today.
anago could have some sort of sanity checking, though I'm not sure how we'd do that without maintaining hardcoded lists, as notes. I guess the "real" fix is having some automated testing somewhere to consume the non-x86 docker images.
After anago dry run, i saw these images on my machine, so it looks fixed.
Thanks and :tada:!


Question: defer cancel() vsi, err := vs.getVSphereInstance(nodeName) if err != nil { <del> return false, err </del> <ins> return false, volPath, err </ins> } // Ensure client is logged in and session is valid err = vs.nodeManager.vcConnect(ctx, vsi) if err != nil { <del> return false, err </del> <ins> return false, volPath, err </ins> } vm, err := vs.getVMFromNodeName(ctx, nodeName) if err != nil { if err == vclib.ErrNoVMFound { klog.Warningf("Node %q does not exist, vsphere CP will assume disk %v is not attached to it.", nodeName, volPath) // make the disk as detached and return false without error. <del> return false, nil </del> <ins> return false, volPath, nil </ins> } klog.Errorf("Failed to get VM object for node: %q. err: +%v", vSphereInstance, err) <del> return false, err </del> <ins> return false, volPath, err </ins> } volPath = vclib.RemoveStorageClusterORFolderNameFromVDiskPath(volPath) <ins> canonicalPath, pathFetchErr := getcanonicalVolumePath(ctx, vm.Datacenter, volPath) // if canonicalPath is not empty string and pathFetchErr is nil then we can use canonical path to perform detach if canonicalPath != "" && pathFetchErr == nil { volPath = canonicalPath } </ins> attached, err := vm.IsDiskAttached(ctx, volPath) if err != nil { klog.Errorf("DiskIsAttached failed to determine whether disk %q is still attached on node %q",

When user creates a static PV using folder name in path: on vSAN backed datastore then detach disk does not work because check fails - The code expects folder name to be UUID but it does not and hence it determines disk is not attached to the node and hence no detach is ever performed. /sig storage cc


Question: // claimOwnerMatchesSetAndPod returns false if the ownerRefs of the claim are not set consistently with the // PVC deletion policy for the StatefulSet. func claimOwnerMatchesSetAndPod(claim *v1.PersistentVolumeClaim, set *apps.StatefulSet, pod *v1.Pod) bool { <del> policy := set.Spec.PersistentVolumeClaimRetentionPolicy </del> <ins> policy := getPersistentVolumeClaimRetentionPolicy(set) </ins> const retain = apps.RetainPersistentVolumeClaimRetentionPolicyType const delete = apps.DeletePersistentVolumeClaimRetentionPolicyType switch {

The replicas is 3 but just create 1 pod 'web-0' and one pvc, then kube-controller-manager panic happend. Should create 3 pod and 3 pvc add '--feature-gates=StatefulSetAutoDeletePVC=true' arg for kube-controller-manager, then create statefulset with 2 replicas and volumeClaimTemplates. No response <details</details<details</details<details</details<details</details<details</details<details</details
/sig apps
I can reproduce the issue and want to fix it. It seems like variables cause . /assign
/triage accepted
Great find! Does this only happen on an upgraded cluster? I'll spin up a cluster and see if the e2e test catches this problem.
The e2e tests are still passing on a 1.24 cluster, so I suspect this is an upgrade problem. I wonder if there are other holes here.
I installed the Kubernetes 1.25 cluster directly using and still reproduced this issue, so it was probably not caused by upgrading the cluster.
Interesting. I thought that would set the policy to be non-nil. Let's keep this bug open after you submit your PR as I think something else is going on here---I don't think we should be seeing the null policy in the first place.
I suspect that somewhere accidentally set it to nil, and I'll keep looking at this code.
I tried the following (and also with parallel pod management), and it didn't seem to cause the panic. This was a 1.24 cluster, maybe this is only happening on head versions?
I just tried this on a kind cluster running on k8s head, and the problem didn't seem to reproduce. But this was setting the feature flag default in I wonder----does the api server need to have the feature gate as well? That would explain why defaulting wasn't happening.
As far as I'm concerned,I just set the in kube-controller-manager manifests to reproduce this issue. I repeat the yaml used as follows: I tested that in statefulset yaml, setting or not would cause the component to panic. The panic logs are the same as above, so ignore them here.


Question: // claimOwnerMatchesSetAndPod returns false if the ownerRefs of the claim are not set consistently with the // PVC deletion policy for the StatefulSet. func claimOwnerMatchesSetAndPod(claim *v1.PersistentVolumeClaim, set *apps.StatefulSet, pod *v1.Pod) bool { <del> policy := set.Spec.PersistentVolumeClaimRetentionPolicy </del> <ins> policy := getPersistentVolumeClaimRetentionPolicy(set) </ins> const retain = apps.RetainPersistentVolumeClaimRetentionPolicyType const delete = apps.DeletePersistentVolumeClaimRetentionPolicyType switch {

statefulset with : The environment
I believe you---this doesn't seem to be well-documented, but it's my understanding that you need to set the feature gate on all of the system components. Obviously failing to do so shouldn't cause a panic, so thank you again for finding & fixing this bug! Let me ask around to see what the expected behavior is. I know there may be skew between kubelets and master versions, but I didn't expect to have to deal with skew between master components.
I read the code, and now I understand what you mean. This feature gate must be enabled in both the and components. We need to describe it in the to prevent someone from ignoring the component, which also needs to enable the feature gate.
I don't think we need all the system components, like the component, right? I read the original KEP documentation. This feature gate only involves components and :
I had a bit of discussion about this with others. Because upgrades don't occur atomically, any feature gate has to work regardless of how the other system components are configured (things shouldn't panic!). I suppose if the cluster administrator wants to track what needs to be enabled on each feature they can try to sequence which component has which feature gate. If it were me I'd try to make all the components consistent though.
This is certainly a method, but it is a bit more cumbersome for the administrator. It is best to have a place for the document to be queried by the administrator and avoid confusion. I don't think everyone is very familiar with the k8s mechanism. I suggest adding a partial description to this section of the : Before: After: I can submit this PR on .
This is going to be true of any api change. When you tried to use it with only adding the controller manager feature gate, I think you would be able to set the retain policy, but there wouldn't be any defaulting or validation (?).
I've seen other feature gates that say what components are needed to enable it to use the new feature. For example: If it's not necessary, then I close .
only adding the controller manager feature gate, I think you would be able to set the retain policy, but there wouldn't be any defaulting or validation (?). I remember just enabling it in kube-controller-manager.


Question: // claimOwnerMatchesSetAndPod returns false if the ownerRefs of the claim are not set consistently with the // PVC deletion policy for the StatefulSet. func claimOwnerMatchesSetAndPod(claim *v1.PersistentVolumeClaim, set *apps.StatefulSet, pod *v1.Pod) bool { <del> policy := set.Spec.PersistentVolumeClaimRetentionPolicy </del> <ins> policy := getPersistentVolumeClaimRetentionPolicy(set) </ins> const retain = apps.RetainPersistentVolumeClaimRetentionPolicyType const delete = apps.DeletePersistentVolumeClaimRetentionPolicyType switch {

it using the retain policy, but the statefulset does not have the policy from output of (It should be the apiserver authentication forbids it)
Yeah I think that makes sense. Thanks for finding this issue! After your PR with the panic fix is submitted, I think we can close this?
Yes. I agree. /close
Closing this issue. <detailsIn response to : Yes. I agree. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: } } else { rawMgr := mngr.(*manager) <del> rawScope := rawMgr.scope.(*containerScope) if rawScope.policy.Name() != tc.expectedPolicy { t.Errorf("Unexpected policy name. Have: %q wants %q", rawScope.policy.Name(), tc.expectedPolicy) </del> <ins> var policyName string if rawScope, ok := rawMgr.scope.(*containerScope); ok { policyName = rawScope.policy.Name() } else if rawScope, ok := rawMgr.scope.(*noneScope); ok { policyName = rawScope.policy.Name() } if policyName != tc.expectedPolicy { t.Errorf("Unexpected policy name. Have: %q wants %q", policyName, tc.expectedPolicy) </ins> } } }

Follow up from: See details there. When is , there is no need to discover the NUMA topology. /sig node /priority backlog /kind feature /help-wanted
/cc
/triage accepted
I would like to take this up also can I get more brief information about it? code review comments from asking that inside the NewManager() implementation, we need to skip NewPolicyOptions and NewNUMAInfo but when the NumaTopology comes kubelet issue also comes. I need some more information to start with this!
/cc
When there is no topology manger policy specified, NewManager should be a no-op. Just skip all checks.
Exactly. Just to add: we may need few iterations over the change to get a clean flow in the call.
/assign


Question: cidr.IP.To4()[3] += 1 json := fmt.Sprintf(NET_CONFIG_TEMPLATE, BridgeName, plugin.MTU, network.DefaultInterfaceName, podCIDR, cidr.IP.String()) <ins> glog.V(2).Infof("CNI network config set to %v", json) </ins> plugin.netConfig, err = libcni.ConfFromBytes([]byte(json)) if err == nil { glog.V(5).Infof("CNI network config:n%s", json)

Event for a pod: The cni plugins do exist at /opt/cni/bin. They cannot be executed and bash is complaining:
cc
I'm guessing this is a shared lib problem, since /usr/grte/v4/lib64/ld-linux-x86- doesn't exist on my gce vm. We should update docs to make it clear how someone would install manually (if they run into such problems again) and statically link (if the problem is indeed the shared lib) with something like Unsure how we'd support different GOOS (vend different tarballs vs checking in and building the plugin inline with our make scripts)
uggh. These binaries need to be ultra-static or else we need to containerize them.
Essentially doing would've prevented this bug :)
And see for the CNI_ARGS problems
also , re ... however, I'm not too sure if your comment was mean to be on this ticket.
cc
I have a working version. I uploaded my cni binary with bug fix on GCS. change source to change sourcehash to Welcome to give it a try.
Also see problem:


Question: framework.ExpectNoError(err) framework.ExpectNoError(e2epod.WaitTimeoutForPodReadyInNamespace(f.ClientSet, webserverPod0.Name, f.Namespace.Name, framework.PodStartTimeout)) <del> validateEndpointsPortsOrFail(cs, ns, serviceName, portsByPodName{webserverPod0.Name: {servicePort}}) </del> <ins> validateEndpointsPortsOrFail(cs, ns, serviceName, portsByPodName{webserverPod0.Name: {endpointPort}}) </ins> <del> ginkgo.By("Creating 2 pause pods that will try to connect to the webservers") </del> <ins> ginkgo.By("Creating 2 pause pods that will try to connect to the webserver") </ins> pausePod0 := e2epod.NewAgnhostPod(ns, "pause-pod-0", nil, nil, nil) e2epod.SetNodeSelection(&pausePod0.Spec, e2epod.NodeSelection{Name: node0.Name})

Failed here /retest Originally posted by in This fails because the pod tries to bind port 80, that is a very demanded port, but there are two tests doing that, that increase the chance to collide
/sig network
Close when merges
/triage accepted


Question: result := make(chan T, 1) // non-blocking c.inFlight[key] = result go func() { <del> // Make potentially slow call </del> <ins> // Make potentially slow call. // While this call is in flight, fillOrWait will // presumably exit. </ins> data := timeCacheEntry{ item:       c.fillFunc(key), lastUpdate: c.clock.Now(),

Failed due to an apparently unrelated PR: https://travis-
Problem here is that I left a small chance of a double-call in my parallelized cache. Didn't realize this was being tested. Will fix.


Golang 1.13 uses TLS 1.13 by default (opt-out), which introduces RSA-PSS. RSA-PSS does not work with keys that are too small. The issue was uncovered during Fedora mass rebuild which uses Golang 1.13: See /sig api-machinery
What is the failure mode if a smaller key is used?
/sig auth /sig scalability
/sub


Question: } // Fail if there are other errors. <del> if len(errors) > 0 { return utilerrors.Flatten(utilerrors.NewAggregate(errors)) </del> <ins> if len(allErrs) > 0 { return utilerrors.Flatten(utilerrors.NewAggregate(allErrs)) </ins> } // Ensure the backendPoolID is also deleted on VMSS itself.

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --/kind bug /sig cloud-provider /area provider/azure Azure in-tree CCM fails to delete a load balancer: The corresponding logs: The clouprovider config is: The Nodes are actually Azure VMs that belong to a VMO (aka vmss with flexible orchestration mode) - regular VMs and managed independently of the vmss. So those VMs are managed on Azure as . LoadBalancer deletion works with the very same setup with v1.21.1 of the in-tree Azure CCM. After recent cherry-picks, with v1.21.4 we observe the failure above. a cluster using VMs that belong to a VMO a Service of type LoadBalancer the Service of type LoadBalancer sure that the deletion fails with: Kubernetes version (use ): v1.21.4 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Gardener Network plugin and version (if this is a network-related bug): Others:
/assign /triage accepted


Question: // the cluster is restored to health. ginkgo.By("waiting for system pods to successfully restart") err := e2epod.WaitForPodsRunningReady(c, metav1.NamespaceSystem, systemPodsNo, 0, framework.PodReadyBeforeTimeout, map[string]string{}) <del> framework.ExpectEqual(err, nil) </del> <ins> framework.ExpectNoError(err) </ins> }) ginkgo.It("node lease should be deleted when corresponding node is deleted", func() { leaseClient := c.CoordinationV1().Leases(v1.NamespaceNodeLease) err := e2enode.WaitForReadyNodes(c, framework.TestContext.CloudConfig.NumNodes, 10*time.Minute) <del> framework.ExpectEqual(err, nil) </del> <ins> framework.ExpectNoError(err) </ins> ginkgo.By("verify node lease exists for every nodes") originalNodes, err := e2enode.GetReadySchedulableNodes(c)

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: kubernetes/node-problem-detector/node-problem-detector-ci: ci-npd-e2e-node See the Which test(s) are failing: [] NodeProblemDetector [NodeFeature:NodeProblemDetector] [] SystemLogMonitor should generate node condition and events for corresponding errors Since when has it been failing: Dec 03. was at 19:50 PST, was at 21:51 PST. Testgrid link: https://k8s- Reason for failure: Apparently is the culprit. Anything else we need to know: This is likely caused by some library upgrade. Nothing big.
/sig node
/assign
Seems like the failure is caused by . /cc /cc /cc /cc
/priority critical-urgent
The particular bug I care about has been fixed in . I'm gonna still work on submitting , as the same error should also be affecting a few other tests.


Question: SCHEDULER_TEST_ARGS="${SCHEDULER_TEST_ARGS:-} ${TEST_CLUSTER_API_CONTENT_TYPE}" KUBEPROXY_TEST_ARGS="${KUBEPROXY_TEST_ARGS:-} ${TEST_CLUSTER_API_CONTENT_TYPE}" <ins> # Historically fluentd was a manifest pod and then was migrated to DaemonSet. # To avoid situation during cluster upgrade when there are two instances # of fluentd running on a node, kubelet need to mark node on which # fluentd is not running as a manifest pod with appropriate label. # TODO(piosz): remove this in 1.8 NODE_LABELS="${KUBE_NODE_LABELS:-alpha.kubernetes.io/fluentd-ds-ready=true}" </ins> # Optional: Enable node logging. ENABLE_NODE_LOGGING="${KUBE_ENABLE_NODE_LOGGING:-true}" LOGGING_DESTINATION="${KUBE_LOGGING_DESTINATION:-gcp}" # options: elasticsearch, gcp

PR introduced an annotation for fluentd migration from static pod to the daemonset to avoid two instances of fluentd on a single node during the transition stage. I didn't read the pr carefully, but based on the code at , every node object has one label be default: "". I think this is wrong. 1) Not everyone use fluentd as the logging agent. 2) It is hard-coded in Kubelet code as part of kubeletnodestatus. No matter running fluentd as a static pod or a daemonset on the node, Kubelet shouldn't treat it different from other pods. The right approach is introducing an external controller to apply this cloud-provider / kubernetes vendor (i.e. GKE) specific label to the node. cc/
The string "fluentd" shouldn't appear anywhere in our code. Please remove it ASAP.
Absolutely agree with you, thanks for bringing this up! Is it possible to apply labels to the node as part of the node startup script/salt?
To clarify, I'm thinking about possible ways to provide a fix for that ASAP. Writing external controller is a best solution, but here we have a release coming and this label is intended to be only a temporary measure to perform upgrade from using fluentd as static pods to using daemon set.
There are many ways to achieve this outside the core system. But I believe GKE already has an existing mechanism to add this kind of labels to the node, and the consistency is important for GKE management. cc/
OK, thanks! However, I believe we should be sure not to break everyone's upgrade process, i.e. after master upgrade to 1.6 on any platform, there should be only one fluentd agent on each node. I propose changing the configure scripts and salt to set up needed labels in kubelet's flag (). WDYT?
It seems to me like upgrades could survive without the explicit dependency on fluentd: I'm not sure if upgrades call initialNode or not, my own ignorance ? If they do, and this is removed, can't an external controller just do the work of re--adding that label if its dropped?
thanks for raising this. Let's sync f2f
cc
WDYT about exposing kubelet version as a label and have a selector in fluentd ds config targeting version 1.6+?


Question: SCHEDULER_TEST_ARGS="${SCHEDULER_TEST_ARGS:-} ${TEST_CLUSTER_API_CONTENT_TYPE}" KUBEPROXY_TEST_ARGS="${KUBEPROXY_TEST_ARGS:-} ${TEST_CLUSTER_API_CONTENT_TYPE}" <ins> # Historically fluentd was a manifest pod and then was migrated to DaemonSet. # To avoid situation during cluster upgrade when there are two instances # of fluentd running on a node, kubelet need to mark node on which # fluentd is not running as a manifest pod with appropriate label. # TODO(piosz): remove this in 1.8 NODE_LABELS="${KUBE_NODE_LABELS:-alpha.kubernetes.io/fluentd-ds-ready=true}" </ins> # Optional: Enable node logging. ENABLE_NODE_LOGGING="${KUBE_ENABLE_NODE_LOGGING:-true}" LOGGING_DESTINATION="${KUBE_LOGGING_DESTINATION:-gcp}" # options: elasticsearch, gcp

and I talked about this issue a bit for migrating kube-proxy. One idea we came up with was that of running a controller that will identify kubelet versions and apply taints to nodes running static pods. The daemonset version would not tolerate that taint and that way we keep daemon set pods off of nodes running static pods. Have you considered that approach?
It doesn't differ from a controller applying labels. I'd like to avoid necessity of introducing such controller, since it's not clear what is it's responsibility (generic controller for applying labels on nodes based on some rules? a custom controller just for fluentd purposes). In GKE we can do it on GKE control plane side, but we need a story for open source.
Move applying labels to startup scripts/salt seems to be the least intrusive and easiest to implement way.
Actually what I proposed in is not an option since nodeSelector in PodSpec is old style selector.
I'm not a big fan of approach with salt, but it seems there is no better option.
Re: Instead of exposing Kubelet version as a label, why not using field selector. Kubelet version is part of NodeStatus already. I just simply don't want to have a duplicate info for the same object.
is it possible to schedule based on fieldselector? I'm only aware of using field selector while listing objects.
Also node-labels option is in alpha
But it's present since 1.2 What is the status of this flag? Can we use it for fluentd purposes or better avoid this?
Maybe it does make sense to implement a temporary ad-hoc controller? It's the easiest to deploy solution, just another in fluentd addon.
I know. It totally doesn't make sense to introduce anything custom. I'll use just the flag. It's rather hacky but unfortunately I can't see anything better.
Cool thing about controller is that you can deploy it into any cluster and it will work, with CA also. You don't have to set up your node labels manually then or restart the cluster.
There is a number of open question with having a new controller: it run on the master or a node? we guarantee that it will be always running? it have any resource requirements? I'd like to avoid discussing it right before the release.
Having a controller will help with kube-proxy transition too in the future.


Question: SCHEDULER_TEST_ARGS="${SCHEDULER_TEST_ARGS:-} ${TEST_CLUSTER_API_CONTENT_TYPE}" KUBEPROXY_TEST_ARGS="${KUBEPROXY_TEST_ARGS:-} ${TEST_CLUSTER_API_CONTENT_TYPE}" <ins> # Historically fluentd was a manifest pod and then was migrated to DaemonSet. # To avoid situation during cluster upgrade when there are two instances # of fluentd running on a node, kubelet need to mark node on which # fluentd is not running as a manifest pod with appropriate label. # TODO(piosz): remove this in 1.8 NODE_LABELS="${KUBE_NODE_LABELS:-alpha.kubernetes.io/fluentd-ds-ready=true}" </ins> # Optional: Enable node logging. ENABLE_NODE_LOGGING="${KUBE_ENABLE_NODE_LOGGING:-true}" LOGGING_DESTINATION="${KUBE_LOGGING_DESTINATION:-gcp}" # options: elasticsearch, gcp

, Piotr Szczesniak <:
is this a 1.6 release blocker?
I think yes. The PR is already sent and approved. I need to fix e2e tests because it doesn't work well with the test infra (e2e tests passes locally). EDIT: I'll do it hopefully today.
The PR is ready to be merged and lgtmed. Waiting for approvals.


Question: return s.converter.Convert(src, dest, flags, s.meta) } <ins> // DefaultConvert continues a conversion, performing a default conversion (no conversion func) // for the current stack frame. func (s *scope) DefaultConvert(src, dest interface{}, flags FieldMatchingFlags) error { return s.converter.DefaultConvert(src, dest, flags, s.meta) } </ins> // SrcTag returns the tag of the struct containing the current source item, if any. func (s *scope) SrcTag() reflect.StructTag { return s.srcStack.top().tag

Files storing latest versions for minors are not updated since infix was to the version string. Example Similar issues can be found in 1.25 and 1.24 builds (last 1.23 build infix so it has not happened there yet): I expect the files to be overwritten with newer version of the build. Manually run build job for one of the minors. No response Not related. Not related. No response No response No response No response
I believe this may be also responsible for breaking kubemark tests: since they fail due to version mismatch
/sig release
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /priority high Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Thanks for raising the issue! /cc who is working on before
I assume this will be fixed automatically with the next patches, because then we would compare with . Anyway, I think we should fix that intermediately by manually updating the version markers to match their correct version. I did that now for the bucket
We had to update the versions markers again, because those were pointing to two commits earlier and didn't include the Go updates. The latest version markers are: v1.24.12-35+ v1.25.8-37+ v1.26.3-41+
latest-1.27 has been affected by this as well. I updated it manually to v1.27.0-9+. See this thread for more details:
/triage accepted
This should be fixed now, closing it. Feel free to reopen if we missed anything.
This issue is still present, see this Slack thread for more details: /reopen
Reopened this issue. <detailsIn response to : This issue is still present, see this Slack thread for more details: /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
We have a similar situation with latest-1.28 which is currently pointing to: . This is one commit before the release commit: ! It's mostly like that the release step didn't update version markers, we should look into this.


Question: return s.converter.Convert(src, dest, flags, s.meta) } <ins> // DefaultConvert continues a conversion, performing a default conversion (no conversion func) // for the current stack frame. func (s *scope) DefaultConvert(src, dest interface{}, flags FieldMatchingFlags) error { return s.converter.DefaultConvert(src, dest, flags, s.meta) } </ins> // SrcTag returns the tag of the struct containing the current source item, if any. func (s *scope) SrcTag() reflect.StructTag { return s.srcStack.top().tag

As of now, I'm going to manually update this version marker to unblock 1.28.1 patch release. I'll also take a look into this. /assign
The PR that fixed this issue got reverted so this is still relevant: /reopen
Reopened this issue. <detailsIn response to : The PR that fixed this issue got reverted so this is still relevant: /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
We're running into this issue again with stable-1.29 marker. It's currently pointing to which is It should instead point to I'm going to manually change it following the same procedure as the last time. Version marker updated to
This issue is labeled with but has not been updated in over 90 days, and should be re-triaged. Important-soon issues must be staffed and worked on either currently, or very soon, ideally in time for the next release. You can: Confirm that this issue is still relevant with (org members only) Deprioritize it with or Close this issue with For more details on the triage process, see /remove-triage accepted
/triage accepted
Same issue for 1.30, where I had to update the version marker manually again.


Question: <ins> ./build/build-image/rsyncd.sh ./build/common.sh ./build/copy-output.sh ./build/lib/release.sh ./build/make-build-image.sh ./build/make-clean.sh ./build/package-tarballs.sh ./build/release-images.sh ./build/release-in-a-container.sh ./build/release.sh ./build/run.sh ./build/shell.sh ./build/util.sh ./cluster/addons/addon-manager/kube-addons.sh ./cluster/addons/fluentd-elasticsearch/es-image/run.sh ./cluster/addons/fluentd-elasticsearch/fluentd-es-image/run.sh ./cluster/centos/build.sh ./cluster/centos/config-build.sh ./cluster/centos/config-default.sh ./cluster/centos/config-test.sh ./cluster/centos/deployAddons.sh ./cluster/centos/make-ca-cert.sh ./cluster/centos/master/scripts/flannel.sh ./cluster/centos/node/bin/mk-docker-opts.sh ./cluster/centos/node/scripts/flannel.sh ./cluster/centos/util.sh ./cluster/clientbin.sh ./cluster/common.sh ./cluster/gce/config-common.sh ./cluster/gce/config-default.sh ./cluster/gce/config-test.sh ./cluster/gce/delete-stranded-load-balancers.sh ./cluster/gce/gci/configure-helper.sh ./cluster/gce/gci/configure.sh ./cluster/gce/gci/flexvolume_node_setup.sh ./cluster/gce/gci/health-monitor.sh ./cluster/gce/gci/master-helper.sh ./cluster/gce/gci/mounter/stage-upload.sh ./cluster/gce/gci/node-helper.sh ./cluster/gce/gci/shutdown.sh ./cluster/gce/list-resources.sh ./cluster/gce/upgrade-aliases.sh ./cluster/gce/upgrade.sh ./cluster/gce/util.sh ./cluster/get-kube-binaries.sh ./cluster/get-kube-local.sh ./cluster/get-kube.sh ./cluster/images/conformance/run_e2e.sh ./cluster/images/etcd-empty-dir-cleanup/etcd-empty-dir-cleanup.sh ./cluster/juju/prereqs/ubuntu-juju.sh ./cluster/juju/util.sh ./cluster/kube-down.sh ./cluster/kube-up.sh ./cluster/kube-util.sh ./cluster/kubeadm.sh ./cluster/kubectl.sh ./cluster/kubemark/gce/config-default.sh ./cluster/kubemark/iks/config-default.sh ./cluster/kubemark/util.sh ./cluster/local/util.sh ./cluster/log-dump/log-dump.sh ./cluster/pre-existing/util.sh ./cluster/restore-from-backup.sh ./cluster/test-e2e.sh ./cluster/test-network.sh ./cluster/test-smoke.sh ./cluster/update-storage-objects.sh ./cluster/validate-cluster.sh ./hack/benchmark-go.sh ./hack/build-cross.sh ./hack/build-go.sh ./hack/build-ui.sh ./hack/cherry_pick_pull.sh ./hack/dev-build-and-push.sh ./hack/dev-build-and-up.sh ./hack/dev-push-conformance.sh ./hack/dev-push-hyperkube.sh ./hack/e2e-internal/e2e-cluster-size.sh ./hack/e2e-internal/e2e-down.sh ./hack/e2e-internal/e2e-grow-cluster.sh ./hack/e2e-internal/e2e-shrink-cluster.sh ./hack/e2e-internal/e2e-status.sh ./hack/e2e-internal/e2e-up.sh ./hack/e2e-node-test.sh ./hack/generate-bindata.sh ./hack/generate-docs.sh ./hack/get-build.sh ./hack/ginkgo-e2e.sh ./hack/godep-restore.sh ./hack/godep-save.sh ./hack/grab-profiles.sh ./hack/install-etcd.sh ./hack/jenkins/benchmark-dockerized.sh ./hack/jenkins/build.sh ./hack/jenkins/test-dockerized.sh ./hack/jenkins/upload-to-gcs.sh ./hack/jenkins/verify-dockerized.sh ./hack/lib/etcd.sh ./hack/lib/golang.sh ./hack/lib/init.sh ./hack/lib/logging.sh ./hack/lib/protoc.sh ./hack/lib/swagger.sh ./hack/lib/test.sh ./hack/lib/util.sh ./hack/lib/version.sh ./hack/list-feature-tests.sh ./hack/local-up-cluster.sh ./hack/make-rules/build.sh ./hack/make-rules/clean.sh ./hack/make-rules/cross.sh ./hack/make-rules/helpers/cache_go_dirs.sh ./hack/make-rules/make-help.sh ./hack/make-rules/test-cmd.sh ./hack/make-rules/test-e2e-node.sh ./hack/make-rules/test-integration.sh ./hack/make-rules/test-kubeadm-cmd.sh ./hack/make-rules/test.sh ./hack/make-rules/update.sh ./hack/make-rules/verify.sh ./hack/make-rules/vet.sh ./hack/print-workspace-status.sh ./hack/run-in-gopath.sh ./hack/test-go.sh ./hack/test-integration.sh ./hack/test-update-storage-objects.sh ./hack/update-all.sh ./hack/update-api-reference-docs.sh ./hack/update-bazel.sh ./hack/update-codegen.sh ./hack/update-generated-device-plugin-dockerized.sh ./hack/update-generated-device-plugin.sh ./hack/update-generated-docs.sh ./hack/update-generated-kms-dockerized.sh ./hack/update-generated-kms.sh ./hack/update-generated-kubelet-plugin-registration-dockerized.sh ./hack/update-generated-kubelet-plugin-registration.sh ./hack/update-generated-pod-resources-dockerized.sh ./hack/update-generated-pod-resources.sh ./hack/update-generated-protobuf-dockerized.sh ./hack/update-generated-protobuf.sh ./hack/update-generated-runtime-dockerized.sh ./hack/update-generated-runtime.sh ./hack/update-generated-swagger-docs.sh ./hack/update-godep-licenses.sh ./hack/update-gofmt.sh ./hack/update-openapi-spec.sh ./hack/update-staging-godeps-dockerized.sh ./hack/update-staging-godeps.sh ./hack/update-swagger-spec.sh ./hack/update-translations.sh ./hack/update-workspace-mirror.sh ./hack/verify-all.sh ./hack/verify-api-groups.sh ./hack/verify-api-reference-docs.sh ./hack/verify-bazel.sh ./hack/verify-boilerplate.sh ./hack/verify-cli-conventions.sh ./hack/verify-codegen.sh ./hack/verify-description.sh ./hack/verify-generated-device-plugin.sh ./hack/verify-generated-docs.sh ./hack/verify-generated-files-remake.sh ./hack/verify-generated-files.sh ./hack/verify-generated-kms.sh ./hack/verify-generated-kubelet-plugin-registration.sh ./hack/verify-generated-pod-resources.sh ./hack/verify-generated-protobuf.sh ./hack/verify-generated-runtime.sh ./hack/verify-generated-swagger-docs.sh ./hack/verify-godep-licenses.sh ./hack/verify-godeps.sh ./hack/verify-gofmt.sh ./hack/verify-golint.sh ./hack/verify-govet.sh ./hack/verify-import-boss.sh ./hack/verify-imports.sh ./hack/verify-linkcheck.sh ./hack/verify-no-vendor-cycles.sh ./hack/verify-openapi-spec.sh ./hack/verify-pkg-names.sh ./hack/verify-readonly-packages.sh ./hack/verify-spelling.sh ./hack/verify-staging-godeps.sh ./hack/verify-staging-meta-files.sh ./hack/verify-swagger-spec.sh ./hack/verify-symbols.sh ./hack/verify-test-featuregates.sh ./hack/verify-test-images.sh ./hack/verify-test-owners.sh ./hack/verify-typecheck.sh ./pkg/kubectl/cmd/edit/testdata/record_testcase.sh ./pkg/util/verify-util-pkg.sh ./plugin/pkg/admission/imagepolicy/gencerts.sh ./staging/src/k8s.io/apiextensions-apiserver/examples/client-go/hack/update-codegen.sh ./staging/src/k8s.io/apiextensions-apiserver/examples/client-go/hack/verify-codegen.sh ./staging/src/k8s.io/apiextensions-apiserver/hack/build-image.sh ./staging/src/k8s.io/apiextensions-apiserver/hack/update-codegen.sh ./staging/src/k8s.io/apiextensions-apiserver/hack/verify-codegen.sh ./staging/src/k8s.io/apiserver/pkg/admission/plugin/webhook/testcerts/gencerts.sh ./staging/src/k8s.io/apiserver/pkg/util/webhook/gencerts.sh ./staging/src/k8s.io/apiserver/plugin/pkg/authenticator/token/oidc/testdata/gen.sh ./staging/src/k8s.io/apiserver/plugin/pkg/authorizer/webhook/gencerts.sh ./staging/src/k8s.io/code-generator/generate-groups.sh ./staging/src/k8s.io/code-generator/generate-internal-groups.sh ./staging/src/k8s.io/code-generator/hack/update-codegen.sh ./staging/src/k8s.io/code-generator/hack/verify-codegen.sh ./staging/src/k8s.io/csi-api/hack/update-codegen.sh ./staging/src/k8s.io/csi-api/hack/verify-codegen.sh ./staging/src/k8s.io/kube-aggregator/hack/build-image.sh ./staging/src/k8s.io/kube-aggregator/hack/local-up-kube-aggregator.sh ./staging/src/k8s.io/kube-aggregator/hack/register-all-apis-from.sh ./staging/src/k8s.io/kube-aggregator/hack/update-codegen.sh ./staging/src/k8s.io/kube-aggregator/hack/verify-codegen.sh ./staging/src/k8s.io/metrics/hack/update-codegen.sh ./staging/src/k8s.io/metrics/hack/verify-codegen.sh ./staging/src/k8s.io/node-api/hack/update-codegen.sh ./staging/src/k8s.io/node-api/hack/verify-codegen.sh ./staging/src/k8s.io/sample-apiserver/hack/build-image.sh ./staging/src/k8s.io/sample-apiserver/hack/update-codegen.sh ./staging/src/k8s.io/sample-apiserver/hack/verify-codegen.sh ./staging/src/k8s.io/sample-controller/hack/update-codegen.sh ./staging/src/k8s.io/sample-controller/hack/verify-codegen.sh ./test/cmd/apply.sh ./test/cmd/apps.sh ./test/cmd/authorization.sh ./test/cmd/batch.sh ./test/cmd/certificate.sh ./test/cmd/core.sh ./test/cmd/crd.sh ./test/cmd/create.sh ./test/cmd/diff.sh ./test/cmd/discovery.sh ./test/cmd/generic-resources.sh ./test/cmd/get.sh ./test/cmd/initializers.sh ./test/cmd/legacy-script.sh ./test/cmd/node-management.sh ./test/cmd/old-print.sh ./test/cmd/proxy.sh ./test/cmd/rbac.sh ./test/cmd/request-timeout.sh ./test/cmd/run.sh ./test/cmd/save-config.sh ./test/cmd/storage.sh ./test/cmd/template-output.sh ./test/cmd/version.sh ./test/e2e_node/conformance/run_test.sh ./test/e2e_node/environment/setup_host.sh ./test/e2e_node/gubernator.sh ./test/e2e_node/jenkins/conformance/conformance-jenkins.sh ./test/e2e_node/jenkins/copy-e2e-image.sh ./test/e2e_node/jenkins/e2e-node-jenkins.sh ./test/e2e_node/jenkins/ubuntu-14.04-nvidia-install.sh ./test/images/image-util.sh ./test/images/pets/redis-installer/on-start.sh ./test/images/pets/zookeeper-installer/install.sh ./test/images/pets/zookeeper-installer/on-start.sh ./test/images/volume/gluster/run_gluster.sh ./test/images/volume/iscsi/create_block.sh ./test/images/volume/nfs/run_nfs.sh ./test/images/volume/rbd/bootstrap.sh ./test/images/volume/rbd/create_block.sh ./test/images/volume/rbd/mon.sh ./test/images/volume/rbd/osd.sh ./test/integration/ipamperf/test-performance.sh ./test/integration/scheduler_perf/test-performance.sh ./test/kubemark/common/util.sh ./test/kubemark/configure-kubectl.sh ./test/kubemark/gce/util.sh ./test/kubemark/iks/shutdown.sh ./test/kubemark/iks/startup.sh ./test/kubemark/iks/util.sh ./test/kubemark/master-log-dump.sh ./test/kubemark/pre-existing/util.sh ./test/kubemark/resources/start-kubemark-master.sh ./test/kubemark/run-e2e-tests.sh ./test/kubemark/start-kubemark.sh ./test/kubemark/stop-kubemark.sh ./third_party/forked/shell2junit/sh2ju.sh ./third_party/intemp/intemp.sh ./third_party/multiarch/qemu-user-static/register/qemu-binfmt-conf.sh ./third_party/multiarch/qemu-user-static/register/register.sh </ins>

is a bash script linter which could help catch common problems in our (numerous) bash scripts. I know we'd like to reduce our use of bash scripts, but until then I think being stricter about script quality would be helpful. /cc
cc you were talking about this earlier. It has to only check modified lines.
False positives are a problem. For instance, not being able to follow a call through an environment variable causes an error. I would be much more comfortable with a non-blocking way to point out these issues, and leave to it human judgement.
Yeah, I've been interested in doing this for a while, but can be pretty persnickety and I haven't yet taken the time to configure it to be less noisy (or figure out if it can be). It's good, though.
Yeah, it would definitely need to be configured to ignore a lot of types of errors, but I still think it could be used to check for some types of errors. We could start with a pretty minimal set.
I would settle for something that just produced a quick-fix format artifact in the GCS bucket, and then the PR munger reading from it and commenting on the PR. We could re-use that for other linters too.
has been pretty accurate historically.
shellcheck, not spellcheck
reading too many issues in a on
/sig testing /priority awaiting-more-evidence Might be more noise than it's worth, but we do pylint our .py scripts now
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with .


Question: <ins> ./build/build-image/rsyncd.sh ./build/common.sh ./build/copy-output.sh ./build/lib/release.sh ./build/make-build-image.sh ./build/make-clean.sh ./build/package-tarballs.sh ./build/release-images.sh ./build/release-in-a-container.sh ./build/release.sh ./build/run.sh ./build/shell.sh ./build/util.sh ./cluster/addons/addon-manager/kube-addons.sh ./cluster/addons/fluentd-elasticsearch/es-image/run.sh ./cluster/addons/fluentd-elasticsearch/fluentd-es-image/run.sh ./cluster/centos/build.sh ./cluster/centos/config-build.sh ./cluster/centos/config-default.sh ./cluster/centos/config-test.sh ./cluster/centos/deployAddons.sh ./cluster/centos/make-ca-cert.sh ./cluster/centos/master/scripts/flannel.sh ./cluster/centos/node/bin/mk-docker-opts.sh ./cluster/centos/node/scripts/flannel.sh ./cluster/centos/util.sh ./cluster/clientbin.sh ./cluster/common.sh ./cluster/gce/config-common.sh ./cluster/gce/config-default.sh ./cluster/gce/config-test.sh ./cluster/gce/delete-stranded-load-balancers.sh ./cluster/gce/gci/configure-helper.sh ./cluster/gce/gci/configure.sh ./cluster/gce/gci/flexvolume_node_setup.sh ./cluster/gce/gci/health-monitor.sh ./cluster/gce/gci/master-helper.sh ./cluster/gce/gci/mounter/stage-upload.sh ./cluster/gce/gci/node-helper.sh ./cluster/gce/gci/shutdown.sh ./cluster/gce/list-resources.sh ./cluster/gce/upgrade-aliases.sh ./cluster/gce/upgrade.sh ./cluster/gce/util.sh ./cluster/get-kube-binaries.sh ./cluster/get-kube-local.sh ./cluster/get-kube.sh ./cluster/images/conformance/run_e2e.sh ./cluster/images/etcd-empty-dir-cleanup/etcd-empty-dir-cleanup.sh ./cluster/juju/prereqs/ubuntu-juju.sh ./cluster/juju/util.sh ./cluster/kube-down.sh ./cluster/kube-up.sh ./cluster/kube-util.sh ./cluster/kubeadm.sh ./cluster/kubectl.sh ./cluster/kubemark/gce/config-default.sh ./cluster/kubemark/iks/config-default.sh ./cluster/kubemark/util.sh ./cluster/local/util.sh ./cluster/log-dump/log-dump.sh ./cluster/pre-existing/util.sh ./cluster/restore-from-backup.sh ./cluster/test-e2e.sh ./cluster/test-network.sh ./cluster/test-smoke.sh ./cluster/update-storage-objects.sh ./cluster/validate-cluster.sh ./hack/benchmark-go.sh ./hack/build-cross.sh ./hack/build-go.sh ./hack/build-ui.sh ./hack/cherry_pick_pull.sh ./hack/dev-build-and-push.sh ./hack/dev-build-and-up.sh ./hack/dev-push-conformance.sh ./hack/dev-push-hyperkube.sh ./hack/e2e-internal/e2e-cluster-size.sh ./hack/e2e-internal/e2e-down.sh ./hack/e2e-internal/e2e-grow-cluster.sh ./hack/e2e-internal/e2e-shrink-cluster.sh ./hack/e2e-internal/e2e-status.sh ./hack/e2e-internal/e2e-up.sh ./hack/e2e-node-test.sh ./hack/generate-bindata.sh ./hack/generate-docs.sh ./hack/get-build.sh ./hack/ginkgo-e2e.sh ./hack/godep-restore.sh ./hack/godep-save.sh ./hack/grab-profiles.sh ./hack/install-etcd.sh ./hack/jenkins/benchmark-dockerized.sh ./hack/jenkins/build.sh ./hack/jenkins/test-dockerized.sh ./hack/jenkins/upload-to-gcs.sh ./hack/jenkins/verify-dockerized.sh ./hack/lib/etcd.sh ./hack/lib/golang.sh ./hack/lib/init.sh ./hack/lib/logging.sh ./hack/lib/protoc.sh ./hack/lib/swagger.sh ./hack/lib/test.sh ./hack/lib/util.sh ./hack/lib/version.sh ./hack/list-feature-tests.sh ./hack/local-up-cluster.sh ./hack/make-rules/build.sh ./hack/make-rules/clean.sh ./hack/make-rules/cross.sh ./hack/make-rules/helpers/cache_go_dirs.sh ./hack/make-rules/make-help.sh ./hack/make-rules/test-cmd.sh ./hack/make-rules/test-e2e-node.sh ./hack/make-rules/test-integration.sh ./hack/make-rules/test-kubeadm-cmd.sh ./hack/make-rules/test.sh ./hack/make-rules/update.sh ./hack/make-rules/verify.sh ./hack/make-rules/vet.sh ./hack/print-workspace-status.sh ./hack/run-in-gopath.sh ./hack/test-go.sh ./hack/test-integration.sh ./hack/test-update-storage-objects.sh ./hack/update-all.sh ./hack/update-api-reference-docs.sh ./hack/update-bazel.sh ./hack/update-codegen.sh ./hack/update-generated-device-plugin-dockerized.sh ./hack/update-generated-device-plugin.sh ./hack/update-generated-docs.sh ./hack/update-generated-kms-dockerized.sh ./hack/update-generated-kms.sh ./hack/update-generated-kubelet-plugin-registration-dockerized.sh ./hack/update-generated-kubelet-plugin-registration.sh ./hack/update-generated-pod-resources-dockerized.sh ./hack/update-generated-pod-resources.sh ./hack/update-generated-protobuf-dockerized.sh ./hack/update-generated-protobuf.sh ./hack/update-generated-runtime-dockerized.sh ./hack/update-generated-runtime.sh ./hack/update-generated-swagger-docs.sh ./hack/update-godep-licenses.sh ./hack/update-gofmt.sh ./hack/update-openapi-spec.sh ./hack/update-staging-godeps-dockerized.sh ./hack/update-staging-godeps.sh ./hack/update-swagger-spec.sh ./hack/update-translations.sh ./hack/update-workspace-mirror.sh ./hack/verify-all.sh ./hack/verify-api-groups.sh ./hack/verify-api-reference-docs.sh ./hack/verify-bazel.sh ./hack/verify-boilerplate.sh ./hack/verify-cli-conventions.sh ./hack/verify-codegen.sh ./hack/verify-description.sh ./hack/verify-generated-device-plugin.sh ./hack/verify-generated-docs.sh ./hack/verify-generated-files-remake.sh ./hack/verify-generated-files.sh ./hack/verify-generated-kms.sh ./hack/verify-generated-kubelet-plugin-registration.sh ./hack/verify-generated-pod-resources.sh ./hack/verify-generated-protobuf.sh ./hack/verify-generated-runtime.sh ./hack/verify-generated-swagger-docs.sh ./hack/verify-godep-licenses.sh ./hack/verify-godeps.sh ./hack/verify-gofmt.sh ./hack/verify-golint.sh ./hack/verify-govet.sh ./hack/verify-import-boss.sh ./hack/verify-imports.sh ./hack/verify-linkcheck.sh ./hack/verify-no-vendor-cycles.sh ./hack/verify-openapi-spec.sh ./hack/verify-pkg-names.sh ./hack/verify-readonly-packages.sh ./hack/verify-spelling.sh ./hack/verify-staging-godeps.sh ./hack/verify-staging-meta-files.sh ./hack/verify-swagger-spec.sh ./hack/verify-symbols.sh ./hack/verify-test-featuregates.sh ./hack/verify-test-images.sh ./hack/verify-test-owners.sh ./hack/verify-typecheck.sh ./pkg/kubectl/cmd/edit/testdata/record_testcase.sh ./pkg/util/verify-util-pkg.sh ./plugin/pkg/admission/imagepolicy/gencerts.sh ./staging/src/k8s.io/apiextensions-apiserver/examples/client-go/hack/update-codegen.sh ./staging/src/k8s.io/apiextensions-apiserver/examples/client-go/hack/verify-codegen.sh ./staging/src/k8s.io/apiextensions-apiserver/hack/build-image.sh ./staging/src/k8s.io/apiextensions-apiserver/hack/update-codegen.sh ./staging/src/k8s.io/apiextensions-apiserver/hack/verify-codegen.sh ./staging/src/k8s.io/apiserver/pkg/admission/plugin/webhook/testcerts/gencerts.sh ./staging/src/k8s.io/apiserver/pkg/util/webhook/gencerts.sh ./staging/src/k8s.io/apiserver/plugin/pkg/authenticator/token/oidc/testdata/gen.sh ./staging/src/k8s.io/apiserver/plugin/pkg/authorizer/webhook/gencerts.sh ./staging/src/k8s.io/code-generator/generate-groups.sh ./staging/src/k8s.io/code-generator/generate-internal-groups.sh ./staging/src/k8s.io/code-generator/hack/update-codegen.sh ./staging/src/k8s.io/code-generator/hack/verify-codegen.sh ./staging/src/k8s.io/csi-api/hack/update-codegen.sh ./staging/src/k8s.io/csi-api/hack/verify-codegen.sh ./staging/src/k8s.io/kube-aggregator/hack/build-image.sh ./staging/src/k8s.io/kube-aggregator/hack/local-up-kube-aggregator.sh ./staging/src/k8s.io/kube-aggregator/hack/register-all-apis-from.sh ./staging/src/k8s.io/kube-aggregator/hack/update-codegen.sh ./staging/src/k8s.io/kube-aggregator/hack/verify-codegen.sh ./staging/src/k8s.io/metrics/hack/update-codegen.sh ./staging/src/k8s.io/metrics/hack/verify-codegen.sh ./staging/src/k8s.io/node-api/hack/update-codegen.sh ./staging/src/k8s.io/node-api/hack/verify-codegen.sh ./staging/src/k8s.io/sample-apiserver/hack/build-image.sh ./staging/src/k8s.io/sample-apiserver/hack/update-codegen.sh ./staging/src/k8s.io/sample-apiserver/hack/verify-codegen.sh ./staging/src/k8s.io/sample-controller/hack/update-codegen.sh ./staging/src/k8s.io/sample-controller/hack/verify-codegen.sh ./test/cmd/apply.sh ./test/cmd/apps.sh ./test/cmd/authorization.sh ./test/cmd/batch.sh ./test/cmd/certificate.sh ./test/cmd/core.sh ./test/cmd/crd.sh ./test/cmd/create.sh ./test/cmd/diff.sh ./test/cmd/discovery.sh ./test/cmd/generic-resources.sh ./test/cmd/get.sh ./test/cmd/initializers.sh ./test/cmd/legacy-script.sh ./test/cmd/node-management.sh ./test/cmd/old-print.sh ./test/cmd/proxy.sh ./test/cmd/rbac.sh ./test/cmd/request-timeout.sh ./test/cmd/run.sh ./test/cmd/save-config.sh ./test/cmd/storage.sh ./test/cmd/template-output.sh ./test/cmd/version.sh ./test/e2e_node/conformance/run_test.sh ./test/e2e_node/environment/setup_host.sh ./test/e2e_node/gubernator.sh ./test/e2e_node/jenkins/conformance/conformance-jenkins.sh ./test/e2e_node/jenkins/copy-e2e-image.sh ./test/e2e_node/jenkins/e2e-node-jenkins.sh ./test/e2e_node/jenkins/ubuntu-14.04-nvidia-install.sh ./test/images/image-util.sh ./test/images/pets/redis-installer/on-start.sh ./test/images/pets/zookeeper-installer/install.sh ./test/images/pets/zookeeper-installer/on-start.sh ./test/images/volume/gluster/run_gluster.sh ./test/images/volume/iscsi/create_block.sh ./test/images/volume/nfs/run_nfs.sh ./test/images/volume/rbd/bootstrap.sh ./test/images/volume/rbd/create_block.sh ./test/images/volume/rbd/mon.sh ./test/images/volume/rbd/osd.sh ./test/integration/ipamperf/test-performance.sh ./test/integration/scheduler_perf/test-performance.sh ./test/kubemark/common/util.sh ./test/kubemark/configure-kubectl.sh ./test/kubemark/gce/util.sh ./test/kubemark/iks/shutdown.sh ./test/kubemark/iks/startup.sh ./test/kubemark/iks/util.sh ./test/kubemark/master-log-dump.sh ./test/kubemark/pre-existing/util.sh ./test/kubemark/resources/start-kubemark-master.sh ./test/kubemark/run-e2e-tests.sh ./test/kubemark/start-kubemark.sh ./test/kubemark/stop-kubemark.sh ./third_party/forked/shell2junit/sh2ju.sh ./third_party/intemp/intemp.sh ./third_party/multiarch/qemu-user-static/register/qemu-binfmt-conf.sh ./third_party/multiarch/qemu-user-static/register/register.sh </ins>

Send feedback to sig-testing, kubernetes/test-infra and/or . /close
I'd like to reopen this, and eventually implement (first we need to deal with the failures ...) I've easily found a lot of bugs in test-infra and kubernetes/kubernetes bash with shellcheck. See also /assign
/reopen /remove-lifecycle rotten /priority important-longterm
Reopening this issue. <detailsIn response to : /reopen /remove-lifecycle rotten /priority important-longterm Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
We also have non-blocking and optionally-run presubmits, even just having an easy way for reviewers to request a shellcheck result for changed files seems like something we should do. I also think we should fix the existing failures. Glancing through some k/k bash with shellcheck in my editor, these generally seem to be things that are actually wrong.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten coming back to this hopefully
This is done now. Only took two years :^)


Question: } func (c *defaultAuthenticationInfoResolver) ClientConfigFor(server string) (*rest.Config, error) { <ins> return c.clientConfig(server) } func (c *defaultAuthenticationInfoResolver) ClientConfigForService(serviceName, serviceNamespace string) (*rest.Config, error) { return c.clientConfig(serviceName + "." + serviceNamespace + ".svc") } func (c *defaultAuthenticationInfoResolver) clientConfig(target string) (*rest.Config, error) { </ins> // exact match <del> if authConfig, ok := c.kubeconfig.AuthInfos[server]; ok { </del> <ins> if authConfig, ok := c.kubeconfig.AuthInfos[target]; ok { </ins> return restConfigFromKubeconfig(authConfig) } // star prefixed match <del> serverSteps := strings.Split(server, ".") </del> <ins> serverSteps := strings.Split(target, ".") </ins> for i := 1; i < len(serverSteps); i++ { nickName := "*." + strings.Join(serverSteps[i:], ".") if authConfig, ok := c.kubeconfig.AuthInfos[nickName]; ok {

We are getting the following error when the apiserver connects the webhook on localhost (configured via URL). We should only use the SSL tunnel for the connections to nodes when the webhooks are running as services. /sig api-machinery /kind bug /assign
/close


Question: utilpointer "k8s.io/utils/pointer" "github.com/onsi/ginkgo/v2" <ins> "github.com/onsi/gomega" </ins> imageutils "k8s.io/kubernetes/test/utils/image" )

Kubernetes e2e suite. Kubernetes e2e suite.[It] [sig-autoscaling] [Feature:HPA] Horizontal pod autoscaling (scale resource: CPU) [Serial] [Slow] ReplicationController Should scale from 1 pod to 3 pods and then from 3 pods to 5 pods and verify decision stability [sig-autoscaling, Feature:HPA, Serial, Slow] Kubernetes e2e suite.[It] [sig-autoscaling] [Feature:HPA] Horizontal pod autoscaling (scale resource: CPU) [Serial] [Slow] ReplicationController Should scale from 5 pods to 3 pods and then from 3 pods to 1 pod and verify decision stability [sig-autoscaling, Feature:HPA, Serial, Slow] Kubernetes e2e suite.[It] [sig-autoscaling] [Feature:HPA] [Serial] [Slow] Horizontal pod autoscaling (non-default behavior) with autoscaling disabled shouldn't scale down [sig-autoscaling, Feature:HPA, Serial, Slow] Kubernetes e2e suite.[It] [sig-autoscaling] [Feature:HPA] [Serial] [Slow] Horizontal pod autoscaling (non-default behavior) with autoscaling disabled shouldn't scale up [sig-autoscaling, Feature:HPA, Serial, Slow] Kubernetes e2e suite.[It] [sig-autoscaling] [Feature:HPA] [Serial] [Slow] Horizontal pod autoscaling (non-default behavior) with both scale up and down controls configured should keep recommendation within the range with stabilization window and pod limit rate [sig-autoscaling, Feature:HPA, Serial, Slow] Kubernetes e2e suite.


Question: utilpointer "k8s.io/utils/pointer" "github.com/onsi/ginkgo/v2" <ins> "github.com/onsi/gomega" </ins> imageutils "k8s.io/kubernetes/test/utils/image" )

[It] [sig-autoscaling] [Feature:HPA] [Serial] [Slow] Horizontal pod autoscaling (non-default behavior) with both scale up and down controls configured should keep recommendation within the range over two stabilization windows [sig-autoscaling, Feature:HPA, Serial, Slow] 11/3/2023 <*fmt.wrapError | 0xc000812080: client rate limiter Wait returned an error: context deadline exceeded { msg: "client rate limiter Wait returned an error: context deadline exceeded", err: <context.deadlineExceededError{}, } looks similiar to /sig autoscaling /sig windows
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: } waitSTSStable(t, c, sts) } <ins> var _ admission.ValidationInterface = &fakePodFailAdmission{} type fakePodFailAdmission struct { limitedPodNumber int succeedPodsCount int } func (f *fakePodFailAdmission) Handles(operation admission.Operation) bool { return operation == admission.Create } func (f *fakePodFailAdmission) Validate(ctx context.Context, attr admission.Attributes, o admission.ObjectInterfaces) (err error) { if attr.GetKind().GroupKind() != api.Kind("Pod") { return nil } if f.succeedPodsCount >= f.limitedPodNumber { return fmt.Errorf("fakePodFailAdmission error") } f.succeedPodsCount++ return nil } </ins>

I create a ResourceQuota with two pods limit in my namespace. Then, I create a statefuleSet with 3 replicas. As expected, two pods are created successfully, and the third pod failed to create. But, the ReadyReplicas is not 2 ,but 1. The ReadyReplicas is correct 1Create a ResourceQuota with two pods limit in a new namespace. 2Create a statefulSet with 3 replicas 3Check if the ReadyReplicas of statefulSet matchs the number of stateful running pods. 1According to the logic of the statefulSet controller, new pod created in current sync is not be immediately counted into the ReadyReplicas. 2Failure to create pod will cause to return error, then skip updating status. <details</details<details</details<detailskind v1.23.3 </details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig apps
/assign


Question: return stringOrDefaultValue(strings.Join(volumes, ","), "<all>") } <ins> func sysctlsToString(sysctls []string) string { return stringOrNone(strings.Join(sysctls, ",")) } </ins> func hostPortRangeToString(ranges []policy.HostPortRange) string { formattedString := "" if ranges != nil {

/kind bug What happened: does not show the description for the new field in What you expected to happen: should describe the new field. How to reproduce it (as minimally and precisely as possible): Environment: Kubernetes version (use ): master
/sig auth
GitHub didn't allow me to assign the following users: fanzhangio. Note that only and repo collaborators can be assigned. <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
If nobody has started yet, I wanna to pick it up.
was working on it AFAIR.
Indeed. if you don't mind, I'd take over.


Question: if err != nil { return nil, err } <ins> volumePath = strings.Replace(volumePath, "040", " ", -1) </ins> glog.V(5).Infof("vSphere volume path is %q", volumePath) vsphereVolume := &v1.Volume{ Name: volumeName,

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.5", GitCommit:"", GitTreeState:"clean", BuildDate:"2016-10-29T01:38:40Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.5", GitCommit:"", GitTreeState:"clean", BuildDate:"2016-10-29T01:32:42Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"} Environment: Cloud provider or hardware configuration: vSphere OS (e.g. from /etc/os-release):Kernel (e.g. ):Install tools: None Others: None What happened: When I create a deployment with vSphere cloud provider dynamic provisioning, it creates a volume spec with volume path - [datastore] kubevols/volume which contains a white space in it. In this case the cloud provider queries the getDeviceNameFromMount() to return the path of the volume mounted. Since getDeviceNameFromMount() queries the filesystem to get the mount references, it returns a volume path "[datastore]040kubevols/volume". Later the kubelet searches for this volume path in both the actual and desired states. Th actual and desired states contains volume with path "[datastore] kubevols/volume". So, it couldn't find such volume path and therefore kubernetes stalls unable to make any progress further similar to one described in .


Question: if err != nil { return nil, err } <ins> volumePath = strings.Replace(volumePath, "040", " ", -1) </ins> glog.V(5).Infof("vSphere volume path is %q", volumePath) vsphereVolume := &v1.Volume{ Name: volumeName,

Also similar issue described in What you expected to happen: All the pods in the recreated deployment comes up. How to reproduce it (as minimally and precisely as possible): See above. Anything else do we need to know: None
cc


Question: if err != nil { return nil, err } <ins> volumePath = strings.Replace(volumePath, "040", " ", -1) </ins> glog.V(5).Infof("vSphere volume path is %q", volumePath) vsphereVolume := &v1.Volume{ Name: volumeName,

I tried to create a kubernetes deployment with vSphere volume with volume path "[datastore] kubevols/redis-master". In this case the cloud provider queries the getDeviceNameFromMount() to return the path of the volume mounted. Since getDeviceNameFromMount() queries the filesystem to get the mount references, it returns a volume path "[datastore]040kubevols/redis-master". Later the kubelet searches for this volume path in both the actual and desired states. Th actual and desired states contains volume with path "[datastore] kubevols/redis-master". So, it couldn't find such volume path and therefore kubernetes stalls unable to make any progress further similar to one described in . However, when I create a kubernetes deployment with volume path - "[datastore]kubevols/redis-master" which doesn't contain any spaces, it works perfectly fine. So, the basic problem is to format and return the correct volume path from getDeviceNameFromMount() function. Alternatively, we can format it properly in the cloud provider code but this has to be done individually in all the cloud provider code base. Inorder to prevent it, it would be better to fix the getDeviceNameFromMount() function to return the correct formatted volume path.
CC Kubernetes should handle volumes with spaces (looks like vsphere is the first such volume) gracefully. Non a blocker issue for 1.5.
you should make sure that the 1.5 release notes capture that "vsphere volumes w/spaces have problems". Known issue accumulator in
getDeviceNameFromMount is function which queries /proc/mount to get mount points. 040 - is an escape sequence. 040 is the ascii code, in octal(base 8) number system, of the space. The mount command interprets 040 as literal space in the mount point. : Can't we provide a fix for this in 1.5 release itself? There a 2 options: can provide a fix in getDeviceNameFromMount. can provide a fix to vSphere cloud provider to fix it after the call to getDeviceNameFromMount. For option 1, we have to make sure it works for all cloud providers where as for option 2 its not needed. Let me know which option you think is feasible. I am ok with anything.


Question: if err != nil { return nil, err } <ins> volumePath = strings.Replace(volumePath, "040", " ", -1) </ins> glog.V(5).Infof("vSphere volume path is %q", volumePath) vsphereVolume := &v1.Volume{ Name: volumeName,

1 is the right answer. But we're very close to 1.5.0 release (<1 week). I'd also like whoever addresses this to ensure the entire system works end-to-end for volumes with spaces, not just patch one little corner. I'm afraid it would be a large change and we're not going to have enough time to validate it in 1.5. You can certainly start working on it. If you get a PR in time, we can consider it for 1.5.0. Otherwise it'll have to go in a patch release.


Question: import ( "context" <ins> "encoding/json" </ins> "fmt" "strconv" "time"

[x] Test PR : [x] Two weeks soak end date : 6 April 2022 [X] Test promotion PR : According to this APIsnoop query, there are still some remaining Job status endpoints which are untested. There are conformance tests for CronJob status endpoints already the existing as a template, a new for job test has been created. The e2e logs for this test are listed below. This query shows the endpoints hit within a short period of running the e2e test If a test with these calls gets merged, test coverage will go up by 3 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/triage accepted
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: } // validateCSINodeDriver tests if CSINodeDriver has valid entries <del> func validateCSINodeDriver(driver storage.CSINodeDriver, driverNamesInSpecs sets.String, fldPath *field.Path, </del> <ins> func validateCSINodeDriver(driver storage.CSINodeDriver, driverNamesInSpecs sets.Set[string], fldPath *field.Path, </ins> validationOpts CSINodeValidationOptions) field.ErrorList { allErrs := field.ErrorList{}

It seems to be valid to add here. Would you like to open a pull request for it? this is like a cleanup, as the set insert will skip and the validation will do twice. is used to track bugs. I suggest you open a PR for that if you like.
/sig storage
I think adding ` == true and we didn't skip, it will insert set twice I believe? wasn't notice that at first glance. However, I can file a PR now
/triage accepted
The label cannot be applied. Only GitHub organization members can add the label. <detailsIn response to : /triage accepted Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/triage accepted
Please feel free to submit a PR.


Question: return nil, nil, nil, err } <del> leaderElectionClient, err := clientset.NewForConfig(restclient.AddUserAgent(kubeConfig, "leader-election")) </del> <ins> // shallow copy, do not modify the kubeConfig.Timeout. restConfig := *kubeConfig restConfig.Timeout = timeout leaderElectionClient, err := clientset.NewForConfig(restclient.AddUserAgent(&restConfig, "leader-election")) </ins> if err != nil { return nil, nil, nil, err }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: kube-cm kube-scheduler does not exit when kube-apiserver suspended What you expected to happen: kube-cm kube-scheduler should exit after LeaderElector stops leading How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig api-machinery /kind bug
/assign
/cc


Question: fieldPath: metadata.namespace # END_PROMETHEUS_TO_SD nodeSelector: <del> # TODO(liggitt): switch to cloud.google.com/metadata-proxy-ready=true in v1.16 beta.kubernetes.io/metadata-proxy-ready: "true" </del> <ins> cloud.google.com/metadata-proxy-ready: "true" </ins> beta.kubernetes.io/os: linux terminationGracePeriodSeconds: 30

xref In 1.16: change the metadata-proxy addon to use change the kube-proxy addon to use change the masq-agent addon to use stop setting , , and in kube-up
Hey I'm trying to start to contribute to kubernetes and this seem to be an easy task. Can I help with this one?


Question: Logf("Getting external IP address for %s", node.Name) host := "" for _, a := range node.Status.Addresses { <del> if a.Type == v1.NodeExternalIP { </del> <ins> if a.Type == v1.NodeExternalIP && a.Address != "" { </ins> host = net.JoinHostPort(a.Address, sshPort) break }

Public IPs are expensive resources. Given that we're running scalability tests on 5k-node clusters, we are unnecessarily consuming 5k IPs which is a lot. We should switch our tests so that nodes have only private IP addresses. We are going to do that by configuring NAT so that nodes can talk to outside world (e.g. to download image from dockerhub). But that also requires changing some correctness tests which assume public IPs.
To summarize recent work: setting up private gce clusters with nat, feature is gated via the env test and dashboard with correctness tests on gce private cluster - https://k8s- 27 tests fail on fixing the tests - should fix majority of them
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale /lifecycle frozen
- is there anything missing to be done here?
No, we can close this one. It was done long time ago. Thanks for catching /close
Closing this issue. <detailsIn response to : No, we can close this one. It was done long time ago. Thanks for catching Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: // GCECloud is an implementation of Interface, LoadBalancer and Instances for Google Compute Engine. type GCECloud struct { <del> service                  *compute.Service containerService         *container.Service projectID                string region                   string localZone                string   // The zone in which we are running managedZones             []string // List of zones we are spanning (for Ubernetes-Lite, primarily when running on master) networkURL               string nodeTags                 []string // List of tags to use on firewall rules for load balancers useMetadataServer        bool operationPollRateLimiter flowcontrol.RateLimiter </del> <ins> service           *compute.Service containerService  *container.Service projectID         string region            string localZone         string   // The zone in which we are running managedZones      []string // List of zones we are spanning (for Ubernetes-Lite, primarily when running on master) networkURL        string nodeTags          []string // List of tags to use on firewall rules for load balancers useMetadataServer bool </ins> } type Config struct {

On a large cluster, the routecontroller basically makes O(n) requests all at once, and most of them error out with "Rate Limit Exceeded". We then wait for operations to complete on the ones that did stick, and damn the rest, they'll be caught next reconcile (which is a while). In fact, in some cases it appears we're spamming hard enough to get rate limited by the upper-level API DOS protections and seeing something akin to , e.g.: We can do better than this!
c.f. , which is the other leg of this problem
I took a look into code and it seems that we are doing full reconciliation every 10s. If we need to create 2000 routes at the beginning it seems to frequently. I think that we need to: first compute how many routes we have to create based on that try to spread them over some time to reduce possibility of causing 'rate limit exceeded' I will try to put some small PR together as a starting point.
I'm reopening this one, since this doesn't seem to be solved. So basically something strange is happening here. With PR we have 10/s limit on API calls send to GCE. And we are still rate-limitted. However, according to documentation we are allowed for 20/s: So something either doesn't work or is weird.
What is super interesting from the logs of recent run: It starts to creating routes: After 2 minutes from that, we are hitting "Rate limit Exceeded" for the first time: And only after 9 minutes we create the first route ever. The second one is created after 15 minutes, and from that point we create a number of them So it seems that nothing useful is happening for the first 2 minutes, and then between 3m30 and 9m30
OK - so I think that what is happening is that all controllers are sharing GCE interface (and thuse they have common throttling). So if one controller is generating a lot of API calls, then other controllers may be throttled. has a hypothesis that it may be caused by nodecontroller.
One thing my PR didn't address (because it was late): Now we have some obnoxious differences in logging when you go to make a request, because the versus could actually be absorbing a fair amount of ratelimiter.


Question: // GCECloud is an implementation of Interface, LoadBalancer and Instances for Google Compute Engine. type GCECloud struct { <del> service                  *compute.Service containerService         *container.Service projectID                string region                   string localZone                string   // The zone in which we are running managedZones             []string // List of zones we are spanning (for Ubernetes-Lite, primarily when running on master) networkURL               string nodeTags                 []string // List of tags to use on firewall rules for load balancers useMetadataServer        bool operationPollRateLimiter flowcontrol.RateLimiter </del> <ins> service           *compute.Service containerService  *container.Service projectID         string region            string localZone         string   // The zone in which we are running managedZones      []string // List of zones we are spanning (for Ubernetes-Lite, primarily when running on master) networkURL        string nodeTags          []string // List of tags to use on firewall rules for load balancers useMetadataServer bool </ins> } type Config struct {

It could before because of the operation rate limiter and operation polling anyways, but now could be worse.
- yes I'm aware of that. And probably this is exactly the case here.
I got fooled by that - I guess we should fix it somehow.
One approach to fixing it is a similar approach to the operation polling code, but in the : warn if we ratelimited for more than N seconds (and dump the request or something? not clear how to clarify what we were sending).
I found something else - Kubelet is also building GCE interface: We have 2000 kubelets, so if all of them send multiple requests to GCE, we may hit limits.
Any idea in what circumstances Kubelet will contact cloud provider?
Hmm - it seems it contacts GCE exactly once at the beginning. So it doesn't explain much.
I think it's only here: I've rarely seen GCE ratelimit readonly calls, in practice, and kubelets come up pretty well staggered.
OK - so with logs that I , it is pretty clear that we are very heavily throttled on GCE api calls. In particular, when I was running 1000-node cluster, I see lines where we were throttle for 1m30. I'm pretty sure it's significantly higher in 2000-node cluster. I'm going to investigate a bit more where all those reuqests come from.
Actually - it seems that we have a pretty big problem here. Basically, Kubelet seems to send a lot of requests to GCE. So to clarify: every time Kubelet is updating NodeStatus (so every 10s by default) it is sending an API call to GCE to get it's not addresses: that means, that if we have 2000 nodes, kubelets generate 200 QPS to GCE (which is significantly more than we can afford) We should think how to solve this issue, but I think the options are: increasing QPS quota or calling GCE only once per X statusUpdates.
The second issue is that we are throttled at the controller level too, and this one I'm tryin to understand right now.


Question: // GCECloud is an implementation of Interface, LoadBalancer and Instances for Google Compute Engine. type GCECloud struct { <del> service                  *compute.Service containerService         *container.Service projectID                string region                   string localZone                string   // The zone in which we are running managedZones             []string // List of zones we are spanning (for Ubernetes-Lite, primarily when running on master) networkURL               string nodeTags                 []string // List of tags to use on firewall rules for load balancers useMetadataServer        bool operationPollRateLimiter flowcontrol.RateLimiter </del> <ins> service           *compute.Service containerService  *container.Service projectID         string region            string localZone         string   // The zone in which we are running managedZones      []string // List of zones we are spanning (for Ubernetes-Lite, primarily when running on master) networkURL        string nodeTags          []string // List of tags to use on firewall rules for load balancers useMetadataServer bool </ins> } type Config struct {

- that said, any rate-limitting on our side, will not solve the kubelet-related problem
Hmm - actually looking into implementation it seems that NodeAddresses are only contacting metadata server, so I'm not longer that convinced...
Also, I think that important thing is that a single call to GCE API can translate to multiple api calls to GCE. In particular - CreateRoute translates to: get instance insert route a bunch of getOperation (until this is finished)
OK - so I think that what is happening here is that since CreateRoute translates to: get instance insert route a bunch of getOperations That means that if we call say 2000 CreateRoute() at the same time, all of them will try to issue: getInstance at the beginning. So we will quickly accumulate 2000 GET requests in the queue. Once they are processed, we generate the "POST route" then for any processed. So It's kind of expected what is happening here. So getting back to the Zach`s PR - I think that throttling POST and GET and OPERATION calls separate is kind of good idea in general.
But I still don't really understand why do we get "RateLimitExceeded". Is that because of Kubelets?
I think that one problem we have at the RouteController level is that if CreateRoute fails, we simply don't react on it. And we will wait at least 5 minutes before even retrying it. We should add some retries at the route controller level too. I will send out some short PR for it.
But I still don't understand why do we get those "RateLimitExceeded" error
OK - so as an update. In GCE, we have a separate rate-limit for number of in-flight CreateRoute call per project. is supposed to address this issue.
OK - so this one is actually fixed. I mean that it is still long, but currently it's purely blocked on GCE.
As a comment - we have an internal bug for it already.


Question: ConsumeMemAddress       = "/ConsumeMem" BumpMetricAddress       = "/BumpMetric" GetCurrentStatusAddress = "/GetCurrentStatus" <del> MetricsAddress          = "/Metrics" </del> <ins> MetricsAddress          = "/metrics" </ins> MillicoresQuery              = "millicores" MegabytesQuery               = "megabytes"

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: Used the endpoint /BumpMetric to create fake metrics for Prometheus. When I tried to curl /metrics I got the following response: Bad request. Not a POST request What you expected to happen: I expected to see something like this: # HELP info message. # TYPE gauge 0. How to reproduce it (as minimally and precisely as possible): Following the docs, after having it running inside kubernetes: Anything else we need to know?: Looks like introduced this by mistake. Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
/sig testing


Question: <ins> /* Copyright 2018 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package cmd import ( "fmt" "io" "sort" "strings" "github.com/spf13/cobra" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/runtime/schema" "k8s.io/apimachinery/pkg/util/sets" "k8s.io/kubernetes/pkg/kubectl/cmd/templates" cmdutil "k8s.io/kubernetes/pkg/kubectl/cmd/util" "k8s.io/kubernetes/pkg/printers" ) var ( apiresources_example = templates.Examples(` # Print the supported API Resources kubectl api-resources # Print the supported API Resources with more information kubectl api-resources -o wide # Print the supported namespaced resources kubectl api-resources --namespaced=true # Print the supported non-namespaced resources kubectl api-resources --namespaced=false # Print the supported API Resources with specific APIGroup kubectl api-resources --api-group=extensions`) ) // ApiResourcesOptions is the start of the data required to perform the operation.  As new fields are added, add them here instead of // referencing the cmd.Flags() type ApiResourcesOptions struct { out io.Writer Output     string APIGroup   string Namespaced bool NoHeaders  bool } // groupResource contains the APIGroup and APIResource type groupResource struct { APIGroup    string APIResource metav1.APIResource } func NewCmdApiResources(f cmdutil.Factory, out io.Writer) *cobra.Command { options := &ApiResourcesOptions{ out: out, } cmd := &cobra.Command{ Use:     "api-resources", Short:   "Print the supported API resources on the server", Long:    "Print the supported API resources on the server", Example: apiresources_example, Run: func(cmd *cobra.Command, args []string) { cmdutil.CheckErr(options.Complete(cmd)) cmdutil.CheckErr(options.Validate(cmd)) cmdutil.CheckErr(options.RunApiResources(cmd, f)) }, } cmdutil.AddOutputFlags(cmd) cmdutil.AddNoHeadersFlags(cmd) cmd.Flags().StringVar(&options.APIGroup, "api-group", "", "The API group to use when talking to the server.") cmd.Flags().BoolVar(&options.Namespaced, "namespaced", true, "Namespaced indicates if a resource is namespaced or not.") return cmd } func (o *ApiResourcesOptions) Complete(cmd *cobra.Command) error { o.Output = cmdutil.GetFlagString(cmd, "output") o.NoHeaders = cmdutil.GetFlagBool(cmd, "no-headers") return nil } func (o *ApiResourcesOptions) Validate(cmd *cobra.Command) error { validOutputTypes := sets.NewString("", "json", "yaml", "wide", "name", "custom-columns", "custom-columns-file", "go-template", "go-template-file", "jsonpath", "jsonpath-file") supportedOutputTypes := sets.NewString("", "wide") outputFormat := cmdutil.GetFlagString(cmd, "output") if !validOutputTypes.Has(outputFormat) { return fmt.Errorf("output must be one of '' or 'wide': %v", outputFormat) } if !supportedOutputTypes.Has(outputFormat) { return fmt.Errorf("--output %v is not available in kubectl api-resources", outputFormat) } return nil } func (o *ApiResourcesOptions) RunApiResources(cmd *cobra.Command, f cmdutil.Factory) error { w := printers.GetNewTabWriter(o.out) defer w.Flush() discoveryclient, err := f.DiscoveryClient() if err != nil { return err } // Always request fresh data from the server discoveryclient.Invalidate() lists, err := discoveryclient.ServerPreferredResources() if err != nil { return err } resources := []groupResource{} groupChanged := cmd.Flags().Changed("api-group") nsChanged := cmd.Flags().Changed("namespaced") for _, list := range lists { if len(list.APIResources) == 0 { continue } gv, err := schema.ParseGroupVersion(list.GroupVersion) if err != nil { continue } for _, resource := range list.APIResources { if len(resource.Verbs) == 0 { continue } // filter apiGroup if groupChanged && o.APIGroup != gv.Group { continue } // filter namespaced if nsChanged && o.Namespaced != resource.Namespaced { continue } resources = append(resources, groupResource{ APIGroup:    gv.Group, APIResource: resource, }) } } if o.NoHeaders == false { if err = printContextHeaders(w, o.Output); err != nil { return err } } sort.Stable(sortableGroupResource(resources)) for _, r := range resources { if o.Output == "wide" { if _, err := fmt.Fprintf(w, "%st%st%st%vt%st%vn", r.APIResource.Name, strings.Join(r.APIResource.ShortNames, ","), r.APIGroup, r.APIResource.Namespaced, r.APIResource.Kind, r.APIResource.Verbs); err != nil { return err } } else { if _, err := fmt.Fprintf(w, "%st%st%st%vt%sn", r.APIResource.Name, strings.Join(r.APIResource.ShortNames, ","), r.APIGroup, r.APIResource.Namespaced, r.APIResource.Kind); err != nil { return err } } } return nil } func printContextHeaders(out io.Writer, output string) error { columnNames := []string{"NAME", "SHORTNAMES", "APIGROUP", "NAMESPACED", "KIND"} if output == "wide" { columnNames = append(columnNames, "VERBS") } _, err := fmt.Fprintf(out, "%sn", strings.Join(columnNames, "t")) return err } type sortableGroupResource []groupResource func (s sortableGroupResource) Len() int      { return len(s) } func (s sortableGroupResource) Swap(i, j int) { s[i], s[j] = s[j], s[i] } func (s sortableGroupResource) Less(i, j int) bool { ret := strings.Compare(s[i].APIGroup, s[j].APIGroup) if ret > 0 { return false } else if ret == 0 { return strings.Compare(s[i].APIResource.Name, s[j].APIResource.Name) < 0 } return true } </ins>

As the RBAC role need to be related to resources. I think we could use a command to get the supported resources. ref to
This would be cool.
I thought we a command that listed resources in the discover doc a long time ago.
is just the versions, but principle is the same. We want to list all the types, their resources, and tie that back to "explain".
What's the best way to determine this information currently? I'm having difficulty figuring out the exact set of permissions (and how they would be written in an RBAC role) for any given kubectl command. The documentation for this right now is split across multiple places and certain things are not explained at all. Specifically, it's difficult to determine which "API verb" is used for which operations. Take as an example. It's unclear if the way to allow this is to use the "proxy" subresource of a pod or service with "get"/"create"/etc. verbs, or to use the "proxy" verb with the resource (pod, service) directly. is a similar case. The almost works for this purpose, but it's still difficult. Operations for a given resource are grouped into categories like write, read, status, and proxy, but these things don't obviously correlate with API verbs. Again, proxy is especially confusing as there is no mention of a proxy "verb". The API operations are only shown using their HTTP verbs. One thing that could help on this document would be a section on the right column under "kubectl command" that shows exactly the rule needed in an RBAC role to allow this command via kubectl. And even then, that assumes an exact 1-to-1 mapping between API endpoints and kubectl commands, which I doubt is always the case. (For example, port fowarding to a pod seems to require both a GET on the pod resource and a CREATE on a port forward subresource.) There also needs to be better documentation about how this information could have been determined in the first place. I've resorted to manually looking through API requests made with but even then it's difficult to figure out. Amusingly, I was the one who wrote which is referenced in the original issue description (as well as the table explaining verbs in ).