Question: t.Fatalf("unexpected error: %v", err) } if !reflect.DeepEqual(testCase.Limiter, client.rateLimiter) { <del> t.Fatalf("unexpected rate limiter: %#v", client.rateLimiter) </del> <ins> t.Fatalf("unexpected rate limiter: %#v, expected %#v at %s", client.rateLimiter, testCase.Limiter, testCase.Name) </ins> } }) }

Reading TestRESTClientLimiter in I noticed a few things. test case struct is defined to have a name field, and its value is used in naming the subtests, but the actual test cases leave the name blank. error statements do not identify which case is failing (do not identify the expected value). Should be clear from the above. Read the code. No response_ <detailsbranch </details<detailsN/A </details<detailsN/A </details<details</details<details</details<details</details
/sig api-machinery
/assign
/triage accepted


Question: set -o nounset set -o pipefail <del> KUBE_ROOT=$(dirname "${BASH_SOURCE}")/.. source "${KUBE_ROOT}/hack/lib/init.sh" </del> <ins> KUBE_HACK_ROOT=$(dirname "${BASH_SOURCE}") </ins> <del> kube::golang::setup_env </del> <ins> echo "WARNING: hack/generate-docs.sh is an alias for hack/update-generated-docs.sh" echo "and will be removed in a future version." </ins> <del> BINS=( cmd/gendocs cmd/genkubedocs cmd/genman cmd/genyaml ) make -C "${KUBE_ROOT}" WHAT="${BINS[*]}" kube::util::ensure-temp-dir kube::util::gen-docs "${KUBE_TEMP}" # remove all of the old docs kube::util::remove-gen-docs # copy fresh docs into the repo. # the shopt is so that we get docs/.generated_docs from the glob. shopt -s dotglob cp -af "${KUBE_TEMP}"/* "${KUBE_ROOT}" shopt -u dotglob </del> <ins> "${KUBE_HACK_ROOT}"/update-generated-docs.sh </ins>

Instead, have a presubmit test that verifies that the docs can be generated. This will reduce the number of conflicts & rebases, therefore increase speed through the queue. Confirmed with that nothing depends on these docs at the moment. fyi, some manual process might have to change once this is in.
Yes, we want to generate the docs into , anyway. cc
Excellent. One closer step to YAML generation, I hope. :)
was interested in working on this.
Want to get started on this?
is this still a goal of ours?
I've seen a thread this week-end about that. I think docs have moved but we still need to generate them there.
can this issue be closed then?
related to ?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale /lifecycle frozen There's still cruft in the docs directory.
/remove-help I don't think this meets . Please add back via if I'm incorrect
/sig docs /sig release /remove-area test-infra /area build-release /area kubeadm /area kubectl All of the following dirs/files have the following content. Do we need them? This file is autogenerated, but we've stopped checking such files into the repository to reduce the need for rebases. Please run hack/generate- to populate this file. docs/admin: .md files for kube components and kubeadm commands man/man1: .1 files for same user-guide/kubectl: .md files for kubectl commands yaml/kubectl: .yaml files for kubectl commands
/help Now that we have /good-first-issue, I think this does meet the criteria for help wanted. Probably need to talk to someone from sig-docs and figure out if/how these are used.
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command.


Question: set -o nounset set -o pipefail <del> KUBE_ROOT=$(dirname "${BASH_SOURCE}")/.. source "${KUBE_ROOT}/hack/lib/init.sh" </del> <ins> KUBE_HACK_ROOT=$(dirname "${BASH_SOURCE}") </ins> <del> kube::golang::setup_env </del> <ins> echo "WARNING: hack/generate-docs.sh is an alias for hack/update-generated-docs.sh" echo "and will be removed in a future version." </ins> <del> BINS=( cmd/gendocs cmd/genkubedocs cmd/genman cmd/genyaml ) make -C "${KUBE_ROOT}" WHAT="${BINS[*]}" kube::util::ensure-temp-dir kube::util::gen-docs "${KUBE_TEMP}" # remove all of the old docs kube::util::remove-gen-docs # copy fresh docs into the repo. # the shopt is so that we get docs/.generated_docs from the glob. shopt -s dotglob cp -af "${KUBE_TEMP}"/* "${KUBE_ROOT}" shopt -u dotglob </del> <ins> "${KUBE_HACK_ROOT}"/update-generated-docs.sh </ins>

<detailsIn response to : /help Now that we have /good-first-issue, I think this does meet the criteria for help wanted. Probably need to talk to someone from sig-docs and figure out if/how these are used. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-priority important-soon /priority important-longterm This hasn't been committed to for any release cycle in a while. I hope someone does commit to it, but I personally cannot, so I'm bumping down to the "hopefully next time" priority. /kind cleanup in addition to docs This impacts merge velocity of any kubectl/kubeadm changes because the docs/OWNERS file is small. We're going to work around by adding OWNERS files for the relevant sigs to the relevant dirs. But if these files don't have to exist in the first place, that'd be even better.
as discovered today this would not work as OWNERS in sub-folders do not solve the problem with the /docs/.generated_docs file which is in the parent folder. i've been looking into a solution to remove the placeholders completely and make verify-docs only consist of kube::util::gen-docs. this will: ensure that godoc errors and such still block presubmits the contributors are not blocked having to commit placeholder files
/lifecycle active
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale /lifecycle active as PR is still in flight.


Question: loc.Path += "/" } <del> // From pkg/genericapiserver/endpoints/handlers/proxy.go#ServeHTTP: // Redirect requests with an empty path to a location that ends with a '/' // This is essentially a hack for http://issue.k8s.io/4958. // Note: Keep this code after tryUpgrade to not break that flow. if len(loc.Path) == 0 { var queryPart string if len(req.URL.RawQuery) > 0 { queryPart = "?" + req.URL.RawQuery } w.Header().Set("Location", req.URL.Path+"/"+queryPart) w.WriteHeader(http.StatusMovedPermanently) </del> <ins> proxyRedirect := proxyRedirectsforRootPath(loc.Path, w, req) if proxyRedirect { </ins> return }

What happened: As part of creating an e2e test for , it was found that the apiserver proxy server is processing a blank redirect for any for 21 endpoints across pods, services and nodes. For a pod, redirecting for any method other than GET or HEAD is a bug, you expected to happen: This is the current behaviour for the apiserver but with a minimal code change the redirect can apply to only the following endpoints; - - - How to reproduce it (as minimally and precisely as possible): Create any standard pod and service in the default namespace run in another terminal the response from running the following commands produces a can be replaced with other http verbs , , , , and which also produce 301 responses. Anything else we need to know?: networking /sig testing /sig architecture
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : What happened: - - - run in another terminal the response from running the following commands produces a testing /sig architecture Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Does this issue and PR address your concerns about the apiserver proxy server doing a blanket direct of any http call to the root proxy endpoints?
commented on the associated PR, thanks
/remove-sig testing
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale Will be addressed late in 1.21
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.


Question: loc.Path += "/" } <del> // From pkg/genericapiserver/endpoints/handlers/proxy.go#ServeHTTP: // Redirect requests with an empty path to a location that ends with a '/' // This is essentially a hack for http://issue.k8s.io/4958. // Note: Keep this code after tryUpgrade to not break that flow. if len(loc.Path) == 0 { var queryPart string if len(req.URL.RawQuery) > 0 { queryPart = "?" + req.URL.RawQuery } w.Header().Set("Location", req.URL.Path+"/"+queryPart) w.WriteHeader(http.StatusMovedPermanently) </del> <ins> proxyRedirect := proxyRedirectsforRootPath(loc.Path, w, req) if proxyRedirect { </ins> return }

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
/remove-lifecycle stale
/remove-lifecycle stale


Question: 97099dccc8807e9159dc28f374a8f0602cab07e1 kube::util::go_install_from_commit  github.com/bazelbuild/bazel-gazelle/cmd/gazelle  <del> 39e8257565d3917c1674bf37d6538c10cdac2260 </del> <ins> a85b63b06c2e0c75931e57c4a1a18d4e566bb6f4 </ins> touch "${KUBE_ROOT}/vendor/BUILD"

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: What you expected to happen: I expected the update to run successfully. How to reproduce it (as minimally and precisely as possible): Checkout. . Anything else we need to know?: I get this in both master and release-1.9. I tested most carefully in release-1.9 (because I am working on a cherry-pick right now). In particular, I blew away the container and image I had used earlier for building. Before running I established this state: Then I ran --- which took a while but eventually completed. It left me this state: After that is when I ran the that is exhibited at the start of this issue. Environment: Kubernetes version (use ): Commit (current release-1.9) Cloud provider or hardware configuration: local VM OS (e.g. from /etc/os-release): Ubuntu 17.10 Kernel (e.g. ): Linux ubu17dev 4.13.0-21-generic -Ubuntu SMP Mon Dec 18 17:29:16 UTC 2017 x8664 x8664 x86_64 GNU/Linux Install tools: Others: Docker 17.12.0-ce
Ah, there's another bug for this, this is being fixed. (same issue, verify uses update internally)
Any workaround?
/reopen It looks like the 1.8 and 1.9 release branches are broken for the same reason, but the fix is a bit less trivial, since we can't just use the latest gazelle, since this would require a huge mass of changes to the BUILD files. (Luckily, 1.7 doesn't use gazelle, so it's unaffected.) Some additional background: gazelle doesn't vendor bazelbuild/buildtools, though it does have a bazel dependency on a particular version. As a result, is always grabbing the latest version of bazelbuild/buildtools, even with our hacks to check out an older version of gazelle. An additional complication for the 1.8 and 1.9 branches is that we're installing gazelle from bazelbuild/rulesgo instead of bazelbuild/bazel-gazelle.


Question: 97099dccc8807e9159dc28f374a8f0602cab07e1 kube::util::go_install_from_commit  github.com/bazelbuild/bazel-gazelle/cmd/gazelle  <del> 39e8257565d3917c1674bf37d6538c10cdac2260 </del> <ins> a85b63b06c2e0c75931e57c4a1a18d4e566bb6f4 </ins> touch "${KUBE_ROOT}/vendor/BUILD"

Various options under consideration to fix these release branches: vendor gazelle, similar to (but not a straight cherrypick, since versions are different) the bazelbuild/rulesgo repo somewhere (where?) and then explicitly vendor and older version of bazelbuild/buildtools, and then use that fork instead of the upstream fork the hack in to also check out a particular bazelbuild/buildtools revision (so hacky) Anyone have a better idea?
you can't re-open an issue/PR unless you authored it or you are assigned to it. <detailsIn response to : /reopen Fully vendor gazelle, similar to (but not a straight cherrypick, since versions are different) Clone the bazelbuild/rulesgo repo somewhere (where?) and then explicitly vendor and older version of bazelbuild/buildtools, and then use that fork instead of the upstream fork Extend the hack in to also check out a particular bazelbuild/buildtools revision (so hacky) Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign /reopen
you can't re-open an issue/PR unless you authored it or you are assigned to it. <detailsIn response to : /assign /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I think we should start with 2) to get things fixed and then solve 1) for the current branch and come back to the release branches if we think we still need that.
can you describe 1) and 2) here?
I ended up implementing option 3 in ; it wasn't quite as gross as I thought it'd be. LMK what you think.
1) and 2) from the bullets in
ahh, thanks.
many thanks... would the same fix work in release-1.7 and release-1.8? trying to get those in shape as well
PRs are out for 1.8 and 1.9. The same thing would work for 1.7 AFAIK Edit: see below, I am wrong WRT 1.7
1.7 is unaffected because it doesn't use gazelle.


Question: 97099dccc8807e9159dc28f374a8f0602cab07e1 kube::util::go_install_from_commit  github.com/bazelbuild/bazel-gazelle/cmd/gazelle  <del> 39e8257565d3917c1674bf37d6538c10cdac2260 </del> <ins> a85b63b06c2e0c75931e57c4a1a18d4e566bb6f4 </ins> touch "${KUBE_ROOT}/vendor/BUILD"

I just opened with the same hacky fix for 1.8. apologies for all these breakages. my technical debt caught up to me. :(
/close this should be fixed everywhere, for now.


Question: 97099dccc8807e9159dc28f374a8f0602cab07e1 kube::util::go_install_from_commit  github.com/bazelbuild/bazel-gazelle/cmd/gazelle  <del> 39e8257565d3917c1674bf37d6538c10cdac2260 </del> <ins> a85b63b06c2e0c75931e57c4a1a18d4e566bb6f4 </ins> touch "${KUBE_ROOT}/vendor/BUILD"

http://k8s- looks like something is indeed busted :- /priority failing-test /kind bug /status approved-for-milestone /sig testing cc /assign
/assign looks like Mike is working on the fix


Question: if [[ -n "${move_docker}" ]]; then # Move docker to e.g. /mnt <ins> # but only if it is a directory, not a symlink left over from a previous run </ins> if [[ -d /var/lib/docker ]]; then mv /var/lib/docker ${move_docker}/ fi mkdir -p ${move_docker}/docker <del> ln -s ${move_docker}/docker /var/lib/docker </del> <ins> # If /var/lib/docker doesn't exist (it will exist if it is already a symlink), # then symlink it to the ephemeral docker area if [[ ! -e /var/lib/docker ]]; then ln -s ${move_docker}/docker /var/lib/docker fi </ins> DOCKER_ROOT="${move_docker}/docker" DOCKER_OPTS="${DOCKER_OPTS:-} -g ${DOCKER_ROOT}" fi

On AWS we (will) now run our bootstrap script on every boot. The function to initialize ephemeral volumes doesn't tolerate being re-run, and we probably shouldn't re-run it anyway for speed. This does preserve the contents, which may be a good thing to save user's data (it doesn't belong on ephemeral drives, but...). On the other hand, if the ephemeral drives are corrupted, we lose the opportunity to reformat them and recover. Arguably, if the ephemeral storage is corrupted the right thing to do is to terminate the instance entirely.


Question: create_storage_class fi <ins> if [[ "${FEATURE_GATES:-}" == "AllAlpha=true" || "${FEATURE_GATES:-}" =~ "CSIDriverRegistry=true" ]]; then create_csi_crd "csidriver" fi if [[ "${FEATURE_GATES:-}" == "AllAlpha=true" || "${FEATURE_GATES:-}" =~ "CSINodeInfo=true" ]]; then create_csi_crd "csinodeinfo" fi </ins> print_success if [[ "${ENABLE_DAEMON}" = false ]]; then

<!-- Please only use this template for submitting enhancement requests --What would you like to be : a new addon to addon manager "that automatically installs CSI CRDs if CSIDriverRegistry or CSINodeInfo feature gates are true". But this is either broken or this addon manager isn't active in a cluster brough up with: This should work automatically also for such local clusters. Why is this needed: The CSI E2E tests are now expecting that the cluster has the CRDs and fail when that isn't the case. /kind feature (or perhaps a bug, not sure) /cc
/assign
Is this expected to be installed in other clusters? A lot of clusters are not going to be using cluster/* scripts.


Question: Close() // DeleteNominatedPodIfExists deletes nominatedPod from internal cache DeleteNominatedPodIfExists(pod *v1.Pod) <ins> // NumUnschedulablePods returns the number of unschedulable pods exist in the SchedulingQueue. NumUnschedulablePods() int </ins> } // NewSchedulingQueue initializes a new scheduling queue. If pod priority is

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Today, the scheduler re-processes unschedulable nodes . Node updates are sent by Kubelet frequently (every 10 seconds) and often times they don't change any of the node properties that could potentially make them feasible. We should be smarter about these updates and move pods to the schedulable queue only when certain properties of nodes change that could make them feasible. These properties include node allocatable, node conditions, taints, labels, .... Why is this needed: Improves performance of the scheduler and avoids doing unnecessary attempts to schedule pods. This is particularly helpful when a cluster has a lot of unschedulable pods. <!-- DO NOT EDIT BELOW THIS LINE --/kind feature This is not a feature really. It is performance optimization, but in the absence of such a label, I need to use "feature" here. /sig scheduling
I can help to fix it. /assign mlmhl
The main reason for this problem is / update their status too frequently and most of these updates are heartbeats without any meaningful change. Recently sig-node introduces a new to alleviate this problem. With this mechanism, kubelet will split heartbeat and node status updates into two independent operations, and reduce the frequency of status updates to once per 1 minute(or more). In the long run this mechanism can indeed alleviate our problems, but at present it does not help us much as this is still an alpha feature and the node status updates frequency is still a bit high(per 1 minute right now).
Maybe we can try comparing old and new nodes after stripping their heartbeat info, like this:
I think that had sent earlier is more efficient. However, that PR does not filter out . In Wei's code, the heartbeat time is removed correctly. I think we should fix this issue and resubmit that PR.
could you resubmit a PR to: disregard in conditions comparison: and (optionally) disregard timestamp in taints comparison:
Thanks for all your advice, a new PR is submitted: This PR does not include your second suggestion(disregard timestamp in taints comparison) as I'm not sure if can be changed when node updating.


Question: Close() // DeleteNominatedPodIfExists deletes nominatedPod from internal cache DeleteNominatedPodIfExists(pod *v1.Pod) <ins> // NumUnschedulablePods returns the number of unschedulable pods exist in the SchedulingQueue. NumUnschedulablePods() int </ins> } // NewSchedulingQueue initializes a new scheduling queue. If pod priority is

If it can be changed, I will add this filter later, otherwise I think we can keep the current implementation to make the compare operation simpler and faster. What's more, just as I said in the PR's description, I'm not sure how much side effects this change has on scheduler's resource consumption, as it always copy node's conditions before comparison.
I am thinking whether we could move this logic(whether node became more schedulable) to kubelet, kubelet reports it through heartbeats.


Question: // BeforeEach gets a client and makes a namespace. func (f *Framework) BeforeEach() { <ins> f.beforeEachStarted = true </ins> // The fact that we need this feels like a bug in ginkgo. // https://github.com/onsi/ginkgo/issues/222 f.cleanupHandle = AddCleanupAction(f.AfterEach)

What happened: When running external CSI Driver tests with the Kubernetes E2E: We seem to be getting the following error. From initial investigation it looks like this happens in the during check but only when the test is being skipped. Hunch is that maybe since the test is skipped some resources are never created or they are cleaned up already and they are nil when the is called and expecting them. Kind of low priority since it doesn't make any tests actually fail. /cc /sig storage
see
I think should address this


Question: } if shareName == "" { <del> // File share name has a length limit of 63, and it cannot contain two consecutive '-'s. </del> <ins> // File share name has a length limit of 63, it cannot contain two consecutive '-'s, and all letters must be lower case. </ins> name := util.GenerateVolumeName(a.options.ClusterName, a.options.PVName, 63) shareName = strings.Replace(name, "--", "-", -1) <ins> shareName = strings.ToLower(shareName) </ins> } if resourceGroup == "" {

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --pull-kubernetes-e2e-aks-engine-azure-file pull-kubernetes-e2e-azure-file-windows AzureFile CSI Driver End-to-End Tests: Dynamic Provisioning should create a volume on demand with mount options [] [] [Windows] AzureFile CSI Driver End-to-End Tests: Dynamic Provisioning should create multiple PV objects, bind to PVCs and attach all to different pods on the same node [] [] [Windows] AzureFile CSI Driver End-to-End Tests: Dynamic Provisioning should create a volume on demand and mount it as readOnly in a pod [] [] [Windows] If the randomly generated kubetest resource group has capital letters, the share name will contain invalid characters and fail provisioning. links to appreciated links to specific failures in spyglass appreciated <!-- Please see the deflaking doc () for more guidance! --
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: func TestUpdateDisruptedPods(t *testing.T) { dc, ps := newFakeDisruptionController() <del> dc.recheckQueue = workqueue.NewNamedDelayingQueue("pdb-queue") </del> <ins> dc.recheckQueue = workqueue.NewNamedDelayingQueue("pdb_queue") </ins> pdb, pdbName := newMinAvailablePodDisruptionBudget(t, intstr.FromInt(1)) currentTime := time.Now() pdb.Status.DisruptedPods = map[string]metav1.Time{

What happened: From : What you expected to happen: Metrics should register successfully. How to reproduce it (as minimally and precisely as possible): Run kubernetes-integration test Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
There are no sig labels on this issue. Please add a sig label by either: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. See the . The in method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: // // alpha: v1.4 // MyFeature() bool <del> // owner @smarterclayton // alpha: v1.16 // beta:  v1.19 // // Enable legacy behavior to vary cluster functionality on the node-role.kubernetes.io labels. On by default (legacy), will be turned off in 1.18. // Original copy from k8s.io/kubernetes/pkg/features/kube_features.go LegacyNodeRoleBehavior featuregate.Feature = "LegacyNodeRoleBehavior" // owner @brendandburns // alpha: v1.9 // beta:  v1.19 // ga:  v1.21 // // Enable nodes to exclude themselves from service load balancers // Original copy from k8s.io/kubernetes/pkg/features/kube_features.go ServiceNodeExclusion featuregate.Feature = "ServiceNodeExclusion" </del> // owner: @khenidak // alpha: v1.15 //

What is missing? Based on SIG call it's probably "tests" but I am not sure. Could use some investigation and followup - how do we get it to GA? /sig network /kind cleanup
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/assign
This one seems not a good first one, I'll take this to myself and start bothering people in Slack to check what's missing, mostly as I've seen this is also related to cloud providers :)
I think this one originally. According to the KEP in This should have went GA in v.19 and be removed in v1.20. Given it's still marked beta, at the very least we should mark it GA in v1.20 and remove it in v1.21. anything I'm missing here?
yeap. Honest question, changing a Feature Gate to GA process is only changing the related line in and also putting a comment that this goes to GA? If so, probably this one can also be a good first issue and we can open a future issue for the removal in v1.21 once v1.20 ships. I think this will also need to be changed in the in v1.21 right? If is ok with that, I would like then to mark this as a good first issue to turn the feature GA in v1.20 :)
Well, taking a look into how secure is to move to GA: In the KEP there are some considerations about removing the IsMasterNode function, which seems to be already dealt in commit E2E tests needs to be also checked about usage of IsMasterNode(), which I could not find anymore and also seems to be removed from the same commit above. About the same Kep, it's directly related to other Feature Gate: ` So what I think that needs to be done: Promote LegacyNodeRoleBehavior to GA and Default it to false as proposed in the Kep -I'll open another issue just for this According to the LegacyNodeRoleBehavior should be removed in v1.19 but according to the Kep in v1.21, so let's stick with v1.21? ServiceNodeExclusion - Should be moved to GA, already defaulted to true. This issue is going to deal ONLY with this.


Question: // // alpha: v1.4 // MyFeature() bool <del> // owner @smarterclayton // alpha: v1.16 // beta:  v1.19 // // Enable legacy behavior to vary cluster functionality on the node-role.kubernetes.io labels. On by default (legacy), will be turned off in 1.18. // Original copy from k8s.io/kubernetes/pkg/features/kube_features.go LegacyNodeRoleBehavior featuregate.Feature = "LegacyNodeRoleBehavior" // owner @brendandburns // alpha: v1.9 // beta:  v1.19 // ga:  v1.21 // // Enable nodes to exclude themselves from service load balancers // Original copy from k8s.io/kubernetes/pkg/features/kube_features.go ServiceNodeExclusion featuregate.Feature = "ServiceNodeExclusion" </del> // owner: @khenidak // alpha: v1.15 //

NodeDisruptionExclusion - Should be moved to GA, already defaulted to true. I'll open another issue for this I'm afraid of this having some sort of direct impact in cloud providers, do you want me to take this to the sig-cloudproviders meet? Otherwise can we move the above Feature Gates to GA, and schedule their removal for v1.21, wdyt?
Per deprecation lifecycle it seems that a feature needs 6 months or 2 releases (the longer one) in Beta before going to GA. So this actually seems to be the missing part before moving to GA. As the 3 FeatureGates are directed related, probably this issue should be: Move all of them to GA in v1.21 Default LegacyNodeRoleBehavior to false Sounds a good approach? I can keep this opened until we reach v1.21
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
/remove-triage unresolved
/retitle Move Feature gate ServiceNodeExclusion to GA and LockToDefault
/triage accepted as this FG is scheduled to be moved to GA in v1.21, do you want to tackle on this?
fixed in /close
Closing this issue. <detailsIn response to : fixed in /close </details


Question: // worker runs a worker thread that just dequeues items, processes them, and marks them done. // It enforces that the syncHandler is never invoked concurrently with the same key. func (sc *ServiceController) clusterServiceWorker() { <del> fedClient := sc.federationClient </del> <ins> // process all pending events in serviceWorkerDoneChan ForLoop: for { select { case clusterName := <-sc.serviceWorkerDoneChan: sc.serviceWorkerMap[clusterName] = false default: // non-blocking, comes here if all existing events are processed break ForLoop } } </ins> for clusterName, cache := range sc.clusterCache.clientMap { <ins> workerExist, found := sc.serviceWorkerMap[clusterName] if found && workerExist { continue } // create a worker only if the previous worker has finished and gone out of scope </ins> go func(cache *clusterCache, clusterName string) { <ins> fedClient := sc.federationClient </ins> for { func() { key, quit := cache.serviceQueue.Get() <del> defer cache.serviceQueue.Done(key) </del> if quit { <ins> // send signal that current worker has finished tasks and is going out of scope sc.serviceWorkerDoneChan <- clusterName </ins> return } <ins> defer cache.serviceQueue.Done(key) </ins> err := sc.clusterCache.syncService(key.(string), clusterName, cache, sc.serviceCache, fedClient, sc) if err != nil { glog.Errorf("Failed to sync service: %+v", err)

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): No Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Environment: Cloud provider or hardware configuration: GCP OS (e.g. from /etc/os-release):Kernel (e.g. ):Install tools: Others: What happened: Memory usage of federation-control-manager keeps on increasing to very large size overtime. What you expected to happen: Memory usage is stable How to reproduce it (as minimally and precisely as possible): Register one or more cluster to federation control plane and watch the top output for the federation-control-manager Also can check the pprof, here is the one such output after federation-control-manager is run for an hour. We can see there are around goroutines created in an hour which are leaked and would consume memory. Anything else do we need to know: I do have a fix for the issue and would be happy to submit it, if this issue is accepted.
plz check the issue
feel free to just submit a pull request, we don't really have a concept of an issue being "accepted" :)
This looks like a legitimate bug. could you please clarify what version of the federated service controller you are running? The image path and version of the container, or a git hash of the source should be sufficient.
i am taking the latest code from master branch.


Question: "type": "string" }, "type": { <del> "description": "Type is the type of the condition.", </del> <ins> "description": "Type is the type of the condition. Types include Established, NamesAccepted and Terminating.", </ins> "type": "string" } },

What would you like to be : A comment in that includes the possible values of . I got the possible values from . The should contain the further explanation available in the linked code that explains what each condition type means. Why is this needed: Currently the output of is incomplete: It is important to include the values of because that's the value used to . It is not user-friendly to require users to guess the possible string values for condition or require them to read the code to deduce what the possible values are.
/sig cli
Please let me do it~
oops, I just notice your message. I already submitted a PR for this.
It's ok~


Question: dockerRuntimeName = "docker" kubeAPIVersion    = "0.1.0" <ins> // https://docs.docker.com/engine/reference/api/docker_remote_api/ // docker version should be at least 1.10.x minimumDockerAPIVersion = "1.22.0" </ins> // String used to detect docker host mode for various namespaces (e.g. // networking). Must match the value returned by docker inspect -f // '{{.HostConfig.NetworkMode}}'.

This mirrors what we do for dockertools. /cc
Is there a maintained list of docker versions supported ?
Is this the proper place to check
That was the proper place to check, but those code are being deprecated. The new one doesn't have a check yet, and that's what this issue is for.
Let me add the check.
/assign


Question: 19: "3.4.13-0", 20: "3.4.13-0", 21: "3.4.13-0", <del> 22: "3.5.0-rc.0-0", 23: "3.5.0-rc.0-0", </del> <ins> 22: "3.5.0-0", 23: "3.5.0-0", </ins> } // KubeadmCertsClusterRoleName sets the name for the ClusterRole that allows

/sig scalability This issue is tracking process of upgrading etcd to 3.5.0 planned for 1.22 K8s release. Etcd has recently released Beta versions and is starting the process of correctness and performance testing. To reduce change of finding performance regression late, we will start qualifying Beta Etcd releases on GKE master branch and will upgrade to 3.5.0 as soon as it is available. Steps: [x] Do preliminary scale testing of etcd v3.5.0-beta.3 release [x] Release etcd v3.5.0-beta.3 image (, ) [x] Release etcd v3.5.0-rc.0 image () [x] Upgrade etcd server to v3.5.0-rc.0 () [x] Wait for 3.5.0 release and upgrade etcd client () [x] Release etcd v3.5.0 image (, ) [x] Upgrade etcd server to v3.5.0 /milestone v1.22
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/area etcd
Does this meant that the 3.5.0 should be promoted like
Yes, created
/area release-eng release-team /sig release For tracking: cc:
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /area release-eng release-team /sig release cc: Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I do think this can be pointed in the Major Themes for the 1.22 release. I would like your opinions on this. If you do think it has a place in it, I will add it to the .
How can I be a contributor to this issue? what are the technical checklist that needs to be achieved to close out on the issue?


Question: 19: "3.4.13-0", 20: "3.4.13-0", 21: "3.4.13-0", <del> 22: "3.5.0-rc.0-0", 23: "3.5.0-rc.0-0", </del> <ins> 22: "3.5.0-0", 23: "3.5.0-0", </ins> } // KubeadmCertsClusterRoleName sets the name for the ClusterRole that allows

PR to update Etcd is ready, but we found a blocker for merging . There is one test failing I will take a look into it on Monday, still more eyes on the problem could speed up resolution.
We'll also need a new kube-cross image w/ etcd v3.5.0. I'm working on that here:
We just need to upgrade the server to use new image like in
ok, looks like is busy. Let me take a look /assign
Thanks.


Question: var _ = SIGDescribe("Mount propagation", func() { f := framework.NewDefaultFramework("mount-propagation") <del> ginkgo.It("should propagate mounts to the host", func() { </del> <ins> ginkgo.It("should propagate mounts within defined scopes", func() { </ins> // This test runs two pods: master and slave with respective mount // propagation on common /var/lib/kubelet/XXXX directory. Both mount a // tmpfs to a subdirectory there. We check that these mounts are

The e2e test should be relaxed. It currently checks 3 different aspects of the volume mount flag: with do not receive mounts from outside their own mount namespace, nor do mount namespaces outside the container receive mounts created within the container. with receive new mounts from the host OS, the container runtime, and other containers with , but no mount namespaces outside the container receive mounts created within the container. with receive new mounts identical to containers with , but additionally any mounts made by these containers are: propagated to the container runtime where they can be propagated to other containers (related to 2) also propagated to the Host OS The first 3 aspects, up to and including 3.i, are critical for container operation and should still be ensured by an e2e test. This last check, 3.ii., is the part that I believe is overly strict. The current e2e test is too opinionated on the underlying node OS configuration. The requirement of propagation to the host OS precludes administrators or solution providers from modifying the Kubernetes installation to hide these Kubernetes-specific mountpoints, which can be for a variety of reasons: While access to container-specific mounts in Kubernetes (like secrets) today are protected by filesystem permissions, the fact that the locations are visible to unprivileged users via the 'mount' command can provide information which could be used in conjunction with other vulnerabilities to decipher secrets or other protected container mount contents. Locking access to even listing the mountpoints behind the ability to enter the mount namespace (which in turn requires and ) removes one level of potential access to container-specific mounts. These container overlay mounts, secret and configmap mounts, and namespace mounts are all internal implementation details around the way the Kubernetes and the Container Runtime allocate resources to containers. For normal day-to-day operation, there is no need for these special-use mounts to be visible in the host OS's top-level 'mount' or 'findmnt' output. After much investigation and conversation on both the sig-node and sig-storage mailing lists, as well as within Red Hat's OpenShift communities, we have yet to turn up a use case that relies on this container-to-host propagation. In fact, it may be seen as an anti-pattern and run contrary to the container-native way of doing things. Relying on a host OS to interact with a container-owned mountpoint seems suspect.


Question: var _ = SIGDescribe("Mount propagation", func() { f := framework.NewDefaultFramework("mount-propagation") <del> ginkgo.It("should propagate mounts to the host", func() { </del> <ins> ginkgo.It("should propagate mounts within defined scopes", func() { </ins> // This test runs two pods: master and slave with respective mount // propagation on common /var/lib/kubelet/XXXX directory. Both mount a // tmpfs to a subdirectory there. We check that these mounts are

Today systemd's CPU usage is adversely affected by a large number of mountpoints, especially versions released prior to December 2020. This feature of systemd, which scans all /proc/1/mount to invoke udev events for every change suffers from two issues. The first was an inefficiency in systemd’s processing logic. This was fixed in December 2020, and this fix reportedly reduced the systemd mount-scanning-related overhead by 30-50%. The second is a lack of granularity in the events the kernel sends to systemd about mountpoint changes, which necessitates a full rescan of all mountpoints even on a minor change or single mountpoint addition. The kernel change to remedy this seems unlikely to move forward in the near term. (See for more details) For example, a fairly small OpenShift deployment today (which does not yet have the systemd December 2020 fix) has upwards of 400 mountpoints on every node owned by Kubelet, the Container Runtime, and infrastructure containers like OVN or CSI operators. Hiding the container mountpoints reduces the steady-state systemd CPU utilization from ~10% to ~0% on every worker node. On a more heavily loaded system, with 1000’s of container mountpoints, systemd’s usage will cap out at 100% of a single CPU core. Even with the 30-50% reduction in systemd usage expected from the December 2020 fix, having systemd take up as much as 50-70% of a single CPU in the worst-case scenario is a lot of unnecessary overhead. Administrators of other OSes with other host OS services that monitor or process mountpoints indiscriminately would be able to hide these container-specific mountpoints from the OS at large, at their option. With this requirement relaxed, it would be possible to run multiple instances of Kubernetes on a single host without the various mountpoints interfering with one another. There may be other inter-instance difficulties besides the mount namespace, but this does move in that direction without resorting to more complicated solutions like trying to run Kubernetes inside of containers.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available .


Question: var _ = SIGDescribe("Mount propagation", func() { f := framework.NewDefaultFramework("mount-propagation") <del> ginkgo.It("should propagate mounts to the host", func() { </del> <ins> ginkgo.It("should propagate mounts within defined scopes", func() { </ins> // This test runs two pods: master and slave with respective mount // propagation on common /var/lib/kubelet/XXXX directory. Both mount a // tmpfs to a subdirectory there. We check that these mounts are

If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig architecture storage /area conformance
I have a proof-of-concept of the relaxed test as well as some more and an OpenShift-based proof-of-concept of an actual implementation with the container mounts in their own namespace
/sig node
There's still a number of other issues, no? Is this a real motivation / use-case? There are other difficulties. Most of the difficulties with running Kubernetes inside containers are just the difficulties in sharing a kernel / host OS, which aren't related to this at all. The security and performance angles seem relevant, in any case.
Update: I mistakenly referred to this as a "conformance test" - Turns out it is not in fact a conformance test; just an e2e test :) Edited the original post to be more correct in this respect.
/remove-area conformance /area test
Fair point. I this (intentionally vague) section since some conversation on the sig-node mailing list with folks working on KIND suggested this may align with their goals in some way I'm not entirely clear on.
A good next step on this issue is to open a PR to the related e2e tests that documents why the tests are not good candidates for future inclusion in conformance. Is that something you can assist with ?
I have a branch that relaxes the test so it works in the current environment and a separated mount namespace environment... Can you explain a bit more what you mean about documenting this? Where would that kind of documentation go?


Question: assert.Error(t, err) } <ins> func TestProberSuccessAndError(t *testing.T) { // Arrange fs := utilfs.NewTempFs() watcher := newFakeWatcher() prober := &flexVolumeProber{ pluginDir: pluginDir, watcher:   watcher, fs:        fs, factory:   fakePluginFactory{errorDriver: errorDriverName}, } installDriver(driverName, fs) prober.Init() installDriver(errorDriverName, fs) driverPath := filepath.Join(pluginDir, errorDriverName) watcher.TriggerEvent(fsnotify.Create, filepath.Join(driverPath, errorDriverName)) // Act events, err := prober.Probe() // Assert assert.Equal(t, 1, len(events)) assert.Equal(t, volume.ProbeAddOrUpdate, events[0].Op) assert.Equal(t, driverName, events[0].PluginName) assert.Error(t, err) } </ins> // Installs a mock driver (an empty file) in the mock fs. func installDriver(driverName string, fs utilfs.Filesystem) { driverPath := filepath.Join(pluginDir, driverName)

When kubelet is running if you create two volume plugin directories here: /usr/libexec/kubernetes/kubelet-plugins/volume/exec /usr/libexec/kubernetes/kubelet-plugins/volume/exec/cifs /usr/libexec/kubernetes/kubelet-plugins/volume/exec/nodeagent~uds You will see these expected logs: Now if you copy over the cifs plugin file to cifs directory and then copy the uds plugin the errors go away. The pods using the uds volume will function but any pods attempting to use the cifs volume will fail with the following error from kubelet: desiredStateOfWorld. err=failed to get Plugin from volumeSpec for volume "pv" err=no volume plugin matched Restarting kubelet fixes this problem as then both plugins are present and properly . A functional volume plugin should be to kubelet even if another volume plugin is missing or invalid. Fixing the invalid volume plugin currently only adds the one that was invalid and fixed and other valid plugins that were previously present are skipped. Start kubelet service on a node. Create two directories for two flex volume plugins in the /usr/libexec/kubernetes/kubelet-plugins/volume/exec/ directory. Eg. nodeagent~uds and cifs Kubelet logs will now start complaining that the plugins are missing. Add the files to the plugin folders and then start a pod using the plugin. Notice that only the last plugin file copied is available to be used by pods. The first plugin will generate this error if used by a volume "err=no volume plugin matched" Performing a touch on the folder of the missing plugin will cause kubelet to properly index it. <details</details<detailsLocal kube cluster </details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig k8s-infra
/sig storage /remove-sig k8s-infra


Question: w.Flush() } <ins> printWithNamespace := allNamespaces if mapping != nil && mapping.Scope.Name() == meta.RESTScopeNameRoot { printWithNamespace = false } </ins> var outputOpts *printers.OutputOptions // if cmd does not specify output format and useOpenAPIPrintColumnFlagLabel flag is true, // then get the default output options for this mapping from OpenAPI schema.

This is wrong, cluster scoped resources should never have a NAMESPACE column. We need to fix and add a test to ensure this doesn't regress.
The presence of the flag is what triggers this.
I would help fix it.
I mean to add a test after modification, but found that current test framework could not support the case: To cover the scene, we need a real NewHumanReadablePrinter in test case, to ensure the output format, and modify the related funcs in FakeFactory to support pass in param withNamespace. I feel this not like behavior of unit test to some extent. The loss may outweighs the gain to have such a big impact on the whole. What's your opinion
[MILESTONENOTIFIER] Milestone Removed Important: This issue was missing the label for more than 7 days. <details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details


Question: <ins> /* Copyright 2018 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package storage import ( "fmt" . "github.com/onsi/ginkgo" . "github.com/onsi/gomega" "k8s.io/api/core/v1" storagev1 "k8s.io/api/storage/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" utilerrors "k8s.io/apimachinery/pkg/util/errors" clientset "k8s.io/client-go/kubernetes" "k8s.io/kubernetes/test/e2e/framework" "k8s.io/kubernetes/test/e2e/storage/utils" ) const ( noProvisioner = "kubernetes.io/no-provisioner" pvNamePrefix  = "pv" ) func generateConfigsForStaticProvisionPVTest(scName string, volBindMode storagev1.VolumeBindingMode, volMode v1.PersistentVolumeMode, pvSource v1.PersistentVolumeSource) (*storagev1.StorageClass, framework.PersistentVolumeConfig, framework.PersistentVolumeClaimConfig) { // StorageClass scConfig := &storagev1.StorageClass{ ObjectMeta: metav1.ObjectMeta{ Name: scName, }, Provisioner:       noProvisioner, VolumeBindingMode: &volBindMode, } // PV pvConfig := framework.PersistentVolumeConfig{ PVSource:         pvSource, NamePrefix:       pvNamePrefix, StorageClassName: scName, VolumeMode:       &volMode, } // PVC pvcConfig := framework.PersistentVolumeClaimConfig{ AccessModes:      []v1.PersistentVolumeAccessMode{v1.ReadWriteOnce}, StorageClassName: &scName, VolumeMode:       &volMode, } return scConfig, pvConfig, pvcConfig } func createPVTestResource(cs clientset.Interface, ns string, scConfig *storagev1.StorageClass, pvConfig framework.PersistentVolumeConfig, pvcConfig framework.PersistentVolumeClaimConfig) (*storagev1.StorageClass, *v1.Pod, *v1.PersistentVolume, *v1.PersistentVolumeClaim) { By("Creating sc") sc, err := cs.StorageV1().StorageClasses().Create(scConfig) Expect(err).NotTo(HaveOccurred()) By("Creating pv and pvc") pv, pvc, err := framework.CreatePVPVC(cs, pvConfig, pvcConfig, ns, false) framework.ExpectNoError(err) framework.ExpectNoError(framework.WaitOnPVandPVC(cs, ns, pv, pvc)) By("Creating a pod") // TODO(mkimuram): Need to set anti-affinity with storage server pod. // Otherwise, storage server pod can also be affected on destructive tests. pod, err := framework.CreateSecPod(cs, ns, []*v1.PersistentVolumeClaim{pvc}, false, "", false, false, framework.SELinuxLabel, nil, framework.PodStartTimeout) Expect(err).NotTo(HaveOccurred()) return sc, pod, pv, pvc } func createPVTestResourceWithFailure(cs clientset.Interface, ns string, scConfig *storagev1.StorageClass, pvConfig framework.PersistentVolumeConfig, pvcConfig framework.PersistentVolumeClaimConfig) (*storagev1.StorageClass, *v1.Pod, *v1.PersistentVolume, *v1.PersistentVolumeClaim) { By("Creating sc") sc, err := cs.StorageV1().StorageClasses().Create(scConfig) Expect(err).NotTo(HaveOccurred()) By("Creating pv and pvc") pv, pvc, err := framework.CreatePVPVC(cs, pvConfig, pvcConfig, ns, false) framework.ExpectNoError(err) framework.ExpectNoError(framework.WaitOnPVandPVC(cs, ns, pv, pvc)) By("Creating a pod") pod, err := framework.CreateSecPod(cs, ns, []*v1.PersistentVolumeClaim{pvc}, false, "", false, false, framework.SELinuxLabel, nil, framework.PodStartTimeout) Expect(err).To(HaveOccurred()) return sc, pod, pv, pvc } func deletePVTestResource(f *framework.Framework, cs clientset.Interface, ns string, sc *storagev1.StorageClass, pod *v1.Pod, pv *v1.PersistentVolume, pvc *v1.PersistentVolumeClaim) { By("Deleting pod") framework.ExpectNoError(framework.DeletePodWithWait(f, cs, pod)) By("Deleting pv and pvc") errs := framework.PVPVCCleanup(cs, ns, pv, pvc) if len(errs) > 0 { framework.Failf("Failed to delete PV and/or PVC: %v", utilerrors.NewAggregate(errs)) } By("Deleting sc") framework.ExpectNoError(cs.StorageV1().StorageClasses().Delete(sc.Name, nil)) } func checkVolumeModeOfPath(pod *v1.Pod, volMode v1.PersistentVolumeMode, path string) { if volMode == v1.PersistentVolumeBlock { // Check if block exists utils.VerifyExecInPodSucceed(pod, fmt.Sprintf("test -b %s", path)) // Double check that it's not directory utils.VerifyExecInPodFail(pod, fmt.Sprintf("test -d %s", path), 1) } else { // Check if directory exists utils.VerifyExecInPodSucceed(pod, fmt.Sprintf("test -d %s", path)) // Double check that it's not block utils.VerifyExecInPodFail(pod, fmt.Sprintf("test -b %s", path), 1) } } func checkReadWriteToPath(pod *v1.Pod, volMode v1.PersistentVolumeMode, path string) { if volMode == v1.PersistentVolumeBlock { // random -> file1 utils.VerifyExecInPodSucceed(pod, "dd if=/dev/urandom of=/tmp/file1 bs=64 count=1") // file1 -> dev (write to dev) utils.VerifyExecInPodSucceed(pod, fmt.Sprintf("dd if=/tmp/file1 of=%s bs=64 count=1", path)) // dev -> file2 (read from dev) utils.VerifyExecInPodSucceed(pod, fmt.Sprintf("dd if=%s of=/tmp/file2 bs=64 count=1", path)) // file1 == file2 (check contents) utils.VerifyExecInPodSucceed(pod, "diff /tmp/file1 /tmp/file2") // Clean up temp files utils.VerifyExecInPodSucceed(pod, "rm -f /tmp/file1 /tmp/file2") // Check that writing file to block volume fails utils.VerifyExecInPodFail(pod, fmt.Sprintf("echo 'Hello world.' > %s/file1.txt", path), 1) } else { // text -> file1 (write to file) utils.VerifyExecInPodSucceed(pod, fmt.Sprintf("echo 'Hello world.' > %s/file1.txt", path)) // grep file1 (read from file and check contents) utils.VerifyExecInPodSucceed(pod, fmt.Sprintf("grep 'Hello world.' %s/file1.txt", path)) // Check that writing to directory as block volume fails utils.VerifyExecInPodFail(pod, fmt.Sprintf("dd if=/dev/urandom of=%s bs=64 count=1", path), 1) } } func skipBlockSupportTestIfUnsupported(volMode v1.PersistentVolumeMode, isBlockSupported bool) { if volMode == v1.PersistentVolumeBlock && !isBlockSupported { framework.Skipf("Skip assertion for block test for block supported plugin.(Block unsupported)") } } func skipBlockUnsupportTestUnlessUnspported(volMode v1.PersistentVolumeMode, isBlockSupported bool) { if !(volMode == v1.PersistentVolumeBlock && !isBlockSupported) { framework.Skipf("Skip assertion for block test for block unsupported plugin.(Block suppported or FileSystem test)") } } var _ = utils.SIGDescribe("PersistentVolumes-volumeMode", func() { f := framework.NewDefaultFramework("pv-volmode") const ( pvTestSCPrefix = "pvtest" ) var ( cs               clientset.Interface ns               string scName           string isBlockSupported bool serverIP         string secret           *v1.Secret serverPod        *v1.Pod pvSource         v1.PersistentVolumeSource sc               *storagev1.StorageClass pod              *v1.Pod pv               *v1.PersistentVolume pvc              *v1.PersistentVolumeClaim volMode          v1.PersistentVolumeMode volBindMode      storagev1.VolumeBindingMode ) BeforeEach(func() { cs = f.ClientSet ns = f.Namespace.Name volBindMode = storagev1.VolumeBindingImmediate }) AssertCreateDeletePodAndReadWriteVolume := func() { // For block supported plugins It("should create sc, pod, pv, and pvc, read/write to the pv, and delete all created resources", func() { skipBlockSupportTestIfUnsupported(volMode, isBlockSupported) scConfig, pvConfig, pvcConfig := generateConfigsForStaticProvisionPVTest(scName, volBindMode, volMode, pvSource) sc, pod, pv, pvc = createPVTestResource(cs, ns, scConfig, pvConfig, pvcConfig) defer deletePVTestResource(f, cs, ns, sc, pod, pv, pvc) By("Checking if persistent volume exists as expected volume mode") checkVolumeModeOfPath(pod, volMode, "/mnt/volume1") By("Checking if read/write to persistent volume works properly") checkReadWriteToPath(pod, volMode, "/mnt/volume1") }) // For block unsupported plugins It("should fail to create pod by failing to mount volume", func() { skipBlockUnsupportTestUnlessUnspported(volMode, isBlockSupported) scConfig, pvConfig, pvcConfig := generateConfigsForStaticProvisionPVTest(scName, volBindMode, volMode, pvSource) sc, pod, pv, pvc = createPVTestResourceWithFailure(cs, ns, scConfig, pvConfig, pvcConfig) deletePVTestResource(f, cs, ns, sc, pod, pv, pvc) }) } verifyAll := func() { AssertCreateDeletePodAndReadWriteVolume() // TODO(mkimuram): Add more tests } Describe("NFS", func() { const pvTestNFSSCSuffix = "nfs" BeforeEach(func() { isBlockSupported = false scName = fmt.Sprintf("%v-%v-%v", pvTestSCPrefix, ns, pvTestNFSSCSuffix) _, serverPod, serverIP = framework.NewNFSServer(cs, ns, []string{}) pvSource = v1.PersistentVolumeSource{ NFS: &v1.NFSVolumeSource{ Server:   serverIP, Path:     "/", ReadOnly: false, }, } }) AfterEach(func() { framework.Logf("AfterEach: deleting NFS server pod %q...", serverPod.Name) err := framework.DeletePodWithWait(f, cs, serverPod) Expect(err).NotTo(HaveOccurred(), "AfterEach: NFS server pod failed to delete") }) Context("FileSystem volume Test", func() { BeforeEach(func() { volMode = v1.PersistentVolumeFilesystem }) verifyAll() }) Context("Block volume Test[Feature:BlockVolume]", func() { BeforeEach(func() { volMode = v1.PersistentVolumeBlock }) verifyAll() }) }) Describe("iSCSI [Feature:Volumes]", func() { const pvTestISCSISCSuffix = "iscsi" BeforeEach(func() { isBlockSupported = true scName = fmt.Sprintf("%v-%v-%v", pvTestSCPrefix, ns, pvTestISCSISCSuffix) _, serverPod, serverIP = framework.NewISCSIServer(cs, ns) pvSource = v1.PersistentVolumeSource{ ISCSI: &v1.ISCSIPersistentVolumeSource{ TargetPortal: serverIP + ":3260", IQN:          "iqn.2003-01.org.linux-iscsi.f21.x8664:sn.4b0aae584f7c", Lun:          0, }, } }) AfterEach(func() { framework.Logf("AfterEach: deleting iSCSI server pod %q...", serverPod.Name) err := framework.DeletePodWithWait(f, cs, serverPod) Expect(err).NotTo(HaveOccurred(), "AfterEach: iSCSI server pod failed to delete") }) Context("FileSystem volume Test", func() { BeforeEach(func() { volMode = v1.PersistentVolumeFilesystem }) verifyAll() }) Context("Block volume Test[Feature:BlockVolume]", func() { BeforeEach(func() { volMode = v1.PersistentVolumeBlock }) verifyAll() }) }) Describe("Ceph-RBD [Feature:Volumes]", func() { const pvTestRBDSCSuffix = "rbd" BeforeEach(func() { isBlockSupported = true scName = fmt.Sprintf("%v-%v-%v", pvTestSCPrefix, ns, pvTestRBDSCSuffix) _, serverPod, secret, serverIP = framework.NewRBDServer(cs, ns) framework.Logf("namespace: %v, secret.Name: %v", ns, secret.Name) pvSource = v1.PersistentVolumeSource{ RBD: &v1.RBDPersistentVolumeSource{ CephMonitors: []string{serverIP}, RBDPool:      "rbd", RBDImage:     "foo", RadosUser:    "admin", SecretRef: &v1.SecretReference{ Name:      secret.Name, Namespace: ns, }, ReadOnly: false, }, } }) AfterEach(func() { framework.Logf("AfterEach: deleting Ceph-RDB server secret %q...", secret.Name) secErr := cs.CoreV1().Secrets(ns).Delete(secret.Name, &metav1.DeleteOptions{}) framework.Logf("AfterEach: deleting Ceph-RDB server pod %q...", serverPod.Name) err := framework.DeletePodWithWait(f, cs, serverPod) if secErr != nil || err != nil { if secErr != nil { framework.Logf("AfterEach: Ceph-RDB delete secret failed: %v", secErr) } if err != nil { framework.Logf("AfterEach: Ceph-RDB server pod delete failed: %v", err) } framework.Failf("AfterEach: cleanup failed") } }) Context("FileSystem volume Test", func() { BeforeEach(func() { volMode = v1.PersistentVolumeFilesystem }) verifyAll() }) Context("Block volume Test[Feature:BlockVolume]", func() { BeforeEach(func() { volMode = v1.PersistentVolumeBlock }) verifyAll() }) }) Describe("CephFS [Feature:Volumes]", func() { const pvTestCephFSSCSuffix = "cephfs" BeforeEach(func() { isBlockSupported = false scName = fmt.Sprintf("%v-%v-%v", pvTestSCPrefix, ns, pvTestCephFSSCSuffix) _, serverPod, secret, serverIP = framework.NewRBDServer(cs, ns) pvSource = v1.PersistentVolumeSource{ CephFS: &v1.CephFSPersistentVolumeSource{ Monitors: []string{serverIP + ":6789"}, User:     "kube", SecretRef: &v1.SecretReference{ Name:      secret.Name, Namespace: ns, }, ReadOnly: false, }, } }) AfterEach(func() { framework.Logf("AfterEach: deleting CephFS server secret %q...", secret.Name) secErr := cs.CoreV1().Secrets(ns).Delete(secret.Name, &metav1.DeleteOptions{}) framework.Logf("AfterEach: deleting CephFS server pod %q...", serverPod.Name) err := framework.DeletePodWithWait(f, cs, serverPod) if secErr != nil || err != nil { if secErr != nil { framework.Logf("AfterEach: CephFS delete secret failed: %v", secErr) } if err != nil { framework.Logf("AfterEach: CephFS server pod delete failed: %v", err) } framework.Failf("AfterEach: cleanup failed") } }) Context("FileSystem volume Test", func() { BeforeEach(func() { volMode = v1.PersistentVolumeFilesystem }) verifyAll() }) Context("Block volume Test[Feature:BlockVolume]", func() { BeforeEach(func() { volMode = v1.PersistentVolumeBlock }) verifyAll() }) }) }) </ins>

It would better to have e2e for iSCSI persistent volume. /sig storage
/assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close


Question: return pod, podStats } <ins> func makePodWithPIDStats(name string, priority int32, processCount uint64) (*v1.Pod, statsapi.PodStats) { pod := newPod(name, priority, []v1.Container{ newContainer(name, nil, nil), }, nil) podStats := newPodProcessStats(pod, processCount) return pod, podStats } </ins> func makePodWithDiskStats(name string, priority int32, requests v1.ResourceList, limits v1.ResourceList, rootFsUsed, logsUsed, perLocalVolumeUsed string) (*v1.Pod, statsapi.PodStats) { pod := newPod(name, priority, []v1.Container{ newContainer(name, requests, limits),

We should add a unit test to Eviction manager that accounts for PID Pressure. The test should cover similar cases as DiskPressure or MemoryPressure. 1) PIDPressure doesn't exist on node. 2) Force PidPressure on node We should include test cases for reclaming resources in terms of pressure and verifying differences between soft and hard eviction.
/sig node /kind feature /help
This request has been marked as needing help from a contributor. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /sig node /kind feature /help </details
/triage accepted /priority important-longterm
/good-first-issue
This request has been marked as suitable for new contributors. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
it looks like has requested to work on this.
/assign
Hi What do you mean by, "verifying differences between soft and hard eviction."?
It’s the same idea as what we do for disk, memory or inode pressure.


Question: return pod, podStats } <ins> func makePodWithPIDStats(name string, priority int32, processCount uint64) (*v1.Pod, statsapi.PodStats) { pod := newPod(name, priority, []v1.Container{ newContainer(name, nil, nil), }, nil) podStats := newPodProcessStats(pod, processCount) return pod, podStats } </ins> func makePodWithDiskStats(name string, priority int32, requests v1.ResourceList, limits v1.ResourceList, rootFsUsed, logsUsed, perLocalVolumeUsed string) (*v1.Pod, statsapi.PodStats) { pod := newPod(name, priority, []v1.Container{ newContainer(name, requests, limits),

Hard eviction means we trigger eviction with no grace period while soft eviction means that we obey a grace period. Test should check for that.
I have opened a PR based on what I could understand using a debugger and the other tests. This is my first time contributing to the codebase. Happy to take brutal feedback.
Hi, i would like to support this issue. Let me know if any help is required.
/assign


Question: o.authProviderArgsToRemove = removePairs } <ins> execArgs, err := cmd.Flags().GetStringSlice(flagExecArg) if err != nil { return fmt.Errorf("Error: %s", err) } if len(execArgs) > 0 { o.execArgs = execArgs } execEnv, err := cmd.Flags().GetStringArray(flagExecEnv) if err != nil { return fmt.Errorf("Error: %s", err) } if len(execEnv) > 0 { newPairs, removePairs, err := cmdutil.ParsePairs(execEnv, flagExecEnv, true) if err != nil { return fmt.Errorf("Error: %s", err) } o.execEnv = newPairs o.execEnvToRemove = removePairs } </ins> o.name = args[0] return nil }

Now it's possible to configure third-party auth providers using command with option. It's nice to have the same possibility for the recently . Example: To specify the executable path we can run This will generate a new config for the user: Various auxiliary configuration options are also could be . For instance: to define an argument for the executable; to set an environment variable; to specify a desired API version. /kind feature
I'm going to start working on this when the API is promoted to beta.
/sig auth
/assign
Can someone share why you can't do : Is there a way to set map values (e.g. something that is -name: value ; versus something that is -context: )
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
I would like to work on it, if this is still persist.
cc
This still seems useful and is outstanding. any objections to taking it over?
No, it's okay, I'm working on it. I'll post the code this week, sorry for the delay
I think this would help out with generating valid kubectl configurations for our EKS developers. I'm using Pulumi to build EKS environments and if all the flags are available, I think I can generate the necessary entries to assume a role. Thanks! Ex: name: k8s.developer user: exec: apiVersion: args: token -i cluster-eksCluster- -r arn:aws:iam:::role/k8sDevAdminRole- command: aws-iam-authenticator env: null
done :)
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
/priority important-longterm


Question: "sync/atomic" "time" <del> "k8s.io/api/core/v1" </del> <ins> v1 "k8s.io/api/core/v1" </ins> discovery "k8s.io/api/discovery/v1alpha1" "k8s.io/apimachinery/pkg/types" "k8s.io/apimachinery/pkg/util/wait"

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Using Kubernetes 1.16.1, I started kube-proxy in iptables mode as a static pod on a master node. The master node was intentionally (and permanently) not available in the API as a Node object. kube-proxy started up and terminated soon after because it failed to (which is supposed to function as the bound address) and eventually returned an error: What you expected to happen: I expected the same behavior as in Kubernetes 1.15 where failure to look up the Node in the API would result in . How to reproduce it (as minimally and precisely as possible): Configure kube-proxy with iptables mode on Kubernetes 1.16 without the hosting Node being available in the Kubernetes API (under the OS hostname). Anything else we need to know?: PR (Fix for kube-proxy to wait for some duration for the node to be defined) introduced an additional . I believe this is the part that is causing the change in behavior: in Kubernetes 1.15, the check would be omitted and cause to set the default bind address to 127.0.0.1 (when iptables mode is set; I haven't checked other modes). The PR's primary purpose seems to be a different one (retrying node IP retrieval from the API), so my understanding is that the check was not really needed; unless I'm misunderstanding things which I don't want to rule out at all. The quick workaround was to explicitly configure kube-proxy with . However, I'm not entirely sure if that's a good solution long-term compared to letting Kubernetes do The Right Thing, which is what I'd like to clarify through this issue. Environment: Kubernetes version (use ): v1.16.1 Cloud provider or hardware configuration: Kubernetes running on DigitalOcean VMs (droplets) OS (e.g: ): Debian GNU/Linux 9 (stretch) Kernel (e.g.


Question: "sync/atomic" "time" <del> "k8s.io/api/core/v1" </del> <ins> v1 "k8s.io/api/core/v1" </ins> discovery "k8s.io/api/discovery/v1alpha1" "k8s.io/apimachinery/pkg/types" "k8s.io/apimachinery/pkg/util/wait"

): Linux master-k8s-1-16-test1 4.19.0-0.bpo.6-amd64 SMP Debian 4.19.67-2+deb10u1~bpo9+1 (2019-09-30) x86_64 GNU/Linux /sig network /cc who were involved in the referenced PR.
I made a similar comment in , wondering if this is a breaking change in v1.16 that we should revert? Previously if is nil, kube-proxy defaults to but now kube-proxy always exits with error instead of defaulting to local host. I think a lot of external cloud providers relied on this behavior because they needed to talk to API server prior to node addresses being set by the provider. Thoughts /assign
Opened to preserve behavior before v1.16 where node IP defaults to 127.0.01 is none is found. PTAL


Question: imageParts := strings.Split(image, "/") return imageParts[0] } <ins> // mergedEnvVars overlays system defined env vars with credential provider env vars, // it gives priority to the credential provider vars allowing user to override system // env vars func mergeEnvVars(sysEnvVars, credProviderVars []string) []string { mergedEnvVars := sysEnvVars for _, credProviderVar := range credProviderVars { mergedEnvVars = append(mergedEnvVars, credProviderVar) } return mergedEnvVars } </ins>

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --I am attempting to pull an image from ECR using the new kubelet image credential provider feature which is an alpha features in Kubernetes v1.20. My CredentialProviderConfig file includes the environment variable AWSPROFILE which resides in the default () directory. The profile references a which allows me to execute a binary that returns a set of temporary AWS credentials. When I try running a pod with a container from ECR, I see the following errors in the kubelet logs: However, when I type the following into the command line on the kubelet: I get the following response: It doesn't appear the environment variables in my Config are being passed to the Kubelet when it executes the ecr-credential-provider binary. Below is Config file: I expect for the Kubelet to pass the environment variables in the Config file when it executes the ecr-credential-provider binary. I am using KinD to test this feature. Here is my config: No Kubernetes version (use ): 1.20 Cloud provider or hardware configuration: None OS (e.g: ): Ubuntu 20.10 Kernel (e.g. ): 5.10 Install tools: KinD Network plugin and version (if this is a network-related bug): Others: cc
/sig Node
/assign
from the code it appears that env variables are passed to the plugin from kubelet logs the plugin is only returning exit status 1, without any details about the error!!
Thanks Yes, I saw that. doesn't work. I wrote the following code to test this: If I uncomment , which imports the environment variables from the OS, it works.
Hello need some clarification. Is the intent of the above code to convey that isn't passing env vars correctly to the spawned process? Below code works as expected for me Also, could you tell if is a proprietary binary or a open source code something similar to because I want to use it in troubleshooting this issue.
Yes it is not passing the Envs to the spawned process.


Question: imageParts := strings.Split(image, "/") return imageParts[0] } <ins> // mergedEnvVars overlays system defined env vars with credential provider env vars, // it gives priority to the credential provider vars allowing user to override system // env vars func mergeEnvVars(sysEnvVars, credProviderVars []string) []string { mergedEnvVars := sysEnvVars for _, credProviderVar := range credProviderVars { mergedEnvVars = append(mergedEnvVars, credProviderVar) } return mergedEnvVars } </ins>

The code for ecr-credential-provider is in the AWS cloud controller repository,
Quick update, I had setup two test beds to reproduce the above issue, but both of them gave expected results and the issue was not reproducible. Below are my test setups: ECR Repository Test Bed 1 Execute from code and pass as environment variable Code Observed Result - Below credentials were generated Expected Result - Credentials should be generated Test Bed 2 Use as the credential provider within the kubelet, I had setup a Provider Config Expected result - Pods should be created based off custom image Observed result - Pods were created based off custom image Though, my test bed runs on kubernetes version , I could see that nothing significant has changed in since version Could you give me any other minor but important detail which you feel you could had missed out earlier which could help me in replicating the issue? Or could you double check if provider exists in your file?
i am using credential_process, in my profile. I am also using KinD for testing, but I don't think that should matter. I don't have a credentials file. I only have a config file.
This is the binary I'm calling with credential_process,
can you check kubelet logs for , It might be possible that is falling back to in-tree aws credential provider.
on my local setup credentials are being fetched. I'll setup another test bed with the Cognito provider which is using.
It is fetching credentials because it is falling back to other options available, It might be fetching from other credential provider if failed from external one. To confirm this you have to check kubelet logs.
In my env I just have one provider and there are no fallbacks. Though I was keeping an eye on kubelet logs while performing this activity and I saw no such logs. The creds were fetched in one shot via the profile which I had specified. And as I had mentioned earlier I don't see any issues in passing env vars to credential process. Also, I did try negative cases as well by passing invalid profile names and it did fail and pods weren't pulled from ECR
kubelet registers intree credential providers while startup, you need not to explicitly add any!! And I recently tried GCP credential provider, It does fall back to intree GCP credential provider after getting failed response from the one i set up.
Sure, makes sense.


Question: imageParts := strings.Split(image, "/") return imageParts[0] } <ins> // mergedEnvVars overlays system defined env vars with credential provider env vars, // it gives priority to the credential provider vars allowing user to override system // env vars func mergeEnvVars(sysEnvVars, credProviderVars []string) []string { mergedEnvVars := sysEnvVars for _, credProviderVar := range credProviderVars { mergedEnvVars = append(mergedEnvVars, credProviderVar) } return mergedEnvVars } </ins>

Though as I mentioned earlier I was keeping an eye on the kubelet logs and I didn't see the logs you had mentioned
i see in the logs when the kubelet starts, before attempting to run a pod that pulls an image from ECR.
Had a light bulb moment and I was able to reproduce this issue, it might not be applicable in your case but it could be worth trying out. In my local setup I ran the cluster as root and failed with error: E0614 18:19:27. ] Failed getting credential from external registry credential provider: error execing credential provider plugin ecr-credential-provider for image exit status 1 I have my aws credentials configure under my user () at path and since there is no similar configuration for the process failed. Since, I have access to code locally, I couple of log statements to print out current env vars and I could see that env var was set to :) Would it be possible for you to check is running as what user and is there any dissonance which can prohibit the user from successfully retrieving the credentials?
I am getting the error you mentioned, So far as I can tell, the kubelet is running as root and the file is in the directory. Again, I'm using KinD for testing. I'm not sure how KinD runs the kubelet. Below is the output from : The Go code I wrote to test this fails when using cmd.Env; it's only when I import the OS environment variables that it works. This may because I am using in my shared config file. credentialprocess is a mechanism to call another binary, e.g. , to get a set of temporary credentials from Cognito.
In order to simplify the sprawl and to pinpoint the offending component,could you remove the credential_process and Cognito and just try fetching creds via plain AWS credentials profile?
sure, however, supporting credential process is critical for my use case. I'm using Cognito as a gating mechanism to ECR. Only licensed users will be allowed to pull images from ECR. These users will not have IAM credentials (access key/secret key). They will authenticate to AWS using Cognito.
The following configuration works when I specify access key/secret key:
When I switch to a profile with credential process I get the following error:
isn't it the issue of ecr-credential-provider?


Question: imageParts := strings.Split(image, "/") return imageParts[0] } <ins> // mergedEnvVars overlays system defined env vars with credential provider env vars, // it gives priority to the credential provider vars allowing user to override system // env vars func mergeEnvVars(sysEnvVars, credProviderVars []string) []string { mergedEnvVars := sysEnvVars for _, credProviderVar := range credProviderVars { mergedEnvVars = append(mergedEnvVars, credProviderVar) } return mergedEnvVars } </ins>

This is the job of ecr-credential-provider to fetch credentials and return result in expected format to kubelet.
why does it work when I run it from the command line? Why does it work when I import envs from the OS?
you might have saved env on command line, same goes for OS.
That doesn't make sense to me. If I run it returns a properly formatted response. Same goes for the Go code I wrote. Does calling another binary, like credentialprocess, confuse the kubelet?
I am not familiar with aws credential process. Kubelet talks to with exec based plugin mechanism, is ecr-credential-provider calling the credential process you are talking about?
I have already validated that env vars are being passed to and there is no issue over there. If that wouldn't be the case then AWS credentials generation process via wouldn't work for you. I had debugged the code of kubelet myself and validated it + I wrote some logs in to print out the env vars and it was logging the passed vars. IMHO, now what remains to be seen here is why it fails with Cognito. I am going to create a test bed tomorrow replicating the exact setup which you have in the mean time if it's possible for you could you create a Linux VM and replicate your setup inside it? My intention is to see whether the Cognito based credential process goes through or not in a clean slate environment if it does then something on your local setup is interfering with passed env vars Thoughts?
Here's the execution flow ---- Does the above paints a fairly accurate picture? If not then feel free to correct :)
I included my KinD configuration in the beginning of this thread. Can you use that The binary has to be built from source, (or I can send it to you along with a temp username/password). There is an accompanying blog post for the binary at The flow is as describes above. The binary returns a json blob to the calling application that looks like this: During my testing (without the kubelet) I am able to get the proper response for the kubelet when referencing the AWS profile using or by setting to . However, when I try to pass the AWS profile in using exec.Env("AWS_PROFILE=default") I get and no further information. I have a feeling that doesn't properly chain or pass information back to the calling application.


Question: imageParts := strings.Split(image, "/") return imageParts[0] } <ins> // mergedEnvVars overlays system defined env vars with credential provider env vars, // it gives priority to the credential provider vars allowing user to override system // env vars func mergeEnvVars(sysEnvVars, credProviderVars []string) []string { mergedEnvVars := sysEnvVars for _, credProviderVar := range credProviderVars { mergedEnvVars = append(mergedEnvVars, credProviderVar) } return mergedEnvVars } </ins>

Let me replicate your setup in my local env and see if I can reproduce this issue. Also, could you tell what your AWS file look like? Is it something like this You can use placeholder values for sensitive information but correct the binary path if it's not
yes, that looks right. Ideally the cognito binary is located in a directory in the shell's PATH but it also works when you specify the full path in the config file like you've done. Are you planning to setup cognito in AWS? You'll need a cognito user pool and configure federation.
Yes, I am planning to setup Cognito Identity pool in my AWS Account.
So I replicated your setup on my local machine with slight change, instead of using AWS Cognito for generating credentials via I am calling a binary written by me which logs env vars to a specific path and prints out credentials in the format as prescribed by AWS. Below was my initial config file After I ran local cluster with this setup I noticed couple of log lines like below: I did some basic validation of perms + I even moved the binary to couple of well know locations but nothing worked Finally I modified my vars section of my deployment spec like below and it worked The logs which I had put in my custom credential provider printed following env vars validating the fact that, env vars are being passed In your logs do you see a log line something like above? If not then could you pass full contents of from start up until you create a Pod?
thanks for doing this. I never tried adding the PATH to the list of environment variables. I honestly didn't think it was necessary because I was referencing the full path to the binary in credential_process. I also never saw an error about not being able to find the executable in $PATH. I'll re-test after adding PATH as an ENV and let you know the results.
Adding the PATH did not work for me. I'm happy to send you the cognito binary with a username/password that you can use for testing. I have only tested this with KinD, but I don't think that is the issue here since it fails with exit code when when I try using .
^^ is my credential provider config
Thanks for the logs. I have cloned your code in my repo and I'll substitute my Cognito creds.


Question: imageParts := strings.Split(image, "/") return imageParts[0] } <ins> // mergedEnvVars overlays system defined env vars with credential provider env vars, // it gives priority to the credential provider vars allowing user to override system // env vars func mergeEnvVars(sysEnvVars, credProviderVars []string) []string { mergedEnvVars := sysEnvVars for _, credProviderVar := range credProviderVars { mergedEnvVars = append(mergedEnvVars, credProviderVar) } return mergedEnvVars } </ins>

If it doesn't work out then I'll reach out to you for creds
After several rounds of troubleshooting I was able to get working in my KinD environment. My credential loading setup is <-<-and below is my configuration of credential process Initially, I got the same error as yours, then I couple of custom log lines to print env vars in in directory and I could verify that the env vars specified in the credentials spec were indeed being passed. What I noticed in my KinD setup is that AWS SDK was logging an error because it couldn't find any "providers", though I could verify in the container that file and were indeed present. I altered my env vars section of the as below and it worked for me Apparently, what I noticed is is being passed ONLY the env vars which are present in the spec and everything else was missing! Or at least this is the behaviour in my local cluster setup + KinD cluster. Can you please add and other env vars necessary for the correct execution of AWS SDK and see if it works for you? If this doesn't work, then I'll send you the modified version of the which puts some additional logs to troubleshoot this issue further.
That did it! Thanks for helping me troubleshoot this. I wonder if this is unique to KinD or whether we would see this behavior in a "normal" cluster (i'll test it later). I will tell the developers who wrote the ecr-credential-provider to include an argument to increase the verbosity or to output logs to a file so this is easier to troubleshoot in the future.
What I have observed is, is only passed env vars which are in credential spec and the spec clearly says that the vars in spec are appended to existing env vars and this looks like an issue which needs deeper investigation. Could you please re-open this issue?
sure thing.
The label cannot be applied. Only GitHub organization members can add the label. <detailsIn response to : /triage accepted Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
To me this looks like an issue with the kubelet and I have observed this behaviour with as well as my local cluster. What are your thoughts on this? If this looks like a bug then can you accept the triage? I would fix this.


Question: imageParts := strings.Split(image, "/") return imageParts[0] } <ins> // mergedEnvVars overlays system defined env vars with credential provider env vars, // it gives priority to the credential provider vars allowing user to override system // env vars func mergeEnvVars(sysEnvVars, credProviderVars []string) []string { mergedEnvVars := sysEnvVars for _, credProviderVar := range credProviderVars { mergedEnvVars = append(mergedEnvVars, credProviderVar) } return mergedEnvVars } </ins>

Thanks for looking into the issue, you are right the spec does says that. Feel free to fix it /triage accepted
Thanks!!!
/area kubelet
: I am facing the similar issue on my bare metal cluster This is my config Can you help. It works when I try to execute the credentials binary from local but doesn't work in the pod. Can you pls guide me
I've been fighting this for days - did you ever figure it out?


Question: defer f.Unlock() f.CalledFunctions = append(f.CalledFunctions, "ListImages") <del> return f.ImageList, f.Err </del> <ins> return snapshot(f.ImageList), f.Err } func snapshot(imageList []kubecontainer.Image) []kubecontainer.Image { result := make([]kubecontainer.Image, len(imageList)) copy(result, imageList) return result </ins> } func (f *FakeRuntime) RemoveImage(_ context.Context, image kubecontainer.ImageSpec) error {

when kubelet restart,all images which notused cleaned. but there is a flag in kubelet,,explainMinimum age for an unused image before it is garbage collected. Examples: '300ms', '10s' or '2h45m'. i set to ,it's means if a image not used,it will live 20m and then clean by kubeletif has disk pressure.if kubelet not restart, it work,but if restart ther is a bug, not work,image will be cleaneven if it's no enough time. work, image even if not used,it will live 20m and then clean by kubelet. 1make node trigger disk pressure. 2pull some image which not used by pod 3restart kubelet 4image cleaned by kubelet after around 10s i see the source code, and maybe there is a bug. in func,well detect image and update firstdetect time for every image. and in func,well check images livetime by firstDetected. if kubelet running normal,it work,because there is a func. in this line,,in first time,ts is zero. so image's firstDetected time is zero,and in freeSpace this check is always false,so image will be cleaned. maybe we can change to ,can fix this bug. <details</details<detailsNone </details<details</details<details</details<details</details<details</details
/sig node
/assign
/triage accepted /priority backlog initial assessment of priority, subject to change


Question: "//staging/src/k8s.io/apimachinery/pkg/runtime:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/types:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/util/json:go_default_library", <ins> "//staging/src/k8s.io/apimachinery/pkg/util/sets:go_default_library", </ins> "//staging/src/k8s.io/client-go/util/workqueue:go_default_library", "//vendor/k8s.io/klog:go_default_library", "//vendor/sigs.k8s.io/yaml:go_default_library",

There is no reason this should be allowed to happen, especially that we can't configure duplicate plugins differently since plugin configs is a separate list . /sig scheduling
/label help-wanted
The label(s) cannot be applied. These labels are supported: <detailsIn response to : /label help-wanted Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/help-wanted
/help /good-first-issue
/assign
/remove-good-first-issue /remove-help


Question: cache.moveNodeInfoToHead(node.Name) } <del> cache.nodeTree.RemoveNode(node) </del> <ins> if err := cache.nodeTree.RemoveNode(node); err != nil { return err } </ins> cache.removeNodeImageStates(node) return nil }

<!-- Please only use this template for submitting enhancement requests --What would you like to be : 1) Increase the log level of info logs to "v=2" when the scheduler or a node from its cache. 2) Increase the log level of info logs to "v=2" when the scheduler can't schedule a pod , , and add a log message for 2) report memory/cpu status of . Why is this needed: With cluster autoscaler now being used in production by some vendors, we want to have more visibility to events that could be related to the auto scaler to help debug customer issues. /sig scheduling /priority important-soon /assign
GitHub didn't allow me to assign the following users: congliu01. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : <!-- Please only use this template for submitting enhancement requests --1) Increase the log level of info logs to "v=2" when the scheduler or a node from its cache. 2) report memory/cpu status of . Why is this needed: With cluster autoscaler now being used in production by some vendors, we want to have more visibility to events that could be related to the auto scaler to help debug customer issues. /priority important-soon /assign </details
will consider this assigned to you.
/assign


Question: import ( "bytes" <del> "encoding/json" "fmt" </del> "io/ioutil" <del> "net/http" </del> "os" "path/filepath" "regexp" <del> "strings" "sync/atomic" </del> "testing" <del> "time" </del> "k8s.io/api/core/v1" "k8s.io/apimachinery/pkg/api/errors"

Just got this while running the tests:
let's please move this to /close /sig cluster-lifecycle


Question: // GetActivePods returns pods that may have a running container (a // terminated pod is one that is known to have no running containers and // will not get any more). <ins> // // TODO: This method must include pods that have been force deleted from // the config source (and thus removed from the pod manager) but are still // terminating. </ins> func (kl *Kubelet) GetActivePods() []*v1.Pod { allPods := kl.podManager.GetPods() activePods := kl.filterOutTerminatedPods(allPods)

There appears to be a change of behavior / possible regression in k8s 1.22 with respect to pod lifecycle behavior. We've started to see some cases when multiple pods are being created (often as part of a k8s job) the following will happen: pods get created by e.g. job controller (e.g. say 5). Each pod in job requests majority of node memory so only one such pod can fit on node at a time. schedules [first] pod to node, rest (4 pods) stay in pending accepts first pod, runs it to completion pod get scheduled to the node rejects the pod during pod admission, e.g. with , despite the fact that the first pod has already completed successfully This results in many of pods backing the job getting into Failed phase and having to be recreated by job controller. The job can often get into BackofLimit exceeded to hitting backoffLimit. The root issue seems to be a mismatch of when kubelet actually considers the pod to complete and what it reports to API server (which causes the followup pods to be scheduled to it). I'm still investigating further, but this may be related to the pod lifecycle changes in 1.22 -- I've created a e2e test as minimal repro of this behavior -- (Test fails) 22 build log -- 22 kubelet log -- (Test passes) 21 build log -- 21 kubelet log -- <!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --- Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository.