Question: for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { <ins> featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, features.SchedulerQueueingHints, tt.qHintEnabled) </ins> nodeInfo := framework.NewNodeInfo(tt.existingPods...) nodeInfo.SetNode(tt.nodeFn()) preCheckFn := preCheckForNode(nodeInfo) <del> var got []bool </del> <ins> got := make([]bool, 0, len(tt.pods)) </ins> for _, pod := range tt.pods { <del> got = append(got, preCheckFn(pod)) </del> <ins> got = append(got, preCheckFn == nil || preCheckFn(pod)) </ins> } if diff := cmp.Diff(tt.want, got); diff != "" {

EDIT: It is a bad habit of mine to always write long sentences even though I am not very good at English. Summarizing the above: I think it's not a good idea to create something that can be achieved in in-tree plugins but not in out-of-tree plugins. If we need a solid advantage to extend like this, then it has the advantage of optimizing the queue per plugin like described above.
I didn't jump in the thread since it's actively on the discussion. But, the issue is the similar one. (This time, we can solve that issue by changing scheduling queue logic since It's the k/k feature though.)
I'm not sure I fully understand your intention. The slack thread was discussing the AssignedPod update case, so it fallls into case 1 in , which we already agreed to improve in upstream.
Hi Are you working on this? I think this is quite important and if you're not working on this, I can follow.
Sorry, leave here for a while. I wanted to wait for (and then forgot to come back here. ) That proposal is quite similar to what I want to see in the conclusion of this issue; every plugin can judge if each Pod in unschedQ can be moved to schedQ/backoffQ. Do you think it makes sense to add "By which event the pod will be moved back to schedQ/backoffQ" on 's argument? I know and isn't needed in DefaultEnqueue plugin for API, but that's the last missing puzzle piece : If has , enqueue plugin can judge if the event makes it possible to schedule the Pod. For example, I believe we can move ↓ this logic to Pod Affinity plugin by implementing in Pod Affinity plugin. Pod Affinity plugin will remember Pods rejected by itself in . And in , if the given Pod was the one rejected by Pod Affinity, is AssignedPod/Created|Updated and is Pod affinity with the rejected Pod, then move the Pod back to activeQ/backoffQ. Otherwise reject the Pod moving. We can resolve (3) by implementing as well. (by exactly the way you said.↓) My original concern in this issue is that having logic for specific plugins in the scheduler core may prevent out-of-tree plugins' extendability. The scheduler itself should be pure, and shouldn't do anything special for any plugins implementation ideally.


Question: for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { <ins> featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, features.SchedulerQueueingHints, tt.qHintEnabled) </ins> nodeInfo := framework.NewNodeInfo(tt.existingPods...) nodeInfo.SetNode(tt.nodeFn()) preCheckFn := preCheckForNode(nodeInfo) <del> var got []bool </del> <ins> got := make([]bool, 0, len(tt.pods)) </ins> for _, pod := range tt.pods { <del> got = append(got, preCheckFn(pod)) </del> <ins> got = append(got, preCheckFn == nil || preCheckFn(pod)) </ins> } if diff := cmp.Diff(tt.want, got); diff != "" {

If we can move all such logic to each plugin, that'd be perfect.
^ is more like a new feature than just a solution to this problem. I created another issue and describe my proposal in detail. Let's have a discussion there.
It can be resolved by moving all preCheck into each plugin's QueueingHint. Let me retitle this issue so that the current status is easy to understand. /retitle scheduler: move all preCheck to QueueingHint
This issue has not been updated in over 1 year, and should be re-triaged. You can: Confirm that this issue is still relevant with (org members only) Close this issue with For more details on the triage process, see /remove-triage accepted
/triage accepted


What happened: As part of creating an e2e test for , it was found that the apiserver proxy server is processing a blank redirect for any for 21 endpoints across pods, services and nodes. For a pod, redirecting for any method other than GET or HEAD is a bug, you expected to happen: This is the current behaviour for the apiserver but with a minimal code change the redirect can apply to only the following endpoints; - - - How to reproduce it (as minimally and precisely as possible): Create any standard pod and service in the default namespace run in another terminal the response from running the following commands produces a can be replaced with other http verbs , , , , and which also produce 301 responses. Anything else we need to know?: networking /sig testing /sig architecture
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : What happened: - - - run in another terminal the response from running the following commands produces a testing /sig architecture Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Does this issue and PR address your concerns about the apiserver proxy server doing a blanket direct of any http call to the root proxy endpoints?
commented on the associated PR, thanks
/remove-sig testing
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale Will be addressed late in 1.21
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.


This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
/remove-lifecycle stale
/remove-lifecycle stale


Question: ] + select({ "@io_bazel_rules_go//go/platform:android": [ "//pkg/util/procfs:go_default_library", <ins> "//staging/src/k8s.io/api/rbac/v1:go_default_library", </ins> "//staging/src/k8s.io/apimachinery/pkg/fields:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/labels:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/types:go_default_library",

Which jobs are failing: Which test(s) are failing: Since when has it been failing: Last successful run (23 Oct 2020, 10:07 PDT): First unsuccessful run (23 Oct 2020, 12:08 PDT): Testgrid link: Reason for failure: Anything else we need to know: presubmit job in repository also started to fail in the same fashion. Attempted to debug it myself, found out the following stuff: last successful run was accessing the apiserver via HTTP: all unsuccessful runs attempted to access the apiserver via HTTPS but to no avail: perused the history of to find out that the PR is almost certainly the reason of the failures (or rather uncovered some problem with the NPD or its tests): first, it coincides with the start of the failures of , second, we changed the default hostname of the apiserver in node tests in that PR. Judging by the code and the changes in the aforementioned PR I think that a quick mitigation would be to add a flag to the of these jobs. /assign /sig node /cc
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Opened for a mitigation that would make those jobs stop failing.
This won't work, unfortunately. Copying response:
I have filed to address this issue, but it seems I could not run the test in advance to verify my patch, did you know how to manually trigger the presubmit job?
Unfortunately I don't :/. My PR was also a wild guess. could you take a look at the proposed fix? Thanks once again for your help - really appreciated :).
the submission of hasn't fixed the problems, unfortunately :/. Sample failing run: We still have some peculiar problem related to health checking: and now NPD panics, too:
Thanks for your report, I believe the reason is admission controller has been disabled in node e2e tests: We need to figure out a new approach now. /reopen
Reopened this issue.


Question: ] + select({ "@io_bazel_rules_go//go/platform:android": [ "//pkg/util/procfs:go_default_library", <ins> "//staging/src/k8s.io/api/rbac/v1:go_default_library", </ins> "//staging/src/k8s.io/apimachinery/pkg/fields:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/labels:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/types:go_default_library",

<detailsIn response to : Thanks for your report, I believe the reason is admission controller has been disabled in node e2e tests: We need to figure out a new approach now. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
filed . I hope we could completely address this issue this time.
/milestone v1.20


Question: if nvcSize > 0 && nvcSize+vcSize >= numCandidates { cancel() } <del> } else { statusesLock.Lock() nodeStatuses[nodeInfoCopy.Node().Name] = status statusesLock.Unlock() </del> <ins> return } if status.IsSuccess() && len(pods) == 0 { status = framework.AsStatus(fmt.Errorf("expected at least one victim pod on node %q", nodeInfoCopy.Node().Name)) </ins> } <ins> statusesLock.Lock() nodeStatuses[nodeInfoCopy.Node().Name] = status statusesLock.Unlock() </ins> } fh.Parallelizer().Until(parallelCtx, len(potentialNodes), checkNode) return append(nonViolatingCandidates.get(), violatingCandidates.get()...), nodeStatuses

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --Kubernetes 1.19 (confirmed with 1.19.7 and 1.19.10) kube-scheduler crashes and restarts with the following errors in . panic: runtime error: index out of range [0] with length 0 The problem code is line 389. When is nil or is empty, the error happens. If we skip when it's nil or empty, the error is gone. panic: runtime error: invalid memory address or nil poster dereference The problem is does not exist and returns nil sometimes. Simply skipping it won't solve the problem. The scheduler will reach either line 456 or 341. The scheduler works without failures. When there are preemptions in a cluster. Kubernetes version (use ): v1.19.10 or v1.19.7 Cloud provider or hardware configuration: bare metal OS (e.g: ): centos 7 Kernel (e.g. ): Linux 5.4.77-7.el7pie SMP Sat Nov 21 01:16:27 UTC 2020 x8664 x8664 x8664 GNU/Linux Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scheduling
/cc
/assign
Are you running the vanilla scheduler? or a customized one that leverages the utilities in
It's a custom scheduler with some internal out-of-tree plugins (just like ), but it doesn't have any custom postFilter plugins. Which utility functions are you referring to? Thanks.
are those custom plugins filter plugins? do they maintain state?


Question: if nvcSize > 0 && nvcSize+vcSize >= numCandidates { cancel() } <del> } else { statusesLock.Lock() nodeStatuses[nodeInfoCopy.Node().Name] = status statusesLock.Unlock() </del> <ins> return } if status.IsSuccess() && len(pods) == 0 { status = framework.AsStatus(fmt.Errorf("expected at least one victim pod on node %q", nodeInfoCopy.Node().Name)) </ins> } <ins> statusesLock.Lock() nodeStatuses[nodeInfoCopy.Node().Name] = status statusesLock.Unlock() </ins> } fh.Parallelizer().Until(parallelCtx, len(potentialNodes), checkNode) return append(nonViolatingCandidates.get(), violatingCandidates.get()...), nodeStatuses

If yes, then one hypothesis is that those custom filters filter the node in the filter phase, but not when executed in the preemption phase, and so this will result in a candidate node with no victims?
Yes, it includes filter plugins and uses cycle state.
you need to make sure that those filter plugins play nicely when executed again in the preemption phase in the same cycle: i.e., produce the same result.
Thanks What do you mean by performing filter in the preemption phase? would you mind elaborating it a little?
What additional logic is needed to handle the preemption case?
in the preemption phase we run the filters again to check if removing lower priority pods make the node schedulable: we later add them one by one to reduce the set of victim pods to the absolute minimum: My hypothesis is that somehow your filter returns success when adding all the pods back, and so you end up with zero victims and a candidate node. This shouldn't happen because if the pod fits the node without removing any victims, then we shouldn't be running preemption in the first place. So the theory is that the custom filter is returning false when run in the filter phase, but returns true when executed in the preemption phase perhaps because it makes some assumptions about cyclestate that makes it behave this way (e.g. that a the filter will be executed once per node in a scheduling cycle).
Disabled the filter plugin, but still see the same issues.
are you able to change the scheduler code and run the test again? I can send a patch tomorrow to add some debugging messages to help us root cause the issue.
yes, thanks a lot! really appreciate it!
You are right. We still use a deprecated Predicate extender, which causes the problem. After disabling it, it's working fine. It used to work with 1.18 though. We are retiring it. Thank you so much!!!
If the test cases can help debug scheduler extender Webhook (Predicate) too, I'd like to try it. Thanks again!
How can we improve the preemption logic to prevent bugs and issues in custom Filters or Predicates crashing the scheduler?
for starter, we should check the list len, the scheduler should not panic in all cases.


Question: if nvcSize > 0 && nvcSize+vcSize >= numCandidates { cancel() } <del> } else { statusesLock.Lock() nodeStatuses[nodeInfoCopy.Node().Name] = status statusesLock.Unlock() </del> <ins> return } if status.IsSuccess() && len(pods) == 0 { status = framework.AsStatus(fmt.Errorf("expected at least one victim pod on node %q", nodeInfoCopy.Node().Name)) </ins> } <ins> statusesLock.Lock() nodeStatuses[nodeInfoCopy.Node().Name] = status statusesLock.Unlock() </ins> } fh.Parallelizer().Until(parallelCtx, len(potentialNodes), checkNode) return append(nonViolatingCandidates.get(), violatingCandidates.get()...), nodeStatuses

Other than that, I think we need to clearly document what the preemption logic does (that it executes the prefilter extensions points and the filter plugins multiple times) and the expectations from filter plugins (that they may get executed more than once in the same cycle)
Is it a predicate extender or preemption extender? If it's a predicate extender, I don't quite think we found the root cause - b/c predicate extenders are not invoked during preemption (). It makes more sense if it's a preemption extender - if the preemption extender mutates the to make it invalid. If it's the case, could you verify it by adding debug info before and after L157 to check the validness of ?
That's always true :)
filed a PR to mitigate it.
It's a predicate extender. I'll debug it more.
Added some debug info. after and . Here's the result. returned 4 candidates, but one of candidates () has zero pod. The candidate happened to be the best candidate and hence caused the "" error. It should have nothing to do with since our extender doesn't support Preemption. Disabling it may change pod placements/candidates selection and bypassed the error. It's like a coincidence. To remove the panic, a simple fix is to check if is empty. If it is, do not add the node to candidates. Alternatively, a change to may work too. Here is the log with debug info.
This looks more promising. Back to digging into the root cause, it seems returns an empty but with . So that implies firstly we have a non-nil , and then in , the is separated into two list: violatingVictims and nonViolatingVictims. Finally the two victims list are tried one by one to run . It looks like returns {fits=true, nil error} for every victim, which is quite abnormal. I think it's problematic that every pod is retrievable and in which we can debug into.
Yes, should not return fit=true with empty pods in the first place, but it's hard to locate the problematic code in this function.
The following code in looks questionable to me. If is empty, should the function return nil, nil? Also, if is true for all extenders, the loop won't do anything and candidates will remain the same.


Question: if nvcSize > 0 && nvcSize+vcSize >= numCandidates { cancel() } <del> } else { statusesLock.Lock() nodeStatuses[nodeInfoCopy.Node().Name] = status statusesLock.Unlock() </del> <ins> return } if status.IsSuccess() && len(pods) == 0 { status = framework.AsStatus(fmt.Errorf("expected at least one victim pod on node %q", nodeInfoCopy.Node().Name)) </ins> } <ins> statusesLock.Lock() nodeStatuses[nodeInfoCopy.Node().Name] = status statusesLock.Unlock() </ins> } fh.Parallelizer().Until(parallelCtx, len(potentialNodes), checkNode) return append(nonViolatingCandidates.get(), violatingCandidates.get()...), nodeStatuses

Why do we bother creating a new candidates later? Can it just return candidates?
It can prevent from returning candidates with empty pods, but what about ? It's possible to add nodes without victim pods to candidates if returned by contain such nodes. How about adding the following check to CallExtender?
Updated the PR based on the finding and discussions.
Yes, we can use a flag to mark if all extenders don't support preemption. If yes, simply return the candidates.
Technically this can exist due to faulty extender implementation. If we really want to guard it, instead of logging the error and continue, I'm more inclined to return the error immediately as this is a fatal error - the victimsMap cannot be used either for latter extender or further preemptor nominating.
Is my understanding correct? In , if victimsMap is empty, should it return instead of ?
The only case victimsMap can be empty is empty candidates, and so return either doesn't quite matter, right?
You are right.
As long as one extender returns an invalid victimsMap (with empty pods), s returns an error immediately? What about ? Should it return an error immediately if it returnswith empty pods?
I think so. Because the result it returned is faulty, and we don't want to continue based on it The same. We may don't want to continue the scheduling cycle based on faulty result.


Question: // Do this multiple times to show that it rate limit rejected requests don't block. for i := 0; i < 2; i++ { <del> expectHTTP(server.URL, errors.StatusTooManyRequests, t) </del> <ins> if err := expectHTTP(server.URL, errors.StatusTooManyRequests); err != nil { t.Error(err) } </ins> } // Validate that non-accounted URLs still work <del> expectHTTP(server.URL+"/dontwait/watch", http.StatusOK, t) </del> <ins> if err := expectHTTP(server.URL+"/dontwait/watch", http.StatusOK); err != nil { t.Error(err) } </ins> // Let all hanging requests finish block.Done() // Show that we recover from being blocked up. // Too avoid flakyness we need to wait until at least one of the requests really finishes. <del> oneAccountedFinished.Wait() expectHTTP(server.URL, http.StatusOK, t) </del> <ins> responses.Wait() if err := expectHTTP(server.URL, http.StatusOK); err != nil { t.Error(err) } </ins> } func TestReadOnly(t *testing.T) {

Have only seen this once, personally.
This flaked in Jenkins as well. [job/kubernetes-test-go/5351]
can you take a look at this? I think has a lot already :)


Question: }, func(client kubeclientset.Interface, obj pkgruntime.Object) error { configmap := obj.(*apiv1.ConfigMap) <del> err := client.Core().ConfigMaps(configmap.Namespace).Delete(configmap.Name, &metav1.DeleteOptions{}) </del> <ins> orphanDependents := false err := client.Core().ConfigMaps(configmap.Namespace).Delete(configmap.Name, &metav1.DeleteOptions{OrphanDependents: &orphanDependents}) </ins> return err })

Problem: When users sends a cascading deletion request to federation i.e they set orphanDependents=false, then federation should also set orphanDependents=false, when it sends the request to underlying clusters.
Fixed by


Question: serviceAccountName: fluentd-es containers: - name: fluentd-es <del> image: gcr.io/google-containers/fluentd-elasticsearch:v2.0.0 </del> <ins> image: gcr.io/google-containers/fluentd-elasticsearch:v2.0.1 </ins> env: - name: FLUENTD_ARGS value: --no-supervisor -q

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: [root EFK]# kubectl logs -f fluentd-es-jrsf7 -n kube-system /var/lib/gems/2.1.0/gems/ffi- block in ffilib': Could not open library '': cannot open shared object file: No such file or directory. (LoadError) Could not open library '': cannot open shared object file: No such file or directory. Could not open library 'libsystemd-': cannot open shared object file: No such file or directory. Could not open library 'libsystemd-': libsystemd- cannot open shared object file: No such file or directory from /var/lib/gems/2.1.0/gems/ffi-map' from /var/lib/gems/2.1.0/gems/ffi- ffilib' from /var/lib/gems/2.1.0/gems/systemd-journal-<module:Native' from /var/lib/gems/2.1.0/gems/systemd-journal- <module:Id128' from /var/lib/gems/2.1.0/gems/systemd-journal-<module:Systemd' from /var/lib/gems/2.1.0/gems/systemd-journal- <top (required)' fromrequire' from require' from /var/lib/gems/2.1.0/gems/systemd-journal-<top (required)' from require' fromrequire' from /var/lib/gems/2.1.0/gems/systemd-journal- <top (required)' fromrequire' from rescue in require' fromrequire' from /var/lib/gems/2.1.0/gems/fluent-plugin-systemd- <top (required)' fromrequire' from require' from /var/lib/gems/2.1.0/gems/fluentd-block in tryloadplugin' from /var/lib/gems/2.1.0/gems/fluentd- each' from /var/lib/gems/2.1.0/gems/fluentd-tryloadplugin' from /var/lib/gems/2.1.


Question: serviceAccountName: fluentd-es containers: - name: fluentd-es <del> image: gcr.io/google-containers/fluentd-elasticsearch:v2.0.0 </del> <ins> image: gcr.io/google-containers/fluentd-elasticsearch:v2.0.1 </ins> env: - name: FLUENTD_ARGS value: --no-supervisor -q

0/gems/fluentd- newimpl' from /var/lib/gems/2.1.0/gems/fluentd-newinput' from /var/lib/gems/2.1.0/gems/fluentd- addsource' from /var/lib/gems/2.1.0/gems/fluentd-block in configure' from /var/lib/gems/2.1.0/gems/fluentd- each' from /var/lib/gems/2.1.0/gems/fluentd-configure' from /var/lib/gems/2.1.0/gems/fluentd- configure' from /var/lib/gems/2.1.0/gems/fluentd-runconfigure' from /var/lib/gems/2.1.0/gems/fluentd- runconfigure' from /var/lib/gems/2.1.0/gems/fluentd-block in start' from /var/lib/gems/2.1.0/gems/fluentd- call' from /var/lib/gems/2.1.0/gems/fluentd-mainprocess' from /var/lib/gems/2.1.0/gems/fluentd- start' from /var/lib/gems/2.1.0/gems/fluentd-<top (required)' from require' fromrequire' from /var/lib/gems/2.1.0/gems/fluentd-0.12.39/bin/fluentd:5:in <top (required)' from /usr/local/bin/fluentd:23:inload' from /usr/local/bin/fluentd:23:in <main' What you expected to happen: fluentd-es run success How to reproduce it (as minimally and precisely as possible): kubectl create -f fluentd-es- the image is google-containers/fluentd-elasticsearch:v2.0.0 Anything else we need to know?


Question: serviceAccountName: fluentd-es containers: - name: fluentd-es <del> image: gcr.io/google-containers/fluentd-elasticsearch:v2.0.0 </del> <ins> image: gcr.io/google-containers/fluentd-elasticsearch:v2.0.1 </ins> env: - name: FLUENTD_ARGS value: --no-supervisor -q

: Environment: Kubernetes version (use ):Client Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-07-21T08:23:22Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-07-21T08:08:00Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration**: OS (e.g. from /etc/os-release):centos 7.3 Kernel (e.g. ):Linux node-21.46 3.10.0-514.26.2.el7.x8664 SMP Tue Jul 4 15:04:05 UTC 2017 x8664 x8664 x86_64 GNU/Linux Install tools: Others:
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
Yup, the situation with fluentd on the system with systemd is very sloppy. There's a hack to make it works sometimes, but it seems that it doesn't work. I'll look into it
/sig instrumentation /assign
Facing same issue.


Question: serviceAccountName: fluentd-es containers: - name: fluentd-es <del> image: gcr.io/google-containers/fluentd-elasticsearch:v2.0.0 </del> <ins> image: gcr.io/google-containers/fluentd-elasticsearch:v2.0.1 </ins> env: - name: FLUENTD_ARGS value: --no-supervisor -q

Environment Details:kubectl version Client Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-07-03T15:31:10Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-07-03T15:31:10Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"} OS (e.g. from /etc/os-release):Centos 7 (Core) Kernel (e.g. uname -a): Linux xxxx 3.10.0-327.el7.x8664 Error for reference /var/lib/gems/2.1.0/gems/ffi- block in ffilib': Could not open library '': cannot open shared object file: No such file or directory. (LoadError) Could not open library '': cannot open shared object file: No such file or directory. Could not open library 'libsystemd-': cannot open shared object file: No such file or directory. Could not open library 'libsystemd-': libsystemd- cannot open shared object file: No such file or directory from /var/lib/gems/2.1.0/gems/ffi-map' from /var/lib/gems/2.1.0/gems/ffi- ffilib' from /var/lib/gems/2.1.0/gems/systemd-journal-<module:Native' from /var/lib/gems/2.1.0/gems/systemd-journal- <module:Id128' from /var/lib/gems/2.1.0/gems/systemd-journal-<module:Systemd' from /var/lib/gems/2.1.0/gems/systemd-journal- <top (required)' fromrequire' from require' from /var/lib/gems/2.1.0/gems/systemd-journal-<top (required)' from require' fromrequire' from /var/lib/gems/2.1.


Question: serviceAccountName: fluentd-es containers: - name: fluentd-es <del> image: gcr.io/google-containers/fluentd-elasticsearch:v2.0.0 </del> <ins> image: gcr.io/google-containers/fluentd-elasticsearch:v2.0.1 </ins> env: - name: FLUENTD_ARGS value: --no-supervisor -q

0/gems/systemd-journal- <top (required)' fromrequire' from rescue in require' fromrequire' from /var/lib/gems/2.1.0/gems/fluent-plugin-systemd- <top (required)' fromrequire' from require' from /var/lib/gems/2.1.0/gems/fluentd-block in tryloadplugin' from /var/lib/gems/2.1.0/gems/fluentd- each' from /var/lib/gems/2.1.0/gems/fluentd-tryloadplugin' from /var/lib/gems/2.1.0/gems/fluentd- newimpl' from /var/lib/gems/2.1.0/gems/fluentd-newinput' from /var/lib/gems/2.1.0/gems/fluentd- addsource' from /var/lib/gems/2.1.0/gems/fluentd-block in configure' from /var/lib/gems/2.1.0/gems/fluentd- each' from /var/lib/gems/2.1.0/gems/fluentd-configure' from /var/lib/gems/2.1.0/gems/fluentd- configure' from /var/lib/gems/2.1.0/gems/fluentd-runconfigure' from /var/lib/gems/2.1.0/gems/fluentd- runconfigure' from /var/lib/gems/2.1.0/gems/fluentd-block in start' from /var/lib/gems/2.1.0/gems/fluentd- call' from /var/lib/gems/2.1.0/gems/fluentd-mainprocess' from /var/lib/gems/2.1.0/gems/fluentd- start' from /var/lib/gems/2.1.0/gems/fluentd-<top (required)' from require' fromrequire' from /var/lib/gems/2.1.0/gems/fluentd-0.12.39/bin/fluentd:5:in <top (required)' from /usr/local/bin/fluentd:23:inload' from /usr/local/bin/fluentd:23:in <


Question: serviceAccountName: fluentd-es containers: - name: fluentd-es <del> image: gcr.io/google-containers/fluentd-elasticsearch:v2.0.0 </del> <ins> image: gcr.io/google-containers/fluentd-elasticsearch:v2.0.1 </ins> env: - name: FLUENTD_ARGS value: --no-supervisor -q

main'
You can solve it temporary by removing I'm working on the solution
A solution here might be to base fluentd-es off an image with journald supporting XZ and LZ4 compression. I'm looking :)
Thanks


Question: package validation import ( <ins> "strings" </ins> "testing" <del> "k8s.io/api/core/v1" </del> <ins> v1 "k8s.io/api/core/v1" </ins> "k8s.io/apimachinery/pkg/api/resource" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/util/sets"

Created limit range object with this parameters then tried to create this pod got message: Error from server (Invalid): error when creating "pod-to-be-": Pod "nginx-to-be-removed-4" is invalid: spec.containers[0].resources.requests: Invalid value: "700m": must be less than or equal to cpu limit Pod should be created with Requests & Limits of cpu equal to 700m (since it is in range of min/max cpu limitrange object) create limitrange object and pod as in description No response <details</details<detailson premise/AKS </details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
You have a default CPU limit of 10m, and no explicit CPU limit on the container, but a request larger than the limit. The request must always be less than or equal to the limit. Your pod needs to also set a higher limit, or the limit range needs to not set a default limit.
/remove-kind bug /kind support
/cc
but this is not obvious behavior since there is min & max set in limit range. This is limits (not min/max) that should be overwritten by pod. limits should be set to requests in this case imho
limit range limits what the user can set, it doesn't perform any mutations other than the defaulting and doesn't claim to. Min/max enforce that pods cannot set higher / lower. The default ensures that if the pod doesn't set a limit it will have one. personally I would remove the default limit, especially at 10m, this will aggressively limit pods. it would be, if the pod set a limit that would be a substantial breaking change, I don't think we could do that short of a v2 API and I'm not sure it would be correct behavior even then. we could make it opt-in, but then, why not just set them to match on the pod to begin with?


Question: <ins> /* Copyright 2017 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package main import ( "fmt" "os" ) func main() { fmt.Printf("Effective uid: %dn", os.Geteuid()) } </ins>

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind feature This issue is for creating image as multiarch
/assign
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig testing


Question: * Internally, the Kubernetes master can decide the defaulting behavior and the kubelet implementation may expected an absolute specification.  For example, if the master decided that "the default is unbounded" it would pass 2^64 to the kubelet. ## Kubernetes-defined resource types The following resource types are predefined ("reserved") by Kubernetes in the `kubernetes.io` namespace, and so cannot be used for user-defined resources.  Note that the syntax of all resource types in the resource spec is deliberately similar, but some resource types (e.g., CPU) may receive significantly more support than simply tracking quantities in the schedulers and/or the Kubelet.

did which addresses some of them, but turns up a lot more.
Alright, I will do that.


Question: t.Fatalf("Failed to pop a pod %v", err) } checkPerPodSchedulingMetrics("Attempt twice with update", t, pInfo, 2, timestamp) <ins> // Case 4: A gated pod is created and scheduled after lifting gate. The queue operations are //  Add gated pod -> check unschedulablePods -> lift gate & update pod -> Pop. c = testingclock.NewFakeClock(timestamp) // Create a queue with PreEnqueuePlugin m := map[string][]framework.PreEnqueuePlugin{"": {&preEnqueuePlugin{allowlists: []string{"foo"}}}} queue = NewTestQueue(ctx, newDefaultQueueSort(), WithClock(c), WithPreEnqueuePluginMap(m), WithPluginMetricsSamplePercent(0)) // Create a pod without PreEnqueuePlugin label. gatedPod := st.MakePod().Name("gated-test-pod").Namespace("test-ns").UID("test-uid").Obj() err = queue.Add(logger, gatedPod) if err != nil { t.Fatalf("Failed to add a pod %v", err) } // Check pod is added to the unschedulablePods queue. if getUnschedulablePod(queue, gatedPod) != gatedPod { t.Errorf("Pod %v was not found in the unschedulablePods.", gatedPod.Name) } // Override clock to get different InitialAttemptTimestamp c.Step(1 * time.Minute) // Update pod with the required label to get it out of unschedulablePods queue. updateGatedPod := gatedPod.DeepCopy() updateGatedPod.Labels = map[string]string{"foo": ""} queue.Update(logger, gatedPod, updateGatedPod) pInfo, err = queue.Pop() if err != nil { t.Fatalf("Failed to pop a pod %v", err) } checkPerPodSchedulingMetrics("Attempt once/gated", t, pInfo, 1, timestamp.Add(1*time.Minute)) </ins> } func TestIncomingPodsMetrics(t *testing.T) {

is recording the time that a Pod is gated. We use the timestamp when the scheduler inserts the pod into the queue: The period of time when a Pod fails PreEnque (like being gated) shouldn't be accounted in the podschedulingdurationseconds. Create a Pod with scheduling gates. Wait some time before removing the gate. Observe the metric No response <details</details<detailsAny </details<details</details<details</details<details</details<details</details
/sig scheduling
/help /triage accepted
This request has been marked as needing help from a contributor. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help /triage accepted Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
would you want to take a stab on this issue?
Yup, I can give it a shot. /assign helayoty
, as we discussed, we can address this issue with different solutions: the pod timestamp to the current time , so the gated time is substituted. timestamp . Use a pointer to the InitialAttemptTimestamp and interpret as the newest timestamp so the pod will go to the back of the queue. I'll go ahead and implement solution no. 3, wdyt?
sgtm. Any concerns ?
I think it should be option 1 + 3, and pInfo.Timestamp should be pInfo.InitialAttemptTimestamp, right? as records the time based on :
I think we should revisit this issue, in terms of its assumption: So is it reasonable to always exclude time spent in PreEnqueue from ? Note that PreEnqueue != SchedulingGates, is one of the PreEnqueue's usecases.


Question: t.Fatalf("Failed to pop a pod %v", err) } checkPerPodSchedulingMetrics("Attempt twice with update", t, pInfo, 2, timestamp) <ins> // Case 4: A gated pod is created and scheduled after lifting gate. The queue operations are //  Add gated pod -> check unschedulablePods -> lift gate & update pod -> Pop. c = testingclock.NewFakeClock(timestamp) // Create a queue with PreEnqueuePlugin m := map[string][]framework.PreEnqueuePlugin{"": {&preEnqueuePlugin{allowlists: []string{"foo"}}}} queue = NewTestQueue(ctx, newDefaultQueueSort(), WithClock(c), WithPreEnqueuePluginMap(m), WithPluginMetricsSamplePercent(0)) // Create a pod without PreEnqueuePlugin label. gatedPod := st.MakePod().Name("gated-test-pod").Namespace("test-ns").UID("test-uid").Obj() err = queue.Add(logger, gatedPod) if err != nil { t.Fatalf("Failed to add a pod %v", err) } // Check pod is added to the unschedulablePods queue. if getUnschedulablePod(queue, gatedPod) != gatedPod { t.Errorf("Pod %v was not found in the unschedulablePods.", gatedPod.Name) } // Override clock to get different InitialAttemptTimestamp c.Step(1 * time.Minute) // Update pod with the required label to get it out of unschedulablePods queue. updateGatedPod := gatedPod.DeepCopy() updateGatedPod.Labels = map[string]string{"foo": ""} queue.Update(logger, gatedPod, updateGatedPod) pInfo, err = queue.Pop() if err != nil { t.Fatalf("Failed to pop a pod %v", err) } checkPerPodSchedulingMetrics("Attempt once/gated", t, pInfo, 1, timestamp.Add(1*time.Minute)) </ins> } func TestIncomingPodsMetrics(t *testing.T) {

In other words, some user of PreEnqueue may want to include the time spent in PreEnqueue.
is a signal that cluster providers can use to build SLOs. The time spent before passing PreEnqueue is not under control of kube-scheduler, so considering it would not allow to build an SLO. Are you aware of other users of PreEnqueue?
True, but comparing to state that metric X can or cannot be used to build SLO (which is hard IMO), I'm in favor of giving suggestion on SLI/SLO's semantic definition, for example: the official SLI definition of has a footnote clearly explaining its definition: So if a pod is gated, it's not "immediately schedulable", and hence doesn't fall into the definition of podstartup_latency.
A viable solution might be providing an extra label like to explicitly exclude the time spent on preenqueue.
That is true, but there is no way today to exclude those pods from the metric, or separate the time. Maybe we need an extra metric, but I think the new metric could be the one that includes the time guarded in PreEnqueue. It should probably be part of the KEP. The old metric is already stable, so we can't add labels. And it might already be in use for SLOs.
cc
I think we should be working on that. And treat it as part of GA requirement for PodSchedulingReadiness.
An edge(maybe not edge) case is a pod's scheduling attempts is 0 but the is a long time, so prefer not to account for the waiting time when gated. For users not using PreEnqueue, they will not be impacted, but for users using PreEnqueue, it's a break change for it increases the duration time. And we can add a new metric like to measure how long a pod stays in a scheduler.
I thought of "attempts", it's just I'm not sure it's a good idea to rely on "attempts" to define a SLI/SLO, b/c "attempts" is obtained from the in-memory object and in a HA config env, it's only updated by the primary scheduler instance. In other words, the value is inaccurate when the lease owner switches.
attempt=0 is good enough for most scenarios
do we have a conclusion?


Question: t.Fatalf("Failed to pop a pod %v", err) } checkPerPodSchedulingMetrics("Attempt twice with update", t, pInfo, 2, timestamp) <ins> // Case 4: A gated pod is created and scheduled after lifting gate. The queue operations are //  Add gated pod -> check unschedulablePods -> lift gate & update pod -> Pop. c = testingclock.NewFakeClock(timestamp) // Create a queue with PreEnqueuePlugin m := map[string][]framework.PreEnqueuePlugin{"": {&preEnqueuePlugin{allowlists: []string{"foo"}}}} queue = NewTestQueue(ctx, newDefaultQueueSort(), WithClock(c), WithPreEnqueuePluginMap(m), WithPluginMetricsSamplePercent(0)) // Create a pod without PreEnqueuePlugin label. gatedPod := st.MakePod().Name("gated-test-pod").Namespace("test-ns").UID("test-uid").Obj() err = queue.Add(logger, gatedPod) if err != nil { t.Fatalf("Failed to add a pod %v", err) } // Check pod is added to the unschedulablePods queue. if getUnschedulablePod(queue, gatedPod) != gatedPod { t.Errorf("Pod %v was not found in the unschedulablePods.", gatedPod.Name) } // Override clock to get different InitialAttemptTimestamp c.Step(1 * time.Minute) // Update pod with the required label to get it out of unschedulablePods queue. updateGatedPod := gatedPod.DeepCopy() updateGatedPod.Labels = map[string]string{"foo": ""} queue.Update(logger, gatedPod, updateGatedPod) pInfo, err = queue.Pop() if err != nil { t.Fatalf("Failed to pop a pod %v", err) } checkPerPodSchedulingMetrics("Attempt once/gated", t, pInfo, 1, timestamp.Add(1*time.Minute)) </ins> } func TestIncomingPodsMetrics(t *testing.T) {

Should we move on with the solution or remove this issue and the linked PR from the current release?
I would like to move forward with this proposal. ?
To move it forward, I think we should add new metric(s) to explicitly tell it's including or excluding the time spent on PreEnqueue. Given is in stable level: We will need a new metric like "podschedulingexcludingpreenqueuedurationseconds".
28 Bug Triage Shadow here. Just checking-in, if this is still on track for the 1.28 release?
We don't have a conclusion yet, but as a bug, it should qualify for changes after the freezes.


Question: - name: db image: mysql resources: <ins> requests: memory: "64Mi" cpu: "250m" </ins> limits: memory: "128Mi" cpu: "500m" - name: wp image: wordpress resources: <del> limits: </del> <ins> requests: memory: "64Mi" cpu: "250m" limits: </ins> memory: "128Mi" cpu: "500m" ``` <del> ## How Pods with Resource Limits are Scheduled </del> <ins> ## How Pods with Resource Requests are Scheduled </ins> When a pod is created, the Kubernetes scheduler selects a node for the pod to run on.  Each node has a maximum capacity for each of the resource types: the amount of CPU and memory it can provide for pods.  The scheduler ensures that, <del> for each resource type (CPU and memory), the sum of the resource limits of the </del> <ins> for each resource type (CPU and memory), the sum of the resource requests of the </ins> containers scheduled to the node is less than the capacity of the node.  Note that although actual memory or CPU resource usage on nodes is very low, the scheduler will still refuse to place pods onto nodes if the capacity check fails.  This protects against a resource shortage on a node when resource usage later increases, such as due to a daily peak in request rate. <del> Note: Although the scheduler normally spreads pods out across nodes, there are currently some cases where pods with no limits (unset values) might all land on the same node. </del> ## How Pods with Resource Limits are Run When kubelet starts a container of a pod, it passes the CPU and memory limits to the container


Question: # TODO: split this out into args when we no longer need to pipe stdout to a file #6428 - sh - -c <del> - '/rescheduler --running-in-cluster=false 1>>/var/log/rescheduler.log 2>&1' </del> <ins> - 'exec /rescheduler --running-in-cluster=false 1>>/var/log/rescheduler.log 2>&1' </ins> volumes: - hostPath: path: /var/log/rescheduler.log

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: When kube-apiserver's livenessProbe suggests that pod is unhealthy, kubelet tries to kill its container with 30 seconds grace period. The container is killed ungracefully with SIGKILL when grace period passes. This happens each time. What you expected to happen: I expected that SIGTERM will be passed to kube-apiserver which will try to gracefully stop before 30 seconds pass (at least in some cases). How to reproduce it (as minimally and precisely as possible): Inject kube-apiserver failure (e.g. ) Watch kubelet logs for container's death You will see that container is killed after 30 seconds each time Anything else we need to know?: I believe the problem is that in kube-apiserver.manifest uses '/bin/sh -c CMD' as a PID1 process which doesn't pass SIGTERM signals to child process. See Environment: Kubernetes version (use ): v1.7.11
/sig node
Looks like without the pipe command will have to be removed which will make the logs hard to track. There is indeed some which makes kube-apiserver run in background and trapping SIGTERM then send a SIGTERM to apiserver's PID.
I think this is the case for all our components, not just kube-apiserver. - FYI
How about using 'exec'-based solution suggested here: In our case it means prepending 'exec': I did changed this in my cluster and the process seems to meet requirements: fd/1 and fd/2 are redirected to /var/log/kube- container is killed instantaneously If everyone is fine with such a solution, I can prepare PR adding 'exec ' in front of all kubernetes/cluster/saltbase/salt/kube-apiserver/*.manifest using '/bin/sh -c CMD' pattern.
Not sure if understand what you mean. Before there was no logs, so /bin/sh -c wasn't needed. -based solution seems to have all required properties. What is your concern about using it?
Umm. just ignore my previous comment. I just tried prepending and it works fine for me.


Question: # TODO: split this out into args when we no longer need to pipe stdout to a file #6428 - sh - -c <del> - '/rescheduler --running-in-cluster=false 1>>/var/log/rescheduler.log 2>&1' </del> <ins> - 'exec /rescheduler --running-in-cluster=false 1>>/var/log/rescheduler.log 2>&1' </ins> volumes: - hostPath: path: /var/log/rescheduler.log

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): No What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): 3 Environment: Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: What happened: All of the salt scripts bundled in kubernetes (for example, ) start up the component in a broken way. They use which means the shell will intercept all signals. So these daemons are not gracefully shutdown when the pod is stopped. This causes serious problems since when apiserver is killed abruptly it doesn't seem to close watches correctly. This was observed in for example. What you expected to happen: Running is incorrect. It should run the binary directly. Otherwise, it needs to perform an so the binary correctly replaces the shell script process. How to reproduce it (as minimally and precisely as possible): Run kube-apiserver inside a container with and try to stop it. You can't - eventually docker will force kill it. Anything else we need to know:
It looks like we changed from running the binaries directly to using in (from a few years ago to enable us to capture logs. or may recall why we had to use a shell since they reviewed that PR.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close.


Question: # TODO: split this out into args when we no longer need to pipe stdout to a file #6428 - sh - -c <del> - '/rescheduler --running-in-cluster=false 1>>/var/log/rescheduler.log 2>&1' </del> <ins> - 'exec /rescheduler --running-in-cluster=false 1>>/var/log/rescheduler.log 2>&1' </ins> volumes: - hostPath: path: /var/log/rescheduler.log

Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


Question: if [[ "${host_arch}" != "amd64" ]]; then echo "Cannot download cfssl on non-amd64 hosts and cfssl does not appear to be installed." echo "Please install cfssl and cfssljson and verify they are in $PATH." <del> echo "Hint: export PATH=$PATH:$GOPATH/bin; go get -u github.com/cloudflare/cfssl/cmd/..." </del> <ins> echo "Hint: export PATH=$PATH:$GOPATH/bin; go install github.com/cloudflare/cfssl/cmd/...@latest" </ins> exit 1 fi

While trying to run with a valid ibmcloud account, the following error happened I also tried to run and got the same error. Looks like a compatibility issue. ibmcloud CLI shouldn't return error Run with a valid IBM cloud account. No response I am trying to build kubemark from source <details</details<detailsIBM cloud IKS </details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scalability
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:


Question: shouldContinueRunning: true, }, { <ins> predicateName: "InsufficientResourceError", </ins> ds: &extensions.DaemonSet{ Spec: extensions.DaemonSetSpec{ Selector: &metav1.LabelSelector{MatchLabels: simpleDaemonSetLabel},

Is this a BUG REPORT or FEATURE REQUEST?: /kind enhancement /sig apps Description: is a key function of DaemonSet; it's better to have more unit test case for it, e.g. node status, pod on the node. There's several test cases to the DaemonSet's finial decision, which are kind of integration test to me; prefer to provide more unit test case for this key function.
GitHub didn't allow me to assign the following users: mathspanda. Note that only can be assigned. <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
do you have any specific cases or shall I figure out something? I can take a look at this issue if you don't mind.
, , thanks very much for our effort on this enhancement; as you known, DaemonSet will use scheduler's predicates to decide which node can place pods; there's several predicates errors , the purpose of this issue is to trigger every error in unit test for code coverage.
Sure, thanks. I'll cover these predicates.
Also I've found interesting issue when I wanted to add test for predicate. It looks like is not handled or maybe I did something wrong, I couldn't find check. According following comments it should fail because AWS EBS forbids any two pods mounting the same volume ID : Test case: Any thoughts? is it correct behavior?


Question: } } <del> func TestSetRemove(t *testing.T) { evictor := NewRateLimitedTimedQueue(flowcontrol.NewFakeAlwaysRateLimiter()) evictor.Add("first", "11111") evictor.SetRemove("first") if evictor.queue.set.Len() != 0 { t.Fatalf("SetRemove should remove element from the set.") } } </del> func TestSwapLimiter(t *testing.T) { evictor := NewRateLimitedTimedQueue(flowcontrol.NewFakeAlwaysRateLimiter()) fakeAlways := flowcontrol.NewFakeAlwaysRateLimiter()

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Nodelifecycle controller does not set NoExecute taint to NotReady node when kubelet has been stopped What you expected to happen: effect: NoExecute key: Nodelifecycle controller sets NoExecute taint to NotReady node when kubelet has been stopped How to reproduce it (as minimally and precisely as possible): systemctl stop kubelet Anything else we need to know?: I use default args to start kube-controller-manager. I see log from kube-controller-manager: Environment: Kubernetes version (use ): [root ~]# kubectl version Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.6", GitCommit:"", GitTreeState:"clean", BuildDate:"2020-07-15T16:58:53Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.6", GitCommit:"", GitTreeState:"clean", BuildDate:"2020-07-15T16:51:04Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: OS (e.g: ): [root ~]# cat /etc/os-release NAME="CentOS Linux" VERSION="8 (Core)" ID="centos" IDLIKE="rhel fedora" VERSIONID="8" PLATFORMID="platform:el8" PRETTYNAME="CentOS Linux 8 (Core)" ANSICOLOR="0;31" CPENAME="cpe:/o:centos:centos:8" HOMEURL="" BUGREPORTURL="" CENTOSMANTISBTPROJECT="CentOS-8" CENTOSMANTISBTPROJECTVERSION="8" REDHATSUPPORTPRODUCT="centos" REDHATSUPPORTPRODUCTVERSION="8" Kernel (e.g.


Question: } } <del> func TestSetRemove(t *testing.T) { evictor := NewRateLimitedTimedQueue(flowcontrol.NewFakeAlwaysRateLimiter()) evictor.Add("first", "11111") evictor.SetRemove("first") if evictor.queue.set.Len() != 0 { t.Fatalf("SetRemove should remove element from the set.") } } </del> func TestSwapLimiter(t *testing.T) { evictor := NewRateLimitedTimedQueue(flowcontrol.NewFakeAlwaysRateLimiter()) fakeAlways := flowcontrol.NewFakeAlwaysRateLimiter()

): [root ~]# uname -a Linux liabio 4.18.0-147.5.1.el81.x8664 SMP Wed Feb 5 02:00:39 UTC 2020 x8664 x8664 x8664 GNU/Linux Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig cluster-lifecycle /wg cluster-lifecycle
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig sig-cluster-lifecycle /wg sig-cluster-lifecycle /committee sig-cluster-lifecycle Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-sig cluster-lifecycle /sig node
Those labels are not set on the issue: <detailsIn response to : /remove-sig cluster-lifecycle /sig node </details
is NoExecute the expected taint for this condition?
- yes it is when running the taint-manager: evictions are usually safe eviction as we don't force delete pods
ref
I see the same, a node which has node condition set to Unknown, does not get an Unreachable taint with NoExecute effect. However I only see this on 1.19, works fine till 1.18.10 Till 1.18.10, such a node gets two Unreachable taints, one with effect NoExecute and one with effect NoSchedule. But on 1.19.3 it only gets the NoSchedule taint. I see that this is the only commit made in that area between 1.18 and 1.19: , could be related?
It totally breaks production readiness of all recent releases, yet doesn't seem to be regarded as a top priority issue. WTF?


Question: } func (c *dynamicResourceClient) Get(name string, opts metav1.GetOptions, subresources ...string) (*unstructured.Unstructured, error) { <ins> if len(name) == 0 { return nil, fmt.Errorf("name is required") } </ins> result := c.client.client.Get().AbsPath(append(c.makeURLSegments(name), subresources...)...).SpecificallyVersionedParams(&opts, dynamicParameterCodec, versionV1).Do() if err := result.Error(); err != nil { return nil, err

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Providing an empty to the dynamic client's method results in a panic. What you expected to happen: An error should be returned on empty . How to reproduce it (as minimally and precisely as possible): Call with Anything else we need to know?: The expects a nonempty to be provided, but does not validate this. If an empty name is provided, this method will panic because it expects an , but the response will actually be an . Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Others: client-go 10.0.0
/sig api-machinery


Question: // CheckServiceAffinityPred defines the name of predicate checkServiceAffinity. CheckServiceAffinityPred = "CheckServiceAffinity" // MaxEBSVolumeCountPred defines the name of predicate MaxEBSVolumeCount. <ins> // DEPRECATED // All cloudprovider specific predicates are deprecated in favour of MaxCSIVolumeCountPred. </ins> MaxEBSVolumeCountPred = "MaxEBSVolumeCount" // MaxGCEPDVolumeCountPred defines the name of predicate MaxGCEPDVolumeCount. <ins> // DEPRECATED // All cloudprovider specific predicates are deprecated in favour of MaxCSIVolumeCountPred. </ins> MaxGCEPDVolumeCountPred = "MaxGCEPDVolumeCount" // MaxAzureDiskVolumeCountPred defines the name of predicate MaxAzureDiskVolumeCount. <ins> // DEPRECATED // All cloudprovider specific predicates are deprecated in favour of MaxCSIVolumeCountPred. </ins> MaxAzureDiskVolumeCountPred = "MaxAzureDiskVolumeCount" // MaxCinderVolumeCountPred defines the name of predicate MaxCinderDiskVolumeCount. <ins> // DEPRECATED // All cloudprovider specific predicates are deprecated in favour of MaxCSIVolumeCountPred. </ins> MaxCinderVolumeCountPred = "MaxCinderVolumeCount" // MaxCSIVolumeCountPred defines the predicate that decides how many CSI volumes should be attached MaxCSIVolumeCountPred = "MaxCSIVolumeCountPred"

We should deprecate existing hardcoded cloudprovider specific volume limit predicates in favour of volume limits coming from CSI and node. We can replace all of the three predicates with CSI predicate. But that needs to take into account migration path for in-tree volumes. /sig storage /sig scheduling
/assign


Question: } func (nsu *nodeStatusUpdater) UpdateNodeStatuses() error { <ins> // TODO: investigate right behavior if nodeName is empty // kubernetes/kubernetes/issues/37777 </ins> nodesToUpdate := nsu.actualStateOfWorld.GetVolumesToReportAttached() for nodeName, attachedVolumes := range nodesToUpdate { nodeObj, exists, err := nsu.nodeInformer.GetStore().GetByKey(string(nodeName))

Is this a BUG REPORT or FEATURE REQUEST? BUG REPORT Environment: Cloud provider or hardware configuration: GCE OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: What happened: $ kubectl describe pods test-ipj93 Name: test-ipj93 Namespace: default Node: kubernetes-minion-group-yxra/10.128.0.3 Start Time: Mon, 28 Nov 2016 14:39:28 -0800 Labels: <noneStatus: Pending IP: Controllers: <noneContainers: busybox: Container ID: Image: busybox Image ID: Port: Command: sh -c while true; do echo (hostname)-(cat /proc/stat cut -f 4 -d" "); sleep 30; done QoS Tier: cpu: Burstable memory: BestEffort Requests: cpu: 100m State: Waiting Reason: ContainerCreating Ready: False Restart Count: 0 Environment Variables: Conditions: Type Status Initialized True Ready False PodScheduled True Volumes: data: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: test ReadOnly: false default-token-kjoqg: Type: Secret (a volume populated by a Secret) SecretName: default-token-kjoqg Events: FirstSeen LastSeen Count From SubobjectPath Type Reason Message --------- -------- ----- ---- ------------- -------- ------ ------- 57m 57m 1 {default-scheduler } Normal Scheduled Successfully assigned test-ipj93 to kubernetes-minion-group-yxra 57m 57m 1 {kubelet kubernetes-minion-group-yxra} spec.containers{busybox} Normal Pulling pulling image "busybox" 57m 57m 1 {kubelet kubernetes-minion-group-yxra} spec.containers{busybox} Normal Pulled Successfully pulled image "busybox" 57m 57m 1 {kubelet kubernetes-minion-group-yxra} spec.containers{busybox} Normal Created Created container with docker id ; Security:[seccomp=unconfined] 57m 57m 1 {kubelet kubernetes-minion-group-yxra} spec.containers{busybox} Normal Started Started container with docker id 47m 47m 6 {kubelet kubernetes-minion-group-yxra} Warning FailedMount MountVolume.


Question: } func (nsu *nodeStatusUpdater) UpdateNodeStatuses() error { <ins> // TODO: investigate right behavior if nodeName is empty // kubernetes/kubernetes/issues/37777 </ins> nodesToUpdate := nsu.actualStateOfWorld.GetVolumesToReportAttached() for nodeName, attachedVolumes := range nodesToUpdate { nodeObj, exists, err := nsu.nodeInformer.GetStore().GetByKey(string(nodeName))

SetUp failed for volume "" (spec.Name: "default-token-kjoqg") pod "-b5bb-11e6-bab0-" (UID: "-b5bb-11e6-bab0-") with: Get https://kubernetes-master/api/v1/namespaces/default/secrets/default-token-kjoqg: dial tcp: lookup kubernetes-master on 169.254.169.254:53: no such host 46m 46m 1 {kubelet kubernetes-minion-group-yxra} Warning FailedMount MountVolume.SetUp failed for volume "" (spec.Name: "default-token-kjoqg") pod "-b5bb-11e6-bab0-" (UID: "-b5bb-11e6-bab0-") with: Get https://kubernetes-master/api/v1/namespaces/default/secrets/default-token-kjoqg: dial tcp 10.128.0.2:443: i/o timeout 45m 45m 1 {kubelet kubernetes-minion-group-yxra} Warning FailedMount MountVolume.SetUp failed for volume "" (spec.Name: "default-token-kjoqg") pod "-b5bb-11e6-bab0-" (UID: "-b5bb-11e6-bab0-") with: Get https://kubernetes-master/api/v1/namespaces/default/secrets/default-token-kjoqg: dial tcp 10.128.0.2:443: getsockopt: connection refused 11m 11m 2 {controller-manager } Warning FailedMount Failed to attach volume "test" on node "kubernetes-minion-group-yxra" with: error getting instance "kubernetes-minion-group-yxra" 9m 9m 1 {kubelet kubernetes-minion-group-yxra} Warning FailedSync Error syncing pod, skipping: network is not ready: [Kubenet does not have netConfig. This is most likely due to lack of PodCIDR] 9m 9m 2 {kubelet kubernetes-minion-group-yxra} Warning FailedSync Error syncing pod, skipping: network is not ready: [Kubenet does not have netConfig.


Question: } func (nsu *nodeStatusUpdater) UpdateNodeStatuses() error { <ins> // TODO: investigate right behavior if nodeName is empty // kubernetes/kubernetes/issues/37777 </ins> nodesToUpdate := nsu.actualStateOfWorld.GetVolumesToReportAttached() for nodeName, attachedVolumes := range nodesToUpdate { nodeObj, exists, err := nsu.nodeInformer.GetStore().GetByKey(string(nodeName))

This is most likely due to lack of PodCIDR] 6m 14s 4 {kubelet kubernetes-minion-group-yxra} Warning FailedMount Unable to mount volumes for pod "test-ipj93_default(-b5bb-11e6-bab0-)": timeout expired waiting for volumes to attach/mount for pod "test-ipj93"/"default". list of unattached/unmounted volumes=[data] 6m 14s 4 {kubelet kubernetes-minion-group-yxra} Warning FailedSync Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod "test-ipj93"/"default". list of unattached/unmounted volumes=[data] What you expected to happen: GCE PD to attach and mount successfully post the upgrade How to reproduce it (as minimally and precisely as possible): Running the upgrade steps in the doc for upgrade of manual test from 1.4 to 1.5 Anything else do we need to know:
Ran the upgrade again. This time the .
kube-controller-
Current theory is because kubelet deletes and recreates the node API object during upgrade (because externalID changed), see the new object does not contain the correct status (it is written by the Attach/Detach controller to the old object but not subseqently updated). If this proves to be the issue, the proposed fix is to force the attach/detach controller to update the node status when the node's external ID changes. is working to verify this theory by executing this code path (without a full upgrade). Once he has validated the theory, he will work on the fix, verify it, merge it to master, and get it cherry-picked to 1.5.
: is expected to fix this problem?
: Yes


Question: framework "k8s.io/kubernetes/pkg/scheduler/framework/v1alpha1" "k8s.io/kubernetes/plugin/pkg/admission/priority" testutils "k8s.io/kubernetes/test/integration/util" <del> utils "k8s.io/kubernetes/test/utils" </del> <ins> "k8s.io/kubernetes/test/utils" </ins> ) var lowPriority, mediumPriority, highPriority = int32(100), int32(200), int32(300)

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: Which test(s) are flaking: Testgrid link: Reason for failure: Unknown yet. Anything else we need to know: links to appreciated links to specific failures in spyglass appreciated /sig scheduling /help
Can I give it a try?
Sure, feel free to take it. /assign ahmad-diaa /remove-help
I keep trying to get it fail locally by running the following command: but it never does, am I doing something wrong? Is there something else that I should be doing and would be more helpful?
Sometimes the flake is hard to reproduce in local env. You may have to read the error log of the flakes thoroughly to get some clues. BTW: you can also run the test multiple times using the directive:
After looking through the logs I came up with some information that might be useful As far as I understand, the reason for the flake is the of the medium priority pod not being cleared That is because the update request for removing returns the following error: So my current guess is that might be outdated in some cases As I am not really familiar with the logic, I would like to know whether or not those are valid points so that I could dig deeper. What do you think?
If that's the case, you can set here: You can raise a PR and manually try for ~20 times to see you can reproduce the flake.
found another issue tracking this too. it seems it has been flaky for a while.
Sorry, didn't mean to take this one for you. I worked the duplicate issue and submitted a PR. Feel free to review that PR, or submit your own if you have a different solution.
This sometimes happens. But we usually take the first PR unless it's clearly worked by someone. In terms of which style to resolve the Conflict error, I'm asking here:
Sorry, didn't mean to take this one for you. I worked the duplicate issue and submitted a PR. Feel free to review that PR, or submit your own if you have a different solution. No worries, I haven't started implementing yet and your PR seems to be following the correct approach as well according to this:


Question: framework "k8s.io/kubernetes/pkg/scheduler/framework/v1alpha1" "k8s.io/kubernetes/plugin/pkg/admission/priority" testutils "k8s.io/kubernetes/test/integration/util" <del> utils "k8s.io/kubernetes/test/utils" </del> <ins> "k8s.io/kubernetes/test/utils" </ins> ) var lowPriority, mediumPriority, highPriority = int32(100), int32(200), int32(300)

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: pull-kubernetes-integration Which test(s) are flaking: TestNominatedNodeCleanUp Testgrid link: Reason for failure: [TODO] Anything else we need to know: links to appreciated links to specific failures in spyglass appreciated
/sig node These SIGs are my best guesses for this issue. Please comment if I am incorrect about one. I am a bot run by vllry.


Question: framework "k8s.io/kubernetes/pkg/scheduler/framework/v1alpha1" "k8s.io/kubernetes/plugin/pkg/admission/priority" testutils "k8s.io/kubernetes/test/integration/util" <del> utils "k8s.io/kubernetes/test/utils" </del> <ins> "k8s.io/kubernetes/test/utils" </ins> ) var lowPriority, mediumPriority, highPriority = int32(100), int32(200), int32(300)

Which jobs are flaking: pull-kubernetes-integration Which test(s) are flaking: TestNominatedNodeCleanUp Testgrid link: Reason for failure: /sig scheduling
/assign
any progress on this one?
I was working on it and tried to find the root cause. I will give my solution within this week
Another one
Still happening cc
/assign
/close dupe
Closing this issue. <detailsIn response to : /close dupe Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: "If the file specified by --kubeconfig does not exist, the bootstrap kubeconfig is used to request a client certificate from the API server. "+ "On success, a kubeconfig file referencing the generated client certificate and key is written to the path specified by --kubeconfig. "+ "The client certificate and key file will be stored in the directory pointed by --cert-dir.") <del> fs.BoolVar(&f.RotateCertificates, "rotate-certificates", f.RotateCertificates, "<Warning: Beta feature> Auto rotate the kubelet client certificates by requesting new certificates from the kube-apiserver when the certificate expiration approaches.") </del> fs.BoolVar(&f.ReallyCrashForTesting, "really-crash-for-testing", f.ReallyCrashForTesting, "If true, when panics occur crash. Intended for testing.") fs.Float64Var(&f.ChaosChance, "chaos-chance", f.ChaosChance, "If > 0.0, introduce random client errors and latency. Intended for testing.")

/kind cleanup What happened: and both require opt-in configuration knobs to the kubelet. They are kinda global settings, you probably either opt into dynamic cert rotation or you don't, so they qualify for being in the struct. The knob for enabling dynamic serving certs was (I fully support that :+1:). What you expected to happen: I'd like the same to happen to  to be consistent. (Also, we're planning to use the config file for the kubelet by default in kubeadm soon, and it'd be nice to be able to set this value then in a structured manner) /assign


Question: "If the file specified by --kubeconfig does not exist, the bootstrap kubeconfig is used to request a client certificate from the API server. "+ "On success, a kubeconfig file referencing the generated client certificate and key is written to the path specified by --kubeconfig. "+ "The client certificate and key file will be stored in the directory pointed by --cert-dir.") <del> fs.BoolVar(&f.RotateCertificates, "rotate-certificates", f.RotateCertificates, "<Warning: Beta feature> Auto rotate the kubelet client certificates by requesting new certificates from the kube-apiserver when the certificate expiration approaches.") </del> fs.BoolVar(&f.ReallyCrashForTesting, "really-crash-for-testing", f.ReallyCrashForTesting, "If true, when panics occur crash. Intended for testing.") fs.Float64Var(&f.ChaosChance, "chaos-chance", f.ChaosChance, "If > 0.0, introduce random client errors and latency. Intended for testing.")

Flag name: Help text: <Warning: Beta featureAuto rotate the kubelet client certificates by requesting new certificates from the kube-apiserver when the certificate expiration approaches. This is part of migrating the Kubelet command-line to a Kubernetes-style API. The flag should either be migrated to the Kubelet's API group, or simply removed from the Kubelet. If this could be considered an instance-specific flag, or a descriptor of local topology managed by the Kubelet, see: If this flag is only registered in os-specific builds, see: /assign /sig node /kind feature /priority important-soon /milestone v1.11 /status approved-for-milestone
[MILESTONENOTIFIER] Milestone Issue: Up-to-date for process <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. : New functionality. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details


Question: secretNamespace = v case "resourcegroup": resourceGroup = v <ins> case "sharename": shareName = v </ins> default: return nil, fmt.Errorf("invalid option %q for volume plugin %s", k, a.plugin.GetPluginName()) }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: use an existing azure file share name should be possible in azure file plugin What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /sig azure /assign


Question: "fmt" "io/ioutil" "os" <ins> "os/exec" </ins> "reflect" "strings" "sync"

A status of Pod mounted NFS PV remain as terminating when a node shutdown and restart. A Pod is completely terminated. Deploy a Pod mounted NFS PV Shutdown the node where the Pod is running Wait for 5 min and the Pod status become Restart the node The Pod status is still and the Pod is never terminated completely <detailsPrepare a NFS Server.(I used ) Deploy a Pod mounted NFS PV like this. Shutdown the node where the Pod is running.(In this case ) And the status of node02 becomes . After 5 min later(by default), the Pod status becomes . (The time a Pod becomes status depends on .) Then, restart the node and the status of node02 becomes . But the Pod status is still and the Pod is never terminated completely. It seems that a faild to delete Pod because of trying to unmount a not mounted directory. Force delete Pod. Delete Pod's directory on the node. </detailsNo response <details</details<detailsnone(on premise) </details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage
/sig node
It dose not happen using , so it seems to depend on NFS volume plugin.
/assign
I try to build and run kubelet with comment out this logic. Then it seems to get well. related: This change checks the result of instead of the mount point check. However, check logic is not implemented in and (these functions are used in NFS volume plugin), despite it is done in .
/remove-sig node /sig storage
Reopening, fix had to be reverted due to unit tests requiring root permissions ()
I am terribly sorry... Please wait a moment. I'll try to fix it.
no worries, just get to review the rework, making sure the units tests don't require root permissions
Thank you! Fixed in .
Are there any plans to backport this to older Kubernetes versions? This seems like a critical fix.


Question: package stats import ( <ins> "fmt" </ins> "time" "github.com/Microsoft/hcsshim"

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: 's memory usage is increasing over time. When more pods are run, it increases even more. What you expected to happen: it should not keep increasing How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): 1.14.0 Cloud provider or hardware configuration: AKS OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: !
/sig windows /sig azure
/assign


Question: // we modify it when the pause container is created since it is the first container created in the pod since it holds // the networking namespace. if container.Name == PodInfraContainerName { <del> if err := rewriteResolvFile(containerInfo.ResolvConfPath, opts.DNS, opts.DNSSearch, useClusterFirstPolicy); err != nil { </del> <ins> if err := RewriteResolvFile(containerInfo.ResolvConfPath, opts.DNS, opts.DNSSearch, useClusterFirstPolicy); err != nil { </ins> return kubecontainer.ContainerID{}, err } }

Kubernetes version (use ): HEAD (~v1.6.0-beta.4) Environment: any What happened: I want to use ClusterFirstWithHostNet which was implemented in , but only dockertools (legacy/non-CRI) was updated to support this feature. The CRI dockershim should be modified to support this rewriting the file of Pods with HostNetwork=true as well. This is a bug in v1.6, I hope it can be prioritized to be fixed in v1.6.1, since it's blocking me from using CRI. Since CRI is enabled by default, it's expected that it should have feature parity with the earlier implementation.
Related
- I guess I didn't realize these crossed paths. Do you have time to take a look?
Will take a look
Thanks! Ping me directly on this.
Filed for supporting it in CRI.


Question: statsProvider: statsProvider, recorder:      recorder, nodeRef:       nodeRef, <del> initialized:   false, </del> tracer:        tracer, }

when kubelet restart,all images which notused cleaned. but there is a flag in kubelet,,explainMinimum age for an unused image before it is garbage collected. Examples: '300ms', '10s' or '2h45m'. i set to ,it's means if a image not used,it will live 20m and then clean by kubeletif has disk pressure.if kubelet not restart, it work,but if restart ther is a bug, not work,image will be cleaneven if it's no enough time. work, image even if not used,it will live 20m and then clean by kubelet. 1make node trigger disk pressure. 2pull some image which not used by pod 3restart kubelet 4image cleaned by kubelet after around 10s i see the source code, and maybe there is a bug. in func,well detect image and update firstdetect time for every image. and in func,well check images livetime by firstDetected. if kubelet running normal,it work,because there is a func. in this line,,in first time,ts is zero. so image's firstDetected time is zero,and in freeSpace this check is always false,so image will be cleaned. maybe we can change to ,can fix this bug. <details</details<detailsNone </details<details</details<details</details<details</details<details</details
/sig node
/assign
/triage accepted /priority backlog initial assessment of priority, subject to change


Question: Image:   busyboxImage, Command: []string{"sh", "-c"}, Args: []string{` <del> _term() { echo "Caught SIGTERM signal!" while true; do sleep 5; done } trap _term SIGTERM while true; do sleep 5; done `}, </del> <ins> sleep 9999999 & PID=$! _term() { echo "Caught SIGTERM signal!" wait $PID } trap _term SIGTERM wait $PID `}, </ins> }, }, TerminationGracePeriodSeconds: &gracePeriod,

Followup feedback from In a few e2e node tests (e.g. tests of deletion grace period) we have some busybox pods that wait for SIGTERM and perform some action like a small sleep to simulate SIGTERM taking some time. These tests follow the pattern of -- see The will block the SIGTERM handler from being run until after the time is elapsed. We should consider to remove the and instead use to avoid blocking the SIGTERM handler, see pattern followed in n/a n/a No response n/a n/a <details</details<details</details<details</details<details</details
/sig node /cc
/assign
/triage accepted


Question: "github.com/onsi/ginkgo" "github.com/onsi/ginkgo/config" "github.com/onsi/ginkgo/reporters" <del> "github.com/onsi/ginkgo/types" </del> "github.com/onsi/gomega" )

We for uploading to GCS on , but not on successful e2e runs or failed test -up. We should definitely be dumping those logs for failed cluster-ups, and it would also be nice to throw in the successes too (good to have recent green for comparison).
Getting logs uploaded on a failed test -up seems really important for debugging.
+1 There were just a bunch of failures in scalability test suite (run 1623-1628) that failed because not all system pods were running. Since there are no logs, there is no easy way to debug it. It would be super useful to have all the logs for failed cluster-ups. cc
Unless I'm confused, it looks as if these logs are still not being uploaded. FYI.
For example: gs://kubernetes-jenkins/logs/kubernetes-e2e-gce/7500
That particular build was a timeout. When Jenkins kills the job, the log-collecting code never runs.
Another case we won't collect logs is if the cluster fails to initialize. (Which is a really important place to collect logs, but it's not a trivial fix.)
Moving this into post, since it seems unlikely to get fixed in the near term. feel free to correct me if I'm wrong.
We also don't collect logs on test job timeouts, which is frustrating.
+1
will collect the logs when the cluster fails to initialize. That's not all the cases listed here, but it's a good one.
I'd like to bump the priority for this further - we don't have any logs to help with debugging , which started to happen more often recently - it seems that didn't help. cc
I have too many P0s assigned to me already. This is at least 3rd in line right now.
This one is half year old and is P0 for a month...
From discussion with on hangouts, a vague plan: move log-dumping logic from into a standalone command in something like (feel free to bikeshed the name) update Jenkins to call this command as a post-build step probably remove and the debug stuff recently to . Right now the stuff is pretty GCE/GKE specific and uses ssh. We should probably change to a more provider-agnostic method, but this would be a good start.
Hm, actually, post-build step actually won't work, since we tear down the cluster first.


Question: "github.com/onsi/ginkgo" "github.com/onsi/ginkgo/config" "github.com/onsi/ginkgo/reporters" <del> "github.com/onsi/ginkgo/types" </del> "github.com/onsi/gomega" )

But making it a separate command can still let us call it in more places.
FYI - we also need logs from timed-out runs (not only the failed ones).
moving out into its own binary should help accomplish this.
I'm worried that the best way to do this is to write a bash script that grabs IPs from 's and functions and just does an ssh and cats the logs. Then we could point at that. It's sorta difficult to make work on its own.
I think at this point we're happy with anything that usually works. We can work on making in nice and shiny later.
We now get logs when kube-up fails, which recently happened :) Still some work to do here, I'll leave this issue open until timeouts are handled properly and points here.
- I can't thank you enough!
Ok, is in so we should get logs for kubernetes-e2e-* jobs and PR Jenkins jobs on timeout.
I'm gonna call this done.


Question: scheme *runtime.Scheme codecs serializer.CodecFactory <ins> // hostnameOverride, if set from the command line flag, takes precedence over the `HostnameOverride` value from the config file hostnameOverride string </ins> } // AddFlags adds flags to fs and binds them to options.

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: kube-proxy came up cleanly but mis-configured iptable rules due to a mis-match in the hostname and node name within the cluster triggering a default behavior. In this case the hostname returned the fqdn of the host and the cluster only had the hostname. This resulted in the following warning messages when kube-proxy started. Which then presents itself as pods not being able to access clusterip ip addresses with no route errors. What you expected to happen: Would it be better to fail to in this scenario rather than try to fall back to some default? I spent a good amount of time trying to track down the Error messages that were a result of this before finally working back to taking a closer at the Warning message. How to reproduce it (as minimally and precisely as possible): Start the kube-proxy with set to an invalid hostname should recreate this issue. Anything else we need to know?: While I think we can chalk up this issue to mis-configuration the purpose of opening this issue is really to have a conversation about the default behavior. I've seen several other issues filed that appear to be related to this behavior: , , . Would throwing an error and exiting non-zero be a better default action when the nodeIP is nil? What use case(s) does a default of support? If there must be a default behavior here maybe the IP address on the interface associated with the default gateway is a better default than ? Maybe the issue isn't in the proxier code but the kube-proxy itself should error when it fails to get node info rather than a warning? Environment: I'm not sure how much of this is important to this but here's what my environment was when I ran into this. Kubernetes version: 1.8.5 Cloud provider or hardware configuration: Bare metal OS (e.g. from /etc/os-release): Centos 7 and Ubuntu 16.04
This is similar (though not the same symptom) to for which was supposed to fix it but was closed. One of the objections was up til now, kube-proxy has had no requirement that the hostname = nodename or even that the hostname be defined.


Question: scheme *runtime.Scheme codecs serializer.CodecFactory <ins> // hostnameOverride, if set from the command line flag, takes precedence over the `HostnameOverride` value from the config file hostnameOverride string </ins> } // AddFlags adds flags to fs and binds them to options.

Making kube-proxy exit on failure to retrieve the node object might cause regressions in some legitimate use-cases. But I have no idea what those are. ?
I have a hard time imagining what the legit cases might be. I am confused, though - I only see this field being used in one very specific place, that would not have the effect you are claiming.
I can't see the error message from log? If kube-proxy failed to create kube-ipvs0 dummy interface, it will log something like "Failed to create dummy interface: kube-ipvs0, error:...".
This breaks Services of type and with . This is problematic for AWS environments where the kubelet is configured to override the hostname (AWS instances default to the short name and for the cloud provider integrations to work this needs to be overridden to the private fqdn of the host).
Any chance we could get this prioritized as a bug fix for v1.11?
cc
After putting a bit more thought into this, would it be possible to standardize on an environment variable that could be set through the downward api to override the default behavior of treating the hostname as the node name in the proxy? That would simplify the configuration on the kubeadm side (no need to override the command with the downward api, only setting the appropriate env variable), and would also provide a more accurate way to retrieve the actual node name for kube-proxy when deployed as a daemonset without the need to interact with the kubelet directly. I would suggest that we could use for this purpose. I'd also be happy to take a stab at implementing the changes needed for the v1.12 cycle.


Question: scheme *runtime.Scheme codecs serializer.CodecFactory <ins> // hostnameOverride, if set from the command line flag, takes precedence over the `HostnameOverride` value from the config file hostnameOverride string </ins> } // AddFlags adds flags to fs and binds them to options.

Pods not getting scheduled on the new node joined to the cluster using kubespray The node joined to the cluster but new pods schedueld on the nodes are failing with error "get https://10.233.0.1 : dial tcp 10.233.0.1: 443 connection refused" Network plugin - calico kube-proxy running in ipvs mode new worker node, can connect to master server ip address over 6443 new worker node, cannot access cluster ip 10.233.0.1 over port 443 , nodelocaldns and kube-proxy pods are in running state on new node -proxy logs show error "invalid nodeIP, initializing kube-proxy with 127.0.0.1 as nodeIP" Unable to write event: 'Post https://127.0.0.1:6443/api/v1/namespaces/default/events dial tcp 127.0.0.1:6443 connection refused" -ln output do not show any services addr show output do not show kube-ipvs0 and calico interfaces Can someone please help. Thanks


Question: - user: root - group: root - mode: 644 <ins> /etc/monit/conf.d/kube-proxy: file: - managed - source: salt://monit/kube-proxy - user: root - group: root - mode: 644 </ins> {% endif %} monit-service: service: - running <del> - name: monit </del> <ins> - name: monit </ins> - watch: <del> - pkg: monit </del> <ins> - pkg: monit </ins> - file: /etc/monit/conf.d/* {% endif %}

We already "healthcheck" kubelet and docker periodically, but miss one for kube-proxy.
It would make sense to do (only bind to localhost) at the same time we add the healthchecking configuration.


Question: import ( "context" <ins> "fmt" </ins> "path/filepath" "regexp" "time"

in W0128 10:46:06.683] [FAIL] [sig-node] Device Plugin [Feature:DevicePluginProbe][NodeFeature:DevicePluginProbe][Serial] DevicePlugin [Serial] [Disruptive] [BeforeEach] Can schedule a pod that requires a device The pod was left by the test case not sure No response No response_ /sig node
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc for (revert of )
these tests started failing after got merged?
The test passed after rebasing with the latest code. It should be fixed. Thanks.
It failed for some other remaining pods again.
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
It is still an issue, see It seems that some pods are sometimes not being cleaned up after the test finishes...
Taking a look at Test fails with On kubelet side it was deleted here: The logs show that API server recorded a delete of the pod at but the next test was already running at which is before the pod was deleted. It seems like the test that created namespace (which is ), completed before it deleted all the resources created in the test... edit: The namespace was deleted here: The namespace deletion in the test framework looks async, i.e. I think it will not block to wait until the namespace is deleted... not sure if this is by design or not...
See related slack thread - Namespace deletion is async by design and the test needs to account for this. For this case in particular, since requires no pods to be left over from other tests, we can adjust the BeforeBegin to wait and poll for a bit while all the resources are cleaned up.


Question: if node.ObjectMeta.Annotations == nil { node.ObjectMeta.Annotations = make(map[string]string) } <del> node.ObjectMeta.Annotations[cloudproviderapi.AnnotationAlphaProvidedIPAddr] = nodeIP.String() </del> <ins> annotation := nodeIP.String() if secondaryNodeIPSpecified { annotation += "," + secondaryNodeIP.String() } node.ObjectMeta.Annotations[cloudproviderapi.AnnotationAlphaProvidedIPAddr] = annotation </ins> } else if node.ObjectMeta.Annotations != nil { // Clean up stale annotations if no longer using a cloud provider or // no longer overriding node IP.

When adding to kubelet and to the OpenStack external cloud provider, which is based on kubernetes v1.27.1, the node annotation only shows one address and consequently the node addresses also only displays one address. The kubelet logs shows that node-ip contains dual-stack addresses. The annotation on the node would contain both IPv4 and IPv6 addresses and the node addresses would list them. Deploy a cluster with to kubelet and external cloud provider. Also make sure the external cloud provider is based on Kubernetes 1.27. Kubelet logs: one of CCMs Pods: one of the nodes annotation: <details</details<details</details<details</details<details</details<details</details<details</details
/sig network /sig cloud-provider /cc
/assign
/triage accepted


Question: apierrors "k8s.io/apimachinery/pkg/api/errors" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/runtime/schema" <ins> "k8s.io/apimachinery/pkg/types" utilrand "k8s.io/apimachinery/pkg/util/rand" </ins> "k8s.io/apimachinery/pkg/util/wait" <ins> "k8s.io/apimachinery/pkg/watch" </ins> applyadmissionregistrationv1 "k8s.io/client-go/applyconfigurations/admissionregistration/v1" clientset "k8s.io/client-go/kubernetes" "k8s.io/client-go/openapi3" <ins> "k8s.io/client-go/util/retry" </ins> "k8s.io/kubernetes/test/e2e/framework" admissionapi "k8s.io/pod-security-admission/api" )

from: <img width="902" alt="image" src=""from:
/kind failing-test /area conformance /sig api-machinery /assign cc
/triage accepted