Kubernetes e2e suite. Kubernetes e2e suite.[It] [sig-autoscaling] [Feature:HPA] Horizontal pod autoscaling (scale resource: CPU) [Serial] [Slow] ReplicationController Should scale from 1 pod to 3 pods and then from 3 pods to 5 pods and verify decision stability [sig-autoscaling, Feature:HPA, Serial, Slow] Kubernetes e2e suite.[It] [sig-autoscaling] [Feature:HPA] Horizontal pod autoscaling (scale resource: CPU) [Serial] [Slow] ReplicationController Should scale from 5 pods to 3 pods and then from 3 pods to 1 pod and verify decision stability [sig-autoscaling, Feature:HPA, Serial, Slow] Kubernetes e2e suite.[It] [sig-autoscaling] [Feature:HPA] [Serial] [Slow] Horizontal pod autoscaling (non-default behavior) with autoscaling disabled shouldn't scale down [sig-autoscaling, Feature:HPA, Serial, Slow] Kubernetes e2e suite.[It] [sig-autoscaling] [Feature:HPA] [Serial] [Slow] Horizontal pod autoscaling (non-default behavior) with autoscaling disabled shouldn't scale up [sig-autoscaling, Feature:HPA, Serial, Slow] Kubernetes e2e suite.[It] [sig-autoscaling] [Feature:HPA] [Serial] [Slow] Horizontal pod autoscaling (non-default behavior) with both scale up and down controls configured should keep recommendation within the range with stabilization window and pod limit rate [sig-autoscaling, Feature:HPA, Serial, Slow] Kubernetes e2e suite.


[It] [sig-autoscaling] [Feature:HPA] [Serial] [Slow] Horizontal pod autoscaling (non-default behavior) with both scale up and down controls configured should keep recommendation within the range over two stabilization windows [sig-autoscaling, Feature:HPA, Serial, Slow] 11/3/2023 <*fmt.wrapError | 0xc000812080: client rate limiter Wait returned an error: context deadline exceeded { msg: "client rate limiter Wait returned an error: context deadline exceeded", err: <context.deadlineExceededError{}, } looks similiar to /sig autoscaling /sig windows
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


/kind bug What happened: does not show the description for the new field in What you expected to happen: should describe the new field. How to reproduce it (as minimally and precisely as possible): Environment: Kubernetes version (use ): master
/sig auth
GitHub didn't allow me to assign the following users: fanzhangio. Note that only and repo collaborators can be assigned. <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
If nobody has started yet, I wanna to pick it up.
was working on it AFAIR.
Indeed. if you don't mind, I'd take over.


Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.5", GitCommit:"", GitTreeState:"clean", BuildDate:"2016-10-29T01:38:40Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.5", GitCommit:"", GitTreeState:"clean", BuildDate:"2016-10-29T01:32:42Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"} Environment: Cloud provider or hardware configuration: vSphere OS (e.g. from /etc/os-release):Kernel (e.g. ):Install tools: None Others: None What happened: When I create a deployment with vSphere cloud provider dynamic provisioning, it creates a volume spec with volume path - [datastore] kubevols/volume which contains a white space in it. In this case the cloud provider queries the getDeviceNameFromMount() to return the path of the volume mounted. Since getDeviceNameFromMount() queries the filesystem to get the mount references, it returns a volume path "[datastore]040kubevols/volume". Later the kubelet searches for this volume path in both the actual and desired states. Th actual and desired states contains volume with path "[datastore] kubevols/volume". So, it couldn't find such volume path and therefore kubernetes stalls unable to make any progress further similar to one described in .


Also similar issue described in What you expected to happen: All the pods in the recreated deployment comes up. How to reproduce it (as minimally and precisely as possible): See above. Anything else do we need to know: None
cc


I tried to create a kubernetes deployment with vSphere volume with volume path "[datastore] kubevols/redis-master". In this case the cloud provider queries the getDeviceNameFromMount() to return the path of the volume mounted. Since getDeviceNameFromMount() queries the filesystem to get the mount references, it returns a volume path "[datastore]040kubevols/redis-master". Later the kubelet searches for this volume path in both the actual and desired states. Th actual and desired states contains volume with path "[datastore] kubevols/redis-master". So, it couldn't find such volume path and therefore kubernetes stalls unable to make any progress further similar to one described in . However, when I create a kubernetes deployment with volume path - "[datastore]kubevols/redis-master" which doesn't contain any spaces, it works perfectly fine. So, the basic problem is to format and return the correct volume path from getDeviceNameFromMount() function. Alternatively, we can format it properly in the cloud provider code but this has to be done individually in all the cloud provider code base. Inorder to prevent it, it would be better to fix the getDeviceNameFromMount() function to return the correct formatted volume path.
CC Kubernetes should handle volumes with spaces (looks like vsphere is the first such volume) gracefully. Non a blocker issue for 1.5.
you should make sure that the 1.5 release notes capture that "vsphere volumes w/spaces have problems". Known issue accumulator in
getDeviceNameFromMount is function which queries /proc/mount to get mount points. 040 - is an escape sequence. 040 is the ascii code, in octal(base 8) number system, of the space. The mount command interprets 040 as literal space in the mount point. : Can't we provide a fix for this in 1.5 release itself? There a 2 options: can provide a fix in getDeviceNameFromMount. can provide a fix to vSphere cloud provider to fix it after the call to getDeviceNameFromMount. For option 1, we have to make sure it works for all cloud providers where as for option 2 its not needed. Let me know which option you think is feasible. I am ok with anything.


1 is the right answer. But we're very close to 1.5.0 release (<1 week). I'd also like whoever addresses this to ensure the entire system works end-to-end for volumes with spaces, not just patch one little corner. I'm afraid it would be a large change and we're not going to have enough time to validate it in 1.5. You can certainly start working on it. If you get a PR in time, we can consider it for 1.5.0. Otherwise it'll have to go in a patch release.


[x] Test PR : [x] Two weeks soak end date : 6 April 2022 [X] Test promotion PR : According to this APIsnoop query, there are still some remaining Job status endpoints which are untested. There are conformance tests for CronJob status endpoints already the existing as a template, a new for job test has been created. The e2e logs for this test are listed below. This query shows the endpoints hit within a short period of running the e2e test If a test with these calls gets merged, test coverage will go up by 3 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/triage accepted
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


It seems to be valid to add here. Would you like to open a pull request for it? this is like a cleanup, as the set insert will skip and the validation will do twice. is used to track bugs. I suggest you open a PR for that if you like.
/sig storage
I think adding ` == true and we didn't skip, it will insert set twice I believe? wasn't notice that at first glance. However, I can file a PR now
/triage accepted
The label cannot be applied. Only GitHub organization members can add the label. <detailsIn response to : /triage accepted Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/triage accepted
Please feel free to submit a PR.


<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: kube-cm kube-scheduler does not exit when kube-apiserver suspended What you expected to happen: kube-cm kube-scheduler should exit after LeaderElector stops leading How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig api-machinery /kind bug
/assign
/cc


xref In 1.16: change the metadata-proxy addon to use change the kube-proxy addon to use change the masq-agent addon to use stop setting , , and in kube-up
Hey I'm trying to start to contribute to kubernetes and this seem to be an easy task. Can I help with this one?


Public IPs are expensive resources. Given that we're running scalability tests on 5k-node clusters, we are unnecessarily consuming 5k IPs which is a lot. We should switch our tests so that nodes have only private IP addresses. We are going to do that by configuring NAT so that nodes can talk to outside world (e.g. to download image from dockerhub). But that also requires changing some correctness tests which assume public IPs.
To summarize recent work: setting up private gce clusters with nat, feature is gated via the env test and dashboard with correctness tests on gce private cluster - https://k8s- 27 tests fail on fixing the tests - should fix majority of them
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale /lifecycle frozen
- is there anything missing to be done here?
No, we can close this one. It was done long time ago. Thanks for catching /close
Closing this issue. <detailsIn response to : No, we can close this one. It was done long time ago. Thanks for catching Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: Used the endpoint /BumpMetric to create fake metrics for Prometheus. When I tried to curl /metrics I got the following response: Bad request. Not a POST request What you expected to happen: I expected to see something like this: # HELP info message. # TYPE gauge 0. How to reproduce it (as minimally and precisely as possible): Following the docs, after having it running inside kubernetes: Anything else we need to know?: Looks like introduced this by mistake. Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
/sig testing


Currently it assumes it can read ContainerManifest - it should try both
Why not hold on this until we have Kubelet-read-Pod in?
Also the HTTP source has the same problem.
It should read Pod. Changed the title.
Ideally, I'd like all Pods created directly via Kubelet to be to etcd and surfaced via the API. If updates are made via the API, I'd like those to propagate back to the node. We can discuss whether the config file should be updated or ignored. At some point, I'd like Kubelet to persistently cache pods received from the apiserver, perhaps by just writing them all out as config files.
is the issue to sync local pods back to apiserver.
this is somewhat related to other issues we talked about.
Detail: We want locally specified pods to be able to use YAML config files, but we also want to rip YAML out of the API server . The config-file behavior should be as similar to the kubectl experience. Kubelet could use parts of the kubectl library to translate YAML to JSON. Vision: Eventually, I'd like Kubelet to use the apiserver library and support exactly the same Pod API as the master, at which time I'd like kubectl be able to create pods directly on Kubelets, as well as reading from config files. I also expect to have a per-node controller . I'd like to be able to use the per-node controller to update pods started via config files (or direct kubectl-kubelet pods), overriding those original files. For availability and self-hosting, we'll eventually need local checkpointing , ideally just using serialized JSON for each object. If Kubelet used apiserver, this could be done using a custom registry implementation.
I think that is a prerequisite for it.
If noone picks it up, I'm going to start working on it tomorrow.
thanks, wojtek-t
fixed it for http channel


<!-- Please only use this template for submitting enhancement requests --What would you like to be : Actually, the parameter depends to all namespaces in the cluster. Filtering is only possible with labels. When running the should only depend on the objects defined in . Why is this needed: A typical use-case is having one repository and namespace per project. When working with declarative objects it should be possible to remove objects and get them pruned like having them described in the respository. This would also help when renaming objects within the repository - which actually needs a manual on the host for the old objects. This even could be a better solution for the problem described in
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig api-machinery
cc
/assign
GitHub didn't allow me to assign the following users: Liujingfang1. Note that only and repo collaborators can be assigned and that issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
We have been aware of the issues in prune and we want to improve it. Recently we opened a for improving pruning in . With the proposed solution, the problem from current will be fixed.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or .


/lifecycle stale
/remove-lifecycle stale
I just deleted a bunch of things in another team's namespace (:blush:). I'd would love to have this feature. I had been using dry-run in another context, but then forgot to check it again after switching contexts.
Is there any workaround for this? I want to prune resources within a single namespace only.
We're still working on improving prune, the current mechanism is broken enough that we don't recommend people us it.
Also look like it's been fixed recently.
Can someone confirm if this is implemented? I tried using this feature with kubectl 1.18.2, which has , yet as with previous versions kubectl it returned an error about pruning a nonNamespaced object of kind Namespace.
I assume the problem is that "Namespace" is a non-namespaced resource, and therefore is not filtered by namespace, so gets deleted by "prune". As a workaround it's possible to manually pass a (whitelist of resources allowed to be deleted), and not include "Namespace" in it: The list of resources is taken from .
Would wrapping this loop with achieve the same result?


Which jobs are failing: pull-kubernetes-e2e-gce Which test(s) are failing: [sig-network] Services should have session affinity work for NodePort service [LinuxOnly] [sig-network] Services should be able to switch session affinity for service with type clusterIP [LinuxOnly] [sig-network] Services should be able to switch session affinity for NodePort service [LinuxOnly] [sig-network] Services should have session affinity work for service with type clusterIP [LinuxOnly] Since when has it been failing: At least since 9/30 (). But that's almost at the end of what's currently visible on the testgrid, so maybe longer Testgrid link: Not sure what I'm supposed to put here. ? Reason for failure: The test case connects to a service every 2 seconds for 1 minute and checks if it hit the same endpoint at least 15 times in a row. But the individual checks are taking unexpectedly long and so sometimes the test doesn't even manage to run 15 times total in 1 minute.


eg: Oct 8 13:32:01.402: INFO: Running ' --server=https://34.82.143.14 -- exec --namespace=services-1060 execpod-affinityklpfn -- /bin/sh -x -c curl -q -s --connect-timeout 2 http://10.40.0.5:/' Oct 8 13:32:07.627: INFO: stderr: "+ curl -q -s --connect-timeout 2 http://10.40.0.5:/n" Oct 8 13:32:07.627: INFO: stdout: "affinity-nodeport-2z9hv" Oct 8 13:32:07.627: INFO: Received response from host: affinity-nodeport-2z9hv Oct 8 13:32:09.402: INFO: Running ' --server=https://34.82.143.14 -- exec --namespace=services-1060 execpod-affinityklpfn -- /bin/sh -x -c curl -q -s --connect-timeout 2 http://10.40.0.5:/' Oct 8 13:32:16.027: INFO: stderr: "+ curl -q -s --connect-timeout 2 http://10.40.0.5:/n" Oct 8 13:32:16.027: INFO: stdout: "affinity-nodeport-2z9hv" Oct 8 13:32:16.027: INFO: Received response from host: affinity-nodeport-2z9hv Note the gap between the "Running" line and the "stderr" line. It's not clear if the slowness is in connecting to the execpod or in connecting from the execpod to the affinity service. It is also not particularly consistent between attempts. (Here it was 6s, 1s, 2s, 3s, 5s, 7s, 7s, 3s, 4s, 6s, 7s, 8s.) Anything else we need to know: It looks the checks are also slow in successful runs, it's just that they're not quite as slow, so they pass.


I'm not sure if the problem is "something is making the cluster slow and so this totally valid test is taking too long and failing" or "e2e-gce has grown over time and now cluster load has reached the point where this formerly-passing test now has unreasonable performance expectations". /sig network


/kind bug What happened: walteraa [6:55 AM] Hi, The controller manager of the federation using Google cloud DNS is crashing, the log showed is that: the hyperkube command is that: I have created this in the Google DNS Someone knows why the DNS zone query is getting the 404 error?? :disappointed: What you expected to happen: It should have created the missing domain. How to reproduce it (as minimally and precisely as possible): As per above. Anything else we need to know?: Environment: Kubernetes version (use ): 1.7.4 Cloud provider or hardware configuration**: OpenStack (irrelevant) OS (e.g. from /etc/os-release): irrelevant Kernel (e.g. ): irrelevant Install tools: irrelevant Others: irrelevant
The bug is related to this: The above code assumes that the provider code returns no error but an empty list if the domain does not exist, but instead the provider returns a 404. Need to fix the provider and/or the above code.
/sig federation
I am unlikely to have time to work on this soon, so if anyone else wants to, go right ahead.
/assign
An additional information: I have created a which is based on the credential JSON file created through instruction. After that, I have configured the Federation controller manager to mount this secret as a volume in the container and set the environment variable to the JSON file inside this volume. Before doing that, I was getting an authentication error so the google DNS provider couldn't be instantiated, and after doing the instructions that I mentioned before, this authentication error was solved but now I'm getting 404 error when Google DNS provider tries access the specified managed zone. The file that I have generated follow this structure:


Which jobs are failing: Which test(s) are failing: Since when has it been failing: Testgrid link: Reason for failure: Anything else we need to know: /sig api-machinery /assign
/cc
want to take a look at the audit event issue first? I can dig in after that if issues remain.
yes, I'm looking at the audit event issue now
actually the failure message suggests that the missing audit log is caused by unexpected admission invocation: I think it's unrelated to auditing Note that there are two tests named "no reinvocation of webhook B when in-tree or prior webhook mutations" :( and from the audit log (), it also suggests that the test is hitting a wrong endpoint
Thanks I'll dig in.


In cluster/saltbase/salt/kube-proxy/kube-proxy.manifest, we have this: But kube-proxy also has a flag. Is there a reason to use the approach instead of using the flag? cc as you wrote the code, in
/cc who's working on moving kube-proxy to a DaemonSet
In general, oomscoreadj is not expected to be directly updated by pods and kubelet is expected to manage it. If kube-proxy runs in the Guaranteed QoS class, then it's OOM score adj will be set to a low number. Until that happens, kube-proxy process may want to be shielded from OOM kills to an extent (this matters only when kube-proxy's memory usage memory request) Meanwhile, it does not make sense to have two means to configure the same thing. I'll let decide which option he prefers and sunset the other option.
Thanks for clarifying. At this point, I prefer to use the built-in flag instead of relying on composing bash commands. Sent for this.


What would you like to be : Periodically remove cleanupInfos for old containers from dockershim's map. One way of doing that would be to just forget about s older than say an hour, as they should always be short-lived. Why is this needed: introduced a new in-memory map for the dockershim, that could potentially (in pathological cases) cause memory leaks - for containers that use GMSA cred specs, get created successfully, but then never get started nor removed. This might be okay for the alpha release of the GMSA feature, but should be addressed in the beta. See
/assign wk8
/sig windows
/milestone v1.15
got moved out. let's move this issue with it as well /milestone v1.16
Hello! I'm bug triage lead for the 1.16 release cycle and considering this issue is tagged for 1.16, but not updated for a long time, I'd like to check its status. The code freeze is starting on August 29th (about 1.5 weeks from now), which means that there should be a PR ready (and merged) until then. Do we still target this issue to be fixed for 1.16?
: yup we'd love to have this in 1.16, still waiting for kind reviewers on :) ...


<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --pull-kubernetes-unit : (PR with no go code changes, should not have failed any unit tests) links to appreciated links to specific failures in spyglass appreciated <!-- Please see the deflaking doc () for more guidance! --/sig storage
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assigne
/assign
/assign tried to fix it in


Here is a list of TODOs that could be fixed for Ginkgo migration from v1 to v2. [x] Lost the validation on the SIGDescribe: [x] Ginkgo V2 currently cannot get the number of specs that will run in , this might be fixed in the Ginkgo's minor release of V2 in the future. [x] <delGinkgo dry-run mode has dropped the test spec in V2 (), consider to switch back to use to generate test spec when the feature is ready in . </del--- This has been fixed in Ginkgo 2.1.4 [x] <delMigrate all the custom reporter in , should be lightweight and not sure where those custom reporters are used. </delFixed here: [x] Replace all the camel case flags to kebab case flags, e.g. to . [x] <delRemove the legacy e2e test binary's parameter and in 1.28, those parameters were used to manage report, this is no longer needed for Ginkgo v2. </delno longer needed. [x] upgrade vendor when is migrated to ginkgo v2, this will help to remove the dependency of - [x] <delrevisit to explore the possibility to cover the test in Ginkgo V2. </delfixed by [x] remove the workaround introduced in Ginkgo V1 ( is not called after hitting ^c): partially addressed by: , some cleanup needed for each sig before removing the hook entirely. I think has the pr to fix this entirely, but I cannot get the link for that, mark it as resolved. : [x] New work item (?): capture only a single output text output stream () in JUnit (this issue was resolved along with another one in ) [ ] Due to the junit format change, some tool which use to grab "value" directly cannot get the failure reason as what it does in v1. [x] Test failures are missing information since GinkgoV2 migration [ ] is clustering stack traces, not failure text () Address those leftovers if there is any update from Ginkgo in the future. see the relevant link on the each of the issues.


No response_ <details</details<detailsall </details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig testing
/cc
/cc
hello! I would like to work on this issues
I think we cannot do anything now except the last two, and those two should be based on top of the:
I would like to work on task 5 of this issue
no problem, pls wait for an while until that PR got merged, then you can create a pr to fix those deprecated parameters.
Regarding : searching for points out the places where the workaround is used and needs to be replaced once is solved.
I just noticed this comment in
seems like is not used in k/k, but in other sigs.
hmm... I just grepped the test code, will push a pr to fix this.
/open
/re-open
okay, this is not resolved actually, but I have no way to re-open it.
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
~ it seems that some more jobs needs to be adapted to not use parallel~ [repo][0]=kubernetes/test-infra~ ~ACTION REQUIRED: Ginkgo V2 doesn't accept go test's flags to parallelize Ginkgo specs, please switch to use or instead.~ Sorry for the noise, the flag is passed to kubetest, and it correctly translate it to , it is not the issue here
np, thanks for the update!
New work item (?): capture only a single output text output stream () in JUnit.
updated the issue, haven't got a chance to take a close look at the item though.
This should be resolved by "open stdout" now should only appear if a test intentionally records SpecReports.
mark it as resolved.
is this complete now?


, Only two of them left, I need to follow this one, here is the issue for this one opened in the Ginkgo: I think this should be fixed in the third party tools instead of Kubernetes, if we agree with that, we can mark it as completed as well. For the first one, I will continue to follow until there is a update from Ginkgo's side, for the later one, there is no consensus yet IIUC, we can open a separate issue for that, I am gonna to close this one since there is no much thing we can do now. /close
We agreed to fix the tool, but had no issue open for it. I just created one:
Another issue for


What happened: Occasionally we get a Daemonset stuck with less Current replicas than Desired, causing a Daemonset sync tends to fix it, e.g (add a new node). This happens to all our Daemonsets. I'm using fluentd as an example. There are no Daemonsets pods in pending nor are there any errors when describing the Daemonset What you expected to happen: The Daemonset Pod to be scheduled How to reproduce it (as minimally and precisely as possible): Reproducing this state is very tricky, I've had the most success by terminating nodes in our ASG causing a new one to come up. However I think I've identified why this is happening. The issue appears to be related to daemonset Pod deletion, a Pod gets deleted causing to trigger, this lowers the expectations by as expected. The delete pod event then triggers a sync of the daemonset, in which it finds which involves fetching pods from the informer cache This is somehow returning the deleted pod. You can see in the logs here thinks that it needs to delete the pod that was deleted. It is also setting the expectation back to after the delete pod event previously lowered it by 1. At this point you're stuck as the function can't get past this line As the expectation can not be satisfied as syncNodes set it back to and there's nothing it to get it back to . So I believe this is what's causing us to get less replicas than expected, the function is stuck due to the expectation so it can't perform a full sync to schedule the new pods. There's also not much to trigger a new sync once the expectation has expired until a new node comes up or one goes down. Environment: Kubernetes version (use ):Cloud provider or hardware configuration: AWS OS (e.g: ):Kernel (e.g. ):
/sig apps
I investigated this as much as I could without getting some external input, it feels like it's some kind of race condition in the informer cache. If anyone has ideas of if places to dig deeper that would be handy.
Reiterating the mentions to trigger a notification: <detailsIn response to : I investigated this as much as I could without getting some external input, it feels like it's some kind of race condition in the informer cache. If anyone has ideas of if places to dig deeper that would be handy.


Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This problem is definitely related to node termination, it's trying to delete the already deleted pod from the node that's been terminated. The problem seems to go away if is set, I think although this kind of race condition may have always been possible, it's being frequently caused via Which is finding the non-existent pod in the pod cache and seeing that it is assigned to a non-running node, resulting in a request to delete the non-existent pod. cc
Reiterating the mentions to trigger a notification: <detailsIn response to : This problem is definitely related to node termination, it's trying to delete the already deleted pod from the node that's been terminated. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign could you please take a look at this? It seems that it is related to the code you in .
Thanks for reporting the bug. Bug was introduced in which deleting pods without nodes. There is race condition between removing pod from cache and listing pods for DaemonSet. Bug occurs during following scenario: is deleted. for node is deleted by PodGCController. handler is executed by informer () set is to queue by delete handler () is processed and pods are taken from cache before deleted pod is removed. for deleted pod is not found so we add deleted pod to podsToDelete list (). are set to del:1() succeeds as pod is not found(). are never met so we need to wait till expectations expire. As fix I will send a change to so only pods that can't be deleted by PodGCController are deleted by DaemonSet controller.


<!-- Please only use this template for submitting enhancement requests --What would you like to be : Node E2E tests for Kubelet lock contention flags: --exit-on-lock-contention --lock-file Why is this needed: These flags are being graduated to KubeletConfiguration on , we should add E2E tests for these flags before proceeding.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node /wg component-standard
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-contributor-experience at . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


What happened: Set up a HA cluster with 1.9 and 1.10 API servers (occurs on any apiserver skew boundary where a cohabitating resource's storage version changed groups) Observed errors when modifying workload objects: when modified via the 1.10 server, saw this error in the 1.9 server: when modified via the 1.9 server, saw this error in the 1.10 server: When this error was encountered, all watchers for the resource in question would be dropped and have to re-establish, and the internal watchers feeding the watch cache would drop and re-establish What you expected to happen: Watch events from all supported storage versions are processed correctly How to reproduce it (as minimally and precisely as possible): Start a server from master (for inspectability, store in json). With Create a deployment (e.g. kube-dns in kube-system) Establish a watch via extensions (kubectl get deployments.extensions --all-namespaces -w) Establish a watch via apps (kubectl get --all-namespaces -w) Tail the kube-apiserver log Change the apps/v1 deployment in etcd to extensions/v1beta1: Observer client watches drop and server log errors: Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: /sig api-machinery /cc /priority critical-urgent
error is triggered by this returning false: constructed here: the info about expected cohabitating groups is not included in the list of acceptable inputs


<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Statefulsets are sometimes continually rolling pods and I don't know why statefulset created: storageclass referenced: kubectl describe statefulset statefulset-test: As you can see from the above describe, the pod in the statefulset has been restarted a lot of times. Observing the created pod, 1) The pod comes up ready 2) immediately transitions to terminating 3) Is terminated 4) A new pod is created 5) Go back to (1) (repeats forever) Despite no updates to the statefulset spec ocurring. Making a change to the statefulset template (for example a trivial change adding a nonsense annotation) stops the rolling and fixes the issue When this statefulset was created, there was a mutating webhook enabled. After the observed pod rolling. I disabled this by making its configuration invalid and making it non required, and the rolling persisted. The status of the statefulset: : You can see that there is a discrepency between currentRevision and updateRevision. currentRevision: statefulset-test- updateRevision: statefulset-test- A short while later the status is changed to contain currentRevision: statefulset-test- updateRevision: statefulset-test- This change happened at Sun Apr 1 19:48:48 2018 Presumably after the rollout of statefulset-test- is complete.


Here are all the controllerrevisions from above: kubectl get controllerrevision statefulset-test- -o yaml kubectl get controllerrevision statefulset-test- -o yaml: kubectl get controllerrevision statefulset-test- -o yaml: The differences between the above controllerrevisions are insignificant (the hash, the creation timestamp, the name, the resource version, the uid, the self link) Interestingly all these controllerrevisions were created very close together in time shortly after I created the statefulset The current controllerrevisions are After watching the state change for a while, it actually seems like the statefulset is just cycling through the updateRevision being one of the above 10 controllerrevisions Leaders during this time From kube-controller-manager logs during this time, here is one excerpt: The error sync in there is suspicous, and as you can see the controllerrevisions are changing in the logs I have the full logs from a 5 minute window on either side of the above if necessary. The reason I'm inclined to think this is a kubernetes bug is that any update to the statefulset template fixes the problem. How to reproduce it (as minimally and precisely as possible): Not really sure. Worth just trying to create the above statefulset (requires creating the I tried deleting and recreating the above statefulset and it didn't happen, so something flaky is going on Anything else we need to know?: kubernetes 1.9.3 (client and server) cloud provider: aws
/sig apps
I realize this will probably need some further debugging, happy to get some direction
Consistent repro: Create a statefulset that creates a controllerrevision whose hash is between max int32 and max uint32 (this shouldn't be that hard to do, possibly a 50/50 chance of this). Manually create a controllerrevision identical to the first (same revision), incrementing both the name and hash by one. See that the statefulset controller will continually switch which controller revision should be the target of the statefulset.
Due to the issue fixed here: , under some circumstances controllerrevisions whose revision is the same were being considered equal.


This ends up causing this bug to manifest as follows: ListRevisions finds many controllerrevisions with the same revision (unclear how this state happens, but it can happen) The sort here randomly sorts them Due to the linked bug , FindEqualRevisions incorrectly finds that all these controller revisions are equal to the proposed udpate. The updateRevision selects the last revision found here Therefore, updateRevision, is randomly selected and causes the bug. This fixes the immediate issue, but the sort should probably be made stable (i.e not just look at the revision number, perhaps just consider the hash for tie breakers)
As an update, fixing the equality check will not be sufficient because it is possible for multiple controller revisions to be created that are actually equal, including the same hash, that have the same revision number. I believe the fix will need to be equal controllerrevisions with the same revision number from being created sort the controllerrevisions so that the controllerrevision that is the target of the statefulset is stable
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
ping I use the K8S which version is 1.11.4 and encounter the same problem. I want to reproduce the problem and find your way: Manually create a controllerrevision identical to the first (same revision), incrementing both the name and hash by one. Could you tell me how to mannually create a same controllerrevision identical to the first? Thanks in anvance.


generation for local KEK based on counter; also rotate once a week even if the counter does not hit 2 million? using AES-GCM encryption of local KEK ref: The implementation would be to the directory.
/sig auth /triage accepted
/assign
GitHub didn't allow me to assign the following users: ibihim. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
thank you !
/milestone v1.26
Hi bug-triage shadow team member for 1.26 here. Looks like this issue is in the 1.26 milestone, but has been moved to 1.27. Is there something else or should this issue go to 1.27 as well? Thank you!
I just forgot to move this issue /milestone v1.27
/reopen is the partial implementation.
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
My suggestion on the reference implementation, after reading 's comments is like so... The GRPC server, which implements the expects an implementation that satisfies . The local key logic can be a middleware. So it expects a it can fallback to, if it doesn't know a key or needs to encrypt its own local KEK. So the local implementation would be need to: have a storage of s have a reference to the current in use (if we put it into the cache, it might vanish) have a reference to another , which should be the remote KMS. So the sub-stories could look like: write Middleware write Transformers that can get exhausted (by time, by usage) write 's storage write some dummy for remote KMS? I have some sketches of what I am talking about .


I would implement that in terms of the interface so that it can be used both in testing (via a local key) and in real code (by delegating to a HSM).
Hi bug-triage team member for 1.27 here. This issue is still open but has had no recent activity, and no active PRs. Is everything on track for the Code Freeze in March 14th? Thank you!
/milestone clear
Could be closed, right? :)
We still need metrics and tracing support for the reference implementation (though it is all a lot less important now).
/retitle [KMSv2] Add metrics and tracing in for plugins
This is duplicate of /close
Closing this issue. <detailsIn response to : This is duplicate of Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: a cluster, And node1 with resource 1 DaemonSet, request resource 4 run , There is no pod created. resource to the node: run , There is still no pod created. What you expected to happen: In step 5, There is one pod created. How to reproduce it (as minimally and precisely as possible): see above description Anything else we need to know?: Environment: Kubernetes version (use ):1.10.5 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release):CentOS7 Kernel (e.g. ): Install tools: Others: /sig app
xref
Thanks! is not work here! Because: DaemonSetsController didn't watch the changes.(we use to updateNode.) we restart kubelet, The DaemonSetsController watch the changes, but it seems that the node info is not updated, and still failed to create pod:
I found it is a bug here, ignore the changes. I made a fix , PTAL! Thanks!
Any workarounds for old cluster versions? Up: looks like in my case helped but after some delay.
, which version of k8s are you using? If it's sunset, you have to merge the pr yourself :). I think restart kubelet trigger delete/add to make sure it sync .
Sorry, I didn't say
I think we still support 1.12 for now :) let's cherry pick it if not and wait for next 1.12 release.


changed the default behavior of scheduler framework's - from stopping at the first failed filter to always running all filters. This change breaks Cluster Autoscaler at scale. Cluster Autoscaler imports the scheduler framework to simulate filters on artificial nodes. In a (pretty common) one-pod-per-node scenario, the number of calls is ~quadratic wrt. the number of nodes needed for unschedulable pods - it can be a lot of calls. Before , in such scenario the vast majority of calls would stop at failing Fit or NodeAffinity filters, and not run the rest of them (e.g. PV-related filters). After the change, every call to runs all filters, which completely breaks Cluster Autoscaler at scale (an example scenario of 300 1-pod-per-node pods and 200 node groups takes 10min in 1.24, and ~50s in <1.22). Can we get the old behavior back? Or re-introduce the config option so that Cluster Autoscaler can set it accordingly? Ideally, we'd also like to get this backported as far as possible (the change was introduced in 1.24).
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scheduling
This seems to be a regression. I would be interested in adding a toggle flag that allows one to toggle on that behavior if this was done on purpose.
That PR wasn't supposed to change the behavior actually. Scheduler is always defaulted to stop upon the first Filter failure, and it's just we offered an option to continue go through other Filter plugins upon failures, and that option was removed recently. Let me check the code to see if there is a regression.
When removed the following changes: It should have kept . I will submit a fix along with test, and backport to 1.24, 1.25, 1.26. Thanks for reporting!


cc /assign /kind bug /kind regression /priority critical-urgent /triage/accepted
Thanks Kuba, this is indeed a regression on the scheduler end.
/open for patches
/re-open /reopen
Reopened this issue. <detailsIn response to : /re-open /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Thanks for the reporting, and thanks for the fix.
Really sorry about this regression, and hope not to happen again. I think there's no e2e tests between kubernetes scheduler and cluster-autoscaler, so can we also have testcases of pods unschedulable in scheduling perf-test, maybe this can also help to prevent this happening again. UT benchmark test also makes sense, but it's not graphical.
Thanks for the quick fix! Unfortunately I'm afraid this kind of issue would only be caught in a large-scale test scenario, which is not cheap to run continuously.
For this particular issue, UT ( in the PR) is able to catch it. And hence doesn't need to be that complicated to test with cluster-autoscaler. However, standing from a broader angle, why we didn't catch the regression in perf test is b/c the triggering actor is unschedulable Pods, instead of schedulable Pods (which are the happy path most perf tests covered). So going forward, I'd like to see more perf tests to mixin unschedulable Pods, and hence more likely to surface this kind of issue. (we have preemptio preemption test suite but doesn't seem adequate)
This is not for the particular issue, but keeping an eye on whether there's any performance degradation between two kubernetes components, like we do for scheduling performance. but it's actually out of the scope of scheduler. I'll add it as one of the goals of
Actually, the schedulerperf has a test case . And, the schedulerperf collects the metric (), which has the label in it (). But, the problem is currently we don't distinguish the metric values by label on each data; we just collect/aggregate it regardless the label.


(We can distinguish the metric values by any labels in the schedulerperf as we do in the metric ()) I'd say just distinguishing the happy path and unhappy path with label in metric should be enough to observe the particular performance degradation in the unhappy path.
could you help share the perf-dash link where this metric () is exposed?
Hm, cannot see any scheduler metrics result from the current perf-dash I see some configuration that seems to be to parse the schedulerperf result file though. https://perf- TBH, I haven't visited the perf-dash so much, so not sure whether it's not working for a long time, or got broken recently. Also, another thing, I noticed the perf-dash doesn't show all test cases. All test cases after haven't been executed.
So... probably, something wrong happens in  job (probably in the test case) and the schedulerperf is failing. The schedulerperf doesn't make the result file when fails, so I'm guessing that's why we cannot see any scheduler metrics results from the perf-dash. Okay, I found it. Let me open another issue for this.
cc is this actually a regression, and is it resolved yet?
It's resolved by , and also cherry-picked to 1.24.x ~ 1.26.x. /close
Closing this issue. <detailsIn response to : It's resolved by , and also cherry-picked to 1.24.x ~ 1.26.x. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: EndpointSlice for masters was not created after enabling EndpointSlice on a pre-existing cluster. Only those EndpointSlices managed by kube-controller-manager were created. What you expected to happen: EndpointSlice for masters should be created after enabling EndpointSlice on a pre-existing cluster, the same as the EndpointSlices managed by kube-controller-manager. How to reproduce it (as minimally and precisely as possible): kubeadm to create a cluster: the EndpointSlice feature by editing the kube-apiserver, kube-controller-manager, kube-proxy manifest kube-apiserver and kube-controller-manager is restarted, the EndpointSlice in namespace will be missing. Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools:Network plugin and version (if this is a network-related bug): Others:
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/remove-triage unresolved /assign


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: is the recommended way to configure the outbound networks together with Azure standard loadbalancer (SLB). However, for load balancer rules are disabled by default. That means all frontend IPs would be used as the SNAT IPs for virtual machines, according to . This would introduce troubles for various security scenarios, as there's no way to get a stable list of egress IPs. Azure cloud provider should support this scenario, and disable the outbound SNAT when outbound rules are configured by provisioning tools. What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Others: /kind bug /sig azure /priority critical-urgent


The server-side validation does not check the type of values with . Create a CRD like: Then create a CR: It is not rejected, but should because must not be a boolean.


Kubernetes e2e suite [It] [sig-network] Firewall rule should have correct firewall rules for e2e cluster See also a similar cluster with the same test failing: /kind failing-test /kind flake /sig network
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


https://k8s- Failed: [] Kubectl client [] Kubectl should be able to talk to api server kubectl running in a pod could talk to api server [Conformance] {Kubernetes e2e suite}
https://k8s- Failed: [] Kubectl client [] Kubectl should be able to talk to api server kubectl running in a pod could talk to api server [Conformance] {Kubernetes e2e suite}
https://k8s- Failed: [] Kubectl client [] Kubectl should be able to talk to api server kubectl running in a pod could talk to api server [Conformance] {Kubernetes e2e suite}
https://k8s- Failed: [] Kubectl client [] Kubectl should be able to talk to api server kubectl running in a pod could talk to api server [Conformance] {Kubernetes e2e suite}
https://k8s- Failed: [] Kubectl client [] Kubectl should be able to talk to api server kubectl running in a pod could talk to api server [Conformance] {Kubernetes e2e suite}
https://k8s- Failed: [] Kubectl client [] Kubectl should be able to talk to api server kubectl running in a pod could talk to api server [Conformance] {Kubernetes e2e suite}
https://k8s- Failed: [] Kubectl client [] Kubectl should be able to talk to api server kubectl running in a pod could talk to api server [Conformance] {Kubernetes e2e suite}
https://k8s- Failed: [] Kubectl client [] Kubectl should be able to talk to api server kubectl running in a pod could talk to api server [Conformance] {Kubernetes e2e suite}
https://k8s- Failed: [] Kubectl client [] Kubectl should be able to talk to api server kubectl running in a pod could talk to api server [Conformance] {Kubernetes e2e suite}
https://k8s- Failed: [] Kubectl client [] Kubectl should be able to talk to api server kubectl running in a pod could talk to api server [Conformance] {Kubernetes e2e suite}
https://k8s- Failed: [] Kubectl client [] Kubectl should be able to talk to api server kubectl running in a pod could talk to api server [Conformance] {Kubernetes


e2e suite}
https://k8s- Failed: [] Kubectl client [] Kubectl should be able to talk to api server kubectl running in a pod could talk to api server [Conformance] {Kubernetes e2e suite}
https://k8s- Failed: [] Kubectl client [] Kubectl should be able to talk to api server kubectl running in a pod could talk to api server [Conformance] {Kubernetes e2e suite}
https://k8s- Failed: [] Kubectl client [] Kubectl should be able to talk to api server kubectl running in a pod could talk to api server [Conformance] {Kubernetes e2e suite}
https://k8s- Failed: [] Kubectl client [] Kubectl should be able to talk to api server kubectl running in a pod could talk to api server [Conformance] {Kubernetes e2e suite}
https://k8s- Failed: [] Kubectl client [] Kubectl should be able to talk to api server kubectl running in a pod could talk to api server [Conformance] {Kubernetes e2e suite}
Are you the correct assignee for this? If so, can you please give an update. If not, can you please reassign it to the correct person. CC
https://k8s- Failed: [] Kubectl client [] Kubectl should be able to talk to api server kubectl running in a pod could talk to api server [Conformance] {Kubernetes e2e suite}
https://k8s- Failed: [] Kubectl client [] Kubectl should be able to talk to api server kubectl running in a pod could talk to api server [Conformance] {Kubernetes e2e suite}
https://k8s- Failed: [] Kubectl client [] Kubectl should be able to talk to api server kubectl running in a pod could talk to api server [Conformance] {Kubernetes e2e suite}
Yes, I guess so. I'm looking into it.


https://k8s- Failed: [] Kubectl client [] Kubectl should be able to talk to api server kubectl running in a pod could talk to api server [Conformance] {Kubernetes e2e suite}
https://k8s- Failed: [] Kubectl client [] Kubectl should be able to talk to api server kubectl running in a pod could talk to api server [Conformance] {Kubernetes e2e suite}
https://k8s- Failed: [] Kubectl client [] Kubectl should be able to talk to api server kubectl running in a pod could talk to api server [Conformance] {Kubernetes e2e suite}
https://k8s- Failed: [] Kubectl client [] Kubectl should be able to talk to api server kubectl running in a pod could talk to api server [Conformance] {Kubernetes e2e suite}
https://k8s- Failed: [] Kubectl client [] Kubectl should be able to talk to api server kubectl running in a pod could talk to api server [Conformance] {Kubernetes e2e suite}
https://k8s- Failed: [] Kubectl client [] Kubectl should be able to talk to api server kubectl running in a pod could talk to api server [Conformance] {Kubernetes e2e suite}
We need to get this fixed soon and picked up in release-1.5 since it is consistently red in
ack i have a PR currently running
https://k8s- Failed: [] Kubectl client [] Kubectl should be able to talk to api server kubectl running in a pod could talk to api server [Conformance] {Kubernetes e2e suite}
If you found the root cause, we will close my pending PR that printing out some debug info. What is the PR # of the commit you mentioned above?


https://k8s- Failed: [] Kubectl client [] Kubectl should be able to talk to api server kubectl running in a pod could talk to api server [Conformance] {Kubernetes e2e suite}
: here it is
https://k8s- Failed: [] Kubectl client [] Kubectl should be able to talk to api server kubectl running in a pod could talk to api server [Conformance] {Kubernetes e2e suite}
https://k8s- Failed: [] Kubectl client [] Kubectl should be able to talk to api server kubectl running in a pod could talk to api server [Conformance] {Kubernetes e2e suite}
We timeout waiting for pod sometimes. Is it reasonable to use a longer timeout? Use for in that test.
https://k8s- Failed: [] Kubectl client [] Kubectl should be able to talk to api server kubectl running in a pod could talk to api server [Conformance] {Kubernetes e2e suite}
https://k8s- Failed: [] Kubectl client [] Kubectl should be able to talk to api server kubectl running in a pod could talk to api server [Conformance] {Kubernetes e2e suite}
: sure thing, do you want to do that in a fresh PR?
I updated my pending PR .
https://k8s- Failed: [] Kubectl client [] Kubectl should be able to talk to api server kubectl running in a pod could talk to api server [Conformance] {Kubernetes e2e suite}


master-blocking: ci-kubernetes-unit It starts flaking today 30-05-2024 The test fail is This failure may be caused by the error message's order sensitivity. The error message's order of Resource Classes versus is different. N/A /sig node /sig scheduling
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-sig testing /sig scheduling (sig testing doesn't own all of the unit tests, just test frameworks, tooling, CI infra, etc) This appears to be a flake in scheduler unit tests which would be sig scheduling ( )
.. and it appears to be DRA specific subset of scheduling so /sig node as well
FYI - looks like this is happening every 3-6 hours give or take. Yesterday it was every six hours and today it seems to be every 3-4 hours. Looking into it a bit more, it looks like it may just be an issue with a connection via the test while it's trying to hit the server. I don't think the test itself is failing, I think the connection to the server is failing. ! CC
Hey All - hope you're well. Michael here from the Release Signal team. Just wanted to touch base and make sure we're good to go for the cut on the 4th. CC /sig-node


is this a Node bug? cc as next oncall to follow up on this.
This is a node bug due to init containers
I may need help debugging - I see:
So I think the problem is in the error case - we're not properly recording that the start failed, and the next time we come in we somehow think the first container succeeded. I'll add some debugging to see if we can't recreate it.
Let's discuss on instead.


Kubelet issues a docker inspect call after creating a container, but for very short lived containers there is a window in which they could exit before being inspected in the docker manager. If the container PID is 0, we should check the ExitCode and other relevant fields and handle them appropriately. [1] This is the suspected cause of several flakes: /cc
looking at the logs 2 weird things: race described second container ran, which suggests inconsistent reporting of initialized status of init1 and choosing init2 as the next container ( vs ). I think 1 got it right and 2 got it wrong. From logs: I'd expect the second one to not have run in this case? Node logs:
Init containers have an implicit assumption that container statuses include results of previous SyncPod invocations. If that's not the case we may need to fix that (or at least mitigate it). On May 18, 2016, at 8:28 PM, Prashanth B wrote: looking at the logs 2 weird things: race described second container ran, which suggests inconsistent reporting of initialized status of init1 and choosing init2 as the next container ( vs ). I think 1 got it right and 2 got it wrong. From logs: May 18 15:58:33.380: INFO: pod-init--1d4b-11e6-80a4- started at 2016-05-18T15:58:04-07:00 (2+1 container statuses recorded) May 18 15:58:33.380: INFO: Init container init1 ready: false, restart count 0 May 18 15:58:33.380: INFO: Init container init2 ready: true, restart count 0 I'd expect the second one to not have run in this case? Node logs: — You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub
This should be true. Is there any evidence that suggests the opposite? only shows conflicting view of whether init1 has run.
If the container exits already and it's not an infra container, kubelet should not error out. Although even if it does, I failed to see how this could affect the readiness of the init container without checking the code further.


Each syncPod invocation, we inspect the list of init container in order looking for lastTerminationState. There could be a bug in that logic, or if we didn't get the full status, we might make a bad decision. When determining which container to run next, we always look for the last init container in the chain that has success, then run the next container (unless the pod is RestartNever). , Yu-Ju Hong wrote:
Saw this flake again. How do we verify that container status is not lost between syn pods?
... between calls to syncPods
The second sync clearly saw the init1 container, and decided to launch init2. See "InitContainersToKeep" in the message below.
containers to keep can include failed containers. , Yu-Ju Hong wrote:
Discussed on slack. In summary, the problem has to do with how we convert the kubecontainer.PodStatus to the api.PodStatus. If a container is terminated, the function makes the decisions: the pod restart policy is "Always", and there is a "failure reason" recorded for the given container, this means the pod worker tried to (re)start the container but failed. Set the state to "Waiting" and populate the reason. , the container state is simply "terminated". Init containers don't obey the restart policy like regular containers, and the logic will need to be amended to work with them. is working on a PR. In addition to that, kubelet should not error out if a user container exited before it could apply the oom score (which is best effort in this case). fixes that.
Ahh, thanks for digging into the issue. This is the exact the concern I had when I reviewed the initial implementation of init container, which blocks pr for a while. I even asked to help me verify this case, unfortunately we didn't reproduce the issue back then. :-(
I wish I'd seen the flakes in the runs prior to merge.
We didn't notice this until we started running the tests against docker v1.11.1, which is much faster and hit the race condition differently.
I suppose that's a good thing...
It is hard to reproduce this.


The extra check of last terminated container in your initial code makes me nervous, but we couldn't reproduce it due to that small race windows until we have a faster docker running on the node. :-)
I got "Pods should invoke init containers on a RestartAlways pod" flake (referenced from ): Logs:
All the fixes went in. Please reopen this issue if you see any more flakes.


<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: In order to migrate to a new version of kubernetes on GKE. I cordoned(Kubectl cordon <node) all nodes in the old node pool(1.8.9) in order to drain the node pool(pods on the node) with an intention to cause the pods to run on a new version of the node pool(1.10.2). As soon as I cordoned all the nodes in the old node pool, it caused the downtime of ingress(nginx). The nginx controller pods where running fine on the cordoned node pool. But the cordon caused the nodes to be removed from the target pool of ingress load balancer. Below are the documents that suggest the above same method that I followed as best practice for zero downtime upgrade. 1). 2). What you expected to happen: I expected "kubectl cordon <node" not to have the node/s removed from the target pool of the loadbalancer because the nginx controller pods where running absolutely fine on these nodes. How to reproduce it (as minimally and precisely as possible): 1).Cordon all the nodes in an old node pool that runs nginx controller. Anything else we need to know?: Environment: Kubernetes version (use ): 1.10.2 Cloud provider or hardware configuration: GKE OS (e.g. from /etc/os-release): Container OS Kernel (e.g. ): 4.14.22+ Install tools: Others: Cordon causes to remove the nodes from the target pool of the ingress load balancer. The health checks fail on the nodes of the target pool, which should not be the case. Because the nginx controller pods run fine on these nodes.
Technically, this isn't a Kubernetes bug as this is working as intended; however, it does beg the question why we remove unschedulable nodes if they're marked as ready since we perform the to determine LB backend nodes. is it scheduable? is it not a master? is it not marked for exclusion? is it marked as ready?


If any of those are a no, then it'll be removed from the pool on the next sync. Perhaps someone from can shed some light. /sig network
cc This problem extends beyond GCP/ingress controller and to anyone using pools to upgrade nodes. A typical workflow I see is you'd cordon a node pool so as you drain each node the pods don't go to another node in the pool that'll shortly be drained as well. The current logic does not support that and complicates cluster administration because cordoning does not respect PDB's.
/remove-sig apps /remove-sig gcp (since the service controller falls under sig networking)
Those labels are not set on the issue: <detailsIn response to : /remove-sig apps /remove-sig gcp Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Ref I thought a general practice for draining the old pool is to move over one node at a time in a rolling fashion. Shouldn't there always be certain backend pods being able to receive & serve traffic? cc
WRT draining, it is recommended to do it one node at a time ; however, in real world scenarios doing it one node at a time can cause a lot of avoidable container churn if you don't cordon off the nodes you at least plan to drain so that pods aren't being scheduled on a node you'll be draining after the next one. According to the comments in the service controller it seems the unschedulable part exists because it was how we differentiated masters in the past; however, we have labels for that now. We also have the service exclusion label (can't find when/if that'll be promoted though as i don't think there's a ticket on it).
I'm having problems with this issue as well. The fact that nodes are removed from target pools when cordoned means that I sometimes get brief interruptions of the traffic to a service when I cordon a node. I'm guessing the node isn't gracefully removed from the load balancer which results in some traffic being cut off trying to route through the recently cordoned node. The same problem occasionally happens when deleting nodes / node pools or when the cluster autoscaler deletes a node. We are running GKE ver. 1.10.2-gke.1.


Our best defence against these issue have so far been to limit node changes as much as possible and not use any auto scaling.
I've posted a question regarding this issue in SO some weeks before it was created. The suggested action was to use node taints () instead of cordon, that way nodes will still be marked as .
Thank you for the link, it Certainly helped. Perhaps, if thats the case(as mentioned in the accepted answer of the above link), then to cordon all the nodes of old node pools should not be a recommended action, at least in the GKE upgrade docs.
only says: And I can't find what more it does in the documentation. So I would expect that it has exactly the same behaviour as a taint. Why is removing running pods from the balancer working as intended? If it isn't a bug then the documentation is certainly unclear in this regard.
This just caused a major outage for us too. I agree with the previous posts, while this may be working as the code intended, it does not follow the patterns and documentation k8s provides. When I cordon a node it should not impact the traffic going to the node or any pods on it. I should be able to gradually remove traffic by draining nodes/pods and respecting PDBs.
this issue touches on the fact we are removing masters as well and inadvertently removing cordoned nodes. The logic in the service controller mentions we only filter out unschedulable nodes since that's how masters were setup.
We wanted to replace all of the kubernetes cluster nodes to different node instance types, so we started off by cordoning all of the nodes, but it surprisingly caused an outage for us. As pointed out, the only documentation around cordoning is , which I took to mean that new pods can not be scheduled on those nodes. It was surprising behavior to find out that, not only does it not allow new pods to be scheduled on those nodes, but that it also causes the service controller to remove unschedulable nodes from load balancing pools. I understand that it's working as intended, but if that was also documented as part of the cordon operation, I would have been able to avoid an outage.
I finally caught up to this bug. Fun one. I agree that this is working-as-intended and that it stinks. We can do better.


As a short-term workaround, you can taint the nodes before cordoning one-by-one. That will stop new work from arriving, but will leave old work there. If we agree on an appropriately named taint, we can have nginx tolerate it by default. Am I missing any reason why this would not work? If that flies, we should fix docs ( those are GCP docs). I am happy to help with that. Longer term maybe we can do something smarter. I am not sure what that would be, just now. Ideas?
Is there any reason we can't just leave the nodes in the load balancer pool when the node is cordoned? And when the pods are terminated the nodes can be removed? I may be missing some context here?
The presumption is that a node that is unschedulable is going away, or needs some other repair. Using it as a target for load-balancers means there could be sessions routed THROUGH that node. Even as the pods are drained. Right up until the node suddenly dies, causing an outage of a different flavor. What should really happen is that a node which becomes unschedulable starts draining connections. Once all the pods are moved off, the node can be removed from the pool entirely. I don't think any cloud providers, even GCP, handle this just right today. David Radcliffe <:
Wait, if you brought up a new nodepool, all those new nodes should be in the TargetPool. Cordoning the old nodepool should only terminate the existing connections, new connections should be fine. Something doesn't add up...
Was this using
Yes
Right, well, that explains it. So even draining won't do what you want - you need new connections to be allowed until you have capacity in the new pool. The good news is that you can do this entirely yourself, with taints. We just need nginx to tolerate the taint, which you can also specify yourself. The only real alternative I see would be, as you said, to leave nodes in the LB pool until the node dies. Given that we never specified this, it is probably different on every provider, which would make it hard to change (especially if the provider already provides clean drains while unschedulable).


IIUC, we don't want new pods to land on the tainted/cordoned nodes. In that case, we can place "NoSchedule" taint on the nodes. We don't need any tolerations on nginx. This will prevent new instances of nginx from landing on the tainted nodes, but the existing ones will keep running there.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
This just surprised us also. I understand the technical reasons behind it but the command name and associated documentation is misleading. To me it isn't intuitive that the command does not remove compute (pods) but does remove some networking. I would expect cordon to keep new pods from getting scheduled to it and keep it from getting to new services/ELBs/etc. The command would then cause pods to be removed and also take the node out of networking paths. Almost need a new state for a node of that is clearly defined.
My main issue is with the docs, and also GKE docs. They should explain this pitfall. After I got aware of the issue I understand why it works as it does. A simple workaround is to taint the nodes you are going to rotate first.
Just chiming in; We as a GKE customer have today been bitten by this issue in production. The documentation of (in the context of GKE) is not extensive enough so we could be aware of this issue. This should be addressed. Luckily the outage was not severe because we were using the node-by-node approach (and accepted the container churn this would cause). We explicitly took this approach because our previous nodepool upgrade was a total outage also caused by in a different way. (for those interested in our total outage, not related to this issue): Due to the fact that also results in the re-calculating the amount of schedulable nodes and thus scaled the amount of pods in our cluster (in our case: from 110 back to 2) resulting for us in major DNS outage internally.


Be warned that for a nodepool upgrade can have a lot of unwanted & unexpected side effects.
It looks like this happens only to AWS and GCP but not on Azure? Can anyone confirm this? I just tested in both AWS and Azure, and while I see logs like in AWS Kubernetes, I do not see any logs like that in Azure. Is it intended? If so isn't behavior different for each cloud provider? i.e.) cordoning a node in AWS Kubernetes prevents loadbalancer from routing traffic to the affected node but cordoning a node in Azure Kubernetes will still let the loadbalancer route traffic to the affected node. Is it intentional?
Besides running a test like the above, how would I determine what the behavior is on AWS? I haven't found anything in the node administration docs or the Service that would imply that would shift traffic away from a Pod. In fact, the doc for explicitly states, emphasis mine: If I hadn't come across this Issue due to reports from GKE, I suppose it would only have been a matter of time until we had a high-impact production outage. There's a disconnect between Kubernetes API and the LoadBalancer as to whether Pods running on a 'd node are Ready. From all the docs I've seen, I would expect that either a) would evict Pods over to a Node before setting the instance as , or b) LoadBalancer would not equate with
Hi All, What approaches people using too workaround this? From reading the comments I was hoping to follow the custom taint approach. However, I've noticed executes a cordon making it difficult to drain the node safely with this approach.
I was lucky in that I experienced this issue while cordoning our testing environment, so we didn't experience a production outage. I think what I'm going to try is the following... it's going to be time-consuming but I think it's the safest bet. Double the number of nodes in the server group to bring online the new nodes first Place a custom taint on each "legacy" node As you said, executes a first, so that's a no-go. This is the time-consuming part... for each "legacy" node I am going to each pod, and let my PDBs take care of maintaining minimum number of pods. I'm hopeful to try this approach this afternoon, I'll comment back here if I do!


Same as myself! Pity neither of us have a story for Since my comment this morning I've been trying a few approaches, I've ended up with the following: Create a new node pool of equal size in GKE with the upgraded kube version Set the current node pool to not be able to grow beyond its current size - if your using regional nodes this can be annoying, let it bring up new nodes anyways and just have them taint as NoSchedule straight away Taint all old node pool nodes with NoSchedule Go through each old node pool node executing a taint of NoExecute which will cause the node to drain and go onto available nodes i.e. your newly provisioned node pool Ensure your nginx ingress is deployed with tolerances:
Hi All, I'm at a bit of a loss on this one. I've started using custom taints to drain the nodes without getting them removed from the load balancer. My application can hold a HTTP connection open for up to 600 seconds. I'm using nginx as a daemonset, have it configured with a keepalive timeout of 620s, and have . I drain the application first using a taint on each node. I have nginx on each node tolerate this taint and remove it separately at a later point using a different taint. I'm waiting 660s between drains of each node for each taint to ensure nothing is dropped - this all works fine as expected. However, once I kill the node pool or do a kubectl drain after the node is empty of all my stuff plus nginx ingress I start seeing connection issues. I'm a bit lost where to go at debugging them, best guess is something to do with kube-proxy. Can anyone shed some light on this? is there anything on the Google Layer 4 load balancer I'm missing here? Is there some way to correctly drain kube-proxy (or does that even make sense?)
This doesn't address the network issues you're seeing, but for those trying to "roll their own cordon" using taints: Using a NoExecute taint to "drain" the node won't be orderly and won't respect any PDBs. You want to replicate the logic from which tries to do evictions of every pod in a loop. (I realize this is a burdensome idea, but it is also the safest idea.)


To add on to ,: One approach that does not use node taints to force migrate, is to change for the requirements of your app. A change in node selector (e.g. based on instance type), along with a PDB (e.g. 66% / out of 3 replicas availability) should migrate you over fine. This worked out better for me than to drain or add taints to nodes to force push apps out. After the migration completes, you should be able to the nodes, and delete the node pool. FWIW, I too saw connection issues during load testing of a migration, and at least on my setup it was due to using an AWS NLB for the NGINX Ingress Controller. Switching to a classic ELB instead worked, as the NGINX LB Service seems to hit some sort of by my accounts when using NLBs. self-plug: I happened to recently cover a similar topic in a that you may find relevant. Hope it helps
For better or worse, for in my case pods that are behind an ingress resource are using node affinity to preferably not schedule them onto the same node. Originally this was done to ensure traffic is balanced across the different nginx ingresses. It has had a positive affect in this case too. Very interesting! Thank you for sharing, I'm not sure that swapping out the Layer4 GCE load balancer is an option here, but you have me thinking. I'm wondering if the TCP keepalive between the LB and nginx is potentially keeping connections open. , and suggest that there may be a 600 second TCP keepalive going on. Will retest doing a sleep of 620s after nginx removal and before node drain.
Has anyone come up with a strategy to safely move workloads between two pools? Reading this bug, it seems there's no way to reliably do this without causing an outage if you follow the recommendation of + . In my case I'm not using an nginx ingress but services with type . From what I can see, adding to a node causes the Service Controller to consider the node not ready and thus removes it from a target group. This would mean that between and there is no way to add a taint to the node that will leave it in the LB until it's drained.


Using doesn't work in this case since the thing I'm draining might just come back and I'm not sure if the service controller will still consider the node ready in that case, but I believe it does. The one strategy I can think of is using labels on the node and an anti-affinity on the pod. You'd then add the label to a node that would push the pod away and then delete the pod. As long as the anti affinity uses this should be safe to do in preparation of a migration. Once the pods have been deleted you could then the nodes.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/lifecycle frozen /remove-lifecycle stale /remove-lifecycle rotten
- As I mentioned in I set up a script that first taints all nodes in the old node pool and then cordon+taint one by one node. If your ingress services are redundant this should work well.
Is there a plan for cordon behavior to improve on GCP? We have relied on this behavior for years on our AWS clusters and it was very surprising to find out.
This is a bug with service load balancers - not something we should document. We defined a label in that cluster admins can use - Unschedulable has nothing to do with SLB inclusion. which is beta and on by default in 1.19.
Disagree. Unschedulable means "no new workloads". It has nothing to do with existing workloads, that is why we have drain (but we should have something in kube that communicates nodes are draining). I have opened to remove this check - we have an alternate approach now and the old logic is wrong (the comment in the code even says "this is for masters!" and that's wrong now too as part of the kep).
A node status condition?
Possibly, or a taint, or a label. Something concrete.
/milestone v1.19


When using a generic ephemeral volume, the volume binder in kube-scheduler typically processes a new pod before the generic volume controller in kube-controller-manager has a chance to create the PVC. This leads to events for pods using generic ephemeral volumes: It would be nice to suppress these warnings for a short while (duration to be determined) after the pod creation. Later they should be emitted because then they indicate that the generic volume controller isn't working properly. This makes nicer to use.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage


/kind bug We're having trouble retrieving images from a private registry on DockerCloud. For nearly a year the following has worked flawlessly. Last Friday I pulled down the 1.9.0 CLI (via brew) and since then have not been able to access any image in a private registry on DockerCloud. I've tried this on 10+ brand new clusters created with kops v1.8.0. The Kubernetes cluster version have been 1.8.0, 1.8.4, 1.8.5 and 1.9.0 (which is not supported by kops yet) and all exhibit the same problem. We create the secret as such: And verify it using: Which yields: And create a test service with with these contents: The pod fails immediatly with: I've reviewed the 1.9.0 release notes and nothing jumps out as the root cause. For reference I've tried putting quotes around the value when creating the secret to no avail. I've also tried prefixing the value with , so the full value would read: also to no avail. Environment: Kubernetes version: Cloud provider and installation info: AWS cluster built with Kops 1.8.0, using (to isolate this issue) nearly the default settings for everything.
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I have same problem, and I "solved" it creating a docker-registry secret in an old kubernetes version and pasting it on new kubernetes
I'm recently (I know it worked perfectly about last month) facing issues with as well. For me it ended in a similar manner saying: I was able to confirm, that this is a kubernetes CLI related issue and not end user. Based on and this I was able to create properly formatted auth information looking like this: . When encoding this one-liner using (MacOS workstation) and apply like this works fine and images can be pulled from private repository again.


However this breaks all automation and can't be considered as solution Confirmed working with kubectl 1.8.5 and server 1.8.4 (guess 1.7.9 as well)
looks like there's an extra "auths" element when adding the secret using 1.9 kubectl
Added in /assign
Many thanks for the quick fix on this. Any idea when this might get rolled out? Also, it might be a good idea if this was to the known issue list for v1.9.0. It would have saved us a bunch of time trying to track down the root cause. Just trying to make sure no one else needs to do the same.
linking to the "v1.9.0 known issues" list
ack, asking Mehdy since he is the patch manager for v1.9 branch. Mehdy, any idea when we can ship 1.9.1 with this fix?
workarounds for folks until 1.9.1 is released: if you have a .dockerconfigjson for your private registry already, you can manually specify the type and data key (e.g. ) if you don't have a .dockerconfigjson file already, you can fix up the secret produced by manually: the type from to the data key from to the modified secret with
1 will be released tomorrow
I can confirm solution works, thanks so much!
I am still having issues in 1.9.1. This worked in the 1.8.x branches: results in the following error:
you have to remove the gitlab-registry secret before you can apply an update... type is immutable, and the fix in 1.9.1 required changing the type of the secret


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: I'm trying to test Eviction functionality in my program, but running into some inconsistencies around how Evictions are handled in the fake SimpleClient in the client-go testing functionality. Most notably, creates a different from , and both are missing information. issues a with set to (seems to be the only Fake that has this mismatch) on the subresource of . It is missing all the extra information in the eviction, i.e. the . issues a on the subresource of the resource. This is wrong, because aren't created, they're only posted and handled immediately. The Action is missing a Namespace. What you expected to happen: I want to be able to add my own reactor to the SimpleClient to handle this special case. I propose we change both cases to a with set to the . This will ensure it's not handled by the current SimpleClient ObjectReaction, and makes it so users can implement whatever they want themselves. I'd be happy to make a PR with these changes. How to reproduce it (as minimally and precisely as possible): Output: Environment: git revision of : <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
POST requests are create requests. It doesn't matter if the object is persisted or not.
/sig api-machinery
cc


Not sure if this concided with a Jenkins restart.
Not sure what we can do about this. Retry a few times?
GitHub seemed to hiccup last night and fail a bunch of our builds. I'm not sure a retry would've helped, unless we kept retrying for many minutes.
This is the sort of thing that just happens occasionally. We've seen this a lot in OpenShift CI runs. I'm not sure there's much you can do other than try to avoid external network access (e.g. bake tools into the VM image).
It seems to have occurred twice in the last 12h so retry w/ timeout can't hurt?
It can't hurt... but it may or may not help depending on the extent of the external issue
If the problem is really bad, we could toss it somewhere on GCS?
Looks like it failed on : We should probably put this in a retry loop or else do something more fancy


<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind feature The is not user friendly. There has been to make it better, but we still print in the pending message. We should change these errors to a more human readable message. For example, should be changed to something like . /sig scheduling


The PR above moved everything related to into because that was where most of the code already was, but didn't renamed the files to avoid a larger diff and merge conflicts. Once is merged, we can do the renaming. Code clarity. /sig node
/triage accepted /priority important-soon
/assign
a PR can be created for this now.
will do


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: When I tried to create a pod with env variables referencing to annotations: What you expected to happen: Should Proceed. As in It stated is a valid option How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): v1.18.2 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig docs
The API docs need to be updated to indicate the correct supported values. See for correct values
You are a lifesaver. Thank you!
/assign I can try to update the api documentation


FYI. Currently the field of TPR is marked as which is wrong, actually if you try to create a TPR with empty versions, it will cause api server panic. Versions should be necessary field of TPR, we should validate this somehow.


Kubernetes version (use ): Environment: Cloud provider or hardware configuration: Vagrant OS (e.g. from /etc/os-release): CoreOS Kernel (e.g. ):Install tools: Vagrant & Others: What happened: What you expected to happen: Success How to reproduce it (as minimally and precisely as possible): Followed instructions at Tried to run Received: Anything else do we need to know: From googling, it seems several of the fields in the config are for when using OpenShift origin. But creating the replication controller is a prerequisite to launching OpenShift Origin. Chicken & egg problem.


Look at one of the directories under (Please don't pick more than one directory for one PR as it will cause issues for reviewers) Find all the files named (example find -name ) See the following link it has a this tells tools how this package needs to be imported. So we should ensure that each has an import Add a description appropriate to the package (similar to the one below) Look through sub directories and see if there are any other important directories that are likely to be used by outside users and add a Why are we doing this? We perfer to use vanity urls instead of directly using to refer to code, so this helps ensure that tools honor our wish. See It would be good to have some package level documentation since these directories under staging/ are meant to be for use by folks outside of kubernetes github org. /good-first-issue
/kind documentation /sig architecture
currently working my through Not going to use / assign since there could be a number of PRs for this issue.
/lifecycle active /good-first-issue edit: hmm..
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /lifecycle active /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
May I try on
IIUC this means that there needs to be a even at the root of the repo? Then how does work? :thinking:
Hi , can I make a PR for this directory ?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or .


/lifecycle rotten
I guess this is still not done, was supposed to partially fix this issue. /reopen
Reopened this issue. <detailsIn response to : I guess this is still not done, was supposed to partially fix this issue. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-lifecycle rotten /area code-organization