Question: config.CloudProviderBackoffDuration = backoffDurationDefault } <del> // Do not add master nodes to standard LB by default. if config.ExcludeMasterFromStandardLB == nil { config.ExcludeMasterFromStandardLB = &defaultExcludeMasterFromStandardLB </del> <ins> if strings.EqualFold(config.LoadBalancerSku, loadBalancerSkuStandard) { // Do not add master nodes to standard LB by default. if config.ExcludeMasterFromStandardLB == nil { config.ExcludeMasterFromStandardLB = &defaultExcludeMasterFromStandardLB } // Enable outbound SNAT by default. if config.DisableOutboundSNAT == nil { config.DisableOutboundSNAT = &defaultDisableOutboundSNAT } } else { if config.DisableOutboundSNAT != nil && *config.DisableOutboundSNAT { return nil, fmt.Errorf("disableOutboundSNAT should only set when loadBalancerSku is standard") } </ins> } azClientConfig := &azClientConfig{

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: is the recommended way to configure the outbound networks together with Azure standard loadbalancer (SLB). However, for load balancer rules are disabled by default. That means all frontend IPs would be used as the SNAT IPs for virtual machines, according to . This would introduce troubles for various security scenarios, as there's no way to get a stable list of egress IPs. Azure cloud provider should support this scenario, and disable the outbound SNAT when outbound rules are configured by provisioning tools. What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Others: /kind bug /sig azure /priority critical-urgent


Question: } } <del> func TestValidateCustomResource(t *testing.T) { type args struct { schema apiextensions.JSONSchemaProps object interface{} </del> <ins> func stripIntOrStringType(x interface{}) interface{} { switch x := x.(type) { case map[string]interface{}: if t, found := x["type"]; found { switch t := t.(type) { case []interface{}: if len(t) == 2 && t[0] == "integer" && t[1] == "string" && x["x-kubernetes-int-or-string"] == true { delete(x, "type") } } } for k := range x { x[k] = stripIntOrStringType(x[k]) } return x case []interface{}: for i := range x { x[i] = stripIntOrStringType(x[i]) } return x default: return x </ins> } <ins> } func TestValidateCustomResource(t *testing.T) { </ins> tests := []struct { <del> name    string args    args wantErr bool </del> <ins> name           string schema         apiextensions.JSONSchemaProps objects        []interface{} failingObjects []interface{} </ins> }{ <del> // TODO: make more complete {"!nullable against non-null", args{ apiextensions.JSONSchemaProps{ </del> <ins> {name: "!nullable", schema: apiextensions.JSONSchemaProps{ </ins> Properties: map[string]apiextensions.JSONSchemaProps{ "field": { Type:     "object",

The server-side validation does not check the type of values with . Create a CRD like: Then create a CR: It is not rejected, but should because must not be a boolean.


Question: {d: 8 * 24 * time.Hour, want: "8d"}, {d: 8*24*time.Hour + 23*time.Hour, want: "8d"}, {d: 2*365*24*time.Hour - time.Millisecond, want: "729d"}, <del> {d: 2 * 365 * 24 * time.Hour, want: "2y0d"}, {d: 2*365*24*time.Hour + 23*time.Hour, want: "2y0d"}, {d: 3 * 365 * 24 * time.Hour, want: "3y0d"}, </del> <ins> {d: 2 * 365 * 24 * time.Hour, want: "2y"}, {d: 2*365*24*time.Hour + 23*time.Hour, want: "2y"}, {d: 2*365*24*time.Hour + 23*time.Hour + 59*time.Minute, want: "2y"}, {d: 2*365*24*time.Hour + 24*time.Hour - time.Millisecond, want: "2y"}, {d: 2*365*24*time.Hour + 24*time.Hour, want: "2y1d"}, {d: 3 * 365 * 24 * time.Hour, want: "3y"}, {d: 4 * 365 * 24 * time.Hour, want: "4y"}, {d: 5 * 365 * 24 * time.Hour, want: "5y"}, {d: 6 * 365 * 24 * time.Hour, want: "6y"}, {d: 7 * 365 * 24 * time.Hour, want: "7y"}, </ins> {d: 8*365*24*time.Hour - time.Millisecond, want: "7y364d"}, {d: 8 * 365 * 24 * time.Hour, want: "8y"}, {d: 8*365*24*time.Hour + 364*24*time.Hour, want: "8y"},

What happened: When the duration is exactly 2 years or 3 years, human readable duration is returned as or . But when the duration is exactly 4 years, 5 years, etc, the human readable duration is returned as , , etc. What you expected to happen: The format of an exact year should be consistent. Remove when duration is an exact year: should be should be Check other years... 4, 5, 6, 7... not sure if they also have the problem. How to reproduce it (as minimally and precisely as possible): Because this is time-dependent, it can be difficult to reproduce via the CLI, but you can see the problem illustrated in which shows the current behavior: The and test cases should be changed to want and , and the code should be changed to make that pass. Anything else we need to know?: Note, this is not a problem when the duration is exactly 1 year because the human readable duration is still shown in days. So nothing needs to change to handle the case of exactly 1 year. Environment: Kubernetes version (use ): 1.18 Cloud provider or hardware configuration: n/a OS (e.g: ): n/a Kernel (e.g. ): n/a Install tools: n/a Network plugin and version (if this is a network-related bug): n/a Others: n/a
/sig api-machinery
/good-first-issue
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Could I pick this up as my first issue?
Absolutely. Just use to assign it to yourself.
Also I just updated the issue description. Check the other exact years 4, 5, 6, 7 because they might also have the problem, I can't remember. I know 8 and beyond don't have the problem because it just stops showing days after 8 years.
/assign bitscuit
might recall a related PR?


Question: {d: 8 * 24 * time.Hour, want: "8d"}, {d: 8*24*time.Hour + 23*time.Hour, want: "8d"}, {d: 2*365*24*time.Hour - time.Millisecond, want: "729d"}, <del> {d: 2 * 365 * 24 * time.Hour, want: "2y0d"}, {d: 2*365*24*time.Hour + 23*time.Hour, want: "2y0d"}, {d: 3 * 365 * 24 * time.Hour, want: "3y0d"}, </del> <ins> {d: 2 * 365 * 24 * time.Hour, want: "2y"}, {d: 2*365*24*time.Hour + 23*time.Hour, want: "2y"}, {d: 2*365*24*time.Hour + 23*time.Hour + 59*time.Minute, want: "2y"}, {d: 2*365*24*time.Hour + 24*time.Hour - time.Millisecond, want: "2y"}, {d: 2*365*24*time.Hour + 24*time.Hour, want: "2y1d"}, {d: 3 * 365 * 24 * time.Hour, want: "3y"}, {d: 4 * 365 * 24 * time.Hour, want: "4y"}, {d: 5 * 365 * 24 * time.Hour, want: "5y"}, {d: 6 * 365 * 24 * time.Hour, want: "6y"}, {d: 7 * 365 * 24 * time.Hour, want: "7y"}, </ins> {d: 8*365*24*time.Hour - time.Millisecond, want: "7y364d"}, {d: 8 * 365 * 24 * time.Hour, want: "8y"}, {d: 8*365*24*time.Hour + 364*24*time.Hour, want: "8y"},

(also, I have to leave the pedantic note: a year isn't always 365 days, so it's surprising to me to see the abbreviation exists at all)
I can't find it. Maybe I was remembering
Yeah it was probably that one. At first I had the tests and the fix. But it sat for a while. I changed it to just the tests of existing behavior so it could be approved more easily. Then I made this issue for the fix but decided it would probably be a good first time issue for someone instead of me doing it.
so should leap years be considered by the code?
There's no way to know how long a year is without knowing the date, so it's not a good unit--in the abstract, there's no way to know if it's a leap year or not. The go language deliberately omits it IIRC. Henry Li <:
What's the intention of HumanDuration? From what I've seen, it is just an approximation of a timestamp solely for humans to understand, so assuming 365 days per year sounds about right to me. This issue is about consistency though, so it could be solved the other way. Instead of abbreviating when it's an exact year (2y, 3y, etc.), exact years could just all have 0d (2y0d, 3y0d, etc.)
I think it is just used for a ballpark duration. It loses accuracy as the duration increases anyway because it changes the units. I guess my intention for opening the issue was to make it consistent with itself. For example it doesn't say 2d0h or 2h0m, so it should not say 2y0d.
(FTR I agree that Xy0d looks worse than Xy.) Brian Pursley <:
Hi I have made the required changes. Shall I raise the PR if you haven't started working in it ?
already assigned it, so I think it would be good to let him submit the PR, unless he unassigns or does not submit a PR in a reasonable amount of time.
Thanks I was planning on submitting the PR over the weekend
Hello, I am a first timer in open source contribution community. I think I made the required changes for that issue. However, I don't know how assignment rules work.


Question: # Wait for last batch of jobs kube::util::wait-for-jobs || { <del> echo -e "${color_red}Some commands failed.${color_norm}" >&2 </del> <ins> code=$? echo -e "${color_red}Failed to create firewall rule.${color_norm}" >&2 exit $code </ins> } }

Kubernetes e2e suite [It] [sig-network] Firewall rule should have correct firewall rules for e2e cluster See also a similar cluster with the same test failing: /kind failing-test /kind flake /sig network
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: framework.SkipUnlessProviderIs("gke") nsFlag := fmt.Sprintf("--namespace=%v", ns) podJson := readTestFileOrDie(kubectlInPodFilename) <ins> // Replace the host path to kubectl in json podString := strings.Replace(string(podJson), "$KUBECTL_PATH", framework.TestContext.KubectlPath, 1) </ins> By("validating api verions") <del> framework.RunKubectlOrDieInput(string(podJson), "create", "-f", "-", nsFlag) </del> <ins> framework.RunKubectlOrDieInput(podString, "create", "-f", "-", nsFlag) </ins> err := wait.PollImmediate(time.Second, time.Minute, func() (bool, error) { output := framework.RunKubectlOrDie("get", "pods/kubectl-in-pod", nsFlag) if strings.Contains(output, "Running") {

https://k8s- Failed: [] Kubectl client [] Kubectl should be able to talk to api server kubectl running in a pod could talk to api server [Conformance] {Kubernetes e2e suite}
https://k8s- Failed: [] Kubectl client [] Kubectl should be able to talk to api server kubectl running in a pod could talk to api server [Conformance] {Kubernetes e2e suite}
https://k8s- Failed: [] Kubectl client [] Kubectl should be able to talk to api server kubectl running in a pod could talk to api server [Conformance] {Kubernetes e2e suite}
https://k8s- Failed: [] Kubectl client [] Kubectl should be able to talk to api server kubectl running in a pod could talk to api server [Conformance] {Kubernetes e2e suite}
https://k8s- Failed: [] Kubectl client [] Kubectl should be able to talk to api server kubectl running in a pod could talk to api server [Conformance] {Kubernetes e2e suite}
https://k8s- Failed: [] Kubectl client [] Kubectl should be able to talk to api server kubectl running in a pod could talk to api server [Conformance] {Kubernetes e2e suite}
https://k8s- Failed: [] Kubectl client [] Kubectl should be able to talk to api server kubectl running in a pod could talk to api server [Conformance] {Kubernetes e2e suite}
https://k8s- Failed: [] Kubectl client [] Kubectl should be able to talk to api server kubectl running in a pod could talk to api server [Conformance] {Kubernetes e2e suite}
https://k8s- Failed: [] Kubectl client [] Kubectl should be able to talk to api server kubectl running in a pod could talk to api server [Conformance] {Kubernetes e2e suite}
https://k8s- Failed: [] Kubectl client [] Kubectl should be able to talk to api server kubectl running in a pod could talk to api server [Conformance] {Kubernetes e2e suite}
https://k8s- Failed: [] Kubectl client [] Kubectl should be able to talk to api server kubectl running in a pod could talk to api server [Conformance] {Kubernetes


Question: framework.SkipUnlessProviderIs("gke") nsFlag := fmt.Sprintf("--namespace=%v", ns) podJson := readTestFileOrDie(kubectlInPodFilename) <ins> // Replace the host path to kubectl in json podString := strings.Replace(string(podJson), "$KUBECTL_PATH", framework.TestContext.KubectlPath, 1) </ins> By("validating api verions") <del> framework.RunKubectlOrDieInput(string(podJson), "create", "-f", "-", nsFlag) </del> <ins> framework.RunKubectlOrDieInput(podString, "create", "-f", "-", nsFlag) </ins> err := wait.PollImmediate(time.Second, time.Minute, func() (bool, error) { output := framework.RunKubectlOrDie("get", "pods/kubectl-in-pod", nsFlag) if strings.Contains(output, "Running") {

e2e suite}
https://k8s- Failed: [] Kubectl client [] Kubectl should be able to talk to api server kubectl running in a pod could talk to api server [Conformance] {Kubernetes e2e suite}
https://k8s- Failed: [] Kubectl client [] Kubectl should be able to talk to api server kubectl running in a pod could talk to api server [Conformance] {Kubernetes e2e suite}
https://k8s- Failed: [] Kubectl client [] Kubectl should be able to talk to api server kubectl running in a pod could talk to api server [Conformance] {Kubernetes e2e suite}
https://k8s- Failed: [] Kubectl client [] Kubectl should be able to talk to api server kubectl running in a pod could talk to api server [Conformance] {Kubernetes e2e suite}
https://k8s- Failed: [] Kubectl client [] Kubectl should be able to talk to api server kubectl running in a pod could talk to api server [Conformance] {Kubernetes e2e suite}
Are you the correct assignee for this? If so, can you please give an update. If not, can you please reassign it to the correct person. CC
https://k8s- Failed: [] Kubectl client [] Kubectl should be able to talk to api server kubectl running in a pod could talk to api server [Conformance] {Kubernetes e2e suite}
https://k8s- Failed: [] Kubectl client [] Kubectl should be able to talk to api server kubectl running in a pod could talk to api server [Conformance] {Kubernetes e2e suite}
https://k8s- Failed: [] Kubectl client [] Kubectl should be able to talk to api server kubectl running in a pod could talk to api server [Conformance] {Kubernetes e2e suite}
Yes, I guess so. I'm looking into it.


Question: framework.SkipUnlessProviderIs("gke") nsFlag := fmt.Sprintf("--namespace=%v", ns) podJson := readTestFileOrDie(kubectlInPodFilename) <ins> // Replace the host path to kubectl in json podString := strings.Replace(string(podJson), "$KUBECTL_PATH", framework.TestContext.KubectlPath, 1) </ins> By("validating api verions") <del> framework.RunKubectlOrDieInput(string(podJson), "create", "-f", "-", nsFlag) </del> <ins> framework.RunKubectlOrDieInput(podString, "create", "-f", "-", nsFlag) </ins> err := wait.PollImmediate(time.Second, time.Minute, func() (bool, error) { output := framework.RunKubectlOrDie("get", "pods/kubectl-in-pod", nsFlag) if strings.Contains(output, "Running") {

https://k8s- Failed: [] Kubectl client [] Kubectl should be able to talk to api server kubectl running in a pod could talk to api server [Conformance] {Kubernetes e2e suite}
https://k8s- Failed: [] Kubectl client [] Kubectl should be able to talk to api server kubectl running in a pod could talk to api server [Conformance] {Kubernetes e2e suite}
https://k8s- Failed: [] Kubectl client [] Kubectl should be able to talk to api server kubectl running in a pod could talk to api server [Conformance] {Kubernetes e2e suite}
https://k8s- Failed: [] Kubectl client [] Kubectl should be able to talk to api server kubectl running in a pod could talk to api server [Conformance] {Kubernetes e2e suite}
https://k8s- Failed: [] Kubectl client [] Kubectl should be able to talk to api server kubectl running in a pod could talk to api server [Conformance] {Kubernetes e2e suite}
https://k8s- Failed: [] Kubectl client [] Kubectl should be able to talk to api server kubectl running in a pod could talk to api server [Conformance] {Kubernetes e2e suite}
We need to get this fixed soon and picked up in release-1.5 since it is consistently red in
ack i have a PR currently running
https://k8s- Failed: [] Kubectl client [] Kubectl should be able to talk to api server kubectl running in a pod could talk to api server [Conformance] {Kubernetes e2e suite}
If you found the root cause, we will close my pending PR that printing out some debug info. What is the PR # of the commit you mentioned above?


Question: framework.SkipUnlessProviderIs("gke") nsFlag := fmt.Sprintf("--namespace=%v", ns) podJson := readTestFileOrDie(kubectlInPodFilename) <ins> // Replace the host path to kubectl in json podString := strings.Replace(string(podJson), "$KUBECTL_PATH", framework.TestContext.KubectlPath, 1) </ins> By("validating api verions") <del> framework.RunKubectlOrDieInput(string(podJson), "create", "-f", "-", nsFlag) </del> <ins> framework.RunKubectlOrDieInput(podString, "create", "-f", "-", nsFlag) </ins> err := wait.PollImmediate(time.Second, time.Minute, func() (bool, error) { output := framework.RunKubectlOrDie("get", "pods/kubectl-in-pod", nsFlag) if strings.Contains(output, "Running") {

https://k8s- Failed: [] Kubectl client [] Kubectl should be able to talk to api server kubectl running in a pod could talk to api server [Conformance] {Kubernetes e2e suite}
: here it is
https://k8s- Failed: [] Kubectl client [] Kubectl should be able to talk to api server kubectl running in a pod could talk to api server [Conformance] {Kubernetes e2e suite}
https://k8s- Failed: [] Kubectl client [] Kubectl should be able to talk to api server kubectl running in a pod could talk to api server [Conformance] {Kubernetes e2e suite}
We timeout waiting for pod sometimes. Is it reasonable to use a longer timeout? Use for in that test.
https://k8s- Failed: [] Kubectl client [] Kubectl should be able to talk to api server kubectl running in a pod could talk to api server [Conformance] {Kubernetes e2e suite}
https://k8s- Failed: [] Kubectl client [] Kubectl should be able to talk to api server kubectl running in a pod could talk to api server [Conformance] {Kubernetes e2e suite}
: sure thing, do you want to do that in a fresh PR?
I updated my pending PR .
https://k8s- Failed: [] Kubectl client [] Kubectl should be able to talk to api server kubectl running in a pod could talk to api server [Conformance] {Kubernetes e2e suite}


Question: podLabels := map[string]string{"name": "nginx"} replicas := 1 Logf("Creating simple deployment %s", deploymentName) <del> _, err := c.Deployments(ns).Create(newDeployment(deploymentName, replicas, podLabels, "nginx", "nginx")) </del> <ins> _, err := c.Deployments(ns).Create(newDeployment(deploymentName, replicas, podLabels, "nginx", "nginx", extensions.RollingUpdateDeploymentStrategyType)) </ins> Expect(err).NotTo(HaveOccurred()) defer func() { deployment, err := c.Deployments(ns).Get(deploymentName)

There's some practical limit on the number of concurrent access tokens that can be used in a Kubernetes cluster before it begins to spend all of its time in the authn stack verifying them. Absent fixes to it was < 500; with the fixes linked there it's still not infinitely scalable-- I still hit (what I believe to be) with ~ 5k unique tokens @ ~700QPS. This request is to this dimension an official name & definition, and list it with others, e.g. here: it it have a useful value (I think it should be 10k) /kind bug /sig scalability
/cc
oh and 4. add tests which verify it is as high as believed.
cc since projected tokens will increase the number of unique tokens
/sig auth
LRUs are not good at scaling and sharding has some major pitfalls as pointed out. I would suggest that we try to drop the LRU cache for successful authentication results in the token cache. The main benefit of using an LRU here is bound cache size. Entries in the token cache have a fast expiration independent of the LRU policy. The records probably take up O(2kb) (guesstimate) of memory. Given that these are successful attempts with presumably valid tokens, I don't think we are getting any benefit from the LRU. If we don't have a constraint on fixed cache size, we can back the token cache with a sync.Map and run a single background goroutine to periodically walk the map and remove expired entries. This would fix the problematic locking and eviction due to nonuniform distribution of tokens over shards. The failure cache is less important and we don't cache failures by default. If we consider this the "slowpath", we can probably leave it as. If that's not good enough, we have a couple options (with various tradeoffs) to improve it.
If off by default is not enough, more reasoning why the failed cache is not super useful: it might help with an accidental DoS but it doesn't prevent anything actively malicious DoS. I think we can focus our improvements on the success cache.
/assign He is already working on adding an explicit test for it.
has taken it over and is actively working on this. Unfortunately, I cannot assign this issue to him as he's not a k8s member yet
It's not closed - all testes must be submitted and appropriate documentation must be
Thanks everyone!!


Question: } return parameters, nil default: <ins> sort.Slice(objs, func(i, j int) bool { obj1, obj2 := objs[i].(*resourcev1alpha2.ResourceClassParameters), objs[j].(*resourcev1alpha2.ResourceClassParameters) if obj1 == nil || obj2 == nil { return false } return obj1.Name < obj2.Name }) </ins> return nil, statusError(logger, fmt.Errorf("multiple generated class parameters for %s.%s %s found: %s", class.ParametersRef.Kind, class.ParametersRef.APIGroup, klog.KRef(class.Namespace, class.ParametersRef.Name), klog.KObjSlice(objs))) } }

master-blocking: ci-kubernetes-unit It starts flaking today 30-05-2024 The test fail is This failure may be caused by the error message's order sensitivity. The error message's order of Resource Classes versus is different. N/A /sig node /sig scheduling
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-sig testing /sig scheduling (sig testing doesn't own all of the unit tests, just test frameworks, tooling, CI infra, etc) This appears to be a flake in scheduler unit tests which would be sig scheduling ( )
.. and it appears to be DRA specific subset of scheduling so /sig node as well
FYI - looks like this is happening every 3-6 hours give or take. Yesterday it was every six hours and today it seems to be every 3-4 hours. Looking into it a bit more, it looks like it may just be an issue with a connection via the test while it's trying to hit the server. I don't think the test itself is failing, I think the connection to the server is failing. ! CC
Hey All - hope you're well. Michael here from the Release Signal team. Just wanted to touch base and make sure we're good to go for the cut on the 4th. CC /sig-node


Question: ExitCode: 5, }, }, <del> map[string]error{"with-reason": testErrorReason, "succeed": testErrorReason}, []api.ContainerStatus{}, map[string]api.ContainerState{ </del> <ins> reasons:     map[string]error{"with-reason": testErrorReason, "succeed": testErrorReason}, oldStatuses: []api.ContainerStatus{}, expectedState: map[string]api.ContainerState{ </ins> "without-reason": {Terminated: &api.ContainerStateTerminated{ ExitCode:    1, ContainerID: emptyContainerID,

is this a Node bug? cc as next oncall to follow up on this.
This is a node bug due to init containers
I may need help debugging - I see:
So I think the problem is in the error case - we're not properly recording that the start failed, and the next time we come in we somehow think the first container succeeded. I'll add some debugging to see if we can't recreate it.
Let's discuss on instead.


Question: ExitCode: 5, }, }, <del> map[string]error{"with-reason": testErrorReason, "succeed": testErrorReason}, []api.ContainerStatus{}, map[string]api.ContainerState{ </del> <ins> reasons:     map[string]error{"with-reason": testErrorReason, "succeed": testErrorReason}, oldStatuses: []api.ContainerStatus{}, expectedState: map[string]api.ContainerState{ </ins> "without-reason": {Terminated: &api.ContainerStateTerminated{ ExitCode:    1, ContainerID: emptyContainerID,

Kubelet issues a docker inspect call after creating a container, but for very short lived containers there is a window in which they could exit before being inspected in the docker manager. If the container PID is 0, we should check the ExitCode and other relevant fields and handle them appropriately. [1] This is the suspected cause of several flakes: /cc
looking at the logs 2 weird things: race described second container ran, which suggests inconsistent reporting of initialized status of init1 and choosing init2 as the next container ( vs ). I think 1 got it right and 2 got it wrong. From logs: I'd expect the second one to not have run in this case? Node logs:
Init containers have an implicit assumption that container statuses include results of previous SyncPod invocations. If that's not the case we may need to fix that (or at least mitigate it). On May 18, 2016, at 8:28 PM, Prashanth B wrote: looking at the logs 2 weird things: race described second container ran, which suggests inconsistent reporting of initialized status of init1 and choosing init2 as the next container ( vs ). I think 1 got it right and 2 got it wrong. From logs: May 18 15:58:33.380: INFO: pod-init--1d4b-11e6-80a4- started at 2016-05-18T15:58:04-07:00 (2+1 container statuses recorded) May 18 15:58:33.380: INFO: Init container init1 ready: false, restart count 0 May 18 15:58:33.380: INFO: Init container init2 ready: true, restart count 0 I'd expect the second one to not have run in this case? Node logs: — You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub
This should be true. Is there any evidence that suggests the opposite? only shows conflicting view of whether init1 has run.
If the container exits already and it's not an infra container, kubelet should not error out. Although even if it does, I failed to see how this could affect the readiness of the init container without checking the code further.


Question: ExitCode: 5, }, }, <del> map[string]error{"with-reason": testErrorReason, "succeed": testErrorReason}, []api.ContainerStatus{}, map[string]api.ContainerState{ </del> <ins> reasons:     map[string]error{"with-reason": testErrorReason, "succeed": testErrorReason}, oldStatuses: []api.ContainerStatus{}, expectedState: map[string]api.ContainerState{ </ins> "without-reason": {Terminated: &api.ContainerStateTerminated{ ExitCode:    1, ContainerID: emptyContainerID,

Each syncPod invocation, we inspect the list of init container in order looking for lastTerminationState. There could be a bug in that logic, or if we didn't get the full status, we might make a bad decision. When determining which container to run next, we always look for the last init container in the chain that has success, then run the next container (unless the pod is RestartNever). , Yu-Ju Hong wrote:
Saw this flake again. How do we verify that container status is not lost between syn pods?
... between calls to syncPods
The second sync clearly saw the init1 container, and decided to launch init2. See "InitContainersToKeep" in the message below.
containers to keep can include failed containers. , Yu-Ju Hong wrote:
Discussed on slack. In summary, the problem has to do with how we convert the kubecontainer.PodStatus to the api.PodStatus. If a container is terminated, the function makes the decisions: the pod restart policy is "Always", and there is a "failure reason" recorded for the given container, this means the pod worker tried to (re)start the container but failed. Set the state to "Waiting" and populate the reason. , the container state is simply "terminated". Init containers don't obey the restart policy like regular containers, and the logic will need to be amended to work with them. is working on a PR. In addition to that, kubelet should not error out if a user container exited before it could apply the oom score (which is best effort in this case). fixes that.
Ahh, thanks for digging into the issue. This is the exact the concern I had when I reviewed the initial implementation of init container, which blocks pr for a while. I even asked to help me verify this case, unfortunately we didn't reproduce the issue back then. :-(
I wish I'd seen the flakes in the runs prior to merge.
We didn't notice this until we started running the tests against docker v1.11.1, which is much faster and hit the race condition differently.
I suppose that's a good thing...
It is hard to reproduce this.


Question: ExitCode: 5, }, }, <del> map[string]error{"with-reason": testErrorReason, "succeed": testErrorReason}, []api.ContainerStatus{}, map[string]api.ContainerState{ </del> <ins> reasons:     map[string]error{"with-reason": testErrorReason, "succeed": testErrorReason}, oldStatuses: []api.ContainerStatus{}, expectedState: map[string]api.ContainerState{ </ins> "without-reason": {Terminated: &api.ContainerStateTerminated{ ExitCode:    1, ContainerID: emptyContainerID,

The extra check of last terminated container in your initial code makes me nervous, but we couldn't reproduce it due to that small race windows until we have a faster docker running on the node. :-)
I got "Pods should invoke init containers on a RestartAlways pod" flake (referenced from ): Logs:
All the fixes went in. Please reopen this issue if you see any more flakes.


Question: // be sure to unset environment variable https_proxy (if exported) before testing, otherwise the testing will fail unexpectedly. func TestRoundTripSocks5AndNewConnection(t *testing.T) { <del> t.Skip("Flake https://issues.k8s.io/107708") </del> localhostPool := localhostCertPool(t) for _, redirect := range []bool{false, true} {

TestRoundTripSocks5AndNewConnection Disabled by No response No response No response /sig
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
there is a race shutting down the socks5 server


Question: input           *v1.Node want            bool }{ <ins> {want: false, input: &v1.Node{}}, {want: true, input: &v1.Node{Status: v1.NodeStatus{Conditions: []v1.NodeCondition{{Type: v1.NodeReady, Status: v1.ConditionTrue}}}}}, {want: false, input: &v1.Node{Status: v1.NodeStatus{Conditions: []v1.NodeCondition{{Type: v1.NodeReady, Status: v1.ConditionFalse}}}}}, {want: true, input: &v1.Node{Spec: v1.NodeSpec{Unschedulable: true}, Status: v1.NodeStatus{Conditions: []v1.NodeCondition{{Type: v1.NodeReady, Status: v1.ConditionTrue}}}}}, </ins> {want: true, input: &v1.Node{Status: validNodeStatus, ObjectMeta: metav1.ObjectMeta{Labels: map[string]string{}}}}, {want: true, input: &v1.Node{Status: validNodeStatus, ObjectMeta: metav1.ObjectMeta{Labels: map[string]string{labelNodeRoleMaster: ""}}}}, {want: true, input: &v1.Node{Status: validNodeStatus, ObjectMeta: metav1.ObjectMeta{Labels: map[string]string{labelNodeRoleExcludeBalancer: ""}}}},

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: In order to migrate to a new version of kubernetes on GKE. I cordoned(Kubectl cordon <node) all nodes in the old node pool(1.8.9) in order to drain the node pool(pods on the node) with an intention to cause the pods to run on a new version of the node pool(1.10.2). As soon as I cordoned all the nodes in the old node pool, it caused the downtime of ingress(nginx). The nginx controller pods where running fine on the cordoned node pool. But the cordon caused the nodes to be removed from the target pool of ingress load balancer. Below are the documents that suggest the above same method that I followed as best practice for zero downtime upgrade. 1). 2). What you expected to happen: I expected "kubectl cordon <node" not to have the node/s removed from the target pool of the loadbalancer because the nginx controller pods where running absolutely fine on these nodes. How to reproduce it (as minimally and precisely as possible): 1).Cordon all the nodes in an old node pool that runs nginx controller. Anything else we need to know?: Environment: Kubernetes version (use ): 1.10.2 Cloud provider or hardware configuration: GKE OS (e.g. from /etc/os-release): Container OS Kernel (e.g. ): 4.14.22+ Install tools: Others: Cordon causes to remove the nodes from the target pool of the ingress load balancer. The health checks fail on the nodes of the target pool, which should not be the case. Because the nginx controller pods run fine on these nodes.
Technically, this isn't a Kubernetes bug as this is working as intended; however, it does beg the question why we remove unschedulable nodes if they're marked as ready since we perform the to determine LB backend nodes. is it scheduable? is it not a master? is it not marked for exclusion? is it marked as ready?


Question: input           *v1.Node want            bool }{ <ins> {want: false, input: &v1.Node{}}, {want: true, input: &v1.Node{Status: v1.NodeStatus{Conditions: []v1.NodeCondition{{Type: v1.NodeReady, Status: v1.ConditionTrue}}}}}, {want: false, input: &v1.Node{Status: v1.NodeStatus{Conditions: []v1.NodeCondition{{Type: v1.NodeReady, Status: v1.ConditionFalse}}}}}, {want: true, input: &v1.Node{Spec: v1.NodeSpec{Unschedulable: true}, Status: v1.NodeStatus{Conditions: []v1.NodeCondition{{Type: v1.NodeReady, Status: v1.ConditionTrue}}}}}, </ins> {want: true, input: &v1.Node{Status: validNodeStatus, ObjectMeta: metav1.ObjectMeta{Labels: map[string]string{}}}}, {want: true, input: &v1.Node{Status: validNodeStatus, ObjectMeta: metav1.ObjectMeta{Labels: map[string]string{labelNodeRoleMaster: ""}}}}, {want: true, input: &v1.Node{Status: validNodeStatus, ObjectMeta: metav1.ObjectMeta{Labels: map[string]string{labelNodeRoleExcludeBalancer: ""}}}},

If any of those are a no, then it'll be removed from the pool on the next sync. Perhaps someone from can shed some light. /sig network
cc This problem extends beyond GCP/ingress controller and to anyone using pools to upgrade nodes. A typical workflow I see is you'd cordon a node pool so as you drain each node the pods don't go to another node in the pool that'll shortly be drained as well. The current logic does not support that and complicates cluster administration because cordoning does not respect PDB's.
/remove-sig apps /remove-sig gcp (since the service controller falls under sig networking)
Those labels are not set on the issue: <detailsIn response to : /remove-sig apps /remove-sig gcp Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Ref I thought a general practice for draining the old pool is to move over one node at a time in a rolling fashion. Shouldn't there always be certain backend pods being able to receive & serve traffic? cc
WRT draining, it is recommended to do it one node at a time ; however, in real world scenarios doing it one node at a time can cause a lot of avoidable container churn if you don't cordon off the nodes you at least plan to drain so that pods aren't being scheduled on a node you'll be draining after the next one. According to the comments in the service controller it seems the unschedulable part exists because it was how we differentiated masters in the past; however, we have labels for that now. We also have the service exclusion label (can't find when/if that'll be promoted though as i don't think there's a ticket on it).
I'm having problems with this issue as well. The fact that nodes are removed from target pools when cordoned means that I sometimes get brief interruptions of the traffic to a service when I cordon a node. I'm guessing the node isn't gracefully removed from the load balancer which results in some traffic being cut off trying to route through the recently cordoned node. The same problem occasionally happens when deleting nodes / node pools or when the cluster autoscaler deletes a node. We are running GKE ver. 1.10.2-gke.1.


Question: input           *v1.Node want            bool }{ <ins> {want: false, input: &v1.Node{}}, {want: true, input: &v1.Node{Status: v1.NodeStatus{Conditions: []v1.NodeCondition{{Type: v1.NodeReady, Status: v1.ConditionTrue}}}}}, {want: false, input: &v1.Node{Status: v1.NodeStatus{Conditions: []v1.NodeCondition{{Type: v1.NodeReady, Status: v1.ConditionFalse}}}}}, {want: true, input: &v1.Node{Spec: v1.NodeSpec{Unschedulable: true}, Status: v1.NodeStatus{Conditions: []v1.NodeCondition{{Type: v1.NodeReady, Status: v1.ConditionTrue}}}}}, </ins> {want: true, input: &v1.Node{Status: validNodeStatus, ObjectMeta: metav1.ObjectMeta{Labels: map[string]string{}}}}, {want: true, input: &v1.Node{Status: validNodeStatus, ObjectMeta: metav1.ObjectMeta{Labels: map[string]string{labelNodeRoleMaster: ""}}}}, {want: true, input: &v1.Node{Status: validNodeStatus, ObjectMeta: metav1.ObjectMeta{Labels: map[string]string{labelNodeRoleExcludeBalancer: ""}}}},

Our best defence against these issue have so far been to limit node changes as much as possible and not use any auto scaling.
I've posted a question regarding this issue in SO some weeks before it was created. The suggested action was to use node taints () instead of cordon, that way nodes will still be marked as .
Thank you for the link, it Certainly helped. Perhaps, if thats the case(as mentioned in the accepted answer of the above link), then to cordon all the nodes of old node pools should not be a recommended action, at least in the GKE upgrade docs.
only says: And I can't find what more it does in the documentation. So I would expect that it has exactly the same behaviour as a taint. Why is removing running pods from the balancer working as intended? If it isn't a bug then the documentation is certainly unclear in this regard.
This just caused a major outage for us too. I agree with the previous posts, while this may be working as the code intended, it does not follow the patterns and documentation k8s provides. When I cordon a node it should not impact the traffic going to the node or any pods on it. I should be able to gradually remove traffic by draining nodes/pods and respecting PDBs.
this issue touches on the fact we are removing masters as well and inadvertently removing cordoned nodes. The logic in the service controller mentions we only filter out unschedulable nodes since that's how masters were setup.
We wanted to replace all of the kubernetes cluster nodes to different node instance types, so we started off by cordoning all of the nodes, but it surprisingly caused an outage for us. As pointed out, the only documentation around cordoning is , which I took to mean that new pods can not be scheduled on those nodes. It was surprising behavior to find out that, not only does it not allow new pods to be scheduled on those nodes, but that it also causes the service controller to remove unschedulable nodes from load balancing pools. I understand that it's working as intended, but if that was also documented as part of the cordon operation, I would have been able to avoid an outage.
I finally caught up to this bug. Fun one. I agree that this is working-as-intended and that it stinks. We can do better.


Question: input           *v1.Node want            bool }{ <ins> {want: false, input: &v1.Node{}}, {want: true, input: &v1.Node{Status: v1.NodeStatus{Conditions: []v1.NodeCondition{{Type: v1.NodeReady, Status: v1.ConditionTrue}}}}}, {want: false, input: &v1.Node{Status: v1.NodeStatus{Conditions: []v1.NodeCondition{{Type: v1.NodeReady, Status: v1.ConditionFalse}}}}}, {want: true, input: &v1.Node{Spec: v1.NodeSpec{Unschedulable: true}, Status: v1.NodeStatus{Conditions: []v1.NodeCondition{{Type: v1.NodeReady, Status: v1.ConditionTrue}}}}}, </ins> {want: true, input: &v1.Node{Status: validNodeStatus, ObjectMeta: metav1.ObjectMeta{Labels: map[string]string{}}}}, {want: true, input: &v1.Node{Status: validNodeStatus, ObjectMeta: metav1.ObjectMeta{Labels: map[string]string{labelNodeRoleMaster: ""}}}}, {want: true, input: &v1.Node{Status: validNodeStatus, ObjectMeta: metav1.ObjectMeta{Labels: map[string]string{labelNodeRoleExcludeBalancer: ""}}}},

As a short-term workaround, you can taint the nodes before cordoning one-by-one. That will stop new work from arriving, but will leave old work there. If we agree on an appropriately named taint, we can have nginx tolerate it by default. Am I missing any reason why this would not work? If that flies, we should fix docs ( those are GCP docs). I am happy to help with that. Longer term maybe we can do something smarter. I am not sure what that would be, just now. Ideas?
Is there any reason we can't just leave the nodes in the load balancer pool when the node is cordoned? And when the pods are terminated the nodes can be removed? I may be missing some context here?
The presumption is that a node that is unschedulable is going away, or needs some other repair. Using it as a target for load-balancers means there could be sessions routed THROUGH that node. Even as the pods are drained. Right up until the node suddenly dies, causing an outage of a different flavor. What should really happen is that a node which becomes unschedulable starts draining connections. Once all the pods are moved off, the node can be removed from the pool entirely. I don't think any cloud providers, even GCP, handle this just right today. David Radcliffe <:
Wait, if you brought up a new nodepool, all those new nodes should be in the TargetPool. Cordoning the old nodepool should only terminate the existing connections, new connections should be fine. Something doesn't add up...
Was this using
Yes
Right, well, that explains it. So even draining won't do what you want - you need new connections to be allowed until you have capacity in the new pool. The good news is that you can do this entirely yourself, with taints. We just need nginx to tolerate the taint, which you can also specify yourself. The only real alternative I see would be, as you said, to leave nodes in the LB pool until the node dies. Given that we never specified this, it is probably different on every provider, which would make it hard to change (especially if the provider already provides clean drains while unschedulable).


Question: input           *v1.Node want            bool }{ <ins> {want: false, input: &v1.Node{}}, {want: true, input: &v1.Node{Status: v1.NodeStatus{Conditions: []v1.NodeCondition{{Type: v1.NodeReady, Status: v1.ConditionTrue}}}}}, {want: false, input: &v1.Node{Status: v1.NodeStatus{Conditions: []v1.NodeCondition{{Type: v1.NodeReady, Status: v1.ConditionFalse}}}}}, {want: true, input: &v1.Node{Spec: v1.NodeSpec{Unschedulable: true}, Status: v1.NodeStatus{Conditions: []v1.NodeCondition{{Type: v1.NodeReady, Status: v1.ConditionTrue}}}}}, </ins> {want: true, input: &v1.Node{Status: validNodeStatus, ObjectMeta: metav1.ObjectMeta{Labels: map[string]string{}}}}, {want: true, input: &v1.Node{Status: validNodeStatus, ObjectMeta: metav1.ObjectMeta{Labels: map[string]string{labelNodeRoleMaster: ""}}}}, {want: true, input: &v1.Node{Status: validNodeStatus, ObjectMeta: metav1.ObjectMeta{Labels: map[string]string{labelNodeRoleExcludeBalancer: ""}}}},

IIUC, we don't want new pods to land on the tainted/cordoned nodes. In that case, we can place "NoSchedule" taint on the nodes. We don't need any tolerations on nginx. This will prevent new instances of nginx from landing on the tainted nodes, but the existing ones will keep running there.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
This just surprised us also. I understand the technical reasons behind it but the command name and associated documentation is misleading. To me it isn't intuitive that the command does not remove compute (pods) but does remove some networking. I would expect cordon to keep new pods from getting scheduled to it and keep it from getting to new services/ELBs/etc. The command would then cause pods to be removed and also take the node out of networking paths. Almost need a new state for a node of that is clearly defined.
My main issue is with the docs, and also GKE docs. They should explain this pitfall. After I got aware of the issue I understand why it works as it does. A simple workaround is to taint the nodes you are going to rotate first.
Just chiming in; We as a GKE customer have today been bitten by this issue in production. The documentation of (in the context of GKE) is not extensive enough so we could be aware of this issue. This should be addressed. Luckily the outage was not severe because we were using the node-by-node approach (and accepted the container churn this would cause). We explicitly took this approach because our previous nodepool upgrade was a total outage also caused by in a different way. (for those interested in our total outage, not related to this issue): Due to the fact that also results in the re-calculating the amount of schedulable nodes and thus scaled the amount of pods in our cluster (in our case: from 110 back to 2) resulting for us in major DNS outage internally.


Question: input           *v1.Node want            bool }{ <ins> {want: false, input: &v1.Node{}}, {want: true, input: &v1.Node{Status: v1.NodeStatus{Conditions: []v1.NodeCondition{{Type: v1.NodeReady, Status: v1.ConditionTrue}}}}}, {want: false, input: &v1.Node{Status: v1.NodeStatus{Conditions: []v1.NodeCondition{{Type: v1.NodeReady, Status: v1.ConditionFalse}}}}}, {want: true, input: &v1.Node{Spec: v1.NodeSpec{Unschedulable: true}, Status: v1.NodeStatus{Conditions: []v1.NodeCondition{{Type: v1.NodeReady, Status: v1.ConditionTrue}}}}}, </ins> {want: true, input: &v1.Node{Status: validNodeStatus, ObjectMeta: metav1.ObjectMeta{Labels: map[string]string{}}}}, {want: true, input: &v1.Node{Status: validNodeStatus, ObjectMeta: metav1.ObjectMeta{Labels: map[string]string{labelNodeRoleMaster: ""}}}}, {want: true, input: &v1.Node{Status: validNodeStatus, ObjectMeta: metav1.ObjectMeta{Labels: map[string]string{labelNodeRoleExcludeBalancer: ""}}}},

Be warned that for a nodepool upgrade can have a lot of unwanted & unexpected side effects.
It looks like this happens only to AWS and GCP but not on Azure? Can anyone confirm this? I just tested in both AWS and Azure, and while I see logs like in AWS Kubernetes, I do not see any logs like that in Azure. Is it intended? If so isn't behavior different for each cloud provider? i.e.) cordoning a node in AWS Kubernetes prevents loadbalancer from routing traffic to the affected node but cordoning a node in Azure Kubernetes will still let the loadbalancer route traffic to the affected node. Is it intentional?
Besides running a test like the above, how would I determine what the behavior is on AWS? I haven't found anything in the node administration docs or the Service that would imply that would shift traffic away from a Pod. In fact, the doc for explicitly states, emphasis mine: If I hadn't come across this Issue due to reports from GKE, I suppose it would only have been a matter of time until we had a high-impact production outage. There's a disconnect between Kubernetes API and the LoadBalancer as to whether Pods running on a 'd node are Ready. From all the docs I've seen, I would expect that either a) would evict Pods over to a Node before setting the instance as , or b) LoadBalancer would not equate with
Hi All, What approaches people using too workaround this? From reading the comments I was hoping to follow the custom taint approach. However, I've noticed executes a cordon making it difficult to drain the node safely with this approach.
I was lucky in that I experienced this issue while cordoning our testing environment, so we didn't experience a production outage. I think what I'm going to try is the following... it's going to be time-consuming but I think it's the safest bet. Double the number of nodes in the server group to bring online the new nodes first Place a custom taint on each "legacy" node As you said, executes a first, so that's a no-go. This is the time-consuming part... for each "legacy" node I am going to each pod, and let my PDBs take care of maintaining minimum number of pods. I'm hopeful to try this approach this afternoon, I'll comment back here if I do!


Question: input           *v1.Node want            bool }{ <ins> {want: false, input: &v1.Node{}}, {want: true, input: &v1.Node{Status: v1.NodeStatus{Conditions: []v1.NodeCondition{{Type: v1.NodeReady, Status: v1.ConditionTrue}}}}}, {want: false, input: &v1.Node{Status: v1.NodeStatus{Conditions: []v1.NodeCondition{{Type: v1.NodeReady, Status: v1.ConditionFalse}}}}}, {want: true, input: &v1.Node{Spec: v1.NodeSpec{Unschedulable: true}, Status: v1.NodeStatus{Conditions: []v1.NodeCondition{{Type: v1.NodeReady, Status: v1.ConditionTrue}}}}}, </ins> {want: true, input: &v1.Node{Status: validNodeStatus, ObjectMeta: metav1.ObjectMeta{Labels: map[string]string{}}}}, {want: true, input: &v1.Node{Status: validNodeStatus, ObjectMeta: metav1.ObjectMeta{Labels: map[string]string{labelNodeRoleMaster: ""}}}}, {want: true, input: &v1.Node{Status: validNodeStatus, ObjectMeta: metav1.ObjectMeta{Labels: map[string]string{labelNodeRoleExcludeBalancer: ""}}}},

Same as myself! Pity neither of us have a story for Since my comment this morning I've been trying a few approaches, I've ended up with the following: Create a new node pool of equal size in GKE with the upgraded kube version Set the current node pool to not be able to grow beyond its current size - if your using regional nodes this can be annoying, let it bring up new nodes anyways and just have them taint as NoSchedule straight away Taint all old node pool nodes with NoSchedule Go through each old node pool node executing a taint of NoExecute which will cause the node to drain and go onto available nodes i.e. your newly provisioned node pool Ensure your nginx ingress is deployed with tolerances:
Hi All, I'm at a bit of a loss on this one. I've started using custom taints to drain the nodes without getting them removed from the load balancer. My application can hold a HTTP connection open for up to 600 seconds. I'm using nginx as a daemonset, have it configured with a keepalive timeout of 620s, and have . I drain the application first using a taint on each node. I have nginx on each node tolerate this taint and remove it separately at a later point using a different taint. I'm waiting 660s between drains of each node for each taint to ensure nothing is dropped - this all works fine as expected. However, once I kill the node pool or do a kubectl drain after the node is empty of all my stuff plus nginx ingress I start seeing connection issues. I'm a bit lost where to go at debugging them, best guess is something to do with kube-proxy. Can anyone shed some light on this? is there anything on the Google Layer 4 load balancer I'm missing here? Is there some way to correctly drain kube-proxy (or does that even make sense?)
This doesn't address the network issues you're seeing, but for those trying to "roll their own cordon" using taints: Using a NoExecute taint to "drain" the node won't be orderly and won't respect any PDBs. You want to replicate the logic from which tries to do evictions of every pod in a loop. (I realize this is a burdensome idea, but it is also the safest idea.)


Question: input           *v1.Node want            bool }{ <ins> {want: false, input: &v1.Node{}}, {want: true, input: &v1.Node{Status: v1.NodeStatus{Conditions: []v1.NodeCondition{{Type: v1.NodeReady, Status: v1.ConditionTrue}}}}}, {want: false, input: &v1.Node{Status: v1.NodeStatus{Conditions: []v1.NodeCondition{{Type: v1.NodeReady, Status: v1.ConditionFalse}}}}}, {want: true, input: &v1.Node{Spec: v1.NodeSpec{Unschedulable: true}, Status: v1.NodeStatus{Conditions: []v1.NodeCondition{{Type: v1.NodeReady, Status: v1.ConditionTrue}}}}}, </ins> {want: true, input: &v1.Node{Status: validNodeStatus, ObjectMeta: metav1.ObjectMeta{Labels: map[string]string{}}}}, {want: true, input: &v1.Node{Status: validNodeStatus, ObjectMeta: metav1.ObjectMeta{Labels: map[string]string{labelNodeRoleMaster: ""}}}}, {want: true, input: &v1.Node{Status: validNodeStatus, ObjectMeta: metav1.ObjectMeta{Labels: map[string]string{labelNodeRoleExcludeBalancer: ""}}}},

To add on to ,: One approach that does not use node taints to force migrate, is to change for the requirements of your app. A change in node selector (e.g. based on instance type), along with a PDB (e.g. 66% / out of 3 replicas availability) should migrate you over fine. This worked out better for me than to drain or add taints to nodes to force push apps out. After the migration completes, you should be able to the nodes, and delete the node pool. FWIW, I too saw connection issues during load testing of a migration, and at least on my setup it was due to using an AWS NLB for the NGINX Ingress Controller. Switching to a classic ELB instead worked, as the NGINX LB Service seems to hit some sort of by my accounts when using NLBs. self-plug: I happened to recently cover a similar topic in a that you may find relevant. Hope it helps
For better or worse, for in my case pods that are behind an ingress resource are using node affinity to preferably not schedule them onto the same node. Originally this was done to ensure traffic is balanced across the different nginx ingresses. It has had a positive affect in this case too. Very interesting! Thank you for sharing, I'm not sure that swapping out the Layer4 GCE load balancer is an option here, but you have me thinking. I'm wondering if the TCP keepalive between the LB and nginx is potentially keeping connections open. , and suggest that there may be a 600 second TCP keepalive going on. Will retest doing a sleep of 620s after nginx removal and before node drain.
Has anyone come up with a strategy to safely move workloads between two pools? Reading this bug, it seems there's no way to reliably do this without causing an outage if you follow the recommendation of + . In my case I'm not using an nginx ingress but services with type . From what I can see, adding to a node causes the Service Controller to consider the node not ready and thus removes it from a target group. This would mean that between and there is no way to add a taint to the node that will leave it in the LB until it's drained.


Question: input           *v1.Node want            bool }{ <ins> {want: false, input: &v1.Node{}}, {want: true, input: &v1.Node{Status: v1.NodeStatus{Conditions: []v1.NodeCondition{{Type: v1.NodeReady, Status: v1.ConditionTrue}}}}}, {want: false, input: &v1.Node{Status: v1.NodeStatus{Conditions: []v1.NodeCondition{{Type: v1.NodeReady, Status: v1.ConditionFalse}}}}}, {want: true, input: &v1.Node{Spec: v1.NodeSpec{Unschedulable: true}, Status: v1.NodeStatus{Conditions: []v1.NodeCondition{{Type: v1.NodeReady, Status: v1.ConditionTrue}}}}}, </ins> {want: true, input: &v1.Node{Status: validNodeStatus, ObjectMeta: metav1.ObjectMeta{Labels: map[string]string{}}}}, {want: true, input: &v1.Node{Status: validNodeStatus, ObjectMeta: metav1.ObjectMeta{Labels: map[string]string{labelNodeRoleMaster: ""}}}}, {want: true, input: &v1.Node{Status: validNodeStatus, ObjectMeta: metav1.ObjectMeta{Labels: map[string]string{labelNodeRoleExcludeBalancer: ""}}}},

Using doesn't work in this case since the thing I'm draining might just come back and I'm not sure if the service controller will still consider the node ready in that case, but I believe it does. The one strategy I can think of is using labels on the node and an anti-affinity on the pod. You'd then add the label to a node that would push the pod away and then delete the pod. As long as the anti affinity uses this should be safe to do in preparation of a migration. Once the pods have been deleted you could then the nodes.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/lifecycle frozen /remove-lifecycle stale /remove-lifecycle rotten
- As I mentioned in I set up a script that first taints all nodes in the old node pool and then cordon+taint one by one node. If your ingress services are redundant this should work well.
Is there a plan for cordon behavior to improve on GCP? We have relied on this behavior for years on our AWS clusters and it was very surprising to find out.
This is a bug with service load balancers - not something we should document. We defined a label in that cluster admins can use - Unschedulable has nothing to do with SLB inclusion. which is beta and on by default in 1.19.
Disagree. Unschedulable means "no new workloads". It has nothing to do with existing workloads, that is why we have drain (but we should have something in kube that communicates nodes are draining). I have opened to remove this check - we have an alternate approach now and the old logic is wrong (the comment in the code even says "this is for masters!" and that's wrong now too as part of the kep).
A node status condition?
Possibly, or a taint, or a label. Something concrete.
/milestone v1.19


Question: hasPVC = true pvc, err := pl.PVCLister.PersistentVolumeClaims(pod.Namespace).Get(pvcName) if err != nil { <del> // The error has already enough context ("persistentvolumeclaim "myclaim" not found") </del> <ins> // The error usually has already enough context ("persistentvolumeclaim "myclaim" not found"), // but we can do better for generic ephemeral inline volumes where that situation // is normal directly after creating a pod. if ephemeral && apierrors.IsNotFound(err) { err = fmt.Errorf("waiting for ephemeral volume controller to create the persistentvolumeclaim %q", pvcName) } </ins> return hasPVC, err }

When using a generic ephemeral volume, the volume binder in kube-scheduler typically processes a new pod before the generic volume controller in kube-controller-manager has a chance to create the PVC. This leads to events for pods using generic ephemeral volumes: It would be nice to suppress these warnings for a short while (duration to be determined) after the pod creation. Later they should be emitted because then they indicate that the generic volume controller isn't working properly. This makes nicer to use.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage


Question: Name: "foo", }, Data: map[string][]byte{ <del> v1.DockerConfigKey: secretDataNoEmail, </del> <ins> v1.DockerConfigJsonKey: secretDataNoEmail, </ins> }, <del> Type: v1.SecretTypeDockercfg, </del> <ins> Type: v1.SecretTypeDockerConfigJson, </ins> }, expectErr: false, },

/kind bug We're having trouble retrieving images from a private registry on DockerCloud. For nearly a year the following has worked flawlessly. Last Friday I pulled down the 1.9.0 CLI (via brew) and since then have not been able to access any image in a private registry on DockerCloud. I've tried this on 10+ brand new clusters created with kops v1.8.0. The Kubernetes cluster version have been 1.8.0, 1.8.4, 1.8.5 and 1.9.0 (which is not supported by kops yet) and all exhibit the same problem. We create the secret as such: And verify it using: Which yields: And create a test service with with these contents: The pod fails immediatly with: I've reviewed the 1.9.0 release notes and nothing jumps out as the root cause. For reference I've tried putting quotes around the value when creating the secret to no avail. I've also tried prefixing the value with , so the full value would read: also to no avail. Environment: Kubernetes version: Cloud provider and installation info: AWS cluster built with Kops 1.8.0, using (to isolate this issue) nearly the default settings for everything.
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I have same problem, and I "solved" it creating a docker-registry secret in an old kubernetes version and pasting it on new kubernetes
I'm recently (I know it worked perfectly about last month) facing issues with as well. For me it ended in a similar manner saying: I was able to confirm, that this is a kubernetes CLI related issue and not end user. Based on and this I was able to create properly formatted auth information looking like this: . When encoding this one-liner using (MacOS workstation) and apply like this works fine and images can be pulled from private repository again.


Question: Name: "foo", }, Data: map[string][]byte{ <del> v1.DockerConfigKey: secretDataNoEmail, </del> <ins> v1.DockerConfigJsonKey: secretDataNoEmail, </ins> }, <del> Type: v1.SecretTypeDockercfg, </del> <ins> Type: v1.SecretTypeDockerConfigJson, </ins> }, expectErr: false, },

However this breaks all automation and can't be considered as solution Confirmed working with kubectl 1.8.5 and server 1.8.4 (guess 1.7.9 as well)
looks like there's an extra "auths" element when adding the secret using 1.9 kubectl
Added in /assign
Many thanks for the quick fix on this. Any idea when this might get rolled out? Also, it might be a good idea if this was to the known issue list for v1.9.0. It would have saved us a bunch of time trying to track down the root cause. Just trying to make sure no one else needs to do the same.
linking to the "v1.9.0 known issues" list
ack, asking Mehdy since he is the patch manager for v1.9 branch. Mehdy, any idea when we can ship 1.9.1 with this fix?
workarounds for folks until 1.9.1 is released: if you have a .dockerconfigjson for your private registry already, you can manually specify the type and data key (e.g. ) if you don't have a .dockerconfigjson file already, you can fix up the secret produced by manually: the type from to the data key from to the modified secret with
1 will be released tomorrow
I can confirm solution works, thanks so much!
I am still having issues in 1.9.1. This worked in the 1.8.x branches: results in the following error:
you have to remove the gitlab-registry secret before you can apply an update... type is immutable, and the fix in 1.9.1 required changing the type of the secret


Question: ) func (c *FakeEvictions) Evict(eviction *policy.Eviction) error { <del> action := core.GetActionImpl{} action.Verb = "post" </del> <ins> action := core.CreateActionImpl{} action.Verb = "create" </ins> action.Namespace = c.ns action.Resource = schema.GroupVersionResource{Group: "", Version: "", Resource: "pods"} action.Subresource = "eviction" <ins> action.Object = eviction </ins> _, err := c.Fake.Invokes(action, eviction) return err }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: I'm trying to test Eviction functionality in my program, but running into some inconsistencies around how Evictions are handled in the fake SimpleClient in the client-go testing functionality. Most notably, creates a different from , and both are missing information. issues a with set to (seems to be the only Fake that has this mismatch) on the subresource of . It is missing all the extra information in the eviction, i.e. the . issues a on the subresource of the resource. This is wrong, because aren't created, they're only posted and handled immediately. The Action is missing a Namespace. What you expected to happen: I want to be able to add my own reactor to the SimpleClient to handle this special case. I propose we change both cases to a with set to the . This will ensure it's not handled by the current SimpleClient ObjectReaction, and makes it so users can implement whatever they want themselves. I'd be happy to make a PR with these changes. How to reproduce it (as minimally and precisely as possible): Output: Environment: git revision of : <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
POST requests are create requests. It doesn't matter if the object is persisted or not.
/sig api-machinery
cc


Question: export PATH=${GOPATH}/bin:${PWD}/third_party/etcd:/usr/local/go/bin:${PATH} <del> go get github.com/tools/godep && godep version go get github.com/jstemmer/go-junit-report </del> <ins> retry go get github.com/tools/godep && godep version retry go get github.com/jstemmer/go-junit-report </ins> # Enable the Go race detector. export KUBE_RACE=-race

Not sure if this concided with a Jenkins restart.
Not sure what we can do about this. Retry a few times?
GitHub seemed to hiccup last night and fail a bunch of our builds. I'm not sure a retry would've helped, unless we kept retrying for many minutes.
This is the sort of thing that just happens occasionally. We've seen this a lot in OpenShift CI runs. I'm not sure there's much you can do other than try to avoid external network access (e.g. bake tools into the VM image).
It seems to have occurred twice in the last 12h so retry w/ timeout can't hurt?
It can't hurt... but it may or may not help depending on the extent of the external issue
If the problem is really bad, we could toss it somewhere on GCS?
Looks like it failed on : We should probably put this in a retry loop or else do something more fancy


Question: // it is not used by Convert() other than storing it in the scope. // Not safe for objects with cyclic references! func (c *Converter) Convert(src, dest interface{}, flags FieldMatchingFlags, meta *Meta) error { <ins> return c.doConversion(src, dest, flags, meta, c.convert) } // DefaultConvert will translate src to dest if it knows how. Both must be pointers. // No conversion func is used. If the default copying mechanism // doesn't work on this type pair, an error will be returned. // Read the comments on the various FieldMatchingFlags constants to understand // what the 'flags' parameter does. // 'meta' is given to allow you to pass information to conversion functions, // it is not used by DefaultConvert() other than storing it in the scope. // Not safe for objects with cyclic references! func (c *Converter) DefaultConvert(src, dest interface{}, flags FieldMatchingFlags, meta *Meta) error { return c.doConversion(src, dest, flags, meta, c.defaultConvert) } type conversionFunc func(sv, dv reflect.Value, scope *scope) error func (c *Converter) doConversion(src, dest interface{}, flags FieldMatchingFlags, meta *Meta, f conversionFunc) error { </ins> dv, err := EnforcePtr(dest) if err != nil { return err

Files storing latest versions for minors are not updated since infix was to the version string. Example Similar issues can be found in 1.25 and 1.24 builds (last 1.23 build infix so it has not happened there yet): I expect the files to be overwritten with newer version of the build. Manually run build job for one of the minors. No response Not related. Not related. No response No response No response No response
I believe this may be also responsible for breaking kubemark tests: since they fail due to version mismatch
/sig release
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /priority high Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Thanks for raising the issue! /cc who is working on before
I assume this will be fixed automatically with the next patches, because then we would compare with . Anyway, I think we should fix that intermediately by manually updating the version markers to match their correct version. I did that now for the bucket
We had to update the versions markers again, because those were pointing to two commits earlier and didn't include the Go updates. The latest version markers are: v1.24.12-35+ v1.25.8-37+ v1.26.3-41+
latest-1.27 has been affected by this as well. I updated it manually to v1.27.0-9+. See this thread for more details:
/triage accepted
This should be fixed now, closing it. Feel free to reopen if we missed anything.
This issue is still present, see this Slack thread for more details: /reopen
Reopened this issue. <detailsIn response to : This issue is still present, see this Slack thread for more details: /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
We have a similar situation with latest-1.28 which is currently pointing to: . This is one commit before the release commit: ! It's mostly like that the release step didn't update version markers, we should look into this.


Question: // it is not used by Convert() other than storing it in the scope. // Not safe for objects with cyclic references! func (c *Converter) Convert(src, dest interface{}, flags FieldMatchingFlags, meta *Meta) error { <ins> return c.doConversion(src, dest, flags, meta, c.convert) } // DefaultConvert will translate src to dest if it knows how. Both must be pointers. // No conversion func is used. If the default copying mechanism // doesn't work on this type pair, an error will be returned. // Read the comments on the various FieldMatchingFlags constants to understand // what the 'flags' parameter does. // 'meta' is given to allow you to pass information to conversion functions, // it is not used by DefaultConvert() other than storing it in the scope. // Not safe for objects with cyclic references! func (c *Converter) DefaultConvert(src, dest interface{}, flags FieldMatchingFlags, meta *Meta) error { return c.doConversion(src, dest, flags, meta, c.defaultConvert) } type conversionFunc func(sv, dv reflect.Value, scope *scope) error func (c *Converter) doConversion(src, dest interface{}, flags FieldMatchingFlags, meta *Meta, f conversionFunc) error { </ins> dv, err := EnforcePtr(dest) if err != nil { return err

As of now, I'm going to manually update this version marker to unblock 1.28.1 patch release. I'll also take a look into this. /assign
The PR that fixed this issue got reverted so this is still relevant: /reopen
Reopened this issue. <detailsIn response to : The PR that fixed this issue got reverted so this is still relevant: /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
We're running into this issue again with stable-1.29 marker. It's currently pointing to which is It should instead point to I'm going to manually change it following the same procedure as the last time. Version marker updated to
This issue is labeled with but has not been updated in over 90 days, and should be re-triaged. Important-soon issues must be staffed and worked on either currently, or very soon, ideally in time for the next release. You can: Confirm that this issue is still relevant with (org members only) Deprioritize it with or Close this issue with For more details on the triage process, see /remove-triage accepted
/triage accepted
Same issue for 1.30, where I had to update the version marker manually again.


Question: FindBoundSatsified:   false, }, eventReason: "FailedScheduling", <del> expectError: makePredicateError("1 VolumeBindingNoMatch, 1 VolumeNodeAffinityConflict"), </del> <ins> expectError: makePredicateError("1 node(s) didn't find available persistent volumes to bind, 1 node(s) had volume node affinity conflict"), </ins> }, "unbound,found-matches": { volumeBinderConfig: &persistentvolume.FakeVolumeBinderConfig{

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind feature The is not user friendly. There has been to make it better, but we still print in the pending message. We should change these errors to a more human readable message. For example, should be changed to something like . /sig scheduling


Golang 1.13 uses TLS 1.13 by default (opt-out), which introduces RSA-PSS. RSA-PSS does not work with keys that are too small. The issue was uncovered during Fedora mass rebuild which uses Golang 1.13: See /sig api-machinery
What is the failure mode if a smaller key is used?
/sig auth /sig scalability
/sub


Question: <del> /* Copyright 2022 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package plugin import ( "context" "errors" "fmt" "net" "sync" "time" "google.golang.org/grpc" "google.golang.org/grpc/connectivity" "google.golang.org/grpc/credentials/insecure" utilversion "k8s.io/apimachinery/pkg/util/version" "k8s.io/klog/v2" drapb "k8s.io/kubelet/pkg/apis/dra/v1alpha4" ) // NewDRAPluginClient returns a wrapper around those gRPC methods of a DRA // driver kubelet plugin which need to be called by kubelet. The wrapper // handles gRPC connection management and logging. Connections are reused // across different NewDRAPluginClient calls. func NewDRAPluginClient(pluginName string) (*Plugin, error) { if pluginName == "" { return nil, fmt.Errorf("plugin name is empty") } existingPlugin := draPlugins.get(pluginName) if existingPlugin == nil { return nil, fmt.Errorf("plugin name %s not found in the list of registered DRA plugins", pluginName) } return existingPlugin, nil } type Plugin struct { backgroundCtx context.Context cancel        func(cause error) mutex                   sync.Mutex conn                    *grpc.ClientConn endpoint                string highestSupportedVersion *utilversion.Version clientCallTimeout       time.Duration } func (p *Plugin) getOrCreateGRPCConn() (*grpc.ClientConn, error) { p.mutex.Lock() defer p.mutex.Unlock() if p.conn != nil { return p.conn, nil } ctx := p.backgroundCtx logger := klog.FromContext(ctx) network := "unix" logger.V(4).Info("Creating new gRPC connection", "protocol", network, "endpoint", p.endpoint) // grpc.Dial is deprecated. grpc.NewClient should be used instead. // For now this gets ignored because this function is meant to establish // the connection, with the one second timeout below. Perhaps that // approach should be reconsidered? //nolint:staticcheck conn, err := grpc.Dial( p.endpoint, grpc.WithTransportCredentials(insecure.NewCredentials()), grpc.WithContextDialer(func(ctx context.Context, target string) (net.Conn, error) { return (&net.Dialer{}).DialContext(ctx, network, target) }), ) if err != nil { return nil, err } ctx, cancel := context.WithTimeout(context.Background(), time.Second) defer cancel() if ok := conn.WaitForStateChange(ctx, connectivity.Connecting); !ok { return nil, errors.New("timed out waiting for gRPC connection to be ready") } p.conn = conn return p.conn, nil } func (p *Plugin) NodePrepareResources( ctx context.Context, req *drapb.NodePrepareResourcesRequest, opts ...grpc.CallOption, ) (*drapb.NodePrepareResourcesResponse, error) { logger := klog.FromContext(ctx) logger.V(4).Info("Calling NodePrepareResources rpc", "request", req) conn, err := p.getOrCreateGRPCConn() if err != nil { return nil, err } ctx, cancel := context.WithTimeout(ctx, p.clientCallTimeout) defer cancel() nodeClient := drapb.NewNodeClient(conn) response, err := nodeClient.NodePrepareResources(ctx, req) logger.V(4).Info("Done calling NodePrepareResources rpc", "response", response, "err", err) return response, err } func (p *Plugin) NodeUnprepareResources( ctx context.Context, req *drapb.NodeUnprepareResourcesRequest, opts ...grpc.CallOption, ) (*drapb.NodeUnprepareResourcesResponse, error) { logger := klog.FromContext(ctx) logger.V(4).Info("Calling NodeUnprepareResource rpc", "request", req) conn, err := p.getOrCreateGRPCConn() if err != nil { return nil, err } ctx, cancel := context.WithTimeout(ctx, p.clientCallTimeout) defer cancel() nodeClient := drapb.NewNodeClient(conn) response, err := nodeClient.NodeUnprepareResources(ctx, req) logger.V(4).Info("Done calling NodeUnprepareResources rpc", "response", response, "err", err) return response, err } </del>

The PR above moved everything related to into because that was where most of the code already was, but didn't renamed the files to avoid a larger diff and merge conflicts. Once is merged, we can do the renaming. Code clarity. /sig node
/triage accepted /priority important-soon
/assign
a PR can be created for this now.
will do


Question: // EnvVarSource represents a source for the value of an EnvVar. message EnvVarSource { <del> // Selects a field of the pod: supports metadata.name, metadata.namespace, metadata.labels, metadata.annotations, </del> <ins> // Selects a field of the pod: supports metadata.name, metadata.namespace, `metadata.labels['<KEY>']`, `metadata.annotations['<KEY>']`, </ins> // spec.nodeName, spec.serviceAccountName, status.hostIP, status.podIP, status.podIPs. // +optional optional ObjectFieldSelector fieldRef = 1;

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: When I tried to create a pod with env variables referencing to annotations: What you expected to happen: Should Proceed. As in It stated is a valid option How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): v1.18.2 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig docs
The API docs need to be updated to indicate the correct supported values. See for correct values
You are a lifesaver. Thank you!
/assign I can try to update the api documentation


Question: allErrs = append(allErrs, apivalidation.ValidateObjectMeta(&obj.ObjectMeta, false, ValidateThirdPartyResourceName, field.NewPath("metadata"))...) versions := sets.String{} <ins> if len(obj.Versions) == 0 { allErrs = append(allErrs, field.Required(field.NewPath("versions"), "must specify at least one version")) } </ins> for ix := range obj.Versions { version := &obj.Versions[ix] if len(version.Name) == 0 {

FYI. Currently the field of TPR is marked as which is wrong, actually if you try to create a TPR with empty versions, it will cause api server panic. Versions should be necessary field of TPR, we should validate this somehow.


Question: } else if exists { glog.Infof("Deleting old LB for previously uncached service %s whose endpoint %s doesn't match the service's desired IPs %v", namespacedName, endpoint, service.Spec.PublicIPs) <del> if err := s.ensureLBDeleted(service); err != nil { </del> <ins> if err := s.balancer.EnsureTCPLoadBalancerDeleted(s.loadBalancerName(service), s.zone.Region); err != nil { </ins> return err, retryable } }

All of our external load balancer tests failed for the better part of a day due to the GCE project that our tests run in hitting its quota for number of target pools. While the target pools have been deleted, the fact that we hit our quota indicates that there must be a leak somewhere, and we need to plug it.
For reference, the previous issue was
Could the leakage be related to ? The quota exhaustion occurred about 22 hrs after that.
I'd guess that this is due to GetTCPLoadBalancer returning that the load balancer doesn't exist if the forwarding rule is gone, ignoring the target pool, while the service controller considers its deleting done if the cloud provider says that the load balancer doesn't exist. I that logic to the cloud provider because the other implementations of DeleteTCPLoadBalancer weren't/won't be idempotent in the sense of returning ok if the LB doesn't exist. To fix it, we'll either need to make sure all Delete implementations are idempotent and remove the Get call in the controller, or add a method along the lines of IsTCPLoadBalancerDeleted to ask the cloud provider whether all pieces of it have been completely cleaned up. We can't make GetTCPLoadBalancer check that because it's relied on to tell us whether or not the balancer is fully created.
Copying over my commentary from on why the GKE e2e tests are leaking so many forwarding rules and target pools vs the GCE tests:
Is there a way for the services test to wait for the service controller to clean up the external resources before finishing (without just doing a sleep)?
Not without polling GCE directly for whether the resources still exist, although that may be worth it as a workaround. What would be perfect is that clean-up-all-cloud-provider-resources API method that we talked about a couple months ago.
Can we not perhaps use GCE namespaces or projects as a cheap way of getting this here?
I don't believe so. We can't use projects due to the fact that projects can't be programatically created (they're intended to be long-lived). I'm not aware of anything along the lines of a GCE namespace other than networks, which won't work because target pools and forwarding rules unfortunately don't seem to be scoped to networks.


Question: } else if exists { glog.Infof("Deleting old LB for previously uncached service %s whose endpoint %s doesn't match the service's desired IPs %v", namespacedName, endpoint, service.Spec.PublicIPs) <del> if err := s.ensureLBDeleted(service); err != nil { </del> <ins> if err := s.balancer.EnsureTCPLoadBalancerDeleted(s.loadBalancerName(service), s.zone.Region); err != nil { </ins> return err, retryable } }

The target pool leak in the code might be fixed, but the forwarding rule + target pool leak from deleting GKE clusters too quickly is certainly still happening. Reopening for the sake of that half of the issue.
Can we just add a short sleep to jenkins after the last test has run before it deletes anything?
Yes, you could add a sleep to Ginkgo's AfterEachSuite(): But I'd be concerned about what the upper bound on the sleep period is. Can we get the tests and code in sync with a 1 minute latency here? (e.g. if the controller garbage collects every 60 seconds, then the test only needs to sleep for 60 sec?) And add a BEEG comment to the test's sleep config noting that increasing the sleep time will not improve things, otherwise I'm pretty sure that someone noting flakes in this test in future will think "maybe I should just increase the sleep period a little bit."
Actually, thinking about this a little more, it would be better to put the "sleep" logic in the k8s master teardown procedure (i.e. not in the e2e tests or in Jenkins), because otherwise users are going to run into the same resource leakage problems as we are. So master deletion blocks on cleanup of backend cloud resources.
There's no way to block master deletion from the master itself. The test framework (or user) just pulls the rug out from underneath the master. See
Robby's right -- is the real solution to this, but until we have that the best we can do is to have the wait happen elsewhere. The options seem to be: GKE itself Ginkgo's post-testing logic Jenkins' teardown logic I don't like putting it in GKE, because arbitrarily slowing down our API makes for a poor experience for the majority of people that don't still have resources being deleted. I'm not eager to put it in Gingko either, since won't that sleep also take effect whenever anyone runs an e2e test locally against a dev cluster? Putting it in the Jenkins logic makes the most sense to me. I'll send a PR and cc you.
We may also want to change the kube-down logic to bring down the master and then nodes in parallel to both speed it up and more closely resemble GKE's teardown logic.
I've deleted the old forwarding rules and target pools from the project.


Question: } else if exists { glog.Infof("Deleting old LB for previously uncached service %s whose endpoint %s doesn't match the service's desired IPs %v", namespacedName, endpoint, service.Spec.PublicIPs) <del> if err := s.ensureLBDeleted(service); err != nil { </del> <ins> if err := s.balancer.EnsureTCPLoadBalancerDeleted(s.loadBalancerName(service), s.zone.Region); err != nil { </ins> return err, retryable } }

I'll periodically check to see whether anything is leaked after this point. I apologize if the deletions affected any running tests - everything looked to only be from GKE and parallel e2e runs.


Question: }, "../examples/openshift-origin": { "openshift-origin-namespace": &api.Namespace{}, <del> "openshift-controller":       &api.ReplicationController{}, </del> <ins> "openshift-controller":       &extensions.Deployment{}, </ins> "openshift-service":          &api.Service{}, <del> "etcd-controller":            &api.ReplicationController{}, </del> <ins> "etcd-controller":            &extensions.Deployment{}, </ins> "etcd-service":               &api.Service{}, <del> "etcd-discovery-controller":  &api.ReplicationController{}, </del> <ins> "etcd-discovery-controller":  &extensions.Deployment{}, </ins> "etcd-discovery-service":     &api.Service{}, "secret":                     nil, },

Kubernetes version (use ): Environment: Cloud provider or hardware configuration: Vagrant OS (e.g. from /etc/os-release): CoreOS Kernel (e.g. ):Install tools: Vagrant & Others: What happened: What you expected to happen: Success How to reproduce it (as minimally and precisely as possible): Followed instructions at Tried to run Received: Anything else do we need to know: From googling, it seems several of the fields in the config are for when using OpenShift origin. But creating the replication controller is a prerequisite to launching OpenShift Origin. Chicken & egg problem.


Question: return time.Since(start).Seconds() } <del> func newPodAndContainerCollector(containerCache kubecontainer.RuntimeCache) *podAndContainerCollector { return &podAndContainerCollector{ containerCache: containerCache, } } // Custom collector for current pod and container counts. type podAndContainerCollector struct { // Cache for accessing information about running containers. containerCache kubecontainer.RuntimeCache } // TODO(vmarmol): Split by source? var ( runningPodCountDesc = prometheus.NewDesc( prometheus.BuildFQName("", KubeletSubsystem, "running_pod_count"), "Number of pods currently running", nil, nil) runningContainerCountDesc = prometheus.NewDesc( prometheus.BuildFQName("", KubeletSubsystem, "running_container_count"), "Number of containers currently running", nil, nil) ) // Describe implements Prometheus' Describe method from the Collector interface. It sends all // available descriptions to the provided channel and retunrs once the last description has been sent. func (pc *podAndContainerCollector) Describe(ch chan<- *prometheus.Desc) { ch <- runningPodCountDesc ch <- runningContainerCountDesc } // Collect implements Prometheus' Collect method from the Collector interface. It's called by the Prometheus // registry when collecting metrics. func (pc *podAndContainerCollector) Collect(ch chan<- prometheus.Metric) { runningPods, err := pc.containerCache.GetPods() if err != nil { klog.Warningf("Failed to get running container information while collecting metrics: %v", err) return } runningContainers := 0 for _, p := range runningPods { runningContainers += len(p.Containers) } ch <- prometheus.MustNewConstMetric( runningPodCountDesc, prometheus.GaugeValue, float64(len(runningPods))) ch <- prometheus.MustNewConstMetric( runningContainerCountDesc, prometheus.GaugeValue, float64(runningContainers)) } </del> const configMapAPIPathFmt = "/api/v1/namespaces/%s/configmaps/%s" func configLabels(source *corev1.NodeConfigSource) (map[string]string, error) {

Some kubelet metrics use non-standard prometheus collectors unnecessarily. Specifically, we have which can be simple gauges. Since we cannot migrate non-standard prometheus collectors using the internal metrics stability framework, we will need to convert these metrics into standard prometheus variants. /sig instrumentation /assign
GitHub didn't allow me to assign the following users: solodov. Note that only and repo collaborators can be assigned and that issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : Some kubelet metrics use non-standard prometheus collectors unnecessarily. Specifically, we have which can be simple gauges. Since we cannot migrate non-standard prometheus collectors using the internal metrics stability framework, we will need to convert these metrics into standard prometheus variants. /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
recommends that we generate those metrics in the GenericPLEG during the relist. We can generate the data directly from the pods call there ().
I would be interested to work on this issue if no one else is working on it currently
Go for it!
/assign


Question: } for n := range visitedNamespaces { <ins> if len(o.Namespace) != 0 && n != o.Namespace { continue } </ins> for _, m := range namespacedRESTMappings { if err := p.prune(n, m); err != nil { return fmt.Errorf("error pruning namespaced object %v: %v", m.GroupVersionKind, err)

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Actually, the parameter depends to all namespaces in the cluster. Filtering is only possible with labels. When running the should only depend on the objects defined in . Why is this needed: A typical use-case is having one repository and namespace per project. When working with declarative objects it should be possible to remove objects and get them pruned like having them described in the respository. This would also help when renaming objects within the repository - which actually needs a manual on the host for the old objects. This even could be a better solution for the problem described in
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig api-machinery
cc
/assign
GitHub didn't allow me to assign the following users: Liujingfang1. Note that only and repo collaborators can be assigned and that issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
We have been aware of the issues in prune and we want to improve it. Recently we opened a for improving pruning in . With the proposed solution, the problem from current will be fixed.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or .


Question: } for n := range visitedNamespaces { <ins> if len(o.Namespace) != 0 && n != o.Namespace { continue } </ins> for _, m := range namespacedRESTMappings { if err := p.prune(n, m); err != nil { return fmt.Errorf("error pruning namespaced object %v: %v", m.GroupVersionKind, err)

/lifecycle stale
/remove-lifecycle stale
I just deleted a bunch of things in another team's namespace (:blush:). I'd would love to have this feature. I had been using dry-run in another context, but then forgot to check it again after switching contexts.
Is there any workaround for this? I want to prune resources within a single namespace only.
We're still working on improving prune, the current mechanism is broken enough that we don't recommend people us it.
Also look like it's been fixed recently.
Can someone confirm if this is implemented? I tried using this feature with kubectl 1.18.2, which has , yet as with previous versions kubectl it returned an error about pruning a nonNamespaced object of kind Namespace.
I assume the problem is that "Namespace" is a non-namespaced resource, and therefore is not filtered by namespace, so gets deleted by "prune". As a workaround it's possible to manually pass a (whitelist of resources allowed to be deleted), and not include "Namespace" in it: The list of resources is taken from .
Would wrapping this loop with achieve the same result?


Question: }, { "ImportPath": "github.com/evanphx/json-patch", <del> "Rev": "ed7cfbae1fffc071f71e068df27bf4f0521402d8" </del> <ins> "Rev": "94e38aa1586e8a6c8a75770bddf5ff84c48a106b" </ins> }, { "ImportPath": "github.com/exponent-io/jsonpath",

/kind bug What happened: Wrote a that returned a faulty json patch which specified an our of range index (e.G. when there is only one container). This resulted in the apiserver crashing. Stacktrace: What you expected to happen: No crash /sig-api-machinery
/sig api-machinery
This maybe a bug in . I will follow this question.
This bug had been fixed in PR: you can checkout it.
Thanks for your comment but I don't think the linked PR fixed the issue, as it got merged almost a year ago and I am running Kube 1.10.1 which does contain that patch but still panics. Just from quickly looking at the code I the linked PR I believe there are two things wrong: The check if the index is valid only happens when a negative index was given should be
Once the PR of is merged I will update kubernetes.
/kind bug


Question: if (endpoints.subsets != null && !endpoints.subsets.isEmpty()){ for (Subset subset : endpoints.subsets) { for (Address address : subset.addresses) { <del> list.add(InetAddress.getByName(address.IP)); </del> <ins> list.add(InetAddress.getByName(address.ip)); </ins> } } } <del> } </del> <ins> logger.info("Available endpoints: " + list); } else { logger.warn("Endpoints are not available"); } </ins> } catch (IOException | NoSuchAlgorithmException | KeyManagementException ex) { logger.warn("Request to kubernetes apiserver failed", ex); }

The namespace seems to be hard coded :
cc:
Thanks. We can improve it after v1 release.