Question: glog.Infof("managing multiple zones: %v", managedZones) } <del> operationPollRateLimiter := flowcontrol.NewTokenBucketRateLimiter(10, 100) // 10 qps, 100 bucket size. </del> return &GCECloud{ <del> service:                  svc, containerService:         containerSvc, projectID:                projectID, region:                   region, localZone:                zone, managedZones:             managedZones, networkURL:               networkURL, nodeTags:                 nodeTags, useMetadataServer:        useMetadataServer, operationPollRateLimiter: operationPollRateLimiter, </del> <ins> service:           svc, containerService:  containerSvc, projectID:         projectID, region:            region, localZone:         zone, managedZones:      managedZones, networkURL:        networkURL, nodeTags:          nodeTags, useMetadataServer: useMetadataServer, </ins> }, nil }

On a large cluster, the routecontroller basically makes O(n) requests all at once, and most of them error out with "Rate Limit Exceeded". We then wait for operations to complete on the ones that did stick, and damn the rest, they'll be caught next reconcile (which is a while). In fact, in some cases it appears we're spamming hard enough to get rate limited by the upper-level API DOS protections and seeing something akin to , e.g.: We can do better than this!
c.f. , which is the other leg of this problem
I took a look into code and it seems that we are doing full reconciliation every 10s. If we need to create 2000 routes at the beginning it seems to frequently. I think that we need to: first compute how many routes we have to create based on that try to spread them over some time to reduce possibility of causing 'rate limit exceeded' I will try to put some small PR together as a starting point.
I'm reopening this one, since this doesn't seem to be solved. So basically something strange is happening here. With PR we have 10/s limit on API calls send to GCE. And we are still rate-limitted. However, according to documentation we are allowed for 20/s: So something either doesn't work or is weird.
What is super interesting from the logs of recent run: It starts to creating routes: After 2 minutes from that, we are hitting "Rate limit Exceeded" for the first time: And only after 9 minutes we create the first route ever. The second one is created after 15 minutes, and from that point we create a number of them So it seems that nothing useful is happening for the first 2 minutes, and then between 3m30 and 9m30
OK - so I think that what is happening is that all controllers are sharing GCE interface (and thuse they have common throttling). So if one controller is generating a lot of API calls, then other controllers may be throttled. has a hypothesis that it may be caused by nodecontroller.
One thing my PR didn't address (because it was late): Now we have some obnoxious differences in logging when you go to make a request, because the versus could actually be absorbing a fair amount of ratelimiter.


Question: glog.Infof("managing multiple zones: %v", managedZones) } <del> operationPollRateLimiter := flowcontrol.NewTokenBucketRateLimiter(10, 100) // 10 qps, 100 bucket size. </del> return &GCECloud{ <del> service:                  svc, containerService:         containerSvc, projectID:                projectID, region:                   region, localZone:                zone, managedZones:             managedZones, networkURL:               networkURL, nodeTags:                 nodeTags, useMetadataServer:        useMetadataServer, operationPollRateLimiter: operationPollRateLimiter, </del> <ins> service:           svc, containerService:  containerSvc, projectID:         projectID, region:            region, localZone:         zone, managedZones:      managedZones, networkURL:        networkURL, nodeTags:          nodeTags, useMetadataServer: useMetadataServer, </ins> }, nil }

It could before because of the operation rate limiter and operation polling anyways, but now could be worse.
- yes I'm aware of that. And probably this is exactly the case here.
I got fooled by that - I guess we should fix it somehow.
One approach to fixing it is a similar approach to the operation polling code, but in the : warn if we ratelimited for more than N seconds (and dump the request or something? not clear how to clarify what we were sending).
I found something else - Kubelet is also building GCE interface: We have 2000 kubelets, so if all of them send multiple requests to GCE, we may hit limits.
Any idea in what circumstances Kubelet will contact cloud provider?
Hmm - it seems it contacts GCE exactly once at the beginning. So it doesn't explain much.
I think it's only here: I've rarely seen GCE ratelimit readonly calls, in practice, and kubelets come up pretty well staggered.
OK - so with logs that I , it is pretty clear that we are very heavily throttled on GCE api calls. In particular, when I was running 1000-node cluster, I see lines where we were throttle for 1m30. I'm pretty sure it's significantly higher in 2000-node cluster. I'm going to investigate a bit more where all those reuqests come from.
Actually - it seems that we have a pretty big problem here. Basically, Kubelet seems to send a lot of requests to GCE. So to clarify: every time Kubelet is updating NodeStatus (so every 10s by default) it is sending an API call to GCE to get it's not addresses: that means, that if we have 2000 nodes, kubelets generate 200 QPS to GCE (which is significantly more than we can afford) We should think how to solve this issue, but I think the options are: increasing QPS quota or calling GCE only once per X statusUpdates.
The second issue is that we are throttled at the controller level too, and this one I'm tryin to understand right now.


Question: glog.Infof("managing multiple zones: %v", managedZones) } <del> operationPollRateLimiter := flowcontrol.NewTokenBucketRateLimiter(10, 100) // 10 qps, 100 bucket size. </del> return &GCECloud{ <del> service:                  svc, containerService:         containerSvc, projectID:                projectID, region:                   region, localZone:                zone, managedZones:             managedZones, networkURL:               networkURL, nodeTags:                 nodeTags, useMetadataServer:        useMetadataServer, operationPollRateLimiter: operationPollRateLimiter, </del> <ins> service:           svc, containerService:  containerSvc, projectID:         projectID, region:            region, localZone:         zone, managedZones:      managedZones, networkURL:        networkURL, nodeTags:          nodeTags, useMetadataServer: useMetadataServer, </ins> }, nil }

- that said, any rate-limitting on our side, will not solve the kubelet-related problem
Hmm - actually looking into implementation it seems that NodeAddresses are only contacting metadata server, so I'm not longer that convinced...
Also, I think that important thing is that a single call to GCE API can translate to multiple api calls to GCE. In particular - CreateRoute translates to: get instance insert route a bunch of getOperation (until this is finished)
OK - so I think that what is happening here is that since CreateRoute translates to: get instance insert route a bunch of getOperations That means that if we call say 2000 CreateRoute() at the same time, all of them will try to issue: getInstance at the beginning. So we will quickly accumulate 2000 GET requests in the queue. Once they are processed, we generate the "POST route" then for any processed. So It's kind of expected what is happening here. So getting back to the Zach`s PR - I think that throttling POST and GET and OPERATION calls separate is kind of good idea in general.
But I still don't really understand why do we get "RateLimitExceeded". Is that because of Kubelets?
I think that one problem we have at the RouteController level is that if CreateRoute fails, we simply don't react on it. And we will wait at least 5 minutes before even retrying it. We should add some retries at the route controller level too. I will send out some short PR for it.
But I still don't understand why do we get those "RateLimitExceeded" error
OK - so as an update. In GCE, we have a separate rate-limit for number of in-flight CreateRoute call per project. is supposed to address this issue.
OK - so this one is actually fixed. I mean that it is still long, but currently it's purely blocked on GCE.
As a comment - we have an internal bug for it already.


Question: } servicePrincipalToken, err := auth.GetServicePrincipalToken(&config.AzureAuthConfig, env) <del> if err != nil { </del> <ins> if err == auth.ErrorNoAuth { if !config.UseInstanceMetadata { // No credentials provided, useInstanceMetadata should be enabled. return nil, fmt.Errorf("useInstanceMetadata must be enabled without Azure credentials") } klog.V(2).Infof("Azure cloud provider is starting without credentials") } else if err != nil { </ins> return nil, err }

Currently there is no way to run without any identity (be it service principal or E/MSI). In reality code running on a worker node can use via - and only via - without the need to access ARM. Here is the proposed change a new mode authN = this mode, is basically telling the cloud provider to work any identity, this means only IMDS calls is allowed. This mode can be detected from if no SPN set and both MSI settings are set to . to use a config map within the cluster. This config map can be loaded on the cluster statically. or created - if not existent - in that then only is used in The net new result of the above is users bootstrapping clusters can configure the worker node to use without any identity on said worker nodes.
/sig azure These SIGs are my best guesses for this issue. Please comment if I am incorrect about one. I am a bot run by
In-tree AzureDisk volume plugin is also depending on Azure credential to fetch lun id. we should figure out how to solve it without credential
fetch disk lun id is also a dependency in kubelet code now, after my investigation, it could be removed on Linux node, while it cannot be removed on Windows node, need to change the k/k upstream code, there is a discussion on going here: , let' wait for the discussion result.
For Linux: 1- use IMDS my understanding is new api versions will have the disk/lun pairs. 2- don't use arm for the return of is the second parameter passed to . For Windows: I don't understand the dep on ARM for this. The above still apply, we just need to find a way to encode the value as ut return from that allow us to find the disk by on a windows machine. might be a good source for this. Ideally speaking. The value should be the same. But i will leave it for you to decide.
could be a good idea since currently it's a LUN num returned by Attach() and then it could be changed to the windows disk num, I will make the code change on v1.15 later, thanks for discussion.
Update: PR would remove the VM API calls when getting the LUN number for AzureDisk.


Question: execAffinityTestForLBService(f, cs, svc) }) <del> // TODO: Get rid of [DisabledForLargeClusters] tag when issue #56138 is fixed. </del> // [LinuxOnly]: Windows does not support session affinity. <del> ginkgo.It("should be able to switch session affinity for LoadBalancer service with ESIPP off [Slow] [DisabledForLargeClusters] [LinuxOnly]", func() { </del> <ins> ginkgo.It("should be able to switch session affinity for LoadBalancer service with ESIPP off [Slow] [LinuxOnly]", func() { </ins> // L4 load balancer affinity `ClientIP` is not supported on AWS ELB. e2eskipper.SkipIfProviderIs("aws")

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug This is a followup from filed separately now that we're not tracking this as part of 1.9 release. The tests have been disabled with [DisabledForLargeClusters] tag for now, and this fact will be mentioned in the scalability validation report for v1.9. This issue tracks fixing the test so the tag can be removed and the test ran again in the large cluster. /assign /kind e2e-test-failure
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Is this e2e test still broken?
Sorry for the confusion here. Currently this e2e test is disabled on extra large cluster due to some scalability issues with the load balancer related components. At the moment the issues haven't been completely resolved yet.
ok, adding priority so that I know it's not a 1.13 blocker. /priority important-longterm
cc - this is why the relevant tests for are not part of the e2e suite.
This issue is still valid --- some action items on top of my head: Figure out a plan to make service controller properly handle operations in parallel. Check on the status for cloud providers (especially GCP, which the test ran against) on handling large number of nodes as LB backends. /remove-triage unresolved
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or .


Question: execAffinityTestForLBService(f, cs, svc) }) <del> // TODO: Get rid of [DisabledForLargeClusters] tag when issue #56138 is fixed. </del> // [LinuxOnly]: Windows does not support session affinity. <del> ginkgo.It("should be able to switch session affinity for LoadBalancer service with ESIPP off [Slow] [DisabledForLargeClusters] [LinuxOnly]", func() { </del> <ins> ginkgo.It("should be able to switch session affinity for LoadBalancer service with ESIPP off [Slow] [LinuxOnly]", func() { </ins> // L4 load balancer affinity `ClientIP` is not supported on AWS ELB. e2eskipper.SkipIfProviderIs("aws")

/lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten /lifecycle frozen
/reopen Reopening as is being reverted in
Reopened this issue. <detailsIn response to : /reopen Reopening as is being reverted in Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
these tests are flaking in xref
cc since these were enabled by
Yeah - though these failures doesn't seem to be scale related, they seem to be test related to me. This basically means nodes without public ips: It's pretty much the same thing as (just for a different test). should be addressing these kinds of errors. [Not a release blocker for sure]
BTW - has been opened directly for this one.
... so since is closed, Can we close this and re-enable for large clusters ? (i.e. remove the Tag in that disables this test) ? I'm not up to speed on the history but 2 related notes I came across digging into related test today... 's a couple of TODOs in the code referencing this specific issue, but your above direct issue is now closed, so we can close this as well right?... And if so, we can remove the TODO on the test... -enabling the test ... not sure wether the timing is good right now (given that is open right now , which sparked my stumbling upon this issue :))
We tried to enable all of them and that caused us a bunch of issues and those had to be reverted: A couple tests were enabled later in: and those seem fine. The remaining (mostly session-affinity ones) had some issue, which so far we didn't have time to debug and I haven't heard a good explanation why those were failing. It's possible that they were improved with , but I doubt those would really cause the failures we've seen in the past. So until this is debugged (or at least understood) I'm reluctant to enabling them.


Question: out.Description = in.Description if in.Type != "" { out.Type = spec.StringOrArray([]string{in.Type}) <del> if in.Nullable { out.Type = append(out.Type, "null") } </del> <ins> } if in.XIntOrString { out.VendorExtensible.AddExtension("x-kubernetes-int-or-string", true) out.Type = spec.StringOrArray{"integer", "string"} } if out.Type != nil && in.Nullable { out.Type = append(out.Type, "null") </ins> } out.Format = in.Format out.Title = in.Title

The server-side validation does not check the type of values with . Create a CRD like: Then create a CR: It is not rejected, but should because must not be a boolean.


Question: local -ra docker_cmd=( "${DOCKER[@]}" run "${docker_run_opts[@]}" "${KUBE_BUILD_IMAGE}") <del> # Remove the container if it is left over from some previous aborted run "${DOCKER[@]}" rm -f -v "${KUBE_BUILD_CONTAINER_NAME}" >/dev/null 2>&1 || true </del> <ins> # Clean up container from any previous run kube::build::destroy_container "${KUBE_BUILD_CONTAINER_NAME}" </ins> "${docker_cmd[@]}" "$@" <del> # Remove the container after we run.  '--rm' might be appropriate but it # appears that sometimes it fails. See # https://github.com/docker/docker/issues/3968 "${DOCKER[@]}" rm -f -v "${KUBE_BUILD_CONTAINER_NAME}" >/dev/null 2>&1 || true </del> <ins> kube::build::destroy_container "${KUBE_BUILD_CONTAINER_NAME}" </ins> } # Test if the output directory is remote (and can only be accessed through

Github automatically adds a artifact to every release, which sounds very similar to the artifact that we generate as the release artifact. Assuming we don't have control over the naming or presence of the autogenerated source code artifact, I think we should consider renaming our published artifact to better disambiguate between the two.
We distribute most of our artifacts outside of GitHub. We should probably stop uploading this one entirely.
cc
We've got a thread going on w/ Tim. /assign
can you please link so folks can follow the bread crumbs? ( cc )
-- It's currently sensitive. Will share once there's an outcome.
ACK. thanks for the heads up
I think it's safe to discuss openly at this point. We had a report that the sha512 hashes on our release artifacts were incorrect. We have confirmed that the mismatch was due to a bug in the release pipeline, and not malicious. This issue here was just a point of confusion that came out of that discussion.
thanks !
/assign
Going one step back since I think the exists for historical reasons: The client and server binaries were part of this bundle, which is not the case any more: So it seems that is mostly a collection of licenses and scripts. Do we really need it or may we cleanup some code paths and remove it completely?
+1 to put the word out and remove it completely if no one depends on it.
I'm not confident that CI is not using these versioned scripts, in fact i'm fairly sure that it is. However it would not be pulling it from github.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale I think this remains as relevant as ever.
/milestone v1.23
/triage accepted
Hi and k8s 1.23 bug triage shadow here.


Question: local -ra docker_cmd=( "${DOCKER[@]}" run "${docker_run_opts[@]}" "${KUBE_BUILD_IMAGE}") <del> # Remove the container if it is left over from some previous aborted run "${DOCKER[@]}" rm -f -v "${KUBE_BUILD_CONTAINER_NAME}" >/dev/null 2>&1 || true </del> <ins> # Clean up container from any previous run kube::build::destroy_container "${KUBE_BUILD_CONTAINER_NAME}" </ins> "${docker_cmd[@]}" "$@" <del> # Remove the container after we run.  '--rm' might be appropriate but it # appears that sometimes it fails. See # https://github.com/docker/docker/issues/3968 "${DOCKER[@]}" rm -f -v "${KUBE_BUILD_CONTAINER_NAME}" >/dev/null 2>&1 || true </del> <ins> kube::build::destroy_container "${KUBE_BUILD_CONTAINER_NAME}" </ins> } # Test if the output directory is remote (and can only be accessed through

There hasn't been any activity or linked PRs for this issue. Just wanted to check in to see if this is still on track for 1.23.
/milestone v1.24
Hi I am part of the bug triage team for v1.24. We noticed that this issue hasn't been updated for a while. Just wanted to check if there is any work planned that would go into this.
this one keeps moving down the milestones. i don't see any activity since november. So kick the can down the road /milestone v1.25
Hi My name Hossein Salahi from Bug-Triage team. I am just checking to ensure if everything is on track for K8s 1.25. Thanks!
That's a question for or
any remarks on that?
the release 1.25 bug triage Lead. I'm reaching out to see the status of this issue and if we are still targeting the current release. Also, kindly if you can update the priority so we can track it in the future.
this one keeps moving down the milestones yet again!. i don't see any activity since i moved this down the line. So kick the can down the road: /milestone clear (instead of v1.26 as this has been moving down the line for a few releases now)
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
What's the current status of this issue? Is there anything that we can do to move this forward?
We got new conversations going on during Kubecon on whether we should publish our release binaries on GitHub. Given that and I think we should rethink now what we publish on the release pages. I think we should start publishing all binaries for one. It is a common practice everywhere and I think users largely expect them to be there.


Question: local -ra docker_cmd=( "${DOCKER[@]}" run "${docker_run_opts[@]}" "${KUBE_BUILD_IMAGE}") <del> # Remove the container if it is left over from some previous aborted run "${DOCKER[@]}" rm -f -v "${KUBE_BUILD_CONTAINER_NAME}" >/dev/null 2>&1 || true </del> <ins> # Clean up container from any previous run kube::build::destroy_container "${KUBE_BUILD_CONTAINER_NAME}" </ins> "${docker_cmd[@]}" "$@" <del> # Remove the container after we run.  '--rm' might be appropriate but it # appears that sometimes it fails. See # https://github.com/docker/docker/issues/3968 "${DOCKER[@]}" rm -f -v "${KUBE_BUILD_CONTAINER_NAME}" >/dev/null 2>&1 || true </del> <ins> kube::build::destroy_container "${KUBE_BUILD_CONTAINER_NAME}" </ins> } # Test if the output directory is remote (and can only be accessed through

let's scope the issue to the actual "Kubernetes Source Code" tarball. We can still remove it from what I can see from the discussion.
I agree with Removing the "Kubernetes Source Code" tarball should be as easy as removing this from krel: However, there are a few questions: is it safe to do so? I believe yes, because this "Kubernetes Source Code" tarball is pointing to the GCS bucket, and we'll still continue publishing that tarball to the bucket The link is also available in the changelog, so it's easy to find do we need any deprecation period? Considering the above two points, I don't think so. If you folks agree with that, I can go ahead and create a PR to update krel.
I agree on that, let's drop an additional note on k-dev that others are aware of the change.
/assign
Update: An announcement to proceed with the removal of the "Kubernetes Source Code" artifact has been sent to k-dev: PR to remove this artifact from being published: Removal will go into effect starting the January patch releases.


Question: if diff := cmp.Diff(tc.wantProfiles, profiles); diff != "" { t.Errorf("unexpected profiles (-want, +got):n%s", diff) } <ins> // Extenders if len(tc.wantExtenders) != 0 { // Scheduler.Extenders extenders := make([]string, 0, len(s.Extenders)) for _, e := range s.Extenders { extenders = append(extenders, e.Name()) } if diff := cmp.Diff(tc.wantExtenders, extenders); diff != "" { t.Errorf("unexpected extenders (-want, +got):n%s", diff) } // framework.Handle.Extenders() for _, p := range s.Profiles { extenders := make([]string, 0, len(p.Extenders())) for _, e := range p.Extenders() { extenders = append(extenders, e.Name()) } if diff := cmp.Diff(tc.wantExtenders, extenders); diff != "" { t.Errorf("unexpected extenders (-want, +got):n%s", diff) } } } </ins> }) } }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --here is supposed to be set through , but is not called from anywhere except tests, so always returns . should be set through here. Am I missing something? N/A None Kubernetes version (use ): master () Cloud provider or hardware configuration: N/A OS (e.g: ): N/A Kernel (e.g. ): N/A Install tools: N/A Network plugin and version (if this is a network-related bug): N/A Others: None
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scheduling
Thanks for pointing that out. The plumbing of extenders to the framework impact calling extenders on preemption. That seems to be a path not covered by tests end to end. Can you submit a PR to fix this? as you mentioned we need to pass the extenders on profile creation. We need to patch the fix back to v1.21 and v1.20 (the release where this was refactored). /cc
I have submitted a PR for a fix.


Question: if err != nil { return "", err } <del> if isLocalInstance { if metadata.Compute.VMSize != "" { return metadata.Compute.VMSize, nil </del> <ins> if !isLocalInstance { if az.vmSet != nil { return az.vmSet.GetInstanceTypeByNodeName(string(name)) </ins> } <ins> // vmSet == nil indicates credentials are not provided. return "", fmt.Errorf("no credentials provided for Azure cloud provider") } if metadata.Compute.VMSize != "" { return metadata.Compute.VMSize, nil </ins> } }


Question: jig.SanityCheckService(svc, v1.ServiceTypeLoadBalancer) defer func() { framework.StopServeHostnameService(cs, f.InternalClientset, f.ScalesGetter, ns, serviceName) <ins> lb := cloudprovider.GetLoadBalancerName(svc) framework.Logf("cleaning gce resource for %s", lb) framework.CleanupServiceGCEResources(cs, lb, framework.TestContext.CloudConfig.Region, framework.TestContext.CloudConfig.Zone) </ins> }() ingressIP := framework.GetIngressPoint(&svc.Status.LoadBalancer.Ingress[0]) port := int(svc.Spec.Ports[0].Port)

Our GKE scalability jobs (https://k8s-) are failing in the cluster teardown step (even though the e2e tests themselves passed). E.g failures: https://k8s- https://k8s- Seems like it's caused by some leaked network-related resources: /assign Krzysiek - Could you PTAL into this when you find time? It'll be great to have our GKE suite back to green. cc
I think this is related to an earlier bug we've seen (similar one for GCE): Basically service-controller is creating some cloud resources (like firewalls) for lb-type services which are not brought down properly when the k8s cluster is brought down.
Also another related issue - There should also be an internal bug for this somewhere. IMO we may be able to fix this in test-infra (without waiting for proper long-term fix on gke side).
I am adding clean-up method. This way all of the firewall rules will be cleaned up after tests. If this method has to remove any firewall rule, the teardown phase will return exception. It's not a fix for the bug - it only allows us to easy isolate tests leaking firewall rules.
Tests: should have session affinity work for LoadBalancer service with ESIPP on should be able to switch session affinity for LoadBalancer service with ESIPP on should have session affinity work for LoadBalancer service with ESIPP off should be able to switch session affinity for LoadBalancer service with ESIPP off leak resources related to the LoadBalancer. PR will fix this.


Question: } } <ins> func TestCertSatisfiesTemplate(t *testing.T) { testCases := []struct { name          string cert          *x509.Certificate template      *x509.CertificateRequest shouldSatisfy bool }{ { name:          "No certificate, no template", cert:          nil, template:      nil, shouldSatisfy: false, }, { name:          "No certificate", cert:          nil, template:      &x509.CertificateRequest{}, shouldSatisfy: false, }, { name: "No template", cert: &x509.Certificate{ Subject: pkix.Name{ CommonName: "system:node:fake-node-name", }, }, template:      nil, shouldSatisfy: true, }, { name: "Mismatched common name", cert: &x509.Certificate{ Subject: pkix.Name{ CommonName: "system:node:fake-node-name-2", }, }, template: &x509.CertificateRequest{ Subject: pkix.Name{ CommonName: "system:node:fake-node-name", }, }, shouldSatisfy: false, }, { name: "Missing orgs in certificate", cert: &x509.Certificate{ Subject: pkix.Name{ Organization: []string{"system:nodes"}, }, }, template: &x509.CertificateRequest{ Subject: pkix.Name{ Organization: []string{"system:nodes", "foobar"}, }, }, shouldSatisfy: false, }, { name: "Extra orgs in certificate", cert: &x509.Certificate{ Subject: pkix.Name{ Organization: []string{"system:nodes", "foobar"}, }, }, template: &x509.CertificateRequest{ Subject: pkix.Name{ Organization: []string{"system:nodes"}, }, }, shouldSatisfy: true, }, { name: "Missing DNS names in certificate", cert: &x509.Certificate{ Subject:  pkix.Name{}, DNSNames: []string{"foo.example.com"}, }, template: &x509.CertificateRequest{ Subject:  pkix.Name{}, DNSNames: []string{"foo.example.com", "bar.example.com"}, }, shouldSatisfy: false, }, { name: "Extra DNS names in certificate", cert: &x509.Certificate{ Subject:  pkix.Name{}, DNSNames: []string{"foo.example.com", "bar.example.com"}, }, template: &x509.CertificateRequest{ Subject:  pkix.Name{}, DNSNames: []string{"foo.example.com"}, }, shouldSatisfy: true, }, { name: "Missing IP addresses in certificate", cert: &x509.Certificate{ Subject:     pkix.Name{}, IPAddresses: []net.IP{net.ParseIP("192.168.1.1")}, }, template: &x509.CertificateRequest{ Subject:     pkix.Name{}, IPAddresses: []net.IP{net.ParseIP("192.168.1.1"), net.ParseIP("192.168.1.2")}, }, shouldSatisfy: false, }, { name: "Extra IP addresses in certificate", cert: &x509.Certificate{ Subject:     pkix.Name{}, IPAddresses: []net.IP{net.ParseIP("192.168.1.1"), net.ParseIP("192.168.1.2")}, }, template: &x509.CertificateRequest{ Subject:     pkix.Name{}, IPAddresses: []net.IP{net.ParseIP("192.168.1.1")}, }, shouldSatisfy: true, }, { name: "Matching certificate", cert: &x509.Certificate{ Subject: pkix.Name{ CommonName:   "system:node:fake-node-name", Organization: []string{"system:nodes"}, }, DNSNames:    []string{"foo.example.com"}, IPAddresses: []net.IP{net.ParseIP("192.168.1.1")}, }, template: &x509.CertificateRequest{ Subject: pkix.Name{ CommonName:   "system:node:fake-node-name", Organization: []string{"system:nodes"}, }, DNSNames:    []string{"foo.example.com"}, IPAddresses: []net.IP{net.ParseIP("192.168.1.1")}, }, shouldSatisfy: true, }, } for _, tc := range testCases { t.Run(tc.name, func(t *testing.T) { var tlsCert *tls.Certificate if tc.cert != nil { tlsCert = &tls.Certificate{ Leaf: tc.cert, } } m := manager{ cert:        tlsCert, getTemplate: func() *x509.CertificateRequest { return tc.template }, } result := m.certSatisfiesTemplate() if result != tc.shouldSatisfy { t.Errorf("cert: %+v, template: %+v, certSatisfiesTemplate returned %v, want %v", m.cert, tc.template, result, tc.shouldSatisfy) } }) } } </ins> func TestRotateCertCreateCSRError(t *testing.T) { now := time.Now() m := manager{

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: I'm testing the new option in kubernetes 1.12 and it seems that kubelet always requests the signing of a new CSR and tries to rotate the certificates when it is started. This is happening even though there's a valid (brand new) certificate on disk issued by kube-controller-manager. From what I can tell, this is happening because kubelet implements and is set after a successful certificate rotation, but is never persisted to disk. If kubelet is restarted, then the last requested CSR is nil. What you expected to happen: Ideally, kubelet should only request the signing of a new CSR if it doesn't already have a serving certificate or it's about to expire. How to reproduce it (as minimally and precisely as possible): Restart a kubelet instance that had it's serving certificate issued by kube-controller-manager Check and see that it requested the signing of a new CSR Anything else we need to know?: kubelet is being run with . , , and are all omitted to use the default values. The TLS bootstrapping for client certificates is currently not being used--just the bootstrapping for serving certificates. Environment: Kubernetes version: v1.12.0 - built from source at the tag () Cloud provider or hardware configuration: Bare metal OS: CentOS 7.5.1804 Kernel: Install tools: Custom RPMs built from this git repo with no additional patches Others:
/sig node
/sig auth
Not sure if this is the ideal fix, but I've patched it so that it compares the current CSR template to the last requested template only if it doesn't already match the current cert.
cc
thanks for the proposed fix. I think it looks good. Could you send a PR? We should also remove entirely and only look at current cert and template. does that sound reasonable?
Sure thing, I'd be happy to do that. EDIT: I will submit a PR after approval from my employer.
I think that's probably ok, as long as it checks everything we include in the template and we consider a certificate missing anything we request to be cause for re-requesting another cert (an issuer couldn't choose to issue a cert that drops some SANs and us not request another cert) left some comments in
This regression is a frequent flyer.


Question: } } <ins> func TestCertSatisfiesTemplate(t *testing.T) { testCases := []struct { name          string cert          *x509.Certificate template      *x509.CertificateRequest shouldSatisfy bool }{ { name:          "No certificate, no template", cert:          nil, template:      nil, shouldSatisfy: false, }, { name:          "No certificate", cert:          nil, template:      &x509.CertificateRequest{}, shouldSatisfy: false, }, { name: "No template", cert: &x509.Certificate{ Subject: pkix.Name{ CommonName: "system:node:fake-node-name", }, }, template:      nil, shouldSatisfy: true, }, { name: "Mismatched common name", cert: &x509.Certificate{ Subject: pkix.Name{ CommonName: "system:node:fake-node-name-2", }, }, template: &x509.CertificateRequest{ Subject: pkix.Name{ CommonName: "system:node:fake-node-name", }, }, shouldSatisfy: false, }, { name: "Missing orgs in certificate", cert: &x509.Certificate{ Subject: pkix.Name{ Organization: []string{"system:nodes"}, }, }, template: &x509.CertificateRequest{ Subject: pkix.Name{ Organization: []string{"system:nodes", "foobar"}, }, }, shouldSatisfy: false, }, { name: "Extra orgs in certificate", cert: &x509.Certificate{ Subject: pkix.Name{ Organization: []string{"system:nodes", "foobar"}, }, }, template: &x509.CertificateRequest{ Subject: pkix.Name{ Organization: []string{"system:nodes"}, }, }, shouldSatisfy: true, }, { name: "Missing DNS names in certificate", cert: &x509.Certificate{ Subject:  pkix.Name{}, DNSNames: []string{"foo.example.com"}, }, template: &x509.CertificateRequest{ Subject:  pkix.Name{}, DNSNames: []string{"foo.example.com", "bar.example.com"}, }, shouldSatisfy: false, }, { name: "Extra DNS names in certificate", cert: &x509.Certificate{ Subject:  pkix.Name{}, DNSNames: []string{"foo.example.com", "bar.example.com"}, }, template: &x509.CertificateRequest{ Subject:  pkix.Name{}, DNSNames: []string{"foo.example.com"}, }, shouldSatisfy: true, }, { name: "Missing IP addresses in certificate", cert: &x509.Certificate{ Subject:     pkix.Name{}, IPAddresses: []net.IP{net.ParseIP("192.168.1.1")}, }, template: &x509.CertificateRequest{ Subject:     pkix.Name{}, IPAddresses: []net.IP{net.ParseIP("192.168.1.1"), net.ParseIP("192.168.1.2")}, }, shouldSatisfy: false, }, { name: "Extra IP addresses in certificate", cert: &x509.Certificate{ Subject:     pkix.Name{}, IPAddresses: []net.IP{net.ParseIP("192.168.1.1"), net.ParseIP("192.168.1.2")}, }, template: &x509.CertificateRequest{ Subject:     pkix.Name{}, IPAddresses: []net.IP{net.ParseIP("192.168.1.1")}, }, shouldSatisfy: true, }, { name: "Matching certificate", cert: &x509.Certificate{ Subject: pkix.Name{ CommonName:   "system:node:fake-node-name", Organization: []string{"system:nodes"}, }, DNSNames:    []string{"foo.example.com"}, IPAddresses: []net.IP{net.ParseIP("192.168.1.1")}, }, template: &x509.CertificateRequest{ Subject: pkix.Name{ CommonName:   "system:node:fake-node-name", Organization: []string{"system:nodes"}, }, DNSNames:    []string{"foo.example.com"}, IPAddresses: []net.IP{net.ParseIP("192.168.1.1")}, }, shouldSatisfy: true, }, } for _, tc := range testCases { t.Run(tc.name, func(t *testing.T) { var tlsCert *tls.Certificate if tc.cert != nil { tlsCert = &tls.Certificate{ Leaf: tc.cert, } } m := manager{ cert:        tlsCert, getTemplate: func() *x509.CertificateRequest { return tc.template }, } result := m.certSatisfiesTemplate() if result != tc.shouldSatisfy { t.Errorf("cert: %+v, template: %+v, certSatisfiesTemplate returned %v, want %v", m.cert, tc.template, result, tc.shouldSatisfy) } }) } } </ins> func TestRotateCertCreateCSRError(t *testing.T) { now := time.Now() m := manager{

I'd love to see an integration test that would catch this (but that shouldn't block a fix).
Thanks for the comments! Now that I have the CLA signed, I'll address those and submit a PR.
Just submitted the PR:


Question: var cpuFractionMap = make(map[string]float64) var memFractionMap = make(map[string]float64) <ins> // For each node, stores its pods info nodeNameToPodList := podListForEachNode(cs) </ins> for _, node := range nodes { <del> cpuFraction, memFraction, _, _ := computeCPUMemFraction(cs, node, requestedResource) </del> <ins> cpuFraction, memFraction, _, _ := computeCPUMemFraction(node, requestedResource, nodeNameToPodList[node.Name]) </ins> cpuFractionMap[node.Name] = cpuFraction memFractionMap[node.Name] = memFraction if cpuFraction > maxCPUFraction {

From After , Multi-AZ Clusters tests in 1.19 and 1.20 started timing out as createBalancedPodForNodes is extremely inefficient in scale: for each node (out of 5k), it lists all pods in a cluster serially (each list takes ~5s), which translates to ~7 hours. Optimizing this function by listing pods once and then in memory by Spec.NodeName /kind bug /priority critical-urgent /sig scheduling
/triage accepted
/assign
/reopen Looks that ComputeCPUMemFraction took only 12 minutes, but it stuck later, on It is scheduling one pod per each node: Which is waiting 5s to be ready: 5k nodes * 5s = ~7h
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign Can you do the pod creations in parallel?
Yes, good idea. I'll update to do that


Question: #     sleep 4 #     i=$[$i+1] #   done <del> apiVersion: v1beta1 </del> <ins> apiVersion: v1beta3 </ins> kind: Pod <del> id: synthetic-logger-0.25lps-pod desiredState: manifest: version: v1beta1 id: synth-logger-0.25lps containers: - name: synth-lgr image: ubuntu:14.04 command: ["bash", "-c", "i="0"; while true; do echo -n "`hostname`: $i: "; date --rfc-3339 ns; sleep 4; i=$[$i+1]; done"] labels: name: synth-logging-source </del> <ins> metadata: labels: name: synth-logging-source name: synthetic-logger-0.25lps-pod spec: containers: - args: - bash - -c - 'i="0"; while true; do echo -n "`hostname`: $i: "; date --rfc-3339 ns; sleep 4; i=$[$i+1]; done' image: ubuntu:14.04 name: synth-lgr </ins> ``` The other YAML file [synthetic_10lps.yaml](synthetic_10lps.yaml) specifies a similar synthetic logger that emits 10 log messages every second. To run both synthetic loggers:

Hi , I followed the instructions in examples/logging-demo, and there was no "external IP" field for elasticsearch-logging or kibana-logging, as shown below: I start up my cluster via cluster/kube- Did I miss something or is it a bug? Thanks.
Correct. This demo does not create an ELB so there is no external IP address. Access is via service proxy at the master. Type: and you should see the URLs for Elasticsearch and Kibana.
please let us know whether: a. Satnam's comment resolves your issue b. the demo should be updated to prevent this misunderstanding c. the issue isn't resolved Thank you!
Yes, Satnam's comment resolve the issue. I updated the demo in .
Awesome, thanks for sending the patch! We'll discuss there. Always fantastic to have docs updated regularly by folks trying them.


Question: if m == nil { return &Timestamp{} } <ins> // truncate precision to microseconds to match JSON marshaling/unmarshaling truncatedNanoseconds := time.Duration(m.Time.Nanosecond()).Truncate(time.Microsecond) </ins> return &Timestamp{ Seconds: m.Time.Unix(), <del> Nanos:   int32(m.Time.Nanosecond()), </del> <ins> Nanos:   int32(truncatedNanoseconds), </ins> } }

Found some error logs when sending series events to apiserver: seems the later eventTime differed from the first one in series events After investigation, I found the problem was in the marshal. MicroTime is supposed to work on microsecond level precision, but init with nanosecond level precision, in this case, generates microsecond level precision data and generates nanosecond level precision, so when unmarshalled, the new time is not considered the exactly same as the original time (they are different at nanosecond level) MicroTime marshal should work at same precision for json and proto, so unmarshal can generate the exactly same MicroTime result: This causes the series events won't be updated on api-server after created master <details</details<details</details<details</details<details</details<details</details
/sig api-machinery
/assign
Good catch... we'd probably need to truncate to microseconds when marshaling and unmarshaling proto to ensure any already persisted data is fixed up when read. Something like this:
Thanks for advice, going to send a PR soon


Question: func (plugin *flexVolumePlugin) getExecutable() string { parts := strings.Split(plugin.driverName, "/") execName := parts[len(parts)-1] <del> return path.Join(plugin.execPath, execName) </del> <ins> execPath := path.Join(plugin.execPath, execName) if runtime.GOOS == "windows" { execPath = volume.GetWindowsPath(execPath) } return execPath </ins> } // Name is part of the volume.VolumePlugin interface.

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: I have tried FlexVolume on a Windows node, it does not work. I am now working on this feature now. Below is the FlexVolume initialization error What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): v1.7.7 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: /sig windows /assign
Detailed steps about how to use flexvolume on Windows with this PR:
cc:


Question: if err := s.validate(); err != nil { return nil, err } <del> dockercfgContent, err := handleDockercfgContent(s.Username, s.Password, s.Email, s.Server) </del> <ins> dockercfgJsonContent, err := handleDockerCfgJsonContent(s.Username, s.Password, s.Email, s.Server) </ins> if err != nil { return nil, err } secret := &v1.Secret{} secret.Name = s.Name <del> secret.Type = v1.SecretTypeDockercfg </del> <ins> secret.Type = v1.SecretTypeDockerConfigJson </ins> secret.Data = map[string][]byte{} <del> secret.Data[v1.DockerConfigKey] = dockercfgContent </del> <ins> secret.Data[v1.DockerConfigJsonKey] = dockercfgJsonContent </ins> if s.AppendHash { h, err := hash.SecretHash(secret) if err != nil {

/kind bug We're having trouble retrieving images from a private registry on DockerCloud. For nearly a year the following has worked flawlessly. Last Friday I pulled down the 1.9.0 CLI (via brew) and since then have not been able to access any image in a private registry on DockerCloud. I've tried this on 10+ brand new clusters created with kops v1.8.0. The Kubernetes cluster version have been 1.8.0, 1.8.4, 1.8.5 and 1.9.0 (which is not supported by kops yet) and all exhibit the same problem. We create the secret as such: And verify it using: Which yields: And create a test service with with these contents: The pod fails immediatly with: I've reviewed the 1.9.0 release notes and nothing jumps out as the root cause. For reference I've tried putting quotes around the value when creating the secret to no avail. I've also tried prefixing the value with , so the full value would read: also to no avail. Environment: Kubernetes version: Cloud provider and installation info: AWS cluster built with Kops 1.8.0, using (to isolate this issue) nearly the default settings for everything.
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I have same problem, and I "solved" it creating a docker-registry secret in an old kubernetes version and pasting it on new kubernetes
I'm recently (I know it worked perfectly about last month) facing issues with as well. For me it ended in a similar manner saying: I was able to confirm, that this is a kubernetes CLI related issue and not end user. Based on and this I was able to create properly formatted auth information looking like this: . When encoding this one-liner using (MacOS workstation) and apply like this works fine and images can be pulled from private repository again.


Question: if err := s.validate(); err != nil { return nil, err } <del> dockercfgContent, err := handleDockercfgContent(s.Username, s.Password, s.Email, s.Server) </del> <ins> dockercfgJsonContent, err := handleDockerCfgJsonContent(s.Username, s.Password, s.Email, s.Server) </ins> if err != nil { return nil, err } secret := &v1.Secret{} secret.Name = s.Name <del> secret.Type = v1.SecretTypeDockercfg </del> <ins> secret.Type = v1.SecretTypeDockerConfigJson </ins> secret.Data = map[string][]byte{} <del> secret.Data[v1.DockerConfigKey] = dockercfgContent </del> <ins> secret.Data[v1.DockerConfigJsonKey] = dockercfgJsonContent </ins> if s.AppendHash { h, err := hash.SecretHash(secret) if err != nil {

However this breaks all automation and can't be considered as solution Confirmed working with kubectl 1.8.5 and server 1.8.4 (guess 1.7.9 as well)
looks like there's an extra "auths" element when adding the secret using 1.9 kubectl
Added in /assign
Many thanks for the quick fix on this. Any idea when this might get rolled out? Also, it might be a good idea if this was to the known issue list for v1.9.0. It would have saved us a bunch of time trying to track down the root cause. Just trying to make sure no one else needs to do the same.
linking to the "v1.9.0 known issues" list
ack, asking Mehdy since he is the patch manager for v1.9 branch. Mehdy, any idea when we can ship 1.9.1 with this fix?
workarounds for folks until 1.9.1 is released: if you have a .dockerconfigjson for your private registry already, you can manually specify the type and data key (e.g. ) if you don't have a .dockerconfigjson file already, you can fix up the secret produced by manually: the type from to the data key from to the modified secret with
1 will be released tomorrow
I can confirm solution works, thanks so much!
I am still having issues in 1.9.1. This worked in the 1.8.x branches: results in the following error:
you have to remove the gitlab-registry secret before you can apply an update... type is immutable, and the fix in 1.9.1 required changing the type of the secret


Question: ``` Note that we are still working on making all containerized the master components run smoothly in rkt. Before that we are not able to run the master node with rkt yet. <ins> [![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/getting-started-guides/rkt/README.md?pixel)]() </ins>

The "service" file in this document contains a trailing comma on the line ' "name": "my-service", ` and thus kubectl fails to parse it properly.


Question: }) It("should schedule a pod w/ a RW PD, remove it, then schedule it on another host", func() { <del> if testContext.Provider != "gce" && testContext.Provider != "aws" { </del> <ins> if !providerIs("gce", "aws") { </ins> By(fmt.Sprintf("Skipping PD test, which is only supported for providers gce & aws (not %s)", testContext.Provider)) return

I noticed while running e2e tests locally that some tests were apparently being incorrectly skipped: I checked, and this is also true for our Jenkins runs. Currently it looks like and have this bug, though only the former actually skips the test, I think. The issue appears to be that the code inside the top-level in these files is being run before the testContext struct is filled out (and before is even called), and so testContext.Provider is empty. I'm not immediately sure what the proper fix is; I'm also worried that there might be other things broken by the fact that these blocks are called before initialization is complete. We also probably investigate if there is a better way to mark these tests as skipped; right now, there's no indication in the test report (, Jenkins test report, etc) that these tests were skipped.
I merged and which tried to disable these tests for non-{gke,gce} but apparently didn't follow-up and verify that they were still running on our platforms. /cc
Seems easy enough to fix. I'll have a PR out to fix this before Monday.


Question: Contexts: map[string]clientcmdapi.Context{ "gothic-context": {AuthInfo: "blue-user", Cluster: "chicken-cluster", Namespace: "plane-ns"}}, } testConfigConflictAlfa = clientcmdapi.Config{ AuthInfos: map[string]clientcmdapi.AuthInfo{ "red-user":    {Token: "a-different-red-token"},

I've been pulling my hair out trying get kubectl talk to my local cluster. It seems that kubectl merges the local file with values pulled down from the cloud. I've done the following: The I just set is silently overridden. Questions: is this documented anywhere? should kubectl merge in my GCE cluster when I set my cloud-provider to something else? should kubectl override my previously set current-context?
workaround is to use flag to override the config file value.
but not a good UX long term
Someone else hit this as well: . Better fix is to have the local-up script set a context, which I will try to add today. As for the overwriting behavior, it's documented in , but based on what you describe, the intended load order is not working; local .kubeconfig should overwrite global. cc
That sounds infuriating. I'll take a look this morning.


Question: cloud                 *Cloud } <del> // AttachDisk attaches a vhd to vm. The vhd must exist, can be identified by diskName, diskURI, and lun. func (c *controllerCommon) AttachDisk(isManagedDisk bool, diskName, diskURI string, nodeName types.NodeName, lun int32, cachingMode compute.CachingTypes) error { // 1. vmType is standard, attach with availabilitySet.AttachDisk. </del> <ins> // getNodeVMSet gets the VMSet interface based on config.VMType and the real virtual machine type. func (c *controllerCommon) getNodeVMSet(nodeName types.NodeName) (VMSet, error) { // 1. vmType is standard, return cloud.vmSet directly. </ins> if c.cloud.VMType == vmTypeStandard { <del> return c.cloud.vmSet.AttachDisk(isManagedDisk, diskName, diskURI, nodeName, lun, cachingMode) </del> <ins> return c.cloud.vmSet, nil </ins> } // 2. vmType is Virtual Machine Scale Set (vmss), convert vmSet to scaleSet. ss, ok := c.cloud.vmSet.(*scaleSet) if !ok { <del> return fmt.Errorf("error of converting vmSet (%q) to scaleSet with vmType %q", c.cloud.vmSet, c.cloud.VMType) </del> <ins> return nil, fmt.Errorf("error of converting vmSet (%q) to scaleSet with vmType %q", c.cloud.vmSet, c.cloud.VMType) </ins> } <del> // 3. If the node is managed by availability set, then attach with availabilitySet.AttachDisk. </del> <ins> // 3. If the node is managed by availability set, then return ss.availabilitySet. </ins> managedByAS, err := ss.isNodeManagedByAvailabilitySet(mapNodeNameToVMName(nodeName)) if err != nil { <del> return err </del> <ins> return nil, err </ins> } if managedByAS { // vm is managed by availability set. <del> return ss.availabilitySet.AttachDisk(isManagedDisk, diskName, diskURI, nodeName, lun, cachingMode) </del> <ins> return ss.availabilitySet, nil </ins> } <del> // 4. Node is managed by vmss, attach with scaleSet.AttachDisk. return ss.AttachDisk(isManagedDisk, diskName, diskURI, nodeName, lun, cachingMode) </del> <ins> // 4. Node is managed by vmss return ss, nil </ins> } <del> // DetachDiskByName detaches a vhd from host. The vhd can be identified by diskName or diskURI. func (c *controllerCommon) DetachDiskByName(diskName, diskURI string, nodeName types.NodeName) error { // 1. vmType is standard, detach with availabilitySet.DetachDiskByName. if c.cloud.VMType == vmTypeStandard { return c.cloud.vmSet.DetachDiskByName(diskName, diskURI, nodeName) </del> <ins> // AttachDisk attaches a vhd to vm. The vhd must exist, can be identified by diskName, diskURI, and lun. func (c *controllerCommon) AttachDisk(isManagedDisk bool, diskName, diskURI string, nodeName types.NodeName, lun int32, cachingMode compute.CachingTypes) error { vmset, err := c.getNodeVMSet(nodeName) if err != nil { return err </ins> } <del> // 2. vmType is Virtual Machine Scale Set (vmss), convert vmSet to scaleSet. ss, ok := c.cloud.vmSet.(*scaleSet) if !ok { return fmt.Errorf("error of converting vmSet (%q) to scaleSet with vmType %q", c.cloud.vmSet, c.cloud.VMType) } </del> <ins> return vmset.AttachDisk(isManagedDisk, diskName, diskURI, nodeName, lun, cachingMode) } </ins> <del> // 3. If the node is managed by availability set, then detach with availabilitySet.DetachDiskByName. managedByAS, err := ss.isNodeManagedByAvailabilitySet(mapNodeNameToVMName(nodeName)) </del> <ins> // DetachDiskByName detaches a vhd from host. The vhd can be identified by diskName or diskURI. func (c *controllerCommon) DetachDiskByName(diskName, diskURI string, nodeName types.NodeName) error { vmset, err := c.getNodeVMSet(nodeName) </ins> if err != nil { return err } <del> if managedByAS { // vm is managed by availability set. return ss.availabilitySet.DetachDiskByName(diskName, diskURI, nodeName) } // 4. Node is managed by vmss, detach with scaleSet.DetachDiskByName. return ss.DetachDiskByName(diskName, diskURI, nodeName) </del> <ins> return vmset.DetachDiskByName(diskName, diskURI, nodeName) </ins> } <del> // GetDiskLun finds the lun on the host that the vhd is attached to, given a vhd's diskName and diskURI. func (c *controllerCommon) GetDiskLun(diskName, diskURI string, nodeName types.NodeName) (int32, error) { // 1. vmType is standard, get with availabilitySet.GetDiskLun. if c.cloud.VMType == vmTypeStandard { return c.cloud.vmSet.GetDiskLun(diskName, diskURI, nodeName) </del> <ins> // getNodeDataDisks invokes vmSet interfaces to get data disks for the node. func (c *controllerCommon) getNodeDataDisks(nodeName types.NodeName) ([]compute.DataDisk, error) { vmset, err := c.getNodeVMSet(nodeName) if err != nil { return nil, err </ins> } <del> // 2. vmType is Virtual Machine Scale Set (vmss), convert vmSet to scaleSet. ss, ok := c.cloud.vmSet.(*scaleSet) if !ok { return -1, fmt.Errorf("error of converting vmSet (%q) to scaleSet with vmType %q", c.cloud.vmSet, c.cloud.VMType) } </del> <ins> return vmset.GetDataDisks(nodeName) } </ins> <del> // 3. If the node is managed by availability set, then get with availabilitySet.GetDiskLun. managedByAS, err := ss.isNodeManagedByAvailabilitySet(mapNodeNameToVMName(nodeName)) </del> <ins> // GetDiskLun finds the lun on the host that the vhd is attached to, given a vhd's diskName and diskURI. func (c *controllerCommon) GetDiskLun(diskName, diskURI string, nodeName types.NodeName) (int32, error) { disks, err := c.getNodeDataDisks(nodeName) </ins> if err != nil { <ins> glog.Errorf("error of getting data disks for node %q: %v", nodeName, err) </ins> return -1, err } <del> if managedByAS { // vm is managed by availability set. return ss.availabilitySet.GetDiskLun(diskName, diskURI, nodeName) } // 4. Node is managed by vmss, get with scaleSet.GetDiskLun. return ss.GetDiskLun(diskName, diskURI, nodeName) </del> <ins> for _, disk := range disks { if disk.Lun != nil && (disk.Name != nil && diskName != "" && *disk.Name == diskName) || (disk.Vhd != nil && disk.Vhd.URI != nil && diskURI != "" && *disk.Vhd.URI == diskURI) || (disk.ManagedDisk != nil && *disk.ManagedDisk.ID == diskURI) { // found the disk glog.V(4).Infof("azureDisk - find disk: lun %d name %q uri %q", *disk.Lun, diskName, diskURI) return *disk.Lun, nil } } return -1, fmt.Errorf("Cannot find Lun for disk %s", diskName) </ins> } // GetNextDiskLun searches all vhd attachment on the host and find unused lun. Return -1 if all luns are used. func (c *controllerCommon) GetNextDiskLun(nodeName types.NodeName) (int32, error) { <del> // 1. vmType is standard, get with availabilitySet.GetNextDiskLun. if c.cloud.VMType == vmTypeStandard { return c.cloud.vmSet.GetNextDiskLun(nodeName) } // 2. vmType is Virtual Machine Scale Set (vmss), convert vmSet to scaleSet. ss, ok := c.cloud.vmSet.(*scaleSet) if !ok { return -1, fmt.Errorf("error of converting vmSet (%q) to scaleSet with vmType %q", c.cloud.vmSet, c.cloud.VMType) } // 3. If the node is managed by availability set, then get with availabilitySet.GetNextDiskLun. managedByAS, err := ss.isNodeManagedByAvailabilitySet(mapNodeNameToVMName(nodeName)) </del> <ins> disks, err := c.getNodeDataDisks(nodeName) </ins> if err != nil { <ins> glog.Errorf("error of getting data disks for node %q: %v", nodeName, err) </ins> return -1, err } <del> if managedByAS { // vm is managed by availability set. return ss.availabilitySet.GetNextDiskLun(nodeName) } // 4. Node is managed by vmss, get with scaleSet.GetNextDiskLun. return ss.GetNextDiskLun(nodeName) </del> <ins> used := make([]bool, maxLUN) for _, disk := range disks { if disk.Lun != nil { used[*disk.Lun] = true } } for k, v := range used { if !v { return int32(k), nil } } return -1, fmt.Errorf("all luns are used") </ins> } // DisksAreAttached checks if a list of volumes are attached to the node with the specified NodeName. func (c *controllerCommon) DisksAreAttached(diskNames []string, nodeName types.NodeName) (map[string]bool, error) { <del> // 1. vmType is standard, check with availabilitySet.DisksAreAttached. if c.cloud.VMType == vmTypeStandard { return c.cloud.vmSet.DisksAreAttached(diskNames, nodeName) } // 2. vmType is Virtual Machine Scale Set (vmss), convert vmSet to scaleSet. ss, ok := c.cloud.vmSet.(*scaleSet) if !ok { return nil, fmt.Errorf("error of converting vmSet (%q) to scaleSet with vmType %q", c.cloud.vmSet, c.cloud.VMType) </del> <ins> attached := make(map[string]bool) for _, diskName := range diskNames { attached[diskName] = false </ins> } <del> // 3. If the node is managed by availability set, then check with availabilitySet.DisksAreAttached. managedByAS, err := ss.isNodeManagedByAvailabilitySet(mapNodeNameToVMName(nodeName)) </del> <ins> disks, err := c.getNodeDataDisks(nodeName) </ins> if err != nil { <del> return nil, err </del> <ins> if err == cloudprovider.InstanceNotFound { // if host doesn't exist, no need to detach glog.Warningf("azureDisk - Cannot find node %q, DisksAreAttached will assume disks %v are not attached to it.", nodeName, diskNames) return attached, nil } return attached, err </ins> } <del> if managedByAS { // vm is managed by availability set. return ss.availabilitySet.DisksAreAttached(diskNames, nodeName) </del> <ins> for _, disk := range disks { for _, diskName := range diskNames { if disk.Name != nil && diskName != "" && *disk.Name == diskName { attached[diskName] = true } } </ins> } <del> // 4. Node is managed by vmss, check with scaleSet.DisksAreAttached. return ss.DisksAreAttached(diskNames, nodeName) </del> <ins> return attached, nil </ins> }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind cleanup What happened: VirtualMachineScaleSetVM and VirtualMachine are different data structures (because of different API versions), so the disk attach operation seems duplicate between vm and vmss. That means although disk operations are in same logic, some duplication are still required now. They should use same api version in the future and should be merged into same code base, e.g. move sharing logic to Refer What you expected to happen: vmss and vm disk operations should share same code logic. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
I have checked with currently we can’t merge, e.g. following two structs are different since they using different sdk-for-azure versions. While in the end when sdk-for-azure version is upgraded, above two versions should be same and at that time, we should merge functions like ‘AttachDisk’, ‘DetachDiskByName’, 'GetDiskLun', GetNextDiskLun, DisksAreAttached


Question: if err != nil { if apierrors.IsNotFound(err) { c.triggerTimeTracker.DeleteService(namespace, name) <ins> // The service has been deleted, return nil so that it won't be retried. return nil </ins> } return err }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: This requires 1.16 alpha feature: Endpoint Slices being enabled. After deleting a , EndpointSliceController kept enqueuing the service after syncing it successfully, until its retry budget exceeded. And the logs are misleading by indicating something wrong happened while the cleanup has been executed successfully: What you expected to happen: The deletion should be triggered only once and there should be no warning and error logs if the cleanup is successfully. How to reproduce it (as minimally and precisely as possible): Endpoint Slices feature a then delete it. kube-controller-manager logs, there would be repeated logs. Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
There are no sig labels on this issue. Please add a sig label by either: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. See the . The in method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/assign


Question: "sort" "strconv" "strings" <ins> "sync" </ins> "testing" "github.com/vmware/govmomi/find"

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: What you expected to happen: No data races How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
for the record as we tend to lose the logs in the gcs bucket
/sig instrumentation


Question: for _, file := range files { fileName := file.Name() // Make a copy It(fmt.Sprintf("tests that %v passes", fileName), func() { <ins> // A number of scripts only work on gce if !providerIs("gce", "gke") { By(fmt.Sprintf("Skipping Shell test %s, which is only supported for provider gce and gke (not %s)", fileName, testContext.Provider)) return } </ins> runCmdTest(filepath.Join(bashE2ERoot, fileName)) }) }


Question: WhenScaled:  apps.DeletePersistentVolumeClaimRetentionPolicyType, WhenDeleted: apps.DeletePersistentVolumeClaimRetentionPolicyType, }, <ins> // tests the case when no policy is set. nil, </ins> } { subtestName := pvcDeletePolicyString(policy) + "/StatefulSetAutoDeletePVCEnabled" if testName != "" {

The replicas is 3 but just create 1 pod 'web-0' and one pvc, then kube-controller-manager panic happend. Should create 3 pod and 3 pvc add '--feature-gates=StatefulSetAutoDeletePVC=true' arg for kube-controller-manager, then create statefulset with 2 replicas and volumeClaimTemplates. No response <details</details<details</details<details</details<details</details<details</details<details</details
/sig apps
I can reproduce the issue and want to fix it. It seems like variables cause . /assign
/triage accepted
Great find! Does this only happen on an upgraded cluster? I'll spin up a cluster and see if the e2e test catches this problem.
The e2e tests are still passing on a 1.24 cluster, so I suspect this is an upgrade problem. I wonder if there are other holes here.
I installed the Kubernetes 1.25 cluster directly using and still reproduced this issue, so it was probably not caused by upgrading the cluster.
Interesting. I thought that would set the policy to be non-nil. Let's keep this bug open after you submit your PR as I think something else is going on here---I don't think we should be seeing the null policy in the first place.
I suspect that somewhere accidentally set it to nil, and I'll keep looking at this code.
I tried the following (and also with parallel pod management), and it didn't seem to cause the panic. This was a 1.24 cluster, maybe this is only happening on head versions?
I just tried this on a kind cluster running on k8s head, and the problem didn't seem to reproduce. But this was setting the feature flag default in I wonder----does the api server need to have the feature gate as well? That would explain why defaulting wasn't happening.
As far as I'm concerned,I just set the in kube-controller-manager manifests to reproduce this issue. I repeat the yaml used as follows: I tested that in statefulset yaml, setting or not would cause the component to panic. The panic logs are the same as above, so ignore them here.


Question: WhenScaled:  apps.DeletePersistentVolumeClaimRetentionPolicyType, WhenDeleted: apps.DeletePersistentVolumeClaimRetentionPolicyType, }, <ins> // tests the case when no policy is set. nil, </ins> } { subtestName := pvcDeletePolicyString(policy) + "/StatefulSetAutoDeletePVCEnabled" if testName != "" {

statefulset with : The environment
I believe you---this doesn't seem to be well-documented, but it's my understanding that you need to set the feature gate on all of the system components. Obviously failing to do so shouldn't cause a panic, so thank you again for finding & fixing this bug! Let me ask around to see what the expected behavior is. I know there may be skew between kubelets and master versions, but I didn't expect to have to deal with skew between master components.
I read the code, and now I understand what you mean. This feature gate must be enabled in both the and components. We need to describe it in the to prevent someone from ignoring the component, which also needs to enable the feature gate.
I don't think we need all the system components, like the component, right? I read the original KEP documentation. This feature gate only involves components and :
I had a bit of discussion about this with others. Because upgrades don't occur atomically, any feature gate has to work regardless of how the other system components are configured (things shouldn't panic!). I suppose if the cluster administrator wants to track what needs to be enabled on each feature they can try to sequence which component has which feature gate. If it were me I'd try to make all the components consistent though.
This is certainly a method, but it is a bit more cumbersome for the administrator. It is best to have a place for the document to be queried by the administrator and avoid confusion. I don't think everyone is very familiar with the k8s mechanism. I suggest adding a partial description to this section of the : Before: After: I can submit this PR on .
This is going to be true of any api change. When you tried to use it with only adding the controller manager feature gate, I think you would be able to set the retain policy, but there wouldn't be any defaulting or validation (?).
I've seen other feature gates that say what components are needed to enable it to use the new feature. For example: If it's not necessary, then I close .
only adding the controller manager feature gate, I think you would be able to set the retain policy, but there wouldn't be any defaulting or validation (?). I remember just enabling it in kube-controller-manager.


Question: WhenScaled:  apps.DeletePersistentVolumeClaimRetentionPolicyType, WhenDeleted: apps.DeletePersistentVolumeClaimRetentionPolicyType, }, <ins> // tests the case when no policy is set. nil, </ins> } { subtestName := pvcDeletePolicyString(policy) + "/StatefulSetAutoDeletePVCEnabled" if testName != "" {

it using the retain policy, but the statefulset does not have the policy from output of (It should be the apiserver authentication forbids it)
Yeah I think that makes sense. Thanks for finding this issue! After your PR with the panic fix is submitted, I think we can close this?
Yes. I agree. /close
Closing this issue. <detailsIn response to : Yes. I agree. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: FindBoundSatsified:   true, }, eventReason: "FailedScheduling", <del> expectError: makePredicateError("1 VolumeBindingNoMatch"), </del> <ins> expectError: makePredicateError("1 node(s) didn't find available persistent volumes to bind"), </ins> }, "bound-and-unbound-unsatisfied": { volumeBinderConfig: &persistentvolume.FakeVolumeBinderConfig{

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind feature The is not user friendly. There has been to make it better, but we still print in the pending message. We should change these errors to a more human readable message. For example, should be changed to something like . /sig scheduling


Question: ginkgo.By("Waiting for container to stop restarting") stableCount := int(0) stableThreshold := int(time.Minute / framework.Poll) <del> err = wait.PollImmediate(framework.Poll, 2*time.Minute, func() (bool, error) { </del> <ins> err = wait.PollImmediate(framework.Poll, framework.PodStartTimeout, func() (bool, error) { </ins> pod, err := f.ClientSet.CoreV1().Pods(f.Namespace.Name).Get(context.TODO(), pod.Name, metav1.GetOptions{}) if err != nil { return false, err

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --ci-kubernetes-e2e-ubuntu-gce [sig-storage] Subpath Container restart should verify that container can restart successfully after configmaps modified links to appreciated links to specific failures in spyglass appreciated Triage shows that this has been flaking quite frequently: It has flaked in 5 of the last 10 runs: /sig storage /priority important-soon <!-- Please see the deflaking doc () for more guidance! --
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Noticed some more flakes on this one
Yep, looks like this is still flaking quite frequently: Going to re-ping SIG Storage.
/assign
Test is flaking maybe half of the time. ~This test is failing in gce-ubuntu-master-default - not sure if it's failing in any other dashboards, nor whether this is run anywhere else.~ edit: This is failing on Ubuntu only. Not sure if this is limited to master branch. Fix to the original issue was in since 1.19. This is a subpath test which a configmap a container that mounts the configmap a liveness probe and waits for the container to restart. liveness probe and waits for the container to stop restarting. The test failed at the last step - the container never stopped restarting. Timeout is 2min. The liveness probe continuously checks whether a "probeFile" exists in (an emptyDir), and if it doesn't exist, whether the configmap file contains the modified value. The probeFile essentially acts as a condition which guards the probe from checking the configmap file. The test fails the liveness probe by removing the probeFile without updating the configmap just yet. It then fixes the liveness probe by updating the configmap to the modified value.


Question: ginkgo.By("Waiting for container to stop restarting") stableCount := int(0) stableThreshold := int(time.Minute / framework.Poll) <del> err = wait.PollImmediate(framework.Poll, 2*time.Minute, func() (bool, error) { </del> <ins> err = wait.PollImmediate(framework.Poll, framework.PodStartTimeout, func() (bool, error) { </ins> pod, err := f.ClientSet.CoreV1().Pods(f.Namespace.Name).Get(context.TODO(), pod.Name, metav1.GetOptions{}) if err != nil { return false, err

So if the container keeps restarting, the liveness probe could've been failing continuously within 2min, due to one of the following: (? not sure about this one) Command can't be executed (for example if container can't start) configmap file was never updated. Within 2min, the pod only restarted 5 times. The liveness probe's period is 2s, initial delay is 1s, and failure threshold is 1, so a restart due to liveness probe failure is triggered every 3s. The image is busybox. The command is just , so it's unlikely for the container to have crashed. There were no liveness probe unhealthy events after the test fixes the probe, and yet it continued to restart. No more events after probe is fixed.
Could we check if the configmap value was updated for sure to rule out that scenario ? I have not tried to run an end to end test locally :) so am not sure how to debug it
Will do! I ran the test in a GKE Ubuntu cluster running 1.19.6 and couldn't reproduce the failure. I'm now trying to run a cluster on GCE with Ubuntu nodes running k8s master, but haven't been successful after trying a number of techniques. If anyone happens to know how to do this let me know; I'll keep asking around otherwise.
I got a repro on a GKE cluster backed by COS. After the test failed the container ends up in Running state eventually, and the configmap file outputs the expected modified value. The fix is to increase the test timeout. PR coming up.


Question: It("should lookup the Schema by its GroupVersionKind", func() { schema = resources.LookupResource(gvk) Expect(schema).ToNot(BeNil()) <del> }) var sarspec *proto.Kind It("should be a Kind and have a spec", func() { </del> sar := schema.(*proto.Kind) Expect(sar).ToNot(BeNil()) Expect(sar.Fields).To(HaveKey("spec")) specRef := sar.Fields["spec"].(proto.Reference) Expect(specRef).ToNot(BeNil()) Expect(specRef.Reference()).To(Equal("io.k8s.api.authorization.v1.SubjectAccessReviewSpec")) <del> sarspec = specRef.SubSchema().(*proto.Kind) Expect(sarspec).ToNot(BeNil()) </del> <ins> Expect(specRef.SubSchema().(*proto.Kind)).ToNot(BeNil()) </ins> }) })

/kind failing-test <!-- If this is a flake, please add: /kind flake --<!-- Please assign a SIG using: /sig SIG-NAME --
/sig cli
There may have two issues to fix: doesn't support run with assertion must check whether schema is nil first
/assign
/triage accepted


Question: kubetypes.HTTPSource, &api.Pod{ ObjectMeta: api.ObjectMeta{ <del> UID:       "111", Name:      "foo" + "-" + hostname, Namespace: "mynamespace", SelfLink: getSelfLink("foo-"+hostname, "mynamespace"), </del> <ins> UID:         "111", Name:        "foo" + "-" + hostname, Namespace:   "mynamespace", Annotations: map[string]string{kubetypes.ConfigHashAnnotationKey: "111"}, SelfLink:    getSelfLink("foo-"+hostname, "mynamespace"), </ins> }, Spec: api.PodSpec{ NodeName:                      hostname,

See for more context. Currently, kubelet compares the spec of a mirror pod (created via apiserver) with that of a static pod (config stored as a file) to decide whether the two are considered identical. If they differ, kubelet would delete the mirror pod and re-create it with the correct spec. Due to the fact that apiserver may set default values for some fields in the spec of the mirror pod, the specs could deviate undesirably. To avoid problems like this, we can store the config of a static pod as an annotation in the mirror pod, and compare the static pod with the annotation.
Would a hash of the config be sufficient if we're only going to compare? The whole spec would generally be more useful, but I wonder if that is too much data to put in an annotation.
This seems totally reasonable to me, but I'm still desperately curious why you can't just bypass defaulting when you generate mirror pod from static pod? (i.e. just use the static pod's configuration verbatim)
It's not kubelet who applies the default values here. The admission plugin was only run at the apiserver, and to bypass that, you have to ask apiserver to treat mirror pods differently and potentially at multiple places.
On May 22, 2015 10:38 AM, "Yu-Ju Hong" wrote: Understood, but is it really multiple places? If so then I agree it's a bad idea. Otherwise it might be worth considering as an alternative.
It's untenable for Kubelet to assume to know/circumvent all fields that will be set by other entities: admission control, defaulting, late initialization, auto-something, whatever. It needs to obey the same rules as other clients.
I don't understand why you would ever want to apply any of the things you mentioned to static pods. Static pods should be copied verbatim into the mirror pods, right? That's what I'm proposing.
What if the master is updated before the kubelet, and the master adds a new API field with a default value?
That case should be discussed in
I agree that mirror pod should reflect the true status of the static pod running on the node, and no default values should be set in this case.


Question: kubetypes.HTTPSource, &api.Pod{ ObjectMeta: api.ObjectMeta{ <del> UID:       "111", Name:      "foo" + "-" + hostname, Namespace: "mynamespace", SelfLink: getSelfLink("foo-"+hostname, "mynamespace"), </del> <ins> UID:         "111", Name:        "foo" + "-" + hostname, Namespace:   "mynamespace", Annotations: map[string]string{kubetypes.ConfigHashAnnotationKey: "111"}, SelfLink:    getSelfLink("foo-"+hostname, "mynamespace"), </ins> }, Spec: api.PodSpec{ NodeName:                      hostname,

However, I think it'd be harder to implement this way since, as stated above, that there are many things to circumvent in the apiserver itself. It could also be harder to keep this special case from being broken by a yet another new feature/plugin. Keeping the hash of the config in the annotation seems like a good compromise even though it may not reflect the true status 100% accurately. I think this is acceptable. What do you think?
Sure, if you think the annotation approach is easier, I would go with that. I guess in the case Brian mentoined (master adds a new API field with a default value), you would not update the mirror pod, because the static pod definition has not changed?
Yes, a hash would be sufficient.
It shouldn't be necessary to update the mirror pod in the case that a new default field was updated, because that field should be populated automatically by the apiserver on decode from etcd.
We should fix this so we don't have this problem again in 1.2.


Question: { code: http.StatusBadRequest, expected: &Error{ <del> Retriable:      true, </del> <ins> Retriable:      false, </ins> HTTPStatusCode: http.StatusBadRequest, RawError:       fmt.Errorf("HTTP response: 400"), },

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: If you create a service that reference a Basic IP in a standard load balancer cluster, the service will stuck at "Pending" state for days without detailed error from Azure response. When you describe the service, it only has following: What you expected to happen: In this case Azure responded 400 with detailed error message that the referenced IP is a basic IP and cannot be used on standard LB. I would expect the detailed error message show up as events for easier diagnostic. How to reproduce it (as minimally and precisely as possible): Create a service that reference a Basic IP in a standard load balancer cluster Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/assign /area provider/azure /sig cloud-provider
/area cloudprovider


Question: const ( operationDeleteCollection operation = "deleteCollection" operationList             operation = "list" <ins> // assume a default estimate for finalizers to complete when found on items pending deletion. finalizerEstimateSeconds int64 = int64(15) </ins> ) // operationKey is an entry in a cache.

Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): No What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): "LRUExpireCache" Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT What happened: requires write lock since needs to manipulate its list to track recently used item. Currently it uses read lock so it may introduce race condition. What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else do we need to know:
This was fixed in


Question: // in https://github.com/kubernetes/kubernetes/issues/120720. // We are already hinting the external cloud provider via the annotation AnnotationAlphaProvidedIPAddr. if !nodeIPSpecified { <ins> node.Status.Addresses = []v1.NodeAddress{ {Type: v1.NodeHostName, Address: hostname}, } </ins> return nil } }

It looks like there was a change that first appeared in Kubernetes 1.29, where in 1.29 the flag on kubelet is no longer being used to populate the Node's Addresses.Hostname value. Here is the relevant output from a for a 1.28 cluster with : And here's the output from a 1.29 cluster with the same configuration: Here is the full list of parameters in case that is helpful: This is from an IBM Cloud Kubernetes Service 1.28 and 1.29 cluster. We do not think there was any cloud-configuration-manager or any other related change that would cause this difference. Is it possible it is a side effect of ? I expected the behavior to be consistent between Kubernetes releases, since I didn't find any reference to this change in Create an IBM Cloud Kubernetes Service cluster at 1.28 and 1.29 and see the difference No response <details</details<details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
yeah, it seems likely my change broke this, it should keep working as you say, let me take a look /assign
/sig network /sig cloud-provider
/assign /unassign Thanks
FYI it was an unintended side effect of:
Do the sig cloud provider folks need to help here? We did make external cloud providers the default as of 1.29, which may have uncovered this. Let us know!
No, it's just a dumb kubelet startup bug. Well, not dumb, but easy to fix.
It seems the problem is that once kubelet sets the hostname the cloud controller manager does not override it so maybe the right behavior is only override it if is actually overridden ?
what problem?


Question: // in https://github.com/kubernetes/kubernetes/issues/120720. // We are already hinting the external cloud provider via the annotation AnnotationAlphaProvidedIPAddr. if !nodeIPSpecified { <ins> node.Status.Addresses = []v1.NodeAddress{ {Type: v1.NodeHostName, Address: hostname}, } </ins> return nil } }

we are discussing if the fix should be backported or not It seems you are relaying on the with type Hostname and this not clearly documented We also find out recently that the behavior of this address will be overriden if the external cloud provider configures it if the external cloud provider returns the hostname , then this problem will be solved without need to backport
(Well, to clarify my comments there: it's not well-defined between different clouds and node linux distros. If you know what cloud and distro you are on, then the behavior might be consistent and usable, but that still makes for a lousy kubernetes API.) Anyway, the question here is: was this actually breaking anything major, or did you just happen to notice this and think "that's weird"? (Also, if anyone else was broken by this bug, feel free to chime in; trying to figure out what to do about backporting this, because the original fix for this bug ended up introducing another bug, and the fix for that_ bug is maybe not backport-able...)
So far I have only heard from a single customer (who was using IBM Cloud Kubernetes Service) who had something in their cluster relying on cluster node's Address.Hostname value, and so when they updated to 1.29 they were broken. I believe they just modified their code to get this "Hostname" value (which was really the node IP) from somewhere else. So, to answer the question, it is somewhere between the "I noticed it and thought 'that's weird'" option and the "it broke something major" option. We have this documented in our 1.29 release notes for IBM Cloud Kubernetes Service, so other customers will have fair warning before updating. So at this point I don't consider fixing this in 1.29 as essential.


Question: return err } glog.V(4).Info("Successfully created federation controller manager deployment") <del> fmt.Println(cmdOut, " done") </del> <ins> fmt.Fprintln(cmdOut, " done") </ins> fmt.Fprint(cmdOut, "Updating kubeconfig...") glog.V(4).Info("Updating kubeconfig")

Kubernetes version (use ): kubernetes/master Environment: na What happened: What you expected to happen: How to reproduce it (as minimally and precisely as possible): see above Anything else we need to know: nope, have a fix.
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list


Question: "//test/e2e_node/services:go_default_library", "//test/e2e_node/system:go_default_library", "//test/utils:go_default_library", <ins> "//vendor:github.com/coreos/go-systemd/util", </ins> "//vendor:github.com/davecgh/go-spew/spew", "//vendor:github.com/golang/glog", "//vendor:github.com/kardianos/osext",

Kubernetes version (use ): 6 Environment: GCE What happened: I created a cluster for Heapster testing purposes to debug failing test in It turned out that Summary API does not export system/misc stats in SystemContainers section (though there are kubelet and docker stats). Everything works as expected on nodes. See also https://k8s-
What node OS were you using? We don't create the system (misc) container on systemd, so it's expected that GCI nodes don't report those stats.
The default configuration for 1.4, which is: master: GCI nodes: Container VM
Are the nodes running 1.4.6 too? There was a bug with this in 1.4.5, but it should have been fixed in 1.4.6.
Yes, both are running in 1.4.6. If you want to reproduce from Heapster current head run . It will take some time and as a side effect you should have a cluster with the mentioned problem. This was catch by integration tests and I temporary disabled the check here
if you have spare cycles, can you look into this?
friendly ping
Looking at it now
I have found a couple things so far. On the containerVM nodes, cadvisor is exporting a container with "name": "/system" On the GCI Master, cadvisor is exporting a container with "name": "" The has this: "{% set system_container = "--system-cgroups=/system" -%}" (/system is the default misc container name) So the real question is: Should cadvisor export "/system" instead of "" on GCI? Or should we be configuring GCI to use "--system-"?
I thought we fix this very issue for 1.4 branch by a while back. Are you running into the same issue with 1.5?
I think I told you the wrong thing earlier. We don't support the misc (/system) cgroup on systemd systems (I can't remember why, but it was an explicit decision - or might know). The is something different, I believe a base systemd cgroup (slice). As for the misc container on ContainerVM nodes, we've been testing for the misc container in 1.5+ in the summary api test (https://k8s-) for a while, and I haven't seen this issue.


Question: "//test/e2e_node/services:go_default_library", "//test/e2e_node/system:go_default_library", "//test/utils:go_default_library", <ins> "//vendor:github.com/coreos/go-systemd/util", </ins> "//vendor:github.com/davecgh/go-spew/spew", "//vendor:github.com/golang/glog", "//vendor:github.com/kardianos/osext",

So if this is still reproducible, it's limited to 1.4. See this the bug was created for the post-fix.
Found the explanation here: The should use the flag to know the location that has the processes that are associated with , but it should not modify the cgroups of existing processes on the system during bootstrapping of the node. This is because is the on the host and it has not delegated authority to the to change how it manages .
IIUC this is intended behavior on systemd nodes. Based on the explanation provided, the kubelet doesnt have the authority to control "misc" processes on systemd.
+1. "misc" is just a bandaid for cvm. , David Ashpole <:
Ok, so I'm closing this issue.


Question: return } <del> klog.V(4).Infof("Updating customresourcedefinition %s", oldCRD.Name) </del> <ins> klog.V(4).Infof("Updating customresourcedefinition %s", newCRD.Name) r.removeStorage_locked(newCRD.UID) } </ins> <del> if oldInfo, ok := storageMap[types.UID(oldCRD.UID)]; ok { </del> <ins> // removeStorage_locked removes the cached storage with the given uid as key from the storage map. This function // updates r.customStorage with the cleaned-up storageMap and tears down the old storage. // NOTE: Caller MUST hold r.customStorageLock to write r.customStorage thread-safely. func (r *crdHandler) removeStorage_locked(uid types.UID) { storageMap := r.customStorage.Load().(crdStorageMap) if oldInfo, ok := storageMap[uid]; ok { </ins> // Copy because we cannot write to storageMap without a race // as it is used without locking elsewhere. storageMap2 := storageMap.clone() // Remove from the CRD info map and store the map <del> delete(storageMap2, types.UID(oldCRD.UID)) </del> <ins> delete(storageMap2, uid) </ins> r.customStorage.Store(storageMap2) // Tear down the old storage

Which jobs are failing: pull-kubernetes-integration Which test(s) are failing: TestWebhookConverterWithDefaulting /assign /priority important-soon /area custom-resources /sig api-machinery /kind flake /milestone v1.16
/cc
several of the webhook converter tests have similar flakes. it looks like the wait for storage version to become effective times out occasionally Rare flakes seen in all of these tests: TestWebhookConverterWithWatchCache/nontrivial-converter TestWebhookConverterWithWatchCache/metadata-mutating-converter TestWebhookConverterWithDefaulting/metadata-mutating-converter TestWebhookConverterWithWatchCache/noop-converter TestWebhookConverterWithDefaulting/empty-response TestWebhookConverterWithoutWatchCache/nontrivial-converter/check-2 TestWebhookConverterWithoutWatchCache/noop-converter/check-0/v1beta2 TestWebhookConverterWithoutWatchCache/nontrivial-converter/check-0/v1beta2
cc
Yikes, I’ll sort it out.
iiuc sending an empty patch gets short-circuited and doesn't bump CR generation, even if the storage version gets changed, which means we do not re-write to etcd and go through the encoding path this would be racing and doing noop's if the first pass of didn't get the expected storage version I will send a fix to make the patch do actual mutation
It doesn't bump generation, but the bytes to persist in etcd would still be different if the storage version had changed, so they should persist. You can verify that by changing the storage version on a CRD yourself, do an empty patch, and see if the resourceVersion changes.
you're right. Generation is for non-metadata change only, and the RV changes. I queried etcd locally and the apiVersion did change for an empty patch (sidetracking: another observation is after changing the storage version (say v1 -v2; noop convertor), empty-patching v2 endpoint would bump generation, while empty-patching v1 endpoint wouldn't)
another theory: our cached storage strategy could be stale if the CRD informer merges multiple events together (e.g. the informer does a re-list).


Question: return } <del> klog.V(4).Infof("Updating customresourcedefinition %s", oldCRD.Name) </del> <ins> klog.V(4).Infof("Updating customresourcedefinition %s", newCRD.Name) r.removeStorage_locked(newCRD.UID) } </ins> <del> if oldInfo, ok := storageMap[types.UID(oldCRD.UID)]; ok { </del> <ins> // removeStorage_locked removes the cached storage with the given uid as key from the storage map. This function // updates r.customStorage with the cleaned-up storageMap and tears down the old storage. // NOTE: Caller MUST hold r.customStorageLock to write r.customStorage thread-safely. func (r *crdHandler) removeStorage_locked(uid types.UID) { storageMap := r.customStorage.Load().(crdStorageMap) if oldInfo, ok := storageMap[uid]; ok { </ins> // Copy because we cannot write to storageMap without a race // as it is used without locking elsewhere. storageMap2 := storageMap.clone() // Remove from the CRD info map and store the map <del> delete(storageMap2, types.UID(oldCRD.UID)) </del> <ins> delete(storageMap2, uid) </ins> r.customStorage.Store(storageMap2) // Tear down the old storage

Two scenarios: -create-update events merged into an update event: strategy with UID:a was created on demand for CRD Foo Foo was deleted and recreated strategy with UID:b was created on demand for the new Foo new Foo got updated (e.g. switching storage version) the strategy with UID:b is supposed to be deleted and , to reflect the last update. But if the CRD informer merges the delete-create-update events into , the strategy with UID:a will be deleted, but the strategy with UID:b won't. -update events merged into a create event: strategy with UID:a was created on demand for CRD Foo Foo was updated (e.g. switching storage version) the strategy with UID:a is supposed to be deleted and , to reflect the last update. But if the CRD informer merges the create-update events into , the strategy with UID:a won't be deleted (as we don't react on create events). another race found in , which could happen in a single update.
I think I :