Question: } factory.CreateFromConfig(policy) <ins> hpa := factory.GetHardPodAffinitySymmetricWeight() if hpa != v1.DefaultHardPodAffinitySymmetricWeight { t.Errorf("Wrong hardPodAffinitySymmetricWeight, ecpected: %d, got: %d", v1.DefaultHardPodAffinitySymmetricWeight, hpa) } } func TestCreateFromConfigWithHardPodAffinitySymmetricWeight(t *testing.T) { var configData []byte var policy schedulerapi.Policy handler := utiltesting.FakeHandler{ StatusCode:   500, ResponseBody: "", T:            t, } server := httptest.NewServer(&handler) defer server.Close() client := clientset.NewForConfigOrDie(&restclient.Config{Host: server.URL, ContentConfig: restclient.ContentConfig{GroupVersion: &api.Registry.GroupOrDie(v1.GroupName).GroupVersion}}) informerFactory := informers.NewSharedInformerFactory(client, 0) factory := NewConfigFactory( v1.DefaultSchedulerName, client, informerFactory.Core().V1().Nodes(), informerFactory.Core().V1().Pods(), informerFactory.Core().V1().PersistentVolumes(), informerFactory.Core().V1().PersistentVolumeClaims(), informerFactory.Core().V1().ReplicationControllers(), informerFactory.Extensions().V1beta1().ReplicaSets(), informerFactory.Apps().V1beta1().StatefulSets(), informerFactory.Core().V1().Services(), v1.DefaultHardPodAffinitySymmetricWeight, ) // Pre-register some predicate and priority functions RegisterFitPredicate("PredicateOne", PredicateOne) RegisterFitPredicate("PredicateTwo", PredicateTwo) RegisterPriorityFunction("PriorityOne", PriorityOne, 1) RegisterPriorityFunction("PriorityTwo", PriorityTwo, 1) configData = []byte(`{ "kind" : "Policy", "apiVersion" : "v1", "predicates" : [ {"name" : "TestZoneAffinity", "argument" : {"serviceAffinity" : {"labels" : ["zone"]}}}, {"name" : "TestRequireZone", "argument" : {"labelsPresence" : {"labels" : ["zone"], "presence" : true}}}, {"name" : "PredicateOne"}, {"name" : "PredicateTwo"} ], "priorities" : [ {"name" : "RackSpread", "weight" : 3, "argument" : {"serviceAntiAffinity" : {"label" : "rack"}}}, {"name" : "PriorityOne", "weight" : 2}, {"name" : "PriorityTwo", "weight" : 1} ], "hardPodAffinitySymmetricWeight" : 10 }`) if err := runtime.DecodeInto(latestschedulerapi.Codec, configData, &policy); err != nil { t.Errorf("Invalid configuration: %v", err) } factory.CreateFromConfig(policy) hpa := factory.GetHardPodAffinitySymmetricWeight() if hpa != 10 { t.Errorf("Wrong hardPodAffinitySymmetricWeight, ecpected: %d, got: %d", 10, hpa) } </ins> } func TestCreateFromEmptyConfig(t *testing.T) {

( noticed this) HardPodAffinitySymmetricWeight should be in the , not in KubeSchedulerConfiguration If/when fixing this, we need to be careful to do it in a backward-compatilble way.
Do you mean ? It seems scheduler policy config's args can not passed to FitPredicates; it need to update related interface. For the : mark to be deprecated and set scheduler policy config to HardPodAffinitySymmetricWeight if policy config is empty.
Are you working on this? if no, I can take it.
, nop; please go ahead.
/ / / , When reviewing the PR, there's one question on : the default value of is 1, which means the weight (1) will always add to priorities if term matched (), does it our expectation? And the user has to set it to 0 to disable it. IMO, the default value of should be 0.
The value 1 (instead of 0) was chosen so that if you say A must run with B, there is an automatic preference that B should run with A. This seems like reasonable behavior.
Thanks very much; reasonable case to me. But it seems we can not disable it: if set to 0, the default funcs will set it to default (1); similar result to un-set. Anyway, no user complain this now :).


Question: opName := op.Name return wait.Poll(operationPollInterval, operationPollTimeoutDuration, func() (bool, error) { start := time.Now() <del> gce.operationPollRateLimiter.Accept() </del> duration := time.Now().Sub(start) if duration > 5*time.Second { glog.Infof("pollOperation: waited %v for %v", duration, opName)

On a large cluster, the routecontroller basically makes O(n) requests all at once, and most of them error out with "Rate Limit Exceeded". We then wait for operations to complete on the ones that did stick, and damn the rest, they'll be caught next reconcile (which is a while). In fact, in some cases it appears we're spamming hard enough to get rate limited by the upper-level API DOS protections and seeing something akin to , e.g.: We can do better than this!
c.f. , which is the other leg of this problem
I took a look into code and it seems that we are doing full reconciliation every 10s. If we need to create 2000 routes at the beginning it seems to frequently. I think that we need to: first compute how many routes we have to create based on that try to spread them over some time to reduce possibility of causing 'rate limit exceeded' I will try to put some small PR together as a starting point.
I'm reopening this one, since this doesn't seem to be solved. So basically something strange is happening here. With PR we have 10/s limit on API calls send to GCE. And we are still rate-limitted. However, according to documentation we are allowed for 20/s: So something either doesn't work or is weird.
What is super interesting from the logs of recent run: It starts to creating routes: After 2 minutes from that, we are hitting "Rate limit Exceeded" for the first time: And only after 9 minutes we create the first route ever. The second one is created after 15 minutes, and from that point we create a number of them So it seems that nothing useful is happening for the first 2 minutes, and then between 3m30 and 9m30
OK - so I think that what is happening is that all controllers are sharing GCE interface (and thuse they have common throttling). So if one controller is generating a lot of API calls, then other controllers may be throttled. has a hypothesis that it may be caused by nodecontroller.
One thing my PR didn't address (because it was late): Now we have some obnoxious differences in logging when you go to make a request, because the versus could actually be absorbing a fair amount of ratelimiter.


Question: opName := op.Name return wait.Poll(operationPollInterval, operationPollTimeoutDuration, func() (bool, error) { start := time.Now() <del> gce.operationPollRateLimiter.Accept() </del> duration := time.Now().Sub(start) if duration > 5*time.Second { glog.Infof("pollOperation: waited %v for %v", duration, opName)

It could before because of the operation rate limiter and operation polling anyways, but now could be worse.
- yes I'm aware of that. And probably this is exactly the case here.
I got fooled by that - I guess we should fix it somehow.
One approach to fixing it is a similar approach to the operation polling code, but in the : warn if we ratelimited for more than N seconds (and dump the request or something? not clear how to clarify what we were sending).
I found something else - Kubelet is also building GCE interface: We have 2000 kubelets, so if all of them send multiple requests to GCE, we may hit limits.
Any idea in what circumstances Kubelet will contact cloud provider?
Hmm - it seems it contacts GCE exactly once at the beginning. So it doesn't explain much.
I think it's only here: I've rarely seen GCE ratelimit readonly calls, in practice, and kubelets come up pretty well staggered.
OK - so with logs that I , it is pretty clear that we are very heavily throttled on GCE api calls. In particular, when I was running 1000-node cluster, I see lines where we were throttle for 1m30. I'm pretty sure it's significantly higher in 2000-node cluster. I'm going to investigate a bit more where all those reuqests come from.
Actually - it seems that we have a pretty big problem here. Basically, Kubelet seems to send a lot of requests to GCE. So to clarify: every time Kubelet is updating NodeStatus (so every 10s by default) it is sending an API call to GCE to get it's not addresses: that means, that if we have 2000 nodes, kubelets generate 200 QPS to GCE (which is significantly more than we can afford) We should think how to solve this issue, but I think the options are: increasing QPS quota or calling GCE only once per X statusUpdates.
The second issue is that we are throttled at the controller level too, and this one I'm tryin to understand right now.


Question: opName := op.Name return wait.Poll(operationPollInterval, operationPollTimeoutDuration, func() (bool, error) { start := time.Now() <del> gce.operationPollRateLimiter.Accept() </del> duration := time.Now().Sub(start) if duration > 5*time.Second { glog.Infof("pollOperation: waited %v for %v", duration, opName)

- that said, any rate-limitting on our side, will not solve the kubelet-related problem
Hmm - actually looking into implementation it seems that NodeAddresses are only contacting metadata server, so I'm not longer that convinced...
Also, I think that important thing is that a single call to GCE API can translate to multiple api calls to GCE. In particular - CreateRoute translates to: get instance insert route a bunch of getOperation (until this is finished)
OK - so I think that what is happening here is that since CreateRoute translates to: get instance insert route a bunch of getOperations That means that if we call say 2000 CreateRoute() at the same time, all of them will try to issue: getInstance at the beginning. So we will quickly accumulate 2000 GET requests in the queue. Once they are processed, we generate the "POST route" then for any processed. So It's kind of expected what is happening here. So getting back to the Zach`s PR - I think that throttling POST and GET and OPERATION calls separate is kind of good idea in general.
But I still don't really understand why do we get "RateLimitExceeded". Is that because of Kubelets?
I think that one problem we have at the RouteController level is that if CreateRoute fails, we simply don't react on it. And we will wait at least 5 minutes before even retrying it. We should add some retries at the route controller level too. I will send out some short PR for it.
But I still don't understand why do we get those "RateLimitExceeded" error
OK - so as an update. In GCE, we have a separate rate-limit for number of in-flight CreateRoute call per project. is supposed to address this issue.
OK - so this one is actually fixed. I mean that it is still long, but currently it's purely blocked on GCE.
As a comment - we have an internal bug for it already.


Question: } if err := proxyHandler.ServeConn(conn); err != nil && !shoulderror { <ins> // If the connection request is closed before the channel is closed // the test will fail with a ServeConn error. Since the test only return // early if expects shouldError=true, the channel is closed at the end of // the test, just before all the deferred connections Close() are executed. </ins> if isClosed() { return }

TestRoundTripSocks5AndNewConnection Disabled by No response No response No response /sig
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
there is a race shutting down the socks5 server


Question: var podClient *framework.PodClient const ( podCheckInterval     = 1 * time.Second <del> podWaitTimeout       = 3 * time.Minute </del> postStartWaitTimeout = 2 * time.Minute preStopWaitTimeout   = 30 * time.Second ) Context("when create a pod with lifecycle hook", func() { <ins> var targetIP string podHandleHookRequest := &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name: "pod-handle-http-request", }, Spec: v1.PodSpec{ Containers: []v1.Container{ { Name:  "pod-handle-http-request", Image: "gcr.io/google_containers/netexec:1.7", Ports: []v1.ContainerPort{ { ContainerPort: 8080, Protocol:      v1.ProtocolTCP, }, }, }, }, }, } </ins> BeforeEach(func() { podClient = f.PodClient() <ins> By("create the container to handle the HTTPGet hook request.") newPod := podClient.CreateSync(podHandleHookRequest) targetIP = newPod.Status.PodIP </ins> }) <del> Context("when it is exec hook", func() { var file string testPodWithExecHook := func(podWithHook *v1.Pod) { podCheckHook := getExecHookTestPod("pod-check-hook", // Wait until the file is created. []string{"sh", "-c", fmt.Sprintf("while [ ! -e %s ]; do sleep 1; done", file)}, ) By("create the pod with lifecycle hook") podClient.CreateSync(podWithHook) if podWithHook.Spec.Containers[0].Lifecycle.PostStart != nil { By("create the hook check pod") podClient.Create(podCheckHook) By("wait for the hook check pod to success") podClient.WaitForSuccess(podCheckHook.Name, postStartWaitTimeout) } By("delete the pod with lifecycle hook") podClient.DeleteSync(podWithHook.Name, metav1.NewDeleteOptions(15), framework.DefaultPodDeletionTimeout) if podWithHook.Spec.Containers[0].Lifecycle.PreStop != nil { By("create the hook check pod") podClient.Create(podCheckHook) By("wait for the prestop check pod to success") podClient.WaitForSuccess(podCheckHook.Name, preStopWaitTimeout) } </del> <ins> testPodWithHook := func(podWithHook *v1.Pod) { By("create the pod with lifecycle hook") podClient.CreateSync(podWithHook) if podWithHook.Spec.Containers[0].Lifecycle.PostStart != nil { By("check poststart hook") Eventually(func() error { return podClient.MatchContainerOutput(podHandleHookRequest.Name, podHandleHookRequest.Spec.Containers[0].Name, `GET /echo?msg=poststart`) }, postStartWaitTimeout, podCheckInterval).Should(BeNil()) </ins> } <del> BeforeEach(func() { file = "/tmp/test-" + string(uuid.NewUUID()) }) AfterEach(func() { By("cleanup the temporary file created in the test.") cleanupPod := getExecHookTestPod("pod-clean-up", []string{"rm", file}) podClient.Create(cleanupPod) podClient.WaitForSuccess(cleanupPod.Name, podWaitTimeout) }) It("should execute poststart exec hook properly [Conformance]", func() { podWithHook := getExecHookTestPod("pod-with-poststart-exec-hook", // Block forever []string{"tail", "-f", "/dev/null"}, ) podWithHook.Spec.Containers[0].Lifecycle = &v1.Lifecycle{ PostStart: &v1.Handler{ Exec: &v1.ExecAction{Command: []string{"touch", file}}, }, } testPodWithExecHook(podWithHook) }) It("should execute prestop exec hook properly [Conformance]", func() { podWithHook := getExecHookTestPod("pod-with-prestop-exec-hook", // Block forever []string{"tail", "-f", "/dev/null"}, ) podWithHook.Spec.Containers[0].Lifecycle = &v1.Lifecycle{ PreStop: &v1.Handler{ Exec: &v1.ExecAction{Command: []string{"touch", file}}, </del> <ins> By("delete the pod with lifecycle hook") podClient.DeleteSync(podWithHook.Name, metav1.NewDeleteOptions(15), framework.DefaultPodDeletionTimeout) if podWithHook.Spec.Containers[0].Lifecycle.PreStop != nil { By("check prestop hook") Eventually(func() error { return podClient.MatchContainerOutput(podHandleHookRequest.Name, podHandleHookRequest.Spec.Containers[0].Name, `GET /echo?msg=prestop`) }, preStopWaitTimeout, podCheckInterval).Should(BeNil()) } } It("should execute poststart exec hook properly [Conformance]", func() { lifecycle := &v1.Lifecycle{ PostStart: &v1.Handler{ Exec: &v1.ExecAction{ Command: []string{"sh", "-c", "curl http://" + targetIP + ":8080/echo?msg=poststart"}, </ins> }, <del> } testPodWithExecHook(podWithHook) }) }) Context("when it is http hook", func() { var targetIP string podHandleHookRequest := &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name: "pod-handle-http-request", </del> }, <del> Spec: v1.PodSpec{ Containers: []v1.Container{ { Name:  "pod-handle-http-request", Image: "gcr.io/google_containers/netexec:1.7", Ports: []v1.ContainerPort{ { ContainerPort: 8080, Protocol:      v1.ProtocolTCP, }, }, }, </del> <ins> } podWithHook := getPodWithHook("pod-with-poststart-exec-hook", "gcr.io/google_containers/hostexec:1.2", lifecycle) testPodWithHook(podWithHook) }) It("should execute prestop exec hook properly [Conformance]", func() { lifecycle := &v1.Lifecycle{ PreStop: &v1.Handler{ Exec: &v1.ExecAction{ Command: []string{"sh", "-c", "curl http://" + targetIP + ":8080/echo?msg=prestop"}, </ins> }, }, } <del> BeforeEach(func() { By("create the container to handle the HTTPGet hook request.") newPod := podClient.CreateSync(podHandleHookRequest) targetIP = newPod.Status.PodIP }) testPodWithHttpHook := func(podWithHook *v1.Pod) { By("create the pod with lifecycle hook") podClient.CreateSync(podWithHook) if podWithHook.Spec.Containers[0].Lifecycle.PostStart != nil { By("check poststart hook") Eventually(func() error { return podClient.MatchContainerOutput(podHandleHookRequest.Name, podHandleHookRequest.Spec.Containers[0].Name, `GET /echo?msg=poststart`) }, postStartWaitTimeout, podCheckInterval).Should(BeNil()) } By("delete the pod with lifecycle hook") podClient.DeleteSync(podWithHook.Name, metav1.NewDeleteOptions(15), framework.DefaultPodDeletionTimeout) if podWithHook.Spec.Containers[0].Lifecycle.PreStop != nil { By("check prestop hook") Eventually(func() error { return podClient.MatchContainerOutput(podHandleHookRequest.Name, podHandleHookRequest.Spec.Containers[0].Name, `GET /echo?msg=prestop`) }, preStopWaitTimeout, podCheckInterval).Should(BeNil()) } } It("should execute poststart http hook properly [Conformance]", func() { podWithHook := &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name: "pod-with-poststart-http-hook", }, Spec: v1.PodSpec{ Containers: []v1.Container{ { Name:  "pod-with-poststart-http-hook", Image: framework.GetPauseImageNameForHostArch(), Lifecycle: &v1.Lifecycle{ PostStart: &v1.Handler{ HTTPGet: &v1.HTTPGetAction{ Path: "/echo?msg=poststart", Host: targetIP, Port: intstr.FromInt(8080), }, }, }, }, }, }, } testPodWithHttpHook(podWithHook) }) It("should execute prestop http hook properly [Conformance]", func() { podWithHook := &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name: "pod-with-prestop-http-hook", </del> <ins> podWithHook := getPodWithHook("pod-with-prestop-exec-hook", "gcr.io/google_containers/hostexec:1.2", lifecycle) testPodWithHook(podWithHook) }) It("should execute poststart http hook properly [Conformance]", func() { lifecycle := &v1.Lifecycle{ PostStart: &v1.Handler{ HTTPGet: &v1.HTTPGetAction{ Path: "/echo?msg=poststart", Host: targetIP, Port: intstr.FromInt(8080), </ins> }, <del> Spec: v1.PodSpec{ Containers: []v1.Container{ { Name:  "pod-with-prestop-http-hook", Image: framework.GetPauseImageNameForHostArch(), Lifecycle: &v1.Lifecycle{ PreStop: &v1.Handler{ HTTPGet: &v1.HTTPGetAction{ Path: "/echo?msg=prestop", Host: targetIP, Port: intstr.FromInt(8080), }, }, }, }, }, </del> <ins> }, } podWithHook := getPodWithHook("pod-with-poststart-http-hook", framework.GetPauseImageNameForHostArch(), lifecycle) testPodWithHook(podWithHook) }) It("should execute prestop http hook properly [Conformance]", func() { lifecycle := &v1.Lifecycle{ PreStop: &v1.Handler{ HTTPGet: &v1.HTTPGetAction{ Path: "/echo?msg=prestop", Host: targetIP, Port: intstr.FromInt(8080), </ins> }, <del> } testPodWithHttpHook(podWithHook) }) </del> <ins> }, } podWithHook := getPodWithHook("pod-with-prestop-http-hook", framework.GetPauseImageNameForHostArch(), lifecycle) testPodWithHook(podWithHook) </ins> }) }) }) <del> func getExecHookTestPod(name string, cmd []string) *v1.Pod { </del> <ins> func getPodWithHook(name string, image string, lifecycle *v1.Lifecycle) *v1.Pod { </ins> return &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name: name,

The prestop and poststart exec tests volume mount /tmp from the host and attempt to create a file there from inside the pod. This is blocked by selinux. Failure:
we need to have a mechanism to say that a test should not be run if selinux is enabled (if legitimately required) or fix each test (where appropriate)


Question: clusterAddonLabelKey   = "k8s-app" kubeAPIServerLabelName = "kube-apiserver" clusterComponentKey    = "component" <ins> svcReadyTimeout = 1 * time.Minute </ins> ) var (

[x] two weeks soak end date : 18 March 2021 According to this APIsnoop query, there are still some remaining RESOURCENAME endpoints which are untested. with this query you can filter untested endpoints by their category and eligiblity for conformance. e.g below shows a query to find all conformance eligible untested,stable,core endpoints Note: Community feedback for the e2e test has lead to a number of improvements including extending the test coverage to all outstanding endpoints listed above. As each endpoint is tested a ’watch’ confirms that the result is valid before testing the next endpoint. readCoreV1NamespacedServiceStatus (get /status) patchCoreV1NamespacedServiceStatus (patch /status) replaceCoreV1NamespacedServiceStatus (update /status) patchCoreV1NamespacedService Create a Service with a static label Patch the Service with a new label and updated data Get the Service to ensure it’s patched Upate the Service with a new label and updated data Get the Service to ensure it’s updated Delete Namespaced Service via a Collection with a LabelSelector This query by apisnoop shows that all outstanding endpoints as list at the start of this document have been hit by the e2e test. Note that the results do include other service endpoints that have been addressed in other conformance tests. If a test with these calls gets merged, test coverage will go up by 4 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
Can you please review the proposed test and advise if we could go ahead with a PR
/assign
It looks like it's patching metadata and spec but not status?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
I've been retesting this mock test and apisnoop does log as been hit. From my testing below it looks like the patchStatus is working as expected or have I missed something?


Question: clusterAddonLabelKey   = "k8s-app" kubeAPIServerLabelName = "kube-apiserver" clusterComponentKey    = "component" <ins> svcReadyTimeout = 1 * time.Minute </ins> ) var (

Using the following code before the patch request Gives this response Then after running the patch ServiceStatus Then running The response shows the patched changes.
During the Conformance Meeting on 12th of January 2021, reviewed approach with Clayton, and we will start on the test PR this week!
/reopen
Reopened this issue. <detailsIn response to : /reopen </details
/triage accepted


Question: _, result.MissingAggregationRuleSelectors = aggregationRuleCovers(existing.GetAggregationRule(), expected.GetAggregationRule()) switch { <ins> case expected.GetAggregationRule() == nil && existing.GetAggregationRule() != nil: // we didn't expect this to be an aggregated role at all, remove the existing aggregation result.Role.SetAggregationRule(nil) result.Operation = ReconcileUpdate </ins> case !removeExtraPermissions && len(result.MissingAggregationRuleSelectors) > 0: // add missing rules in the union case aggregationRule := result.Role.GetAggregationRule()

/kind bug What happened: I deployed a fresh custom HA kubernetes v1.10.2 cluster with 3 nodes (all controller and worker currently, more workers planned). I then created a namespace called "stummi" and a rolebinding with "user=stummi clusterrole=admin" in this namespace. User stummi (successfully authenticated by TLS client certificate) was not able to list pods or anything else in his namespace: While trying to figure things out, I downgraded to v1.8.11 (a version without cluster role aggregation, that worked) and also tried v1.11 alpha2 (which did not work). I never purged my etcd and I sadly do not have logs from before the different versions. Shortly after posting on slack and before writing this ticket, I dumped the cluster roles in question: Note the rules. What you expected to happen: I expected user stummi to be able to do most things in his namespace. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: I posted this problem on slack and helped me and wanted this issue opened. /sig auth The cluster is deployed with ansible and I can give the playbooks when required. Environment: Kubernetes version (use ):Cloud provider or hardware configuration: 3 virtual machines (2 cores, 8GB memory) OS (e.g. from /etc/os-release): Debian Stretch Kernel (e.g. ): Install tools: custom ansible playbooks Others: Please let me know if you need more information. I'm happy to provide everything you need to fix this :)
/assign


Question: {utiliptables.TableNAT, KubeNodePortChain}, {utiliptables.TableNAT, KubeLoadBalancerChain}, {utiliptables.TableNAT, KubeMarkMasqChain}, <del> {utiliptables.TableNAT, KubeMarkDropChain}, </del> {utiliptables.TableFilter, KubeForwardChain}, } <ins> var iptablesEnsureChains = []struct { table utiliptables.Table chain utiliptables.Chain }{ {utiliptables.TableNAT, KubeMarkDropChain}, } </ins> var iptablesCleanupChains = []struct { table utiliptables.Table chain utiliptables.Chain

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: ci-kubernetes-e2e-gci-gce-ipvs Which test(s) are failing: [sig-network] Services should only allow access from service loadbalancer source ranges [Slow] Testgrid link: https://k8s- Reason for failure: Anything else we need to know: The test is failing consistently, however, there is also test that is failing a lot but not always, may it be related?
/sig-network /area ipvs /cc /help wanted
/sig network
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/assign
/remove-triage unresolved
Hi, , turnout this is because mark drop rule in be deleted, I raied a pr to fix this issue, PTAL. : )
/close as duplicate of
Closing this issue. <detailsIn response to : /close as duplicate of Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen this was affecting ipvs only before , let's keep it open until we verify is not failing anymore
Reopened this issue. <detailsIn response to : /reopen this was affecting ipvs only before , let's keep it open until we verify is not failing anymore Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign Ok I tested this out with kind in ipvs-mode with metallb. In iptables mode I had zero failures but ipvs does not pass at all during the timeout window. This is likely due to


Question: continue } // Write the log line into the stream. <del> if err := writer.write(msg); err != nil { </del> <ins> if err := writer.write(msg, isNewLine); err != nil { </ins> if err == errMaximumWrite { klog.V(2).InfoS("Finished parsing log file, hit bytes limit", "path", path, "limit", opts.bytes) return nil

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Docker has a hardcoded 16K limit on log line length, which splits lines longer than that into multiple messages in the log file. in , however, doesn't account for this when is set to and prints the timestamp in front of every entry in the log file instead of every logical line. This leads to random timestamps appearing in the middle of log lines, potentially breaking the applications that parse these logs. What you expected to happen: inserts timestamps once per logical line, so original log data is preserved. Alternatively, allow API to request the logs in a sane format like JSON, which would allow merging the lines back and combining them with the timestamps more intelligently on the application side. How to reproduce it (as minimally and precisely as possible): Create : apiVersion: v1 kind: Pod metadata: name: log-lines spec: containers: name: main image: busybox command: ['sh', '-c', 'while sleep 1; do echo -n ; for i in $(seq 1 ); do echo -n A; done; echo ; done'] Start the pod with Fetch the logs with , or using the API () Observe random timestamps inserted in the middle of the log lines. Environment: Kubernetes version (use ):
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I will work on it. /sig cli /remove-sig node
The logs are being corrupted by the API endpoint in kubelet, so shouldn't it be in scope of SIG node?
The log is not corrupted by k8s. The same log could be got using "docker container logs -t CONTAINERID" I think you should post your requirement to Docker.


Question: continue } // Write the log line into the stream. <del> if err := writer.write(msg); err != nil { </del> <ins> if err := writer.write(msg, isNewLine); err != nil { </ins> if err == errMaximumWrite { klog.V(2).InfoS("Finished parsing log file, hit bytes limit", "path", path, "limit", opts.bytes) return nil

Please refer following code Line 50 ~ 52 if config.Timestamps { logLine = append([]byte(msg.Timestamp.Format(jsonmessage.RFC3339NanoFixed)+" "), logLine...) }
kubelet doesn't rely on Docker when reading logs, it reads the files itself (see the code linked in the issue). I don't think Docker would change the forced line splitting (see for example), so this should be fixed in Kubernetes.
Your are right and kubelet read log file from host such as /var/log/pods/pods/NAMESPACEPOD-NAMExxx/....(JSON format) I will post a PR for it. /remove-sig cli /sig node
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with .


Question: continue } // Write the log line into the stream. <del> if err := writer.write(msg); err != nil { </del> <ins> if err := writer.write(msg, isNewLine); err != nil { </ins> if err == errMaximumWrite { klog.V(2).InfoS("Finished parsing log file, hit bytes limit", "path", path, "limit", opts.bytes) return nil

Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle rotten
I saw the attempt fixing this but it was scary (echo this comment: ). I suggested a way it may be improved. Also the proposed fix didn't introduce any e2e tests to validate that runtime behavior is not changed and runtimes keep the no-newline-on-splitted-log logic. are you or anybody on this thread are still interested to work on this? Please re-open if so. Meanwhile closing this ahead of bot doing it because the inactivity. /close
Closing this issue. <detailsIn response to : I saw the attempt fixing this but it was scary (echo this comment: ). I suggested a way it may be improved. Also the proposed fix didn't introduce any e2e tests to validate that runtime behavior is not changed and runtimes keep the no-newline-on-splitted-log logic. Instructions for interacting with me using PR comments are available .


Question: continue } // Write the log line into the stream. <del> if err := writer.write(msg); err != nil { </del> <ins> if err := writer.write(msg, isNewLine); err != nil { </ins> if err == errMaximumWrite { klog.V(2).InfoS("Finished parsing log file, hit bytes limit", "path", path, "limit", opts.bytes) return nil

If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
You can't reopen an issue/PR unless you authored it or you are a collaborator. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-lifecycle rotten /reopen will fix it.
Reopened this issue. <detailsIn response to : /remove-lifecycle rotten /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/triage accepted
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.


Question: continue } // Write the log line into the stream. <del> if err := writer.write(msg); err != nil { </del> <ins> if err := writer.write(msg, isNewLine); err != nil { </ins> if err == errMaximumWrite { klog.V(2).InfoS("Finished parsing log file, hit bytes limit", "path", path, "limit", opts.bytes) return nil

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available .


Question: continue } // Write the log line into the stream. <del> if err := writer.write(msg); err != nil { </del> <ins> if err := writer.write(msg, isNewLine); err != nil { </ins> if err == errMaximumWrite { klog.V(2).InfoS("Finished parsing log file, hit bytes limit", "path", path, "limit", opts.bytes) return nil

If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I received a bug report about this issue. The root cause is that the buffer reads up to a new line. If the line is greater than 4096 bytes and '--timestamps' is turrned on the kubelet will write the timestamp and the partial log line. We will need to refactor the ReadLogs function to allow for a partial line read.


Question: recorder record.EventRecorder } <del> func (irecorder *innerEventRecorder) shouldRecordEvent(object runtime.Object) (*clientv1.ObjectReference, bool) { </del> <ins> func (irecorder *innerEventRecorder) shouldRecordEvent(object runtime.Object) (*v1.ObjectReference, bool) { </ins> if object == nil { return nil, false } <del> if ref, ok := object.(*clientv1.ObjectReference); ok { if !strings.HasPrefix(ref.FieldPath, ImplicitContainerPrefix) { return ref, true } } // just in case we miss a spot, be sure that we still log something </del> if ref, ok := object.(*v1.ObjectReference); ok { if !strings.HasPrefix(ref.FieldPath, ImplicitContainerPrefix) { <del> return events.ToObjectReference(ref), true </del> <ins> return ref, true </ins> } } return nil, false

Is this a BUG REPORT or FEATURE REQUEST?: What happened: Now the and has been consistent in by PR . There is no need to convert v1.ObjectReference to clientv1.ObjectReference in . What you expected to happen: Remove this function and cleanup the codes which call this function. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?:
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <bre.g., for API Machinery<br(2) specifying the label manually: <bre.g., for sig/scalability<br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
Will submit the cleanup after PR merged.
/sig api-machinery
/sub


Question: Rules:      kubeSchedulerRules, }) <del> externalProvisionerRules := []rbacv1.PolicyRule{ rbacv1helpers.NewRule("create", "delete", "get", "list", "watch").Groups(legacyGroup).Resources("persistentvolumes").RuleOrDie(), rbacv1helpers.NewRule("get", "list", "watch", "update", "patch").Groups(legacyGroup).Resources("persistentvolumeclaims").RuleOrDie(), rbacv1helpers.NewRule("list", "watch").Groups(storageGroup).Resources("storageclasses").RuleOrDie(), rbacv1helpers.NewRule("get", "list", "watch", "create", "update", "patch").Groups(legacyGroup).Resources("events").RuleOrDie(), rbacv1helpers.NewRule("get", "list", "watch").Groups(legacyGroup).Resources("nodes").RuleOrDie(), } if utilfeature.DefaultFeatureGate.Enabled(features.CSINodeInfo) { externalProvisionerRules = append(externalProvisionerRules, rbacv1helpers.NewRule("get", "watch", "list").Groups("storage.k8s.io").Resources("csinodes").RuleOrDie()) } roles = append(roles, rbacv1.ClusterRole{ // a role for the csi external provisioner ObjectMeta: metav1.ObjectMeta{Name: "system:csi-external-provisioner"}, Rules:      externalProvisionerRules, }) </del> addClusterRoleLabel(roles) return roles }

<!-- Please only use this template for submitting enhancement requests --The built-in and cluster roles here are deprecated in 1.13 when this issue was resolved: /sig storage /milestone v1.17 /priorit important-longterm
You must be a member of the GitHub team to set the milestone. If you believe you should be able to issue the /milestone command, please contact your and have them propose you as an additional delegate for this responsibility. <detailsIn response to : <!-- Please only use this template for submitting enhancement requests -- /milestone v1.17 /priorit important-longterm Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc /priority important-longterm
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale /assign
Bug triage for 1.17 here with a gentle reminder that code freeze for this release is on November 14. Is this issue still intended for 1.17?


Question: func restartKubelet() { kubeletServiceName := findRunningKubletServiceName() <del> stdout, err := exec.Command("sudo", "systemctl", "restart", kubeletServiceName).CombinedOutput() </del> <ins> // reset the kubelet service start-limit-hit stdout, err := exec.Command("sudo", "systemctl", "reset-failed", kubeletServiceName).CombinedOutput() framework.ExpectNoError(err, "Failed to reset kubelet start-limit-hit with systemctl: %v, %v", err, stdout) stdout, err = exec.Command("sudo", "systemctl", "restart", kubeletServiceName).CombinedOutput() </ins> framework.ExpectNoError(err, "Failed to restart kubelet with systemctl: %v, %v", err, stdout) }

Which jobs are flaking: ci-kubernetes-node-kubelet-serial-hugepages Which test(s) are flaking: all Testgrid link: https://k8s- Reason for failure:
/sig node
/assign /cc
GitHub didn't allow me to assign the following users: cynepco3hahue. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign /cc Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
testgrid alert:
/cc
I started to look into logs and the issue that one of the kubelet restarts fails on: I will try to reproduce the issue locally.
any news? Have you managed to reproduce the issue locally?
I already have a fix that is waiting for merge -


Question: // We could try a wget the service from the client pod.  But services.sh e2e test covers that pretty well. }) <ins> It("should be restarted with a docker exec "cat /tmp/health" liveness probe", func() { runLivenessTest(c, &api.Pod{ ObjectMeta: api.ObjectMeta{ Name:   "liveness-exec", Labels: map[string]string{"test": "liveness"}, }, Spec: api.PodSpec{ Containers: []api.Container{ { Name:    "liveness", Image:   "busybox", Command: []string{"/bin/sh", "-c", "echo ok >/tmp/health; sleep 10; echo fail >/tmp/health; sleep 600"}, LivenessProbe: &api.Probe{ Handler: api.Handler{ Exec: &api.ExecAction{ Command: []string{"cat", "/tmp/health"}, }, }, InitialDelaySeconds: 15, }, }, }, }, }) }) It("should be restarted with a /healthz http liveness probe", func() { runLivenessTest(c, &api.Pod{ ObjectMeta: api.ObjectMeta{ Name:   "liveness-http", Labels: map[string]string{"test": "liveness"}, }, Spec: api.PodSpec{ Containers: []api.Container{ { Name:    "liveness", Image:   "kubernetes/liveness", Command: []string{"/server"}, LivenessProbe: &api.Probe{ Handler: api.Handler{ HTTPGet: &api.HTTPGetAction{ Path: "/healthz", Port: util.NewIntOrStringFromInt(8080), }, }, InitialDelaySeconds: 15, }, }, }, }, }) }) </ins> })

Park of ginkgo-ization
Per IRC, is working on this
Pushed an update yesterday to address comments.
what's the status of this? There's an issue open to de-flake this test (), but it's tough to do that if the code is in the middle of being moved to ginkgo.
This has been merged.


Question: docs/yaml/kubectl/kubectl.yaml docs/yaml/kubectl/kubectl_alpha.yaml docs/yaml/kubectl/kubectl_annotate.yaml <ins> docs/yaml/kubectl/kubectl_api-resources.yaml </ins> docs/yaml/kubectl/kubectl_api-versions.yaml docs/yaml/kubectl/kubectl_apply.yaml docs/yaml/kubectl/kubectl_attach.yaml

As the RBAC role need to be related to resources. I think we could use a command to get the supported resources. ref to
This would be cool.
I thought we a command that listed resources in the discover doc a long time ago.
is just the versions, but principle is the same. We want to list all the types, their resources, and tie that back to "explain".
What's the best way to determine this information currently? I'm having difficulty figuring out the exact set of permissions (and how they would be written in an RBAC role) for any given kubectl command. The documentation for this right now is split across multiple places and certain things are not explained at all. Specifically, it's difficult to determine which "API verb" is used for which operations. Take as an example. It's unclear if the way to allow this is to use the "proxy" subresource of a pod or service with "get"/"create"/etc. verbs, or to use the "proxy" verb with the resource (pod, service) directly. is a similar case. The almost works for this purpose, but it's still difficult. Operations for a given resource are grouped into categories like write, read, status, and proxy, but these things don't obviously correlate with API verbs. Again, proxy is especially confusing as there is no mention of a proxy "verb". The API operations are only shown using their HTTP verbs. One thing that could help on this document would be a section on the right column under "kubectl command" that shows exactly the rule needed in an RBAC role to allow this command via kubectl. And even then, that assumes an exact 1-to-1 mapping between API endpoints and kubectl commands, which I doubt is always the case. (For example, port fowarding to a pod seems to require both a GET on the pod resource and a CREATE on a port forward subresource.) There also needs to be better documentation about how this information could have been determined in the first place. I've resorted to manually looking through API requests made with but even then it's difficult to figure out. Amusingly, I was the one who wrote which is referenced in the original issue description (as well as the table explaining verbs in ).


Question: docs/yaml/kubectl/kubectl.yaml docs/yaml/kubectl/kubectl_alpha.yaml docs/yaml/kubectl/kubectl_annotate.yaml <ins> docs/yaml/kubectl/kubectl_api-resources.yaml </ins> docs/yaml/kubectl/kubectl_api-versions.yaml docs/yaml/kubectl/kubectl_apply.yaml docs/yaml/kubectl/kubectl_attach.yaml

tl;dr: It's too difficult right now to go from "I want a user to be able to run " to "here is the RBAC rule to use for that" for any case where there isn't a specific example shown in the docs.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
It seem that relative PR stop updated. If this issue still valuable, i will help fix it
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close


Question: return nil } <del> func (lb *LoadBalancer) DeleteTCPLoadBalancer(name, region string) error { glog.V(4).Infof("DeleteTCPLoadBalancer(%v, %v)", name, region) vip, err := getVipByName(lb.network, name) if err != nil { return err } pool, err := pools.Get(lb.network, vip.PoolID).Extract() if err != nil { return err </del> <ins> func (lb *LoadBalancer) EnsureTCPLoadBalancerDeleted(name, region string) error { glog.V(4).Infof("EnsureTCPLoadBalancerDeleted(%v, %v)", name, region) // TODO(#8352): Because we look up the pool using the VIP object, if the VIP // is already gone we can't attempt to delete the pool. We should instead // continue even if the VIP doesn't exist and attempt to delete the pool by // name. vip, vipErr := getVipByName(lb.network, name) if vipErr == ErrNotFound { return nil } else if vipErr != nil { return vipErr } // It's ok if the pool doesn't exist, as we may still need to delete the vip // (although I don't believe the system should ever be in that state). pool, poolErr := pools.Get(lb.network, vip.PoolID).Extract() if poolErr != nil { detailedErr, ok := poolErr.(*gophercloud.UnexpectedResponseCodeError) if !ok || detailedErr.Actual != http.StatusNotFound { return poolErr } </ins> } <ins> poolExists := (poolErr == nil) </ins> <del> // Have to delete VIP before pool can be deleted err = vips.Delete(lb.network, vip.ID).ExtractErr() if err != nil { </del> <ins> // We have to delete the VIP before the pool can be deleted, so we can't // continue on if this fails. // TODO(#8352): Only do this if the VIP exists once we can delete pools by // name rather than by ID. err := vips.Delete(lb.network, vip.ID).ExtractErr() if err != nil && err != ErrNotFound { </ins> return err } // Ignore errors for everything following here <del> for _, monId := range pool.MonitorIDs { pools.DisassociateMonitor(lb.network, pool.ID, monId) </del> <ins> if poolExists { for _, monId := range pool.MonitorIDs { // TODO(#8352): Delete the monitor, don't just disassociate it. pools.DisassociateMonitor(lb.network, pool.ID, monId) } pools.Delete(lb.network, pool.ID) </ins> } <del> pools.Delete(lb.network, pool.ID) </del> return nil }

All of our external load balancer tests failed for the better part of a day due to the GCE project that our tests run in hitting its quota for number of target pools. While the target pools have been deleted, the fact that we hit our quota indicates that there must be a leak somewhere, and we need to plug it.
For reference, the previous issue was
Could the leakage be related to ? The quota exhaustion occurred about 22 hrs after that.
I'd guess that this is due to GetTCPLoadBalancer returning that the load balancer doesn't exist if the forwarding rule is gone, ignoring the target pool, while the service controller considers its deleting done if the cloud provider says that the load balancer doesn't exist. I that logic to the cloud provider because the other implementations of DeleteTCPLoadBalancer weren't/won't be idempotent in the sense of returning ok if the LB doesn't exist. To fix it, we'll either need to make sure all Delete implementations are idempotent and remove the Get call in the controller, or add a method along the lines of IsTCPLoadBalancerDeleted to ask the cloud provider whether all pieces of it have been completely cleaned up. We can't make GetTCPLoadBalancer check that because it's relied on to tell us whether or not the balancer is fully created.
Copying over my commentary from on why the GKE e2e tests are leaking so many forwarding rules and target pools vs the GCE tests:
Is there a way for the services test to wait for the service controller to clean up the external resources before finishing (without just doing a sleep)?
Not without polling GCE directly for whether the resources still exist, although that may be worth it as a workaround. What would be perfect is that clean-up-all-cloud-provider-resources API method that we talked about a couple months ago.
Can we not perhaps use GCE namespaces or projects as a cheap way of getting this here?
I don't believe so. We can't use projects due to the fact that projects can't be programatically created (they're intended to be long-lived). I'm not aware of anything along the lines of a GCE namespace other than networks, which won't work because target pools and forwarding rules unfortunately don't seem to be scoped to networks.


Question: return nil } <del> func (lb *LoadBalancer) DeleteTCPLoadBalancer(name, region string) error { glog.V(4).Infof("DeleteTCPLoadBalancer(%v, %v)", name, region) vip, err := getVipByName(lb.network, name) if err != nil { return err } pool, err := pools.Get(lb.network, vip.PoolID).Extract() if err != nil { return err </del> <ins> func (lb *LoadBalancer) EnsureTCPLoadBalancerDeleted(name, region string) error { glog.V(4).Infof("EnsureTCPLoadBalancerDeleted(%v, %v)", name, region) // TODO(#8352): Because we look up the pool using the VIP object, if the VIP // is already gone we can't attempt to delete the pool. We should instead // continue even if the VIP doesn't exist and attempt to delete the pool by // name. vip, vipErr := getVipByName(lb.network, name) if vipErr == ErrNotFound { return nil } else if vipErr != nil { return vipErr } // It's ok if the pool doesn't exist, as we may still need to delete the vip // (although I don't believe the system should ever be in that state). pool, poolErr := pools.Get(lb.network, vip.PoolID).Extract() if poolErr != nil { detailedErr, ok := poolErr.(*gophercloud.UnexpectedResponseCodeError) if !ok || detailedErr.Actual != http.StatusNotFound { return poolErr } </ins> } <ins> poolExists := (poolErr == nil) </ins> <del> // Have to delete VIP before pool can be deleted err = vips.Delete(lb.network, vip.ID).ExtractErr() if err != nil { </del> <ins> // We have to delete the VIP before the pool can be deleted, so we can't // continue on if this fails. // TODO(#8352): Only do this if the VIP exists once we can delete pools by // name rather than by ID. err := vips.Delete(lb.network, vip.ID).ExtractErr() if err != nil && err != ErrNotFound { </ins> return err } // Ignore errors for everything following here <del> for _, monId := range pool.MonitorIDs { pools.DisassociateMonitor(lb.network, pool.ID, monId) </del> <ins> if poolExists { for _, monId := range pool.MonitorIDs { // TODO(#8352): Delete the monitor, don't just disassociate it. pools.DisassociateMonitor(lb.network, pool.ID, monId) } pools.Delete(lb.network, pool.ID) </ins> } <del> pools.Delete(lb.network, pool.ID) </del> return nil }

The target pool leak in the code might be fixed, but the forwarding rule + target pool leak from deleting GKE clusters too quickly is certainly still happening. Reopening for the sake of that half of the issue.
Can we just add a short sleep to jenkins after the last test has run before it deletes anything?
Yes, you could add a sleep to Ginkgo's AfterEachSuite(): But I'd be concerned about what the upper bound on the sleep period is. Can we get the tests and code in sync with a 1 minute latency here? (e.g. if the controller garbage collects every 60 seconds, then the test only needs to sleep for 60 sec?) And add a BEEG comment to the test's sleep config noting that increasing the sleep time will not improve things, otherwise I'm pretty sure that someone noting flakes in this test in future will think "maybe I should just increase the sleep period a little bit."
Actually, thinking about this a little more, it would be better to put the "sleep" logic in the k8s master teardown procedure (i.e. not in the e2e tests or in Jenkins), because otherwise users are going to run into the same resource leakage problems as we are. So master deletion blocks on cleanup of backend cloud resources.
There's no way to block master deletion from the master itself. The test framework (or user) just pulls the rug out from underneath the master. See
Robby's right -- is the real solution to this, but until we have that the best we can do is to have the wait happen elsewhere. The options seem to be: GKE itself Ginkgo's post-testing logic Jenkins' teardown logic I don't like putting it in GKE, because arbitrarily slowing down our API makes for a poor experience for the majority of people that don't still have resources being deleted. I'm not eager to put it in Gingko either, since won't that sleep also take effect whenever anyone runs an e2e test locally against a dev cluster? Putting it in the Jenkins logic makes the most sense to me. I'll send a PR and cc you.
We may also want to change the kube-down logic to bring down the master and then nodes in parallel to both speed it up and more closely resemble GKE's teardown logic.
I've deleted the old forwarding rules and target pools from the project.


Question: return nil } <del> func (lb *LoadBalancer) DeleteTCPLoadBalancer(name, region string) error { glog.V(4).Infof("DeleteTCPLoadBalancer(%v, %v)", name, region) vip, err := getVipByName(lb.network, name) if err != nil { return err } pool, err := pools.Get(lb.network, vip.PoolID).Extract() if err != nil { return err </del> <ins> func (lb *LoadBalancer) EnsureTCPLoadBalancerDeleted(name, region string) error { glog.V(4).Infof("EnsureTCPLoadBalancerDeleted(%v, %v)", name, region) // TODO(#8352): Because we look up the pool using the VIP object, if the VIP // is already gone we can't attempt to delete the pool. We should instead // continue even if the VIP doesn't exist and attempt to delete the pool by // name. vip, vipErr := getVipByName(lb.network, name) if vipErr == ErrNotFound { return nil } else if vipErr != nil { return vipErr } // It's ok if the pool doesn't exist, as we may still need to delete the vip // (although I don't believe the system should ever be in that state). pool, poolErr := pools.Get(lb.network, vip.PoolID).Extract() if poolErr != nil { detailedErr, ok := poolErr.(*gophercloud.UnexpectedResponseCodeError) if !ok || detailedErr.Actual != http.StatusNotFound { return poolErr } </ins> } <ins> poolExists := (poolErr == nil) </ins> <del> // Have to delete VIP before pool can be deleted err = vips.Delete(lb.network, vip.ID).ExtractErr() if err != nil { </del> <ins> // We have to delete the VIP before the pool can be deleted, so we can't // continue on if this fails. // TODO(#8352): Only do this if the VIP exists once we can delete pools by // name rather than by ID. err := vips.Delete(lb.network, vip.ID).ExtractErr() if err != nil && err != ErrNotFound { </ins> return err } // Ignore errors for everything following here <del> for _, monId := range pool.MonitorIDs { pools.DisassociateMonitor(lb.network, pool.ID, monId) </del> <ins> if poolExists { for _, monId := range pool.MonitorIDs { // TODO(#8352): Delete the monitor, don't just disassociate it. pools.DisassociateMonitor(lb.network, pool.ID, monId) } pools.Delete(lb.network, pool.ID) </ins> } <del> pools.Delete(lb.network, pool.ID) </del> return nil }

I'll periodically check to see whether anything is leaked after this point. I apologize if the deletions affected any running tests - everything looked to only be from GKE and parallel e2e runs.


Question: cgroupRoots = append(cgroupRoots, cm.NodeAllocatableRoot(s.CgroupRoot, s.CgroupDriver)) kubeletCgroup, err := cm.GetKubeletContainer(s.KubeletCgroups) if err != nil { <del> return fmt.Errorf("failed to get the kubelet's cgroup: %v", err) } if kubeletCgroup != "" { </del> <ins> klog.Warningf("failed to get the kubelet's cgroup: %v.  Kubelet system container metrics may be missing.", err) } else if kubeletCgroup != "" { </ins> cgroupRoots = append(cgroupRoots, kubeletCgroup) } runtimeCgroup, err := cm.GetRuntimeContainer(s.ContainerRuntime, s.RuntimeCgroups) if err != nil { <del> return fmt.Errorf("failed to get the container runtime's cgroup: %v", err) } if runtimeCgroup != "" { </del> <ins> klog.Warningf("failed to get the container runtime's cgroup: %v. Runtime system container metrics may be missing.", err) } else if runtimeCgroup != "" { </ins> // RuntimeCgroups is optional, so ignore if it isn't specified cgroupRoots = append(cgroupRoots, runtimeCgroup) }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: kubelet terminated unexpectedly during hack/local-up-cluster kubelet log as a result apiserver cannot find the node to schedule pods What you expected to happen: kubelet runs successfully and hack/local-up-cluster creates a single node cluster How to reproduce it (as minimally and precisely as possible): run hack/local-up-cluster from HEAD Anything else we need to know?: additional cgroups check seemed to be introduced in ; hack/local-up-cluster works when I check out the parent commit have CI for hack/local-up-cluster and , so I'm not sure if it's an error on my side. It would be greatly appreciated if someone could explain it to me Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Debian GNU/Linux Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /sig node cc
Thanks for raising such a detailed issue! cc any thoughts here? Confusingly, I think the kubelet has been assigned a different cgroup for memory and for cpu... however the fact that its using makes me think the user is relying on system defaults. out of curiosity, have you made any mods to how your system assigns cgroups?
It sounds like we might want to handle the error more gracefully. If we can't get the kubelet's cgroup, we should fall back to collecting metrics from all cgroups.
collecting metrics from all the cgroups on the system? Any concerns about how many cgroups there could be? Also, or any interest in implementing this fix (assuming we go with the fallback behavior)? If not, I can try.
thanks for looking into this! Sorry I'm not familiar with cgroups. I'm don't know any modification to my system assigns cgroups in particular
I can put out a PR this week
xref
Thanks!


Question: // CreateAdapter creates Custom Metrics - Stackdriver adapter // adapterDeploymentFile should be a filename for adapter deployment located in StagingDeploymentLocation <del> func CreateAdapter(namespace, adapterDeploymentFile string) error { </del> <ins> func CreateAdapter(adapterDeploymentFile string) error { </ins> // A workaround to make the work on GKE. GKE doesn't normally allow to create cluster roles, // which the adapter deployment does. The solution is to create cluster role binding for // cluster-admin role and currently used service account. <del> err := createClusterAdminBinding(namespace) </del> <ins> err := createClusterAdminBinding() </ins> if err != nil { return err }

/kind failing-test <!-- If this is a flake, please add: /kind flake --/sig autoscaling Looks like it just started happening today (@ 2020-09-15 03:54:19 +0000 UTC) !
cc the namespace from the provided object "kube-system" does not match the namespace "horizontal-pod-autoscaling-830". You must pass '--namespace=kube-system' to perform this operation. the namespace from the provided object "custom-metrics" does not match the namespace "horizontal-pod-autoscaling-830". You must pass '--namespace=custom-metrics' to perform this operation. the namespace from the provided object "custom-metrics" does not match the namespace "horizontal-pod-autoscaling-830". You must pass '--namespace=custom-metrics' to perform this operation.
related fix:
No I don't think that's the fix for this (that is just a little cleanup). Actually I think we have to delete namespace parameter in line : Because before , the will have no actually effect. But now with the change of , it will have namespace : at the end of the command. But the namespace in yaml file is which is different from . We should not have parameter when applying from file because this test didn't have before, right? Related PR:
/assign


Question: } t.Run(fmt.Sprintf("feature enabled=%v, old pod %v, new pod %v", enabled, oldPodInfo.description, newPodInfo.description), func(t *testing.T) { <del> defer featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, features.VolumeSubpathEnvExpansion, enabled)() </del> var oldPodSpec *api.PodSpec if oldPod != nil {

/kind cleanup /sig storage /sig node Description: Feature gate VolumeSubpathEnvExpansion was GA in 1.17 and is marked for removal in 1.19 Related Issues: GA Release:
/assign
/cc


Question: defer featuregatetesting.SetFeatureGateDuringTest(b, utilfeature.DefaultFeatureGate, feature, flag)() } dataItems.DataItems = append(dataItems.DataItems, runWorkload(b, tc, w)...) <ins> // Reset metrics to prevent metrics generated in current workload gets // carried over to the next workload. legacyregistry.Reset() </ins> }) } })

In the new scheduler performance testing framework, (bool) is provided to benchmark performance of pods created in an "operator": (L11 below) However, the initialized Pods created in L5~L7 seem to be also benchmarked, although it doesn't specify . (please see the reproduce steps below). I think it's b/c once scheduler starts, the associated Prometheus artifacts are also up, and it's a single in-memory datastore. So no matter we start to collect the metric in which step, the prometheus data is already up-to-date. In the above workload config, the final data have benchmarked 1500 pods instead of 1000 pods. I guess it won't work either if running in the pod initialization step or not. Unless we manage a breakpoint at the end of each step, and calculate delta histogram for the step that specifies . Pods created in step without are not benchmarked. Apply the following diff: And then run: You will get an output sth. like: /sig scheduling
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Could you please help confirm if it's an issue?
cc
Unfortunately, does not apply for metrics gathered from . As you mentioned. The default gatherer collects all metrics produced by the scheduler. It's the scheduler itself who knows pod specs and would have to decide which pods will get ignored when updating metrics. The question is if computing metrics for a subset of pods processed by the scheduler is a valid use case.
This is probably what we want. Is something like that possible ?
I think being able to absolutely ignore certain pods for metrics is an overkill.
The use case is to ignore the pods created in steps that don't specify , so as to collect the Pod metrics create at particular step(s) (). Otherwise, the semantics of option is quite misleading. Nope, that's not the intention. We don't need to target certain pods, like using label selector or whatever. Just as I mentioned, measure pods created at particular steps.


Question: defer featuregatetesting.SetFeatureGateDuringTest(b, utilfeature.DefaultFeatureGate, feature, flag)() } dataItems.DataItems = append(dataItems.DataItems, runWorkload(b, tc, w)...) <ins> // Reset metrics to prevent metrics generated in current workload gets // carried over to the next workload. legacyregistry.Reset() </ins> }) } })

I would do something like the following: if a step has collectMetrics:false, we wait for them to finish scheduling, and then we take a snapshot of the histogram. Then for the next step that has collectMetrics:true, we take the diff with the latest snapshot to report the results. This might not be perfect, as we don't control if the metrics have finished syncing when we take the snapshot. But I think that's fine. Would doing this be possible? Now, if we have steps that run in parallel, one with collectMetrics:false and another with collectMetrics:true, then this doesn't work. Note that I don't know the details of how the gatherer works.
That adheres to what I have in my mind - we can only manage this on client side. I think it's fine. As we are already fetching the "snapshot" when pods get scheduled when equals true. True. An alternative is to design the case to be like: the step with to employ Pods that are all unschedulable, and then specify a custom metric "schedulere2eschedulingdurationseconds[result="unschedulable"]" to target a particular histogram instead of all histograms (default option) inside a histogramVec. Note this can only be possible when and some subsequent changes get in place.
We might just call Histogram.Clear() method which will clear what has been collected so far at the end of phase: Assuming individual phases have no pods in common and each phase can clear metrics without affecting other phases. Though, as said, it will work only for serial case.
WDYT
I'm afraid doesn't work. That was my initial thought as well, but no luck. I guess it's b/c, theoretically, we cannot reset a metric on the server side - Promoetheus' model doesn't support this. But let me know if I miss anything.
The performance tests assume all the metrics all gathered and accessed in the same binary. As long as that's the case, works.


Question: defer featuregatetesting.SetFeatureGateDuringTest(b, utilfeature.DefaultFeatureGate, feature, flag)() } dataItems.DataItems = append(dataItems.DataItems, runWorkload(b, tc, w)...) <ins> // Reset metrics to prevent metrics generated in current workload gets // carried over to the next workload. legacyregistry.Reset() </ins> }) } })

We already call in the code:
That was my understanding as well, but no luck :( Here is a diff to call in case when "concreteOp.CollectMetrics" is false: <details<summaryDiff:</summary<p</p</detailsAnd below is the output - it's still 1500 (500 init pods plus 1000 measured pods) instead of 1000 being benchmarked: <details<summaryOutput:</summary<p</p</details
Also, if you run the tests and together, the result proves doesn't work as expected:
But I do find a way to fix this: .


Question: "k8s.io/apimachinery/pkg/util/wait" utilfeature "k8s.io/apiserver/pkg/util/feature" "k8s.io/client-go/dynamic" <ins> "k8s.io/client-go/informers" </ins> clientset "k8s.io/client-go/kubernetes" <ins> restclient "k8s.io/client-go/rest" </ins> featuregatetesting "k8s.io/component-base/featuregate/testing" apiservertesting "k8s.io/kubernetes/cmd/kube-apiserver/app/testing" podutil "k8s.io/kubernetes/pkg/api/v1/pod" <ins> "k8s.io/kubernetes/pkg/controller/statefulset" </ins> "k8s.io/kubernetes/pkg/features" "k8s.io/kubernetes/test/integration/framework" )

I create a ResourceQuota with two pods limit in my namespace. Then, I create a statefuleSet with 3 replicas. As expected, two pods are created successfully, and the third pod failed to create. But, the ReadyReplicas is not 2 ,but 1. The ReadyReplicas is correct 1Create a ResourceQuota with two pods limit in a new namespace. 2Create a statefulSet with 3 replicas 3Check if the ReadyReplicas of statefulSet matchs the number of stateful running pods. 1According to the logic of the statefulSet controller, new pod created in current sync is not be immediately counted into the ReadyReplicas. 2Failure to create pod will cause to return error, then skip updating status. <details</details<details</details<detailskind v1.23.3 </details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig apps
/assign


Question: type NodeClient interface { // NodePrepareResources prepares several ResourceClaims // for use on the node. If an error is returned, the <del> // response is ignored. Failures for individidual claims </del> <ins> // response is ignored. Failures for individual claims </ins> // can be reported inside NodePrepareResourcesResponse. NodePrepareResources(ctx context.Context, in *NodePrepareResourcesRequest, opts ...grpc.CallOption) (*NodePrepareResourcesResponse, error) // NodeUnprepareResources is the opposite of NodePrepareResources.

When ResourceClaimParameters or ResourceClassParameters fail validation in resource driver during PodSchedulingContext sync loop, the error is visible only in the events of the PodSchedulingContext. The error should be visible on the object that fails validation as well, ResourceClaim or ResourceClass respectively. Creating ResourceClaim with unsupported parameters' values or unsupported No response <details</details<details</details<details</details<details</details<details</details<details</details
/assign
/sig node
/triage accepted
KEP:


Question: priorityPairs = append(priorityPairs, priorityPair{name: priorityName, value: priorityVal}) _, err := cs.SchedulingV1().PriorityClasses().Create(context.TODO(), &schedulingv1.PriorityClass{ObjectMeta: metav1.ObjectMeta{Name: priorityName}, Value: priorityVal}, metav1.CreateOptions{}) if err != nil { <del> framework.Logf("Failed to create priority '%v/%v': %v", priorityName, priorityVal, err) framework.Logf("Reason: %v. Msg: %v", apierrors.ReasonForError(err), err) </del> <ins> framework.Logf("Failed to create priority '%v/%v'. Reason: %v. Msg: %v", priorityName, priorityVal, apierrors.ReasonForError(err), err) </ins> } framework.ExpectEqual(err == nil || apierrors.IsAlreadyExists(err), true) }

The following PriorityClass endpoints was Promoted it 1.14 without Conformance tests, and remain untested: replace-, read-, patch-, list-, delete-collection- SchedulingV1PriorityClass According to this APIsnoop query, there are still some remaining PriorityClass endpoints which are untested. with this query you can filter untested endpoints by their category and eligiblity for conformance. e.g below shows a query to find all conformance eligible untested,stable,core endpoints ! createSchedulingV1PriorityClass deleteSchedulingV1PriorityClass Promoted to Conformance 1.19 with /sig testing /sig architecture /area conformance
Is the ultimiate goal to use conformance tests to explicitly issue all CRUD requests against PriorityClass? And, is equivalent to "update PriorityClass"?
all APIs must have Conformance tests unless they are not eligible for Conformance due to the There is other APIGroups that have both Replace and Update as endpoints. That is the prerogative of the SIG to decide.
can you please point us to similar conformance tests used for core objects such as pods and nodes?
This is a life cycle test for Create, List, Update, Patch, DeleteCollection More can be found in the test/e2e folder. Also look for examples on the Conformance at previously Promoted Conformance test Reach out if we can help with more information on our approach to Conformance testing.
I remembered about this PR. Last week a few lines of code was to cover the last of the "Lease" endpoint group for conformance. The one file contain the Conformance tests for the complete life cycle of Lease endpoints.
/assign I will work on it.
/reopen Will monitor in test grid to move to Conformance after 2 weeks soak.
Reopened this issue. <detailsIn response to : /reopen Will monitor in test grid to move to Conformance after 2 weeks soak. </details
/triage accepted


Question: }) }) <del> // TODO: Get rid of [DisabledForLargeClusters] tag when issue #56138 is fixed. var _ = SIGDescribe("ESIPP [Slow] [DisabledForLargeClusters]", func() { </del> <ins> var _ = SIGDescribe("ESIPP [Slow]", func() { </ins> f := framework.NewDefaultFramework("esipp") var loadBalancerCreateTimeout time.Duration

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug This is a followup from filed separately now that we're not tracking this as part of 1.9 release. The tests have been disabled with [DisabledForLargeClusters] tag for now, and this fact will be mentioned in the scalability validation report for v1.9. This issue tracks fixing the test so the tag can be removed and the test ran again in the large cluster. /assign /kind e2e-test-failure
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Is this e2e test still broken?
Sorry for the confusion here. Currently this e2e test is disabled on extra large cluster due to some scalability issues with the load balancer related components. At the moment the issues haven't been completely resolved yet.
ok, adding priority so that I know it's not a 1.13 blocker. /priority important-longterm
cc - this is why the relevant tests for are not part of the e2e suite.
This issue is still valid --- some action items on top of my head: Figure out a plan to make service controller properly handle operations in parallel. Check on the status for cloud providers (especially GCP, which the test ran against) on handling large number of nodes as LB backends. /remove-triage unresolved
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or .


Question: }) }) <del> // TODO: Get rid of [DisabledForLargeClusters] tag when issue #56138 is fixed. var _ = SIGDescribe("ESIPP [Slow] [DisabledForLargeClusters]", func() { </del> <ins> var _ = SIGDescribe("ESIPP [Slow]", func() { </ins> f := framework.NewDefaultFramework("esipp") var loadBalancerCreateTimeout time.Duration

/lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten /lifecycle frozen
/reopen Reopening as is being reverted in
Reopened this issue. <detailsIn response to : /reopen Reopening as is being reverted in Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
these tests are flaking in xref
cc since these were enabled by
Yeah - though these failures doesn't seem to be scale related, they seem to be test related to me. This basically means nodes without public ips: It's pretty much the same thing as (just for a different test). should be addressing these kinds of errors. [Not a release blocker for sure]
BTW - has been opened directly for this one.
... so since is closed, Can we close this and re-enable for large clusters ? (i.e. remove the Tag in that disables this test) ? I'm not up to speed on the history but 2 related notes I came across digging into related test today... 's a couple of TODOs in the code referencing this specific issue, but your above direct issue is now closed, so we can close this as well right?... And if so, we can remove the TODO on the test... -enabling the test ... not sure wether the timing is good right now (given that is open right now , which sparked my stumbling upon this issue :))
We tried to enable all of them and that caused us a bunch of issues and those had to be reverted: A couple tests were enabled later in: and those seem fine. The remaining (mostly session-affinity ones) had some issue, which so far we didn't have time to debug and I haven't heard a good explanation why those were failing. It's possible that they were improved with , but I doubt those would really cause the failures we've seen in the past. So until this is debugged (or at least understood) I'm reluctant to enabling them.


Question: proxy_cpu=${KUBEMARK_HOLLOW_PROXY_MILLICPU:-$proxy_cpu} proxy_mem_per_node=${KUBEMARK_HOLLOW_PROXY_MEM_PER_NODE_KB:-50} proxy_mem=$((100 * 1024 + proxy_mem_per_node*NUM_NODES)) <ins> hollow_kubelet_params=$(eval "for param in ${HOLLOW_KUBELET_TEST_ARGS:-}; do echo -n "$param",; done") hollow_kubelet_params=${hollow_kubelet_params%?} hollow_proxy_params=$(eval "for param in ${HOLLOW_PROXY_TEST_ARGS:-}; do echo -n "$param",; done") hollow_proxy_params=${hollow_proxy_params%?} </ins> sed -i'' -e "s@{{hollow_kubelet_millicpu}}@${KUBEMARK_HOLLOW_KUBELET_MILLICPU:-40}@g" "${RESOURCE_DIRECTORY}/hollow-node.yaml" sed -i'' -e "s@{{hollow_kubelet_mem_Ki}}@${KUBEMARK_HOLLOW_KUBELET_MEM_KB:-$((100*1024))}@g" "${RESOURCE_DIRECTORY}/hollow-node.yaml"

There are CVEs happening in different images - we already migrated Kubmeark from busybox in , but we should go one step further and migrate the base image to distroless. There are some things that would need to be changed mentioned here: /sig scalability
Can I work on this? /assign
/remove-help
- sure that would be great.
Hi based on the discussion in , the kubemark image has dependency on and maybe more other utilities. So it seems impossible to migrate the base image to distroless? Or do we need something like debian-hyperkube-base?
The mount one we may get rid of - basically we should be able to switch to FakeMounter:


Question: if len(o.FilenameOptions.Filenames) != 1 { return cmdutil.UsageErrorf(cmd, "--raw can only use a single local file or stdin") } <del> if strings.HasPrefix(o.FilenameOptions.Filenames[0], "http") { </del> <ins> if strings.Index(o.FilenameOptions.Filenames[0], "http://") == 0 || strings.Index(o.FilenameOptions.Filenames[0], "https://") == 0 { </ins> return cmdutil.UsageErrorf(cmd, "--raw cannot read from a url") } if o.FilenameOptions.Recursive {

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: Currently when run kubectl create with --raw, it will raise "--raw cannot read from a url " if the filename has a prefix "http", which is not proper as we may have filenames like "httptest" or similar. What you expected to happen: Restrict the url check with "https://" or "http://" prefix. How to reproduce it (as minimally and precisely as possible): Run 'kubectl create --raw URI -f' with a filename which has a "http" prefix.
/sig cli
--raw should start with
the message could be improved, but I think the check is reasonable (I'd actually change it to error if the raw path doesn't start with )
This is the filename url check, not the raw path. I think you mean this


Question: _, err = client.CoreV1().Nodes().Patch(context.TODO(), old.Name, types.StrategicMergePatchType, patchBytes, metav1.PatchOptions{}, "status") return err } <ins> func patchPriorityClass(cs clientset.Interface, old, new *schedulingv1.PriorityClass) error { oldData, err := json.Marshal(old) if err != nil { return err } newData, err := json.Marshal(new) if err != nil { return err } patchBytes, err := strategicpatch.CreateTwoWayMergePatch(oldData, newData, &schedulingv1.PriorityClass{}) if err != nil { return fmt.Errorf("failed to create merge patch for PriorityClass %q: %v", old.Name, err) } _, err = cs.SchedulingV1().PriorityClasses().Patch(context.TODO(), old.Name, types.StrategicMergePatchType, patchBytes, metav1.PatchOptions{}) return err } </ins>


Question: if p.isPodBackingOff(pod) { if err := p.podBackoffQ.Add(pInfo); err != nil { klog.Errorf("Error adding pod %v to the backoff queue: %v", pod.Name, err) <ins> } else { p.unschedulableQ.delete(pod) </ins> } } else { if err := p.activeQ.Add(pInfo); err != nil { klog.Errorf("Error adding pod %v to the scheduling queue: %v", pod.Name, err) <ins> } else { p.unschedulableQ.delete(pod) </ins> } } <del> p.unschedulableQ.delete(pod) </del> } p.moveRequestCycle = p.schedulingCycle p.cond.Broadcast()

What happened: We will now always delete the pods from unschedulable queue no matter the Add operations succeed or not in function. What you expected to happen: Delete pods from unschedulable queue only when add operations succeeds
/sig scheduling
/assign
Will the pods be to the queues again somewhere if we delete them by mistake here ? It seems that can not make sure to add them again.
I am not sure if we will add them back again somewhere. If not, not only but also need to be updated. I have already send a PR here:


Question: DescribeEvents(events, w) } <ins> printPodsMultiline(w, "Mounted By", mountPods) </ins> return nil }) }

Is this a BUG REPORT or FEATURE REQUEST?: /kind feature What happened: I need to know to which Pod(s) a PVC is being mounted to, without having to go through each Pod one by one checking the Volume mounts and the PersistentVolumeClaim. The kubectl describe pvc and get pvc, only tell you to which Persistent Volume the PVC is bound to, but not if is being used by a Pod(s) What you expected to happen: Add a new field in the describe or get that lists the pods that are using the PVC as a volume mount How to reproduce it (as minimally and precisely as possible): Have a PV/PVC bound and create a pod using a volume mount Anything else we need to know?: Not sure if this is only for Azure Disk, which is the Storage Class I am using. Environment: Kubernetes version (use ): 7 Cloud provider or hardware configuration: Azure OS (e.g. from /etc/os-release): Ubuntu 16.04 Kernel (e.g. ): 4 Install tools: Azure Kubernetes Services Others: -n/a
/sig storage
/sig cli
I think the kubectl describe can just construct this on the fly. Another angle is should we add this information to PVC object itself
That will be great, as right now the only way to correlate a PVC to a Pod is via a "describe" on the Pod.
I can work on a PR for this!
GitHub didn't allow me to assign the following users: clandry94. Note that only and repo collaborators can be assigned. <detailsIn response to : I can work on this! Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
+1 for the feature. cc
How to find "what pods are using particular PVC through command line"
+1 for the feature.
this feature will be in Kubernetes 1.12
ok kubernetes 1.12 released, how to? which command?
Can someone point to the documentation or paste command to get the pods which have the PVC in use
see the associated PR to see an example usage of this The line in should show it.
Workaround for the meantime:
Thanks , this workaround is great for those people who still use old kubernetes version.


Question: DescribeEvents(events, w) } <ins> printPodsMultiline(w, "Mounted By", mountPods) </ins> return nil }) }

However it will meet an error if the pod have no volumes, so I improves it by adding a to check whether the volumes exists or not.
python script to list pods with pvc in namespace(s): Is a part of a cli to set pods with velero/restic annotations. Hope it helps someone :)
For posterity... There is a Used By: field in the output of describe.
client-go didn't give this api... ,PodPVCIndex can solve the problem
Hi thanks worked for me ;) but I want to make it reverse my workarounds with adding after didn't worked do you have any ideas
$ kubectl describe pvc -A Used" Name: ocs-deviceset-lso-volumeset-0-data-1b45dn Namespace: openshift-storage StorageClass: lso-volumeset Used By: rook-ceph-osd-1--8fwpd Also useful ^
the reason it didn't work is because each return does, in point of fact, have a key defined for claimName. That key is just set to null value. So, what you'd actually want to do is test for nulls. As nice as answer is, it's a PITA to parse unstructured human-readable command output, as opposed to command output in a markup language that well known shell parsers like jq and yq can work with. Towards that end, because I had use of this, I came up with the following: That simple not null test at the end plucks out all your nulls. Obviously invert the last conditional if you want all the nulls, but you'll get all sorts of temp things that way that you probably won't care about.


Question: } } } <ins> func TestMinionListConversionToNew(t *testing.T) { oldMinion := func(id string) v1beta1.Minion { return v1beta1.Minion{JSONBase: v1beta1.JSONBase{ID: id}} } newMinion := func(id string) Minion { return Minion{JSONBase: JSONBase{ID: id}} } oldMinions := []v1beta1.Minion{ oldMinion("foo"), oldMinion("bar"), } newMinions := []Minion{ newMinion("foo"), newMinion("bar"), } table := []struct { oldML *v1beta1.MinionList newML *MinionList }{ { oldML: &v1beta1.MinionList{Items: oldMinions}, newML: &MinionList{Items: newMinions}, }, { oldML: &v1beta1.MinionList{Minions: oldMinions}, newML: &MinionList{Items: newMinions}, }, { oldML: &v1beta1.MinionList{ Items:   oldMinions, Minions: []v1beta1.Minion{oldMinion("baz")}, }, newML: &MinionList{Items: newMinions}, }, } for _, item := range table { got := &MinionList{} err := Convert(item.oldML, got) if err != nil { t.Errorf("Unexpected error: %v", err) } if e, a := item.newML, got; !reflect.DeepEqual(e, a) { t.Errorf("Expected: %#v, got %#v", e, a) } } } func TestMinionListConversionToOld(t *testing.T) { oldMinion := func(id string) v1beta1.Minion { return v1beta1.Minion{JSONBase: v1beta1.JSONBase{ID: id}} } newMinion := func(id string) Minion { return Minion{JSONBase: JSONBase{ID: id}} } oldMinions := []v1beta1.Minion{ oldMinion("foo"), oldMinion("bar"), } newMinions := []Minion{ newMinion("foo"), newMinion("bar"), } newML := &MinionList{Items: newMinions} oldML := &v1beta1.MinionList{ Items:   oldMinions, Minions: oldMinions, } got := &v1beta1.MinionList{} err := Convert(newML, got) if err != nil { t.Errorf("Unexpected error: %v", err) } if e, a := oldML, got; !reflect.DeepEqual(e, a) { t.Errorf("Expected: %#v, got %#v", e, a) } } </ins>

should have the JSON attribute name "items", not "minions"
Doh! Fix now and break clients?
Define Items and Minions and use Convertors to resolve it?
Sure. I'll make a fix.


Question: const ConfigSourceAnnotationKey = "kubernetes.io/config.source" const ConfigMirrorAnnotationKey = "kubernetes.io/config.mirror" const ConfigFirstSeenAnnotationKey = "kubernetes.io/config.seen" <ins> const ConfigHashAnnotationKey = "kubernetes.io/config.hash" </ins> // PodOperation defines what changes will be made on a pod configuration. type PodOperation int

See for more context. Currently, kubelet compares the spec of a mirror pod (created via apiserver) with that of a static pod (config stored as a file) to decide whether the two are considered identical. If they differ, kubelet would delete the mirror pod and re-create it with the correct spec. Due to the fact that apiserver may set default values for some fields in the spec of the mirror pod, the specs could deviate undesirably. To avoid problems like this, we can store the config of a static pod as an annotation in the mirror pod, and compare the static pod with the annotation.
Would a hash of the config be sufficient if we're only going to compare? The whole spec would generally be more useful, but I wonder if that is too much data to put in an annotation.
This seems totally reasonable to me, but I'm still desperately curious why you can't just bypass defaulting when you generate mirror pod from static pod? (i.e. just use the static pod's configuration verbatim)
It's not kubelet who applies the default values here. The admission plugin was only run at the apiserver, and to bypass that, you have to ask apiserver to treat mirror pods differently and potentially at multiple places.
On May 22, 2015 10:38 AM, "Yu-Ju Hong" wrote: Understood, but is it really multiple places? If so then I agree it's a bad idea. Otherwise it might be worth considering as an alternative.
It's untenable for Kubelet to assume to know/circumvent all fields that will be set by other entities: admission control, defaulting, late initialization, auto-something, whatever. It needs to obey the same rules as other clients.
I don't understand why you would ever want to apply any of the things you mentioned to static pods. Static pods should be copied verbatim into the mirror pods, right? That's what I'm proposing.
What if the master is updated before the kubelet, and the master adds a new API field with a default value?
That case should be discussed in
I agree that mirror pod should reflect the true status of the static pod running on the node, and no default values should be set in this case.


Question: const ConfigSourceAnnotationKey = "kubernetes.io/config.source" const ConfigMirrorAnnotationKey = "kubernetes.io/config.mirror" const ConfigFirstSeenAnnotationKey = "kubernetes.io/config.seen" <ins> const ConfigHashAnnotationKey = "kubernetes.io/config.hash" </ins> // PodOperation defines what changes will be made on a pod configuration. type PodOperation int

However, I think it'd be harder to implement this way since, as stated above, that there are many things to circumvent in the apiserver itself. It could also be harder to keep this special case from being broken by a yet another new feature/plugin. Keeping the hash of the config in the annotation seems like a good compromise even though it may not reflect the true status 100% accurately. I think this is acceptable. What do you think?
Sure, if you think the annotation approach is easier, I would go with that. I guess in the case Brian mentoined (master adds a new API field with a default value), you would not update the mirror pod, because the static pod definition has not changed?
Yes, a hash would be sufficient.
It shouldn't be necessary to update the mirror pod in the case that a new default field was updated, because that field should be populated automatically by the apiserver on decode from etcd.
We should fix this so we don't have this problem again in 1.2.


Question: decoder := legacyscheme.Codecs.DecoderToVersion( legacyscheme.Codecs.UniversalDeserializer(), runtime.NewMultiGroupVersioner( <del> *defaultGroup.GroupVersion(), schema.GroupKind{Group: defaultGroup.GroupVersion().Group}, schema.GroupKind{Group: extGroup.GroupVersion().Group}, </del> <ins> defaultGroup, schema.GroupKind{Group: defaultGroup.Group}, schema.GroupKind{Group: extGroup.Group}, </ins> ), )

Execute make test WHAT=./pkg/api/testing/ And it failed with information: /kind failing-test
/sig api-machinery
what's the urgency of fixing this? Is it a release blocker?
i don't see it as a release blocker. but the problem is in client-go, so it's important to get fixed.


Question: } func getPatchedJS(contentType string, originalJS, patchJS []byte, obj runtime.Object) ([]byte, error) { <ins> // Remove "; charset=" if included in header. if idx := strings.Index(contentType, ";"); idx > 0 { contentType = contentType[:idx] } </ins> patchType := api.PatchType(contentType) switch patchType { case api.JSONPatchType:

Given a request with a Content-Type header set to , the server will respond with an error because it does not exactly match the existing strings. Some browsers (Firefox, see: ) will automatically add to a Content-Type header when making an XHR request.


Question: dockercontainer "github.com/docker/docker/api/types/container" dockerfilters "github.com/docker/docker/api/types/filters" "github.com/golang/glog" <del> "k8s.io/api/core/v1" </del> runtimeapi "k8s.io/kubernetes/pkg/kubelet/apis/cri/v1alpha1/runtime" )

seems to belong to sig node PTAL and fix ASAP
/assign as it seems like having something to do with