Question: syncLoopPeriod                = 100 * time.Minute maxWaitForUnmountDuration     = 50 * time.Millisecond maxLongWaitForUnmountDuration = 4200 * time.Second <ins> volumeAttachedCheckTimeout    = 5 * time.Second </ins> ) // Calls Run()

pull-kubernetes-unit TestRunOneVolumeDetachFailNodeWithReadWriteOnce Jan 7th, 2022 Check volume <fake-plugin/volume-nameis reported as attached to node <fail-detach-node, got false, expected true No response_ /sig storage
/cc
this is the top unit test flake according to
cc
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
I think this test is no longer flaky by checking the test grid, close it now.
Unfortunately this does still seem to be flaky: Here was the output from : /reopen
Reopened this issue. <detailsIn response to : Unfortunately this does still seem to be flaky: Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-lifecycle stale
still the case
How about adding some logging to help troubleshoot this? I looked at it a little, but it is hard to tell which part of the test is failing. PR
/
This is still happening
/triage accepted
From the logs, in the OneVolumeDetachFailNodeWithReadWriteOnce test, succeeded, but the next verification failed. In verifyVolumeAttachedToNode, the attachState is from calling and within asw, it calls . In verifyVolumeReportedAsAttachedToNode, the volumes are obtained from calling , it seems that the node is not in asw.nodesToUpdateStatusFor at this step. Cannot reproduce locally. I tried to run the test 100 times locally with debug.Stack()... 3 * 100 times all passed.
Tried to run stress test on my linux machine as well but still cannot reproduce. Command: Result: Also tried running n process in parallel. Command: Result:
It seems like it happens more frequently.


Question: syncLoopPeriod                = 100 * time.Minute maxWaitForUnmountDuration     = 50 * time.Millisecond maxLongWaitForUnmountDuration = 4200 * time.Second <ins> volumeAttachedCheckTimeout    = 5 * time.Second </ins> ) // Calls Run()

But I may have no time to dig into the cause. Let's see.
cc as the test author.
The problem is that this test has a lot of dependency on the sleeps that are hardcoded with a very narrow scope, but our CI seems to be able to stress it much more. As soon as you slightly change one of the sleep duration it starts to flake locally The test can not make an assumption that timing is perfect, even more in the sub second range can someone in sig-storage review this test and make it less dependant on those tight sleeps?
According to most failures, it failed at Line 660. Let me run some stress locally. with 250ms sleep with 100ms sleep
I agree this test case is too brittle due to its specific timing dependencies. To fix this, I suggest reworking the test case to: 1) Add retry with timeout to wait for final conditions 2) Remove the intermediate timing-based state transition checks. Instead, we can keep track of state transitions in the Fakes and validate those stats. wdyt?
Friendly ping from 1.27 CI Signal that this test is still a noisy flake on the master-blocking dashboard as CI Signal has flagged it in and


Question: testCases := []struct { name          string ipModeEnabled bool <ins> nonLBAllowed  bool </ins> tweakLBStatus func(s *core.LoadBalancerStatus) <ins> tweakSvcSpec  func(s *core.ServiceSpec) </ins> numErrs       int }{ <del> /* LoadBalancerIPMode*/ </del> { <ins> name:         "type is not LB", nonLBAllowed: false, tweakSvcSpec: func(s *core.ServiceSpec) { s.Type = core.ServiceTypeClusterIP }, tweakLBStatus: func(s *core.LoadBalancerStatus) { s.Ingress = []core.LoadBalancerIngress{{ IP: "1.2.3.4", }} }, numErrs: 1, }, { name:         "type is not LB. back-compat", nonLBAllowed: true, tweakSvcSpec: func(s *core.ServiceSpec) { s.Type = core.ServiceTypeClusterIP }, tweakLBStatus: func(s *core.LoadBalancerStatus) { s.Ingress = []core.LoadBalancerIngress{{ IP: "1.2.3.4", }} }, numErrs: 0, }, { </ins> name:          "valid vip ipMode", ipModeEnabled: true, tweakLBStatus: func(s *core.LoadBalancerStatus) {

xref I think this may be one of the few cases where we can actually change the behavior in a "breaking" way. User creates service with , is empty Controller or user updates the status subresource, sets (not wiped or validated, should be) User updates in any way, is wiped It's clear that the is not meant to be set when is not "LoadBalancer". The question is: 1) Fail validation? 2) Silently discard the value? 3) Warn and then discard the value? 4) Just warn and leave it. Because we allowed it in the past, but it is so volatile, I lean towards 3. and - the people most familair with this type. Edit: PrepareForUpdate() runs before warnings. So: (1) is easy (2) is easy (4) is slightly less easy than (2) (3) is slightly less easy than (4)
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
OMG change it to clusterIP as expected Status is wiped update status as ClusterIP ... and if I edit the service back to loadbalancer it keeps the status??
/kind bug
/retitle Service can be set when !=
We talked about this one at the SIG Network meeting today. suggested we use the feature gate for deprecation option to phase the current (problematic) functionality out.
/triage accept
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /triage accept Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: } client := oauth2.NewClient(oauth2.NoContext, tokenSource) <ins> // Override the transport to make it rate-limited. client.Transport = &rateLimitedRoundTripper{ rt:      client.Transport, limiter: flowcontrol.NewTokenBucketRateLimiter(10, 100), // 10 qps, 100 bucket size. } </ins> svc, err := compute.New(client) if err != nil { return nil, err

On a large cluster, the routecontroller basically makes O(n) requests all at once, and most of them error out with "Rate Limit Exceeded". We then wait for operations to complete on the ones that did stick, and damn the rest, they'll be caught next reconcile (which is a while). In fact, in some cases it appears we're spamming hard enough to get rate limited by the upper-level API DOS protections and seeing something akin to , e.g.: We can do better than this!
c.f. , which is the other leg of this problem
I took a look into code and it seems that we are doing full reconciliation every 10s. If we need to create 2000 routes at the beginning it seems to frequently. I think that we need to: first compute how many routes we have to create based on that try to spread them over some time to reduce possibility of causing 'rate limit exceeded' I will try to put some small PR together as a starting point.
I'm reopening this one, since this doesn't seem to be solved. So basically something strange is happening here. With PR we have 10/s limit on API calls send to GCE. And we are still rate-limitted. However, according to documentation we are allowed for 20/s: So something either doesn't work or is weird.
What is super interesting from the logs of recent run: It starts to creating routes: After 2 minutes from that, we are hitting "Rate limit Exceeded" for the first time: And only after 9 minutes we create the first route ever. The second one is created after 15 minutes, and from that point we create a number of them So it seems that nothing useful is happening for the first 2 minutes, and then between 3m30 and 9m30
OK - so I think that what is happening is that all controllers are sharing GCE interface (and thuse they have common throttling). So if one controller is generating a lot of API calls, then other controllers may be throttled. has a hypothesis that it may be caused by nodecontroller.
One thing my PR didn't address (because it was late): Now we have some obnoxious differences in logging when you go to make a request, because the versus could actually be absorbing a fair amount of ratelimiter.


Question: } client := oauth2.NewClient(oauth2.NoContext, tokenSource) <ins> // Override the transport to make it rate-limited. client.Transport = &rateLimitedRoundTripper{ rt:      client.Transport, limiter: flowcontrol.NewTokenBucketRateLimiter(10, 100), // 10 qps, 100 bucket size. } </ins> svc, err := compute.New(client) if err != nil { return nil, err

It could before because of the operation rate limiter and operation polling anyways, but now could be worse.
- yes I'm aware of that. And probably this is exactly the case here.
I got fooled by that - I guess we should fix it somehow.
One approach to fixing it is a similar approach to the operation polling code, but in the : warn if we ratelimited for more than N seconds (and dump the request or something? not clear how to clarify what we were sending).
I found something else - Kubelet is also building GCE interface: We have 2000 kubelets, so if all of them send multiple requests to GCE, we may hit limits.
Any idea in what circumstances Kubelet will contact cloud provider?
Hmm - it seems it contacts GCE exactly once at the beginning. So it doesn't explain much.
I think it's only here: I've rarely seen GCE ratelimit readonly calls, in practice, and kubelets come up pretty well staggered.
OK - so with logs that I , it is pretty clear that we are very heavily throttled on GCE api calls. In particular, when I was running 1000-node cluster, I see lines where we were throttle for 1m30. I'm pretty sure it's significantly higher in 2000-node cluster. I'm going to investigate a bit more where all those reuqests come from.
Actually - it seems that we have a pretty big problem here. Basically, Kubelet seems to send a lot of requests to GCE. So to clarify: every time Kubelet is updating NodeStatus (so every 10s by default) it is sending an API call to GCE to get it's not addresses: that means, that if we have 2000 nodes, kubelets generate 200 QPS to GCE (which is significantly more than we can afford) We should think how to solve this issue, but I think the options are: increasing QPS quota or calling GCE only once per X statusUpdates.
The second issue is that we are throttled at the controller level too, and this one I'm tryin to understand right now.


Question: } client := oauth2.NewClient(oauth2.NoContext, tokenSource) <ins> // Override the transport to make it rate-limited. client.Transport = &rateLimitedRoundTripper{ rt:      client.Transport, limiter: flowcontrol.NewTokenBucketRateLimiter(10, 100), // 10 qps, 100 bucket size. } </ins> svc, err := compute.New(client) if err != nil { return nil, err

- that said, any rate-limitting on our side, will not solve the kubelet-related problem
Hmm - actually looking into implementation it seems that NodeAddresses are only contacting metadata server, so I'm not longer that convinced...
Also, I think that important thing is that a single call to GCE API can translate to multiple api calls to GCE. In particular - CreateRoute translates to: get instance insert route a bunch of getOperation (until this is finished)
OK - so I think that what is happening here is that since CreateRoute translates to: get instance insert route a bunch of getOperations That means that if we call say 2000 CreateRoute() at the same time, all of them will try to issue: getInstance at the beginning. So we will quickly accumulate 2000 GET requests in the queue. Once they are processed, we generate the "POST route" then for any processed. So It's kind of expected what is happening here. So getting back to the Zach`s PR - I think that throttling POST and GET and OPERATION calls separate is kind of good idea in general.
But I still don't really understand why do we get "RateLimitExceeded". Is that because of Kubelets?
I think that one problem we have at the RouteController level is that if CreateRoute fails, we simply don't react on it. And we will wait at least 5 minutes before even retrying it. We should add some retries at the route controller level too. I will send out some short PR for it.
But I still don't understand why do we get those "RateLimitExceeded" error
OK - so as an update. In GCE, we have a separate rate-limit for number of in-flight CreateRoute call per project. is supposed to address this issue.
OK - so this one is actually fixed. I mean that it is still long, but currently it's purely blocked on GCE.
As a comment - we have an internal bug for it already.


Question: var _ = SIGDescribe("Deployment", func() { var ns string var c clientset.Interface <ins> var dc dynamic.Interface </ins> ginkgo.AfterEach(func() { failureTrap(c, ns)

[x] Original approval issue: [x] Test pr: [x] Two weeks soak end date: 19 November 2020 This issue is created to allow edit by /wip /hold According to this APIsnoop query, there are still some remaining Deployment endpoints which are untested. a Deployment with a static label the Deployment with a new Label and updated data the Deployment to ensure it's patched all Deployments in all Namespaces find the Deployment(1) ensure that the Deployment is found and is patched Namespaced Deployment(1) via a Collection with a LabelSelector Discover useragents: List endpoints hit by the test: Display endpoint coverage change: If a test with these calls gets merged, test coverage will go up by 5 points* This test is also created with the goal of conformance promotion.
/sig testing /sig architecture /area conformance
BobyMcbobs commented in original Issue: Unsure if the endpoint replaceAppsV1NamespacedDeploymentStatus is able to be hit. When this endpoint is attempted to be updated the data after retrieving the update resource doesn't reflect the values in the update
/assign
When using the appv1.Deployment structure instead of maps of interfaces to build the structure in a patch of the Deployment, this error comes up. Still in progress yment-static":"true"}: does not match template , spec.selector: Invalid value: "null": field is immutable]
replaceAppsV1NamespacedDeploymentStatus to test. Increase cover by 1 Endpoint
/reopen PR reverted by due to flaky test
Reopened this issue. <detailsIn response to : /reopen PR reverted by due to flaky test Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/reopen Soaking, ready for promotion 19 November 2020
Reopened this issue. <detailsIn response to : /reopen Soaking, ready for promotion 19 November 2020 Instructions for interacting with me using PR comments are available .


Question: var _ = SIGDescribe("Deployment", func() { var ns string var c clientset.Interface <ins> var dc dynamic.Interface </ins> ginkgo.AfterEach(func() { failureTrap(c, ns)

If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-lifecycle stale
/triage accepted
/merged


Question: load("@io_bazel_rules_docker//container:container.bzl", "container_pull") container_pull( <del> name = "debian_jessie", digest = "sha256:e25703ee6ab5b2fac31510323d959cdae31eebdf48e88891c549e55b25ad7e94", registry = "index.docker.io", repository = "library/debian", tag = "jessie",  # ignored when digest provided, but kept here for documentation. </del> <ins> name = "distroless_base", digest = "sha256:7fa7445dfbebae4f4b7ab0e6ef99276e96075ae42584af6286ba080750d6dfe5", registry = "gcr.io", repository = "distroless/base", tag = "latest",  # ignored when digest provided, but kept here for documentation. </ins> ) load("//build:workspace.bzl", "release_dependencies")

There are CVEs happening in different images - we already migrated Kubmeark from busybox in , but we should go one step further and migrate the base image to distroless. There are some things that would need to be changed mentioned here: /sig scalability
Can I work on this? /assign
/remove-help
- sure that would be great.
Hi based on the discussion in , the kubemark image has dependency on and maybe more other utilities. So it seems impossible to migrate the base image to distroless? Or do we need something like debian-hyperkube-base?
The mount one we may get rid of - basically we should be able to switch to FakeMounter:


Question: } } <del> for _, endpointSlice := range slicesToCreate { addTriggerTimeAnnotation(endpointSlice, triggerTime) createdSlice, err := r.client.DiscoveryV1beta1().EndpointSlices(service.Namespace).Create(context.TODO(), endpointSlice, metav1.CreateOptions{}) if err != nil { // If the namespace is terminating, creates will continue to fail. Simply drop the item. if errors.HasStatusCause(err, corev1.NamespaceTerminatingCause) { return nil </del> <ins> // Don't create new EndpointSlices if the Service is pending deletion. This // is to avoid a potential race condition with the garbage collector where // it tries to delete EndpointSlices as this controller replaces them. if service.DeletionTimestamp == nil { for _, endpointSlice := range slicesToCreate { addTriggerTimeAnnotation(endpointSlice, triggerTime) createdSlice, err := r.client.DiscoveryV1beta1().EndpointSlices(service.Namespace).Create(context.TODO(), endpointSlice, metav1.CreateOptions{}) if err != nil { // If the namespace is terminating, creates will continue to fail. Simply drop the item. if errors.HasStatusCause(err, corev1.NamespaceTerminatingCause) { return nil } errs = append(errs, fmt.Errorf("Error creating EndpointSlice for Service %s/%s: %v", service.Namespace, service.Name, err)) } else { r.endpointSliceTracker.Update(createdSlice) metrics.EndpointSliceChanges.WithLabelValues("create").Inc() </ins> } <del> errs = append(errs, fmt.Errorf("Error creating EndpointSlice for Service %s/%s: %v", service.Namespace, service.Name, err)) } else { r.endpointSliceTracker.Update(createdSlice) metrics.EndpointSliceChanges.WithLabelValues("create").Inc() </del> } }

What happened: In the PMEM-CSI CI, we rapidly deploy and remove a custom resources which then owns all objects created for that resource. Deletion of those objects is handled through ownership references, i.e. deleting the CR is expected to cascade to all objects. This worked fine up to Kubernetes 1.17. When starting to test with 1.18.2, we pretty reliably get into a state where the CR and the service objects owned by it are marked for deletion, but don't go away. Debugging with the garbage collector's dependency graph (see ) showed that endpointslices were blocking the deletion of the services and thus the CR. In the graph dump (= a snaphot in time), they were not even marked for deletion. A closer look showed that endpointslices for the services were getting created and deleted continuously for over 15 minutes. Eventually, the garbage collector won and managed to delete endpointslices and services before new endpointslices were created. What you expected to happen: There seems to be a missing check in if the service is marked for deletion (i.e. has a deletionTimestamp), then the controller should not create new endpointslices. How to reproduce it (as minimally and precisely as possible): I tried to boil it down to a script, but it just ran fine. The "easiest" way to reproduce it is to run PMEM-CSI's E2E testing, which is probably not easy enough :disappointed: Anything else we need to know?: Disabling the EndpointSlice feature makes the problem go away. Environment: Kubernetes version (use ): 1.18.2 Cloud provider or hardware configuration: QEMU virtual machines OS (e.g: ): Fedora Install tools: kubeadm /sig network
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
cc for endpointslice
Thanks for the detailed bug report! Will work on getting a fix in soon. /assign
/remove-triage unresolved
Can you describe the general process to reproduce this bug? Would it involve hundreds of EndpointSlices for a Service? I've tried to recreate this in a , but unfortunately I haven't been able to get anything to break. Any ideas for what I could try?
There's one endpointslice per service. Our operator creates two services.


Question: } } <del> for _, endpointSlice := range slicesToCreate { addTriggerTimeAnnotation(endpointSlice, triggerTime) createdSlice, err := r.client.DiscoveryV1beta1().EndpointSlices(service.Namespace).Create(context.TODO(), endpointSlice, metav1.CreateOptions{}) if err != nil { // If the namespace is terminating, creates will continue to fail. Simply drop the item. if errors.HasStatusCause(err, corev1.NamespaceTerminatingCause) { return nil </del> <ins> // Don't create new EndpointSlices if the Service is pending deletion. This // is to avoid a potential race condition with the garbage collector where // it tries to delete EndpointSlices as this controller replaces them. if service.DeletionTimestamp == nil { for _, endpointSlice := range slicesToCreate { addTriggerTimeAnnotation(endpointSlice, triggerTime) createdSlice, err := r.client.DiscoveryV1beta1().EndpointSlices(service.Namespace).Create(context.TODO(), endpointSlice, metav1.CreateOptions{}) if err != nil { // If the namespace is terminating, creates will continue to fail. Simply drop the item. if errors.HasStatusCause(err, corev1.NamespaceTerminatingCause) { return nil } errs = append(errs, fmt.Errorf("Error creating EndpointSlice for Service %s/%s: %v", service.Namespace, service.Name, err)) } else { r.endpointSliceTracker.Update(createdSlice) metrics.EndpointSliceChanges.WithLabelValues("create").Inc() </ins> } <del> errs = append(errs, fmt.Errorf("Error creating EndpointSlice for Service %s/%s: %v", service.Namespace, service.Name, err)) } else { r.endpointSliceTracker.Update(createdSlice) metrics.EndpointSliceChanges.WithLabelValues("create").Inc() </del> } }

One peculiarity may be that we don't actually want the pods to start during these tests because we only want to verify object creation, not the usage of the result, and therefore all pods reference non-existent images. All objects immediately get deleted once the test has verified that they exist, so the timing is probably very different from "normal" usage patterns. I suspect that it is more important to create and delete services quickly than it is to have many endpointslices per service.
I had tried to come up with a script that triggers the problem, without success. That was before I realized that endpointslices are involved. On Monday I can have another look.
That was because I thought that (the default) enables foreground deletion. It doesn't, and there's also no option to select it. After explicitly selecting it via curl, the following script failed for me eventually: Output: Note that for the endpointslice is greater than the of its service.


Question: } <ins> func TestFailedParseParamsSummaryHandler(t *testing.T) { fw := newServerTest() defer fw.testHTTPServer.Close() resp, err := http.Post(fw.testHTTPServer.URL+"/stats/summary", "invalid/content/type", nil) assert.NoError(t, err) defer resp.Body.Close() v, err := ioutil.ReadAll(resp.Body) assert.NoError(t, err) assert.Equal(t, http.StatusInternalServerError, resp.StatusCode) assert.Contains(t, string(v), "parse form failed") } </ins> func TestTrimURLPath(t *testing.T) { tests := []struct { path, expected string

What happened: In handler#handleSummary, return value from request.Request.ParseForm() is ignored. What you expected to happen: The error return should be checked.
/sig node
Hi this seems to be easy to fix. Could I solve this?
Please go ahead.


Question: ) decoder := opts.StorageSerializer.DecoderToVersion( recognizer.NewDecoder(decoders...), <del> runtime.NewMultiGroupVersioner( </del> <ins> runtime.NewCoercingMultiGroupVersioner( </ins> opts.MemoryVersion, schema.GroupKind{Group: opts.MemoryVersion.Group}, schema.GroupKind{Group: opts.StorageVersion.Group},

What happened: Set up a HA cluster with 1.9 and 1.10 API servers (occurs on any apiserver skew boundary where a cohabitating resource's storage version changed groups) Observed errors when modifying workload objects: when modified via the 1.10 server, saw this error in the 1.9 server: when modified via the 1.9 server, saw this error in the 1.10 server: When this error was encountered, all watchers for the resource in question would be dropped and have to re-establish, and the internal watchers feeding the watch cache would drop and re-establish What you expected to happen: Watch events from all supported storage versions are processed correctly How to reproduce it (as minimally and precisely as possible): Start a server from master (for inspectability, store in json). With Create a deployment (e.g. kube-dns in kube-system) Establish a watch via extensions (kubectl get deployments.extensions --all-namespaces -w) Establish a watch via apps (kubectl get --all-namespaces -w) Tail the kube-apiserver log Change the apps/v1 deployment in etcd to extensions/v1beta1: Observer client watches drop and server log errors: Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: /sig api-machinery /cc /priority critical-urgent
error is triggered by this returning false: constructed here: the info about expected cohabitating groups is not included in the list of acceptable inputs


Question: expectedIPFamilies:     nil, expectError:            true, }, <ins> // preferDualStack services should not be updated // to match cluster config if the user didn't change any // ClusterIPs related fields { name:       "unchanged preferDualStack-1-ClusterUpgraded", modifyRest: fnMakeDualStackStackIPv4IPv6Allocator, oldSvc: &api.Service{ Spec: api.ServiceSpec{ Type:           api.ServiceTypeClusterIP, ClusterIPs:     []string{"1.1.1.1"}, IPFamilies:     []api.IPFamily{api.IPv4Protocol}, IPFamilyPolicy: &preferDualStack, }, }, svc: api.Service{ Spec: api.ServiceSpec{ Type:           api.ServiceTypeClusterIP, ClusterIPs:     []string{"1.1.1.1"}, IPFamilies:     []api.IPFamily{api.IPv4Protocol}, IPFamilyPolicy: &preferDualStack, }, }, expectedIPFamilyPolicy: &preferDualStack, expectedIPFamilies:     []api.IPFamily{api.IPv4Protocol}, expectError:            false, }, { name:       "unchanged preferDualStack-2-ClusterDowngraded", modifyRest: fnMakeSingleStackIPv4Allocator, oldSvc: &api.Service{ Spec: api.ServiceSpec{ Type:           api.ServiceTypeClusterIP, ClusterIPs:     []string{"1.1.1.1", "2001::1"}, IPFamilies:     []api.IPFamily{api.IPv4Protocol, api.IPv6Protocol}, IPFamilyPolicy: &preferDualStack, }, }, svc: api.Service{ Spec: api.ServiceSpec{ Type:           api.ServiceTypeClusterIP, ClusterIPs:     []string{"1.1.1.1", "2001::1"}, IPFamilies:     []api.IPFamily{api.IPv4Protocol, api.IPv6Protocol}, IPFamilyPolicy: &preferDualStack, }, }, expectedIPFamilyPolicy: &preferDualStack, expectedIPFamilies:     []api.IPFamily{api.IPv4Protocol, api.IPv6Protocol}, expectError:            false, }, { name:       "changed preferDualStack-1 (cluster upgraded)", modifyRest: fnMakeDualStackStackIPv4IPv6Allocator, oldSvc: &api.Service{ Spec: api.ServiceSpec{ Type:           api.ServiceTypeClusterIP, ClusterIPs:     []string{"1.1.1.1"}, IPFamilies:     []api.IPFamily{api.IPv4Protocol}, IPFamilyPolicy: &preferDualStack, }, }, svc: api.Service{ Spec: api.ServiceSpec{ Type:           api.ServiceTypeClusterIP, ClusterIPs:     nil, IPFamilies:     []api.IPFamily{api.IPv4Protocol}, IPFamilyPolicy: &requireDualStack, }, }, expectedIPFamilyPolicy: &requireDualStack, expectedIPFamilies:     []api.IPFamily{api.IPv4Protocol, api.IPv6Protocol}, expectError:            false, }, { name:       "changed preferDualStack-2-ClusterDowngraded", modifyRest: fnMakeSingleStackIPv4Allocator, oldSvc: &api.Service{ Spec: api.ServiceSpec{ Type:           api.ServiceTypeClusterIP, ClusterIPs:     []string{"1.1.1.1", "2001::1"}, IPFamilies:     []api.IPFamily{api.IPv4Protocol, api.IPv6Protocol}, IPFamilyPolicy: &preferDualStack, }, }, svc: api.Service{ Spec: api.ServiceSpec{ Type:           api.ServiceTypeClusterIP, ClusterIPs:     []string{"1.1.1.1"}, IPFamilies:     []api.IPFamily{api.IPv4Protocol}, IPFamilyPolicy: &singleStack, }, }, expectedIPFamilyPolicy: &singleStack, expectedIPFamilies:     []api.IPFamily{api.IPv4Protocol}, expectError:            false, }, </ins> } // This func only runs when feature gate is on

1) start with a single stack cluster 2) create a PreferDualStack service 3) configure API server with a 2nd service CIDR 4) Confirm the above service is not dual-stack 5) Add a label to the above service 6) Observe that the service becomes dual-stack I think the correct answer is to have a "repair loop" that upgrades (and downgrades!) PreferDualStack Services to match the config. I think this is a GA blocker.
/triage accepted The current thought is: of type with are special new breed of api objects that I don't think we have any similar precedent in the apis, i could be wrong. Their behavior is tied to how the cluster is configured. Thus their behavior must change as the cluster behavior change. a way the user is stating . If it is cluster then i am good with two ClusterIPs. If it is single stack cluster then I am also good with a single ClusterIP. One can assume that if the user was ok was with getting a additional IP when possible then they must be ok with taking it away when it is no longer possible. Based on the above I believe best course of action is create a new repair loop or piggyback on any of the existing (or better: fix it in rest stack during list and get calls). The new logic takes care of matching to the .
reproducer and regression test if somebody takes this, feel free to add it
ack - Scheduled to tackle 3rd week of May. Will definitely include that patch.


Question: } func testZonalFailover(c clientset.Interface, ns string) { <del> nodes := framework.GetReadySchedulableNodesOrDie(c) nodeCount := len(nodes.Items) </del> cloudZones := getTwoRandomZones(c) class := newRegionalStorageClass(ns, cloudZones) claimTemplate := newClaimTemplate(ns)

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: Testgrid: https://k8s-
/assign
The test's assumption about test parameter is wrong. Previously it assumed the group name is the same across all zones, but they are different in regional clusters. Looking into using node taints as the permanent fix, as suggested by the TODO comment in the test.
Hm I think taint based evictions is still an alpha feature.
OK. I propose the following solution: Get the zone of Schedule the StatefulSet pod to that zone using node affinity; wait for the pod to run Remove the node affinity from StatefulSet Delete
Can it be simplified? Schedule the pod Figure out which zone that pod got scheduled in Delete the instance group of that zone
Unfortunately I don't know of a good way to look up the instance group given the zone, since there could be instance groups from multiple clusters in the same project
How about: Schedule the pod Figure out the zone Taint all the nodes in the zone Kill the pod
Taint with NoSchedule? yeah that sounds good


Question: if err != nil { return nil, err } <del> minion, err := rs.registry.GetMinion(ctx, minion.Name) </del> <ins> minionName := minion.Name minion, err := rs.registry.GetMinion(ctx, minionName) if err == ErrNotHealty { return rs.toApiMinion(minionName), nil } </ins> if minion == nil { return nil, ErrDoesNotExist }

apiserver reports creation failure if a node is not ready (unhealthy, or kubelet is not ready), but the api object is already stored in master. extracted from and A short term solution is to have health_registry check node health upon creation, not just on retrieving. Or ignore unhealthy node. A long term solution might fall into the interface revamp
Why can't apiserver accept an unhealthy minion.
What's the point of having an unhealthy minion that we'll never be able to use? One reason I can think of is that the minion can later becomes healthy and thus a valid node in k8s. But for now, as we don't have the ability to handle unhealthy node, delegate the responsibility to user makes more sense. However, I admit the title is misleading. What I really what to convey is that kubernetes currently allow registrying unhealthy node, by storing it in RESTStorage. But the create operation reports fail because the node can't pass health checking. I don't think this is the right behavior. We can either return success and do health check later, or return failure and do not store it. (i prefer the latter one for the above reason).
I tend to think that we should accept unhealthy minions - minions may go in and out of health, or go up and down at any time - so trying ensure health at registration time seems like a fruitless effort since we have to handle the error cases anyway.
I've no problem with accepting unhealthy minions, and I agree with your point. The real problem here is that the apiserver returns an inconsistent status. For example, if I have an unhealthy minion that I want to registry. From user's perspective, the registration fails, because apiserver returns an error (resource not found). So I retry, but apiserver returns failure again; this time, the error is resource already exists, no matter the minion is healthy or not.
- yes, we should remove the health check as part of the registration and just report success. agree there is inconsistency.
+1 On Nov 7, 2014 8:48 AM, "Derek Carr" wrote:
Seems like it's a clients responsibility to warn the user that a node might not be available prior to it being , but the the master's responsibility to assume that all properly specified nodes are reachable before making decisions regarding using a node.


Question: if err != nil { return nil, err } <del> minion, err := rs.registry.GetMinion(ctx, minion.Name) </del> <ins> minionName := minion.Name minion, err := rs.registry.GetMinion(ctx, minionName) if err == ErrNotHealty { return rs.toApiMinion(minionName), nil } </ins> if minion == nil { return nil, ErrDoesNotExist }

There's very little difference between an admin stopping a VM without clearing Kube and adding a node to the cluster prior to it being fully reachable (I think).
"clients responsibility" ? if we enable kubectl to create minions, it would be kubectl who should check if the minion is reachable and complain? I'm not sure what 'client' means. (but I agree 100%, don't report failure adding a minion just because it isn't reachable)
node controller plugin should also give warning


Question: defer nodePortOp.Finish() // try set ip families (for missing ip families) <del> if err := rs.tryDefaultValidateServiceClusterIPFields(service); err != nil { </del> <ins> if err := rs.tryDefaultValidateServiceClusterIPFields(oldService, service); err != nil { </ins> return nil, false, err }


Question: } /* <del> Testname: allowPrivilegeEscalation unset and uid != 0. Description: Configuring the allowPrivilegeEscalation unset, allows the privilege escalation operation. A container is configured with allowPrivilegeEscalation not specified (nil) and a given uid which is not 0. When the container is run, the container is run using uid=0. This test is marked LinuxOnly since Windows does not support running as UID / GID, or privilege escalation. </del> <ins> Release : v1.15 Testname: Security Context, allowPrivilegeEscalation unset, uid != 0. Description: Configuring the allowPrivilegeEscalation unset, allows the privilege escalation operation. A container is configured with allowPrivilegeEscalation not specified (nil) and a given uid which is not 0. When the container is run, container's output MUST match with expected output verifying container ran with uid=0. This e2e Can not be promoted to Conformance as it is Container Runtime dependent and not all conformant platforms will require this behavior. [LinuxOnly]: This test is marked LinuxOnly since Windows does not support running as UID / GID, or privilege escalation. </ins> */ It("should allow privilege escalation when not explicitly set and uid != 0 [LinuxOnly] [NodeConformance]", func() { podName := "alpine-nnp-nil-" + string(uuid.NewUUID())

/kind cleanup /area conformance /area test /sig node /sig testing /sig architecture /assign /assign We have a number of tests that only have the tag that may make sense as tests, we should identify whether these tests cases meet (or could meet) the . We should open PR's to promote these if so. I would advise starting with the test/common cases
GitHub didn't allow me to assign the following users: mgdevstack, mkurylec. Note that only and repo collaborators can be assigned. For more information please see <detailsIn response to : /kind cleanup /area conformance /area test /sig node /sig testing /sig architecture /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
The tests in test/e2enode don't actually get run as part of Conformance as near as I can tell. If we want to run them as part of conformance, they need to be migrated into test/e2e/common. Some of the tests in e2enode may not be able to be ported over because they assume a level access we don't have in a conformance environment.
regarding this testcase: [sig-storage] HostPath should support r/w [NodeConformance]: It uses hostPath and privileged=true, to read/write from /tmp folder. Which would you suggest is the best orientation to overcome the privilege mode setting and allow its promotion?
/assign
Regarding , the suite runs the test only if the kubelet configuration was required to have the QOS (cgroups-per-qos) enabled. This configuration for the kubelet is selected at the time the cluster is set up. Looking for where this might be configured found: where is set to true. For other settings it is also configured to true. where is set to true. BUT also found where it is mentioned that the flag is no longer available. Questions: would it be ok to promote this test to be conformance, even it can be skipped based on configuration of the cluster? And not supported in the containerized kubelet?
I would consider cgroups to be an implementation detail that Conformance shouldn't be verifying
It uses hostPath and privileged=true, to read/write from /tmp folder.


Question: } /* <del> Testname: allowPrivilegeEscalation unset and uid != 0. Description: Configuring the allowPrivilegeEscalation unset, allows the privilege escalation operation. A container is configured with allowPrivilegeEscalation not specified (nil) and a given uid which is not 0. When the container is run, the container is run using uid=0. This test is marked LinuxOnly since Windows does not support running as UID / GID, or privilege escalation. </del> <ins> Release : v1.15 Testname: Security Context, allowPrivilegeEscalation unset, uid != 0. Description: Configuring the allowPrivilegeEscalation unset, allows the privilege escalation operation. A container is configured with allowPrivilegeEscalation not specified (nil) and a given uid which is not 0. When the container is run, container's output MUST match with expected output verifying container ran with uid=0. This e2e Can not be promoted to Conformance as it is Container Runtime dependent and not all conformant platforms will require this behavior. [LinuxOnly]: This test is marked LinuxOnly since Windows does not support running as UID / GID, or privilege escalation. </ins> */ It("should allow privilege escalation when not explicitly set and uid != 0 [LinuxOnly] [NodeConformance]", func() { podName := "alpine-nnp-nil-" + string(uuid.NewUUID())

Which would you suggest is the best orientation to overcome the privilege mode setting and allow its promotion? Michelle Au If my understanding is correct, on some platforms, writing to /tmp requires privileged. I don't see a way to to remove that requirement.
If my understanding is correct, on some platforms, writing to /tmp requires privileged. I don't see a way to to remove that requirement. Yeah writing to host doesn't seem like a thing we can expect of conformance tests. I wonder if it makes sense to tag this test with something to denote the requirement, ie: I see some tests in test/e2e_node use
The hostpath tests in already have a comment explaining this. Maybe the comment should be updated to be more clear that that is the reason why we can't add it to conformance.
That's /tmp on the host or in the container filesystem?
/tmp on the host
ping what else remains to be done before we close this out? I am guessing that we want to close , is there anything beyond that?
Once tests under (issue- ) and are promoted we are good to close this issue. Short update for reference- None of [NodeConformance] tests from "test/e2e_node" seems eligible for [Conformance] promotion at this moment due to node level dependencies. Remaining [NodeConformance] tests from "test/e2e" has following status - Waiting for to be closed and then below tests would be promoted to Conformance Needs to be Promoted Can not be promoted to Conformance as //Slow by design Can not be promoted due to Node level dependency
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
/lifecycle frozen
/assign


Question: List<InetAddress> list = new ArrayList<InetAddress>(); String host = "https://kubernetes.default.cluster.local"; String serviceName = getEnvOrDefault("CASSANDRA_SERVICE", "cassandra"); <del> String path = "/api/v1/namespaces/default/endpoints/"; </del> <ins> String podNamespace = getEnvOrDefault("POD_NAMESPACE", "default"); String path = String.format("/api/v1/namespaces/%s/endpoints/", podNamespace); </ins> try { String token = getServiceAccountToken();

The namespace seems to be hard coded :
cc:
Thanks. We can improve it after v1 release.


Question: cachedCreds *credentials exp         time.Time <del> onRotate func() </del> <ins> onRotateList []func() </ins> } type credentials struct {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Data race in client-go with below sample code: Data race: What you expected to happen: No data race How to reproduce it (as minimally and precisely as possible): Using above given sample code Kubernetes version (use ):Cloud provider or hardware configuration:OS (e.g: ): OSX
/sig api-machinery
I'd like to take it. /assign
/remove-sig api-machinery /sig auth cli
Bug triage for release 1.17 here. Since this issue has been tagged for the milestone, we would like to check it's status. Code freeze is on Nov. 14. Will this issue be resolved before then?
Another reminder here that code freeze is in less than two weeks (Nov. 14). We will need to move this issue out of the milestone if isn't going to be resolved before then.


Question: #     sleep 4 #     i=$[$i+1] #   done <del> apiVersion: v1beta1 </del> <ins> apiVersion: v1beta3 </ins> kind: Pod <del> id: synthetic-logger-0.25lps-pod desiredState: manifest: version: v1beta1 id: synth-logger-0.25lps containers: - name: synth-lgr image: ubuntu:14.04 command: ["bash", "-c", "i="0"; while true; do echo -n "`hostname`: $i: "; date --rfc-3339 ns; sleep 4; i=$[$i+1]; done"] labels: name: synth-logging-source </del>  No newline at end of file <ins> metadata: labels: name: synth-logging-source name: synthetic-logger-0.25lps-pod spec: containers: - args: - bash - -c - 'i="0"; while true; do echo -n "`hostname`: $i: "; date --rfc-3339 ns; sleep 4; i=$[$i+1]; done' image: ubuntu:14.04 name: synth-lgr </ins>

Hi , I followed the instructions in examples/logging-demo, and there was no "external IP" field for elasticsearch-logging or kibana-logging, as shown below: I start up my cluster via cluster/kube- Did I miss something or is it a bug? Thanks.
Correct. This demo does not create an ELB so there is no external IP address. Access is via service proxy at the master. Type: and you should see the URLs for Elasticsearch and Kibana.
please let us know whether: a. Satnam's comment resolves your issue b. the demo should be updated to prevent this misunderstanding c. the issue isn't resolved Thank you!
Yes, Satnam's comment resolve the issue. I updated the demo in .
Awesome, thanks for sending the patch! We'll discuss there. Always fantastic to have docs updated regularly by folks trying them.


Question: type PredicateFailureError struct { PredicateName string <ins> PredicateDesc string </ins> } <del> func newPredicateFailureError(predicateName string) *PredicateFailureError { return &PredicateFailureError{PredicateName: predicateName} </del> <ins> func newPredicateFailureError(predicateName, predicateDesc string) *PredicateFailureError { return &PredicateFailureError{PredicateName: predicateName, PredicateDesc: predicateDesc} </ins> } func (e *PredicateFailureError) Error() string {

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind feature The is not user friendly. There has been to make it better, but we still print in the pending message. We should change these errors to a more human readable message. For example, should be changed to something like . /sig scheduling


Question: <del> { "kind": "ValidatingAdmissionPolicy", "apiVersion": "admissionregistration.k8s.io/v1alpha1", "metadata": { "name": "nameValue", "generateName": "generateNameValue", "namespace": "namespaceValue", "selfLink": "selfLinkValue", "uid": "uidValue", "resourceVersion": "resourceVersionValue", "generation": 7, "creationTimestamp": "2008-01-01T01:01:01Z", "deletionTimestamp": "2009-01-01T01:01:01Z", "deletionGracePeriodSeconds": 10, "labels": { "labelsKey": "labelsValue" }, "annotations": { "annotationsKey": "annotationsValue" }, "ownerReferences": [ { "apiVersion": "apiVersionValue", "kind": "kindValue", "name": "nameValue", "uid": "uidValue", "controller": true, "blockOwnerDeletion": true } ], "finalizers": [ "finalizersValue" ], "managedFields": [ { "manager": "managerValue", "operation": "operationValue", "apiVersion": "apiVersionValue", "time": "2004-01-01T01:01:01Z", "fieldsType": "fieldsTypeValue", "fieldsV1": {}, "subresource": "subresourceValue" } ] }, "spec": { "paramKind": { "apiVersion": "apiVersionValue", "kind": "kindValue" }, "matchConstraints": { "namespaceSelector": { "matchLabels": { "matchLabelsKey": "matchLabelsValue" }, "matchExpressions": [ { "key": "keyValue", "operator": "operatorValue", "values": [ "valuesValue" ] } ] }, "objectSelector": { "matchLabels": { "matchLabelsKey": "matchLabelsValue" }, "matchExpressions": [ { "key": "keyValue", "operator": "operatorValue", "values": [ "valuesValue" ] } ] }, "resourceRules": [ { "resourceNames": [ "resourceNamesValue" ], "operations": [ "operationsValue" ], "apiGroups": [ "apiGroupsValue" ], "apiVersions": [ "apiVersionsValue" ], "resources": [ "resourcesValue" ], "scope": "scopeValue" } ], "excludeResourceRules": [ { "resourceNames": [ "resourceNamesValue" ], "operations": [ "operationsValue" ], "apiGroups": [ "apiGroupsValue" ], "apiVersions": [ "apiVersionsValue" ], "resources": [ "resourcesValue" ], "scope": "scopeValue" } ], "matchPolicy": "matchPolicyValue" }, "validations": [ { "expression": "expressionValue", "message": "messageValue", "reason": "reasonValue", "messageExpression": "messageExpressionValue" } ], "failurePolicy": "failurePolicyValue", "auditAnnotations": [ { "key": "keyValue", "valueExpression": "valueExpressionValue" } ], "matchConditions": [ { "name": "nameValue", "expression": "expressionValue" } ], "variables": null }, "status": { "observedGeneration": 1, "typeChecking": { "expressionWarnings": [ { "fieldRef": "fieldRefValue", "warning": "warningValue" } ] }, "conditions": [ { "type": "typeValue", "status": "statusValue", "observedGeneration": 3, "lastTransitionTime": "2004-01-01T01:01:01Z", "reason": "reasonValue", "message": "messageValue" } ] } } </del>  No newline at end of file

spec.variables should be marked omitempty. I'm trying to create a validating admission policy that uses in v1.27 as follows: I'm setting : to "Strict" in CreateOptions so that the server will reject requests with unknown fields but it complains that the don't exist in v1.27 and it does make sense because this field was introduced in v1.28: I'm using a 1.28 client.
/sig api-machinery
/assign
/assign /triage accepted
Thanks for raising this! /cc


Question: awsServices := newMockedFakeAWSServices(TestClusterId) c, _ := newAWSCloud(strings.NewReader("[global]"), awsServices) <del> sg1 := "sg-000001" sg2 := "sg-000002" </del> <ins> sg1 := map[string]string{ServiceAnnotationLoadBalancerExtraSecurityGroups: "sg-000001"} sg2 := map[string]string{ServiceAnnotationLoadBalancerExtraSecurityGroups: "sg-000002"} sg3 := map[string]string{ServiceAnnotationLoadBalancerExtraSecurityGroups: "sg-000001, sg-000002"} </ins> tests := []struct { name string <del> extraSGsAnnotation string expectedSGs        []string </del> <ins> annotations map[string]string expectedSGs []string </ins> }{ <del> {"No extra SG annotation", "", []string{}}, {"Empty extra SGs specified", ", ,,", []string{}}, {"SG specified", sg1, []string{sg1}}, {"Multiple SGs specified", fmt.Sprintf("%s, %s", sg1, sg2), []string{sg1, sg2}}, </del> <ins> {"No extra SG annotation", map[string]string{}, []string{}}, {"Empty extra SGs specified", map[string]string{ServiceAnnotationLoadBalancerExtraSecurityGroups: ", ,,"}, []string{}}, {"SG specified", sg1, []string{sg1[ServiceAnnotationLoadBalancerExtraSecurityGroups]}}, {"Multiple SGs specified", sg3, []string{sg1[ServiceAnnotationLoadBalancerExtraSecurityGroups], sg2[ServiceAnnotationLoadBalancerExtraSecurityGroups]}}, </ins> } awsServices.ec2.(*MockedFakeEC2).expectDescribeSecurityGroups(TestClusterId, "k8s-elb-aid", "cluster.test")

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind feature /sig aws What happened: In k8s 1.7, when creating a service of type LoadBalancer, K8S creates the ELB and a Security Group for it. For these resources, it adds a tag, constructed dynamically, containing the cluster id: "${cluster_id}" with the value "true" What you expected to happen: We'd like to have also a statically named tag , as it was in 1.5 ("KubernetesCluster") with the value of the cluster id. How to reproduce it (as minimally and precisely as possible): Running on AWS cloud provider, create a service of type LoadBalancer. Anything else we need to know?: The reason for what we expect is that we have tight security constraints that require us to fine tune the IAM policies for the Instance profiles that the controllers use. With a statically named tag we were able, in 1.5, to restrict the permissions in the IAM policy to act only on the k8s managed resources (ELBs, security groups). Here is the relevant part of the IAM policy: With a dynamically named tag we are not able to do this any more as AWS IAM doesn't support something like: We are sharing the Instance Profiles among different k8s clusters, hence the "" in the value of the tag. Environment: Kubernetes version (use ):Cloud provider or hardware configuration*: AWS OS (e.g. from /etc/os-release):Kernel (e.g. ):Install tools: kube-aws Others:
Reiterating the mentions to trigger a notification: <detailsIn response to : <!-- This form is for bug reports and feature requests ONLY! -- /sig aws In k8s 1.7, when creating a service of type LoadBalancer, K8S creates the ELB and a Security Group for it. For these resources, it adds a tag, constructed dynamically, containing the cluster id: "${cluster_id}" with the value "true" We'd like to have also a statically named tag , as it was in 1.5 ("KubernetesCluster") with the value of the cluster id. Running on AWS cloud provider, create a service of type LoadBalancer.


Question: awsServices := newMockedFakeAWSServices(TestClusterId) c, _ := newAWSCloud(strings.NewReader("[global]"), awsServices) <del> sg1 := "sg-000001" sg2 := "sg-000002" </del> <ins> sg1 := map[string]string{ServiceAnnotationLoadBalancerExtraSecurityGroups: "sg-000001"} sg2 := map[string]string{ServiceAnnotationLoadBalancerExtraSecurityGroups: "sg-000002"} sg3 := map[string]string{ServiceAnnotationLoadBalancerExtraSecurityGroups: "sg-000001, sg-000002"} </ins> tests := []struct { name string <del> extraSGsAnnotation string expectedSGs        []string </del> <ins> annotations map[string]string expectedSGs []string </ins> }{ <del> {"No extra SG annotation", "", []string{}}, {"Empty extra SGs specified", ", ,,", []string{}}, {"SG specified", sg1, []string{sg1}}, {"Multiple SGs specified", fmt.Sprintf("%s, %s", sg1, sg2), []string{sg1, sg2}}, </del> <ins> {"No extra SG annotation", map[string]string{}, []string{}}, {"Empty extra SGs specified", map[string]string{ServiceAnnotationLoadBalancerExtraSecurityGroups: ", ,,"}, []string{}}, {"SG specified", sg1, []string{sg1[ServiceAnnotationLoadBalancerExtraSecurityGroups]}}, {"Multiple SGs specified", sg3, []string{sg1[ServiceAnnotationLoadBalancerExtraSecurityGroups], sg2[ServiceAnnotationLoadBalancerExtraSecurityGroups]}}, </ins> } awsServices.ec2.(*MockedFakeEC2).expectDescribeSecurityGroups(TestClusterId, "k8s-elb-aid", "cluster.test")

The reason for what we expect is that we have tight security constraints that require us to fine tune the IAM policies for the Instance profiles that the controllers use. With a statically named tag we were able, in 1.5, to restrict the permissions in the IAM policy to act only on the k8s managed resources (ELBs, security groups). Here is the relevant part of the IAM policy: Environment: Kubernetes version (use ):OS (e.g. from /etc/os-release):Others: </details
I believe they are moving away from the tag since there are many resources that can be shared by multiple clusters. could you use an annotation?
We removed the condition in roles in the end. Thank you for your suggestion.
That annotation affects the ELB but not any security groups that are created. can you please reopen this? You reported a real issue, and even if it no longer affects you, it needs to be tracked and fixed.
Indeed, this might be a feature that someone else would need.
In fact, this is a feature I need!
Food for thought: should we propagate the tags on the ELB to the SG, or should we introduce an annotation for the SG specifically?


Question: }) } } <ins> func TestConvertToAPIContainerStatusesDataRace(t *testing.T) { pod := podWithUIDNameNs("12345", "test-pod", "test-namespace") testTimestamp := time.Unix(123456789, 987654321) criStatus := &kubecontainer.PodStatus{ ID:        pod.UID, Name:      pod.Name, Namespace: pod.Namespace, ContainerStatuses: []*kubecontainer.Status{ {Name: "containerA", CreatedAt: testTimestamp}, {Name: "containerB", CreatedAt: testTimestamp.Add(1)}, }, } testKubelet := newTestKubelet(t, false) defer testKubelet.Cleanup() kl := testKubelet.kubelet // convertToAPIContainerStatuses is purely transformative and shouldn't alter the state of the kubelet // as there are no synchronisation events in that function (no locks, no channels, ...) each test routine // should have its own vector clock increased independently. Golang race detector uses pure happens-before // detection, so would catch a race condition consistently, despite only spawning 2 goroutines for i := 0; i < 2; i++ { go func() { kl.convertToAPIContainerStatuses(pod, criStatus, []v1.ContainerStatus{}, []v1.Container{}, false, false) }() } } </ins>

What happened: One pod stuck at CreateContainerError and can't get recovered automatically. Kubelet log shows that the requested name (let's say Container2) is already in use. shows the Container2 Running, and it's created before Container1 This should never happen. How to reproduce it (as minimally and precisely as possible): I wish i knew. Anything else we need to know?: I know this can be caused by time shifting, but I didn't find any time shifting sign in system log and chrony log, and there is no rebooting between the create timestamp of Container1 and Container2, so this should be caused by something else. This happens intermittently in our env, like 3 times in half a year, and I didn't find any ime shifting sign each time. Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.6", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-11-13T11:20:18Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.6", GitCommit:"", GitTreeState:"clean", BuildDate:"2020-08-28T11:18:26Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: OS (e.g: ): CentOS Linux release 7.5.1804 (Core) Kernel (e.g. ): Linux node-1 3.10.0-957.21.3.el7.x8664 SMP Wed Jul 17 06:57:50 UTC 2019 x8664 x8664 x86_64 GNU/Linux Install tools: kubeadm Network plugin and version (if this is a network-related bug): Others:
/sig node
Any update on this issue, we are also facing the same issue. EKS version : 1.14.9 Docker version: 19.03.6-ce AWS CNI: 1.6.2 Intermittently we are also facing this , what might the issue, could please guide us here? Below are the kubelet logs. )"


Question: }) } } <ins> func TestConvertToAPIContainerStatusesDataRace(t *testing.T) { pod := podWithUIDNameNs("12345", "test-pod", "test-namespace") testTimestamp := time.Unix(123456789, 987654321) criStatus := &kubecontainer.PodStatus{ ID:        pod.UID, Name:      pod.Name, Namespace: pod.Namespace, ContainerStatuses: []*kubecontainer.Status{ {Name: "containerA", CreatedAt: testTimestamp}, {Name: "containerB", CreatedAt: testTimestamp.Add(1)}, }, } testKubelet := newTestKubelet(t, false) defer testKubelet.Cleanup() kl := testKubelet.kubelet // convertToAPIContainerStatuses is purely transformative and shouldn't alter the state of the kubelet // as there are no synchronisation events in that function (no locks, no channels, ...) each test routine // should have its own vector clock increased independently. Golang race detector uses pure happens-before // detection, so would catch a race condition consistently, despite only spawning 2 goroutines for i := 0; i < 2; i++ { go func() { kl.convertToAPIContainerStatuses(pod, criStatus, []v1.ContainerStatus{}, []v1.Container{}, false, false) }() } } </ins>

with CreatePodSandboxError: "CreatePodSandbox for pod "-f0fe-11ea-9872-)" failed: rpc error: code = Unknown desc = failed to create a sandbox for pod "aiqx-server-dep--b9lmv": operation timeout: context deadline exceeded" Sep 07 11:40:40 kubelet[2899]: W0907 11:40:40. 2899 ] No ref for container "docker://7b9c86b529530f15d941f8e68973a202c8f348ab5db864a8371e8c82ddbf9dc4" (r(-e845-11ea-bd96-) Sep 07 11:40:43 kubelet[2899]: W0907 11:40:43. 2899 ] Unable to create pod sandbox due to conflict. Attempting to remove sandbox "a73bd9a03d56bc7855e4e89e163e13c5e7173397f10edcc200fb686e1b6885c8" Sep 07 11:40:43 kubelet[2899]: E0907 11:40:43. 2899 ] Failed to remove the conflicting container "a73bd9a03d56bc7855e4e89e163e13c5e7173397f10edcc200fb686e1b6885c8": Error: No such container: a73bd9a03d56bc7855e4e89e163e13c5e7173397f10edcc200fb686e1b6885c8
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or .


Question: }) } } <ins> func TestConvertToAPIContainerStatusesDataRace(t *testing.T) { pod := podWithUIDNameNs("12345", "test-pod", "test-namespace") testTimestamp := time.Unix(123456789, 987654321) criStatus := &kubecontainer.PodStatus{ ID:        pod.UID, Name:      pod.Name, Namespace: pod.Namespace, ContainerStatuses: []*kubecontainer.Status{ {Name: "containerA", CreatedAt: testTimestamp}, {Name: "containerB", CreatedAt: testTimestamp.Add(1)}, }, } testKubelet := newTestKubelet(t, false) defer testKubelet.Cleanup() kl := testKubelet.kubelet // convertToAPIContainerStatuses is purely transformative and shouldn't alter the state of the kubelet // as there are no synchronisation events in that function (no locks, no channels, ...) each test routine // should have its own vector clock increased independently. Golang race detector uses pure happens-before // detection, so would catch a race condition consistently, despite only spawning 2 goroutines for i := 0; i < 2; i++ { go func() { kl.convertToAPIContainerStatuses(pod, criStatus, []v1.ContainerStatus{}, []v1.Container{}, false, false) }() } } </ins>

/lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-contributor-experience at . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: mountPath: /device-plugin - name: dev mountPath: /dev <ins> updateStrategy: type: RollingUpdate </ins>

This issue is here to satisfy PR requirements of issue association. A lot of times we need to do version bumps for minor bug fixes that don't deserve a separate issue in the kubernetes/kubernetes repo. /kind bug /sig node
/lifecycle frozen
all the linked PRs are emrged and no activity since more than 3 years /close
Closing this issue. <detailsIn response to : all the linked PRs are emrged and no activity since more than 3 years /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: { "ImportPath": "github.com/GoogleCloudPlatform/kubernetes", <del> "GoVersion": "go1.3", </del> <ins> "GoVersion": "go1.4.1", </ins> "Packages": [ "./..." ],

committed a changed version of go into godeps. Was this deliberate? I don't see any discussion of it ( I've almost done this by mistake as well, so I'm inclined to think it was just an oversight. (I think this is giving me errors building, but I'm not yet 100% sure this is the cause)
Hmm, this is a good point. It does look to be accidental. I don't know. I think and would have context though I believe they are out— do you know offhand whether this is probably an accident? I guess were talking specifically about in the .
I believe it's an accident.
Thanks, The linked PR should fix this (assuming I don't have to do anything else with Godeps).
This was almost certainly an accident. Whenever I've updated godeps it's changed automatically to 1.3 and I've had to adjust it back manually.
Actually, jeff lowdermilk was super helpful and showed me how to update the deps on this PR. it was his commit which I merged into my PR, so not sure it it was accidental or not. Fortunately, now that k8petstore is in the go build, we will know if it breaks anything when you revert the version back. Thanks for noticing this. , Alex Robinson wrote:


Question: import ( "context" <ins> "fmt" </ins> "time" v1 "k8s.io/api/core/v1"

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: aks-engine-azure-windows-master-containerd-serial-slow Which test(s) are failing: [sig-windows] [Feature:Windows] Kubelet-Stats [Serial] Kubelet stats collection for Windows nodes when running 10 pods should return within 10 seconds Since when has it been failing: Unknown. Testgrid link: Reason for failure: The test spawns 10 pods with the same pod name, which contains multiple containers with the same container name. Because of this, the test fails. Logs: Anything else we need to know:
/sig windows These SIGs are my best guesses for this issue. Please comment if I am incorrect about one. I am a bot run by vllry.


Question: func (m *manager) UpdatePodStatus(podUID types.UID, podStatus *v1.PodStatus) { for i, c := range podStatus.ContainerStatuses { <del> var ready bool if c.State.Running == nil { ready = false } else if result, ok := m.readinessManager.Get(kubecontainer.ParseContainerID(c.ContainerID)); ok { ready = result == results.Success } else { // The check whether there is a probe which hasn't run yet. _, exists := m.getWorker(podUID, c.Name, readiness) ready = !exists } podStatus.ContainerStatuses[i].Ready = ready </del> var started bool if c.State.Running == nil { started = false

If I have probe and then all good. Previous version of the pod is terminated after startup probe success, but if wasn't specified the pod becomes ready immediately. It would be nice to add new state (for example starting) which would have pods until a is successful. Also, it would mean that container not ready for connection. Thank you for attention!
/sig cluster-lifecycle
/remove-sig cluster-lifecycle /sig apps try asking in #kubernetes-users on k8s slack.
they are silent in the chat. Who is involved in the development of startup probe?
i think this topic better fits SIG Node. try asking in their channel on slack , but be patient ;) /sig node /remove-sig apps
Those labels are not set on the issue: <detailsIn response to : i think this topic better fits SIG Node. /sig node /remove-sig apps </details
I think this is a feature request, not support, so I changed labels. On its face it SEEMS reasonable, but I am not close enough to the details to say. This is definitely sig-node.
Adding a new pod lifecycle state has been determined to be intractable in the past. Is the proposed change keeping the pod in !Ready until the first success from ?
Where do you usually discuss this? Just wondering. Yes, I want that pod considered not ready until will not be successful. If readiness probe doesn't present then a pod can become ready. How can a pod be ready if it is not running?
how long wait?
Looking at this issue, I can confirm it, at least superficially - a pod with a perma-failing is "ready" if there is no readinessGate defined and "not ready" if it is defined. The question I have is whether we think that is "correct" or not? Initially I thought it was correct, but the more I think about it the less convinced I am. If I have defined a startupProbe and it is not passing, my app should not be ready. Does anyone disagree?
Also, the app should not be ready in status. It's funny, but without a readiness probe, pods in the final state have the status ready and accept requests. This is unexpected.
Whether a pod is ready while terminating is an entirely different question, IMO.


Question: func (m *manager) UpdatePodStatus(podUID types.UID, podStatus *v1.PodStatus) { for i, c := range podStatus.ContainerStatuses { <del> var ready bool if c.State.Running == nil { ready = false } else if result, ok := m.readinessManager.Get(kubecontainer.ParseContainerID(c.ContainerID)); ok { ready = result == results.Success } else { // The check whether there is a probe which hasn't run yet. _, exists := m.getWorker(podUID, c.Name, readiness) ready = !exists } podStatus.ContainerStatuses[i].Ready = ready </del> var started bool if c.State.Running == nil { started = false

Apps that server network traffic probably need to "drain" and to do that they need to stay "ready" for a while as upstream LBs de-program. Let's not conflate issues, please.
The fix seems simple - lets see what CI says.
Sorry, I don't spend much time on slack... you could have checked the release notes in 1.18 ;-) Tim's fix seems good, thanks!
/reopen Until we merge all related PRs...
Reopened this issue. <detailsIn response to : /reopen Until we merge all related PRs... Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
All merged. /close
Closing this issue. <detailsIn response to : All merged. /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: t.Errorf("Invalid content read from cache %q", string(content)) } } <ins> func TestCacheRoundTripperPathPerm(t *testing.T) { assert := assert.New(t) rt := &testRoundTripper{} cacheDir, err := ioutil.TempDir("", "cache-rt") os.RemoveAll(cacheDir) defer os.RemoveAll(cacheDir) if err != nil { t.Fatal(err) } cache := newCacheRoundTripper(cacheDir, rt) // First call, caches the response req := &http.Request{ Method: http.MethodGet, URL:    &url.URL{Host: "localhost"}, } rt.Response = &http.Response{ Header:     http.Header{"ETag": []string{`"123456"`}}, Body:       ioutil.NopCloser(bytes.NewReader([]byte("Content"))), StatusCode: http.StatusOK, } resp, err := cache.RoundTrip(req) if err != nil { t.Fatal(err) } content, err := ioutil.ReadAll(resp.Body) if err != nil { t.Fatal(err) } if string(content) != "Content" { t.Errorf(`Expected Body to be "Content", got %q`, string(content)) } err = filepath.Walk(cacheDir, func(path string, info os.FileInfo, err error) error { if err != nil { return err } if info.IsDir() { assert.Equal(os.FileMode(0750), info.Mode().Perm()) } else { assert.Equal(os.FileMode(0660), info.Mode().Perm()) } return nil }) assert.NoError(err) } </ins>

In kubectl v1.8.0+, schema info is cached in the location specified by (defaulting to ), written with world-writeable permissions (rw-rw-rw-). If is specified and pointed at a different location accessible to other users/groups, the written files may be modified by other users/groups and disrupt the kubectl invocation. CVSS score: CVSS:3.0/AV:L/AC:H/PR:L/UI:R/S:U/C:L/I:L/A:N (3.3, low) What versions are affected? kubectl v1.8.0+ What configurations are affected? Invocations that point at world-writeable locations Impact Malformed responses written to the cache directory can disrupt the kubectl invocation Workaround Use the default location in the $HOME directory or point it at a directory that is only accessible to desired users/groups. (original description follows) ==== What happened: The files inside of ".kube/http-cache" are world writeable (rw-rw-rw-). While the default for these files appears to be the home directory, using the "--cache-dir" flag could put these files into a place where world writeable files would allow any user / process to modify the cache files. Modification of the cache files could influence the kubectl utility in a negative way for other users. What you expected to happen: Apply stricter file permissions to the http-cache files. How to reproduce it (as minimally and precisely as possible): Run any generic kubectl command which is successful and then list the cache directory $ kubectl get pods --all-namespaces $ ls -la Anything else we need to know?


Question: t.Errorf("Invalid content read from cache %q", string(content)) } } <ins> func TestCacheRoundTripperPathPerm(t *testing.T) { assert := assert.New(t) rt := &testRoundTripper{} cacheDir, err := ioutil.TempDir("", "cache-rt") os.RemoveAll(cacheDir) defer os.RemoveAll(cacheDir) if err != nil { t.Fatal(err) } cache := newCacheRoundTripper(cacheDir, rt) // First call, caches the response req := &http.Request{ Method: http.MethodGet, URL:    &url.URL{Host: "localhost"}, } rt.Response = &http.Response{ Header:     http.Header{"ETag": []string{`"123456"`}}, Body:       ioutil.NopCloser(bytes.NewReader([]byte("Content"))), StatusCode: http.StatusOK, } resp, err := cache.RoundTrip(req) if err != nil { t.Fatal(err) } content, err := ioutil.ReadAll(resp.Body) if err != nil { t.Fatal(err) } if string(content) != "Content" { t.Errorf(`Expected Body to be "Content", got %q`, string(content)) } err = filepath.Walk(cacheDir, func(path string, info os.FileInfo, err error) error { if err != nil { return err } if info.IsDir() { assert.Equal(os.FileMode(0750), info.Mode().Perm()) } else { assert.Equal(os.FileMode(0660), info.Mode().Perm()) } return nil }) assert.NoError(err) } </ins>

: I estimate this is a low severity security issue with a CVSS score of "3.3 / CVSS:3.0/AV:L/AC:L/PR:L/UI:N/S:U/C:N/I:L/A:N" - Environment: Linux Kubernetes version (use kubectl version):Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.6", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-02-26T12:49:28Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.6", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-02-26T12:49:28Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: AWS. Running kube api server in hyperkube. OS (e.g: cat /etc/os-release): NAME="CentOS Linux" VERSION="7.1808 (Core)" ID="centos" IDLIKE="rhel fedora" VERSIONID="7" PRETTYNAME="CentOS Linux 7.1808 (Core)" ANSICOLOR="0;31" CPENAME="cpe:/o:centos:centos:7" HOMEURL="" BUGREPORTURL="" CENTOSMANTISBTPROJECT="CentOS-7" CENTOSMANTISBTPROJECTVERSION="7" REDHATSUPPORTPRODUCT="centos" REDHATSUPPORTPRODUCTVERSION="7" OSTREEVERSION=7.1808 Kernel (e.g. uname -a): Linux hackit.internal 3.10.0-862.11.6.el7.x8664 SMP Tue Aug 14 21:49:04 UTC 2018 x8664 x8664 x86_64 GNU/Linux Install tools: Manual installation. Others: n/a
/sig product-security
Closing this issue. <detailsIn response to : if you think this is serious, send an email thanks!


Question: t.Errorf("Invalid content read from cache %q", string(content)) } } <ins> func TestCacheRoundTripperPathPerm(t *testing.T) { assert := assert.New(t) rt := &testRoundTripper{} cacheDir, err := ioutil.TempDir("", "cache-rt") os.RemoveAll(cacheDir) defer os.RemoveAll(cacheDir) if err != nil { t.Fatal(err) } cache := newCacheRoundTripper(cacheDir, rt) // First call, caches the response req := &http.Request{ Method: http.MethodGet, URL:    &url.URL{Host: "localhost"}, } rt.Response = &http.Response{ Header:     http.Header{"ETag": []string{`"123456"`}}, Body:       ioutil.NopCloser(bytes.NewReader([]byte("Content"))), StatusCode: http.StatusOK, } resp, err := cache.RoundTrip(req) if err != nil { t.Fatal(err) } content, err := ioutil.ReadAll(resp.Body) if err != nil { t.Fatal(err) } if string(content) != "Content" { t.Errorf(`Expected Body to be "Content", got %q`, string(content)) } err = filepath.Walk(cacheDir, func(path string, info os.FileInfo, err error) error { if err != nil { return err } if info.IsDir() { assert.Equal(os.FileMode(0750), info.Mode().Perm()) } else { assert.Equal(os.FileMode(0660), info.Mode().Perm()) } return nil }) assert.NoError(err) } </ins>

</details
Adjusted the CVSS score to to indicate user interaction is required (since they must specify a non-default cache option pointed at a shared writeable location)
There are two places this should be done: PathPerm and FilePerm should be set here: the temp file and final file should have stricter permissions (probably 0660) here:
/cc
/help
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/assign I would like to work on release-1.9, as I just made a hot fix for our team's k8s version of 1.9.
GitHub didn't allow me to assign the following users: yuchengwu. Note that only and repo collaborators can be assigned and that issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign I would like to work on release-1.9, as I just made a hot fix for our team's k8s version of 1.9. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
help fixing this is welcome, though it would only be fixed in releases 1.12+, per
/label official-cve-feed (Related to kubernetes/sig-security)


Question: }) } } <ins> func TestSubnet(t *testing.T) { for i, c := range []struct { desc     string service  *v1.Service expected *string }{ { desc:     "No annotation should return nil", service:  &v1.Service{}, expected: nil, }, { desc: "annotation with subnet but no ILB should return nil", service: &v1.Service{ ObjectMeta: metav1.ObjectMeta{ Annotations: map[string]string{ ServiceAnnotationLoadBalancerInternalSubnet: "subnet", }, }, }, expected: nil, }, { desc: "annotation with subnet but ILB=false should return nil", service: &v1.Service{ ObjectMeta: metav1.ObjectMeta{ Annotations: map[string]string{ ServiceAnnotationLoadBalancerInternalSubnet: "subnet", ServiceAnnotationLoadBalancerInternal:       "false", }, }, }, expected: nil, }, { desc: "annotation with empty subnet should return nil", service: &v1.Service{ ObjectMeta: metav1.ObjectMeta{ Annotations: map[string]string{ ServiceAnnotationLoadBalancerInternalSubnet: "", ServiceAnnotationLoadBalancerInternal:       "true", }, }, }, expected: nil, }, { desc: "annotation with subnet and ILB should return subnet", service: &v1.Service{ ObjectMeta: metav1.ObjectMeta{ Annotations: map[string]string{ ServiceAnnotationLoadBalancerInternalSubnet: "subnet", ServiceAnnotationLoadBalancerInternal:       "true", }, }, }, expected: to.StringPtr("subnet"), }, } { real := subnet(c.service) assert.Equal(t, c.expected, real, fmt.Sprintf("TestCase[%d]: %s", i, c.desc)) } } </ins>

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: When service is set with ILB annotation and empty string subnet, e.g. The service would be pending forever with following error: What you expected to happen: ILB should be provisioned successfully. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Others: /sig azure /kind bug /assign
/milestone v1.14 We're entering and this looks relevant to the 1.14 release. Please if I am incorrect
Thanks


Question: "fmt" "os" "path/filepath" <ins> "strings" </ins> "time" "github.com/pkg/errors"

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --When attempting to upgrade kubernetes using kubeadm upgrade apply, kubeadm fails to parse an etcd imagetag containing build metadata in the form of as a valid semver. An example upgrade produces an error similar to: Note this will occur even if the etcd version is unchanged. The +` character. This pattern is used to support custom kubernetes images, for instance kube-apiserver:v1.20.4+patch.0 will be converted to imagetag v1.20.4_patch.0 when running make quick-release-images. Add the following to kubeadm- Install kubernetes. Then upgrade kubernetes with kubeadm upgrade apply. I have tested a fix for this issue and will PR shortly. Kubernetes version (use ): N/A Cloud provider or hardware configuration: N/A OS (e.g: ): N/A Kernel (e.g. ): N/A Install tools: kubeadm Network plugin and version (if this is a network-related bug): N/A Others: N/A
/sig cluster-lifecycle
/area kubeadm
/triage accepted


Question: } // CleanupAdapter deletes Custom Metrics - Stackdriver adapter deployments. <del> func CleanupAdapter(namespace, adapterDeploymentFile string) { stat, err := framework.RunKubectl(namespace, "delete", "-f", adapterDeploymentFile) </del> <ins> func CleanupAdapter(adapterDeploymentFile string) { stat, err := framework.RunKubectl("", "delete", "-f", adapterDeploymentFile) </ins> framework.Logf(stat) if err != nil { framework.Logf("Failed to delete adapter deployments: %s", err)

/kind failing-test <!-- If this is a flake, please add: /kind flake --/sig autoscaling Looks like it just started happening today (@ 2020-09-15 03:54:19 +0000 UTC) !
cc the namespace from the provided object "kube-system" does not match the namespace "horizontal-pod-autoscaling-830". You must pass '--namespace=kube-system' to perform this operation. the namespace from the provided object "custom-metrics" does not match the namespace "horizontal-pod-autoscaling-830". You must pass '--namespace=custom-metrics' to perform this operation. the namespace from the provided object "custom-metrics" does not match the namespace "horizontal-pod-autoscaling-830". You must pass '--namespace=custom-metrics' to perform this operation.
related fix:
No I don't think that's the fix for this (that is just a little cleanup). Actually I think we have to delete namespace parameter in line : Because before , the will have no actually effect. But now with the change of , it will have namespace : at the end of the command. But the namespace in yaml file is which is different from . We should not have parameter when applying from file because this test didn't have before, right? Related PR:
/assign


Question: internalapi "k8s.io/cri-api/pkg/apis" apitest "k8s.io/cri-api/pkg/apis/testing" fakeremote "k8s.io/kubernetes/pkg/kubelet/cri/remote/fake" <ins> "k8s.io/kubernetes/pkg/kubelet/cri/remote/util" </ins> ) const (

Which jobs are flaking: periodics-kubernetes-bazel-test Which test(s) are flaking: //pkg/kubelet/cri/remote TestVersion Testgrid link: Reason for failure: Error: Received unexpected error: rpc error: code = Unavailable desc = transport is closing panic: runtime error: invalid memory address or nil pointer dereference [recovered] panic: runtime error: invalid memory address or nil pointer dereference Anything else we need to know: To reproduce with stress: go test -race -c ./pkg/kubelet/cri/remote stress - TestVersion Detailed logs
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig testing
I 'll give it a see and fix it :)
/assign
/remove-sig testing SIG node owns their unit tests. SIG testing owns the framework, infrastructure, etc.
/sig node


Question: * [Example of dynamic updates](examples/update-demo/README.md) * [Cluster monitoring with heapster and cAdvisor](https://github.com/GoogleCloudPlatform/heapster) * [Community projects](https://github.com/GoogleCloudPlatform/kubernetes/wiki/Kubernetes-Community) <del> * [Development guide](docs/development.md) </del> <ins> * [Development guide](docs/devel/development.md) </ins> Or fork and start hacking!

Right now, we have a mixture of user documentation and developer documentation in the top-level directory. We should make it easier to find a just user documentation. Step 1 would be to move developer documentation into a subdir like docs/development or something like that. User docs would stay in the top-level docs dir.
move to


Question: "description": "EndpointPort is a tuple that describes a single port.", "properties": { "appProtocol": { <del> "description": "The application protocol for this port. This field follows standard Kubernetes label syntax. Un-prefixed names are reserved for IANA standard service names (as per RFC-6335 and http://www.iana.org/assignments/service-names). Non-standard protocols should use prefixed names such as mycompany.com/my-custom-protocol. This is a beta field that is guarded by the ServiceAppProtocol feature gate and enabled by default.", </del> <ins> "description": "The application protocol for this port. This field follows standard Kubernetes label syntax. Un-prefixed names are reserved for IANA standard service names (as per RFC-6335 and https://www.iana.org/assignments/service-names). Non-standard protocols should use prefixed names such as mycompany.com/my-custom-protocol. This is a beta field that is guarded by the ServiceAppProtocol feature gate and enabled by default.", </ins> "type": "string" }, "name": {

We should make the change in PR API reference links to should use HTTPS, not HTTP Visit h and look at the IANA hyperlinks Once this fix is made, a PR to is needed to update the docs (by autogenerating them). This typically would be deferred until the next Kubernetes release. v1.23 not applicable not applicable not applicable not applicable not applicable
/sig docs
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I opened this issue so that someone else can open a new PR that is equivalent to .
/retitle Update links to IANA for Service Name and Transport Protocol Port Number Registry
I would like to work on this issue. May I assign it to myself?
Anyone - including you - is welcome to work on this issue! So long as you've you can open a pull request to propose a fix.
Thanks
/assign
I have made the PR. I hope this works. If not then will be happy to do the necessary changes.
Had to close the previous PR because of issues, here is the new one I made.


Question: return feasibleNodes, nil } <del> type pluginScores struct { Plugin           string Score            int64 AverageNodeScore float64 } </del> // prioritizeNodes prioritizes the nodes by running the score plugins, // which return a score for each node from the call to RunScorePlugins(). // The scores from each plugin are added together to make the score for that node, then

Scheduler throughput drops significantly in http://perf- ! After , seems to be the curlspirit.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/priority critical-urgent /sig scheduling /sig scalability


Question: ) apiserverRequestConcurrencyLimit = compbasemetrics.NewGaugeVec( &compbasemetrics.GaugeOpts{ <del> Namespace:      namespace, Subsystem:      subsystem, Name:           "request_concurrency_limit", Help:           "Shared concurrency limit in the API Priority and Fairness subsystem", StabilityLevel: compbasemetrics.ALPHA, </del> <ins> Namespace: namespace, Subsystem: subsystem, Name:      "request_concurrency_limit", Help:      "Nominal number of execution seats configured for each priority level", // Remove this metric once all suppported releases have the equal nominal_limit_seats metric DeprecatedVersion: "1.30.0", StabilityLevel:    compbasemetrics.ALPHA, </ins> }, []string{priorityLevel}, )

In asked the following question. While the user-facing documentation answers this question clearly enough, John was reading the HELP string in the Prometheus scrapes. That thing only says the following. The metric HELP strings should be clear enough. Scrape or read the code (). No response <detailsThis problem started when we generalized to allow width other than 1 and got worse when we introduced borrowing. </details<detailsN/A </details<detailsN/A </details<details</details<details</details<details</details
/assign
Actually, this metric should be removed. It is a duplicate of but with an inferior name.
Looks good (Sorry I never got around to logging this issue myself. Have been super busy. And your wording is better anyway :-) )
/triage accepted Thank you.


Question: <ins> /* Copyright 2016 The Kubernetes Authors All rights reserved. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package v1 import ( "k8s.io/kubernetes/pkg/api/v1" "k8s.io/kubernetes/pkg/runtime" ) func addDeepCopyFuncs(scheme *runtime.Scheme) { if err := scheme.AddGeneratedDeepCopyFuncs( v1.DeepCopy_v1_DeleteOptions, v1.DeepCopy_v1_ExportOptions, v1.DeepCopy_v1_List, v1.DeepCopy_v1_ListOptions, v1.DeepCopy_v1_ObjectMeta, v1.DeepCopy_v1_ObjectReference, v1.DeepCopy_v1_OwnerReference, v1.DeepCopy_v1_Service, v1.DeepCopy_v1_ServiceList, v1.DeepCopy_v1_ServicePort, v1.DeepCopy_v1_ServiceSpec, v1.DeepCopy_v1_ServiceStatus, ); err != nil { // if one of the deep copy functions is malformed, detect it immediately. panic(err) } } </ins>

I started the federation-apiserver and went to and got the following output: Expected: I should have got an empty v1.ServiceList Any idea on what we are doing wrong? Have you guys seen this error?
I think your deep copiers or conversion functions are not getting registered properly. If you look at my change in , some places are inappropriately registering with api.Scheme instead of the passed in scheme. Maybe you have that problem or a similar one, since I think you guys copied some code. , Nikhil Jindal wrote:
Deep copy generation was slightly broken as well - you may want to double check that they are registered. On May 23, 2016, at 5:50 PM, Daniel Smith wrote: I think your deep copiers or conversion functions are not getting registered properly. If you look at my change in , some places are inappropriately registering with api.Scheme instead of the passed in scheme. Maybe you have that problem or a similar one, since I think you guys copied some code. , Nikhil Jindal wrote: — You are receiving this because you are on a team that was mentioned. Reply to this email directly or view it on GitHub
I got the same error with latest branch, as I was swamping on service controller, so I use a previous version and worked around it.
deep copies were indeed missing, but adding them hasnt fixed the issue. Will debug more. is my PR to add the deep copies cc Please feel free to file issues for any such problem you face. We need to fix all these issues. Its good if we know about them early. Thanks!
fixes this


Question: statuses = append(statuses, "Unknown") } row.Cells = append(row.Cells, obj.Name, strings.Join(statuses, ","), translateTimestamp(obj.CreationTimestamp)) <ins> if options.ShowLabels { row.Cells = append(row.Cells, labels.FormatLabels(obj.Labels)) } </ins> return []metav1alpha1.TableRow{row}, nil }

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: After labeling clusters in a Federation, when you type the field is empty. What you expected to happen: The fiels should contain information about the labels in the cluster How to reproduce it (as minimally and precisely as possible): a federation control pane in any Cloud Provider and any DNS provider your cluster: info aboyut the cluster with the flag: Note: if you describe it, you see the labels being : Environment: Kubernetes version (use ): 1.8.0 Cloud provider or hardware configuration**: anyone OS (e.g. from /etc/os-release): Ubuntu 16.04 Kernel (e.g. ): Linux Install tools: curl Others:
/sig multicluster
I'm working on it.


Question: echo -n Attempt "$(($attempt+1))" to check for salt-master local output local ok=1 <del> output=$(ssh -oStrictHostKeyChecking=no -i "${AWS_SSH_KEY}" ubuntu@${KUBE_MASTER_IP} pgrep salt-master 2> $LOG) || ok=0 </del> <ins> output=$(ssh -oStrictHostKeyChecking=no -i "${AWS_SSH_KEY}" ${SSH_USER}@${KUBE_MASTER_IP} pgrep salt-master 2> $LOG) || ok=0 </ins> if [[ ${ok} == 0 ]]; then if (( attempt > 30 )); then echo

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: I visited the Kubernetes 1.11 release blog post and clicked on the link to the article on resizing persistent volumes. The server returned a 404. What you expected to happen: I expected to navigate to the article on resizing persistent volumes in Kubernetes. How to reproduce it (as minimally and precisely as possible): Visit the release announcement blog post (), scroll down to the "4 Day Features Blog Series" section, and click on the link to the article on resizing persistent volumes. Anything else we need to know?: The link currently points to: The correct link is: Environment: Kubernetes version (use ): N/A Cloud provider or hardware configuration: N/A OS (e.g. from /etc/os-release): N/A Kernel (e.g. ): N/A Install tools: N/A Others: N/A
Reiterating the mentions to trigger a notification: <detailsIn response to : <!-- This form is for bug reports and feature requests ONLY! --/kind bug /kind feature What happened: I visited the Kubernetes 1.11 release blog post and clicked on the link to the article on resizing persistent volumes. The server returned a 404. I expected to navigate to the article on resizing persistent volumes in Kubernetes. Visit the release announcement blog post (), scroll down to the "4 Day Features Blog Series" section, and click on the link to the article on resizing persistent volumes. The link currently points to: The correct link is: Kubernetes version (use ): N/A Cloud provider or hardware configuration: N/A OS (e.g. from /etc/os-release): N/A Kernel (e.g. ): N/A Install tools: N/A Others: N/A Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
It should be reported to kubernetes/website, but doesn't matter.


Question: // Run begins watching and syncing. func (rm *ReplicationManager) Run(workers int, stopCh <-chan struct{}) { defer util.HandleCrash() <ins> glog.Infof("Starting RC Manager") controller.SyncAllPodsWithStore(rm.kubeClient, rm.podStore.Store) </ins> go rm.rcController.Run(stopCh) go rm.podController.Run(stopCh) for i := 0; i < workers; i++ {

The test is in flaky suite and no one seems to care. We need an owner for this test. cc Last failure log:
do you think you can take a look at this?
This is happening because of a bad assumption. The assumption is: if the reflector's store is consistent with etcd, the store given back to clients of the framework.Controller is too. I'd really like a general fix applicable to all controllers for this problem, but fixing it just for the RCs is pretty easy. In more detail: This function reports when the reflector has listed from apiserver. But the store given to the reflector is the fifo of the framework.Controller. The store given back to the caller is still not consistent, and will not be till the framework.Controller puts all the objects from the fifo into the client state store (). The rc specific fix: Before starting the rc manager, seed the expectations store with an expectation like {rc:foo, adds: foo.Status.Replicas, deletes:0}. This will wait for status.Replicas number of ADD events till a timeout. The general fix: Somehow expose this through framework.Controller.HasSynced (clients don't care about the fifo anyway, shouldn't it just be an implementation detail?) ideas? cc'ing people who have written controllers that might be interested I don't think this (and generally anything flaky for 1 release) is a p0, I can fix it if you don't mind dealing with the flakes till I've cleared things off my plate. I'd be more than happy to review a fix.
Well, it's a P0 because test is flaky. E.g last run:
- did you have chance to look into it?
The "DaemonRestart" tests are still labeled at . I will send a PR to promote them.
are you still planning to send aPR to promote the DaemonRestart tests out of flaky? I think this is the right thing to do.
Ack; sorry, I lost this in the muck.


Question: for _, kind := range []string{ "Service", } { <del> err = api.Scheme.AddFieldLabelConversionFunc("v1", kind, </del> <ins> err = scheme.AddFieldLabelConversionFunc("v1", kind, </ins> func(label, value string) (string, string, error) { switch label { case "metadata.namespace",