We merged to allow configurable volume limits for Cinder. But proposed adding such a limit to as well, among other things this will enable us to support node specific limits more easily - /sig openstack /sig storage /sig scheduling
/assign
cc
/kind feature
This will require updating in-tree Openstack cloudprovider code. are we onboard with that?
What would be the extent of the modification? It must not break current compatibility, and the in-tree provider is scheduled for removal this year. I'm ok with it as long as existing users aren't broken by it.
it should be a change into which would add a new configurable field. It would be backward compatible but would allow admins to configure different limits on a node if they must.
/milestone v1.14 We're entering and this looks relevant to the 1.14 release. Please if I am incorrect


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: When running e2e conformance tests against a cluster using a public HTTPS endpoint for the apiserver, you get the following errors: What you expected to happen: The API server endpoint is so the e2e tests should use and connect on port instead of . How to reproduce it (as minimally and precisely as possible): Run e2e conformance tests against a cluster with the apiserver exposed on a public endpoint with a public CA. Anything else we need to know?: When looking at the code here: It looks like the tlsConfig is not correctly derived from the URL. Diving into I see that this uses deep down: And this seem to return unless the kubeconfig has a CA or other explicit TLS config configured: I believe if this check also checked the of the URL, then it would be correct: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig testing


We've noticed a strange behavior of the scheduler in GCE 5K node scalability tests. There were periods of time, lasting about 6s, when scheduler didn't schedule any new pods. It's ruining our 5s pod-startup-latency SLO. See the example below, pod density-latency-pod-2325-dgqrv created on 09:08:59, scheduler didn't schedule it until 09:09:06. An interesting pattern can be found in the around that time. Before 09:09:00 scheduler schedules ~5 pods / s. Then it goes silent for about 6s and at 09:09:06 it outputs about 30 "bound successfully" statements and continues scheduling pods with ~5 pods / s pace. We checked api-server logs for that silent period of time and it turned out that scheduler was doing something, it was constantly sending GET and PUT request for a metrics-server and heapster pod it was trying to schedule. ! We dug deeper and found that scheduler was actually trying to schedule these two pods for over 12h, below is the plot of number of calls per minute that scheduler issued to api-server for these two pods: ! As we can see there are long periods of time when scheduler issues over 1500 requests per minute. That seems strangely a lot given that the requests are only for the two pods. During the whole run scheduler issued over 400K GETs and over 200K PUTs for these two pods. ! We figured out that scheduler wasn't able to schedule these two pods because there was not enough resources in the cluster. While it's understandable that scheduler wasn't able to schedule the pods, I'd expect scheduler to not spam the system with so many retries, which, as we can see in this example, can potentially block / slow-down scheduling of other pods. I've increased the scheduler log verbosity level in , we should have more logs from the tomorrow's run. Will post them here once available.
/sig scheduling
Reiterating the mentions to trigger a notification: <detailsIn response to : /sig scheduling </details
/assign for triage
What is the version of your cluster? If you are using the HEAD, we have a back-off mechanism to the scheduler to delay rescheduling unschedulable pods. It starts with 1 second delay and doubles it every time the pod is not schedulable up to 10 seconds.
It's from head, so apparently something doesn't work...
Yes, we're using head. The test run on Jan 21 and the cluster was built from head on the same day (commit to be more precise). As you can see in the logs it doesn't look like scheduler is using any back-off mechanism. Definitely not the one you described, it issues requests to api server much more often than 1 per second per pod. !
An update on more logs - we won't have more logs. We tried increasing scheduler's log level to v4 and it resulted in the log growing by few orders of magnitude and it was clogged with mostly useless stuff, e.g. it logged "Node gce-scale-cluster-minion-group-XXX is a potential node for preemption." over 250 000 000 times. We reverted the log level change in If more scheduler logs are needed to debug this we probably should add some new, more useful, ones.
cc/
/assign I can look into this next Tuesday
Friendly ping, any updates?
Whoops. Thanks for the ping. Log level v4 should not be needed. Although the log line you called out should probably get pushed into V9... First off, that's some really great debugging. Thanks for the detailed report. If the scheduler is getting into tight busy loops, that could in theory starve the scheduling thread, causing the observed 6 second pause in scheduling. The bug is in  A pod only gets marked for backoff if . And can increment very quickly in a high churn cluster. This would result in a pod being retried in a tight loop. One (unrelated) observation that stood out to me was this: This is being printed by the package and shows a trace with no steps logged to it, which means that we are hitting one of these four error cases in . Most likely, the PVC for those heapster & metrics-server pods are in a bad/unexpected state.
I think fixes the root problem here. If we eliminate the API server spam, the main scheduling thread hopefully will not get blocked for 6 second periods. Are you able to test with this patch?
Thanks Jonathan! I've reserved a scale-testing slot for Thursday (Feb 21) to test the fix. Will update this issue once I have the results.
I reproduced the issue (non-schedulable heapster pod due to not enough resources) on a 5K node cluster. Then I run the load test, first on the version without the fix and the on the version with the fix. Below are the graphs of the number of scheduler requests (for the unschedulable pod) per minute. Without fix: ! With fix: ! As we can see on the graphs, the fix definitely helps, without it there were minutes were scheduler was calling api server up to 2K times per minute (2K QPM). With the fix it didn't exceed 100 QPM. But even with the fix the average QPM is ~60 (per single pod) which is 1 query per second per pod on average. is average 60 QPM (1 QPS) reasonable here or should we expect something less from the rate-limiting mechanism?
By default the scheduler rate limit is 20 qps for all scheduler operations. Default backoff delay for unschedulable pods is 1 second and goes exponentially higher every time a pod is retried, but does not exceed 10 seconds. This 10 second max backoff time also means that if a pod is not retried for 10 seconds, for example because there are many other pods in front of it in the queue, its backoff time is reset to 1 second. So the next time it is retried and is found unschedulable it will be backed off for 1 second. I am basically trying to say that 1 qps is not unreasonable. We also an optimization () that should lower qps for unschedulable pods further down. This last PR got merged only a few hours ago. With that PR merged, I would expect to see even lower qps as the scheduler will not update pod status every time it retries scheduling a pod.
Thanks Bobby! That makes sense. In that case I believe we can close this issue once merges.
/milestone v1.14 We're entering and this looks relevant to the 1.14 release. Please if I am incorrect
Thanks for fixing!


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: test/e2enode/perf tests are referring to adm only images, this need to be replaced with multi-arch fat manifest images. ref:
/assign /sig node


Kubelet and System-reserve are implemented on Windows along with pod/container reserves which can help prevent overprovisioning, but we still need to test OOM behavior. We need a test case that will: kubelet & system reserve pods until node is full something that will take up system ram so that OOM is triggered /sig windows /milestone v1.14
I have started to look into this test case and have a few questions: What is the expected behavior we are testing for? What does "run something that will take up system ram" mean? Does that mean run a pod that uses up memory or run a system service that uses memory? After setting the reserves and as mentioned in we should be able to schedule pods until full and then once we trigger OOM (either by pod or system service) kubelet should continue running becuase of the reserves? Do the two settings need to be tested independently or they should be tested together? I found that seems to be doing something similar but it is in node-e2e. Is sig-windows running node-e2e tests currently?
/assign
GitHub didn't allow me to assign the following users: jsturtevant. Note that only and repo collaborators can be assigned and that issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
After researching this further and talking with I found that the does test the allocatable memory but uses CGroups to set reserves to test the reported vs actual. As windows does not have CGroups this test will not port as is. There is and the is not implemented so will not work either. The proposed testing solution would be two parts: kubelet & system reserve and verify allocatable is reporting properly. Schedule pods with known memory consumption values and verify the allocatable is correclty updated.Schedule until full and verify scheduling is not allowed. kubelet & system reserve and verify allocatable is reporting properly. Schedule Schedule until full and verify scheduling is not into machine and run process that consumes memory and verify kubelet and pods remain running. There is an open question on the second test as to whether kubelet and will be have properly. If it does not we will submit test 1 and open an issue to resolve 2.
I would suggest adding a third case that would test the behavior when excessive memory pressure is exerted on the node from the pods. This is perhaps a more common use case than 2 since the pod workloads are more likely to exhibit greater resource variability than node system processes. It would also carry over as a test for when/if node allocatable enforcement get implemented for windows nodes. Additionally, to complement this work would it be suitable to test windows pod memory limit enforcement? Or is this tested elsewhere?
Agreed this is a more common case. I am not sure that it is. Is this tested on linux somewhere?
I am seeing strange numbers reported when performing a test case. I scheduled a few pods to test memory limits and then schedule a pod with no memory limit and a process that will consume all of the available memory 500mb at a time: After the pod consumes all available memory, When I compare Node Allocatable/Node usage/Pod Usage I am not seeing the numbers add up properly: My node is staying online and kubelet is reporting no memory pressure. The total node memory consumption is over the Allocatable memory. What is the expected behavior and what should the test look like?
I believe node allocatable will always just be capacity minus reserves (system and kubelet) - So in this scenario it looks like you have and/or set? The memory % on the node is relative to the allocatable amount, and still seems to be under the total memory capacity - are you sure that all available memory has been consumed by your oom pod? The node should be tainted with memory-pressure when a hard or soft eviction threshold is met (). What do you have set for and ?
Thanks for the clarification. I currently have these settings: And on the machine I am seeing most memory being used: If I understand correctly, I start to see memory-pressure once I get into the setting above. In that case I should start to see pods get evicted, so for a valid test I could adjust the values to be easier to hit then test to see that pods get evicted properly which would be similar the . I am going to adjust a few of the settings and manually test that this is working then see what it will take to port the PriorityMemoryEviction, which currently uses cgroups which won't work on windows. This combined with testing that Memory Allocatable is being properly set should give us a good test coverage for windows on this scenario.
I performed manual testing and found that even when memory was used up on the machine the "Memory Pressure" notification was not fired. After looking deeper I found is not implemented for windows. The , which uses cgroups, provides the . This means even though is being it isn't used. On Windows, Node allocatable when used with limits and requests does prevent the node from over scheduling. I believe the the test and  (this only shows cpu and I could not find a test that demonstrated memory) demonstrate this works. But this doesn't stop a pod from consuming all the memory on the node and since eviction (hard/soft/node allocatable) is not implemented on windows this means a node can end up resource starved. I think the only valid test to be written for this scenario, as it stands now, is validating that Node Allocatable is being reported properly. Beyond that there is no functionality that can be validated around OOM behavior for the windows node.
Yeah, that sounds to be in agreement with what I've observed on the windows nodes that I run. To prevent resource starvation (as a workaround), would setting mem/cpu requests equal to mem/cpu limits for all windows pods work? I think the request / limit values get passed down into the docker daemon when it starts a container, so is it possible that docker could enforce the memory limit?
Yes, setting limits==requests would help the scheduler not over provision the nodes. I think the best way to approach this particular issue for 1.14 is to make sure docs call out what works/doesn't work and give guidance to on how to prevent resource starvation on nodes. Is there a place in the docs that would make sense for this information? The call outs would be: , , and flags do update Node Allocatable is not implemented and are not implemented Condition is not implemented in Windows There are no OOM actions taken by kubelet As a work around to previous, and should always be equal to prevent memory resource starvation as this allows the scheduler to to use limits and Kubelet running on the windows node does not have memory restrictions. and system-reserve does not set limits on kubelet or processes running the host. This means kubelet or a process on the host could cause memory resource starvation outside the and scheduler
thank you for the thorough investigation


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: We created a LoadBalancer service with type 'nlb' with two listeners forwarding traffic on two different ports. The service was created correctly along with the target groups and was functionning correctly. Around 20 hours later, no traffic was flowing from clients to any k8s nodes. While troubleshooting we discovered that the k8s node instance security group was lacking the input rules allowing traffic to the node ports for this LB. We deleted the service and recreated it. The service started working normally again, but we found out that the two target groups were missing their k8s tags. The logs contain the following error: What you expected to happen: We expected the NLB target groups to have k8s tags, so that k8s could remember it created them. How to reproduce it (as minimally and precisely as possible): Create a k8s service with NLB: Add a simple deployment that is matched by this service. Anything else we need to know?: Looking at AWS cloudtrail we could see the following error event: So the API request was called with two Resource ARNs, which is confirmed by the code: Since we have two target groups, there are two resources to tag. Except that the AWS API while allowing to send more than one resource ARNs doesn't allow it returning a validation error. This could be solved by doing one per target groups. Note that this can't happen on a Classic LB because there's no target groups. I believe that those missing tags on the target groups are the root cause of the deletion of the security group rules allowing traffic to reach the nodeports, with this (hypothetical and unproved) scenario: 8s creates the target groups, the NLB, the listeners, the sg input rules 8s fails to tag the target groups, but doesn't rollback anything a subsequent convergence loops, k8s notices that there are two k8s-managed Security Group input rules, but those don't match any Target Groups or match Target Groups that are not managed by k8s (because they don't have the owner tags) 8s deletes those rules Unfortunately we don't see the rule deletion in the controller manager logs, but it appears in Cloudtrail: Let me know if you have any questions, Environment: Kubernetes version: 1.10.7 Cloud provider or hardware configuration: AWS OS (e.g. from /etc/os-release): CoreOS 1967.3.0 Kernel (e.g. ): Linux 4.14.88-coreos Install tools: home grown Others:
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig aws /assign
Hi, the root cause of your problem should already be solved by It's cherry-picked into old versions and available as v1.11.7, v1.12.5 and v1.13.3(hasn't released yet)
, Thanks, you're most certainly right about the root cause of the issue being . It indeed looks exactly like what we've experienced. Still there's the problem of target groups not being tagged.


What happened: What you expected to happen: Human readable error output. Anything else we need to know?: The same is happening when in duplicate key while creating a with binaries. !!! The error output will be all the of the binary itself !!! Environment: Kubernetes version (use ):
/sig cli
/cc


What happened: We currently unregister the CSI driver if an error happens while registering it. Then, if another error happens, we return only the unregistration error. The problem is that the user will only see the unregistration error if they check the registrar logs. This gives the impression that it never tried to register the driver. What you expected to happen: The user should see the registration error instead. How to reproduce it (as minimally and precisely as possible): Try to install a CSI driver in a cluster where CSINodeInfo and/or CSIDriver CRDs are not created. Check registrar logs /kind bug /sig storage CC
/assign


Which jobs are failing: ci-kubernetes-e2e-kops-aws pull-kubernetes-e2e-kops-aws Which test(s) are failing: [sig-storage] In-tree Volumes [Driver: local][LocalVolumeType: block] [Testpattern: Pre-provisioned PV (xfs)] volumes should allow exec of files on the volume [sig-storage] In-tree Volumes [Driver: local][LocalVolumeType: block] [Testpattern: Pre-provisioned PV (xfs)] volumes should be mountable Since when has it been failing: Don't know.. as the job was down for a while due to Testgrid link: Reason for failure: To be found.. Anything else we need to know: This issue is a blocker for turning on the kops-aws presubmit, see - /cc /kind failing-test /sig storage
cc
By looking at the log this line is suspicious: Both testcases share similar error message. And that comes from when doing I guess its some mount client missing from the AMI so that the mount library can't find it.
Looks like the test needs to skip XFS on environments that do not support XFS
This is where the environment is checked to skip xfs:
Thanks Michelle! - Could you comment if we need to skip it in aws tests?
Looks like this was broken by - shall we revert? cc
Actually looking at the GCE tests for kops it was broken before . I'm wondering if we assumed xfs back in
/priority critical-urgent this is blocker for kops currently example of failing build
do you have idea could break this?
Yes, in , I tests for local volume which use . Because , PR is not blocked by gce e2e test. hi, Quick way to fix is to remove from supported types, I can submit a PR for this.
Alternatively, we can skip for environments which do not support XFS . What do you think?
I do not know what is the best solution in long term but in short term we should get kops e2e tests working. So if removing xfs from supported types is fastest solution - i would go with that. After that tests could be fixed how those should be. Could you do that PR and remove from supported types?
Hard-coding the knowledge which clusters or tests support XFS (because that's what it is - a dependency on a kernel feature, right?) in the test source code is just bound to fail eventually. What if the presence of xfs support depends on the "gci" version? This line doesn't cover that: I would prefer to mark tests which depend on XFS kernel support with or something like it (no idea how descriptive feature tags need to be) and then enable those test cases only in jobs where the target cluster is known to have XFS. If you extend that check as a short-term solution, then please file an issue about it and link to it next to the check. /cc
Agreeing with here... Isn't there a way to check if the kernel we test for/ against is supporting it more directly? Building a huge list of whats compatible and what's not sounds not perfect to me.
Long term we can enhance the check to ssh into a random node and see if the binary exists. Short term let's just remove xfs testing for local volumes.
If these two are newly tests, agree to skip (or remove) xfs testing to unblock PRs.
assuming all these in-tree tests will be less important over the course of the year as CSI moves forward... so should we assume that the XFS tests will never come back . ?
The local volume driver plan is to remain in-tree for the medium-term future. I think we should continue to add the environment check for xfs so we can add it back.


<!-- Please only use this template for submitting enhancement requests --What would you like to be : /sig testing /sig windows In the the PR we're adding a test case for pulling a Windows image from dockerhub (. However, there are no known public Windows images on , so the only option would be to upload it ourselves. Our proposal is to have it uploaded to the using Windows images). The proposed image manifest list to be uploaded is built from [1], which is currently hosted on dockerhub [2] and is currently used for testing Kubernetes with Windows nodes [3]. The manifest list contains the images: e2eteam/busybox:1.29-1803 (for Windows Server 1803) e2eteam/busybox:1.29-1809 (for Windows Server 2019) [1] [2] [3] Why is this needed: The Windows images will be used for the Windows equivalent test case for the following test:
/assign
/area conformance
It is my contention that we really only want one prod repo and that should be folded into the main repo. The authenticated repo should be for testing authenticated pulls and nothing else. To that end, I propose that we start with this. Does the image need to be named or tagged or something to indicate OS? How about * ? Is that too deep? Does it need Windows in the path or should we put it in the tag? Or can we have a multi-os manifest list? Also, is busybox the right image for testing, or can we do something even more minimal (to address the "what f there's a busybox CVE" question)?
Yes, (all images to date have used ), so I'd prefer that route over yet another name or tag. I'd also prefer something other than busybox if possible. I originally created because there were so many images in and it was hard to tell which of them were production images, which were test images, and which had been completely abandoned. Also, only a small number of Googlers could push to this repo since it contained all the production images. To address the first issue, I decided that all e2e test images would be sourced from the directory. To address the second issue, I created a new project with a wider ACL (though still currently limited to Googlers). All that said, given the automation we're building for the main GCR repo, it'd probably be fine to fold these images back in.
The image promoter means that a single repo should be BETTER managed than individual repos - we will have a PR trail of who put images there and why.
We should probably broaden the discussion to non-busybox Windows test images too. (related: ). Do we have a lower bar for test images? If so, having a separate repo (or making it clear that the images are test only) does have clear benefits.
Do we want a lower bar? I don't think we do.
Nobody wants a lower bar until they want to add/update the image quickly for a new test, or fix a test failure to unblock the release. If we do effectively hold the same bar today, this shouldn't be a problem.
i agree. also we don't need schema v2. multi-os, multi-arch image here. just (e.g.) a nanoserver image that satisfies a Windows 1803 and Windows 1809 host. a nanoserver image will obviously imply Windows only. for a test called "should be able to pull image from [WindowsOnly]": all we need is to upload this image to [x] and i guess we need to decide on if we want this test to be / passing for Windows Containers GA.
And indeed, a manifest list with nanoserver images could be an alternative (considering that the ) (we did try to find a command which works on both OSes, and ultimately found curl, but not all Linux images contain curl, check PR ).
an official nanoserver image is a more secure download. do we need to build a nanoserver manifest list or are there official images that we can use? i would like to bring a topic here related to pause. why are all the pause images for Windows that i found 1.1 GB or more? we definitely need the absolute minimal pause container uploaded in a location where users can download it and use that location in the guide. AFAIK, 100 MB is as low as we can go for a native Windows container, but that's still much better... i brought this topic in the other PR. we don't need a command that works on both OSes. if is the command to use on Windows containers only, so be it.
And also, my image? It might be out-of-date.
we need to start planning how Windows containers can used be easily on Windows nodes. the pause image should be pre-built, small and easy to pull by deployment tools, such as kops, kubeadm. that is much better, but still a bit too much? can we go down to ~100MB? this image size was reported by a user in a k/k issue recently: my assumption is that the user was building the image himself.
i am ok to have that conversation, but please file a ticket and let's discuss that in sig-windows
everyone, thank you for your help and responses. trying to bring back this conversation to the original request. what would you like us to do in order for the few folks from google that have permissions to upload the windows images to be used for testing.
I am happy to help publish this via the alias, which I believe is the correct long-term path. We need an image - preferably not one as prone to CVEs as busybox historically is. Hence a suggestion for "pause" or something equivalently useless and tiny (if it is tiny). Then, we need a path. First, we'll be setting precedent for test images, so I need input on that. I propose as the root. Objections from test folks? Second, do we want to squirrel this away as a Windwows-specific binary, or should we not name it in any particular way, so that we can pull the same image for all tests via a multi-os, multi-arch manifest list (which can be done later). E.g. vs How about * ? Is that too deep? Does it need Windows in the path or should we put it in the tag? Michael Michael <:
So, I'll sumarize the previous discussions. The proposed images are: The image currently named image tags:. based on: respectively. image is currently used in Windows E2E tests and should already exist on the nodes, so download should be minimal. The image. a manifest list needs to be created containing the tags: The image is used to tag the image, which is used by kubelet, so it should already exist on the nodes. smaller in size (~350MB). the test should create pods with the command . I'm fine with either one, although I think people will tend towards the nanoserver image, and in fairness, I think it will be the easiest one to maintain long-term. AFAIK, Microsoft updates their container images periodically, so, we should do the same for the image. If not, the test using the image will then have to pull the entire image (350 MB is ok, IMO). E.g. vs Well, a manifest list can contain both Linux and Windows images, So, if the busybox image is picked, the Linux busybox image can be included in the manifest as well. Ideally, most images work on both OSes (with some exceptions, like the nanoserver or alpine images), so I don't think there's a need to put windows in the path. And if a new root path is being for new test images, I would like to ask that images mustn't container slashes in their names (e.g.:). Docker doesn't support images with slashes in their names (see ). Even if all the images on eventually support Windows, it would still be a good idea to be able to use dockerhub for testing purposes.
We need an image - preferably not one as prone to CVEs as busybox WHOAH. That's a 2GB image. For testing. I am in favor of this just based on the size (I was thinking something like kuberneets' pause, which is 240 kB. Over 1000 times smaller. We have different comprehensions of what "nano" means. 350 MB is not OK. It's acceptable if we have no other choice, but honestly we should create our own that is as small as humanly possible. Then, we need a path. First, we'll be setting precedent for test images, so I need input on that. I propose as the root. Objections from test folks? I don't have enough context on e2e future, so I'd like some input from someone who does. Second, do we want to squirrel this away as a Windwows-specific binary, or should we not name it in any particular way, so that we can pull the same image for all tests via a multi-os, multi-arch manifest list (which can be done later). E.g. vs I agree (except for busybox, which is a known vector for contaigion). Having a multi-os, multi-arch "serve_hostname" or something equally dumb seems ideal. Docker HUB doesn't. Docker itself does, and GCR does. If you're using dockerhub for testing, you have to change "" to "e2eteam/whatever", right? So what does it matter if the GCR path is longer? That said, I would be OK to have a single dir if that is what test-team wants. It just seems likely to get messy eventually.
Yes, dockerhub, I forgot the "hub" :) And yes, that's what we're doing and it works for the most part. But, if you take a look here [1], basically, the root is , which, as you know, is not accepted by dockerhub. So, the proposal was to not have "image names" with slashes in them (e.g.: in this case). [1]
I see. I think it would be OK to ask that images don't use subdirs, but the e2eRegistry variable could. e.g. // Docker hub e2eRegistry = "e2eteam" // GCR e2eRegistry = "" ResourceController = Config{e2eRegistry, "resource-consumer-controller", "1.0"} That seems workable, though it doesn't answer the question of the actual path in GCR :) Claudiu Belu <:
That seems workable, though it doesn't answer the question of the Yep, exactly right. :)
what are the next steps on this ticket to get the images to
I am still waiting for which image we are going to use, and for any objections to "" as a root. Michael Michael <:
Well, I think we are in agreement that we should use the nanoserver images. As for the root repo, I have no objections.
can you please explicitly give all the images to upload in an authoritative list? i am hoping we can upload all of these by monday EOD.
Sure. As we've discussed previously, the images are:
no manifest list image that supports both 1809 and 1803? are we OK for only having 1809 for that "pull from GCR" test?
Yeah, I meant those images should be included in a single manifest list (something like or something), like we've previously discussed. As for the second question, AFAIK, the GCE Windows CI is using Windows Server 1803 at the moment, so it would be nice to have the new test running on that too.
see above for the two images we want to push. we also need to repush this image resource-consumer/controller:1.0 to avoid the "/" as resource-consumer-controller cc:
i pulled these images on Windows: size of each is roughly ~350MB. and then saved them to TAR and uploaded them here: to go lower than that we might have hack custom images that trim, for instance a bunch of DLLs that we don't need (for e.g. a minimal pause image), however such a custom image will just require yet another unique download if the users are already basing their workload containers on official images. also for such hacks we need someone to verify if the nanoserver LICENSE allows that. i'm pretty sure it doesn't.
no hacks please. we need to upload the images as they officially come from microsoft. :)
/milestone v1.14 We're entering and this looks relevant to the 1.14 release. Please if I am incorrect
The slash thing is kinda irrelevant to me, it's just a limitation of Docker Hub as pointed out
i understand that. but can you make forward progress in testing that autoscale ticket without this work?
i think you have the two images to upload. let us know when this is complete and thank you so much!
The tars shouldn't be necessary. The nanoserver images include 2 layers that are type and should not be pushed. here's an example manifest from the bazel repo. you could probably use the existing rulesdocker to push nanoserver with no changes if you're building and pushing images with bazel
we can't follow this process due to Windows licensing. i will hide your post so that it is not followed by accident.
just to clarify - you can reference those in manifest lists, and docker push / pull them without actually pushing the layer contents to a manifest v2 complaint registry. That's the right way to do it. That separates out user-owned files in the upper layers from those in the base layers that are always served from a Microsoft server due to licensing restrictions
Oh ok then. Keeping in mind the fact that the nanoserver image is a manifest itself (for amd64 and arm) and that you cannot reference another manifest list in your manifest list, then the the actual images which should be to the manifest list are: 1803 doesn't have multiarch.
the post is still quoted in your response. best to edit that as well? ok, looks like the licensing of these image is slightly more complicated than expected... i hope a tutorial is not required when pushing to GCR. :)
What is the status of this now? I am THOROUGHLY confused
me too according to here and here and as far as i understood, we only have to push the JSON files that hold layer references, but not the actual layers because this violates the licensing of the nanoserver images.
I'll post explicit steps using , etc.
a PowerShell window as an administrator Docker-CE 18.09.2 experimental mode for the Docker CLI Here's the copy & paste steps to pull the existing images, push them to the new registry, and create a manifest. Change $destinationRepo to be something like or your private repo name. The pushes will be nearly instantaneous because they're manifest-only pushes. All layer data is skipped. Now, you can inspect the manifest and it will show multiple images with different os.version And last, push it
Here's the example repo I made on When things are done this way, the original Windows image is still served from Microsoft as required by the license. If I were to add unique layers on top of this image - that content would be pushed to the registry.
Pushed to my staging, will try to get the promoter running ASAP
hmm, it looks like our image promoter is not quite ready for multi-platform yet. Let me try pushing to
yeah, you might be bumping into
Someone test that pull?
does that mean no Windows images can live on GCR? Am I misreading it?
I believe GCR supports it, but some of our tooling (which uses google/containerregistry) may not. OTOH I think tool is just using directly, so it might be unaffected.
!
Awesome, so that seems to work for now. I guess the next step is to FROM that in your own containers.
or just use it in your tests. Do we close this for now, then?
SGTM, leaving the close call to
This looks right. I did a pull on a 1809 machine and the hashes match. is this enough to finish that test case, or are more images needed? We definitely need to get the images into a CI for 1.15. See /
this should be enough in terms of the test case.
^ FYI, there are now Windows test images at GCR. thanks !
Awesomeness, I've made a PR adding the testcase using this new image. This issue will close automatically once that merges. Test works for me. One question remains though. Do we have to have a Windows equivalent for the , so it fails on Windows. If we have to add a Windows equivalent for that too, then the nanoserver manifest list should also be pushed to the registry as well. Anyways. thanks for the help, everyone!
are you asking if can push the nanoserver manifest to the authenticated-image-pulling registry as well? Tim?
Yep. It is a conformance test, so it would be nice to be able to cover it as well.
any chance you push the nanoserver manifest to the authenticated-image-pulling registry as well? thanks for all your help! once you do the above, we can lose this issue
Try ""
thanks it works! !
please, close the issue once you check this.
thanks Tim! you rock!
can you please close this issue if we are done?
The issue is supposed to close when the PR merges. Closing anyways.


<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: ci-kubernetes-e2e-gci-gce-serial Which test(s) are failing: LocalVolumeType: gce-localssd-scsi-fs Since when has it been failing: 2/20 Testgrid link: Reason for failure: unsure yet /assign
After , there are still two failing tests: For the first one, looks like reconstruction is not correctly implemented for local PVs: to check if it's ok for mounter to be initialized with an empty path the panic in the operation_generator
cc Unsure why it's failing only for this local pv type (real disk vs loopback block) I have not investigated the 2nd test yet
/priority critical-urgent /milestone v1.14 We're entering and this looks relevant to the 1.14 release. Please if I am incorrect
/assign
hi, checked , after , "should be mountable" has been fixed. failing test is "should unmount if pod is force deleted while kubelet is down [Disruptive][Slow]" and it fails for all kinds of local volume types
This is because local volume cannot be reconstructed for now (localPlugin.NewMounter requires field is set), so is going to . But uses to , and , so it panics here. I'm not sure how to support reconstruction for local volumes correctly yet. But we can fix this panic first, then reconciler will clean up the existing mount points if the volume is no longer needed. IIUC, this will fix this failing test. After this is fixed, if necessary we can revisit how to reconstruct local volumes correctly.
I see similar failures in and too; are they connected? cc
yes they should be similar
need to keep track of this to see if failures in and also clear up.
The tests are still throwing s in (it's the job there though). I've opened to track the failures in 1.14-blocking. They also seem to not have run since the 24th, I'll look into that in the meantime


In , we check mount on first level directory in pod volume directories. It is true for many volume plugins, but not for CSI plugin which . cc
/assign
Pardon my ignorance of kubelet source code, but this looks very much similar to an issue we have been running to many times recently, when developing a CSI plugin. The problem is that when CSI volume publish is called and the mount point is not propagated from the CSI plugin pod (either missing bidirectional flag in yaml or not supported by underlaying k8s cluster implementation), the kubelet nevertheless starts a pod using the volume. The volume is not there, so the app writes to root filesystem in fact, leading to all other kinds of problems - i.e. losing data, filling up root fs. IMHO what kubelet should do is to check that the mounted filesystem is indeed visible after publish call and if not it should fail the volume attach request. If this ticket was fixed, would it solve the issue we are seeing or should I open a new ticket? thanks
I think we can potentially add that check but we need to make sure first it can work with ephemeral CSI drivers. Ephemeral drivers would have to "stage" their directories somewhere, like /var/lib/kubelet/plugins/<csi driver/, and then bind mount it to the publish path. cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Hmm... That is a good question. Does csi require a mount to be made or only that the specified directory be created? Will have to go read through the spec...
/remove-lifecycle stale
The only place in the spec I could find was this comment: // For volumes with an access type of mount, the SP SHALL place the // mounted directory at target_path. So I think having Kubelet enforce that check would be ok with the spec, and a good thing to do.
cc CSI spec says that the target path has to be a mount. Shall we go ahead and enforce it then? Are we going to break drivers that may not be doing this?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten
The issue mentioned here involves two functions . Now with PR , the issue in CheckVolumeExistenceOperation is solved by checking mount path instead of volume path. The only issue here is for getMountedVolumePathListFromDisk which is used during pod deletion. This function was introduced by PR . Now I am questioning whether we need to check the mount point or just need to check pod dirs exist before deleting the pod. Firstly, for some plugins, there might not be mount point. Secondly, without checking mount point, it means that pod cannot be deleted as long as there are pod volume dirs. In normal case, volume dirs should be cleanedup during volume teardown. For cases that kubelet restarts, we have reconciler try to clean up volume mounts if possible. So we can propose a couple of solutions here mount point check. It solves the problem, the downside is that there might be some rare case that reconciler could not clean up dirs after kubelet restarts. check mount point, and also check the next level "/mount" dir. This might still cause problem for some plugins that do not have mounts. cc
Discussed offline with For proposal 1., pod will not be deleted from api server if there are still volume dirs exist. So to make sure it works well, we need to have reliable way to clean up dirs. Normally when kubelet restarts, reconciler will either add it into actual state (if reconstruction works) or try to clean it up directly once unless volume is still in desired state. So if reconstruction does not work, and it fails to clean up during reconciler sync, there is no other chance to clean up the dirs. Or if volume was in desired state and removed quickly afterwords, the dir might not get the chance to be cleaned up at all. We are thinking to improve this part, we could use mark volume "uncertain" in actual state feature. During reconstruction, we can remove the check for mounted points (since the mount might be gone during node reboots or driver does not have moutns at all). Reconciler could mark those volumes as uncertain (even if it was in desired state) to make sure those volume got cleaned up or remount whenever needed with retries. With this change, I think it is safe to remove the mount point check for pod deletion too.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


Which jobs are failing: Tests which exec into containers and alter lifecycles Which test(s) are failing: All new tests introduced for api change which exec into containers New alpha subpathExpr tests [Feature:VolumeSubpathEnvExpansion][NodeAlphaFeature:VolumeSubpathEnvExpansion] Since when has it been failing: 27/February/2019 Testgrid link: https://k8s- Reason for failure: is only set in e2e tests and the binary is not available in node e2e tests Need to restructure and call the API rather than through the framework convenience function Anything else we need to know:
/sig node /sig storage /assign /cc


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Running the vSphere cloud provider on a windows node throws the following error: This is because the vm identifier is retrieved with the command - , which assumes that the uuid and serial are the same. On some vmware vm versions, the uuid and serial are not the same. vmware (comment), vmware (comment) What you expected to happen: Kubelet is able to find vm in vCenter from the identifier looked up on the node. or should be used instead to find the vm identifier. How to reproduce it (as minimally and precisely as possible): Issue only occurs on some vm versions, such as 13. Anything else we need to know?: On the linux side, this issue was resolved by switching from looking up the id from to . See Some discussion about this issue here - Environment: Kubernetes version (use ): 14.0-beta.1 Cloud provider or hardware configuration: Vsphere OS (e.g: ): Windows 2019 Kernel (e.g. ): Install tools: Others:
/sig windows /sig vmware
/milestone v1.14
You must be a member of the GitHub team to set the milestone. If you believe you should be able to issue the /milestone command, please contact your and have them propose you as an additional delegate for this responsibility. <detailsIn response to : /milestone v1.14 Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign


xref In 1.16: change the metadata-proxy addon to use change the kube-proxy addon to use change the masq-agent addon to use stop setting , , and in kube-up
Hey I'm trying to start to contribute to kubernetes and this seem to be an easy task. Can I help with this one?


have worked since 1.0 as a way to pass basic auth credentials to the API server. In 1.11, these flags were unintentionally removed in Environment: Kubernetes version (use ): kubectl 1.11+ /sig cli /sig auth
/assign


What happened: The rbd plugin use "rbd info" error: The messages before and after json result is warning string and no effect to rbd info, but it is effect to parse the output. So the kubernetes will failed: What you expected to happen: Kubernete no failed and errors How to reproduce it (as minimally and precisely as possible): Remove the config file
/sig storage
We are blocked by this issue, can't resize any PVCs
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
That is another situation that cluster configuation is empty, aslo will return lots of warnning information: Also will trigger expand volume failed. But should we add -c=/dev/null in rbd info command?
Folks, the pull request that would have fixed this just got closed as it was rotting away. This is an actual problem in production, can we please fix this?
+1 we are still being plagued by this issue...
+1 {"name":"kubernetes-dynamic-pvc--147e-422c-9e67-","size":,"objects":2560,"order":22,"objectsize":,"blocknameprefix":"rbddata.","format":2,"features":["layering"],"flags":[]}, invalid character '-' after top-level value
I solved by exceuting this command
Wow...I was not expecting this to work ... but it did ;) Any insights on why is it working with just the presence of the file? This is very strange... Appreciate any comments...


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: I recently saw an issue in which the scheduler hit this log line many times: After some digging, it turns out that PodAffinityChecker has a and . The points at the scheduler's cache instance, while the points at a light wrapper around a . In the scheduler, we use the cache to provide a globally consistent view of the cluster, even if that view is slightly out of date. Data read from the core may not have been processed by the scheduler cache yet. Letting data "skip ahead" like this can cause stale info to get persisted in the scheduler cache, and other subtle bugs. While the original problem is not likely caused by this issue, I think it needs to be cleaned up to avoid potential bugs down the line. What you expected to happen: Predicates should only read state from the scheduler's cache in order to maintain consistency. How to reproduce it (as minimally and precisely as possible): This was observed in a cluster with high rate of pod creation and node creation/deletion. Anything else we need to know?: Environment: Kubernetes version (use ): 1.12.5 Cloud provider or hardware configuration: GKE with preemptible nodes OS (e.g: ): Kernel (e.g. ): Install tools: Others:
/sig scheduling
We do have this kind of issues, and here is another one:
I'm not sure the urgency of this fix, but if its not super urgent, I would love to pick it up, if there's an eng more experienced with the scheduler who could give me some guidance :)
Hi I dug into the source code, and find out is used by the scheduler and kubelet. Maybe we could change the internal struct of , but I do not know why it appears in kubelet which seems irrelevant to the scheduler.
is just a wrapper of corelisters.NodeLister. kubelet can get node info by it.
I think the point is that the kubernetes scheduler was designed to support single instance at the beginning. Even with the addition of multiple scheduler support, the cache between each scheduler is isolated. So at this stage, we can accept this without a shared state solution. After all, the node information is not actively modified by the kubernetes scheduler.
So is it necessary to ensure the consistency by using node lister instead of cache in the scheduler, or we do not need to fix this issue which seems like to caused by inconsistent cache.
We've recently hit this issue with the scheduler. It appears that our scheduler somehow gets out of sync with the nodes, and start scheduling pods on tainted nodes. The node's kubelet then (rightfully) refuse to admit the pod. Looking at the scheduler logs, they are flooded with the following : Similar scenario, this is a high churn cluster with thousands of pods scheduled per minute - and nodes being autoscaled. Can we try to get a fix ?
I'm glad to pick this up if we decide to fix this issue.
It would be awesome if you could fix this issue for us, we are currently seeing 100's of pods incorrectly being scheduled onto nodes with taints.
The code cited for logging "Node not found, %v" is now in :
/assign
Thanks for the interest on fixing this bug. Sorry I haven't had time to engage on this issue. Feel free to send PRs my way. I'm not 100% certain that this will fix the problem of pods getting scheduled to tainted nodes. How is the taint being applied to the node? Even if the scheduler is internally consistent, there could be a time between node creation and the taint being applied where we send pods to that node.


What happened: I tried to set up bandwidth limiting by booting a cluster with , which by default sets up kubenet. I used the annotations xM xM and then used to test the ingress and egress bandwidths, with no effect. Does anyone know if this feature is still supported in kubenet? And if so what are the steps I need to take to boot a cluster that can use pod annotations to do tc shaping. What you expected to happen: Per traffic shaping should be implemented on Kubenet as of 4/12/2018. Anything else we need to know?: There are a lot of issues regarding traffic shaping not working. None have a concrete answer of how to get it to work on kubenet: Relevant: I believe this is where it was first implemented. Environment: Kubernetes version (use ): 1.11.8 Cloud provider or hardware configuration: AWS CC:
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
cc
/triage unresolved I am a bot run by
can we put this on sig-network agenda? the discussion item would be: what is the current status of this feature (alpha, beta, GA?) what should we do next with this API?
I have this to the agenda, will grab some info.
AFAIK traffic shaping is still supported in kubenet. Does a pod with these annotations come up successfully but has no traffic shaping applied? Or do things fail earlier? If it comes up successfully, could you run ?
On GKE we do not officially support this, our default image even don't have tc installed. I tried with ubuntu, the kubenet is failed to parse the tc output. kubenet expect. // filter parent 1: protocol ip pref 1 u32 fh 800::800 order 2048 key ht 800 bkt 0 flowid 1:1 tc return.// filter parent 1: protocol ip pref 1 u32 chain 0 fh 800::800 order 2048 key ht 800 bkt 0 flowid 1:1 notinhw
If this issue has been triaged, please comment . Meta: /athenabot mark-triage-reminder I am a bot run by
Depending on which CNI plugin you use you need to install the bandwidth plugin, see:
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
since it is in progress /priority critical-urgent /milestone v1.17
/lifecycle frozen
/assign
Bug triage for release 1.17 here. Since this issue has been tagged for the milestone, we would like to check it's status. Code freeze is on Nov. 14. Will this issue be resolved before then? If not, we should move it out of the milestone.
The fix is still pending in review.
Will it be merged by this Thursday for v1.17 code freeze?
/milestone v1.18 (v1.17 code freeze)
matching assigned priority on PR
Hello I'm one of the Bug Triage shadows for the 1.18 release. Since this issue is tagged for 1.18, I'd like to check its status. Is this issue intended for milestone 1.18? Friendly reminder: Code freeze is scheduled for March 5th.
Hello Checking in again. Do you think this will make it in for 1.18 release?
Hey Checking in again as this issue is marked as . Are you close to completing this for v1.18 release? Another reminder that code freeze is just around the corner! (March 5th).
Friendly reminder that code freeze is in 2 days and this issue is marked as
The fix got approved by ,should be able to catch the freeze.


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: I see only amd64 architecture etcd is used for the which will create issues for the other architectures like ppc64le, arm etc. We need to replace this etcd image with the k8s maintained fat manifest image present in
/sig testing


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Following the documentation at the , I tried enabling quota for a namespace other than but I was still not able to create a pod with priorityClass in my namespace with following error: Error from server (Forbidden): error when creating "": pods "memory-demo" is forbidden: pods with system-cluster-critical priorityClass is not permitted in my-namespace namespace What you expected to happen: Expected pod to be created with namespace where I allowed quota. I can see that both ResourceQuota and Priority Plugins are enabled from logs. I0408 17:15:08. ] Loaded 8 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,MutatingAdmissionWebhook. I0408 17:15:08. ] Loaded 6 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,ValidatingAdmissionWebhook,ResourceQuota. How to reproduce it (as minimally and precisely as possible): Follow the steps at Anything else we need to know?: I think, the issue is with priorityPlugin where we are rejecting instead of letting resourceQuota plugin to handle the admission when is enabled. Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Others:
/sig scheduling /cc
Same happens to me. Correction, it does not allow to use the default permissions outside of kube-system
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
See also:
/remove-lifecycle stale
/assign Per offline discussion with
Same issue here. We are using namespaces other than kube-system for system pods as it is getting too crowded, therefore we would like to run cluster-node-critical pods in other namespaces.
Waiting for this to be available myself. No reason my Node Exporter dataset has to run in the namespace instead of my namespace.


Public IPs are expensive resources. Given that we're running scalability tests on 5k-node clusters, we are unnecessarily consuming 5k IPs which is a lot. We should switch our tests so that nodes have only private IP addresses. We are going to do that by configuring NAT so that nodes can talk to outside world (e.g. to download image from dockerhub). But that also requires changing some correctness tests which assume public IPs.
To summarize recent work: setting up private gce clusters with nat, feature is gated via the env test and dashboard with correctness tests on gce private cluster - https://k8s- 27 tests fail on fixing the tests - should fix majority of them
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale /lifecycle frozen
- is there anything missing to be done here?
No, we can close this one. It was done long time ago. Thanks for catching /close
Closing this issue. <detailsIn response to : No, we can close this one. It was done long time ago. Thanks for catching Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: In Azure disk attach/detach function, it uses slice() to store the disk list: And after changed, the original would also be changed, so original azure data disk list is corrupted after attach/detach disk example code as below: What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /sig azure /assign


What happened: We created a Deployment and continuously updated an annotation on it periodically for a week. Eventually, we noticed errors in the controller-manager logs due to the annotation growing too large. Example annotations: controller-manager error: What you expected to happen: that do not change the revision number should not be to the revision history. revision history should be capped at a fixed number of elements, it should not be possible to have unbounded growth for Deployments that are frequently updated. How to reproduce it (as minimally and precisely as possible): a Deployment modify an annotation on it and note that the annotation gets the same number appended every time. Environment: Kubernetes version (use ):
/sig scalability Not 100% sure where this one fits, I couldn't determine which group owns the Deployment/ReplicaSet controllers, but this is a scalability problem
/sig apps
Agree with these, and if the limit is hit, the beginning of the history can be removed.
/assign


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Docker has a hardcoded 16K limit on log line length, which splits lines longer than that into multiple messages in the log file. in , however, doesn't account for this when is set to and prints the timestamp in front of every entry in the log file instead of every logical line. This leads to random timestamps appearing in the middle of log lines, potentially breaking the applications that parse these logs. What you expected to happen: inserts timestamps once per logical line, so original log data is preserved. Alternatively, allow API to request the logs in a sane format like JSON, which would allow merging the lines back and combining them with the timestamps more intelligently on the application side. How to reproduce it (as minimally and precisely as possible): Create : apiVersion: v1 kind: Pod metadata: name: log-lines spec: containers: name: main image: busybox command: ['sh', '-c', 'while sleep 1; do echo -n ; for i in $(seq 1 ); do echo -n A; done; echo ; done'] Start the pod with Fetch the logs with , or using the API () Observe random timestamps inserted in the middle of the log lines. Environment: Kubernetes version (use ):
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I will work on it. /sig cli /remove-sig node
The logs are being corrupted by the API endpoint in kubelet, so shouldn't it be in scope of SIG node?
The log is not corrupted by k8s. The same log could be got using "docker container logs -t CONTAINERID" I think you should post your requirement to Docker. Please refer following code Line 50 ~ 52 if config.Timestamps { logLine = append([]byte(msg.Timestamp.Format(jsonmessage.RFC3339NanoFixed)+" "), logLine...) }
kubelet doesn't rely on Docker when reading logs, it reads the files itself (see the code linked in the issue). I don't think Docker would change the forced line splitting (see for example), so this should be fixed in Kubernetes.
Your are right and kubelet read log file from host such as /var/log/pods/pods/NAMESPACEPOD-NAMExxx/....(JSON format) I will post a PR for it. /remove-sig cli /sig node
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle rotten
I saw the attempt fixing this but it was scary (echo this comment: ). I suggested a way it may be improved. Also the proposed fix didn't introduce any e2e tests to validate that runtime behavior is not changed and runtimes keep the no-newline-on-splitted-log logic. are you or anybody on this thread are still interested to work on this? Please re-open if so. Meanwhile closing this ahead of bot doing it because the inactivity. /close
Closing this issue. <detailsIn response to : I saw the attempt fixing this but it was scary (echo this comment: ). I suggested a way it may be improved. Also the proposed fix didn't introduce any e2e tests to validate that runtime behavior is not changed and runtimes keep the no-newline-on-splitted-log logic. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
You can't reopen an issue/PR unless you authored it or you are a collaborator. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-lifecycle rotten /reopen will fix it.
Reopened this issue. <detailsIn response to : /remove-lifecycle rotten /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/triage accepted
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I received a bug report about this issue. The root cause is that the buffer reads up to a new line. If the line is greater than 4096 bytes and '--timestamps' is turrned on the kubelet will write the timestamp and the partial log line. We will need to refactor the ReadLogs function to allow for a partial line read.


We observed in an e2e that if attach is performed on a volume that was detached before from same node, sometimes the mount of the volume (for 2nd pod) could be stuck waiting on invalid devicePath in call. The problem is - when attach for second pod failed with this caused an dangling volume error from AWS cloudprovider(implemented by us). The dangling volume error is supposed to auto-correct any volumes which are attached to a node that it shouldn't be. But in this case - the dangling volume error was triggered by the fact that, volume was not actually attached to the node but because call for 2nd attachment returned stale data (because of AWS eventual consistency). So, dangling volume exception caused volume to be to actual state of world (so as it can be detached later), but this caused volume to be reported to node's Status field. And that caused second pod to assume volume as attached to invalid device path and hence 2nd pod was stuck for 10 minutes waiting for volume to attach on a device path (/dev/xvdbl) which does not exist.
/sig storage /sig aws cc
Here is k8s logs that confirm that, did return stale data for volume: See detach succeeds at 19.30.25: and yet: so second DescribeVolume API call is actually returning stale data.
There are couple of ways of fixing this: error is used by AWS and Openstack cloudproviders to fix incorrectly attached volumes. We could fix mechanism of dangling volumes, so as when it adds volumes to actual state of the world - it does so without reporting the volume to node's Status. can make a local fix inside AWS so that when we return error, before returning that error we also verify volume attachment by querying the instance. We already have instance information in that call. Verifying volume attachment by and should reduce likelyhood of returning incorrect errors. There is still a chance though that, both API calls would land on a AWS backend which does not yet have updated information.
expanding on fix - I am thinking when we add a volume to actual state of the world via DanglingVolume error, then we should add the volume as "uncertain". We will update the code to not report "uncertain" volumes in node's status and hence in this case volume state will be fixed correctly. If I understand correctly - did not intend to report "uncertain" volumes to node's status anyways.
This looks like another symptom of . I tried that, it does not always help. Both and may return the same stale data, see IMO, is the right way to go. Unless it break something else, all these AWS corner cases were complicated already and now anything can break.
/cc


Example:
/assign
cc
For all those PD instances, tests are being skipped due to this line: Fixing it by removing that skip as suggested by
/open until we verify the issue is fixed
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Looks like diffResources became green on 5/27 /close
Closing this issue. <detailsIn response to : Looks like diffResources became green on 5/27 Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


/sig api-machinery /kind bug /assign sttts because the controller that maintains the condition runs multiple instances in HA apiserver clusters, the controller must not duel in cases where the validation messages change slightly between releases


What would you like to be : I would like a method of seeing more clear data while tests are running, e.g.: How many tests will be run, have been run, have passed, have failed, etc. Name of the tests passed/failed/skipped Why is this needed: While a run is going we only have the output from stdout to investigate. There is a LOT of noise in those logs which is great for debugging after the fact, but difficult to consume most of the time. you can't tell how many tests have been run or get a sense of how much time is left you cant easily get a list of all tests that have failed so far My proposal Adding a new, custom ginkgo reporter that will post the data to a given URL (passed via a flag) if the URL isn't set, the reporter is a noop and doesn't waste time/resources Use cases: A user runs the tests dumps stdout to a file for debugging later, meanwhile they run a local server that will just echo the status updates it gets. At any point they look at the output they can get good data about what would be run, how far along it is, how successful it has been so far. Using the conformance test image and Sonobuoy, the Sonobuoy worker can act as the server, taking those status updates and reporting them to the aggregator. This would allow Sonobuoy to present that information to users on demand or stream it depending on the context. The work of adding this is pretty minimal and is clearly useful. In the case of Sonobuoy; this has been the most requested feature for a long time showing that people in the community really do want to have a better high-level picture of the progress of these tests. I think emitting the data to an endpoint like this is a flexible implementation that can help any consumer of the tests (not just Sonobuoy).
/sig testing /area test /area test-framework
One notification URL should work fine. Can you give me details on the data model for the status data to be published?
That is one, slightly TBD area. The reporter in the test framework has a bunch of data and I'm not sure what we exactly Sonobuoy may want from any given plugin (not just the e2e plugin). I don't want this implementation to be overly Sonobuoy specific or overly e2e specific. Will update when something more clear is pinned down.
Brought this up in the testing commons meeting and it was agreed that this approach seemed reasonable and that the customReporters seemed like the only way to generate this data. In addition, it was requested that we find a way to also emit this (or some variation) into stdout/logs so that it is easier for a human to navigate those verbose logs.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: ci-kubernetes-e2e-gci-gce-serial Which test(s) are failing: [sig-network] DNS configMap nameserver [IPv4] Forward external name lookup should forward externalname lookup to upstream nameserver [Slow][Serial] Since when has it been failing: Failing since 6/10 at 16:58. . Decent chance that the failure is due to Testgrid link: Reason for failure: /cc /priority critical-urgent /kind failing-test /milestone v1.15 /cc
This is blocking the release /cc
I noticed in the it thinks it's testing 1.16 alpha:
/assign
should fix it. Sorry for the mess-up!


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: run on macOS, all md5 hashes will be stripped from the output and the script will not fail or warn. What you expected to happen: The script should work or detect this issue and fail. Currently it silently ignores not existing on the host. We have some warning logic, but solely based on bash version, if the user upgraded their bash installation but did not install md5sum we don't warn them. How to reproduce it (as minimally and precisely as possible): run the script Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /sig testing /priority important-longterm /assign
cc


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: If a pod's restart policy is "Never", and kubelet failed to create the PodSandbox (for whatever reason) once, the pod will be stuck at "ContainerCreating" forever. What you expected to happen: Since the user/application containers were never created, kubelet should retry the PodSandbox creation. The fact that the pod remains in a ContainerCreating state while kubelet explicitly does nothing to resolve the situation also confuses the user. How to reproduce it (as minimally and precisely as possible): I don't have an easy way to trigger a podsandbox creation failure off the top of my head, but this should be doable by messing with the CNI binaries/configurations. Anything else we need to know?: Relevant code snippet: Environment: Kubernetes version (use ): Only checked the master branch (1.16), but all supported release branches should have the issue Cloud provider or hardware configuration: not relevant OS (e.g: ): not relevant Kernel (e.g. ): not relevant Install tools: Network plugin and version (if this is a network-related bug): not relevant; though the initial PodSandbox creation failure could be caused by networking issues. Others:
thanks for posting! I'm looking at the code, and it seems like one fix could be checking if this pod has an associated sandbox - if it does, we want to respect the current behavior. If it doesn't, we want to attempt to create the sandbox. I was thinking something similar to: Thoughts? Is there a simplex fix?
Hi! Thank you for filing this - it matches the behaviour we've been seeing. Am I correct that this is a regression from (), and so only affects Kubernetes =1.13 ?
Your solution may not work for the one specific failure I checked, where the PodSandbox container was created, but call to the CNI plugin failed (for some reason; could be caused by a race condition). This left us a not-ready PodSandbox with no IP. Probably have to actually determine whether the user containers were created before. Need to check the code closer though.
Thanks. That's probably correct. I saw this in happening in 1.13
Gotcha - thanks! Taking a look at your pr now :)


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: I was trying to write a test in a service I'm working on to catch if an error returns from . When I overrode the result of the fake client with a reactor to return an error I did not get the error back. What you expected to happen: If the reactor is set to return an error, it should come back. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: This only happens with the fake. Environment: Kubernetes version (use ): 1.14 Cloud provider or hardware configuration: any OS (e.g: ): any Kernel (e.g. ): any Install tools: any Network plugin and version (if this is a network-related bug): n/a Others:
/sig testing
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Let's take a look at this /priority important-longterm /sig api-machinery /remove-sig testing /remove-lifecycle rotten
Is there any progress?
I submited a PR to fix this issue. PTAL .
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
^^^ would it be possible to reconsider this PR? I ran into the same issue today.
this hasn't been fixed and we recently ran into this issue. Can we /reopen? Happy to bring up to date.
/reopen still relevant as above
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-lifecycle rotten /triage accepted
/assign


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: I was using kubeadm to generate cluster ca: The CA generation failed because of 502 bad gateway returned and the response body isn't empty when it's trying to fetch stable version from internet. In this case, the function returns error instead of falling back to local client version: is how kubeadm fetches stable version and is how the error is handled for . Should kubeadm always fall back to client version when there is any internet issue? Otherwise, this sounds defeat the purpose of fallback. /sig cluster-lifecycle What you expected to happen: kubeadm falls back to local client version when there is any internet issue. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): 1.13.7 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
I'd like to take this issue. It would be better if you could give some directions. /assign
recreate the issue in kubernetes/kubeadm: /close
Closing this issue. <detailsIn response to : recreate the issue in kubernetes/kubeadm: Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
the fix will be removing the when failed to and add some unit test if I don't missing anything
Got it. Thanks.


Found this when trying to see which vendored dependencies does not use a proper tag (and is using a random SHA). The dependency is very old and there have been a lot of updates to it: Looks like the However there does not seem to be anyone around to take care of the portworx plugin. Is it time to deprecate it? is anyone using it? /sig storage
cc ?
/area code-organization
Hello The portworx volume plugin is definitely in use. The portworx volume plugin doesn't need frequent updates since the plugin acts as a simple client to all provisioning calls. In the same view, the vendored openstorage package doesn't need to be updated too. The server side implementation in Portworx handles all the latest Portworx features.
cc:
in that case, can you please update the dependency to a specific tag instead of a random SHA?
/assign updated the title of the issue as well
my request is not for a new release or tag in libopenstorage/openstorage repository. Just to update the SHA we use in our k/k repository to one of the released versions ( ). This is not asking or demanding anything to be done in libopenstorage/openstorage, just that we should update what we use from there (and avoid a random SHA). We are trying to do this for all vendored libraries (one by one … slowly as part of the sig-arch code organization sub project, so it’s not a random request)
/assign
GitHub didn't allow me to assign the following users: lpabon. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
After a chat with - apparently this request is coming across the wrong way I am closing this out as it seems to add a over head /close
Closing this issue. <detailsIn response to : After a chat with - apparently this request is coming across the wrong way /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


What would you like to be : The current error message is vague and could be improved. (credit to Why is this needed: Improve contributor experience, make it easier to identify change to be fixed. /cc
/assign


What happened: Ref , we noticed that there are some unnecessary check around limit in kubelet, specifically at: and is moving this check to outside of the pods loop make it better, but we should be looking at whether this check can exist in an even higher layer, as it doesn't make much sense in either or . cc
/sig network
/assign Pending outcome of the discussion here, I would be happy to present a PR
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/remove-triage unresolved
poke :-)
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
How about lifting the check to the beginning of Kubelet#syncLoop (outside the for loop) ? All the events emitted by CheckLimitsForResolvConf are at EventTypeWarning level (not error). The CheckLimitsForResolvConf call in HandlePodUpdates would be removed as well.


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Data race in client-go with below sample code: Data race: What you expected to happen: No data race How to reproduce it (as minimally and precisely as possible): Using above given sample code Kubernetes version (use ):Cloud provider or hardware configuration:OS (e.g: ): OSX
/sig api-machinery
I'd like to take it. /assign
/remove-sig api-machinery /sig auth cli
Bug triage for release 1.17 here. Since this issue has been tagged for the milestone, we would like to check it's status. Code freeze is on Nov. 14. Will this issue be resolved before then?
Another reminder here that code freeze is in less than two weeks (Nov. 14). We will need to move this issue out of the milestone if isn't going to be resolved before then.