<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Tried to create an ingress with a backend serviceport pointing to a service with a port name with length 15 chars that is valid from the service point of view. But failed because the validation for Ingress does not match the one for service. Service Validation (uses DNS Label): ) Ingress Service Port Validation (uses IANA format like for container ports): What you expected to happen: Service port validation should match Ingress ServicePort validation How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.4", GitCommit:"", GitTreeState:"clean", BuildDate:"2020-11-16T17:18:52Z", GoVersion:"go1.15.4", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.10", GitCommit:"", GitTreeState:"clean", BuildDate:"2020-10-15T01:43:56Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig network
It's not friendly to users who create service with long service port name, to create ingress with port reference. Although, the port name of service is editable. Maybe, we can add warning message for service creation and update when the port name is not a valid ingress port name.
I'm fine with a doc change mentioning the restriction similar to the restriction in the doc for container port () where it explicitly says about the IANASVCNAME constraint
/triage accepted /assign


What happened: The latest release of the debian-base image, v2.1.3, contains a vulnerable version of the apt package. CVE-2020- impacts apt version 1.8.2.1. What you expected to happen: Image should be updated to use at least apt 1.8.2.2 to clear the vulnerability. How to reproduce it (as minimally and precisely as possible):
/sig release
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
kube-proxy is using debain as base image. /assign Let me check other components
checked the base image in and I seems that no new debian or ubuntu official image is built for this CVE ?
That is correct, the purpose of this issue is to get a new image built and released, one which contains a resolution for the CVE.
per comments of @ tianon in Will update it once debain releases it.


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: ListByResourceGroup method in the azure diskClient mock makes a call to Delete What you expected to happen: ListByResourceGroup method in the azure diskClient mock should call ListByResourceGroup How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: azure OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/sig cloud-provider-azure
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig cloud-provider-azure Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage
/area provider/azure
fixed by /close
Closing this issue. <detailsIn response to : fixed by /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Cluster was upgraded from 1.19.5 to 1.20.1 using kubeadm. After that, kube-apiserver cpu usage went from 50% to 300%. What you expected to happen: Expected that apiserver has the same cpu usage. How to reproduce it (as minimally and precisely as possible): Update a cluster from 1.19.5 to 1.20.1. Anything else we need to know?: The cluster is running on sbc hardware, thats why the original 50% cpu usage, and thats why 300% cpu usage is noticeable and causes service disruption. Environment: Kubernetes version (use ): (the apiserver is rolled back to 1.19.5)Cloud provider or hardware configuration: bare metal on SBC hardware. OS (e.g: ):Kernel (e.g. ):Install tools: kubeadm Network plugin and version (if this is a network-related bug): kube-router Others: Please check profiling of each apiserver.
/sig api-machinery
please do a and show the contents here. (hide sensitive IPs if needed). this is the static pod for the kube-apiserver. also, anything interesting in the logs of the api server?
On another cluster I could not reproduce the issue. But, on the original one, it happens again if i switch to the newer apiserver image. So, now I am bisecting apiserver to find the commit since it behaves this way, hopefully that will help. I could not spot any interesting in the logs. I will be able to provide more after the bisecting procedure.
post the above output whenever possible.
The static pod manifest has been moved to avoid kubelet starting it: Still bisecting, trying to find the cause. May it relate to the go1.15 upgrade? Building the apiserver with go1.15.6 seems to be working, while building with 1.15.5 results in a wrong binary. But, still I am working to confirm this.
With apiserver built with go1.15.5 I get 300% cpu usage most of the time, and the following log entries appear:
unclear whether we have perf tests to catch similar problems: /sig scalability for other readers, go1.15.5 is the version used to build 1.20: /sig auth this and similar come when doing rest storage on various apis. e.g.
/retitle performance regression: 1.20 apiserver has high cpu usage
/kind regression
Definitely it seems that the binary built with go1.15.5 is broken. It ends up on 300% cpu usage and with log messages posted above. Note that I am building apiserver this way: Building with go1.15.6 is better, even if not all of the time. Perhaps I am on the wrong way?
this is a detail i've missed from the OP. since we don't have k8s ARM tests, it is likely why we are not catching this and it could be a golang compiler issue on ARM only. in the past we have seen some weird issues from ARM users, that have hinted about compiler bugs.
Sorry for that, I did not mention that explicitly. If it turns out to be a compiler bug, will it trigger a golang upgrade for k8s images?
I will make a build with golang 1.15.2 too and post the results later.
if we know that a certain golang version is buggy, we can update a specific k8s version to use a golang version that works. in this case the latest 1.20.x k8s can be updated to not use 1.15.5.
Simply compiling 1.20.1 with golang 1.15.2 still produces misbehaving binaries. So, I will do a bisecting again with building binaries as documented, and try to find the bad commit this way. This will take longer on my hardware.
So, apiserver behavior is a bit random regarding this issue. What is true that on my hardware (orange pi pc) the apiserver needs around 85 cpu seconds to boot up, after that it consumes ~10% cpu time. When a build is bad, apiserver passes the 85 cpu seconds and still uses more cpu than 10%, sometimes even 300%. So, based on this, bisecting shows is first bad commit.
Reverting that commit on 1.20.1 makes apiserver behaving as before. Consistently it finishes its bootup in 90 seconds, and then cpu usage gets to its normal 10-50%, it depends on load-balancer.
By the way, the problem is not just high cpu usage, but during bootup and high cpu usage, apiserver stays in not ready state, thus kubelet restarts it after failed liveness probes. So it may cause service outage.
we had long discussions in the past where RPI users complained about the kubeadm probe defaults. we claimed something else is not right if the probes are failing, so i think you are confirming that. i can ping the release team and api-machinery about this ticket, but note post the holidays responses will be slow.
/retitle performance regression on ARM: 1.20 apiserver has high cpu usage
I suspect, that the mentioned commit fixes things in a way that it actually places more cpu load, I mean expression evaluates to true more often. The attached profiling data may be a proof for this.
It seems that on slower hardware, operations are distributed into more sample periods (time.Millisecond), which makes more calls to the expensive metrics.Observe functions. Before the mentioned fix, due to quantization bug, those calls were not made every time.Milliseconds. Now they are, and profiling data proves this. Can we raise the samplePeriod, can it be made configurable?
sig api machinery can provide feedback on this.
is that accurate? my reading of the fix is that the quantize method returns results with the same granularity. The bug that was fixed was that quantization could return numbers that resulted in negative calculations if wall clock time changed, and now it respects the monotonic clock.
Perhaps not. That was just my assumption. But still, profiling data shows (at least for me) that metric.Observe calls take most of the time, while reverting the mentioned commit solves my problem. Somehow less calls are made to them.
/assign
We have numerous K3s users reporting perf issues (and crashes in the cronjob controller) on armv7l. I can reproduce it locally on my Pi nodes as well. I hadn't gotten around to digging into it yet, so I'm glad someone else has done so. We also had issues getting arm to build properly starting with Kubernetes 1.19.4; some builds would crash with what looked like a corrupted stack on armv7l unless we set for the build. See: Prior to doing that it was fairly random if a build would work or not; changing unrelated bits of code (ie bumping a vendored module version) would give us a clean build, and then changing something else would break it, and then changing a 3rd thing would give us a clean build again. It felt very much like golang arm toolchain issues. This sounds similar to what reported with good vs bad builds.
I've a little debugging: And I got strange results, a snippet from stdout: Check out the high on the 4th line. That causes a long-long loop to start. And there are more of such lines. If I refactor the code to remove the inline func() and do the Unlock() and the end of the block, this behavior goes away, apiserver goes back to normal. May it be a compiler bug?
earlier you've mentioned: Building the apiserver with go1.15.6 seems to be working, while building with 1.15.5 results in a wrong binary But, still I am working to confirm this. did you confirm that the newer toolchain does not produce code that exhibits the problem? seems likely. in the past i've seen at least one compiler bug-like report from a k8s user (on ARM)
Unfortunately, I think I got confused regarding compilers. Now it seems that I got the same results with go1.15.2, 1.15.5, 1.15.6. So, now I think it does not matter which toolchain I use.
I've applied the following patch on release-1.20: And this produces strange results: It can be observed that from the third line, the function receives strange values. Later normal values appear, but sometimes the calculated is a large positive integer, which triggers a long running loop. Unfortunately, I am not familiar with golang's internals, and could not reproduce this in a few lines of code. But indeed, it seems that something weird is happening. If I place a before the call to , then the function gets the correct value passed all the time. How to proceed?
I think this is an arm32 int64 division overflow/underflow issue? I'm currently testing a minor change to confirm/deny this hunch, but building for arm takes forever so it's still running.
It looks like the time value is being corrupted when being passed into quantize somehow, would be nice to try and reproduce with a very small executable. k3s has a long history of arm32 issues like this that I suspect is a core golang issue, but hard to reproduce and isolate, arbitrary code changes could change the result.
you could try running something like this, which exercises the local logic of the old and new quantize methods and checks that they output roughly similar values (some drift is expected because one uses wall clock and the other uses monotonic clock) I was able to run this for several minutes on linux/amd64 with no problematic values output
Yes, it looks like parameter corruption rather than a division problem. very strange.
Yeah my simple change to the quantize didn't make any difference. That combined with the strange output from the debug printing does indicate some sort of corruption to the when time struct that's being passed in, rather than bad math on the parameter.
Presumably all the unit tests are passing on the arch & OS you are using.
But there are no unit tests focused on that. It's on the to-do list.
If we created unit tests for this file, would somebody be running them on the arch & OS involved here?
I put in a check in quantize to grab a stack trace if when is before t0:
This seems like papering over a compiler problem, but this fixes it for me:
: that is interesting info. I would prefer to not merge that change, we have another mystery where the monotonic time seems to go backwards, and I would hate to lose half the debug info from that. I will add unit tests for the sample-and-watermark histograms; can somebody please run them on the arch & OS involved here?
The time going backwards stuff all seems to be related to VM time sync, isn't it? In this case the compiler seems to be doing something that's corrupting the time struct out from under the anonymous function; the fact that it appears to be going backwards is not (as far as I can tell) due to host time actually jumping backwards into the dark ages with a weird TZ offset. FWIW I considered moving the Before check-and-err into the quantize function, but it wasn't necessary for a minimal fix for this issue don't I left it alone. I have a couple armhf (Pi3b/Pi4b) systems I can run tests on, assuming it's not too difficult to set up.
I am not sure whether the monotonic time going backwards is the same bug. It was observed on Intel arch. I suspect another way to paper over the compiler bug would be to switch the argument to from pass-by-value to pass-by-reference. I suspect a compiler bug would be a valuable thing to find. I am working on unit tests for this package. They will involve dependency on the Prometheus go client library, but nothing more. Will that be simple enough?
The incidents I have personally seen or helped users with time sync issues have all been VMs on Intel. If there are non-virtualized platforms with this issue it'd be interesting to see. Simply preventing the sampling time bucket from going backwards seems reasonable, which appears to be what the fix was attempting to do. Changing it to a pointer is an interesting idea. I can give it a try later this evening. I was figuring I'd have to run the entire Kubernetes test suite, so anything less than that is great!
Your small program is running on arm for minutes now without any errors. It needs more to trigger the compiler bug.
I will be able, though I will need instructions how to do that.
Here is a typescript of me running the new unit test: (the test does exercise the case of the clock going backwards a little) That's using the branch in
On ARM, I get similar results:
/remove-sig auth
: dang. Can you try it a few times? If that does not provoke the problem, perhaps you can edit the test to do a lot more iterations? You can also cut down on the warning noise by making a much smaller fraction of the cases have time going backwards.
Running multiple times results in some failing. Sometimes there are cases when 3 consecutive runs are successful. So basically, the test triggers the issue, but not deterministically. Even with iterations set to 1000, there are still successful test runs.
Is that close enough to submit as a compiler bug report? BTW, what do the failures look like?
is the logic non-flaky on AMD64? for a compiler bug, the maintainers are likely to request the failing code disassembled, so that they don't have to clone k8s, or a minimal code example that fails.
I am getting non-flaky results on this: I think I have gotten the test to be about as simple as makes sense for a kube unit test. I will have to leave the compiler bug hunting to people more familiar with ARM.
I tried to make a stdlib-only file that exercises the same call patterns w.r.t. variable scope as the original at what does that do when run on ARM?
Interesting, running the tests by on an rpi4 (using arm32) I am able to trigger two different issues: clock is NOT monotonic and can go backwards time IS corrupted periodically when being passed into I am also not able to trigger any issues running those tests on Darwin (using x86_64). However, running the stand-alone go file by I am not able to trigger any problems on the rpi4
: sorry if it wasn't clear, but the test introduced by uses a fake clock that is set backwards in some cases just to test how the metrics code reacts to that.
Same here, on an OrangePI armv7 board, the single stdlib-only test does not produce any errors.
Later I will do a test on arm64 as well.
woops, ah thank you I had to add to use the flag with go test to see those messages in Darwin and missed that part of the test code. At least the arm32 corruption issue is reproducible with that unit test, but I suspect that attempts to reduce the code space for the problem may cause it to go away, so maybe an issue of memory management or executable code layout. Unfortunately seems like debugging tools can be hard to come by on arm32 (eg no delve), am open to any ideas or suggestions on how to dig into it more
BTW, I updated to always log an info message explaining what's up with the warnings.
i don't know how to handle go mod imports with and some of the golang intrinsics are weird, but you could instead: build the unit test as a binary with the binary and for the object code of interest log a ticket @ golang and share the dissasembly you might also be able to use gdb over the test binary - set some breakpoints and try to catch what instruction is stomping the register / stack address that holds the value that ends up being bogus:
Thanks a bunch Was just looking into the compiled test, that is a great guide & will submit an issue with golang.
On arm64 the tests succeeded multiple times, no failures.
Compiled the test code for i386 resulted in: On i386 the test code triggers the bug too.
Different GOGC settings results in different test failure rate:
I am able to reproduce that. ... and if I build without the variant GOARCH then the test passes all 300 times.
When I merge and , the problem is visibly gone on the 386 arch on linux.
should we merge the pass-by-reference bit of in the meantime until golang/go is resolved?
I would say that we could wait a few days for response on the bug report. On the other side, I dont know how fast a resolution will get into a stable golang release, and what is needed for k8s to be built using that. Can these take a long time?
yes, I would not expect it faster than O(weeks), potentially O(months) if the golang investigation takes a while
It would be great if the fix could be merged and backported in time for 1.20.2; as it currently stands 1.20 is basically unusable on arm.
BTW, contains only the pass-by-reference change and logging of an error if there is a wild jump anyway; I would recommend taking all of that PR.
the deadline for cherry-picks to be merged for 1.20.2 is tomorrow
i wonder if it's also related w/ , the latter issue seems to be stambling on the memory pressure tho. but cpu&memory consumption can be also connected.
I think yes, I've seen similar log messages.
For what it is worth this seems like a very high impact bug, as it can potentially effect 64-bit architectures also. The following example of the bug that was submitted as part of the fix also fails on Darwin/x86_64: It seems like the best mitigation until a new version of golang is released is to disable optimizations for the compiled binaries.
I created as a cut-down alternative to , containing only the work-around and not the checking and logging for the argument passing problem.
: What do you recommend for how to proceed on this? We should probably get something into 1.20.2
agree. I'm fine with a temporary commented pass-by-reference workaround.
/triage accepted
ping , On arm64 I got the same error like . and kube-api-server restart many times.
the error you are getting looks like a connection problem, so not clear if related. the bug discussed here is in golang. so if we update to the latest PATCH version golang in our k8s PATCH versions it will be resolved.
thanks. but In my 3 masters kubernetse clusters. Only one master got this Error . others works well. this confused me. this problem like connection working with ipv6 not ipv4?
you can try asking in the #sig-network channel on slack or log a new issue.


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Creating a pod with a nested subpath volume will cause the pod to fail to start on Windows containerd nodes. What you expected to happen: The pods should start normally. How to reproduce it (as minimally and precisely as possible): Create the following pod on a Windows node with containerd: apiVersion: v1 kind: Pod metadata: annotations: e2e-test-privileged-psp name: pod-subpath-test-downwardapi spec: containers: args: mounttest --filecontentinloop=/test-volume --retrytime=20 image: claudiubelu/agnhost:2.21 name: test-container-subpath-downwardapi volumeMounts: mountPath: /test-volume name: test-volume subPath: downward/podname volumes: downwardAPI: defaultMode: 420 items: fieldRef: apiVersion: v1 fieldPath: path: downward/podname name: test-volume Anything else we need to know?: This issue is causing the following tests to fail on Windows Containerd nodes: [sig-storage] Subpath Atomic writer volumes should support subpaths with downward pod [LinuxOnly] [Conformance] [sig-storage] Subpath Atomic writer volumes should support subpaths with projected pod [LinuxOnly] [Conformance] The tests have the ` tag because we did not support Windows Containerd nodes at the time, and the tests still do not work for Windows Docker nodes. After a fix for this bug is merged, these tests will have to be renamed to enable them for Windows Containerd environments but not for Windows Docker environments. /sig windows Environment: Kubernetes version (use ): v1.21.0-alpha.0.493+ Cloud provider or hardware configuration: N/A OS (e.g: ): Windows Server 1809, Containerd Kernel (e.g. ): 10.0. N/A Build Install tools: N/A Network plugin and version (if this is a network-related bug): N/A Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale


<!-- Please only use this template for submitting enhancement requests --What would you like to be : Remove the folder in favor of . Why is this needed: With the refactoring from the Netpol validation , the tests from folder are deprecated and duplicated at this point. Action items: tests tests
/area testing /sig network
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /area testing /sig network Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
thank you ... !
/triage accepted


Many e2e tests today have a tag in their name, which lets us know which sig is responsible for fixing failing or flaky e2e tests. However, not all tests have such a tag. We should: assign tags to all e2e tests that don't yet have them enforce that no new e2e tests can be without a tag in their name EDIT: proposed path forward (pulled out of and ) rearrange e2e tests into sig-owned packages, - e2e tests that run as part of nodee2e - e2e tests that run as part of e2e and nodee2e add that imports sub-packages so e2e binaries can continue to import to pick up test definitions () ensure each sig-owned package has: a package-local () an OWNERS files with sig members and ( follow these conventions: top-level calls are required, nested calls are not allowed remove usage of , , , and in favor of nested calls are ok, nested calls are not enforce these conventions via some combination of: static analysis regex enforcement against names output during a run while we are here, is vestigial and can be removed (ref: ()) EDIT: Everything now conforms to policies defined above will close this out by adding a blocking test and docs to /sig testing /area test for enforcement (FYI /sig architecture /area conformance some of these are conformance tests (FYI /wg reliability for helping determine appropriate sig ownership (FYI ) /sig release since this is relevant to figuring out who should fix release-blocking tests (FYI /priority important-soon /milestone v1.21 I think it's reasonable to aim to accomplish this prior to code freeze for v1.21
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : Many e2e tests today have a tag in their name, which lets us know which sig is responsible for fixing failing or flaky e2e tests. assign tags to all e2e tests that don't yet have them enforce that no new e2e tests can be without a tag in their name /sig testing /area test for enforcement (FYI /sig architecture /area conformance some of these are conformance tests (FYI /wg reliability for helping determine appropriate sig ownership (FYI ) /sig release since this is relevant to figuring out who should fix release-blocking tests (FYI /priority important-soon /milestone v1.21 I think it's reasonable to aim to accomplish this prior to code freeze for v1.21 Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign Assigning an arbitrary lead from each SIG/WG mentioned, please assign to someone more appropriate if you don't have the bandwidth
I can't see anything sig-network related :thinking: there are a lot of NodeConformance things though ,
/assign for visibility - lots of these are node /sig node
/assign (for sig-node)
I've just sent out for autoscaling tests. It seems that most of the remaining tests are node-related. We should try to fix most of these to see if there is anything that is hidden between those.
/unassign
Sent to address rest of the tests. Echoing the review comment from I agree, we can remove prefix once all the tests are tagged. wdyt ? I'll raise a follow up to clean the prefix.
re: /assign (also maybe knows) at one time I thought Openshift was relying on to filter out upstream tests for certain jobs? I don't see that it's consistently applied some I'm assuming we could drop from our tests? (I said as much here: )
We synthetically apply that (if the source package is in ) so can be dropped. Also note in generally I am massively in favor of this change - I forced this to happen in openshift so some of the unassigned upstream tests got blanket assigned to equivalent teams (i.e. any test in that didn't have a sig had to get one synthetically ) so this will remove the need for it. Having owners just clarified everything, and then the ratcheting "you may not add one without" also ensured folks thought even a tiny bit about ownership.
Fleshing out what I suggested in and Can we agree to do the following? rearrange e2e tests into sig-owned packages, - e2e tests that run as part of nodee2e - e2e tests that run as part of e2e and node_e2e ensure each sig-owned package has: a package-local () an OWNERS files with sig members and ( remove usage of , , , and in favor of add that imports sub-packages so e2e binaries can continue to import to pick up test definitions () Enforcing test ownership then looks like: an e2e test must live in a sig-owned package (or subpackage) an e2e test must be defined by that package's PRs to that package will be assigned / auto-labeled to the appropriate SIG changing test ownership is a move from one package to another, requires approval from both the old SIG and new SIG Implications I don't have the foresight to see as blockers, can you think of any? we disallow the concept of e2e packages containing tests owned by multiple sigs some moves may require imports to change or things like helper functions and fixtures to get extracted Folks seemed to like the idea at the time but I'm seeing PRs move in different directions. How can I help get us on the same page?
+1 for that I think that the PR above as an example is an intermediate step (it's not a different direction, rather just one step instead of multiple at once).
There is one question that I have at this point - what do we do with nested Described, e.g.: this one: [there is KubeDescribe nested in SIGDescribe]
I think that having SIGDescribe twice in those cases doesn't seem to be the best option, and those cases seem to be significant percentage of remaining cases...
I don't have strong opinion here. Mostly curious why prefer the non-explicit variant? Is it to optimize for tests migrations?
There is one question that I have at this point - what do we do with nested Described I can't think of a strong reason to ban them, and I think it would be more trouble than it's worth to come up with alternatives for how they're used now. I can't think of a compile-time way to enforce that "[sig-foo]" doesn't show up in their text, so I'm not sure it's worth keeping/enforcing use of wrappers (they definitely make error reporting worse) So something like: replace nested calls with follow these conventions: top-level calls are required, nested calls are not allowed nested calls are ok, nested calls are not enforcement of these conventions can be done via some combination of static analysis and regex check against the full names generated by a run
That makes sense to me. I will start looking into migrating existing tests to satisfy these rules in free cycle, but i'm leaving the enforcement to you :)
I do consider not having to modify test code to change ownership a benefit. But it's more that we'd have to change a lot more code to conform to than . Since we mostly already have the pattern in place, I'd rather stick to it.
Thinking of timeline (ref: ) This work can continue past Code Freeze on Mar 9, but that's a useful milestone to evaluate our progress I'll update the issue description as we get tests owned My goal is to get enforcement landed sometime after Code Freeze This work can't continue past Test Freeze on Mar 24
- also one more thing to do would be to ensure that no tests are tagged with more then one SIG (which I think isn't true now for a few tests)
also one more thing to do would be to ensure that no tests are tagged with more then one SIG (which I think isn't true now for a few tests) Good point. There are tinier corner cases, but let's start with the big one. /sig storage /sig windows /assign There are a bunch of storage tests that look like the following: Which SIG should own these?
We make us of for many e2e tests that are intended to run on windows machines. My suggestion here is probably to have sig-storage own these tests as they reside under /test/e2e/storage/* and switch to using instead of in the test names. SIG-windows will always be happy to help debug issues here. - thought?
I think it makes sense for storage to own the windows storage tests. Using [Feature:Windows] seems reasonable although we'll have to update a number of test jobs.
Working on in-tree enforcement before I update description. tl;dr the majority of tests listed in the description are still not tagged (with at least one SIG), waiting on SIG Node to sign off on usage of KubeDescribe has gone way down, but not at zero yet
is the biggest bit now (once I get feedback from SIG node, we will be almost there with both)
refreshing description based on script in
Regarding "multi-ownership", with windows test figured (the PR didn't merge yet, but hopefully will soon), the biggest question is now about the upgrade tests. - there are a number of apps related upgrade tests. Even though they are currently running on GCP, I suggest that we officially make them sig-apps tests (I'm sure sig-cloud-provider-gcp will be happy to help on infrastructural problems). Thoughts?
can you be more specific about which particular tests? I quickly checked and those are already marked as sig-apps. I don't have any objections as long as they are part of sig-apps territory but I'd like to know a concrete list before saying so :wink:
- sorry, I should have been more specific. Many of those tests are actually being run from here: and the tests there are marked as [sig-cloud-provider-gcp] As part of ensuring that each test has a one single sig ownership, I suggest making it explicit that all of those are sig-apps. Does that make more sense now?
most - yes, but I see a few exceptions: upgradeTest: ServiceUpgradeTest - seems like sig-network SecretUpgradeTest - looks like assigned to sig-storage/sig-apimachinery ConfigMapUpgradeTest - looks like assigned to sig-storage/sig-apimachinery HPAUpgradeTest - sig-autoscaling PersistentVolumeUpgradeTest - sig-storage AppArmorUpgradeTest - sig-auth (?) VolumeModeDowngradeTest - sig-storage gpuUpgradeTests - looks like sig-node/sig-scheduling kubeProxyUpgradeTests - both sig-network kubeProxyDowngradeTests - both sig-network serviceaccountAdmissionControllerMigrationTests - looks like sig-apimachinery the remaining less than half :+1: for sig-apps.
Sure - I didn't mean to put all of them to sig-apps. There is just the biggest group of apps related one so wanted to start with this group.
I'm just making sure :wink: Go for it and thx :100:
Hello! The code freeze is starting March 9th, 2021 (about 1 week from now). As the Issue is tagged for 1.21, is it still planned for this release?
Yes this is planned for this release, and PRs related to this are not subject to code freeze. This will be completed by test freeze (see timeline here: )
- calling your script from head, I have: So there are just couple exceptions now (and a mess with upgrade tests :) )
- when you have time, there are couple PRs waiting for you now
Not reviewing on the weekend, but will check them out first thing next week. Also ok if someone else gets to them before me
Run in master. The is due to The are all from It seems it is done.
The two opened PRs + the one that I have internally (close to ready, though dependent on the firs from the above) should actually close this.
will close this out by adding a blocking test and docs to
Nice work ... that jq stuff is something :)
/triage accepted


is a very useful option in scheduler perf test framework to define an operation without waiting for it to finish. However, the current testing logic still waits for all the pods in this operation to be schedulable, in the last step: the number for Pods in this operation to be schedulable This can be problematic as some case just want to generate Pods (for example, to test internal queuing or sorting behavior/latency). The following test command times out as those large-cpu Pods are expected to not fit on any node: So, given is commented as: I'm proposing to relax the semantics to not wait for the Pods to be schedulable if the operation is specified as . /sig scheduling
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign WDYT?


What happened: Sweeping uses of extensions/v1beta1 in preparation for , the addon manager stood out as using extensions/v1beta1 Ingress by default: What you expected to happen: All components shipped in 1.21 must use Ingress in order to work properly against a 1.22 API server. /sig networking cc per cluster/addons/addon-manager/OWNERS cc I'm not sure what the addon manager does if given a prune list of Ingress objects while still using a manifest containing extensions/v1 Ingress objects... this needs to be tested. /milestone v1.21 /priority important-soon /lifecycle frozen
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : What happened: cc /priority important-soon /lifecycle frozen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig network
/triage accepted
how "urgent" is this? Can wait until next tuesday? Otherwise I can work on the tests this weekend :)
important, not urgent. it should happen early in the 1.21 cycle, but does not require work out of business hours
Hello! I am from the Bug Triage team! This issue has not been updated for a few weeks, so I'd like to check on the status. Code freeze is starting March 9th, 2021 (about 1 week from now). We want to ensure that each PR has a chance to be merged before code freeze. As the Issue is tagged for 1.21, is it still planned for this release?
we're waiting to the PR to get merged :) I guess it's important for the v1.21 release, for the sake of extensions apiGroup deprecation :) Will ping reviewers on the PR


When both and parameters are set on Kubelet, they act as the upper and lower thresholds that define how the state is entered and exited. When trying to remediate a , Kubelet performs different operations to free up space. In order to decide if it has freed up enough, it compares the currently available disk space with the triggering threshold instead of the exiting threshold. Kubelet ends up in a long lasting state which can only be left with Kubelet restart, or if by luck the Pods that were using disk space are rescheduled somewhere else. Kubelet periodically does some housekeeping in order to ensure its node has enough resources available. If that isn’t the case, it tries to free up resources, updates the node’s condition (e.g. ) and in last resort starts evicting Pods. By default, the housekeeping is done every 10 seconds for a set of “signals” which represent node level resources (e.g. , , , ). There are a couple of parameters that comes into play in this process and I'll ignore all but two for simplicity: Signal thresholds: defines at which point there is not enough of a resource available, e.g. Signal minimum reclaim: how much resources Kubelet should aim to free up before a threshold is considered as not met. It’s the lower bound of an hysteresis, the threshold being the upper bound (e.g. ) The housekeeping is implemented in Here is my understanding of what this method does, filtering out what I think does matter in this issue: the signals (e.g. how much is available) a list of thresholds that are met (e.g. observed < ) there were thresholds met on a previous run: a. Check if those previous threshold are still met taking the into account. If = 15% and , our value to unmet the threshold is actually 25% percent b. Merge them with (2), the thresholds that are met from the new observation (1) out thresholds for which the individual grace period is not reached. They might be taken into account in another run if observed again. the previously met threshold with the new computed list of thresholds. Those met threshold are stored across runs. no threshold is not actually met, exits. If at least one is met, Kubelet has work to do! Try to reclaim node level resources. In other terms, try to free up resources without having to evict any Pod. For that’s done by removing stopped containers, and unused images. a new observation. If the threshold are not met anymore, ignoring , exit with Pods as last resort Ignoring the in (7) seems wrong. If the amount of resources freed is somewhere between and , Kubelet can get stuck. On the next run, as part of (3a) the threshold taking into account is not unmet. Therefore Kubelet tries again to free up node level resources in (6), eventually does nothing but exits (6) pretending it free up enough resources without doing anything. after having cleaned up the node and in order to decide if Pod eviction is the step, seems to do the trick. Kubelet should try to get out of a by evicting almost all Pods. In this case, it means it should try to evict Pods until the threshold is reached and the can be removed. I expect Kubelet to stay in permanent only if the resources that are used are beyond its control. Preparation: a node with a different from (the problem might be the same when they are mixed but I didn't try) to 15% and to 30% in the Kubelet’s configuration into a Pod running on that node, and use to write a file inside the container. The amount of data should be enough to bring the underlying disk at 86% usage for Kubelet to notice the threshold and to cleanup stopped containers and remove unused images Remove your file. The node is now clean. Restart Kubelet and notice how it sees everything is normal Getting Kubelet stuck: into your Pod and write a file to reach 86% usage again for Kubelet to notice, cleanup stopped containers and images. Since the cleanup was done in the preparation step a minute ago, there is almost nothing to be removed and your usage should stay more or less the same the Pod's file by a new one to reach between 55% and 85% disk usage. how every 10 seconds, Kubelet pretends it was able to . The Node is stuck in until you restart Kubelet and it forgets it had to ensure there is at least available in . Kubernetes version (use ): 1.18.8 Cloud provider or hardware configuration: vSphere OS (e.g: ): 18.04.2 LTS Kernel (e.g. ): 4.15.0-130-generic We have been running a slightly modified version of Kubelet on one of our clusters for the past 5 days, and since then we had no node staying in state for more than 20 minutes. From the disk usage graph, it looks like both image garbage collection and the disk pressure remedy operations are working properly. I'll open a PR with the little patch we're using.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node


<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --/sig windows /cc After was merged I think.
I agree that looks like the issue. It started failing between the two commits: on this run:
/triage accepted
/assign


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --Start the scheduler with a scheduler config file which has a custom configuration for a specific plugin, the scheduler will fail to start and throw below exception, kube- error message, custom plugin config should take precedence over default plugin config. start a scheduler with a config file. Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scheduling /assign
Does disabling the plugin and re-enabling the same plugin with a different weight work as a workaround?
This is WAI. You have to explicitly disable the default configuration.
I think so.
I not sure I agree with this, particularly, for someone who doesn't know details of code IMPL, e.g. an operator, Why bother to disable the plugin first and then re-enable the same plugin? there is no pointer to show they are different in a config file, as a developer, we know the reason behind the scenes, can we assume operator also know what that means to define a schedule config? By disabling the plugin and re-enable it means the operator has the knowledge this plugin is a default enabled plugin, should the operator check the source code to confirm this plugin's configuration? When this happens, given a different config for a specific plugin, should this deserve a corruption? I think scheduler should more robust to tolerate this.
We can improve the docs to include the default plugins. It's better for the API to be more predictable (always do the same thing). One thing to think about is order: if we do what your PR proposes, we are altering the order in which plugins are run, not just modifying the "weight" of the plugin.
I'm leaning towards closing this ?
It's the key point I'm concerned about: commented here So let me ask a question: for a given input like: What's the expected output? {DefaultPlugin1, 1}, {DefaultPlugin2, 200}, {DefaultPlugin3, 3}, OR {DefaultPlugin1, 1}, {DefaultPlugin3, 3}, {DefaultPlugin2, 200}
Today, it is 2.
So in other words, if a user wants to customize a plugin's weight and keep its order, the only way is to use and give a full list of plugins? If so, that's not a good UX IMO.
Well, order doesn't matter for Score, so not really a problem.
All right, so the only case users could disable PluginXYZ while also enabling PluginXYZ is Score plugins, I'm fine with the order issue. Back to this issue, my 2 cents is to change the semantics of to a more enforced manner (proposed in this issue) in a new API version - v1beta2 or GA. As IMO it's a real UX improvement. WDYT?
I'm not convinced it's a UX improvement. If the user "enables" a filter plugin that was a default (say, by mistake), then its order changes. At least first disabling the plugin makes it more explicit. cc who has been working on decisions based on CC version
I guess it depends on the implementation. What is the full proposal?
/remove-kind bug /kind feature
That's a fair point, esp. if we adjust default plugins from version to version. I think a good UX is to throw an error/warning to indicate that a default Filter plugin is being re-enabled - this's useful no matter which semantic employes.
I'd like to: keep current logic to warn users that "PluginXYZ" is already registered (by default), if it's a non-Score plugin apply the "overridden" semantics to directive, for Score plugins only btw: let's get a consensus first and better adopt it in new CC version.
I propose (only for new version): Fail when plugin is already registered in the custom configuration, and only warning if the plugin was already part of defaults. Override any weight, preserving order I think that keeps everything consistent. I think this is what the last iteration of the PR is doing, but I only skimmed through it.
SGTM. To confirm, by "Fail when plugin is already registered in the custom configuration", this's only possible when users implement an out-of-tree plugin with a name identical to a default one, right?
I'm just referring to failing when the user repeats the same value twice. I think we can't prevent what you say.
This might be the reason why the order of the plugins should be reserved.
Unless it's score plugin like this issue, I don't know if there is any point to enable the default plugin in a config file. I think there is no need to distinguish score plugin with others, this will make the life easier, correct me?
Yeah, overall, is following this ideas, except that PR doesn't handle the repeated configuration, because if there is repeated value after the default plugin is overridden, it will hit the error below,
thanks for those valuable suggestion! So, we are still with beta1, and you suggest we can make some change in the new version, beta2? Is there any plan when this could happen? v1.22?
You are correct, there is no point, but it's good to have consistency. If it's allowed for Score, it should be allowed for other extension points. I agree Yes, that is the plan. It was 1.21, but we fall behind due to newer plans for deprecating plugins
let's say there are 3 default non-Score plugins A, B and C (in that order). I want to move B to the end of the list but I want to custom enable C too (for whatever reason; since we want to consistent with score plugins, let's allow this for other extension points too as said). if a non-Score plugin is custom disabled AND enabled, we should move it to the end of the list but if it's simply just enabled, then we should preserve order. if it's a score plugin, disabling and re-enabling has the same semantics as just simply enabling; they both will only change the weight. (idk if this has already been proposed; if it has been, consider this a vote of support :))
Are you saying that "disabled and enabled" should be treated the same as just "enabled"? That is, always move to the end of the list?
No, just "enabled" would NOT move things to the end of the list (it will preserve order), but "disabled and enabled" would move to the end of the list.
ok, that sounds good
Another problem here is if we don't want to overridden anything but just config a plugin which is accidently a default plugin, something like below, scheduler will fail as well, I think we need to De-dup here instead of just a warning here. I guess this PR should already cover this, I will go back to check it again, at least one more testcase is needed to cover this case. keep current logic to warn users that "PluginXYZ" is already registered (by default), if it's a non-Score plugin
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/lifecycle stale let me revive the PR later.
/remove-lifecycle stale


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --kube-up tries to add metadata to the wrong project. It is trying to add metadata to the prow project running the tests instead of the target project. This is because is missing a kube-up uses the target project to add metadata Don't set/set a different default project and run cluster/kube-up. originally in kubetest hides the project acquired from boskos under the covers by setting it as the default project uncovered as part of migration to kubetest2 xref: /cc
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --When is passed a ref while logging, it panics: or an empty reference should be logged. I have fixed this in so we will need to bump klog. ht for reporting in /priority critical-urgent /sig instrumentation
need update klog dependencies later. milestone 1.21
/milestone v1.21 Will set the milestone for the issue and let people review the PR during regular burndown.
Scope check: potentially affected components are: Kubelet Controller manager Scheduler (grep for KObj: )
I looked in current implementation of handling object. Please remember how checking works. Current fix should work on simple nil (): but it will fail on structs implementing KMetadata interface, but not handling nil pointer (). Reason is that interface with nil value is not nil. Resulting call to 'Log' function will panic on call. In stacktrace presented above it was function that panicked which leads to thinking that the problem is in function which panics if pointer is empty. Potential solutions: Replace all calls with until we solve this issue Fix this particular case where pod can be nil (solves only local problem, but not the long term issue that every log call can be a panic) Use reflection to verify if object under interface is an nil (expected overhead EDIT: got results only 7% increased cpu usage) Fix all and functions to check for nil (not enough time before the freeze, some overhead on calls) Do runtime checks for K8s types (would require klog to import api types) Looking at all options I think that first one makes most sense as all other have bad impact on whole k/k repo
Did benchmarking for using reflections, looks like performance overhead is not as significant as I previously anticipated, around 20%. I think we could consider using reflections.
/cc For feedback
/cc
There are hundreds of instances of in k/k now, and every single one would need to be carefully checked when obtaining the pod name/namespace, and we'd need a top-level approver to do the replacement. I think this really needs to be fixed in klog.
With benchmark results being not as bad as first estimated I think we could use reflection in klog like proposed in kubernetes/klog
/reopen needs to get bumped in k/k to close
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/triage accepted


The method doesn't check taints and tolerations. This is problematic in 2 scenarios: If the users bypass scheduler by setting a value in If the node restarts. The node would try to restart scheduled pods once it comes back up, even if it's not ready (has a not-ready taint). Kubelet to check taints and tolerations. a taint in a node. a Pod with set to the Node above, without a matching toleration. /sig node /sig scheduling
this reminds me of your kubelet sync PR ...
I don't think this is a syncing issue. The logic doesn't check taints at all.
scenario 1 is pretty bad, i will let the #sig-node slack know about this.
It doesn't sound like a syncing issue, but we seem to have multiple issues with pod admission/management on restart currently
I'm aware of that issue. This should be unrelated.
I would like to address this. /assign
I think it would be great to proceed after merging ref:
Please add me as reviewer
/triage accepted
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale PR in review:
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale


[x] test approval issue : [X] test pr : 26 May [x] two weeks soak end date : 9 June [x] test promotion pr : kubernetes/kubernetes According to this APIsnoop query, there are still some remaining DaemonSet endpoints which are untested. Create a watch to track daemon set events that DaemonSet with a static label Create a daemon set with a static label. Confirm that the pods are running. Get the daemon set status. Parse the response and confirm that the daemon set status conditions can be listed. Update the daemon set status. Confirm via the watch that the status has been updated. Patch the daemon set status. Confirm via the watch that the status has been patched. Using an existing as a template for a new which validates that three new endpoints are hit. This query shows the endpoints hit within a short period of running the e2e test If a test with these calls gets merged, test coverage will go up by 3 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/triage accepted
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Originally tracked at . After fixing "not include test files in import-boss" in , we have a regression here: is using . We need remove dependencies from integration tests to e2e tests again. /assign /sig testing
/sig auth as per the required change
/kind cleanup /area test
It would be helpful to outline an argument as to why this is the case. I can imagine reasons, but none are mentioned here.
Description updated.
/triage accepted /priority backlog /milestone v1.21 This seems a simple enough fix to land (and thank you for doing this) but marking low enough priority to make it clear we may back out and wait for v1.22 if we don't have bandwidth for this
So while I think we should probably do this anyhow I want to note: This issue lists the reasons: we have no intention to stage e2e (there's been much discussion about this since then) we generally don't want anything non-informational (required jobs) running only sometimes anyhow, it's just not reliable Decoupling these seems like a good idea from a purely code-organization perspective, but the reasons in the previous issue are no longer valid.
/milestone v1.22 The PR didn't land in time. At this point I view this as nice to have. Even if it's low impact, let's wait until next release.


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --e2e test fails with ECDSA Service Account key e2e test to succeed with ECDSA Service Account key Use ECDSA key for the service account key. Logs from the started test container: library defaults to by default as the only supported method while should be explicitly: Relevant code: On the API server side: Kubernetes version (use ): 1.21.0-rc.0 Cloud provider or hardware configuration: OS (e.g: ): Talos Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig testing
Changing the service account key type to RSA fixes the test:
this is a problem for us, as this test is part of test suite, and it fails with 1.21-rc.0 e2e tests
SIG Auth own their test cases (it's tagged in the test name), SIG Architecture own conformance. Testing owns the frameworks, CI, tooling, etc. EDIT: just clarifying and making sure the right people see it. I raised this to the conformance work group.
/assign /priority critical-urgent /milestone v1.21 because this is in the conformance suite, it must not restrict valid+supported cluster configurations
/reopen the image needs to be promoted and bumped in the e2e test
Reopened this issue. <detailsIn response to : /reopen the image needs to be promoted and bumped in the e2e test Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
promote image PR:
I believe with merged this can be closed as all the fixes were done and backported to 1.21
Thanks!


<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --pull-kubernetes-e2e-aks-engine-azure-file pull-kubernetes-e2e-azure-file-windows AzureFile CSI Driver End-to-End Tests: Dynamic Provisioning should create a volume on demand with mount options [] [] [Windows] AzureFile CSI Driver End-to-End Tests: Dynamic Provisioning should create multiple PV objects, bind to PVCs and attach all to different pods on the same node [] [] [Windows] AzureFile CSI Driver End-to-End Tests: Dynamic Provisioning should create a volume on demand and mount it as readOnly in a pod [] [] [Windows] If the randomly generated kubetest resource group has capital letters, the share name will contain invalid characters and fail provisioning. links to appreciated links to specific failures in spyglass appreciated <!-- Please see the deflaking doc () for more guidance! --
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


With recent migration of Kubelet to structured logging introduced e2e tests that enable JSON in Kubelet. Organized migration of other components haven't yet started and is on best effort bases, yet some line were already migrated. As code-base will be slowly migrated, we need to expand the test to avoid errors like I propose to expand existing e2e test suite ( and ) to cover all control plane components (apiserver, controller-manager and scheduler). /cc
/sig instrumentation
/help
If all components have this flag do we just want to enable it on all components? We already increase the log level on all components in CI, it should be pretty straightforward to expand this to all: pick a new environment variable name for the script with a more appropriate name for all components instead of just kubelet set it on the job in addition to the old kubelet one, by sending a PR to test-infra send a PR to the kind script to set the value in all places where we set verbosity instead of just kubelet, switching to the new variable name
/triage accepted
/area logging
So currently the e2e tests do not actually read and test what the logs return the current job just makes sure that kubelet is running Would we want to write similar tests for the different control planes while we start migration or do you have something else in mind with respect to the automated testing ?
Yes, currently we just test if Kubelet is running to test that flag is working and if enabling json doesn't cause panics. This exact issue proposes to expand this to other control plane, but your idea would be a great follow up. Would be great to propose a test plan that will specify what we want to test and how. If you have ideas some ideas could you collect hem in some issue so we can discuss.
There's no portable way to get these logs. There are non portable ways but those cannot be used in a standard e2e test.
Oh so for the standard e2e tests it might be good enough to just test the startups of these control planes with the logging enabled.
I was thinking that we could get started with maybe writing similar to one for the remaining components and then maybe find a way of checking the logs generated as we have bigger discussions a) kube-apiserver --Completed b) etcd --- TODO c) kube-scheduler ---TODO d) kube-controller-manager --TODO
sound good. FYI Etcd is not a kubernetes component as so it's out of scope of this issue.
/wg structured-logging As getting core components logs is not portable, we could test a container with same logging library and using Kubenetes logs API to access logs. Created to track it
it still might not be a terrible idea to write a simple test script to probe the exported logs from a cluster with json logging enabled for any non-json lines once we know that everything should be. I think this could probably be hacked up with jq or something, but I wouldn't position it like a normal e2e test.
Could you give a hint how to implement and run such script?
/priority important-soon This issue is based on feedback from Kubelet migration retrospection, it is required before organizing migration of next component. /cc
Would this follow the same flow as in where we check if json logging is enabled and then run jq separately or within the same flow. To be more specific: Seems to be getting the data and validating it so in that case would we be running after grabbing the logs ?
For now let's focus on the original tasks of having ( and ) test scenarios, also configure other components. We just need to update Kind script to also pass to other components. Example:
Would like to take this one! /cc
Great, let me know if you need any help? /remove-help
just created a PR:
Components to be picked: [x] kube-scheduler [ ] kube-apiserver [ ] controller-manager
Attempted to cover the remaining components in this
It would be great if we had e2e setup for all components before we start structured migration next week. will you be able to work on it this week? Please let me know if there is anything I can help with.
Thanks for help with control plane components, for 1.23 we are planning to migrate kube-proxy to structured logging. Could you also try to enable JSON logging on kube-proxy?
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Confirmed that in last CI run Logs from kubelet and control plane components are written in JSON: Great Job Could you take a look into changing log format of kube-proxy?
Filed for kube-proxy. Are we good to close this issue ?
sg


<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --pull-kubernetes-integration runkubectldiff_tests It only shows failure cases in recent 2 days. The last date is I tried to reproduce it locally, but can't
/sig testing
/sig cli
/cc
I’ve only briefly taken a look at this, but my understanding is that server-side diff is flaking because the timestamp is different (by like a second) between the one already applied and the diff apply. do you think this could be related to this recent change Could delaying when we update the timestamp cause it to sometimes get misaligned with what’s being diffed? I’m not super familiar with diff, but it feels like it should ignore differences between timestamps that get updated on each call, right? Is there some sort of mechanism we have in that knows to ignore timestamps that just isn’t being done for fields?
This error probably shows that we have a path that causes the object to be re-written to etcd when it shouldn't.
As we can see, the diff only shows a difference in the timestamp but not in the fields that are managed. We are not supposed to update the timestamp if the managed fields haven't changed. That timestamp only gets updated if the two steps happen in different seconds (that's why it's flaky). That is definitely not a flake but an actual failure.
Failed 81/236 times today in /priority critical-urgent I think, correct it if I am wrong.
/triage accepted
correct. who should be assigned as the owner of this?
I can take it, I'm looking into it now
/assign


<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --k8s-e2e-ltsc2019-containerd-flannel-sdnoverlay-master [] Container Lifecycle Hook when create a pod with lifecycle hook should execute prestart http hook properly [NodeConformance] [Conformance] [] Container Lifecycle Hook when create a pod with lifecycle hook should execute poststart http hook properly [NodeConformance] [Conformance] Note: the tests have been excluded until they are fixed. In the case of multinode clusters, the http server pod and the test cluster can spawn on different nodes, which can be problematic for poststart / prestop hooks, as they are executed by the kubelet itself, and the cross-node lifecycle hook might fail (according to the [Kubernetes network model], it is not mandatory for kubelet to be able to access pods on a different node). However, that is not always the case, different Networking setups allow cross-node communication between kubelet and pods, which means that the above tests are passing properly (for example: ), which begs the question if this behaviour should be universal or not, especially since the tests are marked as , which would imply that it should work in all cases. [Kubernetes network model]: links to appreciated links to specific failures in spyglass appreciated Slack: /sig testing /sig networking /sig windows <!-- Please see the deflaking doc () for more guidance! --
/sig network /sig node /area conformance /assign /priority critical-urgent /cc This is important since affects conformance and defines the networking behaviour for the poststart/ prestop hooks
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /sig network /sig node /sig conformance /assign /priority critical-urgent /cc Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
i'd personally let folks have first dibs on what we need to do here. Though from the name of the test itself, it seems like the test is NOT a sig-network test :)
Indeed, it does not have any tag in its name, but the tests rely on networking and on how CNIs work. As I've stated above, it's not a problem for flannel sdnbridge, but it is for flannel sdnoverlay; other CNIs might have a similar issue too.
because the network is not a thing :) let´s tag the sig owner then /sig node I think that the key here is to decide if kubelet should be able to access pod in other nodes, and based on the link Claudiu is part of the Kubernetes networking model. I saw users that doesn't want kubelet to access even the pods on same host, so I think that is legit to say kubelet MAY reach pods in other nodes but not MUST ... but I'm not familiar with the to fully understand the context. If kubelet "may" reach pods in the other nodes, the test must only test lifecycle hooks for pods in the same node as the kubelet under test, right?
This is correct. The model also says: "For those platforms that support Pods running in the host network (e.g. Linux): pods in the host network of a node can communicate with all pods on all nodes without NAT" I believe that Windows host network is not supported, right?
Right, and there are specific tests that checks that exact scenario you've mentioned: However, the flaky tests I've mentioned in the issue require kubelet to be able to dial-in into a webserver pod for a different pod's poststart / prestop HTTP lifecycle hooks. There are couple of things regarding this: of all, in order to ensure kubelet would be able to run the HTTP lifecycle hooks, we'd have to make sure both pods are on the same node, as that is a supported scenario, according to the Kubernetes Network Model. I have sent a PR for this above. Alternatively, a Cluster IP could probably work as well, but IMO, ensuring that both pods end up on the same node is simpler and easier to clean up. , it's interesting that those HTTP lifecycle hook tests work on different networking setups, without any pods having HostNetwork enabled, even though you've mentioned they should (e.g.: Linux / Windows with flannel l2bridge, and even Linux with flannel overlay). Now, my question is: is this behaviour a feature, or a bug, especially since the tests are marked as `?
oh, I missed that part, so in Linux kubelet must be able to communicate with pods on all nodes (becuase it is running in the hostnetwork) If that is correct, it seems that one solution is to keep the test for Linux and tag it as LinuxOnly, and a new test for WindowsOnly that only spawn the pods under test in the same node EDIT Discussed in sig-net meeting , no clear answer... can be kubelet considered as a hostnetwork pod?
For the purposes of testing, I don't mind forcing the pods onto the same node., but let's document the heck out of that at the code-site?
correct but it will be supported once HostProcess containers are merged in 1.22: Only in Overlay networks (ie Flannel where these tests are flaking) does the pod have to be on the same node. It doesn't require host network mode and these tests are . I would be more inclined to disable this test for Windows Overlay test setups and have a sig-windows test for the overlay functionality. I would also question if the lifecycle hooks only running on the same node is useful and if it should just be documented that it doesn't work in the Windows Overlay setup.
is looking at test PRs from SIG Node. I will let sig-net triage this issue
Next steps on this?
/triage accepted
We still haven't merged this: I also had this, but it has been decided to be blocked:
requires conformance approval, at this point I think that has to be postponed to 1.23 right, since that behavior is not required, we shouldn't test it, because is legit to fail and having a test for something that can fail can create confusion :smile:


kubelet logs misleading when volume limits not an
/sig storage /tirage accepted /assign
Note that it is set as info level in the log above: However, this is definitely confusing. /remove-kind bug /kind cleanup /triage accepted /sig node
I also have this problem Excuse me, how did you solve it
Which version do you use .. This was fixed here Check k8s version you use please


The test only validates that the API objects are created E2e tests must verify the behaviour, i.e. that the Service and Endpoints actually work sending traffic. The file is not well organized and there is a lot of duplicate code, one possibility is to use current Services helper has a function to test the reachability that iterates over all service ports However, something that needs to be addressed is that the pods that are backing the service doesn't have any process listening on those ports
/help wanted /sig network /priority important-soon /triage accepted
I think that there are more tests that doesn't verify the reachability , those should be modified too /help-wanted
/help-wanted
/assign
/triage accepted
/assign /unassign I will take on this since it was more difficult than I expected, and is a Conformance test.. I assigned you to , that is similar to this, but creating a new test from scratch


<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --post-kubernetes-push-e2e-agnhost-test-images Image build unhappy on s390x arch: Building image for agnhost OS/ARCH: linux/s390x... [...] [ 3/10] RUN apk --update add bind-tools curl netcat-openbsd iproute2 ipe... 0.165 fetch http://dl- 0.725 fetch http://dl- 1.719 (1/35) Installing ncurses-terminfo-base (6.2p20200523-r0) 1.758 (2/35) Installing ncurses-libs (6.2p20200523-r0) 1.811 (3/35) Installing readline (8.0.4-r0) 1.851 (4/35) Installing bash (5.0.17-r0) 1.939 Executing bash-5.0.17- 1.945 ERROR: bash-5.0.17- script exited with error 1 1.945 (5/35) Installing fstrm (0.6.0-r1) 1.987 (6/35) Installing libgcc (9.3.0-r2) 2.013 (7/35) Installing krb5-conf (1.0-r2) 2.053 (8/35) Installing libcomerr (1.45.6-r0) 2.093 (9/35) Installing keyutils-libs (1.6.1-r1) 2.133 (10/35) Installing libverto (0.3.1-r1) 2.174 (11/35) Installing krb5-libs (1.18.3-r0) 2.334 (12/35) Installing json-c (0.14-r1) 2.376 (13/35) Installing libstdc++ (9.3.0-r2) 2.493 (14/35) Installing libprotobuf (3.12.2-r0) 2.667 (15/35) Installing libprotoc (3.12.2-r0) 2.807 (16/35) Installing protobuf-c (1.3.3-r1) 2.854 (17/35) Installing libuv (1.38.1-r0) 2.906 (18/35) Installing xz-libs (5.2.5-r0) 2.954 (19/35) Installing libxml2 (2.9.10-r5) 3.071 (20/35) Installing bind-libs (9.16.15-r0) 3.287 (21/35) Installing bind-tools (9.16.15-r0) 3.363 (22/35) Installing ca-certificates (-r4) 3.426 (23/35) Installing nghttp2-libs (1.41.0-r0) 3.456 (24/35) Installing libcurl (7.76.1-r0) 3.510 (25/35) Installing curl (7.76.1-r0) 3.561 (26/35) Installing iperf (2.0.13-r2) 3.607 (27/35) Installing libbz2 (1.0.8-r1) 3.649 (28/35) Installing fts (1.2.7-r1) 3.691 (29/35) Installing libelf (0.179-r0) 3.774 (30/35) Installing libmnl (1.0.4-r0) 3.814 (31/35) Installing libnftnl-libs (1.1.6-r0) 3.865 (32/35) Installing iptables (1.8.4-r2) 4.387 (33/35) Installing iproute2 (5.6.0-r0) 4.535 Executing iproute2-5.6.0- 4.540 ERROR: iproute2-5.6.0- script exited with error 1 4.540 (34/35) Installing libbsd (0.10.0-r0) 4.584 (35/35) Installing netcat-openbsd (1.130-r1) 4.627 Executing busybox-1.31.1-r20.trigger 4.632 ERROR: busybox-1.31.1-r20.trigger: script exited with error 1 4.632 Executing ca-certificates--r4.trigger 4.686 2 errors; 29 MiB in 49 packages ERROR: executor failed running [/dev/.buildkitqemuemulator /bin/sh -c apk --update add bind-tools curl netcat-openbsd iproute2 iperf bash && rm -rf /var/cache/apk/ && ln -s /usr/bin/iperf /usr/local/bin/iperf && ls -altrh /usr/local/bin/iperf]: exit code: 2 failed to solve: rpc error: code = Unknown desc = executor failed running [/dev/.buildkitqemu_emulator /bin/sh -c apk --update add bind-tools curl netcat-openbsd iproute2 iperf bash && rm -rf /var/cache/apk/ && ln -s /usr/bin/iperf /usr/local/bin/iperf && ls -altrh /usr/local/bin/iperf]: exit code: 2 make: ** [Makefile:43: all-build-and-push] Error 1 ERROR ERROR: build step 0 "" failed: step exited with non-zero status: 2 ERROR: () build -78ae-4086-bc26- completed with status "FAILURE"
2021/04/30 16:58:31 Failed to run some build jobs: [error running [gcloud builds submit --verbosity info --config --substitutions PULLBASEREF=master,WHAT=agnhost,GITTAG=v20210430-v1.22.0-alpha.1-58-gf039f94a6fe --project k8s-staging-e2e-test-images --gcs-log-dir gs://k8s-staging-e2e-test-images-gcb/logs --gcs-source-staging-dir gs://k8s-staging-e2e-test-images-gcb/source gs://k8s-staging-e2e-test-images-gcb/source/-1c8a-4141-9bbb-]: exit status 1] Need this fixed to finish image version dance to fix conformance tests for 1.21.1:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This has failed 3 times consecutively now. All because s390x issues ... I've manually re-triggered it the fourth time. /sig testing but also /sig release because this is a major problem for continuing to ship this platform. cc We've seen this before with flakes across other image builds for this platform.
looks like the latest run passed -
there's still underlying flakiness i need to get to the bottom of
Also what latest run? Were new changes made without bumping the version? Or did someone else manually trigger it yet again?
I just saw a green run at and the image was pullable, so I thought it had succeeded
I think someone must have retried it again manually. Postsubmits only run on merge, and we've had nothing else merge
Guirish tried to see whether our local s390x environment has the same problem too, he'll update us here.
the issue seams to be intermittent caused while pulling respective binaries from Alpine repo. I tried to reproduce the issue locally, I am observing similar build failures while cross building the image for other ARCH too, so the issue is not specific to s390x and it is to do with alpine repo.
thanks i've filed a PR that may help mitigate please review!
looks like a long pending problem
Interesting. FWIW 4/4 consecutive failures appeared to be s390x in CI. We've had previous flakes relating to symlinks affecting this platform only.


kubelet increased more than 100MB in memory in 4 days. kubelet to not used that amount of RAM I have reproduced it using 3 cronjobs which create hello world pods each minute and another cronjob that writes a line to a file each minute. Kubernetes version (use ): v1.18.1 Cloud provider or hardware configuration happens on both virtualbox and baremetal OS (e.g: ): centos 7 Kernel (e.g. ): 18.0-147.3.1 I have seen some existing issues like but I do not have any dangling resources associated with the cgroup.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
/assign This is a duplicate issue, there has been a fixed PR but it has not been merged.


<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --ci-kubernetes-e2e-gci-gce-slow [sig-storage] Dynamic Provisioning DynamicProvisioner [Slow] should provision storage with different parameters 5/13 https://k8s- v1beta1.StorageClass got removed in We should remove the v1beta1 test.
/assign can you help take a look?
/triage accepted
/sig storage
I run the test with the latest from master without removing the beta test and it failed with a different error before getting to the beta test part: Do you have any idea what would've happened in my setup? I redeployed k/k using master HEAD
For this testcase: I saw that the provisioned PD has a size of 1.5Gi according to kubernetes: And in gcloud It's really interesting that the test was expecting 2Gi to be created, is this something that was changed recently?
The logs you posted are showing pd csi as the provisioner. Were you testing with the intree driver or csi driver? The CSI driver should to the nearest Gb. I believe intree does the .
I think I did intree because I didn't send as an argument in my command: It's interesting that the PVC has this annotation too:
mmm something interesting about my setup is that I created the cluster with CSIMigration turned on Do you think that might be forcing the usage of ?
Yes likely. Try with CSIMigration off and see if that helps. If so, then this potentially seems like a rounding bug with the csi driver
Thanks I tried with and it did indeed round up to 2Gi.


/sig scalability This issue is tracking process of upgrading etcd to 3.5.0 planned for 1.22 K8s release. Etcd has recently released Beta versions and is starting the process of correctness and performance testing. To reduce change of finding performance regression late, we will start qualifying Beta Etcd releases on GKE master branch and will upgrade to 3.5.0 as soon as it is available. Steps: [x] Do preliminary scale testing of etcd v3.5.0-beta.3 release [x] Release etcd v3.5.0-beta.3 image (, ) [x] Release etcd v3.5.0-rc.0 image () [x] Upgrade etcd server to v3.5.0-rc.0 () [x] Wait for 3.5.0 release and upgrade etcd client () [x] Release etcd v3.5.0 image (, ) [x] Upgrade etcd server to v3.5.0 /milestone v1.22
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/area etcd
Does this meant that the 3.5.0 should be promoted like
Yes, created
/area release-eng release-team /sig release For tracking: cc:
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /area release-eng release-team /sig release cc: Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I do think this can be pointed in the Major Themes for the 1.22 release. I would like your opinions on this. If you do think it has a place in it, I will add it to the .
How can I be a contributor to this issue? what are the technical checklist that needs to be achieved to close out on the issue?
PR to update Etcd is ready, but we found a blocker for merging . There is one test failing I will take a look into it on Monday, still more eyes on the problem could speed up resolution.
We'll also need a new kube-cross image w/ etcd v3.5.0. I'm working on that here:
We just need to upgrade the server to use new image like in
ok, looks like is busy. Let me take a look /assign
Thanks.


Creating a service with fails to be created when trying to use a TCP health probe. Example service: Failed as follows: Health probe works correctly and is registered properly as TCP health probe. Take the mentioned service yaml and apply it, together with a corresponding pod / deployment exposing port 445. E.g. like this: The respective code seems to take the wrong path to me (it's +/- the same code on the in-tree as on the out of tree cloud provider): My reading of the code and passed values are like this: returns returns -- because returns returns therefore: So the requestPath is not at this point and the azure API server returns a 400 bad request. Kubernetes version (use ): 1.20.6 Cloud provider or hardware configuration: azure OS (e.g: ): linux Network plugin and version (if this is a network-related bug): vxlan & flannel
/sig cloud-provider
related PR introducing the feature:
/assign /triage accepted
/assign


When we introduced we an additional check during attach disk call to check for canonical path - Now this is creating a regression in many deployments because when volume is created, canonical path gets replaced - and hence when attach is attempted k8s can't find canonical path of the volume anymore. It throws following error: There are two ways of fixing this. We could decide to use old in attach code, if we can't fetch canonical path or we could decide not to replace canonical path during provisioning. cc /sig storage
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
cc:
The code that replaces canonical Path during provisioning is pretty old and hence I don't remember/know why we are doing that. But since there could be many existing PVs that now have different canonical path - I think it might be safest to merge - to prevent any regressions.


<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --https://prow.ppc64le- Below two test cases started failing on ppc64le: 6th June 2021 https://k8s- ... was the delta between the passing and failing tests Started failing after
/assign
/sig cloud-provider /priority important-soon /triage accepted /assign
Thanks for reporting the issue. This was caused by vendor changes in , which introduced external dependency on IMDS. If IMDS endpoint is not reachable, then the new logic would report error.
Filed to clean up the failed tests.


! no panic Kubernetes version (use ): k8s 1.16
/sig Network
Could you fix this? I think the function needs a lock while being executed.
/triage accepted /good-first-issue todo: add a lock around the body of this function: Similar to
/assign
Oh I see, thank you for letting me know, I'll keep that in mind going forward. :)
Anyone actually on this? It seems like a good fix to have for 23 and maybe backport
I'll have a try.
There is a reproducer here
Oops, I just peeked at the time of the pull request and not realized that you've got such deep conversation.
/remove-good-first-issue /unassign
/assign
hi ! someone work on this now? I would like to try it.
Yes, working on it currently.
are you still planning to give this a shot?
Is there no one working on this? If so, I'd submit a PR for it this week. :)


/kind bug The logs of pods with label will be output expected without errors. create the following resources: kubectl logs -l Kubernetes version (use ): latest master.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cli
I am able to pull logs from multiple containers in my test setup, this doesn't seem to be an issue with Could you check if all the containers in the specified pod are healthy?
yes, all of these pods' containers are healthy. I the section.
This is working as expected, was not able to find the default container which in both deployments you explicitly said is but only the first deployment has container, the second deployment has . /close
Closing this issue. <detailsIn response to : This is working as expected, was not able to find the default container which in both deployments you explicitly said is but only the first deployment has container, the second deployment has . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


we have one in k/k and one in k/utils, both need to be in LICENSES /assign
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/area code-organization
I think we should include anything matching the LICENSE/COPYRIGHT/COPYING patthers under or subdirs in k/k or dependencies... a quick sweep showed these:
/sig release ?


<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI -- [sig-node] Pods should delete a collection of pods [Conformance] TBD links to appreciated links to specific failures in spyglass appreciated <!-- Please see the deflaking doc () for more guidance! --
cc /sig windows
I've looked through the logs for a bit and the test itself. What the test is doing: create 3 pods. expect to find 3 pods (it doesn't seem to wait for the to start, they only need to exist). delete all 3 of them as a collection. expect all 3 pods to dissapear. All 3 pods ended up on the same node, plus some other tests were running in parallel, which could also contribute to the failure. Looking at the kubelet logs, it seems that was the remaining pod to be deleted. Filtering the logs, we can see: The test failed at, while the other 2 pods did: Those times correspond with the number of pods observed by the test: Now, my question is, why are there such big gaps between the ` events? It's 12s for test-pod-2 and 21s for test-pod-3? Seems like a long time for pods / containers to linger after they've been killed. PS: It seems that Containers are still being created / started / stopped, even if the delete request came less than 0.2s after their creation. It doesn't seem to me like a reasonable thing to do, especially if that Pod would have to pull a huge image; I would have expected to be able to cancel that.
/triage accepted
Let's discuss at the 8/17 SIG-Windows community meeting
Hit again on 8/15
I think was planning on opening up a PR to use longer timeouts in this test. We see that some Windows containers do not get scheduled until 1+ minutes into the test - especially when there are many tests running in parallel.