<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Deploy a Windows cluster with containerd configured, SMB mount does not work by trying following project: It reports following error: What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): v1.18.6 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /sig windows /sig storage
/cc
I was able to repro your issue using the azurefile-csi-driver for Windows. I also tried to set up a repo where I would mount an azure file share to a Windows node manually, create symlinks to the share and then schedule a pod to the node using a hostPath volume and the container was able to start up fine using contianerd. Here is the pod-spec i used I tried to mount the Azure file share with both New-PsDrive and New-SmbGlobalMapping and saw date being written the the in the azure file share in both instances. I used mklink /D to create the symlinks - and also tried creating a symlink chain because it looks like that is how they get set up when when using azurefile-csi-driver and csi-proxy Right now I think this issue may be specific to either how the csi-proxy is setting up the SMB share for Windows or how paths are getting passed around between the various components.
this is not related to csi-proxy, in-tree azure file does not use csi-proxy, and it does not work
hostPath on containerd Windows does not work too:
I captured containerd logs for this and see I spoke with and he thinks this may be related to a known issue where go-lang is getting confused on Windows if a symlink points directly to the root of a SBM share (which is does in this case). He suggested adding a trailing  to see if this fixes the issue. I will try that out now and report back.
I tested some updates to CSI proxy to ensure the symlink to the sharepath had a trailing backslash and it looks like that resolved the issue. On my node running contianerd i have the following symlinks Previously the  on pvc--f195-4d9e-841b- was missing.
The core issue here is that containerd calls on the mount source path, and that function has a bug which prevents it from working when the path is a UNC share root. If the path has a trailing appended, then works properly. I have filed a bug on Go to fix this, and am working on a fix: golang/go
/close
Closing this issue. <detailsIn response to : /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is not resolved, right?
I see this issue is resolved in csi-proxy v0.2.2, while it's not resolved for the in-tree azure file driver.
/triage accepted
Ah, good point. yes, this is not fully resolved.


Note: this issue should be viewed in the context of . If this happens, this is very likely a typo where someone used a parameter in a way they didn't mean to (for example, ). /sig scheduling /assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale
/good-first-issue
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hi this is my first time contributing to kubernetes. can you provide me some guidance on how to to get started?
Hi, is this issue still valid to work on?
yes, this is still valid. I recommend starting with the and reading from there.
are you still working on this issue or should I work on it.
I would like to work on this issue if you haven't started already.
would it be ok if I took a look at this?
/assign
Hi Can you help me in why its not assigning the values to createNodes and createPods directly and using the opcode and countParam in performance-
We use a field to differentiate between different operations. The value of the field (e.g. "createNodes") determines the operation. The field defines the number of pods/nodes to create. What do you mean by "assigning the values to createNodes and createPods directly"?
Thanks for the information. By assigning, I just meant that whether we can assign createNodes directly like ( createNodes: $initNodes ).
I don't think that's possible since there can potentially be other parameters besides the countParam. For example: Different operations could potentially have different parameters.
createNodes and createPods aren't the only operations. Other operations (e.g. churn and barrier), including future ones, may not fit that format. It's also easier to parse and more explicit this way.
Thanks , Having a doubt more, in performance- under the name:SchedulingPodAntiAffinity, ! why are we giving the same values in 5000Nodes as given in 500Nodes, or this is an error.
that sounds like a typo indeed
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
Hello Are you still going to work on this issue?
Hi I have worked on the issue but not working for a while as was unsure about the exact problem.
, no problem if you are willing to work on this issue, shall unassign myself
/unassign
Thanks :)
/assign
/remove-lifecycle stale
hey! I am beginner and I interested in scheduling and I know basics of scheduling , so how can I start to contribute on this issue.
Sorry, PR for this issue has already created.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:


<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: /test pull-kubernetes-node-crio1-18-e2e Which test(s) are failing: pull-kubernetes-node-crio1-18-e2e Reason for failure: Just before node e2e tests are started, we verify that the runtime (containerd or docker) is properly running or not on the target system. This check doesn't include crio, so the targeted node is not reported ready if it's using crio as the runtime.
/sig testing
/sig node /remove-sig testing This check is owned by SIG node
The code has also not allowed this sort of dependency (beyond some legacy dockershim specific bits), but I guess it's somewhat more understandable to check in .


Warning: The v0.1 and v1beta1 metadata server endpoints are deprecated and scheduled for shutdown on September 30, 2020. This test (which runs in most e2e jobs that run on GCE): Schedules a container that runs this: Resulting in a number of legacy metadata endpoints getting hit. To see if this test is going to start failing across all release branches on Sep 30th, we could try disabling the endpoints ahead of time via: In fact and support for expecting different results based on whether is set. My concern is the test expects 200 for some root endpoints, and 403 for disabled endpoints. However the e-mail I received as a GCP customer tells me any legacy endpoints may return 404. I'm inclined to suggest we remove the legacy endpoints from the check, bump the image version, and cherry-pick use of the new image version back to all release-branches. WDYT since you were involved with the earlier PRs /sig auth /sig cloud-provider /provider gcp /sig testing /priority important-soon (I'm tempted to critical-urgent this but it's not drop-everything yet) /milestone v1.20
/kind cleanup
Heyho I wanted to check how it is going? The code freeze is starting 12.11.2020 (about 3-4 weeks from now) and while there is still plenty of time, we want to ensure that each relevant issue has a chance to be considered. As the issue is tagged for 1.20, is it still planned for this release?
Bug Triage here, we're pretty late into code freeze and it doesn't look like there has been much activity recently. I am going to clear the milestone on this issue. /milestone clear
any update here?
Try to cleanup the legacy endpoints in
/milestone v1.21
/reopen This needs to stay open until the new image has been verified to work and has been promoted (ref: )
Reopened this issue. <detailsIn response to : /reopen This needs to stay open until the new image has been verified to work and has been promoted (ref: ) Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/triage accepted /assign


<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: ci-kubernetes-node-kubelet-serial node-kubelet-serial job Which test(s) are failing: Hard to tell, but everything at one point or another. Since when has it been failing: A month or two, by recollection. would indicate several years... ci-kubernetes-node-kubelet-serial | failing_days | 992 Testgrid link: Reason for failure: Hard to tell, some is timeout, but I think timeouts due to test failures. Anything else we need to know: Nothing consistent about the failures that I can tell besides timeouts. This is the only place these serial tests are running.
/sig node
/cc
I will take a look at what happens with the serial lane.
So the kubelet failed to restart because of I think it can be related to the topology manager scope PR, I will dig into the code tomorrow. FYI
It actually looks like it was introduced by: With the same issue reported here: And Alexey has a fix for it here (I haven't reviewed it):
Thanks for the update.
Some of these failures are definitely due to , which is causing tests to fail since pods aren't terminating in time. See e.g.
/assign
Under the latest run, I saw that the kubelet failed to restart because of The problem that one of the topology manager tests failed in the middle and did not remove the state file, I can not guarantee that it a single problem, but let's start from it.


<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: Which test(s) are flaking: Testgrid link: Reason for failure: Timeout while - trying to gave a 404 or 502 Anything else we need to know: 4 flakes in the past week: Triage: /sig network <!-- Please see the deflaking doc () for more guidance! --
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/assign
Hi and any update on this ?
/assign
Minhan -- can you take a look?
/triage accepted
Looks like the LB did not stabilize before timeout. Still looking. We are considering disabling some of the tests here
This test is still flaking a bit -- recent failures from the past 2 days:
It looks like this test is still flaking quite frequently: Any ideas on how to move forward?
I will start disabling the tests.


<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: bazel-test-master (periodic-kubernetes-bazel-test-master ) Which test(s) are flaking: gotest: Testgrid link: Reason for failure: Anything else we need to know: links to appreciated links to specific failures in spyglass appreciated <!-- Please see the deflaking doc () for more guidance! --cc
/sig api-machinery
This is continuing to be flaky with more recent failure:
/assign Would you mind taking a look here? thanks! /triage accepted
- can we somehow uncover when this started happening? This test is very old and it seems it started happening very recently, so it would be helpful to see when this started happening...
Triage shows flakes starting ~Sept 30: Before Sept 30 it seems the job was flaking much less frequently.
Might be related to I will take a look.
It seems it's not related to that PR (neither to the previous: ) As there were already failures like that on Sep 24th (before this one was merged). I will take a deeper look.
OK - it's not related to any changes mentioned before. I can reproduce this particular failure by simply adding somewhere in this block: So it's probably connected to cpu-starvation at the test level and the solution is to switch to less aggressive timeouts. Will send out a PR.
Actually - it appeared to be slightly more subtle then that. But I already fully understand the problem. It's purely test issue, but require a bit more changes (and I won't get to it today/tomorrow). I will send a fix for that early next week.
- if that changes anything from your tracking perspective ^^
No board changes yet - the next stage is "Observing" once this issue gets closed :) If you have time - I'd love to pick your brain/understand the error more thoroughly though! Hoping you drop a good explanation in your PR next week
Thanks for sharing those insights, on your discovery of the solution to this bug.
It seems the PR didn't address that fully. Looking.
out for review


<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: gci-gce-serial-kube-dns-nodecache Which test(s) are failing: N/A Since when has it been failing: 10/13 Testgrid link: https://k8s- Reason for failure: Cluster failed to start: Anything else we need to know: /cc
So, is a failing job, we can see that there is a error around binding to a specific IP address. In passing jobs, the initial core file isn't printed out () so not sure how to do an apples to apples comparison of what is broken here yet.
/kind bug I have been looking at this offline with and The only difference between a failing and a passing job is the arguments to the bind plugin. Failing Job: Link - Image - Config: Passing Job: Link - Image: Config:
Issue is reproducible when spinning up a cluster via kubetest using:
changes to lines 907-910 in hack/local-up- and lines 2467 to 2469 in configure- seem like the issue.
What we see is substitution works correctly for line , but for line , it replaces "PILLARLOCALDNS" with empty string.
/assign Can you investigate?
Yeah lookin into it
Regex is matching the entire line :). Should be easy fix . Will verify in hack local up and issue PR shortly
/sig network /triage accepted


What would you like to be : Add integration tests for Why is this needed: This is part of the kubernetes/enhancements /sig cli /priority important-soon
/assign
/triage accepted


What would you like to be : e2e test for service ExternalIPs Why is this needed: There is no test covering Service ExternalIPs Since we can not assume that the e2e binary will have connectivity against the nodes and that we can add routes to the environment, we are going to test it from a pod. I've verified that this works locally with the following commands: The test should leverage current framework, maybe something as A good example is here, maybe adding a ?
/sig network /help wanted /triage accepted
/help wanted /good-first-issue
/assign
I'd like to pick this up!
Ok,I'm not started :) /unassign /assign
There are two different externalIP codepaths, one for the case where the claimed external IP exists as an IP on the default interface, and one for where it does not. I suppose we could just temporarily add an IP to the default interface...
good idea Dan Please remember, it has to work for IPv6 and for IPv4 ... anyway, we can address these things during the review
This has been partially implemented with . Covering the code path for when an IP is does not exist on a node. I'll assign this back to you so you can split this issue or close it in favor of another issue. Thanks! /unassign /assign
what is the different codepath for ExternalIPs that already exist on the host? ~My reading is that the only difference is that it opens a local socket on that port, can you expand on this?~ Found it
so, it seems that this work in some environments because they are not using the bridge plugin why is not allowed that pods can reach ExternalIPs?
Wow, this is ancient and needs some fixing. This is not about preventing pod traffic from reaching those IPs. I had to really stare at it, and I am not 100% in my interpretation. First, we recognize such traffic from "off machine" (!physdev-is-in && !src=LOCAL, which we should probably switch to localDetector) and steer it to the Service. Second, we recognize ANY such traffic that is destined for a LOCAL address and shortcut that to the Service, Implicitly, traffic destined for this IP where the IP is not local goes to the system's default routing logic. The reliance on the bridge plugin is awful and archaic and I will buy a beverage of your choice for whomever gets rid of that. I think localDetector can cover this, but not 100% sure
but the ExternalIP belongs to a service and we(kube-proxy) "knows" where to send it (to the service) , that is the part I'm not sure to understand, because whatever the user is doing with that ExternalIP it has to send it back to the cluster to reach the service. However, there is always context that is important and maybe I'm missing something. Assuming we want to handle ALL the ExternalIP traffic with kube-proxy, the logic will be much simpler: traffic to an externalIP will be ALWAYS sent to the Service (DNAT) if traffic is NOT LOCAL, we also SNAT
/remove-good-first-issue /remove-help-wanted


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Using example for from documentation failed with: What you expected to happen: Example from documentation works. How to reproduce it (as minimally and precisely as possible): How should configuration look like for ? Environment: Kubernetes version (use ):OS (e.g: ):Kernel (e.g. ):
/sig scheduling
/triage accepted oh, I missed a dash. It should be: Would you mind sending a PR to kubernetes/website?


What happened: Process running as non-root will fail to work as it cannot access /etc/hosts inside the container when umask of kubelet is configured as 027. It is because the ioutil.WriteFile() used to create /etc/hosts is impacted by umask. Even though kubelet specify the permission of 0644 explicitly, with umask of 0027, it will be configured as 0640. As a result, application running inside the pods with non-root user, failed to access it. What you expected to happen: kubelet should pre-configure required umask by itself, instead of depending on the environment. At least we should document the required umask. How to reproduce it (as minimally and precisely as possible): Lauch kubelet with umask of 027 Anything else we need to know?: It's the same problem figured out. Environment: Kubernetes version (use ): 1.19 Cloud provider or hardware configuration: N/A OS (e.g: ): Ubuntu 18.04.4 LTS Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
/area kubelet
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-contributor-experience at . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: Which test(s) are failing: Since when has it been failing: Testgrid link: Reason for failure: Anything else we need to know: Introduced in /cc /sig node /assign /milestone v1.20
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: When I register a device plugin and set the option in true, then when I create a pod and specify this resource as 0 in yaml, an error will be reported. Examples are as follows. The request to register the device pluginA pod yaml with the resource 0kubelet return error What you expected to happen: I think when the kubelet calls , the resource value should be checked. If it is 0, the step should be skipped to avoid errors. How to reproduce it (as minimally and precisely as possible): Register a device plugin which set option is true Create a pod with the above resource 0 Anything else we need to know?: In fact, when the amount of requested resources is 0, devicemanager will get the number of allocated devices as nil, resulting in the internal cache not storing the application resource information corresponding to the container. But if is set to true at the same time, then devicemanager will check podDevices cache. The two processing steps contradict each other so it will return an error.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Below k8s UT fail when run with master-golang. TestCheckpointStateRestore/RestorecheckpointwithinvalidJSON error:=== RUN TestCheckpointStateRestore/RestorecheckpointwithinvalidJSON I1124 05:38:14. ] [cpumanager] initializing new in-memory state store unexpected error while creatng checkpointState: could not restore state from checkpoint: json: unexpected end of JSON input, please drain this node and delete the CPU manager checkpoint file "/tmp/cpumanagerstatetest479660663/cpumanagercheckpointtest" before restarting Kubelet --- FAIL: TestCheckpointStateRestore/RestorecheckpointwithinvalidJSON (0.00s) TestStrategicMergePatchInvalid error:=== RUN TestStrategicMergePatchInvalid expected "invalid character 'b' looking for beginning of value", got "json: invalid character 'b' looking for beginning of value" --- FAIL: TestStrategicMergePatchInvalid (0.00s) TestJSONPatch error:=== RUN TestJSONPatch invalid-syntax: expected error invalid character 'i' looking for beginning of value, but got json: invalid character 'i' looking for beginning of value --- FAIL: TestJSONPatch (0.00s) TestV4ErrorDecoder expandless error:=== RUN TestV4ErrorDecoder wrong error for message "{": want="error stream protocol error: unexpected end of JSON input in "{"", got="error stream protocol error: json: unexpected end of JSON input in "{"" --- FAIL: TestV4ErrorDecoder (0.00s What you expected to happen: The tests should pass with no error. How to reproduce it (as minimally and precisely as possible): a stable version of golang installed master go from upstream git clone cd go/src cd ../.. cp -r ./go /usr/local export PATH=/usr/local/go/bin:$PATH kubernetes repo and run the unit tests. () Anything else we need to know?: These failures are occurring due to mismatch of error messages between upstream go and the k8s tests. There is an addition "json:" prefix to SyntaxError messages after this below commit: Environment: Kubernetes version (use ): latest source code from upstream Cloud provider or hardware configuration: OS (e.g: ): "CentOS Linux 8 (Core)" Kernel (e.g. ): Linux rajalakshmi-workspace 4.18.0-193.19.1.el82.ppc64le SMP Mon Sep 14 14:48:04 UTC 2020 ppc64le ppc64le ppc64le GNU/Linux Install tools: [root go]# go version go version devel + Fri Nov 20 00:54:43 2020 +0000 linux/ppc64le [root go]# Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig testing
will fix it in Go 1.16 as release-blocker.
I would fix the test here to be more tolerant. exact string matches on error messages aren't really an API… I would not expect that to block the golang release
Agree too, we can think about having a substring match to avoid these issues instead!
/assign Lets see if we can fix this in the k8s tests


<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: bazel-test-master Which test(s) are flaking: //pkg/kubelet/cm/devicemanager/godefaulttest:run2of_2 Testgrid link: Reason for failure: Anything else we need to know: links to appreciated Triage shows nothings (seems to not have updated data), but there are multiple failures in the job bazel-test-master. links to specific failures in spyglass appreciated <!-- Please see the deflaking doc () for more guidance! --
/sig release
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig sig-release Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Digging further as the step has another recent test fail ... So on the two runs reported by has 2 test failures has 1 test failure And the most recent failure in this area is a TestAdmit/match test failure but on a different test.
/sig node
/assign /triage accepted I'll take a look at them and fix.


What happened: I was running the subpath testsuite against Azurefile. This test failed and can be easily reproduced: Basically what this test does is: a PVC use azurefile a pod that has two containers. One of it mounts the volume itself. The second container mounts the subpath of the volume. the subpath dir from the container that has the whole volume the pod, the pod will stuck in terminating status and cannot be cleaned up other than What you expected to happen: The pod can be deleted successfully. How to reproduce it (as minimally and precisely as possible): the pvc the pod the subpath dir from the parent pod the pod Anything else we need to know?: The volume is being formatted as cifs type Environment: Kubernetes version (use ): v1.18.6 Cloud provider or hardware configuration: Azurefile-csi-driver 0.9.0 OS (e.g: ): Ubuntu 20.04 Kernel (e.g. ): Linux nodepool-- 5.4.0-1016-azure -Ubuntu SMP Thu Jun 4 22:49:15 UTC 2020 x8664 x8664 x86_64 GNU/Linux Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig storage /cc /cc
log After debugging with and offline, the issue is that: After the dir from the subpath was removed, the subpath mountpoint on the host machine becomes into a weird state where you can still find the dir by doing a readdir to its parent. But doing or something like that will result in a "No such file or directory". When you do a ls on the path:
/triage accepted
So the issue from code is that It will fail the path exists check while the actual mountpoint exists. So when the code logic runs to here: It will fail to remove the container dir because it somehow detect there is still file there. To fix this as a workaround, user can manually , and then the kubelet can take care of this. To fix it ultimately, we need to see if this is an issue specific to cifs filesystem? Or we should consider do an umount regardless if the path exists.
/assign
/assign
Tried the same scenario using NFS. It works as expected using NFS.
Yeah, that's why I am wondering if this is something special with CIFS?
Looks like it has something to do with the azurefile-csi storage class mount options. I was getting the below error when I was trying to do ls on the subpath. So I realized there might be some issue with the file permissions. root ls ls: cannot open directory '.': No such file or directory When I removed the dirmode and filemode properties from the storageclass and performed the same steps as mentioned in the issue, the pod got deleted successfully.(Although it took a lot of time 10-15 mins for the pod to get deleted)
FYI, I am planning to use this PR to fix the unmount issue in Azure File & SMB CSI driver: Not sure how to handle gracefully in k8s subPath code since that's common code.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
So this PR does not fix the Azurefile subpath test issue?
no, that PR only fixed the unmount issue inside CSI driver. While for the subPath issue, we should fix it in k8s upstream code as you mentioned above:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
back to original discussion, shall we use instead of inside func? That will solve the issue of smb file PathExists check issue.
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
this subPath issue was fixed in , ,


What happened: Adding a VolumeMount to a container with a multi-part subpath results in only the leaf directory having 777 permissions. If you define e.g. then within the volume, and will have permissions, whereas will have permissions. This means that if you want to remove and eventually from within a pod, you will need to use a privileged pod. If the sub-directories already exist, their permissions are unchanged, and the order in which these subpaths are created appears to be the order in which they appear within the spec -- if we see first, then will be created with 755 permissions and will be created with 777 permissions, even if there's another volumeMount for further down. What you expected to happen: Sub-directories created to support a volumeMount should all have the same permissions as the leaf directory. How to reproduce it (as minimally and precisely as possible): the yaml below. The first example has volume mounts so that is created before ; the second example has volume mounts in reversed order, so is encountered before , which results in being unwritable. <details<summaryThis pod will complete</summary</details<details<summaryThis pod will fail</summary</detailsAnything else we need to know?: I believe this is the return of Environment: Reproduces on minikube and a (OpenShift) development cluster I have access to Kubernetes version (use ): minikube: OpenShift cluster: output: Cloud provider or hardware configuration: Minikube: v1.15.1; docker driver; hostpath volumes (all default) OpenShift cluster: AWS, EBS volumes OS (e.g: ): Minikube: Fedora 32 OpenShift cluster: RHEL 8.2 Kernel (e.g. ): Minikube: OpenShift cluster: Install tools: minikube v1.15.1 / Not sure Network plugin and version (if this is a network-related bug): N/A Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage
/assign
GitHub didn't allow me to assign the following users: mengjiao-liu. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign


In reference to conversation on #sig-network-policy-api.  What would you like to be : Add a test to the netpol validation framework to validate a scenario where traffic between two pods has egress and ingress rules defined. Why is this needed: To ensure consistent behavior between different implementations of networkPolicy. For example, describes different behavior between Calico networkPolicy and Azure networkPolicy
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig sig-network
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig sig-network Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig network
thanks for offering to hack on this
/sig network
cc
I reviewed the current networkpolicy tests, and I believe this case is already captured in the currents tests.
Thanks for digging into this ... so Your case, yesterday was more then just combining egress and ingress. Your case , was one where Egress blocked a traffic that was allowed by other rules. Which is quite interesting and confuses alot of people. ... It highlights the 'podselector centric' nature of policies.... I think you want to do , right? In that case, Egress rule is specifically created that disallows but Ingress rules explicitly allow And the result is that Egress wins, and traffic is blocked ...
This is an interesting test case, I'd be happy to hack on this and look at adding a table-driven netpol test case but don't want to step on your toes if you're already looking into ! Definitely see the value of illuminating this corner case with a clear test scenario!
Thx for the feedback. You are right that it's still a different case. : I'm looking at what I can throw together today. Stay tuned, and would love your input.
I created the initial version of the test.  The code (and comments) are likely a bit verbose at this point. Should I open a PR against this repo now, or do you want me to make changes to that code prior to the PR?
Looks awesome, I'd go ahead with the PR!
can you reference this issue in your new PR ?
/assign


We could either: where appropriate - least amount of code change but cumbersome OS version in the tag instead of the repository /cc
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig windows
/kind bug


[ ] two weeks soak start date : testgrid-link [ ] two weeks soak end date : [ ] test promotion pr : kubernetes/kubernetes#? According to this APIsnoop query, there are still some remaining StatefulsetScale endpoints that are untested. with this query, you can filter untested endpoints by their category and eligibility for conformance. e.g below shows a query to find all conformance eligible untested, stable, apps endpoints sql-mode select distinct endpoint, right(useragent,73) AS useragent from testing.auditevent -- where useragent ilike '%subresource%' where endpoint ilike '%AppsV1NamespacedStatefulSetScale%' and releasedate::BIGINT round(((EXTRACT(EPOCH FROM NOW()))::numeric)1000,0) - and useragent like 'e2e%' order by endpoint limit 30; example endpoint | useragent -----------------------------------------|--------------------------------------------------------------------------- patchAppsV1NamespacedStatefulSetScale | [StatefulSetBasic] should have a working scale subresource [Conformance] readAppsV1NamespacedStatefulSetScale | [StatefulSetBasic] should have a working scale subresource [Conformance] replaceAppsV1NamespacedStatefulSetScale | [StatefulSetBasic] should have a working scale subresource [Conformance] (3 rows) If a test with these calls gets merged, test coverage will go up by 1 point* This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


What happened: with --cpu-manager-policy=static kubelet will panic when pause container lose during kubelet restarting What you expected to happen: kubelet can start normally How to reproduce it (as minimally and precisely as possible): 1,config kubelet with --cpu-manager-policy=static 2, Run a large number of pods on the test node 3,stop kubelet use command: 4,kill and remove all the pause container like this: ! 5,start kubelet use command: and then kubelet maybe panic, log is : Anything else we need to know?: When we actually encountered this problem, only one pause container was lost. In order to reproduce this bug, I deleted all pause containers. Environment: Kubernetes version (use ): 1.20.1 Cloud provider or hardware configuration: x8664 OS (e.g: ):18.04.5 LTS (Bionic Beaver) Kernel (e.g. ): Linux ecs-linux-0008-43ee 4.15.0-123-generic -Ubuntu SMP Wed Oct 21 09:40:11 UTC 2020 x8664 x8664 x8664 GNU/Linux Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig node /assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-contributor-experience at . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I think it makes sense to reopen this issue as the referenced PR is still open.
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-lifecycle rotten
/triage accepted
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
I am seeing this issue with a customer also. What is the timeline for this PR to be reviewed and merged?


The based container images for are shipped with the wrong architecture within the manifest if not amd64: Refers to We updated the images a few minutes ago to buster-v1.3.0, which are now ready to be used as base for the multi-architecture images. If the images contain the right architectures, then it may magically happen that the kube-proxy images are fixed, too. This is because we already use the build syntax for it: /cc
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig release


What happened: The current in-tree PV topology is still using beta version label "failure-". This was intentionally not upgrade because of CSI migration. However, since the CSI migration timeline is likely to slip, and with the fix of , it should be safe to update the in-tree PVC to GA version () So that we can unblock removing the beta label from node. What you expected to happen: Replace all beta topology label to GA version once merged. /sig storage /cc
/assign
GitHub didn't allow me to assign the following users: kassarl. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
gce-pd topology update already merged:
aws-ebs topology update is here:
/triage accepted
cinder topology fix is here: Waiting for topology label update.
OpenStack Cinder:
/reopen Has vsphere made the fix?
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Not yet. is going to make the fix.


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: When reviewing Windows kubelet logs noticed errors like: What you expected to happen: The stats should be returned How to reproduce it (as minimally and precisely as possible): This occurs when the pods has just started or during termination. To simulate the issue: Then deploy pod: Anything else we need to know?: The container is in a state but is has not started running so the call to get the stats fails: This is handled in other places like moby: and Environment: Kubernetes version (use ): 1.18+ Cloud provider or hardware configuration: aks-engine OS (e.g: ): Windows Kernel (e.g. ): 2019 Install tools: Network plugin and version (if this is a network-related bug): Others: Dockershim /sig windows /priority critical-urgent /assign
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
cc:


What happened: What you expected to happen: How to reproduce it (as minimally and precisely as possible): Run the netpol test suite. Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/triage accepted
/sig network


MetricsGrabber tests failing when scheduler & controller-manager not visible. In managed offerings, like GKE, scheduler and controller-manager are not visible causing BeforeSuite of MetricsGrabber to fail. Test is failing then control plane pods can't be found (as on GKE): HasControlPlanePods was in /assign
/sig instrumentation
Just removing this check allows tests to run without problems (some skipped, one passing).
/triage accepted


[x] two weeks soak end date : 26 Feb 2021 According to this APIsnoop query, there are still some remaining RESOURCENAME endpoints which are untested. with this query you can filter untested endpoints by their category and eligiblity for conformance. e.g below shows a query to find all conformance eligible untested,stable,core endpoints Create a ReplicaSet with a static label Read the ReplicaSetScale Update the ReplicaSetScale Get the ReplicaSetScale to ensure it is updated Patch the ReplicaSetScale Read the ReplicaSetScale to ensure it is Patch Delete the Namespace and ReplicaSet Due to the complexity of setting up the resources for APIService we have used the current e2e test as template. It has been extended in a for review. example endpoint | useragent ----------------------------------------|
patchAppsV1NamespacedReplicaSetScale | (linux/amd64) kubernetes/$Format -- [sig-apps] ReplicaSet Replicaset should have a working scale subresource readAppsV1NamespacedReplicaSetScale | (linux/amd64) kubernetes/$Format -- [sig-apps] ReplicaSet Replicaset should have a working scale subresource replaceAppsV1NamespacedReplicaSetScale | (linux/amd64) kubernetes/$Format -- [sig-apps] ReplicaSet Replicaset should have a working scale subresource (3 rows) If a test with these calls gets merged, test coverage will go up by 3 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture
/triage accepted
Presented in Conformance meeting of 9 Feb 2021 and it was agreed that that method see to achieve the purpose and a PR can be created. A similar approach would be taken for the DepoymentScale endpoints.
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: What you expected to happen: The test should pass How to reproduce it (as minimally and precisely as possible): I think the key here is being on a linux host with cgroupsv2 enabled. Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /sig node
/assign
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


file was found 'empty'. It blocked running etcd in a fail loop with Before this issue: Data disk (etcd) run out of space due to . .txt be not corrupted. It should be updated: only when needed (infrequently) safely (rename swap or writing on pre-allocated space) migrator script that runs etcd in case of the file not being found. Use whole space of etcd data disk and restart migrator+etcd when concurrent process is disk hungry. Requires changes of this logic: This logic (file write) should not trigger if the minor version matches: The write itself: should be defensive: Please notice that truncates file before writingKubernetes version: 1.18 Cloud provider or hardware configuration: GCE
/sig api-machinery
/assign /triage accepted
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale /lifecycle frozen


was removed in 1.20 in favor of a plugin. However, this is not included in release artifacts. The binary is included in release artifacts, and ideally easily installable via kubectl plugin managers like . /sig cli /assign
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This question came up in a docs review of a page referencing () Once this is resolved, please ensure the website is updated to document installation/use of this tool.
/sig release


on the new node fails sometimes. The following error appears in the log: on the new node succeeds every time Join a control plane node The node has over 40 GB of free space on its root partition. I some more logging to the following functions: , , , , , , It appears that the ephemeral storage is detected a bit too late: /sig node storage Kubernetes version (use ): 1.20.4, commit Kernel (e.g. ): 4.9.252-1.ph2
/assign I will try to reproduce it today.
LocalStorageCapacityIsolation is a feature gate for kubelet. Can you check if you disable it? ReadyCondition will check ephemeral-storage as required capacities if this feature gate is enabled in kubelet configuration. Would you check it first?
[x] 1.8 alpha ephemeral-storage [x] 1.10 Promote LocalStorageCapacityIsolation feature to beta [x] 1.15 KEPs Quotas for ephemeral-storage for LocalStorageCapacityIsolationFSQuotaMonitoring default to false is Alpha since 1.15 [x] Failed to promote LocalStorageCapacityIsolationFSQuotaMonitoring feature to beta [x] 1.20 Honor disabled LocalStorageCapacityIsolation in scheduling I am no sure whether there is a plan to GA LocalStorageCapacityIsolation?
I tried to fix it in . Need some testing later.
It appears that the pod fails before sees the storage capacity:
Would you try this ? I build it with on PR branch.
I tried to apply the PR to 1.20.4. After four successful s, one failed. I got the following error:
! It means that during the node resource updated the first time, the cadvisor is not available yet. If we wait here, it would be a problem. I would think more of how to fix it.
Hi, Same problem here. I'm using K8s v1.20.4 with CentOS7 (2 cores, 4GB and 20GB storage) Cluster creation with kubeadm init always works, but when I try to add several master nodes with kubeadm join it takes a lot of time waiting for etcd pod to be created. In most of cases, etcd pod never runs. The rest of components (APIserver, controller, scheduler and proxy) exists and works (except APIServer which reboots trying to connect the missing etcd) Checking the kubelet logs in the missing etcd hosts I have found this: Mar 10 14:23:27 kmaster-02 kubelet[9448]: W0310 14:23:27. 9448 ] Failed to admit pod etcd-kmaster-02_kube-system() - Unexpected error while attempting to recover from admission failure: preemption: error finding a set of pods to preempt: no set of running pods found to reclaim resources: [(res: ephemeral-storage, q: ), ] That node have 17GB storage free. If kubelet is restarted, the etcd (pod and container in docker ps -a) appears. If I could help doing something please tell me. I would like to help
For anyone who is experiencing this issue and still wants to use kubeadm until it is fixed: You can remove the ephemeral-storage request from the etcd static pod and the join will work. Just make sure that your nodes actually have more than 100MB free storage left. You can reinsert the storage request by patching the etcd manifests after the nodes are up. To patch etcd during a create the following file (the filename matters, the file location doesn't) and use to join additional control plane nodes.
Thanks for the workaround I'm encountering this same issue and the workaround worked for me. Edit: the patch I tried doesn't work though (not sure if this is really a problem). I issued the command: and the response shows that the resource can't be patched this way: There may be another way but I'm still fairly new to Kubernetes so it is not obvious to me at the moment. Anyhow, thanks for the suggestion!
I updated Add a start-delay of 10 seconds for a condition that the capacity of node ephemeral-storage is 0. Not sure if this will fix it.
we are starting to see reports about this every week.
cc
cc
From it sounds like the issue is kubeadm etcd ephemeral-storage requests are too big for some environments? So far this is sounding like a kubeadm issue, not a kubernetes issue.
it seems to be something else. i doubt that users are trying machines that don't satisfy 100Mi. for example, the report in the OP points out that the machine has 40GB.
I can confirm that the VMs on which I installed k8s control plane components had sufficient disk space when I encountered the problem. I'm running Kubernetes v1.20.4 and kubeadm v1.20.4.
I think the root cause here sounds right. There is kind of race here. adding delay will help, but it does not have 100% guarantee. Not sure whether it is easy to add logic to make sure capacity is reported correctly before starting pods..
Did something change in 1.20 regarding the ordering of cadvisor and node startup?
If the reason would be a disk problem(or something we can figure out) and the delay can solve it, I think we should add a warning besides the delay.
We need to figure out what triggers the race condition. This only happens on some machines. I have not seen this myself.
We're able to repro this nearly 100% of the time with Cluster API Provider Azure + k8s 1.20.x + Ubuntu 18.04 + default template, which makes k8s 1.20 pretty much unusable at this point. Trying to see if this repros with other non-Azure cluster-api providers with kubeadm 1.20.
Does anyone know why this only repros on and not ? The ephemeral storage resource requirement here is used in common by both and : In practice I can workaround this issue in cluster-api + Azure by adding a sleep command to my spec, but it requires a significant number to make work ~100% of the time (like 90 seconds). Does the flow take significantly longer before it gets to the part where it submits an etcd pod to the apiserver? Is there any other explanation for this being a -only symptom?
i can see this failing for any static pod with ephemeral storage requests. that is, unless there is proof that this is somehow specific to the kubeadm static pod for etcd that kubeadm deploys.
I agree that the specific kubeadm-delivered etcd pod is not interesting. (In fact it's very permissive, it's only asking for 100Mi.) What definitely seems to be happening (in Azure) is that kubelet takes a long time to know about its underlying node's ephemeral-storage availability. Kubelet starts: ~2 mins later, it complains that it can't schedule etcd: Again, why does this not repro on nodes created by ?
it also seems to be an even more scoped problem because it's not reproducible for me locally (i create a number of clusters every day, usually), the kubeadm CI or in other high level projects that use kubeadm (e.g. kind, minikube, etc). we should try to find a common reason / condition from all the reporting users that are seeing this.
Experiencing same problem on openSuSE Kubic/Tumbletweed snapshot with kubeadm/kubelet 1.20.2, kernel 5.11.15 running on Xen, 2 CPU, 4GB ram 30GB disk (BRTFS)
Problem consistently appear/disappear when switching between snapshots (Working) and , so a package update in that period has probably triggered this. Following files contains changes between those two snapshots.
are you suggesting a root cause in underlying Linux storage driver surface area that is adding delay/uncertainty to kubelet's ability to introspect available ephemeral-storage on the node host OS?
A bit of luck combined with 6th sense i got it narrowed down to systemd update here Looking at the changelog, one item sticks out: Anyone else enabled cgroupsv2 on i.e. the CentOS cases (it needs to be enabled explicitly for that AFAIK)?
ACK on your latest respond regarding unified cgroup hierarchy. Is there a quick, cross-distro way to verify that any folks who are repro'ing have a similar systemd config?
The precence of file/dir /sys/fs/cgroup/cgroup.controllers should mean you are using cgroupsv2 ( Ref: ) I can confirm that the file is present on the kubic snapshot which is not working. It is not present on the one that does work. ( I'm using cri-o as runtime which should support cgroupsv2 ) cgroup2 support was introduced for K8S in 1.19 ( ).
Thanks. So tl;dr your working theory is that cgroupsv2 + kubelet ephemeral-storage admission has some pathologies. I'll post quickly with what my repro containerd + Ubuntu 18.04-LTS OS surface area looks like.
I ran into this issue consistently on Ubuntu 20.04.2 LTS using cri-o but is not present on my systems.
I can also confirm that cgroupsv2 is not a factor in my repro'ing environment:
Also, I'm not using cri-o, containerd here:
Ok - so much for that theory. But still: What I did to narrow it to systemd-update was taking the working kubic snapshot () and run zypper update systemd which updates systemd (along with systemd-sysvinit and udev packages to same version). That consistently made it fail. So I think a pointer to the root-cause might still be hiding somewhere around this changelog - even if it is not cgroupsv2. That part was just a theory (seemed like a "big" thing), but the update of systemd package is fairly hard evidence.
I have identified where these particular symptoms (probably) originate, and cursory testing suggests something like this eliminates the issue: I'll be advocating for a few mins in today's sig-node discussion if any folks following this thread want to chime in. cc
/sig cluster-lifecycle /triage accepted
I've also observed that on some occasions the node allocatable storage is in a continual state of reporting "0". Restarting kubelet immediately fixed that. More debugging to do....
did you manage to reproduce this with an arbitrary static pod that is not the etcd one?
I was not able to reproduce in a non-kubeadm test, but I suspect that the way kubelet bootstrap is implemented there (and possibly environmental reasons) the static pod handling was always happening after this code executed: (the above is in the containerManager method) kubeadm w/ ephemeral-storage requirements in etcd is the most effective repro at this time
the writing of the etcd pod manifest in during is after the kubelet has already started and i wonder if that is contributing to the observed failures. probably not given this is a cadvisor related problem. to verify that i would write an arbitrary static pod manifest with the same request in : before (which also starts the kubelet) right after the kubelet has started if is causing such racy problems i don't think it should be graduated to GA. though, it has been Beta for ~12 releases (on by default), so it's already used everywhere.
Based on your assessment do you think we should remove the ephemeral-storage requirement from default kubeadm etcd static pod? I can make that change.
based on the investigations here (thanks all), it feels like is a feature that may or may not work (for static pods at least). it doesn't feel right to mitigate a kubelet bug on the side of the deployer, but given how many reports there are about this i think it should be done at this point. happy to LGTM approve your PRs. note it must also be backported to all versions in support (back to 1.19). suggesting the following release note: we should keep this issue open, too.
Sounds good. I'll submit a PR soon with the kubeadm change, and I'll continue to drive toward a root cause solution on the kubelet side. I'll bring this up at sig-node tomorrow. Thanks!
is a possible duplicate of this issue - seeing a CrashLoopBackOff failure for kube-apiserver under a similar scenario to those mentioned here, e.g. failure to join a third control plane node with mentions of in the kubelet logs.
An ugly as sin workaroud: (please don't hate me :D )
It seems like this could be closed with and related cherry picks now being part of the newest releases? Bumping to 1.21.2 allowed me to remove the hack for etcd and several test deployments have steered clear entirely of the original issue. Thanks
great news. I'd prefer to pin this issue to this PR: That PR will deliver more race condition resilience for all static pod + ephemeral-storage requirement scenarios.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:


Currently whenever a vSphere volume is provisioned, we are looking at all nodes in the cluster and then fetching their hosts and then their datastores and then ensuring that selected datastore in present on all the nodes. This is done every time a new PVC is provisioned. It can be pretty expensive in large clusters and number of involved API calls can be big. /sig storage /area vsphere cc
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : Currently whenever a vSphere volume is provisioned, we are looking at all nodes in the cluster and then fetching their hosts and then their datastores and then ensuring that selected datastore in present on all the nodes. This is done every time a new PVC is provisioned. /area vsphere Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/triage accepted
/assign


Seen while running on If hack/update- fails, the output should make it clear Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /area code-organization /sig architecture /cc /assign
Maybe we can add a to check the exit code.
/cc
something like that seems like a good idea
usually when we want to do stuff like that we instead record the exit code for specific steps with something like so we can check after and inform the user of the specific failure (and suggest what to do), then
/assign


Scheduler throughput drops significantly in http://perf- ! After , seems to be the curlspirit.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/priority critical-urgent /sig scheduling /sig scalability


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --Serializing a v2beta2 HorizontalPodAutoscaler with (and ) produces yaml that cannot be applied due to a ValidationError on the field which is under the optional field . A valid HPA serialized with should be able to be applied to the cluster with no validation errors, without filling out the optional field. Other valid objects (such as Namespaces and Deployments) can be applied directly, without filling out their optional fields. Run this to generate the serialized yaml (same outcome with both and functions) Running the go code should yield this yaml: Writing the above yaml to a file and applying it (with ) will result in: Kubernetes version (use ): Server: v1.20.2, Client: v1.20.4 go version: go1.16 darwin/amd64 has: cc cc
/sig api-machinery /sig autoscaling /sig cli
/assign /triage accepted
marking conditions as +optional in the API type would resolve the client-side apply error
It would remove the validation error, but I'd be concerned about any side effects or unwanted outcomes when moving that field from required to optional. A somewhat analogous field is required. But if you anticipate this won't change much else, that will work.
Client-side validation is non-authoritative, and problematic for the reasons described in this issue ... the client is disallowing a request the server would actually have no problems with. No field the server would permit to be empty (e.g. succeeds with ) should be marked as required for this reason. fwiw, openapi schema generation marks fields tagged as as optional:
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
/remove-lifecycle rotten


<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --ci-kubernetes-e2e-ubuntu-gce [sig-storage] Subpath Container restart should verify that container can restart successfully after configmaps modified links to appreciated links to specific failures in spyglass appreciated Triage shows that this has been flaking quite frequently: It has flaked in 5 of the last 10 runs: /sig storage /priority important-soon <!-- Please see the deflaking doc () for more guidance! --
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Noticed some more flakes on this one
Yep, looks like this is still flaking quite frequently: Going to re-ping SIG Storage.
/assign
Test is flaking maybe half of the time. ~This test is failing in gce-ubuntu-master-default - not sure if it's failing in any other dashboards, nor whether this is run anywhere else.~ edit: This is failing on Ubuntu only. Not sure if this is limited to master branch. Fix to the original issue was in since 1.19. This is a subpath test which a configmap a container that mounts the configmap a liveness probe and waits for the container to restart. liveness probe and waits for the container to stop restarting. The test failed at the last step - the container never stopped restarting. Timeout is 2min. The liveness probe continuously checks whether a "probeFile" exists in (an emptyDir), and if it doesn't exist, whether the configmap file contains the modified value. The probeFile essentially acts as a condition which guards the probe from checking the configmap file. The test fails the liveness probe by removing the probeFile without updating the configmap just yet. It then fixes the liveness probe by updating the configmap to the modified value. So if the container keeps restarting, the liveness probe could've been failing continuously within 2min, due to one of the following: (? not sure about this one) Command can't be executed (for example if container can't start) configmap file was never updated. Within 2min, the pod only restarted 5 times. The liveness probe's period is 2s, initial delay is 1s, and failure threshold is 1, so a restart due to liveness probe failure is triggered every 3s. The image is busybox. The command is just , so it's unlikely for the container to have crashed. There were no liveness probe unhealthy events after the test fixes the probe, and yet it continued to restart. No more events after probe is fixed.
Could we check if the configmap value was updated for sure to rule out that scenario ? I have not tried to run an end to end test locally :) so am not sure how to debug it
Will do! I ran the test in a GKE Ubuntu cluster running 1.19.6 and couldn't reproduce the failure. I'm now trying to run a cluster on GCE with Ubuntu nodes running k8s master, but haven't been successful after trying a number of techniques. If anyone happens to know how to do this let me know; I'll keep asking around otherwise.
I got a repro on a GKE cluster backed by COS. After the test failed the container ends up in Running state eventually, and the configmap file outputs the expected modified value. The fix is to increase the test timeout. PR coming up.


found this in while finding beta API dependencies for 1.22. This will block from running at all in 1.22 I think this is for this deployment: , and in ingress-gce I found
/sig network /triage accepted I pinged some GCE sig-network folks in the PR
given that uses extensions/v1beta1 Deployments, that particular manifest seems unused
once we land the fix in , anyone know how to update the image we use in e2e tests?
/assign The update to use v1 APIs is merged to in What version should be tagged we can pick up in Does need to be picked back to one or more release branches?
It would be helpful to get a version containing the fix tagged, so the manifest can be updated in to get e2e signal even before opens for 1.22.
Ack. We are planning to cut a release soon. Once the release is cut, I will update the k/k to use that
what's the ETA for cutting ingress-gce release? I think it might be related to failing pull-kubernetes-e2e-gce-100-performance run in


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --In , client-go don't share transport when c.Dial is not nil, but kubelet custorm Dial by default, After PR is integrated, the connections between kubelet and kube-apiserver in our cluster with 4000 nodes increases by five times. kubelet share transport by default, one kubelet only keep one connection to kube-apiserver. start kubelet with rotate-certificates=false Kubernetes version (use ): v1.19.4 Cloud provider or hardware configuration: OS (e.g: ):EulerOS 2.9 Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig api-machinery /sig node
/triage accepted /assign
how many connections between kubelet and kube apiserver on one node after PR in your case? it's mentioned it increases by five times, does that mean before the PR, there is one connection, after the PR, there are 5 or 6 connections?
before the PR, there is one connection, after the PR, there 3 connections, there are some customization capabilities in our product, and we'll have another two connections. In our kubelet startup parameters, rotate-certificates=false
Some increase was expected, since multiple REST clients constructed from a config with a custom dialer cannot safely share a transport, and client-go constructs a REST client for each API group/version accessed. Something like would be required to rework how client-go constructs clients to start sharing REST clients above the transport level between API groups/versions. I think this is working as intended until client-go client construction is reworked.
do you know where in the code the 5 connections are created? are they all safe to share the transport? also are the 5 connections all active all the time? if not, IdleConnTimeout should cause the connection to be closed.
I test the code on master branch, if I change rotateCertificates to false, there are 6 connections between kubelet and kube-apiserver ! !
transport will share when config.Transport not nil if rotateCertificates is true, config.Transport will been set in kubeletcertificate.UpdateTransport but if rotateCertificates is false, config.Transport will not been set.
/area kubelet Any update on this issue?
I will work on this issue
Hi if rotate-certificates=false kubelet customizes the dialer to provide a closeAllConns to close the connection when the connection is dead but not been closed. In , there is a discuss about another solution after client-go has HTTP 2.0 health check, so we don't need this change. So we can solve the current problem by reverting . I have submitted to revert , and this change requires your final confirmation. Thanks you.
are we guaranteed kubelets always speak to kube-apiserver over http/2? if they use a load-balancer that is not http/2 capable, isn't the "close all connections" fix still required as a backstop?
I thought http1.1 wasn't affected because it will always try to connect a new connection
I'm not confident in that. The original issue dated back to before client-go defaulted to http/2, and we definitely saw issues with stuck TCP connections then.
yeah, I could reproduce it with http1.1, however, it can be solved using the transport CloseIdleConnections method please take a look , the last commit needs more work, but I think that the 2 first commits exposeing the CloseIdleConnections() are useful
ok, this seems to work , but it will be nice to have more feedback would you be able to test and see if this solve your issue?
Hi aojea, I have test on v1.21.1 before merge , there are 6 connections to kube-apiserver from kubelet ! after merge , there is only 1 connection to kube-apiserver from kubelet ! so, can solve my issue. thank you very much.


is wrong as it says default label can be specified, but KEP and other lines of changelog mentions annotation. Changelog is correct and consistent about default container feature for kubectl. I'm still not able to get this feature to work actually for . Kubernetes version (use ): v1.21.0
/triage accepted thanks for reporting early!
/remove-kind bug /kind documentation
/sig release


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --panic as below when an Endpoint doesn't have zone. should print zone as if an Endpoint's zone is unset. not set "" for Nodes. Pods and export them as a service. kubectl to describe the corresponding EndpointSlice. Kubernetes version (use ): v1.21.0 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig network /sig cli /assign


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --The default fluent-bit configuration for GCE Windows nodes produces unparsed JSON logs and does not forward container logs. Container log output is produced. fluent-bit on a given windows image. a GCE Windows e2e cluster and view the stackdriver output for pods. Kubernetes version (use ): 1.20+ Cloud provider or hardware configuration: GKE/GCE OS (e.g: ): Windows Kernel (e.g. ): Any version Install tools: fluent-bit Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig windows


I am aware that the following is a corner case that does not occur in day-to-day business, but it took me a long while to debug, so I thought I would raise this issue to save others from the same pain. Deploying the following job with empty imagePullSecrets leads to it being stuck in forever, and status looking the following way: The status should look something like: This would allow my autoscaler to kick in and add the corresponding nodes. Run Kubernetes version (use ):Cloud provider or hardware configuration: AWS EKS OS (e.g: ): MacOSX 11.3 Kernel (e.g. ): Darwin Kernel Version 20.4.0 Install tools: brew
/sig scheduling (I assume)
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /sig scheduling (I assume) Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign Scheduler log
Should we instead forbid empty items in ? Is there a use case for using as ?
For me personally that would be totally fine I had no use case for it. It was more produced by chance as I wasn't aware that I had to handle the case separately. An error which led to an early failure would have been fine by me and solved all my problems in tracking this down.
/assign as
Forbidding it is the appropriate solution. However, we can't do it for backwards compatibility of the API. /triage accepted


This issue is a bucket placeholder for collaborating on the "Known Issues" additions for the 1.22 Release Notes(). If you know of issues or API changes that are going out in 1.22, please comment here so that we can coordinate incorporating information about these changes in the Release Notes. /assign /sig release /milestone v1.22
GitHub didn't allow me to assign the following users: simcard0000. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : This issue is a bucket placeholder for collaborating on the "Known Issues" additions for the 1.22 Release Notes(). If you know of issues or API changes that are going out in 1.19, please comment here so that we can coordinate incorporating information about these changes in the Release Notes. /sig release /milestone v1.22 Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/triage accepted
1.19 or 1.22 ?
/assign
For the 1.21 example in , should be changed to
Please add the following to the Known Issues section of the 1.22 Release Notes: CC
Can you also mention the CSIMigrationVSphere feature gate in the note as well?
what should I mention about the CSIMigrationVSphere feature gate? If someone uses vSphere CSI Driver, he/she should not upgrade to v1.22 whether CSIMigrationVSphere is on or off. If someone does not use vSphere CSI Driver, then we also don't need to worry about this feature gate.
If the feature gate is enabled, they should also hold off on upgrading to 1.22
The main concern as far as k8s release is - it is intree to CSI migration of vSphere volumes that is broken in 1.22. So I think our release note should say something like:


Expose totalAnnotationSizeLimitB in the validation package: This value is useful for validating annotations outside of the validation package, such as in SSA: Otherwise, size validation on annotations outside of the package would diverge and cause issues later. /kind cleanup /wg api-expression cc
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign


[X] Test PR: kuberenetes/kubernetes [x] Two weeks soak end date: 1 July 2021 [x] Test promotion PR: kubernetes/kubernetes According to this APIsnoop query, there are still some remaining StatefulSet endpoints which are untested. Create a watch to track deployment events. Create a deployment with a static label. Confirm that the pods are running. Get the deployment status. Parse the response and confirm that the deployment status conditions can be listed. Update the deployment status. Confirm via the watch that the status has been updated. Patch the deployment status. Confirm via the watch that the status has been patched. Using an existing as a template for a new for deployment lifecycle test. This query shows the endpoints hit within a short period of running the e2e test If a test with these calls gets merged, test coverage will go up by 1 point This test is also created with the goal of conformance promotion. /sig apps /sig testing /sig architecture /area conformance
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


As part of the Kubernetes upgrade workflow we have in place, we are upgrading kubernetes from 1.15.3 -1.18.15 (both versions and all versions in the middle are custom patched for different behavior of the and . None of them having anything to do with kube-proxy's IPVS), and as part of this upgrade we are also migrating the from to mode. We have calico 3.12 running on the cluster as part of the CNI stack. Once the pods are migrated from to mode by applying the new DaemonSet manifest, the cluster has a series of stale rules for chain left around that breaks the runtime state of the pods. have cluster running in HA mode (3 Node cluster) where all nodes of the cluster are running as leaders and take the workload as well. There is no non-leader Pods. have customized the kubernetes and calico infra to enable the usage of Link local IP address in range for internal usecases Most of the IPs and the names have been mangled. Stale IP table rules to be cleaned up once the pods and endpoints are stable with all endpoints in Ready address mode. down the pods on a given namespace to 0 (leave the service and Endpoints around) the control plane to 1.18.15 (take the intermediate hop path. 1.15.3 -1.16.7 -1.17.5 -1.18.15) apply the kube-proxy manifest provided above. up the pods scaled down in Step 1. Current workaround we have been putting it use the as an init container and run the and then bring up the in IPVS mode and that seem to take care of the stale entries (but this seem to be too much of a big hammer approach of cleaning up) Kubernetes version (use ):Cloud provider or hardware configuration:OS (e.g: ):Kernel (e.g. ):Install tools: kubeadmNetwork plugin and version (if this is a network-related bug):Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The label(s) cannot be applied. These labels are supported: <detailsIn response to : /label area/kube-proxy Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /sig sig/network Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig network
I don't think that the live migration between kube-proxy implementations iptables<-ipvs is supported, at least I never heard about that and seems tricky. you use to have all the historical context, what do you think?
We tried implementing a cleanup mode but it was problematic and removed. (see ) You can run but that's pretty coarse. Maybe we should add a value to that flag to say which mode to try to cleanup? PRs welcome. The simplest answer is to reboot.
Let's make this a docs issues - we should write this down
can we merge the above so that we can close this issue?
not just yet :-) wouldn't signal registration and calling clean-up would at least clean things up on graceful shutdown? Granted rules will not be there until kube-proxy is restarted.


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --kubelet panics on creation of inline ephemeral volume with FSType specified. creation of inline ephemeral volume succeeds with FSType specified. no existing isolated test case. Happens during e2e testing of pmem-csi. Kubernetes version (use ): The pmem-csi CI system where this was detected, runs same tests on k8s systems with 1.19 and 1.20. The problem occurs in testing on 1.20 system, problem does not occur on 1.19 system.Cloud provider or hardware configuration: pmem-csi tests run on cluster formed by few qemu-based systems running Fedora-33 and emulated pmem storage. The issue has been seen in e2e testing in , where a new test was recently introduced that creates a pod with an inline ephemeral volume. If the related VolumeSource has filesystem type set to empty "", or FSType is omitted in spec, then the pod creation works OK. In that case, pmem-csi chooses it's default fstype which is . Example of working definition of VolumeSource: If the filesystem type in the VolumeSource is set to or then kubelet fails to create the volume and pod. A panic trace is written in the node kubelet log and the creation will be retried few times, On pmem-csi side: the 1st NodePublishVolume is received, namespace and volume are created, formatted with mkfs, mounted and NodePublishVolume returns OK. Same request is received few times with increasing time interval. e2e testing code which tries to create the pod, will fail with:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/sig node storage
... and ownership changed for those volumes.