Question:
What would you like to be : e2e test for service ExternalIPs Why is this needed: There is no test covering Service ExternalIPs Since we can not assume that the e2e binary will have connectivity against the nodes and that we can add routes to the environment, we are going to test it from a pod. I've verified that this works locally with the following commands: The test should leverage current framework, maybe something as A good example is here, maybe adding a ?
/sig network /help wanted /triage accepted
/help wanted /good-first-issue
/assign
I'd like to pick this up!
Ok,I'm not started :) /unassign /assign
There are two different externalIP codepaths, one for the case where the claimed external IP exists as an IP on the default interface, and one for where it does not. I suppose we could just temporarily add an IP to the default interface...
good idea Dan Please remember, it has to work for IPv6 and for IPv4 ... anyway, we can address these things during the review
This has been partially implemented with . Covering the code path for when an IP is does not exist on a node. I'll assign this back to you so you can split this issue or close it in favor of another issue. Thanks! /unassign /assign
what is the different codepath for ExternalIPs that already exist on the host? ~My reading is that the only difference is that it opens a local socket on that port, can you expand on this?~ Found it
so, it seems that this work in some environments because they are not using the bridge plugin why is not allowed that pods can reach ExternalIPs?
Wow, this is ancient and needs some fixing. This is not about preventing pod traffic from reaching those IPs. I had to really stare at it, and I am not 100% in my interpretation. First, we recognize such traffic from "off machine" (!physdev-is-in && !src=LOCAL, which we should probably switch to localDetector) and steer it to the Service. Second, we recognize ANY such traffic that is destined for a LOCAL address and shortcut that to the Service, Implicitly, traffic destined for this IP where the IP is not local goes to the system's default routing logic. The reliance on the bridge plugin is awful and archaic and I will buy a beverage of your choice for whomever gets rid of that. I think localDetector can cover this, but not 100% sure
but the ExternalIP belongs to a service and we(kube-proxy) "knows" where to send it (to the service) , that is the part I'm not sure to understand, because whatever the user is doing with that ExternalIP it has to send it back to the cluster to reach the service. However, there is always context that is important and maybe I'm missing something. Assuming we want to handle ALL the ExternalIP traffic with kube-proxy, the logic will be much simpler: traffic to an externalIP will be ALWAYS sent to the Service (DNAT) if traffic is NOT LOCAL, we also SNAT
/remove-good-first-issue /remove-help-wanted

) destChain := svcXlbChain <del> // We have to SNAT packets to external IPs if externalTrafficPolicy is cluster. </del> <ins> // We have to SNAT packets to external IPs if externalTrafficPolicy is cluster // and the traffic is NOT Local. Local traffic coming from Pods and Nodes will // be always forwarded to the corresponding Service, so no need to SNAT // If we can't differentiate the local traffic we always SNAT. </ins> if !svcInfo.OnlyNodeLocalEndpoints() { destChain = svcChain <del> writeLine(proxier.natRules, append(args, "-j", string(KubeMarkMasqChain))...) </del> <ins> // This masquerades off-cluster traffic to a External IP. if proxier.localDetector.IsImplemented() { writeLine(proxier.natRules, proxier.localDetector.JumpIfNotLocal(args, string(KubeMarkMasqChain))...) } else { writeLine(proxier.natRules, append(args, "-j", string(KubeMarkMasqChain))...) } </ins> } <ins> // Sent traffic bound for external IPs to the service chain. writeLine(proxier.natRules, append(args, "-j", string(destChain))...) </ins> <del> // Allow traffic for external IPs that does not come from a bridge (i.e. not from a container) // nor from a local process to be forwarded to the service. // This rule roughly translates to "all traffic from off-machine". // This is imperfect in the face of network plugins that might not use a bridge, but we can revisit that later. externalTrafficOnlyArgs := append(args, "-m", "physdev", "!", "--physdev-is-in", "-m", "addrtype", "!", "--src-type", "LOCAL") writeLine(proxier.natRules, append(externalTrafficOnlyArgs, "-j", string(destChain))...) dstLocalOnlyArgs := append(args, "-m", "addrtype", "--dst-type", "LOCAL") // Allow traffic bound for external IPs that happen to be recognized as local IPs to stay local. // This covers cases like GCE load-balancers which get added to the local routing table. writeLine(proxier.natRules, append(dstLocalOnlyArgs, "-j", string(destChain))...) </del> } else { // No endpoints. writeLine(proxier.filterRules,


Question:
What would you like to be : e2e test for service ExternalIPs Why is this needed: There is no test covering Service ExternalIPs Since we can not assume that the e2e binary will have connectivity against the nodes and that we can add routes to the environment, we are going to test it from a pod. I've verified that this works locally with the following commands: The test should leverage current framework, maybe something as A good example is here, maybe adding a ?
/sig network /help wanted /triage accepted
/help wanted /good-first-issue
/assign
I'd like to pick this up!
Ok,I'm not started :) /unassign /assign
There are two different externalIP codepaths, one for the case where the claimed external IP exists as an IP on the default interface, and one for where it does not. I suppose we could just temporarily add an IP to the default interface...
good idea Dan Please remember, it has to work for IPv6 and for IPv4 ... anyway, we can address these things during the review
This has been partially implemented with . Covering the code path for when an IP is does not exist on a node. I'll assign this back to you so you can split this issue or close it in favor of another issue. Thanks! /unassign /assign
what is the different codepath for ExternalIPs that already exist on the host? ~My reading is that the only difference is that it opens a local socket on that port, can you expand on this?~ Found it
so, it seems that this work in some environments because they are not using the bridge plugin why is not allowed that pods can reach ExternalIPs?
Wow, this is ancient and needs some fixing. This is not about preventing pod traffic from reaching those IPs. I had to really stare at it, and I am not 100% in my interpretation. First, we recognize such traffic from "off machine" (!physdev-is-in && !src=LOCAL, which we should probably switch to localDetector) and steer it to the Service. Second, we recognize ANY such traffic that is destined for a LOCAL address and shortcut that to the Service, Implicitly, traffic destined for this IP where the IP is not local goes to the system's default routing logic. The reliance on the bridge plugin is awful and archaic and I will buy a beverage of your choice for whomever gets rid of that. I think localDetector can cover this, but not 100% sure
but the ExternalIP belongs to a service and we(kube-proxy) "knows" where to send it (to the service) , that is the part I'm not sure to understand, because whatever the user is doing with that ExternalIP it has to send it back to the cluster to reach the service. However, there is always context that is important and maybe I'm missing something. Assuming we want to handle ALL the ExternalIP traffic with kube-proxy, the logic will be much simpler: traffic to an externalIP will be ALWAYS sent to the Service (DNAT) if traffic is NOT LOCAL, we also SNAT
/remove-good-first-issue /remove-help-wanted

return nil } <ins> func testReachabilityOverExternalIP(externalIP string, sp v1.ServicePort, execPod *v1.Pod) error { return testEndpointReachability(externalIP, sp.Port, sp.Protocol, execPod) } </ins> func testReachabilityOverNodePorts(nodes *v1.NodeList, sp v1.ServicePort, pod *v1.Pod, clusterIP string) error { internalAddrs := e2enode.CollectAddresses(nodes, v1.NodeInternalIP) externalAddrs := e2enode.CollectAddresses(nodes, v1.NodeExternalIP)


Question:
What would you like to be : e2e test for service ExternalIPs Why is this needed: There is no test covering Service ExternalIPs Since we can not assume that the e2e binary will have connectivity against the nodes and that we can add routes to the environment, we are going to test it from a pod. I've verified that this works locally with the following commands: The test should leverage current framework, maybe something as A good example is here, maybe adding a ?
/sig network /help wanted /triage accepted
/help wanted /good-first-issue
/assign
I'd like to pick this up!
Ok,I'm not started :) /unassign /assign
There are two different externalIP codepaths, one for the case where the claimed external IP exists as an IP on the default interface, and one for where it does not. I suppose we could just temporarily add an IP to the default interface...
good idea Dan Please remember, it has to work for IPv6 and for IPv4 ... anyway, we can address these things during the review
This has been partially implemented with . Covering the code path for when an IP is does not exist on a node. I'll assign this back to you so you can split this issue or close it in favor of another issue. Thanks! /unassign /assign
what is the different codepath for ExternalIPs that already exist on the host? ~My reading is that the only difference is that it opens a local socket on that port, can you expand on this?~ Found it
so, it seems that this work in some environments because they are not using the bridge plugin why is not allowed that pods can reach ExternalIPs?
Wow, this is ancient and needs some fixing. This is not about preventing pod traffic from reaching those IPs. I had to really stare at it, and I am not 100% in my interpretation. First, we recognize such traffic from "off machine" (!physdev-is-in && !src=LOCAL, which we should probably switch to localDetector) and steer it to the Service. Second, we recognize ANY such traffic that is destined for a LOCAL address and shortcut that to the Service, Implicitly, traffic destined for this IP where the IP is not local goes to the system's default routing logic. The reliance on the bridge plugin is awful and archaic and I will buy a beverage of your choice for whomever gets rid of that. I think localDetector can cover this, but not 100% sure
but the ExternalIP belongs to a service and we(kube-proxy) "knows" where to send it (to the service) , that is the part I'm not sure to understand, because whatever the user is doing with that ExternalIP it has to send it back to the cluster to reach the service. However, there is always context that is important and maybe I'm missing something. Assuming we want to handle ALL the ExternalIP traffic with kube-proxy, the logic will be much simpler: traffic to an externalIP will be ALWAYS sent to the Service (DNAT) if traffic is NOT LOCAL, we also SNAT
/remove-good-first-issue /remove-help-wanted

func (j *TestJig) checkClusterIPServiceReachability(svc *v1.Service, pod *v1.Pod) error { clusterIP := svc.Spec.ClusterIP servicePorts := svc.Spec.Ports <ins> externalIPs := svc.Spec.ExternalIPs </ins> err := j.waitForAvailableEndpoint(ServiceEndpointsTimeout) if err != nil {


Question:
What would you like to be : e2e test for service ExternalIPs Why is this needed: There is no test covering Service ExternalIPs Since we can not assume that the e2e binary will have connectivity against the nodes and that we can add routes to the environment, we are going to test it from a pod. I've verified that this works locally with the following commands: The test should leverage current framework, maybe something as A good example is here, maybe adding a ?
/sig network /help wanted /triage accepted
/help wanted /good-first-issue
/assign
I'd like to pick this up!
Ok,I'm not started :) /unassign /assign
There are two different externalIP codepaths, one for the case where the claimed external IP exists as an IP on the default interface, and one for where it does not. I suppose we could just temporarily add an IP to the default interface...
good idea Dan Please remember, it has to work for IPv6 and for IPv4 ... anyway, we can address these things during the review
This has been partially implemented with . Covering the code path for when an IP is does not exist on a node. I'll assign this back to you so you can split this issue or close it in favor of another issue. Thanks! /unassign /assign
what is the different codepath for ExternalIPs that already exist on the host? ~My reading is that the only difference is that it opens a local socket on that port, can you expand on this?~ Found it
so, it seems that this work in some environments because they are not using the bridge plugin why is not allowed that pods can reach ExternalIPs?
Wow, this is ancient and needs some fixing. This is not about preventing pod traffic from reaching those IPs. I had to really stare at it, and I am not 100% in my interpretation. First, we recognize such traffic from "off machine" (!physdev-is-in && !src=LOCAL, which we should probably switch to localDetector) and steer it to the Service. Second, we recognize ANY such traffic that is destined for a LOCAL address and shortcut that to the Service, Implicitly, traffic destined for this IP where the IP is not local goes to the system's default routing logic. The reliance on the bridge plugin is awful and archaic and I will buy a beverage of your choice for whomever gets rid of that. I think localDetector can cover this, but not 100% sure
but the ExternalIP belongs to a service and we(kube-proxy) "knows" where to send it (to the service) , that is the part I'm not sure to understand, because whatever the user is doing with that ExternalIP it has to send it back to the cluster to reach the service. However, there is always context that is important and maybe I'm missing something. Assuming we want to handle ALL the ExternalIP traffic with kube-proxy, the logic will be much simpler: traffic to an externalIP will be ALWAYS sent to the Service (DNAT) if traffic is NOT LOCAL, we also SNAT
/remove-good-first-issue /remove-help-wanted

if err != nil { return err } <ins> if len(externalIPs) > 0 { for _, externalIP := range externalIPs { err = testReachabilityOverExternalIP(externalIP, servicePort, pod) if err != nil { return err } } } </ins> } return nil }


Question:
What would you like to be : e2e test for service ExternalIPs Why is this needed: There is no test covering Service ExternalIPs Since we can not assume that the e2e binary will have connectivity against the nodes and that we can add routes to the environment, we are going to test it from a pod. I've verified that this works locally with the following commands: The test should leverage current framework, maybe something as A good example is here, maybe adding a ?
/sig network /help wanted /triage accepted
/help wanted /good-first-issue
/assign
I'd like to pick this up!
Ok,I'm not started :) /unassign /assign
There are two different externalIP codepaths, one for the case where the claimed external IP exists as an IP on the default interface, and one for where it does not. I suppose we could just temporarily add an IP to the default interface...
good idea Dan Please remember, it has to work for IPv6 and for IPv4 ... anyway, we can address these things during the review
This has been partially implemented with . Covering the code path for when an IP is does not exist on a node. I'll assign this back to you so you can split this issue or close it in favor of another issue. Thanks! /unassign /assign
what is the different codepath for ExternalIPs that already exist on the host? ~My reading is that the only difference is that it opens a local socket on that port, can you expand on this?~ Found it
so, it seems that this work in some environments because they are not using the bridge plugin why is not allowed that pods can reach ExternalIPs?
Wow, this is ancient and needs some fixing. This is not about preventing pod traffic from reaching those IPs. I had to really stare at it, and I am not 100% in my interpretation. First, we recognize such traffic from "off machine" (!physdev-is-in && !src=LOCAL, which we should probably switch to localDetector) and steer it to the Service. Second, we recognize ANY such traffic that is destined for a LOCAL address and shortcut that to the Service, Implicitly, traffic destined for this IP where the IP is not local goes to the system's default routing logic. The reliance on the bridge plugin is awful and archaic and I will buy a beverage of your choice for whomever gets rid of that. I think localDetector can cover this, but not 100% sure
but the ExternalIP belongs to a service and we(kube-proxy) "knows" where to send it (to the service) , that is the part I'm not sure to understand, because whatever the user is doing with that ExternalIP it has to send it back to the cluster to reach the service. However, there is always context that is important and maybe I'm missing something. Assuming we want to handle ALL the ExternalIP traffic with kube-proxy, the logic will be much simpler: traffic to an externalIP will be ALWAYS sent to the Service (DNAT) if traffic is NOT LOCAL, we also SNAT
/remove-good-first-issue /remove-help-wanted

framework.ExpectNoError(err) }) <ins> /* Create a ClusterIP service with an External IP that is not assigned to an interface. The IP ranges here are reserved for documentation according to [RFC 5737](https://tools.ietf.org/html/rfc5737) Section 3 and should not be used by any host. */ ginkgo.It("should be possible to connect to a service via ExternalIP when the external IP is not assigned to a node", func() { serviceName := "externalip-test" ns := f.Namespace.Name externalIP := "203.0.113.250" if framework.TestContext.ClusterIsIPv6() { externalIP = "2001:DB8::cb00:71fa" } jig := e2eservice.NewTestJig(cs, ns, serviceName) ginkgo.By("creating service " + serviceName + " with type=clusterIP in namespace " + ns) clusterIPService, err := jig.CreateTCPService(func(svc *v1.Service) { svc.Spec.Type = v1.ServiceTypeClusterIP svc.Spec.ExternalIPs = []string{externalIP} svc.Spec.Ports = []v1.ServicePort{ {Port: 80, Name: "http", Protocol: v1.ProtocolTCP, TargetPort: intstr.FromInt(9376)}, } }) framework.ExpectNoError(err) err = jig.CreateServicePods(2) framework.ExpectNoError(err) execPod := e2epod.CreateExecPodOrFail(cs, ns, "execpod", nil) err = jig.CheckServiceReachability(clusterIPService, execPod) framework.ExpectNoError(err) }) </ins> // TODO: Get rid of [DisabledForLargeClusters] tag when issue #56138 is fixed. ginkgo.It("should be able to change the type and ports of a service [Slow] [DisabledForLargeClusters]", func() { // requires cloud load-balancer support


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Using example for from documentation failed with: What you expected to happen: Example from documentation works. How to reproduce it (as minimally and precisely as possible): How should configuration look like for ? Environment: Kubernetes version (use ):OS (e.g: ):Kernel (e.g. ):
/sig scheduling
/triage accepted oh, I missed a dash. It should be: Would you mind sending a PR to kubernetes/website?

<del> ## Contributor guidelines 1. Please read our [contributor guidelines](https://github.com/kubernetes/kubernetes/blob/master/CONTRIBUTING.md). 1. See our [developer guide](https://github.com/kubernetes/kubernetes/blob/master/docs/devel/development.md). 1. Follow the instructions for [labeling and writing a release note for this PR](https://github.com/kubernetes/kubernetes/blob/master/docs/devel/pull-requests.md#release-notes) in the block below. (_this would also contain updated detail later about this new template_) ```release-note * Use the release-note-* labels to set the release note state * Clear this block to use the PR title as the release note -OR- * Enter your extended release note here ``` </del>


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Using example for from documentation failed with: What you expected to happen: Example from documentation works. How to reproduce it (as minimally and precisely as possible): How should configuration look like for ? Environment: Kubernetes version (use ):OS (e.g: ):Kernel (e.g. ):
/sig scheduling
/triage accepted oh, I missed a dash. It should be: Would you mind sending a PR to kubernetes/website?

<ins> ## Pull Request Guidelines 1. Please read our [contributor guidelines](https://github.com/kubernetes/kubernetes/blob/master/CONTRIBUTING.md). 1. See our [developer guide](https://github.com/kubernetes/kubernetes/blob/master/docs/devel/development.md). 1. Follow the instructions for [labeling and writing a release note for this PR](https://github.com/kubernetes/kubernetes/blob/master/docs/devel/pull-requests.md#release-notes) in the block below. ```release-note * Use the release-note-* labels to set the release note state * Clear this block to use the PR title as the release note -OR- * Enter your extended release note here ``` [![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/.github/PULL_REQUEST_TEMPLATE.md?pixel)]() </ins>


Question:
What happened: Process running as non-root will fail to work as it cannot access /etc/hosts inside the container when umask of kubelet is configured as 027. It is because the ioutil.WriteFile() used to create /etc/hosts is impacted by umask. Even though kubelet specify the permission of 0644 explicitly, with umask of 0027, it will be configured as 0640. As a result, application running inside the pods with non-root user, failed to access it. What you expected to happen: kubelet should pre-configure required umask by itself, instead of depending on the environment. At least we should document the required umask. How to reproduce it (as minimally and precisely as possible): Lauch kubelet with umask of 027 Anything else we need to know?: It's the same problem figured out. Environment: Kubernetes version (use ): 1.19 Cloud provider or hardware configuration: N/A OS (e.g: ): Ubuntu 18.04.4 LTS Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
/area kubelet
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-contributor-experience at . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

hostsFileContent = managedHostsFileContent(hostIPs, hostName, hostDomainName, hostAliases) } <del> return os.WriteFile(fileName, hostsFileContent, 0644) </del> <ins> hostsFilePerm := os.FileMode(0644) if err := os.WriteFile(fileName, hostsFileContent, hostsFilePerm); err != nil { return err } return os.Chmod(fileName, hostsFilePerm) </ins> } // nodeHostsFileContent reads the content of node's hosts file.


Question:
<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: Which test(s) are failing: Since when has it been failing: Testgrid link: Reason for failure: Anything else we need to know: Introduced in /cc /sig node /assign /milestone v1.20
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

fmt.Sprintf("%s::%s::%s", f.Namespace.Name, pod1, "busybox-container"): boundedSample(10*e2evolume.Kb, 80*e2evolume.Mb), }), <del> "pod_cpu_usage_seconds_total": gstruct.MatchElements(containerID, gstruct.IgnoreExtras, gstruct.Elements{ </del> <ins> "pod_cpu_usage_seconds_total": gstruct.MatchElements(podID, gstruct.IgnoreExtras, gstruct.Elements{ </ins> fmt.Sprintf("%s::%s", f.Namespace.Name, pod0): boundedSample(0, 100), fmt.Sprintf("%s::%s", f.Namespace.Name, pod1): boundedSample(0, 100), }), <del> "pod_memory_working_set_bytes": gstruct.MatchAllElements(containerID, gstruct.Elements{ </del> <ins> "pod_memory_working_set_bytes": gstruct.MatchElements(podID, gstruct.IgnoreExtras, gstruct.Elements{ </ins> fmt.Sprintf("%s::%s", f.Namespace.Name, pod0): boundedSample(10*e2evolume.Kb, 80*e2evolume.Mb), fmt.Sprintf("%s::%s", f.Namespace.Name, pod1): boundedSample(10*e2evolume.Kb, 80*e2evolume.Mb), }),


Question:
<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: Which test(s) are failing: Since when has it been failing: Testgrid link: Reason for failure: Anything else we need to know: Introduced in /cc /sig node /assign /milestone v1.20
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

return "" } <ins> func podID(element interface{}) string { el := element.(*model.Sample) return fmt.Sprintf("%s::%s", el.Metric["namespace"], el.Metric["pod"]) } </ins> func containerID(element interface{}) string { el := element.(*model.Sample) return fmt.Sprintf("%s::%s::%s", el.Metric["namespace"], el.Metric["pod"], el.Metric["container"])


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: When I register a device plugin and set the option in true, then when I create a pod and specify this resource as 0 in yaml, an error will be reported. Examples are as follows. The request to register the device pluginA pod yaml with the resource 0kubelet return error What you expected to happen: I think when the kubelet calls , the resource value should be checked. If it is 0, the step should be skipped to avoid errors. How to reproduce it (as minimally and precisely as possible): Register a device plugin which set option is true Create a pod with the above resource 0 Anything else we need to know?: In fact, when the amount of requested resources is 0, devicemanager will get the number of allocated devices as nil, resulting in the internal cache not storing the application resource information corresponding to the container. But if is set to true at the same time, then devicemanager will check podDevices cache. The two processing steps contradict each other so it will return an error.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node

podUID := string(pod.UID) contName := container.Name needsReAllocate := false <del> for k := range container.Resources.Limits { </del> <ins> for k, v := range container.Resources.Limits { </ins> resource := string(k) <del> if !m.isDevicePluginResource(resource) { </del> <ins> if !m.isDevicePluginResource(resource) || v.Value() == 0 { </ins> continue } err := m.callPreStartContainerIfNeeded(podUID, contName, resource)


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: When I register a device plugin and set the option in true, then when I create a pod and specify this resource as 0 in yaml, an error will be reported. Examples are as follows. The request to register the device pluginA pod yaml with the resource 0kubelet return error What you expected to happen: I think when the kubelet calls , the resource value should be checked. If it is 0, the step should be skipped to avoid errors. How to reproduce it (as minimally and precisely as possible): Register a device plugin which set option is true Create a pod with the above resource 0 Anything else we need to know?: In fact, when the amount of requested resources is 0, devicemanager will get the number of allocated devices as nil, resulting in the internal cache not storing the application resource information corresponding to the container. But if is set to true at the same time, then devicemanager will check podDevices cache. The two processing steps contradict each other so it will return an error.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node

as.Equal(len(runContainerOpts.Devices), len(expectedResp.Devices)) as.Equal(len(runContainerOpts.Mounts), len(expectedResp.Mounts)) as.Equal(len(runContainerOpts.Envs), len(expectedResp.Envs)) <ins> pod2 := makePod(v1.ResourceList{ v1.ResourceName(res1.resourceName): *resource.NewQuantity(int64(0), resource.DecimalSI)}) activePods = append(activePods, pod2) podsStub.updateActivePods(activePods) err = testManager.Allocate(pod2, &pod2.Spec.Containers[0]) as.Nil(err) _, err = testManager.GetDeviceRunContainerOptions(pod2, &pod2.Spec.Containers[0]) as.Nil(err) select { case <-time.After(time.Millisecond): t.Log("When pod resourceQuantity is 0,  PreStartContainer RPC stub will be skipped") case <-ch: break } </ins> } func TestResetExtendedResource(t *testing.T) {


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Below k8s UT fail when run with master-golang. TestCheckpointStateRestore/RestorecheckpointwithinvalidJSON error:=== RUN TestCheckpointStateRestore/RestorecheckpointwithinvalidJSON I1124 05:38:14. ] [cpumanager] initializing new in-memory state store unexpected error while creatng checkpointState: could not restore state from checkpoint: json: unexpected end of JSON input, please drain this node and delete the CPU manager checkpoint file "/tmp/cpumanagerstatetest479660663/cpumanagercheckpointtest" before restarting Kubelet --- FAIL: TestCheckpointStateRestore/RestorecheckpointwithinvalidJSON (0.00s) TestStrategicMergePatchInvalid error:=== RUN TestStrategicMergePatchInvalid expected "invalid character 'b' looking for beginning of value", got "json: invalid character 'b' looking for beginning of value" --- FAIL: TestStrategicMergePatchInvalid (0.00s) TestJSONPatch error:=== RUN TestJSONPatch invalid-syntax: expected error invalid character 'i' looking for beginning of value, but got json: invalid character 'i' looking for beginning of value --- FAIL: TestJSONPatch (0.00s) TestV4ErrorDecoder expandless error:=== RUN TestV4ErrorDecoder wrong error for message "{": want="error stream protocol error: unexpected end of JSON input in "{"", got="error stream protocol error: json: unexpected end of JSON input in "{"" --- FAIL: TestV4ErrorDecoder (0.00s What you expected to happen: The tests should pass with no error. How to reproduce it (as minimally and precisely as possible): a stable version of golang installed master go from upstream git clone cd go/src cd ../.. cp -r ./go /usr/local export PATH=/usr/local/go/bin:$PATH kubernetes repo and run the unit tests. () Anything else we need to know?: These failures are occurring due to mismatch of error messages between upstream go and the k8s tests. There is an addition "json:" prefix to SyntaxError messages after this below commit: Environment: Kubernetes version (use ): latest source code from upstream Cloud provider or hardware configuration: OS (e.g: ): "CentOS Linux 8 (Core)" Kernel (e.g. ): Linux rajalakshmi-workspace 4.18.0-193.19.1.el82.ppc64le SMP Mon Sep 14 14:48:04 UTC 2020 ppc64le ppc64le ppc64le GNU/Linux Install tools: [root go]# go version go version devel + Fri Nov 20 00:54:43 2020 +0000 linux/ppc64le [root go]# Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig testing
will fix it in Go 1.16 as release-blocker.
I would fix the test here to be more tolerant. exact string matches on error messages aren't really an API… I would not expect that to block the golang release
Agree too, we can think about having a substring match to avoid these issues instead!
/assign Lets see if we can fix this in the k8s tests

restoredState, err := NewCheckpointState(testingDir, testingCheckpoint, tc.policyName, tc.initialContainers) if err != nil { if strings.TrimSpace(tc.expectedError) != "" { <del> tc.expectedError = "could not restore state from checkpoint: " + tc.expectedError if strings.HasPrefix(err.Error(), tc.expectedError) { </del> <ins> if strings.Contains(err.Error(), "could not restore state from checkpoint") && strings.Contains(err.Error(), tc.expectedError) { </ins> t.Logf("got expected error: %v", err) return }


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Below k8s UT fail when run with master-golang. TestCheckpointStateRestore/RestorecheckpointwithinvalidJSON error:=== RUN TestCheckpointStateRestore/RestorecheckpointwithinvalidJSON I1124 05:38:14. ] [cpumanager] initializing new in-memory state store unexpected error while creatng checkpointState: could not restore state from checkpoint: json: unexpected end of JSON input, please drain this node and delete the CPU manager checkpoint file "/tmp/cpumanagerstatetest479660663/cpumanagercheckpointtest" before restarting Kubelet --- FAIL: TestCheckpointStateRestore/RestorecheckpointwithinvalidJSON (0.00s) TestStrategicMergePatchInvalid error:=== RUN TestStrategicMergePatchInvalid expected "invalid character 'b' looking for beginning of value", got "json: invalid character 'b' looking for beginning of value" --- FAIL: TestStrategicMergePatchInvalid (0.00s) TestJSONPatch error:=== RUN TestJSONPatch invalid-syntax: expected error invalid character 'i' looking for beginning of value, but got json: invalid character 'i' looking for beginning of value --- FAIL: TestJSONPatch (0.00s) TestV4ErrorDecoder expandless error:=== RUN TestV4ErrorDecoder wrong error for message "{": want="error stream protocol error: unexpected end of JSON input in "{"", got="error stream protocol error: json: unexpected end of JSON input in "{"" --- FAIL: TestV4ErrorDecoder (0.00s What you expected to happen: The tests should pass with no error. How to reproduce it (as minimally and precisely as possible): a stable version of golang installed master go from upstream git clone cd go/src cd ../.. cp -r ./go /usr/local export PATH=/usr/local/go/bin:$PATH kubernetes repo and run the unit tests. () Anything else we need to know?: These failures are occurring due to mismatch of error messages between upstream go and the k8s tests. There is an addition "json:" prefix to SyntaxError messages after this below commit: Environment: Kubernetes version (use ): latest source code from upstream Cloud provider or hardware configuration: OS (e.g: ): "CentOS Linux 8 (Core)" Kernel (e.g. ): Linux rajalakshmi-workspace 4.18.0-193.19.1.el82.ppc64le SMP Mon Sep 14 14:48:04 UTC 2020 ppc64le ppc64le ppc64le GNU/Linux Install tools: [root go]# go version go version devel + Fri Nov 20 00:54:43 2020 +0000 linux/ppc64le [root go]# Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig testing
will fix it in Go 1.16 as release-blocker.
I would fix the test here to be more tolerant. exact string matches on error messages aren't really an API… I would not expect that to block the golang release
Agree too, we can think about having a substring match to avoid these issues instead!
/assign Lets see if we can fix this in the k8s tests

if !apierrors.IsBadRequest(err) { t.Errorf("expected HTTP status: BadRequest, got: %#v", apierrors.ReasonForError(err)) } <del> if err.Error() != expectedError { </del> <ins> if !strings.Contains(err.Error(), expectedError) { </ins> t.Errorf("expected %#v, got %#v", expectedError, err.Error()) } }


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Below k8s UT fail when run with master-golang. TestCheckpointStateRestore/RestorecheckpointwithinvalidJSON error:=== RUN TestCheckpointStateRestore/RestorecheckpointwithinvalidJSON I1124 05:38:14. ] [cpumanager] initializing new in-memory state store unexpected error while creatng checkpointState: could not restore state from checkpoint: json: unexpected end of JSON input, please drain this node and delete the CPU manager checkpoint file "/tmp/cpumanagerstatetest479660663/cpumanagercheckpointtest" before restarting Kubelet --- FAIL: TestCheckpointStateRestore/RestorecheckpointwithinvalidJSON (0.00s) TestStrategicMergePatchInvalid error:=== RUN TestStrategicMergePatchInvalid expected "invalid character 'b' looking for beginning of value", got "json: invalid character 'b' looking for beginning of value" --- FAIL: TestStrategicMergePatchInvalid (0.00s) TestJSONPatch error:=== RUN TestJSONPatch invalid-syntax: expected error invalid character 'i' looking for beginning of value, but got json: invalid character 'i' looking for beginning of value --- FAIL: TestJSONPatch (0.00s) TestV4ErrorDecoder expandless error:=== RUN TestV4ErrorDecoder wrong error for message "{": want="error stream protocol error: unexpected end of JSON input in "{"", got="error stream protocol error: json: unexpected end of JSON input in "{"" --- FAIL: TestV4ErrorDecoder (0.00s What you expected to happen: The tests should pass with no error. How to reproduce it (as minimally and precisely as possible): a stable version of golang installed master go from upstream git clone cd go/src cd ../.. cp -r ./go /usr/local export PATH=/usr/local/go/bin:$PATH kubernetes repo and run the unit tests. () Anything else we need to know?: These failures are occurring due to mismatch of error messages between upstream go and the k8s tests. There is an addition "json:" prefix to SyntaxError messages after this below commit: Environment: Kubernetes version (use ): latest source code from upstream Cloud provider or hardware configuration: OS (e.g: ): "CentOS Linux 8 (Core)" Kernel (e.g. ): Linux rajalakshmi-workspace 4.18.0-193.19.1.el82.ppc64le SMP Mon Sep 14 14:48:04 UTC 2020 ppc64le ppc64le ppc64le GNU/Linux Install tools: [root go]# go version go version devel + Fri Nov 20 00:54:43 2020 +0000 linux/ppc64le [root go]# Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig testing
will fix it in Go 1.16 as release-blocker.
I would fix the test here to be more tolerant. exact string matches on error messages aren't really an API… I would not expect that to block the golang release
Agree too, we can think about having a substring match to avoid these issues instead!
/assign Lets see if we can fix this in the k8s tests

t.Errorf("%s: expect no error when applying json patch, but got %v", test.name, err) continue } <del> if err.Error() != test.expectedError { </del> <ins> if !strings.Contains(err.Error(), test.expectedError) { </ins> t.Errorf("%s: expected error %v, but got %v", test.name, test.expectedError, err) } if test.expectedErrorType != apierrors.ReasonForError(err) {


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Below k8s UT fail when run with master-golang. TestCheckpointStateRestore/RestorecheckpointwithinvalidJSON error:=== RUN TestCheckpointStateRestore/RestorecheckpointwithinvalidJSON I1124 05:38:14. ] [cpumanager] initializing new in-memory state store unexpected error while creatng checkpointState: could not restore state from checkpoint: json: unexpected end of JSON input, please drain this node and delete the CPU manager checkpoint file "/tmp/cpumanagerstatetest479660663/cpumanagercheckpointtest" before restarting Kubelet --- FAIL: TestCheckpointStateRestore/RestorecheckpointwithinvalidJSON (0.00s) TestStrategicMergePatchInvalid error:=== RUN TestStrategicMergePatchInvalid expected "invalid character 'b' looking for beginning of value", got "json: invalid character 'b' looking for beginning of value" --- FAIL: TestStrategicMergePatchInvalid (0.00s) TestJSONPatch error:=== RUN TestJSONPatch invalid-syntax: expected error invalid character 'i' looking for beginning of value, but got json: invalid character 'i' looking for beginning of value --- FAIL: TestJSONPatch (0.00s) TestV4ErrorDecoder expandless error:=== RUN TestV4ErrorDecoder wrong error for message "{": want="error stream protocol error: unexpected end of JSON input in "{"", got="error stream protocol error: json: unexpected end of JSON input in "{"" --- FAIL: TestV4ErrorDecoder (0.00s What you expected to happen: The tests should pass with no error. How to reproduce it (as minimally and precisely as possible): a stable version of golang installed master go from upstream git clone cd go/src cd ../.. cp -r ./go /usr/local export PATH=/usr/local/go/bin:$PATH kubernetes repo and run the unit tests. () Anything else we need to know?: These failures are occurring due to mismatch of error messages between upstream go and the k8s tests. There is an addition "json:" prefix to SyntaxError messages after this below commit: Environment: Kubernetes version (use ): latest source code from upstream Cloud provider or hardware configuration: OS (e.g: ): "CentOS Linux 8 (Core)" Kernel (e.g. ): Linux rajalakshmi-workspace 4.18.0-193.19.1.el82.ppc64le SMP Mon Sep 14 14:48:04 UTC 2020 ppc64le ppc64le ppc64le GNU/Linux Install tools: [root go]# go version go version devel + Fri Nov 20 00:54:43 2020 +0000 linux/ppc64le [root go]# Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig testing
will fix it in Go 1.16 as release-blocker.
I would fix the test here to be more tolerant. exact string matches on error messages aren't really an API… I would not expect that to block the golang release
Agree too, we can think about having a substring match to avoid these issues instead!
/assign Lets see if we can fix this in the k8s tests

import ( "fmt" <ins> "strings" </ins> "testing" )


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Below k8s UT fail when run with master-golang. TestCheckpointStateRestore/RestorecheckpointwithinvalidJSON error:=== RUN TestCheckpointStateRestore/RestorecheckpointwithinvalidJSON I1124 05:38:14. ] [cpumanager] initializing new in-memory state store unexpected error while creatng checkpointState: could not restore state from checkpoint: json: unexpected end of JSON input, please drain this node and delete the CPU manager checkpoint file "/tmp/cpumanagerstatetest479660663/cpumanagercheckpointtest" before restarting Kubelet --- FAIL: TestCheckpointStateRestore/RestorecheckpointwithinvalidJSON (0.00s) TestStrategicMergePatchInvalid error:=== RUN TestStrategicMergePatchInvalid expected "invalid character 'b' looking for beginning of value", got "json: invalid character 'b' looking for beginning of value" --- FAIL: TestStrategicMergePatchInvalid (0.00s) TestJSONPatch error:=== RUN TestJSONPatch invalid-syntax: expected error invalid character 'i' looking for beginning of value, but got json: invalid character 'i' looking for beginning of value --- FAIL: TestJSONPatch (0.00s) TestV4ErrorDecoder expandless error:=== RUN TestV4ErrorDecoder wrong error for message "{": want="error stream protocol error: unexpected end of JSON input in "{"", got="error stream protocol error: json: unexpected end of JSON input in "{"" --- FAIL: TestV4ErrorDecoder (0.00s What you expected to happen: The tests should pass with no error. How to reproduce it (as minimally and precisely as possible): a stable version of golang installed master go from upstream git clone cd go/src cd ../.. cp -r ./go /usr/local export PATH=/usr/local/go/bin:$PATH kubernetes repo and run the unit tests. () Anything else we need to know?: These failures are occurring due to mismatch of error messages between upstream go and the k8s tests. There is an addition "json:" prefix to SyntaxError messages after this below commit: Environment: Kubernetes version (use ): latest source code from upstream Cloud provider or hardware configuration: OS (e.g: ): "CentOS Linux 8 (Core)" Kernel (e.g. ): Linux rajalakshmi-workspace 4.18.0-193.19.1.el82.ppc64le SMP Mon Sep 14 14:48:04 UTC 2020 ppc64le ppc64le ppc64le GNU/Linux Install tools: [root go]# go version go version devel + Fri Nov 20 00:54:43 2020 +0000 linux/ppc64le [root go]# Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig testing
will fix it in Go 1.16 as release-blocker.
I would fix the test here to be more tolerant. exact string matches on error messages aren't really an API… I would not expect that to block the golang release
Agree too, we can think about having a substring match to avoid these issues instead!
/assign Lets see if we can fix this in the k8s tests

}, { message: "{", <del> err:     "error stream protocol error: unexpected end of JSON input in "{"", </del> <ins> err:     "unexpected end of JSON input in "{"", </ins> }, { message: `{"status": "Success" }`,


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Below k8s UT fail when run with master-golang. TestCheckpointStateRestore/RestorecheckpointwithinvalidJSON error:=== RUN TestCheckpointStateRestore/RestorecheckpointwithinvalidJSON I1124 05:38:14. ] [cpumanager] initializing new in-memory state store unexpected error while creatng checkpointState: could not restore state from checkpoint: json: unexpected end of JSON input, please drain this node and delete the CPU manager checkpoint file "/tmp/cpumanagerstatetest479660663/cpumanagercheckpointtest" before restarting Kubelet --- FAIL: TestCheckpointStateRestore/RestorecheckpointwithinvalidJSON (0.00s) TestStrategicMergePatchInvalid error:=== RUN TestStrategicMergePatchInvalid expected "invalid character 'b' looking for beginning of value", got "json: invalid character 'b' looking for beginning of value" --- FAIL: TestStrategicMergePatchInvalid (0.00s) TestJSONPatch error:=== RUN TestJSONPatch invalid-syntax: expected error invalid character 'i' looking for beginning of value, but got json: invalid character 'i' looking for beginning of value --- FAIL: TestJSONPatch (0.00s) TestV4ErrorDecoder expandless error:=== RUN TestV4ErrorDecoder wrong error for message "{": want="error stream protocol error: unexpected end of JSON input in "{"", got="error stream protocol error: json: unexpected end of JSON input in "{"" --- FAIL: TestV4ErrorDecoder (0.00s What you expected to happen: The tests should pass with no error. How to reproduce it (as minimally and precisely as possible): a stable version of golang installed master go from upstream git clone cd go/src cd ../.. cp -r ./go /usr/local export PATH=/usr/local/go/bin:$PATH kubernetes repo and run the unit tests. () Anything else we need to know?: These failures are occurring due to mismatch of error messages between upstream go and the k8s tests. There is an addition "json:" prefix to SyntaxError messages after this below commit: Environment: Kubernetes version (use ): latest source code from upstream Cloud provider or hardware configuration: OS (e.g: ): "CentOS Linux 8 (Core)" Kernel (e.g. ): Linux rajalakshmi-workspace 4.18.0-193.19.1.el82.ppc64le SMP Mon Sep 14 14:48:04 UTC 2020 ppc64le ppc64le ppc64le GNU/Linux Install tools: [root go]# go version go version devel + Fri Nov 20 00:54:43 2020 +0000 linux/ppc64le [root go]# Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig testing
will fix it in Go 1.16 as release-blocker.
I would fix the test here to be more tolerant. exact string matches on error messages aren't really an API… I would not expect that to block the golang release
Agree too, we can think about having a substring match to avoid these issues instead!
/assign Lets see if we can fix this in the k8s tests

if want == "" { want = "<nil>" } <del> if got := fmt.Sprintf("%v", err); got != want { </del> <ins> if got := fmt.Sprintf("%v", err); !strings.Contains(got, want) { </ins> t.Errorf("wrong error for message %q: want=%q, got=%q", test.message, want, got) } }


Question:
<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: bazel-test-master Which test(s) are flaking: //pkg/kubelet/cm/devicemanager/godefaulttest:run2of_2 Testgrid link: Reason for failure: Anything else we need to know: links to appreciated Triage shows nothings (seems to not have updated data), but there are multiple failures in the job bazel-test-master. links to specific failures in spyglass appreciated <!-- Please see the deflaking doc () for more guidance! --
/sig release
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig sig-release Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Digging further as the step has another recent test fail ... So on the two runs reported by has 2 test failures has 1 test failure And the most recent failure in this area is a TestAdmit/match test failure but on a different test.
/sig node
/assign /triage accepted I'll take a look at them and fix.

func setupInProbeMode(t *testing.T, devs []*pluginapi.Device, callback monitorCallback, socketName string, pluginSocketName string) (Manager, <-chan interface{}, *Stub, pluginmanager.PluginManager) { m, updateChan := setupDeviceManager(t, devs, callback, socketName) <del> pm := setupPluginManager(t, pluginSocketName, m) </del> p := setupDevicePlugin(t, devs, pluginSocketName) <ins> pm := setupPluginManager(t, pluginSocketName, m) </ins> return m, updateChan, p, pm }


Question:
<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: bazel-test-master Which test(s) are flaking: //pkg/kubelet/cm/devicemanager/godefaulttest:run2of_2 Testgrid link: Reason for failure: Anything else we need to know: links to appreciated Triage shows nothings (seems to not have updated data), but there are multiple failures in the job bazel-test-master. links to specific failures in spyglass appreciated <!-- Please see the deflaking doc () for more guidance! --
/sig release
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig sig-release Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Digging further as the step has another recent test fail ... So on the two runs reported by has 2 test failures has 1 test failure And the most recent failure in this area is a TestAdmit/match test failure but on a different test.
/sig node
/assign /triage accepted I'll take a look at them and fix.

rc.RLock() defer rc.RUnlock() <del> return rc.handlers </del> <ins> var copyHandlers = make(map[string]cache.PluginHandler) for pluginType, handler := range rc.handlers { copyHandlers[pluginType] = handler } return copyHandlers </ins> } func (rc *reconciler) reconcile() {


Question:
What happened: I was running the subpath testsuite against Azurefile. This test failed and can be easily reproduced: Basically what this test does is: a PVC use azurefile a pod that has two containers. One of it mounts the volume itself. The second container mounts the subpath of the volume. the subpath dir from the container that has the whole volume the pod, the pod will stuck in terminating status and cannot be cleaned up other than What you expected to happen: The pod can be deleted successfully. How to reproduce it (as minimally and precisely as possible): the pvc the pod the subpath dir from the parent pod the pod Anything else we need to know?: The volume is being formatted as cifs type Environment: Kubernetes version (use ): v1.18.6 Cloud provider or hardware configuration: Azurefile-csi-driver 0.9.0 OS (e.g: ): Ubuntu 20.04 Kernel (e.g. ): Linux nodepool-- 5.4.0-1016-azure -Ubuntu SMP Thu Jun 4 22:49:15 UTC 2020 x8664 x8664 x86_64 GNU/Linux Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig storage /cc /cc
log After debugging with and offline, the issue is that: After the dir from the subpath was removed, the subpath mountpoint on the host machine becomes into a weird state where you can still find the dir by doing a readdir to its parent. But doing or something like that will result in a "No such file or directory". When you do a ls on the path:
/triage accepted
So the issue from code is that It will fail the path exists check while the actual mountpoint exists. So when the code logic runs to here: It will fail to remove the container dir because it somehow detect there is still file there. To fix this as a workaround, user can manually , and then the kubelet can take care of this. To fix it ultimately, we need to see if this is an issue specific to cifs filesystem? Or we should consider do an umount regardless if the path exists.
/assign
/assign
Tried the same scenario using NFS. It works as expected using NFS.
Yeah, that's why I am wondering if this is something special with CIFS?
Looks like it has something to do with the azurefile-csi storage class mount options. I was getting the below error when I was trying to do ls on the subpath. So I realized there might be some issue with the file permissions. root ls ls: cannot open directory '.': No such file or directory When I removed the dirmode and filemode properties from the storageclass and performed the same steps as mentioned in the issue, the pod got deleted successfully.(Although it took a lot of time 10-15 mins for the pod to get deleted)
FYI, I am planning to use this PR to fix the unmount issue in Azure File & SMB CSI driver: Not sure how to handle gracefully in k8s subPath code since that's common code.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
So this PR does not fix the Azurefile subpath test issue?
no, that PR only fixed the unmount issue inside CSI driver. While for the subPath issue, we should fix it in k8s upstream code as you mentioned above:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
back to original discussion, shall we use instead of inside func? That will solve the issue of smb file PathExists check issue.
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
this subPath issue was fixed in , ,

} return notMnt, nil } <del> // PathExists returns true if the specified path exists. // TODO: clean this up to use pkg/util/file/FileExists func PathExists(path string) (bool, error) { _, err := os.Stat(path) if err == nil { return true, nil } else if os.IsNotExist(err) { return false, nil } else if IsCorruptedMnt(err) { return true, err } return false, err } </del>


Question:
What happened: I was running the subpath testsuite against Azurefile. This test failed and can be easily reproduced: Basically what this test does is: a PVC use azurefile a pod that has two containers. One of it mounts the volume itself. The second container mounts the subpath of the volume. the subpath dir from the container that has the whole volume the pod, the pod will stuck in terminating status and cannot be cleaned up other than What you expected to happen: The pod can be deleted successfully. How to reproduce it (as minimally and precisely as possible): the pvc the pod the subpath dir from the parent pod the pod Anything else we need to know?: The volume is being formatted as cifs type Environment: Kubernetes version (use ): v1.18.6 Cloud provider or hardware configuration: Azurefile-csi-driver 0.9.0 OS (e.g: ): Ubuntu 20.04 Kernel (e.g. ): Linux nodepool-- 5.4.0-1016-azure -Ubuntu SMP Thu Jun 4 22:49:15 UTC 2020 x8664 x8664 x86_64 GNU/Linux Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig storage /cc /cc
log After debugging with and offline, the issue is that: After the dir from the subpath was removed, the subpath mountpoint on the host machine becomes into a weird state where you can still find the dir by doing a readdir to its parent. But doing or something like that will result in a "No such file or directory". When you do a ls on the path:
/triage accepted
So the issue from code is that It will fail the path exists check while the actual mountpoint exists. So when the code logic runs to here: It will fail to remove the container dir because it somehow detect there is still file there. To fix this as a workaround, user can manually , and then the kubelet can take care of this. To fix it ultimately, we need to see if this is an issue specific to cifs filesystem? Or we should consider do an umount regardless if the path exists.
/assign
/assign
Tried the same scenario using NFS. It works as expected using NFS.
Yeah, that's why I am wondering if this is something special with CIFS?
Looks like it has something to do with the azurefile-csi storage class mount options. I was getting the below error when I was trying to do ls on the subpath. So I realized there might be some issue with the file permissions. root ls ls: cannot open directory '.': No such file or directory When I removed the dirmode and filemode properties from the storageclass and performed the same steps as mentioned in the issue, the pod got deleted successfully.(Although it took a lot of time 10-15 mins for the pod to get deleted)
FYI, I am planning to use this PR to fix the unmount issue in Azure File & SMB CSI driver: Not sure how to handle gracefully in k8s subPath code since that's common code.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
So this PR does not fix the Azurefile subpath test issue?
no, that PR only fixed the unmount issue inside CSI driver. While for the subPath issue, we should fix it in k8s upstream code as you mentioned above:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
back to original discussion, shall we use instead of inside func? That will solve the issue of smb file PathExists check issue.
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
this subPath issue was fixed in , ,

package mount import ( <ins> "errors" </ins> "fmt" <ins> "io/fs" </ins> "os" "strconv" "strings" "syscall" <ins> "k8s.io/klog/v2" </ins> utilio "k8s.io/utils/io" )


Question:
What happened: I was running the subpath testsuite against Azurefile. This test failed and can be easily reproduced: Basically what this test does is: a PVC use azurefile a pod that has two containers. One of it mounts the volume itself. The second container mounts the subpath of the volume. the subpath dir from the container that has the whole volume the pod, the pod will stuck in terminating status and cannot be cleaned up other than What you expected to happen: The pod can be deleted successfully. How to reproduce it (as minimally and precisely as possible): the pvc the pod the subpath dir from the parent pod the pod Anything else we need to know?: The volume is being formatted as cifs type Environment: Kubernetes version (use ): v1.18.6 Cloud provider or hardware configuration: Azurefile-csi-driver 0.9.0 OS (e.g: ): Ubuntu 20.04 Kernel (e.g. ): Linux nodepool-- 5.4.0-1016-azure -Ubuntu SMP Thu Jun 4 22:49:15 UTC 2020 x8664 x8664 x86_64 GNU/Linux Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig storage /cc /cc
log After debugging with and offline, the issue is that: After the dir from the subpath was removed, the subpath mountpoint on the host machine becomes into a weird state where you can still find the dir by doing a readdir to its parent. But doing or something like that will result in a "No such file or directory". When you do a ls on the path:
/triage accepted
So the issue from code is that It will fail the path exists check while the actual mountpoint exists. So when the code logic runs to here: It will fail to remove the container dir because it somehow detect there is still file there. To fix this as a workaround, user can manually , and then the kubelet can take care of this. To fix it ultimately, we need to see if this is an issue specific to cifs filesystem? Or we should consider do an umount regardless if the path exists.
/assign
/assign
Tried the same scenario using NFS. It works as expected using NFS.
Yeah, that's why I am wondering if this is something special with CIFS?
Looks like it has something to do with the azurefile-csi storage class mount options. I was getting the below error when I was trying to do ls on the subpath. So I realized there might be some issue with the file permissions. root ls ls: cannot open directory '.': No such file or directory When I removed the dirmode and filemode properties from the storageclass and performed the same steps as mentioned in the issue, the pod got deleted successfully.(Although it took a lot of time 10-15 mins for the pod to get deleted)
FYI, I am planning to use this PR to fix the unmount issue in Azure File & SMB CSI driver: Not sure how to handle gracefully in k8s subPath code since that's common code.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
So this PR does not fix the Azurefile subpath test issue?
no, that PR only fixed the unmount issue inside CSI driver. While for the subPath issue, we should fix it in k8s upstream code as you mentioned above:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
back to original discussion, shall we use instead of inside func? That will solve the issue of smb file PathExists check issue.
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
this subPath issue was fixed in , ,

underlyingError = pe.Err case *os.SyscallError: underlyingError = pe.Err <ins> case syscall.Errno: underlyingError = err </ins> } return underlyingError == syscall.ENOTCONN || underlyingError == syscall.ESTALE || underlyingError == syscall.EIO || underlyingError == syscall.EACCES || underlyingError == syscall.EHOSTDOWN


Question:
What happened: I was running the subpath testsuite against Azurefile. This test failed and can be easily reproduced: Basically what this test does is: a PVC use azurefile a pod that has two containers. One of it mounts the volume itself. The second container mounts the subpath of the volume. the subpath dir from the container that has the whole volume the pod, the pod will stuck in terminating status and cannot be cleaned up other than What you expected to happen: The pod can be deleted successfully. How to reproduce it (as minimally and precisely as possible): the pvc the pod the subpath dir from the parent pod the pod Anything else we need to know?: The volume is being formatted as cifs type Environment: Kubernetes version (use ): v1.18.6 Cloud provider or hardware configuration: Azurefile-csi-driver 0.9.0 OS (e.g: ): Ubuntu 20.04 Kernel (e.g. ): Linux nodepool-- 5.4.0-1016-azure -Ubuntu SMP Thu Jun 4 22:49:15 UTC 2020 x8664 x8664 x86_64 GNU/Linux Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig storage /cc /cc
log After debugging with and offline, the issue is that: After the dir from the subpath was removed, the subpath mountpoint on the host machine becomes into a weird state where you can still find the dir by doing a readdir to its parent. But doing or something like that will result in a "No such file or directory". When you do a ls on the path:
/triage accepted
So the issue from code is that It will fail the path exists check while the actual mountpoint exists. So when the code logic runs to here: It will fail to remove the container dir because it somehow detect there is still file there. To fix this as a workaround, user can manually , and then the kubelet can take care of this. To fix it ultimately, we need to see if this is an issue specific to cifs filesystem? Or we should consider do an umount regardless if the path exists.
/assign
/assign
Tried the same scenario using NFS. It works as expected using NFS.
Yeah, that's why I am wondering if this is something special with CIFS?
Looks like it has something to do with the azurefile-csi storage class mount options. I was getting the below error when I was trying to do ls on the subpath. So I realized there might be some issue with the file permissions. root ls ls: cannot open directory '.': No such file or directory When I removed the dirmode and filemode properties from the storageclass and performed the same steps as mentioned in the issue, the pod got deleted successfully.(Although it took a lot of time 10-15 mins for the pod to get deleted)
FYI, I am planning to use this PR to fix the unmount issue in Azure File & SMB CSI driver: Not sure how to handle gracefully in k8s subPath code since that's common code.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
So this PR does not fix the Azurefile subpath test issue?
no, that PR only fixed the unmount issue inside CSI driver. While for the subPath issue, we should fix it in k8s upstream code as you mentioned above:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
back to original discussion, shall we use instead of inside func? That will solve the issue of smb file PathExists check issue.
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
this subPath issue was fixed in , ,

deletedDir := fmt.Sprintf("%s040(deleted)", dir) return ((mp.Path == dir) || (mp.Path == deletedDir)) } <ins> // PathExists returns true if the specified path exists. // TODO: clean this up to use pkg/util/file/FileExists func PathExists(path string) (bool, error) { _, err := os.Stat(path) if err == nil { return true, nil } else if errors.Is(err, fs.ErrNotExist) { err = syscall.Access(path, syscall.F_OK) if err == nil { // The access syscall says the file exists, the stat syscall says it // doesn't. This was observed on CIFS when the path was removed at // the server somehow. POSIX calls this a stale file handle, let's fake // that error and treat the path as existing but corrupted. klog.Warningf("Potential stale file handle detected: %s", path) return true, syscall.ESTALE } return false, nil } else if IsCorruptedMnt(err) { return true, err } return false, err } </ins>


Question:
What happened: I was running the subpath testsuite against Azurefile. This test failed and can be easily reproduced: Basically what this test does is: a PVC use azurefile a pod that has two containers. One of it mounts the volume itself. The second container mounts the subpath of the volume. the subpath dir from the container that has the whole volume the pod, the pod will stuck in terminating status and cannot be cleaned up other than What you expected to happen: The pod can be deleted successfully. How to reproduce it (as minimally and precisely as possible): the pvc the pod the subpath dir from the parent pod the pod Anything else we need to know?: The volume is being formatted as cifs type Environment: Kubernetes version (use ): v1.18.6 Cloud provider or hardware configuration: Azurefile-csi-driver 0.9.0 OS (e.g: ): Ubuntu 20.04 Kernel (e.g. ): Linux nodepool-- 5.4.0-1016-azure -Ubuntu SMP Thu Jun 4 22:49:15 UTC 2020 x8664 x8664 x86_64 GNU/Linux Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig storage /cc /cc
log After debugging with and offline, the issue is that: After the dir from the subpath was removed, the subpath mountpoint on the host machine becomes into a weird state where you can still find the dir by doing a readdir to its parent. But doing or something like that will result in a "No such file or directory". When you do a ls on the path:
/triage accepted
So the issue from code is that It will fail the path exists check while the actual mountpoint exists. So when the code logic runs to here: It will fail to remove the container dir because it somehow detect there is still file there. To fix this as a workaround, user can manually , and then the kubelet can take care of this. To fix it ultimately, we need to see if this is an issue specific to cifs filesystem? Or we should consider do an umount regardless if the path exists.
/assign
/assign
Tried the same scenario using NFS. It works as expected using NFS.
Yeah, that's why I am wondering if this is something special with CIFS?
Looks like it has something to do with the azurefile-csi storage class mount options. I was getting the below error when I was trying to do ls on the subpath. So I realized there might be some issue with the file permissions. root ls ls: cannot open directory '.': No such file or directory When I removed the dirmode and filemode properties from the storageclass and performed the same steps as mentioned in the issue, the pod got deleted successfully.(Although it took a lot of time 10-15 mins for the pod to get deleted)
FYI, I am planning to use this PR to fix the unmount issue in Azure File & SMB CSI driver: Not sure how to handle gracefully in k8s subPath code since that's common code.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
So this PR does not fix the Azurefile subpath test issue?
no, that PR only fixed the unmount issue inside CSI driver. While for the subPath issue, we should fix it in k8s upstream code as you mentioned above:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
back to original discussion, shall we use instead of inside func? That will solve the issue of smb file PathExists check issue.
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
this subPath issue was fixed in , ,

func isMountPointMatch(mp MountPoint, dir string) bool { return mp.Path == dir } <ins> // PathExists returns true if the specified path exists. // TODO: clean this up to use pkg/util/file/FileExists func PathExists(path string) (bool, error) { _, err := os.Stat(path) if err == nil { return true, nil } else if os.IsNotExist(err) { return false, nil } else if IsCorruptedMnt(err) { return true, err } return false, err } </ins>


Question:
What happened: Adding a VolumeMount to a container with a multi-part subpath results in only the leaf directory having 777 permissions. If you define e.g. then within the volume, and will have permissions, whereas will have permissions. This means that if you want to remove and eventually from within a pod, you will need to use a privileged pod. If the sub-directories already exist, their permissions are unchanged, and the order in which these subpaths are created appears to be the order in which they appear within the spec -- if we see first, then will be created with 755 permissions and will be created with 777 permissions, even if there's another volumeMount for further down. What you expected to happen: Sub-directories created to support a volumeMount should all have the same permissions as the leaf directory. How to reproduce it (as minimally and precisely as possible): the yaml below. The first example has volume mounts so that is created before ; the second example has volume mounts in reversed order, so is encountered before , which results in being unwritable. <details<summaryThis pod will complete</summary</details<details<summaryThis pod will fail</summary</detailsAnything else we need to know?: I believe this is the return of Environment: Reproduces on minikube and a (OpenShift) development cluster I have access to Kubernetes version (use ): minikube: OpenShift cluster: output: Cloud provider or hardware configuration: Minikube: v1.15.1; docker driver; hostpath volumes (all default) OpenShift cluster: AWS, EBS volumes OS (e.g: ): Minikube: Fedora 32 OpenShift cluster: RHEL 8.2 Kernel (e.g. ): Minikube: OpenShift cluster: Install tools: minikube v1.15.1 / Not sure Network plugin and version (if this is a network-related bug): N/A Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage
/assign
GitHub didn't allow me to assign the following users: mengjiao-liu. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign

} parentFD = childFD childFD = -1 <del> } // Everything was created. mkdirat(..., perm) above was affected by current // umask and we must apply the right permissions to the last directory // (that's the one that will be available to the container as subpath) // so user can read/write it. This is the behavior of previous code. // TODO: chmod all created directories, not just the last one. // parentFD is the last created directory. </del> <ins> // Everything was created. mkdirat(..., perm) above was affected by current // umask and we must apply the right permissions to the all created directory. // (that's the one that will be available to the container as subpath) // so user can read/write it. // parentFD is the last created directory. </ins> <del> // Translate perm (os.FileMode) to uint32 that fchmod() expects kernelPerm := uint32(perm & os.ModePerm) if perm&os.ModeSetgid > 0 { kernelPerm |= syscall.S_ISGID } if perm&os.ModeSetuid > 0 { kernelPerm |= syscall.S_ISUID } if perm&os.ModeSticky > 0 { kernelPerm |= syscall.S_ISVTX } if err = syscall.Fchmod(parentFD, kernelPerm); err != nil { return fmt.Errorf("chmod %q failed: %s", currentPath, err) </del> <ins> // Translate perm (os.FileMode) to uint32 that fchmod() expects kernelPerm := uint32(perm & os.ModePerm) if perm&os.ModeSetgid > 0 { kernelPerm |= syscall.S_ISGID } if perm&os.ModeSetuid > 0 { kernelPerm |= syscall.S_ISUID } if perm&os.ModeSticky > 0 { kernelPerm |= syscall.S_ISVTX } if err = syscall.Fchmod(parentFD, kernelPerm); err != nil { return fmt.Errorf("chmod %q failed: %s", currentPath, err) } </ins> } return nil }