Question:
Kubelet and System-reserve are implemented on Windows along with pod/container reserves which can help prevent overprovisioning, but we still need to test OOM behavior. We need a test case that will: kubelet & system reserve pods until node is full something that will take up system ram so that OOM is triggered /sig windows /milestone v1.14
I have started to look into this test case and have a few questions: What is the expected behavior we are testing for? What does "run something that will take up system ram" mean? Does that mean run a pod that uses up memory or run a system service that uses memory? After setting the reserves and as mentioned in we should be able to schedule pods until full and then once we trigger OOM (either by pod or system service) kubelet should continue running becuase of the reserves? Do the two settings need to be tested independently or they should be tested together? I found that seems to be doing something similar but it is in node-e2e. Is sig-windows running node-e2e tests currently?
/assign
GitHub didn't allow me to assign the following users: jsturtevant. Note that only and repo collaborators can be assigned and that issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
After researching this further and talking with I found that the does test the allocatable memory but uses CGroups to set reserves to test the reported vs actual. As windows does not have CGroups this test will not port as is. There is and the is not implemented so will not work either. The proposed testing solution would be two parts: kubelet & system reserve and verify allocatable is reporting properly. Schedule pods with known memory consumption values and verify the allocatable is correclty updated.Schedule until full and verify scheduling is not allowed. kubelet & system reserve and verify allocatable is reporting properly. Schedule Schedule until full and verify scheduling is not into machine and run process that consumes memory and verify kubelet and pods remain running. There is an open question on the second test as to whether kubelet and will be have properly. If it does not we will submit test 1 and open an issue to resolve 2.
I would suggest adding a third case that would test the behavior when excessive memory pressure is exerted on the node from the pods. This is perhaps a more common use case than 2 since the pod workloads are more likely to exhibit greater resource variability than node system processes. It would also carry over as a test for when/if node allocatable enforcement get implemented for windows nodes. Additionally, to complement this work would it be suitable to test windows pod memory limit enforcement? Or is this tested elsewhere?
Agreed this is a more common case. I am not sure that it is. Is this tested on linux somewhere?
I am seeing strange numbers reported when performing a test case. I scheduled a few pods to test memory limits and then schedule a pod with no memory limit and a process that will consume all of the available memory 500mb at a time: After the pod consumes all available memory, When I compare Node Allocatable/Node usage/Pod Usage I am not seeing the numbers add up properly: My node is staying online and kubelet is reporting no memory pressure. The total node memory consumption is over the Allocatable memory. What is the expected behavior and what should the test look like?
I believe node allocatable will always just be capacity minus reserves (system and kubelet) - So in this scenario it looks like you have and/or set? The memory % on the node is relative to the allocatable amount, and still seems to be under the total memory capacity - are you sure that all available memory has been consumed by your oom pod? The node should be tainted with memory-pressure when a hard or soft eviction threshold is met (). What do you have set for and ?
Thanks for the clarification. I currently have these settings: And on the machine I am seeing most memory being used: If I understand correctly, I start to see memory-pressure once I get into the setting above. In that case I should start to see pods get evicted, so for a valid test I could adjust the values to be easier to hit then test to see that pods get evicted properly which would be similar the . I am going to adjust a few of the settings and manually test that this is working then see what it will take to port the PriorityMemoryEviction, which currently uses cgroups which won't work on windows. This combined with testing that Memory Allocatable is being properly set should give us a good test coverage for windows on this scenario.
I performed manual testing and found that even when memory was used up on the machine the "Memory Pressure" notification was not fired. After looking deeper I found is not implemented for windows. The , which uses cgroups, provides the . This means even though is being it isn't used. On Windows, Node allocatable when used with limits and requests does prevent the node from over scheduling. I believe the the test and  (this only shows cpu and I could not find a test that demonstrated memory) demonstrate this works. But this doesn't stop a pod from consuming all the memory on the node and since eviction (hard/soft/node allocatable) is not implemented on windows this means a node can end up resource starved. I think the only valid test to be written for this scenario, as it stands now, is validating that Node Allocatable is being reported properly. Beyond that there is no functionality that can be validated around OOM behavior for the windows node.
Yeah, that sounds to be in agreement with what I've observed on the windows nodes that I run. To prevent resource starvation (as a workaround), would setting mem/cpu requests equal to mem/cpu limits for all windows pods work? I think the request / limit values get passed down into the docker daemon when it starts a container, so is it possible that docker could enforce the memory limit?
Yes, setting limits==requests would help the scheduler not over provision the nodes. I think the best way to approach this particular issue for 1.14 is to make sure docs call out what works/doesn't work and give guidance to on how to prevent resource starvation on nodes. Is there a place in the docs that would make sense for this information? The call outs would be: , , and flags do update Node Allocatable is not implemented and are not implemented Condition is not implemented in Windows There are no OOM actions taken by kubelet As a work around to previous, and should always be equal to prevent memory resource starvation as this allows the scheduler to to use limits and Kubelet running on the windows node does not have memory restrictions. and system-reserve does not set limits on kubelet or processes running the host. This means kubelet or a process on the host could cause memory resource starvation outside the and scheduler
thank you for the thorough investigation

return controller } <del> // newTestPods creates a list of pods (specification) for test. func newTestPods(numPods int, volume bool, imageName, podType string) []*v1.Pod { </del> <ins> // newDensityTestPods creates a list of pods (specification) for test. func newDensityTestPods(numPods int, volume bool, imageName, podType string) []*v1.Pod { </ins> var pods []*v1.Pod for i := 0; i < numPods; i++ { podName := "test-" + string(uuid.NewUUID()) <del> labels := map[string]string{ "type": podType, "name": podName, </del> <ins> pod := v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name: podName, Labels: map[string]string{ "type": podType, "name": podName, }, }, Spec: v1.PodSpec{ // Restart policy is always (default). Containers: []v1.Container{ { Image: imageName, Name:  podName, }, }, NodeSelector: map[string]string{ "beta.kubernetes.io/os": "windows", }, }, </ins> } if volume { <del> pods = append(pods, &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name:   podName, Labels: labels, }, Spec: v1.PodSpec{ // Restart policy is always (default). Containers: []v1.Container{ { Image: imageName, Name:  podName, VolumeMounts: []v1.VolumeMount{ {MountPath: "/test-volume-mnt", Name: podName + "-volume"}, }, }, }, NodeSelector: map[string]string{ "beta.kubernetes.io/os": "windows", }, Volumes: []v1.Volume{ {Name: podName + "-volume", VolumeSource: v1.VolumeSource{EmptyDir: &v1.EmptyDirVolumeSource{}}}, }, }, }) } else { pods = append(pods, &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name:   podName, Labels: labels, }, Spec: v1.PodSpec{ // Restart policy is always (default). Containers: []v1.Container{ { Image: imageName, Name:  podName, }, }, NodeSelector: map[string]string{ "beta.kubernetes.io/os": "windows", }, }, }) </del> <ins> pod.Spec.Containers[0].VolumeMounts = []v1.VolumeMount{ {MountPath: "/test-volume-mnt", Name: podName + "-volume"}, } pod.Spec.Volumes = []v1.Volume{ {Name: podName + "-volume", VolumeSource: v1.VolumeSource{EmptyDir: &v1.EmptyDirVolumeSource{}}}, } </ins> } <ins> pods = append(pods, &pod) </ins> } return pods }


Question:
Kubelet and System-reserve are implemented on Windows along with pod/container reserves which can help prevent overprovisioning, but we still need to test OOM behavior. We need a test case that will: kubelet & system reserve pods until node is full something that will take up system ram so that OOM is triggered /sig windows /milestone v1.14
I have started to look into this test case and have a few questions: What is the expected behavior we are testing for? What does "run something that will take up system ram" mean? Does that mean run a pod that uses up memory or run a system service that uses memory? After setting the reserves and as mentioned in we should be able to schedule pods until full and then once we trigger OOM (either by pod or system service) kubelet should continue running becuase of the reserves? Do the two settings need to be tested independently or they should be tested together? I found that seems to be doing something similar but it is in node-e2e. Is sig-windows running node-e2e tests currently?
/assign
GitHub didn't allow me to assign the following users: jsturtevant. Note that only and repo collaborators can be assigned and that issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
After researching this further and talking with I found that the does test the allocatable memory but uses CGroups to set reserves to test the reported vs actual. As windows does not have CGroups this test will not port as is. There is and the is not implemented so will not work either. The proposed testing solution would be two parts: kubelet & system reserve and verify allocatable is reporting properly. Schedule pods with known memory consumption values and verify the allocatable is correclty updated.Schedule until full and verify scheduling is not allowed. kubelet & system reserve and verify allocatable is reporting properly. Schedule Schedule until full and verify scheduling is not into machine and run process that consumes memory and verify kubelet and pods remain running. There is an open question on the second test as to whether kubelet and will be have properly. If it does not we will submit test 1 and open an issue to resolve 2.
I would suggest adding a third case that would test the behavior when excessive memory pressure is exerted on the node from the pods. This is perhaps a more common use case than 2 since the pod workloads are more likely to exhibit greater resource variability than node system processes. It would also carry over as a test for when/if node allocatable enforcement get implemented for windows nodes. Additionally, to complement this work would it be suitable to test windows pod memory limit enforcement? Or is this tested elsewhere?
Agreed this is a more common case. I am not sure that it is. Is this tested on linux somewhere?
I am seeing strange numbers reported when performing a test case. I scheduled a few pods to test memory limits and then schedule a pod with no memory limit and a process that will consume all of the available memory 500mb at a time: After the pod consumes all available memory, When I compare Node Allocatable/Node usage/Pod Usage I am not seeing the numbers add up properly: My node is staying online and kubelet is reporting no memory pressure. The total node memory consumption is over the Allocatable memory. What is the expected behavior and what should the test look like?
I believe node allocatable will always just be capacity minus reserves (system and kubelet) - So in this scenario it looks like you have and/or set? The memory % on the node is relative to the allocatable amount, and still seems to be under the total memory capacity - are you sure that all available memory has been consumed by your oom pod? The node should be tainted with memory-pressure when a hard or soft eviction threshold is met (). What do you have set for and ?
Thanks for the clarification. I currently have these settings: And on the machine I am seeing most memory being used: If I understand correctly, I start to see memory-pressure once I get into the setting above. In that case I should start to see pods get evicted, so for a valid test I could adjust the values to be easier to hit then test to see that pods get evicted properly which would be similar the . I am going to adjust a few of the settings and manually test that this is working then see what it will take to port the PriorityMemoryEviction, which currently uses cgroups which won't work on windows. This combined with testing that Memory Allocatable is being properly set should give us a good test coverage for windows on this scenario.
I performed manual testing and found that even when memory was used up on the machine the "Memory Pressure" notification was not fired. After looking deeper I found is not implemented for windows. The , which uses cgroups, provides the . This means even though is being it isn't used. On Windows, Node allocatable when used with limits and requests does prevent the node from over scheduling. I believe the the test and  (this only shows cpu and I could not find a test that demonstrated memory) demonstrate this works. But this doesn't stop a pod from consuming all the memory on the node and since eviction (hard/soft/node allocatable) is not implemented on windows this means a node can end up resource starved. I think the only valid test to be written for this scenario, as it stands now, is validating that Node Allocatable is being reported properly. Beyond that there is no functionality that can be validated around OOM behavior for the windows node.
Yeah, that sounds to be in agreement with what I've observed on the windows nodes that I run. To prevent resource starvation (as a workaround), would setting mem/cpu requests equal to mem/cpu limits for all windows pods work? I think the request / limit values get passed down into the docker daemon when it starts a container, so is it possible that docker could enforce the memory limit?
Yes, setting limits==requests would help the scheduler not over provision the nodes. I think the best way to approach this particular issue for 1.14 is to make sure docs call out what works/doesn't work and give guidance to on how to prevent resource starvation on nodes. Is there a place in the docs that would make sense for this information? The call outs would be: , , and flags do update Node Allocatable is not implemented and are not implemented Condition is not implemented in Windows There are no OOM actions taken by kubelet As a work around to previous, and should always be equal to prevent memory resource starvation as this allows the scheduler to to use limits and Kubelet running on the windows node does not have memory restrictions. and system-reserve does not set limits on kubelet or processes running the host. This means kubelet or a process on the host could cause memory resource starvation outside the and scheduler
thank you for the thorough investigation

<ins> /* Copyright 2019 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package windows import ( "crypto/tls" "encoding/json" "fmt" "io/ioutil" "net/http" "regexp" "strconv" "time" v1 "k8s.io/api/core/v1" "k8s.io/apimachinery/pkg/api/resource" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/labels" "k8s.io/apimachinery/pkg/util/uuid" "k8s.io/client-go/kubernetes/scheme" kubeletconfigv1beta1 "k8s.io/kubelet/config/v1beta1" kubeletconfig "k8s.io/kubernetes/pkg/kubelet/apis/config" "k8s.io/kubernetes/test/e2e/framework" imageutils "k8s.io/kubernetes/test/utils/image" "github.com/onsi/ginkgo" "github.com/onsi/gomega" ) var _ = SIGDescribe("[Feature:Windows] Memory Limits [Serial] [Slow]", func() { f := framework.NewDefaultFramework("memory-limit-test-windows") ginkgo.BeforeEach(func() { // NOTE(vyta): these tests are Windows specific framework.SkipUnlessNodeOSDistroIs("windows") }) ginkgo.Context("Allocatable node memory", func() { ginkgo.It("should be equal to a calculated allocatable memory value", func() { checkNodeAllocatableTest(f) }) }) ginkgo.Context("attempt to deploy past allocatable memory limits", func() { ginkgo.It("should fail deployments of pods once there isn't enough memory", func() { overrideAllocatableMemoryTest(f, 4) }) }) }) type nodeMemory struct { // capacity capacity resource.Quantity // allocatable memory allocatable resource.Quantity // memory reserved for OS level processes systemReserve resource.Quantity // memory reserved for kubelet (not implemented) kubeReserve resource.Quantity // grace period memory limit (not implemented) softEviction resource.Quantity // no grace period memory limit hardEviction resource.Quantity } // runDensityBatchTest runs the density batch pod creation test // checks that a calculated value for NodeAllocatable is equal to the reported value func checkNodeAllocatableTest(f *framework.Framework) { nodeMem := getNodeMemory(f) framework.Logf("nodeMem says: %+v", nodeMem) // calculate the allocatable mem based on capacity - reserved amounts calculatedNodeAlloc := nodeMem.capacity.Copy() calculatedNodeAlloc.Sub(nodeMem.systemReserve) calculatedNodeAlloc.Sub(nodeMem.kubeReserve) calculatedNodeAlloc.Sub(nodeMem.softEviction) calculatedNodeAlloc.Sub(nodeMem.hardEviction) ginkgo.By(fmt.Sprintf("Checking stated allocatable memory %v against calculated allocatable memory %v", &nodeMem.allocatable, calculatedNodeAlloc)) // sanity check against stated allocatable gomega.Expect(calculatedNodeAlloc.Cmp(nodeMem.allocatable)).To(gomega.Equal(0)) } // Deploys `allocatablePods + 1` pods, each with a memory limit of `1/allocatablePods` of the total allocatable // memory, then confirms that the last pod failed because of failedScheduling func overrideAllocatableMemoryTest(f *framework.Framework, allocatablePods int) { const ( podType = "memory_limit_test_pod" ) totalAllocatable := getTotalAllocatableMemory(f) memValue := totalAllocatable.Value() memPerPod := memValue / int64(allocatablePods) ginkgo.By(fmt.Sprintf("Deploying %d pods with mem limit %v, then one additional pod", allocatablePods, memPerPod)) // these should all work pods := newMemLimitTestPods(allocatablePods, imageutils.GetPauseImageName(), podType, strconv.FormatInt(memPerPod, 10)) f.PodClient().CreateBatch(pods) failurePods := newMemLimitTestPods(1, imageutils.GetPauseImageName(), podType, strconv.FormatInt(memPerPod, 10)) f.PodClient().Create(failurePods[0]) gomega.Eventually(func() bool { eventList, err := f.ClientSet.CoreV1().Events(f.Namespace.Name).List(metav1.ListOptions{}) framework.ExpectNoError(err) for _, e := range eventList.Items { // Look for an event that shows FailedScheduling if e.Type == "Warning" && e.Reason == "FailedScheduling" && e.InvolvedObject.Name == failurePods[0].ObjectMeta.Name { framework.Logf("Found %+v event with message %+v", e.Reason, e.Message) return true } } return false }, 3*time.Minute, 10*time.Second).Should(gomega.Equal(true)) } // newMemLimitTestPods creates a list of pods (specification) for test. func newMemLimitTestPods(numPods int, imageName, podType string, memoryLimit string) []*v1.Pod { var pods []*v1.Pod memLimitQuantity, err := resource.ParseQuantity(memoryLimit) framework.ExpectNoError(err) for i := 0; i < numPods; i++ { podName := "test-" + string(uuid.NewUUID()) pod := v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name: podName, Labels: map[string]string{ "type": podType, "name": podName, }, }, Spec: v1.PodSpec{ // Restart policy is always (default). Containers: []v1.Container{ { Image: imageName, Name:  podName, Resources: v1.ResourceRequirements{ Limits: v1.ResourceList{ v1.ResourceMemory: memLimitQuantity, }, }, }, }, NodeSelector: map[string]string{ "beta.kubernetes.io/os": "windows", }, }, } pods = append(pods, &pod) } return pods } // getNodeMemory populates a nodeMemory struct with information from the first func getNodeMemory(f *framework.Framework) nodeMemory { selector := labels.Set{"beta.kubernetes.io/os": "windows"}.AsSelector() nodeList, err := f.ClientSet.CoreV1().Nodes().List(metav1.ListOptions{ LabelSelector: selector.String(), }) framework.ExpectNoError(err) // Assuming that agent nodes have the same config // Make sure there is >0 agent nodes, then use the first one for info gomega.Expect(nodeList.Size()).NotTo(gomega.Equal(0)) ginkgo.By("Getting memory details from node status and kubelet config") status := nodeList.Items[0].Status nodeName := nodeList.Items[0].ObjectMeta.Name kubeletConfig, err := getCurrentKubeletConfig(nodeName) framework.ExpectNoError(err) systemReserve, err := resource.ParseQuantity(kubeletConfig.SystemReserved["memory"]) if err != nil { systemReserve = *resource.NewQuantity(0, resource.BinarySI) } kubeReserve, err := resource.ParseQuantity(kubeletConfig.KubeReserved["memory"]) if err != nil { kubeReserve = *resource.NewQuantity(0, resource.BinarySI) } hardEviction, err := resource.ParseQuantity(kubeletConfig.EvictionHard["memory.available"]) if err != nil { hardEviction = *resource.NewQuantity(0, resource.BinarySI) } softEviction, err := resource.ParseQuantity(kubeletConfig.EvictionSoft["memory.available"]) if err != nil { softEviction = *resource.NewQuantity(0, resource.BinarySI) } nodeMem := nodeMemory{ capacity:      status.Capacity[v1.ResourceMemory], allocatable:   status.Allocatable[v1.ResourceMemory], systemReserve: systemReserve, hardEviction:  hardEviction, // these are not implemented and are here for future use - will always be 0 at the moment kubeReserve:  kubeReserve, softEviction: softEviction, } return nodeMem } // getTotalAllocatableMemory gets the sum of all agent node's allocatable memory func getTotalAllocatableMemory(f *framework.Framework) *resource.Quantity { selector := labels.Set{"beta.kubernetes.io/os": "windows"}.AsSelector() nodeList, err := f.ClientSet.CoreV1().Nodes().List(metav1.ListOptions{ LabelSelector: selector.String(), }) framework.ExpectNoError(err) ginkgo.By("Summing allocatable memory across all agent nodes") totalAllocatable := resource.NewQuantity(0, resource.BinarySI) for _, node := range nodeList.Items { status := node.Status totalAllocatable.Add(status.Allocatable[v1.ResourceMemory]) } return totalAllocatable } // getCurrentKubeletConfig modified from test/e2e_node/util.go func getCurrentKubeletConfig(nodeName string) (*kubeletconfig.KubeletConfiguration, error) { resp := pollConfigz(5*time.Minute, 5*time.Second, nodeName) kubeCfg, err := decodeConfigz(resp) if err != nil { return nil, err } return kubeCfg, nil } // Causes the test to fail, or returns a status 200 response from the /configz endpoint func pollConfigz(timeout time.Duration, pollInterval time.Duration, nodeName string) *http.Response { // start local proxy, so we can send graceful deletion over query string, rather than body parameter ginkgo.By("Opening proxy to cluster") cmd := framework.KubectlCmd("proxy", "-p", "0") stdout, stderr, err := framework.StartCmdAndStreamOutput(cmd) framework.ExpectNoError(err) defer stdout.Close() defer stderr.Close() defer framework.TryKill(cmd) buf := make([]byte, 128) var n int n, err = stdout.Read(buf) framework.ExpectNoError(err) output := string(buf[:n]) proxyRegexp := regexp.MustCompile("Starting to serve on 127.0.0.1:([0-9]+)") match := proxyRegexp.FindStringSubmatch(output) gomega.Expect(len(match)).To(gomega.Equal(2)) port, err := strconv.Atoi(match[1]) framework.ExpectNoError(err) ginkgo.By("http requesting node kubelet /configz") endpoint := fmt.Sprintf("http://127.0.0.1:%d/api/v1/nodes/%s/proxy/configz", port, nodeName) tr := &http.Transport{ TLSClientConfig: &tls.Config{InsecureSkipVerify: true}, } client := &http.Client{Transport: tr} req, err := http.NewRequest("GET", endpoint, nil) framework.ExpectNoError(err) req.Header.Add("Accept", "application/json") var resp *http.Response gomega.Eventually(func() bool { resp, err = client.Do(req) if err != nil { framework.Logf("Failed to get /configz, retrying. Error: %v", err) return false } if resp.StatusCode != 200 { framework.Logf("/configz response status not 200, retrying. Response was: %+v", resp) return false } return true }, timeout, pollInterval).Should(gomega.Equal(true)) return resp } // Decodes the http response from /configz and returns a kubeletconfig.KubeletConfiguration (internal type). func decodeConfigz(resp *http.Response) (*kubeletconfig.KubeletConfiguration, error) { // This hack because /configz reports the following structure: // {"kubeletconfig": {the JSON representation of kubeletconfigv1beta1.KubeletConfiguration}} type configzWrapper struct { ComponentConfig kubeletconfigv1beta1.KubeletConfiguration `json:"kubeletconfig"` } configz := configzWrapper{} kubeCfg := kubeletconfig.KubeletConfiguration{} contentsBytes, err := ioutil.ReadAll(resp.Body) if err != nil { return nil, err } err = json.Unmarshal(contentsBytes, &configz) if err != nil { return nil, err } err = scheme.Scheme.Convert(&configz.ComponentConfig, &kubeCfg, nil) if err != nil { return nil, err } return &kubeCfg, nil } </ins>


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: We created a LoadBalancer service with type 'nlb' with two listeners forwarding traffic on two different ports. The service was created correctly along with the target groups and was functionning correctly. Around 20 hours later, no traffic was flowing from clients to any k8s nodes. While troubleshooting we discovered that the k8s node instance security group was lacking the input rules allowing traffic to the node ports for this LB. We deleted the service and recreated it. The service started working normally again, but we found out that the two target groups were missing their k8s tags. The logs contain the following error: What you expected to happen: We expected the NLB target groups to have k8s tags, so that k8s could remember it created them. How to reproduce it (as minimally and precisely as possible): Create a k8s service with NLB: Add a simple deployment that is matched by this service. Anything else we need to know?: Looking at AWS cloudtrail we could see the following error event: So the API request was called with two Resource ARNs, which is confirmed by the code: Since we have two target groups, there are two resources to tag. Except that the AWS API while allowing to send more than one resource ARNs doesn't allow it returning a validation error. This could be solved by doing one per target groups. Note that this can't happen on a Classic LB because there's no target groups. I believe that those missing tags on the target groups are the root cause of the deletion of the security group rules allowing traffic to reach the nodeports, with this (hypothetical and unproved) scenario: 8s creates the target groups, the NLB, the listeners, the sg input rules 8s fails to tag the target groups, but doesn't rollback anything a subsequent convergence loops, k8s notices that there are two k8s-managed Security Group input rules, but those don't match any Target Groups or match Target Groups that are not managed by k8s (because they don't have the owner tags) 8s deletes those rules Unfortunately we don't see the rule deletion in the controller manager logs, but it appears in Cloudtrail: Let me know if you have any questions, Environment: Kubernetes version: 1.10.7 Cloud provider or hardware configuration: AWS OS (e.g. from /etc/os-release): CoreOS 1967.3.0 Kernel (e.g. ): Linux 4.14.88-coreos Install tools: home grown Others:
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig aws /assign
Hi, the root cause of your problem should already be solved by It's cherry-picked into old versions and available as v1.11.7, v1.12.5 and v1.13.3(hasn't released yet)
, Thanks, you're most certainly right about the root cause of the issue being . It indeed looks exactly like what we've experienced. Still there's the problem of target groups not being tagged.

loadBalancer = createResponse.LoadBalancers[0] // Create Target Groups <del> addTagsInput := &elbv2.AddTagsInput{ ResourceArns: []*string{}, Tags:         []*elbv2.Tag{}, } </del> <ins> resourceArns := make([]*string, 0, len(mappings)) </ins> for i := range mappings { // It is easier to keep track of updates by having possibly


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: We created a LoadBalancer service with type 'nlb' with two listeners forwarding traffic on two different ports. The service was created correctly along with the target groups and was functionning correctly. Around 20 hours later, no traffic was flowing from clients to any k8s nodes. While troubleshooting we discovered that the k8s node instance security group was lacking the input rules allowing traffic to the node ports for this LB. We deleted the service and recreated it. The service started working normally again, but we found out that the two target groups were missing their k8s tags. The logs contain the following error: What you expected to happen: We expected the NLB target groups to have k8s tags, so that k8s could remember it created them. How to reproduce it (as minimally and precisely as possible): Create a k8s service with NLB: Add a simple deployment that is matched by this service. Anything else we need to know?: Looking at AWS cloudtrail we could see the following error event: So the API request was called with two Resource ARNs, which is confirmed by the code: Since we have two target groups, there are two resources to tag. Except that the AWS API while allowing to send more than one resource ARNs doesn't allow it returning a validation error. This could be solved by doing one per target groups. Note that this can't happen on a Classic LB because there's no target groups. I believe that those missing tags on the target groups are the root cause of the deletion of the security group rules allowing traffic to reach the nodeports, with this (hypothetical and unproved) scenario: 8s creates the target groups, the NLB, the listeners, the sg input rules 8s fails to tag the target groups, but doesn't rollback anything a subsequent convergence loops, k8s notices that there are two k8s-managed Security Group input rules, but those don't match any Target Groups or match Target Groups that are not managed by k8s (because they don't have the owner tags) 8s deletes those rules Unfortunately we don't see the rule deletion in the controller manager logs, but it appears in Cloudtrail: Let me know if you have any questions, Environment: Kubernetes version: 1.10.7 Cloud provider or hardware configuration: AWS OS (e.g. from /etc/os-release): CoreOS 1967.3.0 Kernel (e.g. ): Linux 4.14.88-coreos Install tools: home grown Others:
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig aws /assign
Hi, the root cause of your problem should already be solved by It's cherry-picked into old versions and available as v1.11.7, v1.12.5 and v1.13.3(hasn't released yet)
, Thanks, you're most certainly right about the root cause of the issue being . It indeed looks exactly like what we've experienced. Still there's the problem of target groups not being tagged.

if err != nil { return nil, fmt.Errorf("Error creating listener: %q", err) } <del> addTagsInput.ResourceArns = append(addTagsInput.ResourceArns, targetGroupArn) </del> <ins> resourceArns = append(resourceArns, targetGroupArn) </ins> } // Add tags to targets <ins> targetGroupTags := make([]*elbv2.Tag, 0, len(tags)) </ins> for k, v := range tags { <del> addTagsInput.Tags = append(addTagsInput.Tags, &elbv2.Tag{ </del> <ins> targetGroupTags = append(targetGroupTags, &elbv2.Tag{ </ins> Key: aws.String(k), Value: aws.String(v), }) } <del> if len(addTagsInput.ResourceArns) > 0 && len(addTagsInput.Tags) > 0 { _, err = c.elbv2.AddTags(addTagsInput) if err != nil { return nil, fmt.Errorf("Error adding tags after creating Load Balancer: %q", err) </del> <ins> if len(resourceArns) > 0 && len(targetGroupTags) > 0 { // elbv2.AddTags doesn't allow to tag multiple resources at once for _, arn := range resourceArns { _, err = c.elbv2.AddTags(&elbv2.AddTagsInput{ ResourceArns: []*string{arn}, Tags:         targetGroupTags, }) if err != nil { return nil, fmt.Errorf("Error adding tags after creating Load Balancer: %q", err) } </ins> } } } else {


Question:
What happened: What you expected to happen: Human readable error output. Anything else we need to know?: The same is happening when in duplicate key while creating a with binaries. !!! The error output will be all the of the binary itself !!! Environment: Kubernetes version (use ):
/sig cli
/cc

} if _, exists := configMap.Data[keyName]; exists { <del> return fmt.Errorf("cannot add key %s, another key by that name already exists in data: %v", keyName, configMap.Data) </del> <ins> return fmt.Errorf("cannot add key %q, another key by that name already exists in Data for ConfigMap %q", keyName, configMap.Name) </ins> } if _, exists := configMap.BinaryData[keyName]; exists { <del> return fmt.Errorf("cannot add key %s, another key by that name already exists in binaryData: %v", keyName, configMap.BinaryData) </del> <ins> return fmt.Errorf("cannot add key %q, another key by that name already exists in BinaryData for ConfigMap %q", keyName, configMap.Name) </ins> } return nil }


Question:
What happened: What you expected to happen: Human readable error output. Anything else we need to know?: The same is happening when in duplicate key while creating a with binaries. !!! The error output will be all the of the binary itself !!! Environment: Kubernetes version (use ):
/sig cli
/cc

} if _, entryExists := secret.Data[keyName]; entryExists { <del> return fmt.Errorf("cannot add key %s, another key by that name already exists: %v", keyName, secret.Data) </del> <ins> return fmt.Errorf("cannot add key %s, another key by that name already exists", keyName) </ins> } secret.Data[keyName] = data return nil


Question:
What happened: We currently unregister the CSI driver if an error happens while registering it. Then, if another error happens, we return only the unregistration error. The problem is that the user will only see the unregistration error if they check the registrar logs. This gives the impression that it never tried to register the driver. What you expected to happen: The user should see the registration error instead. How to reproduce it (as minimally and precisely as possible): Try to install a CSI driver in a cluster where CSINodeInfo and/or CSIDriver CRDs are not created. Check registrar logs /kind bug /sig storage CC
/assign

driverNodeID, maxVolumePerNode, accessibleTopology, err := csi.NodeGetInfo(ctx) if err != nil { <del> klog.Error(log("registrationHandler.RegisterPlugin failed at CSI.NodeGetInfo: %v", err)) </del> if unregErr := unregisterDriver(pluginName); unregErr != nil { klog.Error(log("registrationHandler.RegisterPlugin failed to unregister plugin due to previous error: %v", unregErr)) <del> return unregErr </del> } return err } err = nim.InstallCSIDriver(pluginName, driverNodeID, maxVolumePerNode, accessibleTopology) if err != nil { <del> klog.Error(log("registrationHandler.RegisterPlugin failed at AddNodeInfo: %v", err)) </del> if unregErr := unregisterDriver(pluginName); unregErr != nil { klog.Error(log("registrationHandler.RegisterPlugin failed to unregister plugin due to previous error: %v", unregErr)) <del> return unregErr </del> } return err }


Question:
Which jobs are failing: ci-kubernetes-e2e-kops-aws pull-kubernetes-e2e-kops-aws Which test(s) are failing: [sig-storage] In-tree Volumes [Driver: local][LocalVolumeType: block] [Testpattern: Pre-provisioned PV (xfs)] volumes should allow exec of files on the volume [sig-storage] In-tree Volumes [Driver: local][LocalVolumeType: block] [Testpattern: Pre-provisioned PV (xfs)] volumes should be mountable Since when has it been failing: Don't know.. as the job was down for a while due to Testgrid link: Reason for failure: To be found.. Anything else we need to know: This issue is a blocker for turning on the kops-aws presubmit, see - /cc /kind failing-test /sig storage
cc
By looking at the log this line is suspicious: Both testcases share similar error message. And that comes from when doing I guess its some mount client missing from the AMI so that the mount library can't find it.
Looks like the test needs to skip XFS on environments that do not support XFS
This is where the environment is checked to skip xfs:
Thanks Michelle! - Could you comment if we need to skip it in aws tests?
Looks like this was broken by - shall we revert? cc
Actually looking at the GCE tests for kops it was broken before . I'm wondering if we assumed xfs back in
/priority critical-urgent this is blocker for kops currently example of failing build
do you have idea could break this?
Yes, in , I tests for local volume which use . Because , PR is not blocked by gce e2e test. hi, Quick way to fix is to remove from supported types, I can submit a PR for this.
Alternatively, we can skip for environments which do not support XFS . What do you think?
I do not know what is the best solution in long term but in short term we should get kops e2e tests working. So if removing xfs from supported types is fastest solution - i would go with that. After that tests could be fixed how those should be. Could you do that PR and remove from supported types?
Hard-coding the knowledge which clusters or tests support XFS (because that's what it is - a dependency on a kernel feature, right?) in the test source code is just bound to fail eventually. What if the presence of xfs support depends on the "gci" version? This line doesn't cover that: I would prefer to mark tests which depend on XFS kernel support with or something like it (no idea how descriptive feature tags need to be) and then enable those test cases only in jobs where the target cluster is known to have XFS. If you extend that check as a short-term solution, then please file an issue about it and link to it next to the check. /cc
Agreeing with here... Isn't there a way to check if the kernel we test for/ against is supporting it more directly? Building a huge list of whats compatible and what's not sounds not perfect to me.
Long term we can enhance the check to ssh into a random node and see if the binary exists. Short term let's just remove xfs testing for local volumes.
If these two are newly tests, agree to skip (or remove) xfs testing to unblock PRs.
assuming all these in-tree tests will be less important over the course of the year as CSI moves forward... so should we assume that the XFS tests will never come back . ?
The local volume driver plan is to remain in-tree for the medium-term future. I think we should continue to add the environment check for xfs so we can add it back.

"ext2", "ext3", "ext4", <del> "xfs", </del> <ins> //"xfs", disabled see issue https://github.com/kubernetes/kubernetes/issues/74095 </ins> ), } // max file size


Question:
<!-- Please only use this template for submitting enhancement requests --What would you like to be : /sig testing /sig windows In the the PR we're adding a test case for pulling a Windows image from dockerhub (. However, there are no known public Windows images on , so the only option would be to upload it ourselves. Our proposal is to have it uploaded to the using Windows images). The proposed image manifest list to be uploaded is built from [1], which is currently hosted on dockerhub [2] and is currently used for testing Kubernetes with Windows nodes [3]. The manifest list contains the images: e2eteam/busybox:1.29-1803 (for Windows Server 1803) e2eteam/busybox:1.29-1809 (for Windows Server 2019) [1] [2] [3] Why is this needed: The Windows images will be used for the Windows equivalent test case for the following test:
/assign
/area conformance
It is my contention that we really only want one prod repo and that should be folded into the main repo. The authenticated repo should be for testing authenticated pulls and nothing else. To that end, I propose that we start with this. Does the image need to be named or tagged or something to indicate OS? How about * ? Is that too deep? Does it need Windows in the path or should we put it in the tag? Or can we have a multi-os manifest list? Also, is busybox the right image for testing, or can we do something even more minimal (to address the "what f there's a busybox CVE" question)?
Yes, (all images to date have used ), so I'd prefer that route over yet another name or tag. I'd also prefer something other than busybox if possible. I originally created because there were so many images in and it was hard to tell which of them were production images, which were test images, and which had been completely abandoned. Also, only a small number of Googlers could push to this repo since it contained all the production images. To address the first issue, I decided that all e2e test images would be sourced from the directory. To address the second issue, I created a new project with a wider ACL (though still currently limited to Googlers). All that said, given the automation we're building for the main GCR repo, it'd probably be fine to fold these images back in.
The image promoter means that a single repo should be BETTER managed than individual repos - we will have a PR trail of who put images there and why.
We should probably broaden the discussion to non-busybox Windows test images too. (related: ). Do we have a lower bar for test images? If so, having a separate repo (or making it clear that the images are test only) does have clear benefits.
Do we want a lower bar? I don't think we do.
Nobody wants a lower bar until they want to add/update the image quickly for a new test, or fix a test failure to unblock the release. If we do effectively hold the same bar today, this shouldn't be a problem.
i agree. also we don't need schema v2. multi-os, multi-arch image here. just (e.g.) a nanoserver image that satisfies a Windows 1803 and Windows 1809 host. a nanoserver image will obviously imply Windows only. for a test called "should be able to pull image from [WindowsOnly]": all we need is to upload this image to [x] and i guess we need to decide on if we want this test to be / passing for Windows Containers GA.
And indeed, a manifest list with nanoserver images could be an alternative (considering that the ) (we did try to find a command which works on both OSes, and ultimately found curl, but not all Linux images contain curl, check PR ).
an official nanoserver image is a more secure download. do we need to build a nanoserver manifest list or are there official images that we can use? i would like to bring a topic here related to pause. why are all the pause images for Windows that i found 1.1 GB or more? we definitely need the absolute minimal pause container uploaded in a location where users can download it and use that location in the guide. AFAIK, 100 MB is as low as we can go for a native Windows container, but that's still much better... i brought this topic in the other PR. we don't need a command that works on both OSes. if is the command to use on Windows containers only, so be it.
And also, my image? It might be out-of-date.
we need to start planning how Windows containers can used be easily on Windows nodes. the pause image should be pre-built, small and easy to pull by deployment tools, such as kops, kubeadm. that is much better, but still a bit too much? can we go down to ~100MB? this image size was reported by a user in a k/k issue recently: my assumption is that the user was building the image himself.
i am ok to have that conversation, but please file a ticket and let's discuss that in sig-windows
everyone, thank you for your help and responses. trying to bring back this conversation to the original request. what would you like us to do in order for the few folks from google that have permissions to upload the windows images to be used for testing.
I am happy to help publish this via the alias, which I believe is the correct long-term path. We need an image - preferably not one as prone to CVEs as busybox historically is. Hence a suggestion for "pause" or something equivalently useless and tiny (if it is tiny). Then, we need a path. First, we'll be setting precedent for test images, so I need input on that. I propose as the root. Objections from test folks? Second, do we want to squirrel this away as a Windwows-specific binary, or should we not name it in any particular way, so that we can pull the same image for all tests via a multi-os, multi-arch manifest list (which can be done later). E.g. vs How about * ? Is that too deep? Does it need Windows in the path or should we put it in the tag? Michael Michael <:
So, I'll sumarize the previous discussions. The proposed images are: The image currently named image tags:. based on: respectively. image is currently used in Windows E2E tests and should already exist on the nodes, so download should be minimal. The image. a manifest list needs to be created containing the tags: The image is used to tag the image, which is used by kubelet, so it should already exist on the nodes. smaller in size (~350MB). the test should create pods with the command . I'm fine with either one, although I think people will tend towards the nanoserver image, and in fairness, I think it will be the easiest one to maintain long-term. AFAIK, Microsoft updates their container images periodically, so, we should do the same for the image. If not, the test using the image will then have to pull the entire image (350 MB is ok, IMO). E.g. vs Well, a manifest list can contain both Linux and Windows images, So, if the busybox image is picked, the Linux busybox image can be included in the manifest as well. Ideally, most images work on both OSes (with some exceptions, like the nanoserver or alpine images), so I don't think there's a need to put windows in the path. And if a new root path is being for new test images, I would like to ask that images mustn't container slashes in their names (e.g.:). Docker doesn't support images with slashes in their names (see ). Even if all the images on eventually support Windows, it would still be a good idea to be able to use dockerhub for testing purposes.
We need an image - preferably not one as prone to CVEs as busybox WHOAH. That's a 2GB image. For testing. I am in favor of this just based on the size (I was thinking something like kuberneets' pause, which is 240 kB. Over 1000 times smaller. We have different comprehensions of what "nano" means. 350 MB is not OK. It's acceptable if we have no other choice, but honestly we should create our own that is as small as humanly possible. Then, we need a path. First, we'll be setting precedent for test images, so I need input on that. I propose as the root. Objections from test folks? I don't have enough context on e2e future, so I'd like some input from someone who does. Second, do we want to squirrel this away as a Windwows-specific binary, or should we not name it in any particular way, so that we can pull the same image for all tests via a multi-os, multi-arch manifest list (which can be done later). E.g. vs I agree (except for busybox, which is a known vector for contaigion). Having a multi-os, multi-arch "serve_hostname" or something equally dumb seems ideal. Docker HUB doesn't. Docker itself does, and GCR does. If you're using dockerhub for testing, you have to change "" to "e2eteam/whatever", right? So what does it matter if the GCR path is longer? That said, I would be OK to have a single dir if that is what test-team wants. It just seems likely to get messy eventually.
Yes, dockerhub, I forgot the "hub" :) And yes, that's what we're doing and it works for the most part. But, if you take a look here [1], basically, the root is , which, as you know, is not accepted by dockerhub. So, the proposal was to not have "image names" with slashes in them (e.g.: in this case). [1]
I see. I think it would be OK to ask that images don't use subdirs, but the e2eRegistry variable could. e.g. // Docker hub e2eRegistry = "e2eteam" // GCR e2eRegistry = "" ResourceController = Config{e2eRegistry, "resource-consumer-controller", "1.0"} That seems workable, though it doesn't answer the question of the actual path in GCR :) Claudiu Belu <:
That seems workable, though it doesn't answer the question of the Yep, exactly right. :)
what are the next steps on this ticket to get the images to
I am still waiting for which image we are going to use, and for any objections to "" as a root. Michael Michael <:
Well, I think we are in agreement that we should use the nanoserver images. As for the root repo, I have no objections.
can you please explicitly give all the images to upload in an authoritative list? i am hoping we can upload all of these by monday EOD.
Sure. As we've discussed previously, the images are:
no manifest list image that supports both 1809 and 1803? are we OK for only having 1809 for that "pull from GCR" test?
Yeah, I meant those images should be included in a single manifest list (something like or something), like we've previously discussed. As for the second question, AFAIK, the GCE Windows CI is using Windows Server 1803 at the moment, so it would be nice to have the new test running on that too.
see above for the two images we want to push. we also need to repush this image resource-consumer/controller:1.0 to avoid the "/" as resource-consumer-controller cc:
i pulled these images on Windows: size of each is roughly ~350MB. and then saved them to TAR and uploaded them here: to go lower than that we might have hack custom images that trim, for instance a bunch of DLLs that we don't need (for e.g. a minimal pause image), however such a custom image will just require yet another unique download if the users are already basing their workload containers on official images. also for such hacks we need someone to verify if the nanoserver LICENSE allows that. i'm pretty sure it doesn't.
no hacks please. we need to upload the images as they officially come from microsoft. :)
/milestone v1.14 We're entering and this looks relevant to the 1.14 release. Please if I am incorrect
The slash thing is kinda irrelevant to me, it's just a limitation of Docker Hub as pointed out
i understand that. but can you make forward progress in testing that autoscale ticket without this work?
i think you have the two images to upload. let us know when this is complete and thank you so much!
The tars shouldn't be necessary. The nanoserver images include 2 layers that are type and should not be pushed. here's an example manifest from the bazel repo. you could probably use the existing rulesdocker to push nanoserver with no changes if you're building and pushing images with bazel
we can't follow this process due to Windows licensing. i will hide your post so that it is not followed by accident.
just to clarify - you can reference those in manifest lists, and docker push / pull them without actually pushing the layer contents to a manifest v2 complaint registry. That's the right way to do it. That separates out user-owned files in the upper layers from those in the base layers that are always served from a Microsoft server due to licensing restrictions
Oh ok then. Keeping in mind the fact that the nanoserver image is a manifest itself (for amd64 and arm) and that you cannot reference another manifest list in your manifest list, then the the actual images which should be to the manifest list are: 1803 doesn't have multiarch.
the post is still quoted in your response. best to edit that as well? ok, looks like the licensing of these image is slightly more complicated than expected... i hope a tutorial is not required when pushing to GCR. :)
What is the status of this now? I am THOROUGHLY confused
me too according to here and here and as far as i understood, we only have to push the JSON files that hold layer references, but not the actual layers because this violates the licensing of the nanoserver images.
I'll post explicit steps using , etc.
a PowerShell window as an administrator Docker-CE 18.09.2 experimental mode for the Docker CLI Here's the copy & paste steps to pull the existing images, push them to the new registry, and create a manifest. Change $destinationRepo to be something like or your private repo name. The pushes will be nearly instantaneous because they're manifest-only pushes. All layer data is skipped. Now, you can inspect the manifest and it will show multiple images with different os.version And last, push it
Here's the example repo I made on When things are done this way, the original Windows image is still served from Microsoft as required by the license. If I were to add unique layers on top of this image - that content would be pushed to the registry.
Pushed to my staging, will try to get the promoter running ASAP
hmm, it looks like our image promoter is not quite ready for multi-platform yet. Let me try pushing to
yeah, you might be bumping into
Someone test that pull?
does that mean no Windows images can live on GCR? Am I misreading it?
I believe GCR supports it, but some of our tooling (which uses google/containerregistry) may not. OTOH I think tool is just using directly, so it might be unaffected.
!
Awesome, so that seems to work for now. I guess the next step is to FROM that in your own containers.
or just use it in your tests. Do we close this for now, then?
SGTM, leaving the close call to
This looks right. I did a pull on a 1809 machine and the hashes match. is this enough to finish that test case, or are more images needed? We definitely need to get the images into a CI for 1.15. See /
this should be enough in terms of the test case.
^ FYI, there are now Windows test images at GCR. thanks !
Awesomeness, I've made a PR adding the testcase using this new image. This issue will close automatically once that merges. Test works for me. One question remains though. Do we have to have a Windows equivalent for the , so it fails on Windows. If we have to add a Windows equivalent for that too, then the nanoserver manifest list should also be pushed to the registry as well. Anyways. thanks for the help, everyone!
are you asking if can push the nanoserver manifest to the authenticated-image-pulling registry as well? Tim?
Yep. It is a conformance test, so it would be nice to be able to cover it as well.
any chance you push the nanoserver manifest to the authenticated-image-pulling registry as well? thanks for all your help! once you do the above, we can lose this issue
Try ""
thanks it works! !
please, close the issue once you check this.
thanks Tim! you rock!
can you please close this issue if we are done?
The issue is supposed to close when the PR merges. Closing anyways.

// Images used for ConformanceContainer are not added into NodeImageWhiteList, because this test is // testing image pulling, these images don't need to be prepulled. The ImagePullPolicy // is v1.PullAlways, so it won't be blocked by framework image white list check. <del> imagePullTest := func(image string, hasSecret bool, expectedPhase v1.PodPhase, expectedPullStatus bool) { </del> <ins> imagePullTest := func(image string, hasSecret bool, expectedPhase v1.PodPhase, expectedPullStatus bool, windowsImage bool) { command := []string{"/bin/sh", "-c", "while true; do sleep 1; done"} if windowsImage { // -t: Ping the specified host until stopped. command = []string{"ping", "-t", "localhost"} } </ins> container := ConformanceContainer{ PodClient: f.PodClient(), Container: v1.Container{ Name:            "image-pull-test", Image:           image, <del> Command:         []string{"/bin/sh", "-c", "while true; do sleep 1; done"}, </del> <ins> Command:         command, </ins> ImagePullPolicy: v1.PullAlways, }, RestartPolicy: v1.RestartPolicyNever,


Question:
<!-- Please only use this template for submitting enhancement requests --What would you like to be : /sig testing /sig windows In the the PR we're adding a test case for pulling a Windows image from dockerhub (. However, there are no known public Windows images on , so the only option would be to upload it ourselves. Our proposal is to have it uploaded to the using Windows images). The proposed image manifest list to be uploaded is built from [1], which is currently hosted on dockerhub [2] and is currently used for testing Kubernetes with Windows nodes [3]. The manifest list contains the images: e2eteam/busybox:1.29-1803 (for Windows Server 1803) e2eteam/busybox:1.29-1809 (for Windows Server 2019) [1] [2] [3] Why is this needed: The Windows images will be used for the Windows equivalent test case for the following test:
/assign
/area conformance
It is my contention that we really only want one prod repo and that should be folded into the main repo. The authenticated repo should be for testing authenticated pulls and nothing else. To that end, I propose that we start with this. Does the image need to be named or tagged or something to indicate OS? How about * ? Is that too deep? Does it need Windows in the path or should we put it in the tag? Or can we have a multi-os manifest list? Also, is busybox the right image for testing, or can we do something even more minimal (to address the "what f there's a busybox CVE" question)?
Yes, (all images to date have used ), so I'd prefer that route over yet another name or tag. I'd also prefer something other than busybox if possible. I originally created because there were so many images in and it was hard to tell which of them were production images, which were test images, and which had been completely abandoned. Also, only a small number of Googlers could push to this repo since it contained all the production images. To address the first issue, I decided that all e2e test images would be sourced from the directory. To address the second issue, I created a new project with a wider ACL (though still currently limited to Googlers). All that said, given the automation we're building for the main GCR repo, it'd probably be fine to fold these images back in.
The image promoter means that a single repo should be BETTER managed than individual repos - we will have a PR trail of who put images there and why.
We should probably broaden the discussion to non-busybox Windows test images too. (related: ). Do we have a lower bar for test images? If so, having a separate repo (or making it clear that the images are test only) does have clear benefits.
Do we want a lower bar? I don't think we do.
Nobody wants a lower bar until they want to add/update the image quickly for a new test, or fix a test failure to unblock the release. If we do effectively hold the same bar today, this shouldn't be a problem.
i agree. also we don't need schema v2. multi-os, multi-arch image here. just (e.g.) a nanoserver image that satisfies a Windows 1803 and Windows 1809 host. a nanoserver image will obviously imply Windows only. for a test called "should be able to pull image from [WindowsOnly]": all we need is to upload this image to [x] and i guess we need to decide on if we want this test to be / passing for Windows Containers GA.
And indeed, a manifest list with nanoserver images could be an alternative (considering that the ) (we did try to find a command which works on both OSes, and ultimately found curl, but not all Linux images contain curl, check PR ).
an official nanoserver image is a more secure download. do we need to build a nanoserver manifest list or are there official images that we can use? i would like to bring a topic here related to pause. why are all the pause images for Windows that i found 1.1 GB or more? we definitely need the absolute minimal pause container uploaded in a location where users can download it and use that location in the guide. AFAIK, 100 MB is as low as we can go for a native Windows container, but that's still much better... i brought this topic in the other PR. we don't need a command that works on both OSes. if is the command to use on Windows containers only, so be it.
And also, my image? It might be out-of-date.
we need to start planning how Windows containers can used be easily on Windows nodes. the pause image should be pre-built, small and easy to pull by deployment tools, such as kops, kubeadm. that is much better, but still a bit too much? can we go down to ~100MB? this image size was reported by a user in a k/k issue recently: my assumption is that the user was building the image himself.
i am ok to have that conversation, but please file a ticket and let's discuss that in sig-windows
everyone, thank you for your help and responses. trying to bring back this conversation to the original request. what would you like us to do in order for the few folks from google that have permissions to upload the windows images to be used for testing.
I am happy to help publish this via the alias, which I believe is the correct long-term path. We need an image - preferably not one as prone to CVEs as busybox historically is. Hence a suggestion for "pause" or something equivalently useless and tiny (if it is tiny). Then, we need a path. First, we'll be setting precedent for test images, so I need input on that. I propose as the root. Objections from test folks? Second, do we want to squirrel this away as a Windwows-specific binary, or should we not name it in any particular way, so that we can pull the same image for all tests via a multi-os, multi-arch manifest list (which can be done later). E.g. vs How about * ? Is that too deep? Does it need Windows in the path or should we put it in the tag? Michael Michael <:
So, I'll sumarize the previous discussions. The proposed images are: The image currently named image tags:. based on: respectively. image is currently used in Windows E2E tests and should already exist on the nodes, so download should be minimal. The image. a manifest list needs to be created containing the tags: The image is used to tag the image, which is used by kubelet, so it should already exist on the nodes. smaller in size (~350MB). the test should create pods with the command . I'm fine with either one, although I think people will tend towards the nanoserver image, and in fairness, I think it will be the easiest one to maintain long-term. AFAIK, Microsoft updates their container images periodically, so, we should do the same for the image. If not, the test using the image will then have to pull the entire image (350 MB is ok, IMO). E.g. vs Well, a manifest list can contain both Linux and Windows images, So, if the busybox image is picked, the Linux busybox image can be included in the manifest as well. Ideally, most images work on both OSes (with some exceptions, like the nanoserver or alpine images), so I don't think there's a need to put windows in the path. And if a new root path is being for new test images, I would like to ask that images mustn't container slashes in their names (e.g.:). Docker doesn't support images with slashes in their names (see ). Even if all the images on eventually support Windows, it would still be a good idea to be able to use dockerhub for testing purposes.
We need an image - preferably not one as prone to CVEs as busybox WHOAH. That's a 2GB image. For testing. I am in favor of this just based on the size (I was thinking something like kuberneets' pause, which is 240 kB. Over 1000 times smaller. We have different comprehensions of what "nano" means. 350 MB is not OK. It's acceptable if we have no other choice, but honestly we should create our own that is as small as humanly possible. Then, we need a path. First, we'll be setting precedent for test images, so I need input on that. I propose as the root. Objections from test folks? I don't have enough context on e2e future, so I'd like some input from someone who does. Second, do we want to squirrel this away as a Windwows-specific binary, or should we not name it in any particular way, so that we can pull the same image for all tests via a multi-os, multi-arch manifest list (which can be done later). E.g. vs I agree (except for busybox, which is a known vector for contaigion). Having a multi-os, multi-arch "serve_hostname" or something equally dumb seems ideal. Docker HUB doesn't. Docker itself does, and GCR does. If you're using dockerhub for testing, you have to change "" to "e2eteam/whatever", right? So what does it matter if the GCR path is longer? That said, I would be OK to have a single dir if that is what test-team wants. It just seems likely to get messy eventually.
Yes, dockerhub, I forgot the "hub" :) And yes, that's what we're doing and it works for the most part. But, if you take a look here [1], basically, the root is , which, as you know, is not accepted by dockerhub. So, the proposal was to not have "image names" with slashes in them (e.g.: in this case). [1]
I see. I think it would be OK to ask that images don't use subdirs, but the e2eRegistry variable could. e.g. // Docker hub e2eRegistry = "e2eteam" // GCR e2eRegistry = "" ResourceController = Config{e2eRegistry, "resource-consumer-controller", "1.0"} That seems workable, though it doesn't answer the question of the actual path in GCR :) Claudiu Belu <:
That seems workable, though it doesn't answer the question of the Yep, exactly right. :)
what are the next steps on this ticket to get the images to
I am still waiting for which image we are going to use, and for any objections to "" as a root. Michael Michael <:
Well, I think we are in agreement that we should use the nanoserver images. As for the root repo, I have no objections.
can you please explicitly give all the images to upload in an authoritative list? i am hoping we can upload all of these by monday EOD.
Sure. As we've discussed previously, the images are:
no manifest list image that supports both 1809 and 1803? are we OK for only having 1809 for that "pull from GCR" test?
Yeah, I meant those images should be included in a single manifest list (something like or something), like we've previously discussed. As for the second question, AFAIK, the GCE Windows CI is using Windows Server 1803 at the moment, so it would be nice to have the new test running on that too.
see above for the two images we want to push. we also need to repush this image resource-consumer/controller:1.0 to avoid the "/" as resource-consumer-controller cc:
i pulled these images on Windows: size of each is roughly ~350MB. and then saved them to TAR and uploaded them here: to go lower than that we might have hack custom images that trim, for instance a bunch of DLLs that we don't need (for e.g. a minimal pause image), however such a custom image will just require yet another unique download if the users are already basing their workload containers on official images. also for such hacks we need someone to verify if the nanoserver LICENSE allows that. i'm pretty sure it doesn't.
no hacks please. we need to upload the images as they officially come from microsoft. :)
/milestone v1.14 We're entering and this looks relevant to the 1.14 release. Please if I am incorrect
The slash thing is kinda irrelevant to me, it's just a limitation of Docker Hub as pointed out
i understand that. but can you make forward progress in testing that autoscale ticket without this work?
i think you have the two images to upload. let us know when this is complete and thank you so much!
The tars shouldn't be necessary. The nanoserver images include 2 layers that are type and should not be pushed. here's an example manifest from the bazel repo. you could probably use the existing rulesdocker to push nanoserver with no changes if you're building and pushing images with bazel
we can't follow this process due to Windows licensing. i will hide your post so that it is not followed by accident.
just to clarify - you can reference those in manifest lists, and docker push / pull them without actually pushing the layer contents to a manifest v2 complaint registry. That's the right way to do it. That separates out user-owned files in the upper layers from those in the base layers that are always served from a Microsoft server due to licensing restrictions
Oh ok then. Keeping in mind the fact that the nanoserver image is a manifest itself (for amd64 and arm) and that you cannot reference another manifest list in your manifest list, then the the actual images which should be to the manifest list are: 1803 doesn't have multiarch.
the post is still quoted in your response. best to edit that as well? ok, looks like the licensing of these image is slightly more complicated than expected... i hope a tutorial is not required when pushing to GCR. :)
What is the status of this now? I am THOROUGHLY confused
me too according to here and here and as far as i understood, we only have to push the JSON files that hold layer references, but not the actual layers because this violates the licensing of the nanoserver images.
I'll post explicit steps using , etc.
a PowerShell window as an administrator Docker-CE 18.09.2 experimental mode for the Docker CLI Here's the copy & paste steps to pull the existing images, push them to the new registry, and create a manifest. Change $destinationRepo to be something like or your private repo name. The pushes will be nearly instantaneous because they're manifest-only pushes. All layer data is skipped. Now, you can inspect the manifest and it will show multiple images with different os.version And last, push it
Here's the example repo I made on When things are done this way, the original Windows image is still served from Microsoft as required by the license. If I were to add unique layers on top of this image - that content would be pushed to the registry.
Pushed to my staging, will try to get the promoter running ASAP
hmm, it looks like our image promoter is not quite ready for multi-platform yet. Let me try pushing to
yeah, you might be bumping into
Someone test that pull?
does that mean no Windows images can live on GCR? Am I misreading it?
I believe GCR supports it, but some of our tooling (which uses google/containerregistry) may not. OTOH I think tool is just using directly, so it might be unaffected.
!
Awesome, so that seems to work for now. I guess the next step is to FROM that in your own containers.
or just use it in your tests. Do we close this for now, then?
SGTM, leaving the close call to
This looks right. I did a pull on a 1809 machine and the hashes match. is this enough to finish that test case, or are more images needed? We definitely need to get the images into a CI for 1.15. See /
this should be enough in terms of the test case.
^ FYI, there are now Windows test images at GCR. thanks !
Awesomeness, I've made a PR adding the testcase using this new image. This issue will close automatically once that merges. Test works for me. One question remains though. Do we have to have a Windows equivalent for the , so it fails on Windows. If we have to add a Windows equivalent for that too, then the nanoserver manifest list should also be pushed to the registry as well. Anyways. thanks for the help, everyone!
are you asking if can push the nanoserver manifest to the authenticated-image-pulling registry as well? Tim?
Yep. It is a conformance test, so it would be nice to be able to cover it as well.
any chance you push the nanoserver manifest to the authenticated-image-pulling registry as well? thanks for all your help! once you do the above, we can lose this issue
Try ""
thanks it works! !
please, close the issue once you check this.
thanks Tim! you rock!
can you please close this issue if we are done?
The issue is supposed to close when the PR merges. Closing anyways.

It("should not be able to pull image from invalid registry [NodeConformance]", func() { image := "invalid.com/invalid/alpine:3.1" <del> imagePullTest(image, false, v1.PodPending, true) </del> <ins> imagePullTest(image, false, v1.PodPending, true, false) </ins> }) It("should not be able to pull non-existing image from gcr.io [NodeConformance]", func() { image := "k8s.gcr.io/invalid-image:invalid-tag" <del> imagePullTest(image, false, v1.PodPending, true) </del> <ins> imagePullTest(image, false, v1.PodPending, true, false) </ins> }) <del> // TODO(claudiub): Add a Windows equivalent test. </del> It("should be able to pull image from gcr.io [LinuxOnly] [NodeConformance]", func() { image := "gcr.io/google-containers/debian-base:0.4.1" <del> imagePullTest(image, false, v1.PodRunning, false) </del> <ins> imagePullTest(image, false, v1.PodRunning, false, false) }) It("should be able to pull image from gcr.io [NodeConformance]", func() { framework.SkipUnlessNodeOSDistroIs("windows") image := "gcr.io/kubernetes-e2e-test-images/windows-nanoserver:v1" imagePullTest(image, false, v1.PodRunning, false, true) </ins> }) It("should be able to pull image from docker hub [LinuxOnly] [NodeConformance]", func() { image := "alpine:3.7" <del> imagePullTest(image, false, v1.PodRunning, false) </del> <ins> imagePullTest(image, false, v1.PodRunning, false, false) </ins> }) <del> It("should be able to pull image from docker hub [WindowsOnly] [NodeConformance]", func() { </del> <ins> It("should be able to pull image from docker hub [NodeConformance]", func() { </ins> framework.SkipUnlessNodeOSDistroIs("windows") <ins> // TODO(claudiub): Switch to nanoserver image manifest list. </ins> image := "e2eteam/busybox:1.29" <del> imagePullTest(image, false, v1.PodRunning, false) </del> <ins> imagePullTest(image, false, v1.PodRunning, false, true) </ins> }) It("should not be able to pull from private registry without secret [NodeConformance]", func() { image := "gcr.io/authenticated-image-pulling/alpine:3.7" <del> imagePullTest(image, false, v1.PodPending, true) </del> <ins> imagePullTest(image, false, v1.PodPending, true, false) </ins> }) It("should be able to pull from private registry with secret [LinuxOnly] [NodeConformance]", func() { image := "gcr.io/authenticated-image-pulling/alpine:3.7" <del> imagePullTest(image, true, v1.PodRunning, false) </del> <ins> imagePullTest(image, true, v1.PodRunning, false, false) }) It("should be able to pull from private registry with secret [NodeConformance]", func() { framework.SkipUnlessNodeOSDistroIs("windows") image := "gcr.io/authenticated-image-pulling/windows-nanoserver:v1" imagePullTest(image, true, v1.PodRunning, false, true) </ins> }) }) })


Question:
<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: ci-kubernetes-e2e-gci-gce-serial Which test(s) are failing: LocalVolumeType: gce-localssd-scsi-fs Since when has it been failing: 2/20 Testgrid link: Reason for failure: unsure yet /assign
After , there are still two failing tests: For the first one, looks like reconstruction is not correctly implemented for local PVs: to check if it's ok for mounter to be initialized with an empty path the panic in the operation_generator
cc Unsure why it's failing only for this local pv type (real disk vs loopback block) I have not investigated the 2nd test yet
/priority critical-urgent /milestone v1.14 We're entering and this looks relevant to the 1.14 release. Please if I am incorrect
/assign
hi, checked , after , "should be mountable" has been fixed. failing test is "should unmount if pod is force deleted while kubelet is down [Disruptive][Slow]" and it fails for all kinds of local volume types
This is because local volume cannot be reconstructed for now (localPlugin.NewMounter requires field is set), so is going to . But uses to , and , so it panics here. I'm not sure how to support reconstruction for local volumes correctly yet. But we can fix this panic first, then reconciler will clean up the existing mount points if the volume is no longer needed. IIUC, this will fix this failing test. After this is fixed, if necessary we can revisit how to reconstruct local volumes correctly.
I see similar failures in and too; are they connected? cc
yes they should be similar
need to keep track of this to see if failures in and also clear up.
The tests are still throwing s in (it's the job there though). I've opened to track the failures in 1.14-blocking. They also seem to not have run since the 24th, I'll look into that in the meantime

podsDir string) (volumetypes.GeneratedOperations, error) { var pluginName string <del> if useCSIPlugin(og.volumePluginMgr, volumeToUnmount.VolumeSpec) { </del> <ins> if volumeToUnmount.VolumeSpec != nil && useCSIPlugin(og.volumePluginMgr, volumeToUnmount.VolumeSpec) { </ins> pluginName = csi.CSIPluginName } else { pluginName = volumeToUnmount.PluginName


Question:
In , we check mount on first level directory in pod volume directories. It is true for many volume plugins, but not for CSI plugin which . cc
/assign
Pardon my ignorance of kubelet source code, but this looks very much similar to an issue we have been running to many times recently, when developing a CSI plugin. The problem is that when CSI volume publish is called and the mount point is not propagated from the CSI plugin pod (either missing bidirectional flag in yaml or not supported by underlaying k8s cluster implementation), the kubelet nevertheless starts a pod using the volume. The volume is not there, so the app writes to root filesystem in fact, leading to all other kinds of problems - i.e. losing data, filling up root fs. IMHO what kubelet should do is to check that the mounted filesystem is indeed visible after publish call and if not it should fail the volume attach request. If this ticket was fixed, would it solve the issue we are seeing or should I open a new ticket? thanks
I think we can potentially add that check but we need to make sure first it can work with ephemeral CSI drivers. Ephemeral drivers would have to "stage" their directories somewhere, like /var/lib/kubelet/plugins/<csi driver/, and then bind mount it to the publish path. cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Hmm... That is a good question. Does csi require a mount to be made or only that the specified directory be created? Will have to go read through the spec...
/remove-lifecycle stale
The only place in the spec I could find was this comment: // For volumes with an access type of mount, the SP SHALL place the // mounted directory at target_path. So I think having Kubelet enforce that check would be ok with the spec, and a good thing to do.
cc CSI spec says that the target path has to be a mount. Shall we go ahead and enforce it then? Are we going to break drivers that may not be doing this?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten
The issue mentioned here involves two functions . Now with PR , the issue in CheckVolumeExistenceOperation is solved by checking mount path instead of volume path. The only issue here is for getMountedVolumePathListFromDisk which is used during pod deletion. This function was introduced by PR . Now I am questioning whether we need to check the mount point or just need to check pod dirs exist before deleting the pod. Firstly, for some plugins, there might not be mount point. Secondly, without checking mount point, it means that pod cannot be deleted as long as there are pod volume dirs. In normal case, volume dirs should be cleanedup during volume teardown. For cases that kubelet restarts, we have reconciler try to clean up volume mounts if possible. So we can propose a couple of solutions here mount point check. It solves the problem, the downside is that there might be some rare case that reconciler could not clean up dirs after kubelet restarts. check mount point, and also check the next level "/mount" dir. This might still cause problem for some plugins that do not have mounts. cc
Discussed offline with For proposal 1., pod will not be deleted from api server if there are still volume dirs exist. So to make sure it works well, we need to have reliable way to clean up dirs. Normally when kubelet restarts, reconciler will either add it into actual state (if reconstruction works) or try to clean it up directly once unless volume is still in desired state. So if reconstruction does not work, and it fails to clean up during reconciler sync, there is no other chance to clean up the dirs. Or if volume was in desired state and removed quickly afterwords, the dir might not get the chance to be cleaned up at all. We are thinking to improve this part, we could use mark volume "uncertain" in actual state feature. During reconstruction, we can remove the check for mounted points (since the mount might be gone during node reboots or driver does not have moutns at all). Reconciler could mark those volumes as uncertain (even if it was in desired state) to make sure those volume got cleaned up or remount whenever needed with retries. With this change, I think it is safe to remove the mount point check for pod deletion too.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale

"//vendor/k8s.io/utils/mount:go_default_library", "//vendor/k8s.io/utils/net:go_default_library", "//vendor/k8s.io/utils/path:go_default_library", <ins> "//vendor/k8s.io/utils/strings:go_default_library", </ins> ] + select({ "@io_bazel_rules_go//go/platform:windows": [ "//pkg/kubelet/winstats:go_default_library",


Question:
In , we check mount on first level directory in pod volume directories. It is true for many volume plugins, but not for CSI plugin which . cc
/assign
Pardon my ignorance of kubelet source code, but this looks very much similar to an issue we have been running to many times recently, when developing a CSI plugin. The problem is that when CSI volume publish is called and the mount point is not propagated from the CSI plugin pod (either missing bidirectional flag in yaml or not supported by underlaying k8s cluster implementation), the kubelet nevertheless starts a pod using the volume. The volume is not there, so the app writes to root filesystem in fact, leading to all other kinds of problems - i.e. losing data, filling up root fs. IMHO what kubelet should do is to check that the mounted filesystem is indeed visible after publish call and if not it should fail the volume attach request. If this ticket was fixed, would it solve the issue we are seeing or should I open a new ticket? thanks
I think we can potentially add that check but we need to make sure first it can work with ephemeral CSI drivers. Ephemeral drivers would have to "stage" their directories somewhere, like /var/lib/kubelet/plugins/<csi driver/, and then bind mount it to the publish path. cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Hmm... That is a good question. Does csi require a mount to be made or only that the specified directory be created? Will have to go read through the spec...
/remove-lifecycle stale
The only place in the spec I could find was this comment: // For volumes with an access type of mount, the SP SHALL place the // mounted directory at target_path. So I think having Kubelet enforce that check would be ok with the spec, and a good thing to do.
cc CSI spec says that the target path has to be a mount. Shall we go ahead and enforce it then? Are we going to break drivers that may not be doing this?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten
The issue mentioned here involves two functions . Now with PR , the issue in CheckVolumeExistenceOperation is solved by checking mount path instead of volume path. The only issue here is for getMountedVolumePathListFromDisk which is used during pod deletion. This function was introduced by PR . Now I am questioning whether we need to check the mount point or just need to check pod dirs exist before deleting the pod. Firstly, for some plugins, there might not be mount point. Secondly, without checking mount point, it means that pod cannot be deleted as long as there are pod volume dirs. In normal case, volume dirs should be cleanedup during volume teardown. For cases that kubelet restarts, we have reconciler try to clean up volume mounts if possible. So we can propose a couple of solutions here mount point check. It solves the problem, the downside is that there might be some rare case that reconciler could not clean up dirs after kubelet restarts. check mount point, and also check the next level "/mount" dir. This might still cause problem for some plugins that do not have mounts. cc
Discussed offline with For proposal 1., pod will not be deleted from api server if there are still volume dirs exist. So to make sure it works well, we need to have reliable way to clean up dirs. Normally when kubelet restarts, reconciler will either add it into actual state (if reconstruction works) or try to clean it up directly once unless volume is still in desired state. So if reconstruction does not work, and it fails to clean up during reconciler sync, there is no other chance to clean up the dirs. Or if volume was in desired state and removed quickly afterwords, the dir might not get the chance to be cleaned up at all. We are thinking to improve this part, we could use mark volume "uncertain" in actual state feature. During reconstruction, we can remove the check for mounted points (since the mount might be gone during node reboots or driver does not have moutns at all). Reconciler could mark those volumes as uncertain (even if it was in desired state) to make sure those volume got cleaned up or remount whenever needed with retries. With this change, I think it is safe to remove the mount point check for pod deletion too.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale

"k8s.io/klog" "k8s.io/utils/mount" utilpath "k8s.io/utils/path" <ins> utilstrings "k8s.io/utils/strings" </ins> v1 "k8s.io/api/core/v1" "k8s.io/apimachinery/pkg/types"


Question:
In , we check mount on first level directory in pod volume directories. It is true for many volume plugins, but not for CSI plugin which . cc
/assign
Pardon my ignorance of kubelet source code, but this looks very much similar to an issue we have been running to many times recently, when developing a CSI plugin. The problem is that when CSI volume publish is called and the mount point is not propagated from the CSI plugin pod (either missing bidirectional flag in yaml or not supported by underlaying k8s cluster implementation), the kubelet nevertheless starts a pod using the volume. The volume is not there, so the app writes to root filesystem in fact, leading to all other kinds of problems - i.e. losing data, filling up root fs. IMHO what kubelet should do is to check that the mounted filesystem is indeed visible after publish call and if not it should fail the volume attach request. If this ticket was fixed, would it solve the issue we are seeing or should I open a new ticket? thanks
I think we can potentially add that check but we need to make sure first it can work with ephemeral CSI drivers. Ephemeral drivers would have to "stage" their directories somewhere, like /var/lib/kubelet/plugins/<csi driver/, and then bind mount it to the publish path. cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Hmm... That is a good question. Does csi require a mount to be made or only that the specified directory be created? Will have to go read through the spec...
/remove-lifecycle stale
The only place in the spec I could find was this comment: // For volumes with an access type of mount, the SP SHALL place the // mounted directory at target_path. So I think having Kubelet enforce that check would be ok with the spec, and a good thing to do.
cc CSI spec says that the target path has to be a mount. Shall we go ahead and enforce it then? Are we going to break drivers that may not be doing this?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten
The issue mentioned here involves two functions . Now with PR , the issue in CheckVolumeExistenceOperation is solved by checking mount path instead of volume path. The only issue here is for getMountedVolumePathListFromDisk which is used during pod deletion. This function was introduced by PR . Now I am questioning whether we need to check the mount point or just need to check pod dirs exist before deleting the pod. Firstly, for some plugins, there might not be mount point. Secondly, without checking mount point, it means that pod cannot be deleted as long as there are pod volume dirs. In normal case, volume dirs should be cleanedup during volume teardown. For cases that kubelet restarts, we have reconciler try to clean up volume mounts if possible. So we can propose a couple of solutions here mount point check. It solves the problem, the downside is that there might be some rare case that reconciler could not clean up dirs after kubelet restarts. check mount point, and also check the next level "/mount" dir. This might still cause problem for some plugins that do not have mounts. cc
Discussed offline with For proposal 1., pod will not be deleted from api server if there are still volume dirs exist. So to make sure it works well, we need to have reliable way to clean up dirs. Normally when kubelet restarts, reconciler will either add it into actual state (if reconstruction works) or try to clean it up directly once unless volume is still in desired state. So if reconstruction does not work, and it fails to clean up during reconciler sync, there is no other chance to clean up the dirs. Or if volume was in desired state and removed quickly afterwords, the dir might not get the chance to be cleaned up at all. We are thinking to improve this part, we could use mark volume "uncertain" in actual state feature. During reconstruction, we can remove the check for mounted points (since the mount might be gone during node reboots or driver does not have moutns at all). Reconciler could mark those volumes as uncertain (even if it was in desired state) to make sure those volume got cleaned up or remount whenever needed with retries. With this change, I think it is safe to remove the mount point check for pod deletion too.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale

kubecontainer "k8s.io/kubernetes/pkg/kubelet/container" kubelettypes "k8s.io/kubernetes/pkg/kubelet/types" utilnode "k8s.io/kubernetes/pkg/util/node" <ins> "k8s.io/kubernetes/pkg/volume/csi" </ins> ) // getRootDir returns the full path to the directory under which kubelet can