Question: as.Nil(err) runContainerOpts3 := testManager.GetDeviceRunContainerOptions(newPod, &newPod.Spec.Containers[0]) as.Equal(1, len(runContainerOpts3.Envs)) <ins> // Requesting to create a pod that requests resourceName1 in init containers and normal containers // should succeed with devices allocated to init containers reallocated to normal containers. podWithPluginResourcesInInitContainers := &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ UID: uuid.NewUUID(), }, Spec: v1.PodSpec{ InitContainers: []v1.Container{ { Name: string(uuid.NewUUID()), Resources: v1.ResourceRequirements{ Limits: v1.ResourceList{ v1.ResourceName(resourceName1): resourceQuantity2, }, }, }, { Name: string(uuid.NewUUID()), Resources: v1.ResourceRequirements{ Limits: v1.ResourceList{ v1.ResourceName(resourceName1): resourceQuantity1, }, }, }, }, Containers: []v1.Container{ { Name: string(uuid.NewUUID()), Resources: v1.ResourceRequirements{ Limits: v1.ResourceList{ v1.ResourceName(resourceName1): resourceQuantity2, v1.ResourceName(resourceName2): resourceQuantity2, }, }, }, { Name: string(uuid.NewUUID()), Resources: v1.ResourceRequirements{ Limits: v1.ResourceList{ v1.ResourceName(resourceName1): resourceQuantity2, v1.ResourceName(resourceName2): resourceQuantity2, }, }, }, }, }, } podsStub.updateActivePods([]*v1.Pod{podWithPluginResourcesInInitContainers}) err = testManager.Allocate(nodeInfo, &lifecycle.PodAdmitAttributes{Pod: podWithPluginResourcesInInitContainers}) as.Nil(err) podUID := string(podWithPluginResourcesInInitContainers.UID) initCont1 := podWithPluginResourcesInInitContainers.Spec.InitContainers[0].Name initCont2 := podWithPluginResourcesInInitContainers.Spec.InitContainers[1].Name normalCont1 := podWithPluginResourcesInInitContainers.Spec.Containers[0].Name normalCont2 := podWithPluginResourcesInInitContainers.Spec.Containers[1].Name initCont1Devices := testManager.podDevices.containerDevices(podUID, initCont1, resourceName1) initCont2Devices := testManager.podDevices.containerDevices(podUID, initCont2, resourceName1) normalCont1Devices := testManager.podDevices.containerDevices(podUID, normalCont1, resourceName1) normalCont2Devices := testManager.podDevices.containerDevices(podUID, normalCont2, resourceName1) as.True(initCont2Devices.IsSuperset(initCont1Devices)) as.True(initCont2Devices.IsSuperset(normalCont1Devices)) as.True(initCont2Devices.IsSuperset(normalCont2Devices)) as.Equal(0, normalCont1Devices.Intersection(normalCont2Devices).Len()) </ins> } func TestSanitizeNodeAllocatable(t *testing.T) {

do you think if you have time to implement the solution you mentioned in the above comment We can try to get it into a 1.9 sub release. I think for now, lets just support device reuse among initContainers and normal containers. We can revisit this part when we need to guarantee an Allocate call per container.
Right what I am mentioning here is that it is not possible to do that. The point of the device plugin is so that we can execute vendor specific code before making the device available in the container (e.g: Scrub the GPU memory). I don't think the proposed solution addresses this problem.
Sure. But to clarify, I think if we dont't expect big shock on current implementation, we could only provide limited support on the point, that is, without re-generate runtime information for regular containers, only those whose requested resource is equal to or is a subset of that in init containers could make a resuse(e.g. if we have two init containers, each requests 2 devices, then a regular container that request 3 devices still could not reuse the devices.) Besides, if a initcontainer requests 2 devices, and we have two regular containers, each requests 1 device, then due to the lack of runtime info re-generation, the devices could only be reused by one of the containers, not both of them. In sum, since we don't re-generate runtime information, we could only reuse devices per container, not per device, like we do in scheduler. Is that what you mean?
Hmm, that is a good question. I forgot that we now return the combined runtime information in AllocResponse. Then for now, perhaps we should just return an error in device plugin manager Allocate() in case both init containers and containers request the same device plugin resource?
You mean we give up reuse and prohibit users to request same kind of device in init containers and regular containers? that seems a bit weird. And if we decide to do so, we might need to do the check at the first beginning, i.e. in API validation, not wait until it goes into DevicePlugin.
I think for 1.9, we just return an error when both init containers and normal containers of a pod request the same device resource, and document this in the device plugin user guide.


Question: as.Nil(err) runContainerOpts3 := testManager.GetDeviceRunContainerOptions(newPod, &newPod.Spec.Containers[0]) as.Equal(1, len(runContainerOpts3.Envs)) <ins> // Requesting to create a pod that requests resourceName1 in init containers and normal containers // should succeed with devices allocated to init containers reallocated to normal containers. podWithPluginResourcesInInitContainers := &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ UID: uuid.NewUUID(), }, Spec: v1.PodSpec{ InitContainers: []v1.Container{ { Name: string(uuid.NewUUID()), Resources: v1.ResourceRequirements{ Limits: v1.ResourceList{ v1.ResourceName(resourceName1): resourceQuantity2, }, }, }, { Name: string(uuid.NewUUID()), Resources: v1.ResourceRequirements{ Limits: v1.ResourceList{ v1.ResourceName(resourceName1): resourceQuantity1, }, }, }, }, Containers: []v1.Container{ { Name: string(uuid.NewUUID()), Resources: v1.ResourceRequirements{ Limits: v1.ResourceList{ v1.ResourceName(resourceName1): resourceQuantity2, v1.ResourceName(resourceName2): resourceQuantity2, }, }, }, { Name: string(uuid.NewUUID()), Resources: v1.ResourceRequirements{ Limits: v1.ResourceList{ v1.ResourceName(resourceName1): resourceQuantity2, v1.ResourceName(resourceName2): resourceQuantity2, }, }, }, }, }, } podsStub.updateActivePods([]*v1.Pod{podWithPluginResourcesInInitContainers}) err = testManager.Allocate(nodeInfo, &lifecycle.PodAdmitAttributes{Pod: podWithPluginResourcesInInitContainers}) as.Nil(err) podUID := string(podWithPluginResourcesInInitContainers.UID) initCont1 := podWithPluginResourcesInInitContainers.Spec.InitContainers[0].Name initCont2 := podWithPluginResourcesInInitContainers.Spec.InitContainers[1].Name normalCont1 := podWithPluginResourcesInInitContainers.Spec.Containers[0].Name normalCont2 := podWithPluginResourcesInInitContainers.Spec.Containers[1].Name initCont1Devices := testManager.podDevices.containerDevices(podUID, initCont1, resourceName1) initCont2Devices := testManager.podDevices.containerDevices(podUID, initCont2, resourceName1) normalCont1Devices := testManager.podDevices.containerDevices(podUID, normalCont1, resourceName1) normalCont2Devices := testManager.podDevices.containerDevices(podUID, normalCont2, resourceName1) as.True(initCont2Devices.IsSuperset(initCont1Devices)) as.True(initCont2Devices.IsSuperset(normalCont1Devices)) as.True(initCont2Devices.IsSuperset(normalCont2Devices)) as.Equal(0, normalCont1Devices.Intersection(normalCont2Devices).Len()) </ins> } func TestSanitizeNodeAllocatable(t *testing.T) {

The reason is because in the current device plugin implementation, we don't have a good way to free devices used by init containers of an active pod. It is hard to do this kind of validation at API level or even scheduler because we can't know whether an extended resource is device resource or not. So for now, I think we should probably just return a clear error message during admission handler. I am not sure yet whether there will be strong use cases that need to share devices between init containers and normal containers. We can revisit this based on user feedback. If there is strong user demand to support this, something we could do is to differentiate init containers and normal containers in podDevices, and free up devices used by init containers when any normal container is about to run. That would request some code structure change, and we may wait to see whether there is strong demand to support this.
It is hard to do this at API level or even scheduler because we can't know whether an extended resource is device resource or not. Oh, you are right, I ignored the resource awareness issue. I also agree that it's hard to address the issue without big change to current implementation. But I still feel the inconsistency of behavior between API and kubelet might confusing users. However, its effect should be sustainable, since we haven't received feedback about init containers up to now.
Right. So for now, perhaps just return an error when this happens, to give clear error signals.
Oh, I'm afraid we miss something, the check may not work as expected. As I commented, because init-containers run sequentially, we did reuse between init containers in , that is, if we have two init containers, each request 1 device, then we expect 1 device in total in scheduler, but expect 2 in deviceplugin. The check could not cover the scenario.
Thinking about this problem a bit more, I wonder whether we may tackle this issue from the following approaches, ordered by their complexities: 1) Just return an error whenever an init container requests a device plugin resource, and document in the device plugin user guide that requiring device plugin resource in init containers are not supported. This may be the short-term change we can try to get into 1.9. 2) Maintain a list of devices allocated to init containers in ManagerImpl.Allocate().


Question: as.Nil(err) runContainerOpts3 := testManager.GetDeviceRunContainerOptions(newPod, &newPod.Spec.Containers[0]) as.Equal(1, len(runContainerOpts3.Envs)) <ins> // Requesting to create a pod that requests resourceName1 in init containers and normal containers // should succeed with devices allocated to init containers reallocated to normal containers. podWithPluginResourcesInInitContainers := &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ UID: uuid.NewUUID(), }, Spec: v1.PodSpec{ InitContainers: []v1.Container{ { Name: string(uuid.NewUUID()), Resources: v1.ResourceRequirements{ Limits: v1.ResourceList{ v1.ResourceName(resourceName1): resourceQuantity2, }, }, }, { Name: string(uuid.NewUUID()), Resources: v1.ResourceRequirements{ Limits: v1.ResourceList{ v1.ResourceName(resourceName1): resourceQuantity1, }, }, }, }, Containers: []v1.Container{ { Name: string(uuid.NewUUID()), Resources: v1.ResourceRequirements{ Limits: v1.ResourceList{ v1.ResourceName(resourceName1): resourceQuantity2, v1.ResourceName(resourceName2): resourceQuantity2, }, }, }, { Name: string(uuid.NewUUID()), Resources: v1.ResourceRequirements{ Limits: v1.ResourceList{ v1.ResourceName(resourceName1): resourceQuantity2, v1.ResourceName(resourceName2): resourceQuantity2, }, }, }, }, }, } podsStub.updateActivePods([]*v1.Pod{podWithPluginResourcesInInitContainers}) err = testManager.Allocate(nodeInfo, &lifecycle.PodAdmitAttributes{Pod: podWithPluginResourcesInInitContainers}) as.Nil(err) podUID := string(podWithPluginResourcesInInitContainers.UID) initCont1 := podWithPluginResourcesInInitContainers.Spec.InitContainers[0].Name initCont2 := podWithPluginResourcesInInitContainers.Spec.InitContainers[1].Name normalCont1 := podWithPluginResourcesInInitContainers.Spec.Containers[0].Name normalCont2 := podWithPluginResourcesInInitContainers.Spec.Containers[1].Name initCont1Devices := testManager.podDevices.containerDevices(podUID, initCont1, resourceName1) initCont2Devices := testManager.podDevices.containerDevices(podUID, initCont2, resourceName1) normalCont1Devices := testManager.podDevices.containerDevices(podUID, normalCont1, resourceName1) normalCont2Devices := testManager.podDevices.containerDevices(podUID, normalCont2, resourceName1) as.True(initCont2Devices.IsSuperset(initCont1Devices)) as.True(initCont2Devices.IsSuperset(normalCont1Devices)) as.True(initCont2Devices.IsSuperset(normalCont2Devices)) as.Equal(0, normalCont1Devices.Intersection(normalCont2Devices).Len()) </ins> } func TestSanitizeNodeAllocatable(t *testing.T) {

Try to allocate from this list first for later init containers and normal containers. The required no. of devices we pass to ManagerImpl.devicesToAllocate() should subtract the no. of devices we have allocated to init containers. This may require some code structure change around ManagerImpl.Allocate(), ManagerImpl.allocateContainerResources(), and ManagerImpl.devicesToAllocate(). However, Allocate gRPC call is still issued during Admission all together, and it is possible that we issue an Allocate gRPC for [dev1] (say dev1 allocated to initContianer1), followed by another Allocate gRPC call for [dev1, dev2] (say dev1 and dev2 allocated to initContainer2), with none of containers started or finished. This assumes containers running in the same pod share the same security context, and device plugin doesn't need to perform per-container allocation operations. We just use Allocate gRPC call to learn container runtime settings. 3) Take a further step from 2), during Admission, we just determine what devices to allocate to containers without issuing the actual Allocate gRPC calls. I.e., we only fill deviceIds field in podDevices.deviceAllocateInfo but leaves allocResp field empty. During ManagerImpl.GetDeviceRunContainerOptions(), we check for this case and issue the actual Allocate gRPC call for the container. This ensures we can issue a gRPC Allocate call per container. However, if a pod is restarted multiple times, we don't guarantee Allocate gRPC call will be issued during subsequent pod runs. 4) Take a further step from 3), we can check for overlapping device allocations in ManagerImpl.GetDeviceRunContainerOptions() and invalidate their allocResp so that an Allocate gRPC call will be issued next time for containers using re-allocated devices. E.g., for containerA, if we find it is allocated with [dev1, dev2] and dev1 is also allocated to containerB, we set containerB's allocResp to nil so that next time before containerB is running, we will issue an Allocate gRPC call for dev1. Note that this approach does have the drawback that device plugin failure would cause previously running pods not be able to restart, so we need to carefully evaluate our use cases to see whether we need to implement this. My suggestion is to implement 1) now and gather some user feedbacks to determine whether we need to implement the following approaches.


Question: as.Nil(err) runContainerOpts3 := testManager.GetDeviceRunContainerOptions(newPod, &newPod.Spec.Containers[0]) as.Equal(1, len(runContainerOpts3.Envs)) <ins> // Requesting to create a pod that requests resourceName1 in init containers and normal containers // should succeed with devices allocated to init containers reallocated to normal containers. podWithPluginResourcesInInitContainers := &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ UID: uuid.NewUUID(), }, Spec: v1.PodSpec{ InitContainers: []v1.Container{ { Name: string(uuid.NewUUID()), Resources: v1.ResourceRequirements{ Limits: v1.ResourceList{ v1.ResourceName(resourceName1): resourceQuantity2, }, }, }, { Name: string(uuid.NewUUID()), Resources: v1.ResourceRequirements{ Limits: v1.ResourceList{ v1.ResourceName(resourceName1): resourceQuantity1, }, }, }, }, Containers: []v1.Container{ { Name: string(uuid.NewUUID()), Resources: v1.ResourceRequirements{ Limits: v1.ResourceList{ v1.ResourceName(resourceName1): resourceQuantity2, v1.ResourceName(resourceName2): resourceQuantity2, }, }, }, { Name: string(uuid.NewUUID()), Resources: v1.ResourceRequirements{ Limits: v1.ResourceList{ v1.ResourceName(resourceName1): resourceQuantity2, v1.ResourceName(resourceName2): resourceQuantity2, }, }, }, }, }, } podsStub.updateActivePods([]*v1.Pod{podWithPluginResourcesInInitContainers}) err = testManager.Allocate(nodeInfo, &lifecycle.PodAdmitAttributes{Pod: podWithPluginResourcesInInitContainers}) as.Nil(err) podUID := string(podWithPluginResourcesInInitContainers.UID) initCont1 := podWithPluginResourcesInInitContainers.Spec.InitContainers[0].Name initCont2 := podWithPluginResourcesInInitContainers.Spec.InitContainers[1].Name normalCont1 := podWithPluginResourcesInInitContainers.Spec.Containers[0].Name normalCont2 := podWithPluginResourcesInInitContainers.Spec.Containers[1].Name initCont1Devices := testManager.podDevices.containerDevices(podUID, initCont1, resourceName1) initCont2Devices := testManager.podDevices.containerDevices(podUID, initCont2, resourceName1) normalCont1Devices := testManager.podDevices.containerDevices(podUID, normalCont1, resourceName1) normalCont2Devices := testManager.podDevices.containerDevices(podUID, normalCont2, resourceName1) as.True(initCont2Devices.IsSuperset(initCont1Devices)) as.True(initCont2Devices.IsSuperset(normalCont1Devices)) as.True(initCont2Devices.IsSuperset(normalCont2Devices)) as.Equal(0, normalCont1Devices.Intersection(normalCont2Devices).Len()) </ins> } func TestSanitizeNodeAllocatable(t *testing.T) {

I see this a very big limitation. Special devices/resources in init containers is a very common use case IMO. Therefore we should try to fix this. This approach perfectly makes sense to me. What is your concern in implementing this? Even today, IIUC, Allocate call is made per container. Is there any other value that this approach is adding over 1) ? I am not able to see any gains with 2) or 3) over 1). Comments please.
I feel we may not be able to implement 2) for 1.9 timeframe. Option 2) also doesn't address the case mentioned above: Option 3) covers that case for pods without restarts. Option 4) fully covers the requirement we we will scrub old container data before giving it to a new container, even if they both run in the same pod, bug it does have the mentioned drawback during device plugin failure.
Take a further step from 3), we can check for overlapping device allocations in ManagerImpl.GetDeviceRunContainerOptions() and invalidate their allocResp so that an Allocate gRPC call will be issued next time for containers using re-allocated devices. E.g., for containerA, if we find it is allocated with [dev1, dev2] and dev1 is also allocated to containerB, we set containerB's allocResp to nil so that next time before containerB is running, we will issue an Allocate gRPC call for dev1. Note that this approach does have the drawback that device plugin failure would cause previously running pods not be able to restart, so we need to carefully evaluate our use cases to see whether we need to implement this. How about making the allocation call optional? That is , if we find a contaienr using re-allocated devices, we'd try to issue an Allocate gRPC call first, and still use the cached setting if allocation failed. And, if we decide not to address the issue in v1.9, which means we have enough time to work on it later, I think approach 2) might not be a good choice. Because it has a premise that devices are stateless, recycle actions before reuse(e.g: Scrub the GPU memory mentioned above) could not be supported. And this way, as I commented above, we could only reuse devices per container, not per device, like we do in scheduler.


Question: as.Nil(err) runContainerOpts3 := testManager.GetDeviceRunContainerOptions(newPod, &newPod.Spec.Containers[0]) as.Equal(1, len(runContainerOpts3.Envs)) <ins> // Requesting to create a pod that requests resourceName1 in init containers and normal containers // should succeed with devices allocated to init containers reallocated to normal containers. podWithPluginResourcesInInitContainers := &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ UID: uuid.NewUUID(), }, Spec: v1.PodSpec{ InitContainers: []v1.Container{ { Name: string(uuid.NewUUID()), Resources: v1.ResourceRequirements{ Limits: v1.ResourceList{ v1.ResourceName(resourceName1): resourceQuantity2, }, }, }, { Name: string(uuid.NewUUID()), Resources: v1.ResourceRequirements{ Limits: v1.ResourceList{ v1.ResourceName(resourceName1): resourceQuantity1, }, }, }, }, Containers: []v1.Container{ { Name: string(uuid.NewUUID()), Resources: v1.ResourceRequirements{ Limits: v1.ResourceList{ v1.ResourceName(resourceName1): resourceQuantity2, v1.ResourceName(resourceName2): resourceQuantity2, }, }, }, { Name: string(uuid.NewUUID()), Resources: v1.ResourceRequirements{ Limits: v1.ResourceList{ v1.ResourceName(resourceName1): resourceQuantity2, v1.ResourceName(resourceName2): resourceQuantity2, }, }, }, }, }, } podsStub.updateActivePods([]*v1.Pod{podWithPluginResourcesInInitContainers}) err = testManager.Allocate(nodeInfo, &lifecycle.PodAdmitAttributes{Pod: podWithPluginResourcesInInitContainers}) as.Nil(err) podUID := string(podWithPluginResourcesInInitContainers.UID) initCont1 := podWithPluginResourcesInInitContainers.Spec.InitContainers[0].Name initCont2 := podWithPluginResourcesInInitContainers.Spec.InitContainers[1].Name normalCont1 := podWithPluginResourcesInInitContainers.Spec.Containers[0].Name normalCont2 := podWithPluginResourcesInInitContainers.Spec.Containers[1].Name initCont1Devices := testManager.podDevices.containerDevices(podUID, initCont1, resourceName1) initCont2Devices := testManager.podDevices.containerDevices(podUID, initCont2, resourceName1) normalCont1Devices := testManager.podDevices.containerDevices(podUID, normalCont1, resourceName1) normalCont2Devices := testManager.podDevices.containerDevices(podUID, normalCont2, resourceName1) as.True(initCont2Devices.IsSuperset(initCont1Devices)) as.True(initCont2Devices.IsSuperset(normalCont1Devices)) as.True(initCont2Devices.IsSuperset(normalCont2Devices)) as.Equal(0, normalCont1Devices.Intersection(normalCont2Devices).Len()) </ins> } func TestSanitizeNodeAllocatable(t *testing.T) {

I noticed that you have already add the limitation into the , so I think it's already the decision that we add the limitation as a short-time solution in v1.9 So I posted a PR to add the check in code,, PTAL :)
Use Case: Let say there is a which node has two Devices of a resource type X and the device IDs are X1 and X2. User pod has one init container, lets call it cont-init and one regular container, lets call it cont-reg. Both cont-init and cont-reg requests 2 unit of resource X. cont-init will run some vendor specific code on allocated Devices and then cont-reg will use these 'initiated' devices. I think this is the use case that mentioned. And i feel 2) can handle this. 2) approach will allocate both the devices, X1 and X2, to both the containers. When the containers start, init container, cont-init, will be run first and will run to completion and here devices will get initialized. Strictly after this cont-reg will be started and now this regular container has initialized devices. Where i am wrong?
I guess in RenaudWasTaken 's case, the initialization task is considered to be managed in the plugin, not in init-container, and in init-container we only use the device, so then we might need to call plugin to do recycle action before the device reused by reguler containers. In practical use, both of the two scenarios may exist, it depends on the request of user. Besides, if we don't issue an Allocate gRPC call, we could not ensure a ”complete reuse“: say an init-container requests 2 devices, and we have two regular containers, each requests 1 device, then due to the lack of runtime info re-generation, the devices could only be reused by one of the containers, not both of them.
It is the init containers' job to initialize things and if in some special case device plugin is doing the initialization part, in that case, init containers will not be needed and serve no purpose in pod spec. Specific to example of memory scrubbing, i think this also makes more sense to be done in init container. Once Device's memory is scrubbed, it can be used by regular container. Let try to run your use case with approach 2).


Question: as.Nil(err) runContainerOpts3 := testManager.GetDeviceRunContainerOptions(newPod, &newPod.Spec.Containers[0]) as.Equal(1, len(runContainerOpts3.Envs)) <ins> // Requesting to create a pod that requests resourceName1 in init containers and normal containers // should succeed with devices allocated to init containers reallocated to normal containers. podWithPluginResourcesInInitContainers := &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ UID: uuid.NewUUID(), }, Spec: v1.PodSpec{ InitContainers: []v1.Container{ { Name: string(uuid.NewUUID()), Resources: v1.ResourceRequirements{ Limits: v1.ResourceList{ v1.ResourceName(resourceName1): resourceQuantity2, }, }, }, { Name: string(uuid.NewUUID()), Resources: v1.ResourceRequirements{ Limits: v1.ResourceList{ v1.ResourceName(resourceName1): resourceQuantity1, }, }, }, }, Containers: []v1.Container{ { Name: string(uuid.NewUUID()), Resources: v1.ResourceRequirements{ Limits: v1.ResourceList{ v1.ResourceName(resourceName1): resourceQuantity2, v1.ResourceName(resourceName2): resourceQuantity2, }, }, }, { Name: string(uuid.NewUUID()), Resources: v1.ResourceRequirements{ Limits: v1.ResourceList{ v1.ResourceName(resourceName1): resourceQuantity2, v1.ResourceName(resourceName2): resourceQuantity2, }, }, }, }, }, } podsStub.updateActivePods([]*v1.Pod{podWithPluginResourcesInInitContainers}) err = testManager.Allocate(nodeInfo, &lifecycle.PodAdmitAttributes{Pod: podWithPluginResourcesInInitContainers}) as.Nil(err) podUID := string(podWithPluginResourcesInInitContainers.UID) initCont1 := podWithPluginResourcesInInitContainers.Spec.InitContainers[0].Name initCont2 := podWithPluginResourcesInInitContainers.Spec.InitContainers[1].Name normalCont1 := podWithPluginResourcesInInitContainers.Spec.Containers[0].Name normalCont2 := podWithPluginResourcesInInitContainers.Spec.Containers[1].Name initCont1Devices := testManager.podDevices.containerDevices(podUID, initCont1, resourceName1) initCont2Devices := testManager.podDevices.containerDevices(podUID, initCont2, resourceName1) normalCont1Devices := testManager.podDevices.containerDevices(podUID, normalCont1, resourceName1) normalCont2Devices := testManager.podDevices.containerDevices(podUID, normalCont2, resourceName1) as.True(initCont2Devices.IsSuperset(initCont1Devices)) as.True(initCont2Devices.IsSuperset(normalCont1Devices)) as.True(initCont2Devices.IsSuperset(normalCont2Devices)) as.Equal(0, normalCont1Devices.Intersection(normalCont2Devices).Len()) </ins> } func TestSanitizeNodeAllocatable(t *testing.T) {

There are three containers in pod, cont-init(requested 2 unit or resource X), cont-reg1(requested 1 unit of resource X) and cont-reg2(requested 1 unit of resource X). Now in approach 2), gRPC Allocate call is made for cont-init requesting X1 and X2. Next, gRPC Allocate call is made for cont-reg1. This time resource id will be send in Allocate from the previously allocated [X1, X2]. Lets say X1 is requested for cont-reg1 Next, gRPC Allocate for cont-reg2 and resource with ID X2. All this allocation info gets stored in internal cache. Now when container runs, runtimeOptions will be collected from internal cache. Where do you see a problem in this flow?
/cc
Yep, it could work if the plugin behavior the way you described, but I'm not sure if all kinds of devices could support "pre-allocation", that is, doing allocation on a same device multiple times at the same time, to prepare runtime informations for several containers that reuse the device later. Also I have no idea if all plugin developers consider their plugins "stateless", that is, managing initialization in init-containers. As I don't have enough knowledge on hardware, would like to hear opinions from other guys :)
This would defeat the purpose of a container. In the case of GPUs, when you run a container you expect the GPU to always be in the same state. Furthermore GPUs can't be shared (exclusive usage) between containers meaning that Init containers should behave in the same way as regular containers. With the only difference being that they are executed sequentially and before regular containers. Additionally, operations to setup a device before making them available in a container, whether functionally (load the driver, insert libraries, ....) or security wise (scrub the memory), should be handled by the infrastructure not the users pod spec.
I agree that it's a sane behavior to forbid the use of Device Plugin Resources in Init Containers for the 1.9 release
moving forward,
/cc I think what long-term solution we decide to implement will largely depend on use-case requirements from different device plugins. Would like to hear some thoughts from different device plugin developers on this. Can people cc-ed in this issue take a look at the proposed options mentioned in and provide some feedbacks on which one may better fit your use cases?


Question: as.Nil(err) runContainerOpts3 := testManager.GetDeviceRunContainerOptions(newPod, &newPod.Spec.Containers[0]) as.Equal(1, len(runContainerOpts3.Envs)) <ins> // Requesting to create a pod that requests resourceName1 in init containers and normal containers // should succeed with devices allocated to init containers reallocated to normal containers. podWithPluginResourcesInInitContainers := &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ UID: uuid.NewUUID(), }, Spec: v1.PodSpec{ InitContainers: []v1.Container{ { Name: string(uuid.NewUUID()), Resources: v1.ResourceRequirements{ Limits: v1.ResourceList{ v1.ResourceName(resourceName1): resourceQuantity2, }, }, }, { Name: string(uuid.NewUUID()), Resources: v1.ResourceRequirements{ Limits: v1.ResourceList{ v1.ResourceName(resourceName1): resourceQuantity1, }, }, }, }, Containers: []v1.Container{ { Name: string(uuid.NewUUID()), Resources: v1.ResourceRequirements{ Limits: v1.ResourceList{ v1.ResourceName(resourceName1): resourceQuantity2, v1.ResourceName(resourceName2): resourceQuantity2, }, }, }, { Name: string(uuid.NewUUID()), Resources: v1.ResourceRequirements{ Limits: v1.ResourceList{ v1.ResourceName(resourceName1): resourceQuantity2, v1.ResourceName(resourceName2): resourceQuantity2, }, }, }, }, }, } podsStub.updateActivePods([]*v1.Pod{podWithPluginResourcesInInitContainers}) err = testManager.Allocate(nodeInfo, &lifecycle.PodAdmitAttributes{Pod: podWithPluginResourcesInInitContainers}) as.Nil(err) podUID := string(podWithPluginResourcesInInitContainers.UID) initCont1 := podWithPluginResourcesInInitContainers.Spec.InitContainers[0].Name initCont2 := podWithPluginResourcesInInitContainers.Spec.InitContainers[1].Name normalCont1 := podWithPluginResourcesInInitContainers.Spec.Containers[0].Name normalCont2 := podWithPluginResourcesInInitContainers.Spec.Containers[1].Name initCont1Devices := testManager.podDevices.containerDevices(podUID, initCont1, resourceName1) initCont2Devices := testManager.podDevices.containerDevices(podUID, initCont2, resourceName1) normalCont1Devices := testManager.podDevices.containerDevices(podUID, normalCont1, resourceName1) normalCont2Devices := testManager.podDevices.containerDevices(podUID, normalCont2, resourceName1) as.True(initCont2Devices.IsSuperset(initCont1Devices)) as.True(initCont2Devices.IsSuperset(normalCont1Devices)) as.True(initCont2Devices.IsSuperset(normalCont2Devices)) as.Equal(0, normalCont1Devices.Intersection(normalCont2Devices).Len()) </ins> } func TestSanitizeNodeAllocatable(t *testing.T) {

Answering on behalf of and Solution 1 makes sense for the moment Solution 2 defeats the purpose of the device plugin, which is setting up a device before making it available to a container. solution 4 seems to be a sane solution. As mentioned in my previous comment, as long as devices are exclusive, I expect the devices exposed in my container to always be in the same state at the start of my container. When / If we implement a mechanism for sharing a GPU / devices across containers then the scenario described by is something that we can have. It doesn't however fit the current situation.
Solution number 1 seems totally reasonable short-term. Long term, simplifying and documenting the relationship between plugin callbacks and container lifecycle would help plugin authors.
For reference, I implemented option 2 solution in PR The change itself is small. This is an alternative short-term fix we may consider to get into a 1.9 patch release. I think the main benefit of taking this fix compared to Option 1 solution is that we can change the underlying implementation to support init containers later with less user-facing impact.
No mater we take PR or PR for 1.9, I think we need to have this issue approved for 1.9 milestone. or or can any of you approve this issue for 1.9? Thanks!


Question: Namespace: config.ns, }, Data: map[string]string{ <del> config.scName: string(data), </del> <ins> "storageClassMap": string(data), </ins> }, } _, err = config.client.CoreV1().ConfigMaps(config.ns).Create(&configMap)

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: The local-volume bootstrapper/provisioner configuration format changed in kubernetes-incubator/external-storage, and its related e2e tests will fail when the provisioner & bootstrapper images are updated to v2.0.0. What you expected to happen: I'd expect the e2e tests to continue to work properly with the new version of the provisioner and boostrapper. How to reproduce it (as minimally and precisely as possible): Update local-volume provisioner and boostrapper images to v2.0.0, and then the provisioner e2e test will fail.
/sig storage
/assign


Question: // If it is, initialize all probed plugins and replace the cache with them. func (pm *VolumePluginMgr) refreshProbedPlugins() { events, err := pm.prober.Probe() if err != nil { klog.ErrorS(err, "Error dynamically probing plugins") <del> return // Use cached plugins upon failure. </del> } <ins> // because the probe function can return a list of valid plugins // even when an error is present we still must add the plugins // or they will be skipped because each event only fires once </ins> for _, event := range events { if event.Op == ProbeAddOrUpdate { if err := pm.initProbedPlugin(event.Plugin); err != nil {

When kubelet is running if you create two volume plugin directories here: /usr/libexec/kubernetes/kubelet-plugins/volume/exec /usr/libexec/kubernetes/kubelet-plugins/volume/exec/cifs /usr/libexec/kubernetes/kubelet-plugins/volume/exec/nodeagent~uds You will see these expected logs: Now if you copy over the cifs plugin file to cifs directory and then copy the uds plugin the errors go away. The pods using the uds volume will function but any pods attempting to use the cifs volume will fail with the following error from kubelet: desiredStateOfWorld. err=failed to get Plugin from volumeSpec for volume "pv" err=no volume plugin matched Restarting kubelet fixes this problem as then both plugins are present and properly . A functional volume plugin should be to kubelet even if another volume plugin is missing or invalid. Fixing the invalid volume plugin currently only adds the one that was invalid and fixed and other valid plugins that were previously present are skipped. Start kubelet service on a node. Create two directories for two flex volume plugins in the /usr/libexec/kubernetes/kubelet-plugins/volume/exec/ directory. Eg. nodeagent~uds and cifs Kubelet logs will now start complaining that the plugins are missing. Add the files to the plugin folders and then start a pod using the plugin. Notice that only the last plugin file copied is available to be used by pods. The first plugin will generate this error if used by a volume "err=no volume plugin matched" Performing a touch on the folder of the missing plugin will cause kubelet to properly index it. <details</details<detailsLocal kube cluster </details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig k8s-infra
/sig storage /remove-sig k8s-infra


Question: want:                          `issuer.claimMappings.extra[0].key: Invalid value: "example.org/Foo": key must be lowercase`, }, { <ins> name: "extra mapping key prefix is k8.io", in: api.ClaimMappings{ Username: api.PrefixedClaimOrExpression{Expression: "claims.username"}, Groups:   api.PrefixedClaimOrExpression{Expression: "claims.groups"}, Extra: []api.ExtraMapping{ {Key: "k8s.io/foo", ValueExpression: "claims.extra"}, }, }, structuredAuthnFeatureEnabled: true, want:                          `issuer.claimMappings.extra[0].key: Invalid value: "k8s.io/foo": k8s.io, kubernetes.io and their subdomains are reserved for Kubernetes use`, }, { name: "extra mapping key prefix contains k8.io", in: api.ClaimMappings{ Username: api.PrefixedClaimOrExpression{Expression: "claims.username"}, Groups:   api.PrefixedClaimOrExpression{Expression: "claims.groups"}, Extra: []api.ExtraMapping{ {Key: "example.k8s.io/foo", ValueExpression: "claims.extra"}, }, }, structuredAuthnFeatureEnabled: true, want:                          `issuer.claimMappings.extra[0].key: Invalid value: "example.k8s.io/foo": k8s.io, kubernetes.io and their subdomains are reserved for Kubernetes use`, }, { name: "extra mapping key prefix is kubernetes.io", in: api.ClaimMappings{ Username: api.PrefixedClaimOrExpression{Expression: "claims.username"}, Groups:   api.PrefixedClaimOrExpression{Expression: "claims.groups"}, Extra: []api.ExtraMapping{ {Key: "kubernetes.io/foo", ValueExpression: "claims.extra"}, }, }, structuredAuthnFeatureEnabled: true, want:                          `issuer.claimMappings.extra[0].key: Invalid value: "kubernetes.io/foo": k8s.io, kubernetes.io and their subdomains are reserved for Kubernetes use`, }, { name: "extra mapping key prefix contains kubernetes.io", in: api.ClaimMappings{ Username: api.PrefixedClaimOrExpression{Expression: "claims.username"}, Groups:   api.PrefixedClaimOrExpression{Expression: "claims.groups"}, Extra: []api.ExtraMapping{ {Key: "example.kubernetes.io/foo", ValueExpression: "claims.extra"}, }, }, structuredAuthnFeatureEnabled: true, want:                          `issuer.claimMappings.extra[0].key: Invalid value: "example.kubernetes.io/foo": k8s.io, kubernetes.io and their subdomains are reserved for Kubernetes use`, }, { name: "extra mapping key prefix with ak8s.io, *.ak8s.io, bkubernetes.io, *.bkubernetes.io are still valid", in: api.ClaimMappings{ Username: api.PrefixedClaimOrExpression{Expression: "claims.username"}, Groups:   api.PrefixedClaimOrExpression{Expression: "claims.groups"}, Extra: []api.ExtraMapping{ {Key: "ak8s.io/foo", ValueExpression: "claims.extra"}, {Key: "example.ak8s.io/foo", ValueExpression: "claims.extra"}, {Key: "bkubernetes.io/foo", ValueExpression: "claims.extra"}, {Key: "example.bkubernetes.io/foo", ValueExpression: "claims.extra"}, }, }, structuredAuthnFeatureEnabled: true, want:                          "", }, { </ins> name: "valid claim mappings but uses email without verification", in: api.ClaimMappings{ Username: api.PrefixedClaimOrExpression{Expression: "claims.email"},

Disallow setting all and namespaced extra info in custom ways for now. /assign aramase enj /sig auth
/triage accepted


Question: <ins> /* Copyright 2020 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package storage import ( "context" "fmt" "path" "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/fields" "k8s.io/kubernetes/pkg/kubelet/events" "k8s.io/kubernetes/test/e2e/framework" e2eevents "k8s.io/kubernetes/test/e2e/framework/events" e2epod "k8s.io/kubernetes/test/e2e/framework/pod" "k8s.io/kubernetes/test/e2e/storage/utils" imageutils "k8s.io/kubernetes/test/utils/image" "github.com/onsi/ginkgo" ) var _ = utils.SIGDescribe("HostPathType Directory [Slow]", func() { f := framework.NewDefaultFramework("host-path-type-directory") var ( ns           string hostBaseDir  string mountBaseDir string basePod      *v1.Pod targetDir    string hostPathUnset             = v1.HostPathUnset hostPathDirectoryOrCreate = v1.HostPathDirectoryOrCreate hostPathDirectory         = v1.HostPathDirectory hostPathFile              = v1.HostPathFile hostPathSocket            = v1.HostPathSocket hostPathCharDev           = v1.HostPathCharDev hostPathBlockDev          = v1.HostPathBlockDev ) ginkgo.BeforeEach(func() { ns = f.Namespace.Name ginkgo.By("Create a pod for further testing") hostBaseDir = path.Join("/tmp", ns) mountBaseDir = "/mnt/test" basePod = f.PodClient().CreateSync(newHostPathTypeTestPod(map[string]string{}, hostBaseDir, mountBaseDir, &hostPathDirectoryOrCreate)) ginkgo.By(fmt.Sprintf("running on node %s", basePod.Spec.NodeName)) targetDir = path.Join(hostBaseDir, "adir") ginkgo.By("Should automatically create a new directory 'adir' when HostPathType is HostPathDirectoryOrCreate") verifyPodHostPathType(f, map[string]string{"kubernetes.io/hostname": basePod.Spec.NodeName}, targetDir, &hostPathDirectoryOrCreate) }) ginkgo.It("Should fail on mounting non-existent directory 'does-not-exist-dir' when HostPathType is HostPathDirectory", func() { dirPath := path.Join(hostBaseDir, "does-not-exist-dir") verifyPodHostPathTypeFailure(f, map[string]string{"kubernetes.io/hostname": basePod.Spec.NodeName}, dirPath, fmt.Sprintf("%s is not a directory", dirPath), &hostPathDirectory) }) ginkgo.It("Should be able to mount directory 'adir' successfully when HostPathType is HostPathDirectory", func() { verifyPodHostPathType(f, map[string]string{"kubernetes.io/hostname": basePod.Spec.NodeName}, targetDir, &hostPathDirectory) }) ginkgo.It("Should be able to mount directory 'adir' successfully when HostPathType is HostPathUnset", func() { verifyPodHostPathType(f, map[string]string{"kubernetes.io/hostname": basePod.Spec.NodeName}, targetDir, &hostPathUnset) }) ginkgo.It("Should fail on mounting directory 'adir' when HostPathType is HostPathFile", func() { verifyPodHostPathTypeFailure(f, map[string]string{"kubernetes.io/hostname": basePod.Spec.NodeName}, targetDir, fmt.Sprintf("%s is not a file", targetDir), &hostPathFile) }) ginkgo.It("Should fail on mounting directory 'adir' when HostPathType is HostPathSocket", func() { verifyPodHostPathTypeFailure(f, map[string]string{"kubernetes.io/hostname": basePod.Spec.NodeName}, targetDir, fmt.Sprintf("%s is not a socket", targetDir), &hostPathSocket) }) ginkgo.It("Should fail on mounting directory 'adir' when HostPathType is HostPathCharDev", func() { verifyPodHostPathTypeFailure(f, map[string]string{"kubernetes.io/hostname": basePod.Spec.NodeName}, targetDir, fmt.Sprintf("%s is not a character device", targetDir), &hostPathCharDev) }) ginkgo.It("Should fail on mounting directory 'adir' when HostPathType is HostPathBlockDev", func() { verifyPodHostPathTypeFailure(f, map[string]string{"kubernetes.io/hostname": basePod.Spec.NodeName}, targetDir, fmt.Sprintf("%s is not a block device", targetDir), &hostPathBlockDev) }) }) var _ = utils.SIGDescribe("HostPathType File [Slow]", func() { f := framework.NewDefaultFramework("host-path-type-file") var ( ns           string hostBaseDir  string mountBaseDir string basePod      *v1.Pod targetFile   string hostPathUnset             = v1.HostPathUnset hostPathDirectoryOrCreate = v1.HostPathDirectoryOrCreate hostPathDirectory         = v1.HostPathDirectory hostPathFileOrCreate      = v1.HostPathFileOrCreate hostPathFile              = v1.HostPathFile hostPathSocket            = v1.HostPathSocket hostPathCharDev           = v1.HostPathCharDev hostPathBlockDev          = v1.HostPathBlockDev ) ginkgo.BeforeEach(func() { ns = f.Namespace.Name ginkgo.By("Create a pod for further testing") hostBaseDir = path.Join("/tmp", ns) mountBaseDir = "/mnt/test" basePod = f.PodClient().CreateSync(newHostPathTypeTestPod(map[string]string{}, hostBaseDir, mountBaseDir, &hostPathDirectoryOrCreate)) ginkgo.By(fmt.Sprintf("running on node %s", basePod.Spec.NodeName)) targetFile = path.Join(hostBaseDir, "afile") ginkgo.By("Should automatically create a new file 'afile' when HostPathType is HostPathFileOrCreate") verifyPodHostPathType(f, map[string]string{"kubernetes.io/hostname": basePod.Spec.NodeName}, targetFile, &hostPathFileOrCreate) }) ginkgo.It("Should fail on mounting non-existent file 'does-not-exist-file' when HostPathType is HostPathFile", func() { filePath := path.Join(hostBaseDir, "does-not-exist-file") verifyPodHostPathTypeFailure(f, map[string]string{"kubernetes.io/hostname": basePod.Spec.NodeName}, filePath, fmt.Sprintf("%s is not a file", filePath), &hostPathFile) }) ginkgo.It("Should be able to mount file 'afile' successfully when HostPathType is HostPathFile", func() { verifyPodHostPathType(f, map[string]string{"kubernetes.io/hostname": basePod.Spec.NodeName}, targetFile, &hostPathFile) }) ginkgo.It("Should be able to mount file 'afile' successfully when HostPathType is HostPathUnset", func() { verifyPodHostPathType(f, map[string]string{"kubernetes.io/hostname": basePod.Spec.NodeName}, targetFile, &hostPathUnset) }) ginkgo.It("Should fail on mounting file 'afile' when HostPathType is HostPathDirectory", func() { verifyPodHostPathTypeFailure(f, map[string]string{"kubernetes.io/hostname": basePod.Spec.NodeName}, targetFile, fmt.Sprintf("%s is not a directory", targetFile), &hostPathDirectory) }) ginkgo.It("Should fail on mounting file 'afile' when HostPathType is HostPathSocket", func() { verifyPodHostPathTypeFailure(f, map[string]string{"kubernetes.io/hostname": basePod.Spec.NodeName}, targetFile, fmt.Sprintf("%s is not a socket", targetFile), &hostPathSocket) }) ginkgo.It("Should fail on mounting file 'afile' when HostPathType is HostPathCharDev", func() { verifyPodHostPathTypeFailure(f, map[string]string{"kubernetes.io/hostname": basePod.Spec.NodeName}, targetFile, fmt.Sprintf("%s is not a character device", targetFile), &hostPathCharDev) }) ginkgo.It("Should fail on mounting file 'afile' when HostPathType is HostPathBlockDev", func() { verifyPodHostPathTypeFailure(f, map[string]string{"kubernetes.io/hostname": basePod.Spec.NodeName}, targetFile, fmt.Sprintf("%s is not a block device", targetFile), &hostPathBlockDev) }) }) var _ = utils.SIGDescribe("HostPathType Socket [Slow]", func() { f := framework.NewDefaultFramework("host-path-type-socket") var ( ns           string hostBaseDir  string mountBaseDir string basePod      *v1.Pod targetSocket string hostPathUnset             = v1.HostPathUnset hostPathDirectoryOrCreate = v1.HostPathDirectoryOrCreate hostPathDirectory         = v1.HostPathDirectory hostPathFile              = v1.HostPathFile hostPathSocket            = v1.HostPathSocket hostPathCharDev           = v1.HostPathCharDev hostPathBlockDev          = v1.HostPathBlockDev ) ginkgo.BeforeEach(func() { ns = f.Namespace.Name ginkgo.By("Create a pod for further testing") hostBaseDir = path.Join("/tmp", ns) mountBaseDir = "/mnt/test" basePod = f.PodClient().CreateSync(newHostPathTypeTestPod(map[string]string{}, hostBaseDir, mountBaseDir, &hostPathDirectoryOrCreate)) ginkgo.By(fmt.Sprintf("running on node %s", basePod.Spec.NodeName)) targetSocket = path.Join(hostBaseDir, "asocket") ginkgo.By("Create a socket for further testing") _, err := utils.PodExec(f, basePod, fmt.Sprintf("nc -lU %s &", path.Join(mountBaseDir, "asocket"))) framework.ExpectNoError(err) }) ginkgo.It("Should fail on mounting non-existent socket 'does-not-exist-socket' when HostPathType is HostPathSocket", func() { socketPath := path.Join(hostBaseDir, "does-not-exist-socket") verifyPodHostPathTypeFailure(f, map[string]string{"kubernetes.io/hostname": basePod.Spec.NodeName}, socketPath, fmt.Sprintf("%s is not a socket", socketPath), &hostPathSocket) }) ginkgo.It("Should be able to mount socket 'asocket' successfully when HostPathType is HostPathSocket", func() { verifyPodHostPathType(f, map[string]string{"kubernetes.io/hostname": basePod.Spec.NodeName}, targetSocket, &hostPathSocket) }) ginkgo.It("Should be able to mount socket 'asocket' successfully when HostPathType is HostPathUnset", func() { verifyPodHostPathType(f, map[string]string{"kubernetes.io/hostname": basePod.Spec.NodeName}, targetSocket, &hostPathUnset) }) ginkgo.It("Should fail on mounting socket 'asocket' when HostPathType is HostPathDirectory", func() { verifyPodHostPathTypeFailure(f, map[string]string{"kubernetes.io/hostname": basePod.Spec.NodeName}, targetSocket, fmt.Sprintf("%s is not a directory", targetSocket), &hostPathDirectory) }) ginkgo.It("Should fail on mounting socket 'asocket' when HostPathType is HostPathFile", func() { verifyPodHostPathTypeFailure(f, map[string]string{"kubernetes.io/hostname": basePod.Spec.NodeName}, targetSocket, fmt.Sprintf("%s is not a file", targetSocket), &hostPathFile) }) ginkgo.It("Should fail on mounting socket 'asocket' when HostPathType is HostPathCharDev", func() { verifyPodHostPathTypeFailure(f, map[string]string{"kubernetes.io/hostname": basePod.Spec.NodeName}, targetSocket, fmt.Sprintf("%s is not a character device", targetSocket), &hostPathCharDev) }) ginkgo.It("Should fail on mounting socket 'asocket' when HostPathType is HostPathBlockDev", func() { verifyPodHostPathTypeFailure(f, map[string]string{"kubernetes.io/hostname": basePod.Spec.NodeName}, targetSocket, fmt.Sprintf("%s is not a block device", targetSocket), &hostPathBlockDev) }) }) var _ = utils.SIGDescribe("HostPathType Character Device [Slow]", func() { f := framework.NewDefaultFramework("host-path-type-char-dev") var ( ns            string hostBaseDir   string mountBaseDir  string basePod       *v1.Pod targetCharDev string hostPathUnset             = v1.HostPathUnset hostPathDirectoryOrCreate = v1.HostPathDirectoryOrCreate hostPathDirectory         = v1.HostPathDirectory hostPathFile              = v1.HostPathFile hostPathSocket            = v1.HostPathSocket hostPathCharDev           = v1.HostPathCharDev hostPathBlockDev          = v1.HostPathBlockDev ) ginkgo.BeforeEach(func() { ns = f.Namespace.Name ginkgo.By("Create a pod for further testing") hostBaseDir = path.Join("/tmp", ns) mountBaseDir = "/mnt/test" basePod = f.PodClient().CreateSync(newHostPathTypeTestPod(map[string]string{}, hostBaseDir, mountBaseDir, &hostPathDirectoryOrCreate)) ginkgo.By(fmt.Sprintf("running on node %s", basePod.Spec.NodeName)) targetCharDev = path.Join(hostBaseDir, "achardev") ginkgo.By("Create a character device for further testing") _, err := utils.PodExec(f, basePod, fmt.Sprintf("mknod %s c 89 1", path.Join(mountBaseDir, "achardev"))) framework.ExpectNoError(err) }) ginkgo.It("Should fail on mounting non-existent character device 'does-not-exist-char-dev' when HostPathType is HostPathCharDev", func() { charDevPath := path.Join(hostBaseDir, "does-not-exist-char-dev") verifyPodHostPathTypeFailure(f, map[string]string{"kubernetes.io/hostname": basePod.Spec.NodeName}, charDevPath, fmt.Sprintf("%s is not a character device", charDevPath), &hostPathCharDev) }) ginkgo.It("Should be able to mount character device 'achardev' successfully when HostPathType is HostPathCharDev", func() { verifyPodHostPathType(f, map[string]string{"kubernetes.io/hostname": basePod.Spec.NodeName}, targetCharDev, &hostPathCharDev) }) ginkgo.It("Should be able to mount character device 'achardev' successfully when HostPathType is HostPathUnset", func() { verifyPodHostPathType(f, map[string]string{"kubernetes.io/hostname": basePod.Spec.NodeName}, targetCharDev, &hostPathUnset) }) ginkgo.It("Should fail on mounting character device 'achardev' when HostPathType is HostPathDirectory", func() { verifyPodHostPathTypeFailure(f, map[string]string{"kubernetes.io/hostname": basePod.Spec.NodeName}, targetCharDev, fmt.Sprintf("%s is not a directory", targetCharDev), &hostPathDirectory) }) ginkgo.It("Should fail on mounting character device 'achardev' when HostPathType is HostPathFile", func() { verifyPodHostPathTypeFailure(f, map[string]string{"kubernetes.io/hostname": basePod.Spec.NodeName}, targetCharDev, fmt.Sprintf("%s is not a file", targetCharDev), &hostPathFile) }) ginkgo.It("Should fail on mounting character device 'achardev' when HostPathType is HostPathSocket", func() { verifyPodHostPathTypeFailure(f, map[string]string{"kubernetes.io/hostname": basePod.Spec.NodeName}, targetCharDev, fmt.Sprintf("%s is not a socket", targetCharDev), &hostPathSocket) }) ginkgo.It("Should fail on mounting character device 'achardev' when HostPathType is HostPathBlockDev", func() { verifyPodHostPathTypeFailure(f, map[string]string{"kubernetes.io/hostname": basePod.Spec.NodeName}, targetCharDev, fmt.Sprintf("%s is not a block device", targetCharDev), &hostPathBlockDev) }) }) var _ = utils.SIGDescribe("HostPathType Block Device [Slow]", func() { f := framework.NewDefaultFramework("host-path-type-block-dev") var ( ns             string hostBaseDir    string mountBaseDir   string basePod        *v1.Pod targetBlockDev string hostPathUnset             = v1.HostPathUnset hostPathDirectoryOrCreate = v1.HostPathDirectoryOrCreate hostPathDirectory         = v1.HostPathDirectory hostPathFile              = v1.HostPathFile hostPathSocket            = v1.HostPathSocket hostPathCharDev           = v1.HostPathCharDev hostPathBlockDev          = v1.HostPathBlockDev ) ginkgo.BeforeEach(func() { ns = f.Namespace.Name ginkgo.By("Create a pod for further testing") hostBaseDir = path.Join("/tmp", ns) mountBaseDir = "/mnt/test" basePod = f.PodClient().CreateSync(newHostPathTypeTestPod(map[string]string{}, hostBaseDir, mountBaseDir, &hostPathDirectoryOrCreate)) ginkgo.By(fmt.Sprintf("running on node %s", basePod.Spec.NodeName)) targetBlockDev = path.Join(hostBaseDir, "ablkdev") ginkgo.By("Create a block device for further testing") _, err := utils.PodExec(f, basePod, fmt.Sprintf("mknod %s b 89 1", path.Join(mountBaseDir, "ablkdev"))) framework.ExpectNoError(err) }) ginkgo.It("Should fail on mounting non-existent block device 'does-not-exist-blk-dev' when HostPathType is HostPathBlockDev", func() { blkDevPath := path.Join(hostBaseDir, "does-not-exist-blk-dev") verifyPodHostPathTypeFailure(f, map[string]string{"kubernetes.io/hostname": basePod.Spec.NodeName}, blkDevPath, fmt.Sprintf("%s is not a block device", blkDevPath), &hostPathBlockDev) }) ginkgo.It("Should be able to mount block device 'ablkdev' successfully when HostPathType is HostPathBlockDev", func() { verifyPodHostPathType(f, map[string]string{"kubernetes.io/hostname": basePod.Spec.NodeName}, targetBlockDev, &hostPathBlockDev) }) ginkgo.It("Should be able to mount block device 'ablkdev' successfully when HostPathType is HostPathUnset", func() { verifyPodHostPathType(f, map[string]string{"kubernetes.io/hostname": basePod.Spec.NodeName}, targetBlockDev, &hostPathUnset) }) ginkgo.It("Should fail on mounting block device 'ablkdev' when HostPathType is HostPathDirectory", func() { verifyPodHostPathTypeFailure(f, map[string]string{"kubernetes.io/hostname": basePod.Spec.NodeName}, targetBlockDev, fmt.Sprintf("%s is not a directory", targetBlockDev), &hostPathDirectory) }) ginkgo.It("Should fail on mounting block device 'ablkdev' when HostPathType is HostPathFile", func() { verifyPodHostPathTypeFailure(f, map[string]string{"kubernetes.io/hostname": basePod.Spec.NodeName}, targetBlockDev, fmt.Sprintf("%s is not a file", targetBlockDev), &hostPathFile) }) ginkgo.It("Should fail on mounting block device 'ablkdev' when HostPathType is HostPathSocket", func() { verifyPodHostPathTypeFailure(f, map[string]string{"kubernetes.io/hostname": basePod.Spec.NodeName}, targetBlockDev, fmt.Sprintf("%s is not a socket", targetBlockDev), &hostPathSocket) }) ginkgo.It("Should fail on mounting block device 'ablkdev' when HostPathType is HostPathCharDev", func() { verifyPodHostPathTypeFailure(f, map[string]string{"kubernetes.io/hostname": basePod.Spec.NodeName}, targetBlockDev, fmt.Sprintf("%s is not a character device", targetBlockDev), &hostPathCharDev) }) }) func newHostPathTypeTestPod(nodeSelector map[string]string, hostDir, mountDir string, hostPathType *v1.HostPathType) *v1.Pod { pod := &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ GenerateName: "test-hostpath-type-", }, Spec: v1.PodSpec{ NodeSelector:  nodeSelector, RestartPolicy: v1.RestartPolicyNever, Containers: []v1.Container{ { Name:  "host-path-testing", Image: imageutils.GetE2EImage(imageutils.Agnhost), VolumeMounts: []v1.VolumeMount{ { Name:      "host", MountPath: mountDir, ReadOnly:  false, }, }, }, }, Volumes: []v1.Volume{ { Name: "host", VolumeSource: v1.VolumeSource{ HostPath: &v1.HostPathVolumeSource{ Path: hostDir, Type: hostPathType, }, }, }, }, }, } return pod } func verifyPodHostPathTypeFailure(f *framework.Framework, nodeSelector map[string]string, hostDir, pattern string, hostPathType *v1.HostPathType) { pod := newHostPathTypeTestPod(nodeSelector, hostDir, "/mnt/test", hostPathType) ginkgo.By(fmt.Sprintf("Creating pod %s", pod.Name)) pod, err := f.ClientSet.CoreV1().Pods(f.Namespace.Name).Create(context.TODO(), pod, metav1.CreateOptions{}) framework.ExpectNoError(err) ginkgo.By("Checking for HostPathType error event") eventSelector := fields.Set{ "involvedObject.kind":      "Pod", "involvedObject.name":      pod.Name, "involvedObject.namespace": f.Namespace.Name, "reason":                   events.FailedMountVolume, }.AsSelector().String() msg := "hostPath type check failed" err = e2eevents.WaitTimeoutForEvent(f.ClientSet, f.Namespace.Name, eventSelector, msg, framework.PodStartTimeout) // Events are unreliable, don't depend on the event. It's used only to speed up the test. if err != nil { framework.Logf("Warning: did not get event about FailedMountVolume") } // Check the pod is still not running p, err := f.ClientSet.CoreV1().Pods(f.Namespace.Name).Get(context.TODO(), pod.Name, metav1.GetOptions{}) framework.ExpectNoError(err, "could not re-read the pod after event (or timeout)") framework.ExpectEqual(p.Status.Phase, v1.PodPending, "Pod phase isn't pending") f.ClientSet.CoreV1().Pods(f.Namespace.Name).Delete(context.TODO(), pod.Name, *metav1.NewDeleteOptions(0)) } func verifyPodHostPathType(f *framework.Framework, nodeSelector map[string]string, hostDir string, hostPathType *v1.HostPathType) { newPod, err := f.ClientSet.CoreV1().Pods(f.Namespace.Name).Create(context.TODO(), newHostPathTypeTestPod(nodeSelector, hostDir, "/mnt/test", hostPathType), metav1.CreateOptions{}) framework.ExpectNoError(err) framework.ExpectNoError(e2epod.WaitTimeoutForPodRunningInNamespace(f.ClientSet, newPod.Name, newPod.Namespace, framework.PodStartShortTimeout)) f.ClientSet.CoreV1().Pods(f.Namespace.Name).Delete(context.TODO(), newPod.Name, *metav1.NewDeleteOptions(0)) } </ins>

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: This feature is lacking coverage, especially for containerized kubelet environments.
/assign since I'm the owner of this feature.
And just fixes a bug for containerized kubelet when the mount file is an empty file.
Are we going to enable these tests in v1.11? We're getting code freeze soon.
There is a local containerized cluster pull and CI job that we can probably leverage. Right now it only runs conformance tests, we may be able to add new suites to enable other tests. Cc who can point us to the owner for the containerized testing effort
yes, seems to be holding up pretty well - https://k8s-,%20master%20(dev)
In I made pull-kubernetes-local-e2e-containerized to run also all test with string. I'm enabling some tests (subpath!) in , feel free to suggest more tests.
Ref
IIUC, the default host path type is . And when host path type is , we expect container runtime to create the host path if it doesn't exist (docker is doing that today). Is that true? If that is, we should revert part of , because container runtime should still create host path if it does not exist. /cc I tried to enforce , and got a lot of errors , e.g.:
Correct. This is mainly for backwards compatibility.
I see. If so, I'll remove the CRI definition "If the hostPath doesn't exist (e.g. hostPath volume), runtimes should report errors". Thanks!
SGTM. Refer to .
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale will help fix this.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close.


Question: <ins> /* Copyright 2020 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package storage import ( "context" "fmt" "path" "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/fields" "k8s.io/kubernetes/pkg/kubelet/events" "k8s.io/kubernetes/test/e2e/framework" e2eevents "k8s.io/kubernetes/test/e2e/framework/events" e2epod "k8s.io/kubernetes/test/e2e/framework/pod" "k8s.io/kubernetes/test/e2e/storage/utils" imageutils "k8s.io/kubernetes/test/utils/image" "github.com/onsi/ginkgo" ) var _ = utils.SIGDescribe("HostPathType Directory [Slow]", func() { f := framework.NewDefaultFramework("host-path-type-directory") var ( ns           string hostBaseDir  string mountBaseDir string basePod      *v1.Pod targetDir    string hostPathUnset             = v1.HostPathUnset hostPathDirectoryOrCreate = v1.HostPathDirectoryOrCreate hostPathDirectory         = v1.HostPathDirectory hostPathFile              = v1.HostPathFile hostPathSocket            = v1.HostPathSocket hostPathCharDev           = v1.HostPathCharDev hostPathBlockDev          = v1.HostPathBlockDev ) ginkgo.BeforeEach(func() { ns = f.Namespace.Name ginkgo.By("Create a pod for further testing") hostBaseDir = path.Join("/tmp", ns) mountBaseDir = "/mnt/test" basePod = f.PodClient().CreateSync(newHostPathTypeTestPod(map[string]string{}, hostBaseDir, mountBaseDir, &hostPathDirectoryOrCreate)) ginkgo.By(fmt.Sprintf("running on node %s", basePod.Spec.NodeName)) targetDir = path.Join(hostBaseDir, "adir") ginkgo.By("Should automatically create a new directory 'adir' when HostPathType is HostPathDirectoryOrCreate") verifyPodHostPathType(f, map[string]string{"kubernetes.io/hostname": basePod.Spec.NodeName}, targetDir, &hostPathDirectoryOrCreate) }) ginkgo.It("Should fail on mounting non-existent directory 'does-not-exist-dir' when HostPathType is HostPathDirectory", func() { dirPath := path.Join(hostBaseDir, "does-not-exist-dir") verifyPodHostPathTypeFailure(f, map[string]string{"kubernetes.io/hostname": basePod.Spec.NodeName}, dirPath, fmt.Sprintf("%s is not a directory", dirPath), &hostPathDirectory) }) ginkgo.It("Should be able to mount directory 'adir' successfully when HostPathType is HostPathDirectory", func() { verifyPodHostPathType(f, map[string]string{"kubernetes.io/hostname": basePod.Spec.NodeName}, targetDir, &hostPathDirectory) }) ginkgo.It("Should be able to mount directory 'adir' successfully when HostPathType is HostPathUnset", func() { verifyPodHostPathType(f, map[string]string{"kubernetes.io/hostname": basePod.Spec.NodeName}, targetDir, &hostPathUnset) }) ginkgo.It("Should fail on mounting directory 'adir' when HostPathType is HostPathFile", func() { verifyPodHostPathTypeFailure(f, map[string]string{"kubernetes.io/hostname": basePod.Spec.NodeName}, targetDir, fmt.Sprintf("%s is not a file", targetDir), &hostPathFile) }) ginkgo.It("Should fail on mounting directory 'adir' when HostPathType is HostPathSocket", func() { verifyPodHostPathTypeFailure(f, map[string]string{"kubernetes.io/hostname": basePod.Spec.NodeName}, targetDir, fmt.Sprintf("%s is not a socket", targetDir), &hostPathSocket) }) ginkgo.It("Should fail on mounting directory 'adir' when HostPathType is HostPathCharDev", func() { verifyPodHostPathTypeFailure(f, map[string]string{"kubernetes.io/hostname": basePod.Spec.NodeName}, targetDir, fmt.Sprintf("%s is not a character device", targetDir), &hostPathCharDev) }) ginkgo.It("Should fail on mounting directory 'adir' when HostPathType is HostPathBlockDev", func() { verifyPodHostPathTypeFailure(f, map[string]string{"kubernetes.io/hostname": basePod.Spec.NodeName}, targetDir, fmt.Sprintf("%s is not a block device", targetDir), &hostPathBlockDev) }) }) var _ = utils.SIGDescribe("HostPathType File [Slow]", func() { f := framework.NewDefaultFramework("host-path-type-file") var ( ns           string hostBaseDir  string mountBaseDir string basePod      *v1.Pod targetFile   string hostPathUnset             = v1.HostPathUnset hostPathDirectoryOrCreate = v1.HostPathDirectoryOrCreate hostPathDirectory         = v1.HostPathDirectory hostPathFileOrCreate      = v1.HostPathFileOrCreate hostPathFile              = v1.HostPathFile hostPathSocket            = v1.HostPathSocket hostPathCharDev           = v1.HostPathCharDev hostPathBlockDev          = v1.HostPathBlockDev ) ginkgo.BeforeEach(func() { ns = f.Namespace.Name ginkgo.By("Create a pod for further testing") hostBaseDir = path.Join("/tmp", ns) mountBaseDir = "/mnt/test" basePod = f.PodClient().CreateSync(newHostPathTypeTestPod(map[string]string{}, hostBaseDir, mountBaseDir, &hostPathDirectoryOrCreate)) ginkgo.By(fmt.Sprintf("running on node %s", basePod.Spec.NodeName)) targetFile = path.Join(hostBaseDir, "afile") ginkgo.By("Should automatically create a new file 'afile' when HostPathType is HostPathFileOrCreate") verifyPodHostPathType(f, map[string]string{"kubernetes.io/hostname": basePod.Spec.NodeName}, targetFile, &hostPathFileOrCreate) }) ginkgo.It("Should fail on mounting non-existent file 'does-not-exist-file' when HostPathType is HostPathFile", func() { filePath := path.Join(hostBaseDir, "does-not-exist-file") verifyPodHostPathTypeFailure(f, map[string]string{"kubernetes.io/hostname": basePod.Spec.NodeName}, filePath, fmt.Sprintf("%s is not a file", filePath), &hostPathFile) }) ginkgo.It("Should be able to mount file 'afile' successfully when HostPathType is HostPathFile", func() { verifyPodHostPathType(f, map[string]string{"kubernetes.io/hostname": basePod.Spec.NodeName}, targetFile, &hostPathFile) }) ginkgo.It("Should be able to mount file 'afile' successfully when HostPathType is HostPathUnset", func() { verifyPodHostPathType(f, map[string]string{"kubernetes.io/hostname": basePod.Spec.NodeName}, targetFile, &hostPathUnset) }) ginkgo.It("Should fail on mounting file 'afile' when HostPathType is HostPathDirectory", func() { verifyPodHostPathTypeFailure(f, map[string]string{"kubernetes.io/hostname": basePod.Spec.NodeName}, targetFile, fmt.Sprintf("%s is not a directory", targetFile), &hostPathDirectory) }) ginkgo.It("Should fail on mounting file 'afile' when HostPathType is HostPathSocket", func() { verifyPodHostPathTypeFailure(f, map[string]string{"kubernetes.io/hostname": basePod.Spec.NodeName}, targetFile, fmt.Sprintf("%s is not a socket", targetFile), &hostPathSocket) }) ginkgo.It("Should fail on mounting file 'afile' when HostPathType is HostPathCharDev", func() { verifyPodHostPathTypeFailure(f, map[string]string{"kubernetes.io/hostname": basePod.Spec.NodeName}, targetFile, fmt.Sprintf("%s is not a character device", targetFile), &hostPathCharDev) }) ginkgo.It("Should fail on mounting file 'afile' when HostPathType is HostPathBlockDev", func() { verifyPodHostPathTypeFailure(f, map[string]string{"kubernetes.io/hostname": basePod.Spec.NodeName}, targetFile, fmt.Sprintf("%s is not a block device", targetFile), &hostPathBlockDev) }) }) var _ = utils.SIGDescribe("HostPathType Socket [Slow]", func() { f := framework.NewDefaultFramework("host-path-type-socket") var ( ns           string hostBaseDir  string mountBaseDir string basePod      *v1.Pod targetSocket string hostPathUnset             = v1.HostPathUnset hostPathDirectoryOrCreate = v1.HostPathDirectoryOrCreate hostPathDirectory         = v1.HostPathDirectory hostPathFile              = v1.HostPathFile hostPathSocket            = v1.HostPathSocket hostPathCharDev           = v1.HostPathCharDev hostPathBlockDev          = v1.HostPathBlockDev ) ginkgo.BeforeEach(func() { ns = f.Namespace.Name ginkgo.By("Create a pod for further testing") hostBaseDir = path.Join("/tmp", ns) mountBaseDir = "/mnt/test" basePod = f.PodClient().CreateSync(newHostPathTypeTestPod(map[string]string{}, hostBaseDir, mountBaseDir, &hostPathDirectoryOrCreate)) ginkgo.By(fmt.Sprintf("running on node %s", basePod.Spec.NodeName)) targetSocket = path.Join(hostBaseDir, "asocket") ginkgo.By("Create a socket for further testing") _, err := utils.PodExec(f, basePod, fmt.Sprintf("nc -lU %s &", path.Join(mountBaseDir, "asocket"))) framework.ExpectNoError(err) }) ginkgo.It("Should fail on mounting non-existent socket 'does-not-exist-socket' when HostPathType is HostPathSocket", func() { socketPath := path.Join(hostBaseDir, "does-not-exist-socket") verifyPodHostPathTypeFailure(f, map[string]string{"kubernetes.io/hostname": basePod.Spec.NodeName}, socketPath, fmt.Sprintf("%s is not a socket", socketPath), &hostPathSocket) }) ginkgo.It("Should be able to mount socket 'asocket' successfully when HostPathType is HostPathSocket", func() { verifyPodHostPathType(f, map[string]string{"kubernetes.io/hostname": basePod.Spec.NodeName}, targetSocket, &hostPathSocket) }) ginkgo.It("Should be able to mount socket 'asocket' successfully when HostPathType is HostPathUnset", func() { verifyPodHostPathType(f, map[string]string{"kubernetes.io/hostname": basePod.Spec.NodeName}, targetSocket, &hostPathUnset) }) ginkgo.It("Should fail on mounting socket 'asocket' when HostPathType is HostPathDirectory", func() { verifyPodHostPathTypeFailure(f, map[string]string{"kubernetes.io/hostname": basePod.Spec.NodeName}, targetSocket, fmt.Sprintf("%s is not a directory", targetSocket), &hostPathDirectory) }) ginkgo.It("Should fail on mounting socket 'asocket' when HostPathType is HostPathFile", func() { verifyPodHostPathTypeFailure(f, map[string]string{"kubernetes.io/hostname": basePod.Spec.NodeName}, targetSocket, fmt.Sprintf("%s is not a file", targetSocket), &hostPathFile) }) ginkgo.It("Should fail on mounting socket 'asocket' when HostPathType is HostPathCharDev", func() { verifyPodHostPathTypeFailure(f, map[string]string{"kubernetes.io/hostname": basePod.Spec.NodeName}, targetSocket, fmt.Sprintf("%s is not a character device", targetSocket), &hostPathCharDev) }) ginkgo.It("Should fail on mounting socket 'asocket' when HostPathType is HostPathBlockDev", func() { verifyPodHostPathTypeFailure(f, map[string]string{"kubernetes.io/hostname": basePod.Spec.NodeName}, targetSocket, fmt.Sprintf("%s is not a block device", targetSocket), &hostPathBlockDev) }) }) var _ = utils.SIGDescribe("HostPathType Character Device [Slow]", func() { f := framework.NewDefaultFramework("host-path-type-char-dev") var ( ns            string hostBaseDir   string mountBaseDir  string basePod       *v1.Pod targetCharDev string hostPathUnset             = v1.HostPathUnset hostPathDirectoryOrCreate = v1.HostPathDirectoryOrCreate hostPathDirectory         = v1.HostPathDirectory hostPathFile              = v1.HostPathFile hostPathSocket            = v1.HostPathSocket hostPathCharDev           = v1.HostPathCharDev hostPathBlockDev          = v1.HostPathBlockDev ) ginkgo.BeforeEach(func() { ns = f.Namespace.Name ginkgo.By("Create a pod for further testing") hostBaseDir = path.Join("/tmp", ns) mountBaseDir = "/mnt/test" basePod = f.PodClient().CreateSync(newHostPathTypeTestPod(map[string]string{}, hostBaseDir, mountBaseDir, &hostPathDirectoryOrCreate)) ginkgo.By(fmt.Sprintf("running on node %s", basePod.Spec.NodeName)) targetCharDev = path.Join(hostBaseDir, "achardev") ginkgo.By("Create a character device for further testing") _, err := utils.PodExec(f, basePod, fmt.Sprintf("mknod %s c 89 1", path.Join(mountBaseDir, "achardev"))) framework.ExpectNoError(err) }) ginkgo.It("Should fail on mounting non-existent character device 'does-not-exist-char-dev' when HostPathType is HostPathCharDev", func() { charDevPath := path.Join(hostBaseDir, "does-not-exist-char-dev") verifyPodHostPathTypeFailure(f, map[string]string{"kubernetes.io/hostname": basePod.Spec.NodeName}, charDevPath, fmt.Sprintf("%s is not a character device", charDevPath), &hostPathCharDev) }) ginkgo.It("Should be able to mount character device 'achardev' successfully when HostPathType is HostPathCharDev", func() { verifyPodHostPathType(f, map[string]string{"kubernetes.io/hostname": basePod.Spec.NodeName}, targetCharDev, &hostPathCharDev) }) ginkgo.It("Should be able to mount character device 'achardev' successfully when HostPathType is HostPathUnset", func() { verifyPodHostPathType(f, map[string]string{"kubernetes.io/hostname": basePod.Spec.NodeName}, targetCharDev, &hostPathUnset) }) ginkgo.It("Should fail on mounting character device 'achardev' when HostPathType is HostPathDirectory", func() { verifyPodHostPathTypeFailure(f, map[string]string{"kubernetes.io/hostname": basePod.Spec.NodeName}, targetCharDev, fmt.Sprintf("%s is not a directory", targetCharDev), &hostPathDirectory) }) ginkgo.It("Should fail on mounting character device 'achardev' when HostPathType is HostPathFile", func() { verifyPodHostPathTypeFailure(f, map[string]string{"kubernetes.io/hostname": basePod.Spec.NodeName}, targetCharDev, fmt.Sprintf("%s is not a file", targetCharDev), &hostPathFile) }) ginkgo.It("Should fail on mounting character device 'achardev' when HostPathType is HostPathSocket", func() { verifyPodHostPathTypeFailure(f, map[string]string{"kubernetes.io/hostname": basePod.Spec.NodeName}, targetCharDev, fmt.Sprintf("%s is not a socket", targetCharDev), &hostPathSocket) }) ginkgo.It("Should fail on mounting character device 'achardev' when HostPathType is HostPathBlockDev", func() { verifyPodHostPathTypeFailure(f, map[string]string{"kubernetes.io/hostname": basePod.Spec.NodeName}, targetCharDev, fmt.Sprintf("%s is not a block device", targetCharDev), &hostPathBlockDev) }) }) var _ = utils.SIGDescribe("HostPathType Block Device [Slow]", func() { f := framework.NewDefaultFramework("host-path-type-block-dev") var ( ns             string hostBaseDir    string mountBaseDir   string basePod        *v1.Pod targetBlockDev string hostPathUnset             = v1.HostPathUnset hostPathDirectoryOrCreate = v1.HostPathDirectoryOrCreate hostPathDirectory         = v1.HostPathDirectory hostPathFile              = v1.HostPathFile hostPathSocket            = v1.HostPathSocket hostPathCharDev           = v1.HostPathCharDev hostPathBlockDev          = v1.HostPathBlockDev ) ginkgo.BeforeEach(func() { ns = f.Namespace.Name ginkgo.By("Create a pod for further testing") hostBaseDir = path.Join("/tmp", ns) mountBaseDir = "/mnt/test" basePod = f.PodClient().CreateSync(newHostPathTypeTestPod(map[string]string{}, hostBaseDir, mountBaseDir, &hostPathDirectoryOrCreate)) ginkgo.By(fmt.Sprintf("running on node %s", basePod.Spec.NodeName)) targetBlockDev = path.Join(hostBaseDir, "ablkdev") ginkgo.By("Create a block device for further testing") _, err := utils.PodExec(f, basePod, fmt.Sprintf("mknod %s b 89 1", path.Join(mountBaseDir, "ablkdev"))) framework.ExpectNoError(err) }) ginkgo.It("Should fail on mounting non-existent block device 'does-not-exist-blk-dev' when HostPathType is HostPathBlockDev", func() { blkDevPath := path.Join(hostBaseDir, "does-not-exist-blk-dev") verifyPodHostPathTypeFailure(f, map[string]string{"kubernetes.io/hostname": basePod.Spec.NodeName}, blkDevPath, fmt.Sprintf("%s is not a block device", blkDevPath), &hostPathBlockDev) }) ginkgo.It("Should be able to mount block device 'ablkdev' successfully when HostPathType is HostPathBlockDev", func() { verifyPodHostPathType(f, map[string]string{"kubernetes.io/hostname": basePod.Spec.NodeName}, targetBlockDev, &hostPathBlockDev) }) ginkgo.It("Should be able to mount block device 'ablkdev' successfully when HostPathType is HostPathUnset", func() { verifyPodHostPathType(f, map[string]string{"kubernetes.io/hostname": basePod.Spec.NodeName}, targetBlockDev, &hostPathUnset) }) ginkgo.It("Should fail on mounting block device 'ablkdev' when HostPathType is HostPathDirectory", func() { verifyPodHostPathTypeFailure(f, map[string]string{"kubernetes.io/hostname": basePod.Spec.NodeName}, targetBlockDev, fmt.Sprintf("%s is not a directory", targetBlockDev), &hostPathDirectory) }) ginkgo.It("Should fail on mounting block device 'ablkdev' when HostPathType is HostPathFile", func() { verifyPodHostPathTypeFailure(f, map[string]string{"kubernetes.io/hostname": basePod.Spec.NodeName}, targetBlockDev, fmt.Sprintf("%s is not a file", targetBlockDev), &hostPathFile) }) ginkgo.It("Should fail on mounting block device 'ablkdev' when HostPathType is HostPathSocket", func() { verifyPodHostPathTypeFailure(f, map[string]string{"kubernetes.io/hostname": basePod.Spec.NodeName}, targetBlockDev, fmt.Sprintf("%s is not a socket", targetBlockDev), &hostPathSocket) }) ginkgo.It("Should fail on mounting block device 'ablkdev' when HostPathType is HostPathCharDev", func() { verifyPodHostPathTypeFailure(f, map[string]string{"kubernetes.io/hostname": basePod.Spec.NodeName}, targetBlockDev, fmt.Sprintf("%s is not a character device", targetBlockDev), &hostPathCharDev) }) }) func newHostPathTypeTestPod(nodeSelector map[string]string, hostDir, mountDir string, hostPathType *v1.HostPathType) *v1.Pod { pod := &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ GenerateName: "test-hostpath-type-", }, Spec: v1.PodSpec{ NodeSelector:  nodeSelector, RestartPolicy: v1.RestartPolicyNever, Containers: []v1.Container{ { Name:  "host-path-testing", Image: imageutils.GetE2EImage(imageutils.Agnhost), VolumeMounts: []v1.VolumeMount{ { Name:      "host", MountPath: mountDir, ReadOnly:  false, }, }, }, }, Volumes: []v1.Volume{ { Name: "host", VolumeSource: v1.VolumeSource{ HostPath: &v1.HostPathVolumeSource{ Path: hostDir, Type: hostPathType, }, }, }, }, }, } return pod } func verifyPodHostPathTypeFailure(f *framework.Framework, nodeSelector map[string]string, hostDir, pattern string, hostPathType *v1.HostPathType) { pod := newHostPathTypeTestPod(nodeSelector, hostDir, "/mnt/test", hostPathType) ginkgo.By(fmt.Sprintf("Creating pod %s", pod.Name)) pod, err := f.ClientSet.CoreV1().Pods(f.Namespace.Name).Create(context.TODO(), pod, metav1.CreateOptions{}) framework.ExpectNoError(err) ginkgo.By("Checking for HostPathType error event") eventSelector := fields.Set{ "involvedObject.kind":      "Pod", "involvedObject.name":      pod.Name, "involvedObject.namespace": f.Namespace.Name, "reason":                   events.FailedMountVolume, }.AsSelector().String() msg := "hostPath type check failed" err = e2eevents.WaitTimeoutForEvent(f.ClientSet, f.Namespace.Name, eventSelector, msg, framework.PodStartTimeout) // Events are unreliable, don't depend on the event. It's used only to speed up the test. if err != nil { framework.Logf("Warning: did not get event about FailedMountVolume") } // Check the pod is still not running p, err := f.ClientSet.CoreV1().Pods(f.Namespace.Name).Get(context.TODO(), pod.Name, metav1.GetOptions{}) framework.ExpectNoError(err, "could not re-read the pod after event (or timeout)") framework.ExpectEqual(p.Status.Phase, v1.PodPending, "Pod phase isn't pending") f.ClientSet.CoreV1().Pods(f.Namespace.Name).Delete(context.TODO(), pod.Name, *metav1.NewDeleteOptions(0)) } func verifyPodHostPathType(f *framework.Framework, nodeSelector map[string]string, hostDir string, hostPathType *v1.HostPathType) { newPod, err := f.ClientSet.CoreV1().Pods(f.Namespace.Name).Create(context.TODO(), newHostPathTypeTestPod(nodeSelector, hostDir, "/mnt/test", hostPathType), metav1.CreateOptions{}) framework.ExpectNoError(err) framework.ExpectNoError(e2epod.WaitTimeoutForPodRunningInNamespace(f.ClientSet, newPod.Name, newPod.Namespace, framework.PodStartShortTimeout)) f.ClientSet.CoreV1().Pods(f.Namespace.Name).Delete(context.TODO(), newPod.Name, *metav1.NewDeleteOptions(0)) } </ins>

If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with .