Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: When I change default hugepage size 2Mi to 1Gi and reboot, kubelet can't update node status What you expected to happen: kubelet can update node info and node status was ready How to reproduce it (as minimally and precisely as possible): change the node condition validate Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration:OS (e.g: ):Kernel (e.g. ):Install tools:Network plugin and version (if this is a network-related bug):Others:
/sig node
Hi, Ref. the error message, it looks like you have allocated hugepages for multiple sizes, . Can you run: to verify? If more than one of the numbers is , that is the reason. This is currently being discussed in , and the reason for implementing the check is here: Imo. this should be supported, and it would only require removing an if statement in order to implement it. I see no reason to limit the node from allocating huge pages with more than one size. Because of how it is implemented, adding support for allowing a pod to request huge pages with multiple sizes isn't trivial, and will require some potentially breaking changes, but that is not relevant to this problem. /cc
Also, can you also provide the result of , that will list the supported huge page sizes (both when is set, and when it is not.).
Thanks ! Looks like your kernel only enables huge pages with the default size. My kernel 5.0+ enable all (atleast 2MiB and 1GiB on x8664) by default. When the node booted with defaulthugepagesz=1GB it "removed support" the 2MiB one, and becuse kubelet use patch on the node status when connecting to the api server, you ended up with both and , even tho 2Mi wast supported by your machine (the sw, not the hw). Even if the error is removed, this will result in a crash when kubelet tries to write to the 2MiB hugetlb cgroup files. I think using should work, otherwise the only current solution is to remove the node from the cluster, and rejoin afterwords. Ref. it looks like you have sorted out the problem, but I still think this is a valid issue, so don't close it. The correct behaviour for kubelet is to remove unsupported sizes, and only allow use of the supported ones.
"Imo. this should be supported, and it would only require removing an if statement in order to implement it. I see no reason to limit the node from allocating huge pages with more than one size." Could you fix this one as well? We use 2M pages only for pods but we need 1G pages for system processes. So it was little surprising when k8s doesn't start because two different pages were configured no matter they are really used or not. Thanks!
I am hitting the same, I used tuned to change the default size from 2Mi to 1Gi, i hit the same problem, after the reboot node is NotReady and it cannot rejoin , giving the error: Node "test-worker-0" is invalid: [status.capacity.hugepages-2Mi: Invalid value: resource.Quantity{i:resource.int64Amount{value:, scale:0}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:"20Mi", Format:"BinarySI"}: may not have pre-allocated hugepages for multiple page sizes, status.allocatable.hugepages-2Mi: Invalid value: resource.Quantity{i:resource.int64Amount{value:, scale:0}, My system just reports information for 1Gi, not 2Mi anymore. Removing the node and rebooting fixed it, the node rejoined again reporting the right capacity. But this is not a solution for us, changing sizes should be supported.
Hitting same K8s installed with my ubuntu 18 enabled 2Mi hugepage by default. As I change my master node's hugepage to 1Gi, this happened. I have only one master, so I have to rebuild the k8s cluster.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
I'm a little confused. when i used tuned to change the hugepagesz from "defaulthugepagesz=2M hugepagesz=2M hugepages=8192" to " defaulthugepagesz=1G hugepagesz=1G hugepages=16", i hit the same problem, after the reboot node is NotReadybut when i used tuned to change the hugepagesz from " defaulthugepagesz=1G hugepagesz=1G hugepages=16" to "default_hugepagesz=2M hugepagesz=2M hugepages=8192", after the rebootkubelet status can still keep Ready. why?
Hi This will be fixed in , but it is still not merged. Hoping to get it into next release. In order to solve the issue (and use 1GiB huge pages) you have to delete the node in kubernetes via , and join the node again.
Thanks
Thanks
I think this is already fixed by can you confirm?
Hi WIth this would not cause a crash, but is still required in order to fix this properly.
I am facing the same issue. After changing the hugepage setting to . I stopped the kubelet process on the worker node and removed the node from the cluster using command and rebooted it. But after it comes back online and kubelet starts, it struggles to join the cluster . Do I have to run the join command again ?

"k8s.io/apimachinery/pkg/api/resource" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/types" <ins> "k8s.io/apimachinery/pkg/util/sets" </ins> cloudprovider "k8s.io/cloud-provider" cloudproviderapi "k8s.io/cloud-provider/api" "k8s.io/klog"


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: When I change default hugepage size 2Mi to 1Gi and reboot, kubelet can't update node status What you expected to happen: kubelet can update node info and node status was ready How to reproduce it (as minimally and precisely as possible): change the node condition validate Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration:OS (e.g: ):Kernel (e.g. ):Install tools:Network plugin and version (if this is a network-related bug):Others:
/sig node
Hi, Ref. the error message, it looks like you have allocated hugepages for multiple sizes, . Can you run: to verify? If more than one of the numbers is , that is the reason. This is currently being discussed in , and the reason for implementing the check is here: Imo. this should be supported, and it would only require removing an if statement in order to implement it. I see no reason to limit the node from allocating huge pages with more than one size. Because of how it is implemented, adding support for allowing a pod to request huge pages with multiple sizes isn't trivial, and will require some potentially breaking changes, but that is not relevant to this problem. /cc
Also, can you also provide the result of , that will list the supported huge page sizes (both when is set, and when it is not.).
Thanks ! Looks like your kernel only enables huge pages with the default size. My kernel 5.0+ enable all (atleast 2MiB and 1GiB on x8664) by default. When the node booted with defaulthugepagesz=1GB it "removed support" the 2MiB one, and becuse kubelet use patch on the node status when connecting to the api server, you ended up with both and , even tho 2Mi wast supported by your machine (the sw, not the hw). Even if the error is removed, this will result in a crash when kubelet tries to write to the 2MiB hugetlb cgroup files. I think using should work, otherwise the only current solution is to remove the node from the cluster, and rejoin afterwords. Ref. it looks like you have sorted out the problem, but I still think this is a valid issue, so don't close it. The correct behaviour for kubelet is to remove unsupported sizes, and only allow use of the supported ones.
"Imo. this should be supported, and it would only require removing an if statement in order to implement it. I see no reason to limit the node from allocating huge pages with more than one size." Could you fix this one as well? We use 2M pages only for pods but we need 1G pages for system processes. So it was little surprising when k8s doesn't start because two different pages were configured no matter they are really used or not. Thanks!
I am hitting the same, I used tuned to change the default size from 2Mi to 1Gi, i hit the same problem, after the reboot node is NotReady and it cannot rejoin , giving the error: Node "test-worker-0" is invalid: [status.capacity.hugepages-2Mi: Invalid value: resource.Quantity{i:resource.int64Amount{value:, scale:0}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:"20Mi", Format:"BinarySI"}: may not have pre-allocated hugepages for multiple page sizes, status.allocatable.hugepages-2Mi: Invalid value: resource.Quantity{i:resource.int64Amount{value:, scale:0}, My system just reports information for 1Gi, not 2Mi anymore. Removing the node and rebooting fixed it, the node rejoined again reporting the right capacity. But this is not a solution for us, changing sizes should be supported.
Hitting same K8s installed with my ubuntu 18 enabled 2Mi hugepage by default. As I change my master node's hugepage to 1Gi, this happened. I have only one master, so I have to rebuild the k8s cluster.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
I'm a little confused. when i used tuned to change the hugepagesz from "defaulthugepagesz=2M hugepagesz=2M hugepages=8192" to " defaulthugepagesz=1G hugepagesz=1G hugepages=16", i hit the same problem, after the reboot node is NotReadybut when i used tuned to change the hugepagesz from " defaulthugepagesz=1G hugepagesz=1G hugepages=16" to "default_hugepagesz=2M hugepagesz=2M hugepages=8192", after the rebootkubelet status can still keep Ready. why?
Hi This will be fixed in , but it is still not merged. Hoping to get it into next release. In order to solve the issue (and use 1GiB huge pages) you have to delete the node in kubernetes via , and join the node again.
Thanks
Thanks
I think this is already fixed by can you confirm?
Hi WIth this would not cause a crash, but is still required in order to fix this properly.
I am facing the same issue. After changing the hugepage setting to . I stopped the kubelet process on the worker node and removed the node from the cluster using command and rebooted it. But after it comes back online and kubelet starts, it struggles to join the cluster . Do I have to run the join command again ?

requiresUpdate := kl.reconcileCMADAnnotationWithExistingNode(node, existingNode) requiresUpdate = kl.updateDefaultLabels(node, existingNode) || requiresUpdate requiresUpdate = kl.reconcileExtendedResource(node, existingNode) || requiresUpdate <ins> requiresUpdate = kl.reconcileHugePageResource(node, existingNode) || requiresUpdate </ins> if requiresUpdate { if _, _, err := nodeutil.PatchNodeStatus(kl.kubeClient.CoreV1(), types.NodeName(kl.nodeName), originalNode, existingNode); err != nil { klog.Errorf("Unable to reconcile node %q with API server: error updating node: %v", kl.nodeName, err)


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: When I change default hugepage size 2Mi to 1Gi and reboot, kubelet can't update node status What you expected to happen: kubelet can update node info and node status was ready How to reproduce it (as minimally and precisely as possible): change the node condition validate Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration:OS (e.g: ):Kernel (e.g. ):Install tools:Network plugin and version (if this is a network-related bug):Others:
/sig node
Hi, Ref. the error message, it looks like you have allocated hugepages for multiple sizes, . Can you run: to verify? If more than one of the numbers is , that is the reason. This is currently being discussed in , and the reason for implementing the check is here: Imo. this should be supported, and it would only require removing an if statement in order to implement it. I see no reason to limit the node from allocating huge pages with more than one size. Because of how it is implemented, adding support for allowing a pod to request huge pages with multiple sizes isn't trivial, and will require some potentially breaking changes, but that is not relevant to this problem. /cc
Also, can you also provide the result of , that will list the supported huge page sizes (both when is set, and when it is not.).
Thanks ! Looks like your kernel only enables huge pages with the default size. My kernel 5.0+ enable all (atleast 2MiB and 1GiB on x8664) by default. When the node booted with defaulthugepagesz=1GB it "removed support" the 2MiB one, and becuse kubelet use patch on the node status when connecting to the api server, you ended up with both and , even tho 2Mi wast supported by your machine (the sw, not the hw). Even if the error is removed, this will result in a crash when kubelet tries to write to the 2MiB hugetlb cgroup files. I think using should work, otherwise the only current solution is to remove the node from the cluster, and rejoin afterwords. Ref. it looks like you have sorted out the problem, but I still think this is a valid issue, so don't close it. The correct behaviour for kubelet is to remove unsupported sizes, and only allow use of the supported ones.
"Imo. this should be supported, and it would only require removing an if statement in order to implement it. I see no reason to limit the node from allocating huge pages with more than one size." Could you fix this one as well? We use 2M pages only for pods but we need 1G pages for system processes. So it was little surprising when k8s doesn't start because two different pages were configured no matter they are really used or not. Thanks!
I am hitting the same, I used tuned to change the default size from 2Mi to 1Gi, i hit the same problem, after the reboot node is NotReady and it cannot rejoin , giving the error: Node "test-worker-0" is invalid: [status.capacity.hugepages-2Mi: Invalid value: resource.Quantity{i:resource.int64Amount{value:, scale:0}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:"20Mi", Format:"BinarySI"}: may not have pre-allocated hugepages for multiple page sizes, status.allocatable.hugepages-2Mi: Invalid value: resource.Quantity{i:resource.int64Amount{value:, scale:0}, My system just reports information for 1Gi, not 2Mi anymore. Removing the node and rebooting fixed it, the node rejoined again reporting the right capacity. But this is not a solution for us, changing sizes should be supported.
Hitting same K8s installed with my ubuntu 18 enabled 2Mi hugepage by default. As I change my master node's hugepage to 1Gi, this happened. I have only one master, so I have to rebuild the k8s cluster.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
I'm a little confused. when i used tuned to change the hugepagesz from "defaulthugepagesz=2M hugepagesz=2M hugepages=8192" to " defaulthugepagesz=1G hugepagesz=1G hugepages=16", i hit the same problem, after the reboot node is NotReadybut when i used tuned to change the hugepagesz from " defaulthugepagesz=1G hugepagesz=1G hugepages=16" to "default_hugepagesz=2M hugepagesz=2M hugepages=8192", after the rebootkubelet status can still keep Ready. why?
Hi This will be fixed in , but it is still not merged. Hoping to get it into next release. In order to solve the issue (and use 1GiB huge pages) you have to delete the node in kubernetes via , and join the node again.
Thanks
Thanks
I think this is already fixed by can you confirm?
Hi WIth this would not cause a crash, but is still required in order to fix this properly.
I am facing the same issue. After changing the hugepage setting to . I stopped the kubelet process on the worker node and removed the node from the cluster using command and rebooted it. But after it comes back online and kubelet starts, it struggles to join the cluster . Do I have to run the join command again ?

return true } <ins> // reconcileHugePageResource will update huge page capacity for each page size and remove huge page sizes no longer supported func (kl *Kubelet) reconcileHugePageResource(initialNode, existingNode *v1.Node) bool { requiresUpdate := false supportedHugePageResources := sets.String{} for resourceName := range initialNode.Status.Capacity { if !v1helper.IsHugePageResourceName(resourceName) { continue } supportedHugePageResources.Insert(string(resourceName)) initialCapacity := initialNode.Status.Capacity[resourceName] initialAllocatable := initialNode.Status.Allocatable[resourceName] capacity, resourceIsSupported := existingNode.Status.Capacity[resourceName] allocatable := existingNode.Status.Allocatable[resourceName] // Add or update capacity if it the size was previously unsupported or has changed if !resourceIsSupported || capacity.Cmp(initialCapacity) != 0 { existingNode.Status.Capacity[resourceName] = initialCapacity.DeepCopy() requiresUpdate = true } // Add or update allocatable if it the size was previously unsupported or has changed if !resourceIsSupported || allocatable.Cmp(initialAllocatable) != 0 { existingNode.Status.Allocatable[resourceName] = initialAllocatable.DeepCopy() requiresUpdate = true } } for resourceName := range existingNode.Status.Capacity { if !v1helper.IsHugePageResourceName(resourceName) { continue } // If huge page size no longer is supported, we remove it from the node if !supportedHugePageResources.Has(string(resourceName)) { delete(existingNode.Status.Capacity, resourceName) delete(existingNode.Status.Allocatable, resourceName) klog.Infof("Removing now unsupported huge page resource named: %s", resourceName) requiresUpdate = true } } return requiresUpdate } </ins> // Zeros out extended resource capacity during reconciliation. func (kl *Kubelet) reconcileExtendedResource(initialNode, node *v1.Node) bool { requiresUpdate := false


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: When I change default hugepage size 2Mi to 1Gi and reboot, kubelet can't update node status What you expected to happen: kubelet can update node info and node status was ready How to reproduce it (as minimally and precisely as possible): change the node condition validate Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration:OS (e.g: ):Kernel (e.g. ):Install tools:Network plugin and version (if this is a network-related bug):Others:
/sig node
Hi, Ref. the error message, it looks like you have allocated hugepages for multiple sizes, . Can you run: to verify? If more than one of the numbers is , that is the reason. This is currently being discussed in , and the reason for implementing the check is here: Imo. this should be supported, and it would only require removing an if statement in order to implement it. I see no reason to limit the node from allocating huge pages with more than one size. Because of how it is implemented, adding support for allowing a pod to request huge pages with multiple sizes isn't trivial, and will require some potentially breaking changes, but that is not relevant to this problem. /cc
Also, can you also provide the result of , that will list the supported huge page sizes (both when is set, and when it is not.).
Thanks ! Looks like your kernel only enables huge pages with the default size. My kernel 5.0+ enable all (atleast 2MiB and 1GiB on x8664) by default. When the node booted with defaulthugepagesz=1GB it "removed support" the 2MiB one, and becuse kubelet use patch on the node status when connecting to the api server, you ended up with both and , even tho 2Mi wast supported by your machine (the sw, not the hw). Even if the error is removed, this will result in a crash when kubelet tries to write to the 2MiB hugetlb cgroup files. I think using should work, otherwise the only current solution is to remove the node from the cluster, and rejoin afterwords. Ref. it looks like you have sorted out the problem, but I still think this is a valid issue, so don't close it. The correct behaviour for kubelet is to remove unsupported sizes, and only allow use of the supported ones.
"Imo. this should be supported, and it would only require removing an if statement in order to implement it. I see no reason to limit the node from allocating huge pages with more than one size." Could you fix this one as well? We use 2M pages only for pods but we need 1G pages for system processes. So it was little surprising when k8s doesn't start because two different pages were configured no matter they are really used or not. Thanks!
I am hitting the same, I used tuned to change the default size from 2Mi to 1Gi, i hit the same problem, after the reboot node is NotReady and it cannot rejoin , giving the error: Node "test-worker-0" is invalid: [status.capacity.hugepages-2Mi: Invalid value: resource.Quantity{i:resource.int64Amount{value:, scale:0}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:"20Mi", Format:"BinarySI"}: may not have pre-allocated hugepages for multiple page sizes, status.allocatable.hugepages-2Mi: Invalid value: resource.Quantity{i:resource.int64Amount{value:, scale:0}, My system just reports information for 1Gi, not 2Mi anymore. Removing the node and rebooting fixed it, the node rejoined again reporting the right capacity. But this is not a solution for us, changing sizes should be supported.
Hitting same K8s installed with my ubuntu 18 enabled 2Mi hugepage by default. As I change my master node's hugepage to 1Gi, this happened. I have only one master, so I have to rebuild the k8s cluster.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
I'm a little confused. when i used tuned to change the hugepagesz from "defaulthugepagesz=2M hugepagesz=2M hugepages=8192" to " defaulthugepagesz=1G hugepagesz=1G hugepages=16", i hit the same problem, after the reboot node is NotReadybut when i used tuned to change the hugepagesz from " defaulthugepagesz=1G hugepagesz=1G hugepages=16" to "default_hugepagesz=2M hugepagesz=2M hugepages=8192", after the rebootkubelet status can still keep Ready. why?
Hi This will be fixed in , but it is still not merged. Hoping to get it into next release. In order to solve the issue (and use 1GiB huge pages) you have to delete the node in kubernetes via , and join the node again.
Thanks
Thanks
I think this is already fixed by can you confirm?
Hi WIth this would not cause a crash, but is still required in order to fix this properly.
I am facing the same issue. After changing the hugepage setting to . I stopped the kubelet process on the worker node and removed the node from the cluster using command and rebooted it. But after it comes back online and kubelet starts, it struggles to join the cluster . Do I have to run the join command again ?

} } <ins> func TestReconcileHugePageResource(t *testing.T) { testKubelet := newTestKubelet(t, false /* controllerAttachDetachEnabled */) hugePageResourceName64Ki := v1.ResourceName("hugepages-64Ki") hugePageResourceName2Mi := v1.ResourceName("hugepages-2Mi") hugePageResourceName1Gi := v1.ResourceName("hugepages-1Gi") cases := []struct { name         string testKubelet  *TestKubelet initialNode  *v1.Node existingNode *v1.Node expectedNode *v1.Node needsUpdate  bool }{ { name:        "no update needed when all huge page resources are similar", testKubelet: testKubelet, needsUpdate: false, initialNode: &v1.Node{ Status: v1.NodeStatus{ Capacity: v1.ResourceList{ v1.ResourceCPU:              *resource.NewMilliQuantity(2000, resource.DecimalSI), v1.ResourceMemory:           *resource.NewQuantity(10e9, resource.BinarySI), v1.ResourceEphemeralStorage: *resource.NewQuantity(5000, resource.BinarySI), hugePageResourceName2Mi:     resource.MustParse("100Mi"), hugePageResourceName64Ki:    *resource.NewQuantity(0, resource.BinarySI), }, Allocatable: v1.ResourceList{ v1.ResourceCPU:              *resource.NewMilliQuantity(2000, resource.DecimalSI), v1.ResourceMemory:           *resource.NewQuantity(10e9, resource.BinarySI), v1.ResourceEphemeralStorage: *resource.NewQuantity(5000, resource.BinarySI), hugePageResourceName2Mi:     resource.MustParse("100Mi"), hugePageResourceName64Ki:    *resource.NewQuantity(0, resource.BinarySI), }, }, }, existingNode: &v1.Node{ Status: v1.NodeStatus{ Capacity: v1.ResourceList{ v1.ResourceCPU:              *resource.NewMilliQuantity(2000, resource.DecimalSI), v1.ResourceMemory:           *resource.NewQuantity(10e9, resource.BinarySI), v1.ResourceEphemeralStorage: *resource.NewQuantity(5000, resource.BinarySI), hugePageResourceName2Mi:     resource.MustParse("100Mi"), hugePageResourceName64Ki:    *resource.NewQuantity(0, resource.BinarySI), }, Allocatable: v1.ResourceList{ v1.ResourceCPU:              *resource.NewMilliQuantity(2000, resource.DecimalSI), v1.ResourceMemory:           *resource.NewQuantity(10e9, resource.BinarySI), v1.ResourceEphemeralStorage: *resource.NewQuantity(5000, resource.BinarySI), hugePageResourceName2Mi:     resource.MustParse("100Mi"), hugePageResourceName64Ki:    *resource.NewQuantity(0, resource.BinarySI), }, }, }, expectedNode: &v1.Node{ Status: v1.NodeStatus{ Capacity: v1.ResourceList{ v1.ResourceCPU:              *resource.NewMilliQuantity(2000, resource.DecimalSI), v1.ResourceMemory:           *resource.NewQuantity(10e9, resource.BinarySI), v1.ResourceEphemeralStorage: *resource.NewQuantity(5000, resource.BinarySI), hugePageResourceName2Mi:     resource.MustParse("100Mi"), hugePageResourceName64Ki:    *resource.NewQuantity(0, resource.BinarySI), }, Allocatable: v1.ResourceList{ v1.ResourceCPU:              *resource.NewMilliQuantity(2000, resource.DecimalSI), v1.ResourceMemory:           *resource.NewQuantity(10e9, resource.BinarySI), v1.ResourceEphemeralStorage: *resource.NewQuantity(5000, resource.BinarySI), hugePageResourceName2Mi:     resource.MustParse("100Mi"), hugePageResourceName64Ki:    *resource.NewQuantity(0, resource.BinarySI), }, }, }, }, { name:        "update needed when new huge page resources is supported", testKubelet: testKubelet, needsUpdate: true, initialNode: &v1.Node{ Status: v1.NodeStatus{ Capacity: v1.ResourceList{ v1.ResourceCPU:              *resource.NewMilliQuantity(2000, resource.DecimalSI), v1.ResourceMemory:           *resource.NewQuantity(10e9, resource.BinarySI), v1.ResourceEphemeralStorage: *resource.NewQuantity(5000, resource.BinarySI), hugePageResourceName2Mi:     *resource.NewQuantity(0, resource.BinarySI), hugePageResourceName1Gi:     resource.MustParse("2Gi"), }, Allocatable: v1.ResourceList{ v1.ResourceCPU:              *resource.NewMilliQuantity(2000, resource.DecimalSI), v1.ResourceMemory:           *resource.NewQuantity(10e9, resource.BinarySI), v1.ResourceEphemeralStorage: *resource.NewQuantity(5000, resource.BinarySI), hugePageResourceName2Mi:     *resource.NewQuantity(0, resource.BinarySI), hugePageResourceName1Gi:     resource.MustParse("2Gi"), }, }, }, existingNode: &v1.Node{ Status: v1.NodeStatus{ Capacity: v1.ResourceList{ v1.ResourceCPU:              *resource.NewMilliQuantity(2000, resource.DecimalSI), v1.ResourceMemory:           *resource.NewQuantity(10e9, resource.BinarySI), v1.ResourceEphemeralStorage: *resource.NewQuantity(5000, resource.BinarySI), hugePageResourceName2Mi:     resource.MustParse("100Mi"), }, Allocatable: v1.ResourceList{ v1.ResourceCPU:              *resource.NewMilliQuantity(2000, resource.DecimalSI), v1.ResourceMemory:           *resource.NewQuantity(10e9, resource.BinarySI), v1.ResourceEphemeralStorage: *resource.NewQuantity(5000, resource.BinarySI), hugePageResourceName2Mi:     resource.MustParse("100Mi"), }, }, }, expectedNode: &v1.Node{ Status: v1.NodeStatus{ Capacity: v1.ResourceList{ v1.ResourceCPU:              *resource.NewMilliQuantity(2000, resource.DecimalSI), v1.ResourceMemory:           *resource.NewQuantity(10e9, resource.BinarySI), v1.ResourceEphemeralStorage: *resource.NewQuantity(5000, resource.BinarySI), hugePageResourceName2Mi:     *resource.NewQuantity(0, resource.BinarySI), hugePageResourceName1Gi:     resource.MustParse("2Gi"), }, Allocatable: v1.ResourceList{ v1.ResourceCPU:              *resource.NewMilliQuantity(2000, resource.DecimalSI), v1.ResourceMemory:           *resource.NewQuantity(10e9, resource.BinarySI), v1.ResourceEphemeralStorage: *resource.NewQuantity(5000, resource.BinarySI), hugePageResourceName2Mi:     *resource.NewQuantity(0, resource.BinarySI), hugePageResourceName1Gi:     resource.MustParse("2Gi"), }, }, }, }, { name:        "update needed when huge page resource quantity has changed", testKubelet: testKubelet, needsUpdate: true, initialNode: &v1.Node{ Status: v1.NodeStatus{ Capacity: v1.ResourceList{ v1.ResourceCPU:              *resource.NewMilliQuantity(2000, resource.DecimalSI), v1.ResourceMemory:           *resource.NewQuantity(10e9, resource.BinarySI), v1.ResourceEphemeralStorage: *resource.NewQuantity(5000, resource.BinarySI), hugePageResourceName1Gi:     resource.MustParse("4Gi"), }, Allocatable: v1.ResourceList{ v1.ResourceCPU:              *resource.NewMilliQuantity(2000, resource.DecimalSI), v1.ResourceMemory:           *resource.NewQuantity(10e9, resource.BinarySI), v1.ResourceEphemeralStorage: *resource.NewQuantity(5000, resource.BinarySI), hugePageResourceName1Gi:     resource.MustParse("4Gi"), }, }, }, existingNode: &v1.Node{ Status: v1.NodeStatus{ Capacity: v1.ResourceList{ v1.ResourceCPU:              *resource.NewMilliQuantity(2000, resource.DecimalSI), v1.ResourceMemory:           *resource.NewQuantity(10e9, resource.BinarySI), v1.ResourceEphemeralStorage: *resource.NewQuantity(5000, resource.BinarySI), hugePageResourceName1Gi:     resource.MustParse("2Gi"), }, Allocatable: v1.ResourceList{ v1.ResourceCPU:              *resource.NewMilliQuantity(2000, resource.DecimalSI), v1.ResourceMemory:           *resource.NewQuantity(10e9, resource.BinarySI), v1.ResourceEphemeralStorage: *resource.NewQuantity(5000, resource.BinarySI), hugePageResourceName1Gi:     resource.MustParse("2Gi"), }, }, }, expectedNode: &v1.Node{ Status: v1.NodeStatus{ Capacity: v1.ResourceList{ v1.ResourceCPU:              *resource.NewMilliQuantity(2000, resource.DecimalSI), v1.ResourceMemory:           *resource.NewQuantity(10e9, resource.BinarySI), v1.ResourceEphemeralStorage: *resource.NewQuantity(5000, resource.BinarySI), hugePageResourceName1Gi:     resource.MustParse("4Gi"), }, Allocatable: v1.ResourceList{ v1.ResourceCPU:              *resource.NewMilliQuantity(2000, resource.DecimalSI), v1.ResourceMemory:           *resource.NewQuantity(10e9, resource.BinarySI), v1.ResourceEphemeralStorage: *resource.NewQuantity(5000, resource.BinarySI), hugePageResourceName1Gi:     resource.MustParse("4Gi"), }, }, }, }, { name:        "update needed when a huge page resources is no longer supported", testKubelet: testKubelet, needsUpdate: true, initialNode: &v1.Node{ Status: v1.NodeStatus{ Capacity: v1.ResourceList{ v1.ResourceCPU:              *resource.NewMilliQuantity(2000, resource.DecimalSI), v1.ResourceMemory:           *resource.NewQuantity(10e9, resource.BinarySI), v1.ResourceEphemeralStorage: *resource.NewQuantity(5000, resource.BinarySI), hugePageResourceName1Gi:     resource.MustParse("2Gi"), }, Allocatable: v1.ResourceList{ v1.ResourceCPU:              *resource.NewMilliQuantity(2000, resource.DecimalSI), v1.ResourceMemory:           *resource.NewQuantity(10e9, resource.BinarySI), v1.ResourceEphemeralStorage: *resource.NewQuantity(5000, resource.BinarySI), hugePageResourceName1Gi:     resource.MustParse("2Gi"), }, }, }, existingNode: &v1.Node{ Status: v1.NodeStatus{ Capacity: v1.ResourceList{ v1.ResourceCPU:              *resource.NewMilliQuantity(2000, resource.DecimalSI), v1.ResourceMemory:           *resource.NewQuantity(10e9, resource.BinarySI), v1.ResourceEphemeralStorage: *resource.NewQuantity(5000, resource.BinarySI), hugePageResourceName2Mi:     *resource.NewQuantity(0, resource.BinarySI), hugePageResourceName1Gi:     resource.MustParse("2Gi"), }, Allocatable: v1.ResourceList{ v1.ResourceCPU:              *resource.NewMilliQuantity(2000, resource.DecimalSI), v1.ResourceMemory:           *resource.NewQuantity(10e9, resource.BinarySI), v1.ResourceEphemeralStorage: *resource.NewQuantity(5000, resource.BinarySI), hugePageResourceName2Mi:     *resource.NewQuantity(0, resource.BinarySI), hugePageResourceName1Gi:     resource.MustParse("2Gi"), }, }, }, expectedNode: &v1.Node{ Status: v1.NodeStatus{ Capacity: v1.ResourceList{ v1.ResourceCPU:              *resource.NewMilliQuantity(2000, resource.DecimalSI), v1.ResourceMemory:           *resource.NewQuantity(10e9, resource.BinarySI), v1.ResourceEphemeralStorage: *resource.NewQuantity(5000, resource.BinarySI), hugePageResourceName1Gi:     resource.MustParse("2Gi"), }, Allocatable: v1.ResourceList{ v1.ResourceCPU:              *resource.NewMilliQuantity(2000, resource.DecimalSI), v1.ResourceMemory:           *resource.NewQuantity(10e9, resource.BinarySI), v1.ResourceEphemeralStorage: *resource.NewQuantity(5000, resource.BinarySI), hugePageResourceName1Gi:     resource.MustParse("2Gi"), }, }, }, }, } for _, tc := range cases { t.Run(tc.name, func(T *testing.T) { defer testKubelet.Cleanup() kubelet := testKubelet.kubelet needsUpdate := kubelet.reconcileHugePageResource(tc.initialNode, tc.existingNode) assert.Equal(t, tc.needsUpdate, needsUpdate, tc.name) assert.Equal(t, tc.expectedNode, tc.existingNode, tc.name) }) } } </ins> func TestReconcileExtendedResource(t *testing.T) { testKubelet := newTestKubelet(t, false /* controllerAttachDetachEnabled */) testKubelet.kubelet.kubeClient = nil // ensure only the heartbeat client is used


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: When I change default hugepage size 2Mi to 1Gi and reboot, kubelet can't update node status What you expected to happen: kubelet can update node info and node status was ready How to reproduce it (as minimally and precisely as possible): change the node condition validate Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration:OS (e.g: ):Kernel (e.g. ):Install tools:Network plugin and version (if this is a network-related bug):Others:
/sig node
Hi, Ref. the error message, it looks like you have allocated hugepages for multiple sizes, . Can you run: to verify? If more than one of the numbers is , that is the reason. This is currently being discussed in , and the reason for implementing the check is here: Imo. this should be supported, and it would only require removing an if statement in order to implement it. I see no reason to limit the node from allocating huge pages with more than one size. Because of how it is implemented, adding support for allowing a pod to request huge pages with multiple sizes isn't trivial, and will require some potentially breaking changes, but that is not relevant to this problem. /cc
Also, can you also provide the result of , that will list the supported huge page sizes (both when is set, and when it is not.).
Thanks ! Looks like your kernel only enables huge pages with the default size. My kernel 5.0+ enable all (atleast 2MiB and 1GiB on x8664) by default. When the node booted with defaulthugepagesz=1GB it "removed support" the 2MiB one, and becuse kubelet use patch on the node status when connecting to the api server, you ended up with both and , even tho 2Mi wast supported by your machine (the sw, not the hw). Even if the error is removed, this will result in a crash when kubelet tries to write to the 2MiB hugetlb cgroup files. I think using should work, otherwise the only current solution is to remove the node from the cluster, and rejoin afterwords. Ref. it looks like you have sorted out the problem, but I still think this is a valid issue, so don't close it. The correct behaviour for kubelet is to remove unsupported sizes, and only allow use of the supported ones.
"Imo. this should be supported, and it would only require removing an if statement in order to implement it. I see no reason to limit the node from allocating huge pages with more than one size." Could you fix this one as well? We use 2M pages only for pods but we need 1G pages for system processes. So it was little surprising when k8s doesn't start because two different pages were configured no matter they are really used or not. Thanks!
I am hitting the same, I used tuned to change the default size from 2Mi to 1Gi, i hit the same problem, after the reboot node is NotReady and it cannot rejoin , giving the error: Node "test-worker-0" is invalid: [status.capacity.hugepages-2Mi: Invalid value: resource.Quantity{i:resource.int64Amount{value:, scale:0}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:"20Mi", Format:"BinarySI"}: may not have pre-allocated hugepages for multiple page sizes, status.allocatable.hugepages-2Mi: Invalid value: resource.Quantity{i:resource.int64Amount{value:, scale:0}, My system just reports information for 1Gi, not 2Mi anymore. Removing the node and rebooting fixed it, the node rejoined again reporting the right capacity. But this is not a solution for us, changing sizes should be supported.
Hitting same K8s installed with my ubuntu 18 enabled 2Mi hugepage by default. As I change my master node's hugepage to 1Gi, this happened. I have only one master, so I have to rebuild the k8s cluster.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
I'm a little confused. when i used tuned to change the hugepagesz from "defaulthugepagesz=2M hugepagesz=2M hugepages=8192" to " defaulthugepagesz=1G hugepagesz=1G hugepages=16", i hit the same problem, after the reboot node is NotReadybut when i used tuned to change the hugepagesz from " defaulthugepagesz=1G hugepagesz=1G hugepages=16" to "default_hugepagesz=2M hugepagesz=2M hugepages=8192", after the rebootkubelet status can still keep Ready. why?
Hi This will be fixed in , but it is still not merged. Hoping to get it into next release. In order to solve the issue (and use 1GiB huge pages) you have to delete the node in kubernetes via , and join the node again.
Thanks
Thanks
I think this is already fixed by can you confirm?
Hi WIth this would not cause a crash, but is still required in order to fix this properly.
I am facing the same issue. After changing the hugepage setting to . I stopped the kubelet process on the worker node and removed the node from the cluster using command and rebooted it. But after it comes back online and kubelet starts, it struggles to join the cluster . Do I have to run the join command again ?

v1 "k8s.io/api/core/v1" "k8s.io/apimachinery/pkg/api/resource" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" <ins> "k8s.io/apimachinery/pkg/types" </ins> "k8s.io/apimachinery/pkg/util/uuid" "k8s.io/kubernetes/pkg/kubelet/cm" "k8s.io/kubernetes/test/e2e/framework"


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: When I change default hugepage size 2Mi to 1Gi and reboot, kubelet can't update node status What you expected to happen: kubelet can update node info and node status was ready How to reproduce it (as minimally and precisely as possible): change the node condition validate Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration:OS (e.g: ):Kernel (e.g. ):Install tools:Network plugin and version (if this is a network-related bug):Others:
/sig node
Hi, Ref. the error message, it looks like you have allocated hugepages for multiple sizes, . Can you run: to verify? If more than one of the numbers is , that is the reason. This is currently being discussed in , and the reason for implementing the check is here: Imo. this should be supported, and it would only require removing an if statement in order to implement it. I see no reason to limit the node from allocating huge pages with more than one size. Because of how it is implemented, adding support for allowing a pod to request huge pages with multiple sizes isn't trivial, and will require some potentially breaking changes, but that is not relevant to this problem. /cc
Also, can you also provide the result of , that will list the supported huge page sizes (both when is set, and when it is not.).
Thanks ! Looks like your kernel only enables huge pages with the default size. My kernel 5.0+ enable all (atleast 2MiB and 1GiB on x8664) by default. When the node booted with defaulthugepagesz=1GB it "removed support" the 2MiB one, and becuse kubelet use patch on the node status when connecting to the api server, you ended up with both and , even tho 2Mi wast supported by your machine (the sw, not the hw). Even if the error is removed, this will result in a crash when kubelet tries to write to the 2MiB hugetlb cgroup files. I think using should work, otherwise the only current solution is to remove the node from the cluster, and rejoin afterwords. Ref. it looks like you have sorted out the problem, but I still think this is a valid issue, so don't close it. The correct behaviour for kubelet is to remove unsupported sizes, and only allow use of the supported ones.
"Imo. this should be supported, and it would only require removing an if statement in order to implement it. I see no reason to limit the node from allocating huge pages with more than one size." Could you fix this one as well? We use 2M pages only for pods but we need 1G pages for system processes. So it was little surprising when k8s doesn't start because two different pages were configured no matter they are really used or not. Thanks!
I am hitting the same, I used tuned to change the default size from 2Mi to 1Gi, i hit the same problem, after the reboot node is NotReady and it cannot rejoin , giving the error: Node "test-worker-0" is invalid: [status.capacity.hugepages-2Mi: Invalid value: resource.Quantity{i:resource.int64Amount{value:, scale:0}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:"20Mi", Format:"BinarySI"}: may not have pre-allocated hugepages for multiple page sizes, status.allocatable.hugepages-2Mi: Invalid value: resource.Quantity{i:resource.int64Amount{value:, scale:0}, My system just reports information for 1Gi, not 2Mi anymore. Removing the node and rebooting fixed it, the node rejoined again reporting the right capacity. But this is not a solution for us, changing sizes should be supported.
Hitting same K8s installed with my ubuntu 18 enabled 2Mi hugepage by default. As I change my master node's hugepage to 1Gi, this happened. I have only one master, so I have to rebuild the k8s cluster.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
I'm a little confused. when i used tuned to change the hugepagesz from "defaulthugepagesz=2M hugepagesz=2M hugepages=8192" to " defaulthugepagesz=1G hugepagesz=1G hugepages=16", i hit the same problem, after the reboot node is NotReadybut when i used tuned to change the hugepagesz from " defaulthugepagesz=1G hugepagesz=1G hugepages=16" to "default_hugepagesz=2M hugepagesz=2M hugepages=8192", after the rebootkubelet status can still keep Ready. why?
Hi This will be fixed in , but it is still not merged. Hoping to get it into next release. In order to solve the issue (and use 1GiB huge pages) you have to delete the node in kubernetes via , and join the node again.
Thanks
Thanks
I think this is already fixed by can you confirm?
Hi WIth this would not cause a crash, but is still required in order to fix this properly.
I am facing the same issue. After changing the hugepage setting to . I stopped the kubelet process on the worker node and removed the node from the cluster using command and rebooted it. But after it comes back online and kubelet starts, it struggles to join the cluster . Do I have to run the join command again ?

err := e2epod.WaitForPodSuccessInNamespace(f.ClientSet, verifyPod.Name, f.Namespace.Name) framework.ExpectNoError(err) }) <ins> ginkgo.It("should add resources for new huge page sizes on kubelet restart", func() { ginkgo.By("Stopping kubelet") startKubelet := stopKubelet() ginkgo.By(`Patching away support for hugepage resource "hugepages-2Mi"`) patch := []byte(`[{"op": "remove", "path": "/status/capacity/hugepages-2Mi"}, {"op": "remove", "path": "/status/allocatable/hugepages-2Mi"}]`) result := f.ClientSet.CoreV1().RESTClient().Patch(types.JSONPatchType).Resource("nodes").Name(framework.TestContext.NodeName).SubResource("status").Body(patch).Do(context.TODO()) framework.ExpectNoError(result.Error(), "while patching") ginkgo.By("Starting kubelet again") startKubelet() ginkgo.By("verifying that the hugepages-2Mi resource is present") gomega.Eventually(func() bool { node, err := f.ClientSet.CoreV1().Nodes().Get(context.TODO(), framework.TestContext.NodeName, metav1.GetOptions{}) framework.ExpectNoError(err, "while getting node status") _, isPresent := node.Status.Capacity["hugepages-2Mi"] return isPresent }, 30*time.Second, framework.Poll).Should(gomega.Equal(true)) }) </ins> } // Serial because the test updates kubelet configuration. var _ = SIGDescribe("HugePages [Serial] [Feature:HugePages][NodeFeature:HugePages]", func() { f := framework.NewDefaultFramework("hugepages-test") <ins> ginkgo.It("should remove resources for huge page sizes no longer supported", func() { ginkgo.By("mimicking support for 9Mi of 3Mi huge page memory by patching the node status") patch := []byte(`[{"op": "add", "path": "/status/capacity/hugepages-3Mi", "value": "9Mi"}, {"op": "add", "path": "/status/allocatable/hugepages-3Mi", "value": "9Mi"}]`) result := f.ClientSet.CoreV1().RESTClient().Patch(types.JSONPatchType).Resource("nodes").Name(framework.TestContext.NodeName).SubResource("status").Body(patch).Do(context.TODO()) framework.ExpectNoError(result.Error(), "while patching") node, err := f.ClientSet.CoreV1().Nodes().Get(context.TODO(), framework.TestContext.NodeName, metav1.GetOptions{}) framework.ExpectNoError(err, "while getting node status") ginkgo.By("Verifying that the node now supports huge pages with size 3Mi") value, ok := node.Status.Capacity["hugepages-3Mi"] framework.ExpectEqual(ok, true, "capacity should contain resouce hugepages-3Mi") framework.ExpectEqual(value.String(), "9Mi", "huge pages with size 3Mi should be supported") ginkgo.By("restarting the node and verifying that huge pages with size 3Mi are not supported") restartKubelet() ginkgo.By("verifying that the hugepages-3Mi resource no longer is present") gomega.Eventually(func() bool { node, err = f.ClientSet.CoreV1().Nodes().Get(context.TODO(), framework.TestContext.NodeName, metav1.GetOptions{}) framework.ExpectNoError(err, "while getting node status") _, isPresent := node.Status.Capacity["hugepages-3Mi"] return isPresent }, 30*time.Second, framework.Poll).Should(gomega.Equal(false)) }) </ins> ginkgo.Context("With config updated with hugepages feature enabled", func() { ginkgo.BeforeEach(func() { ginkgo.By("verifying hugepages are supported")


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: When I change default hugepage size 2Mi to 1Gi and reboot, kubelet can't update node status What you expected to happen: kubelet can update node info and node status was ready How to reproduce it (as minimally and precisely as possible): change the node condition validate Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration:OS (e.g: ):Kernel (e.g. ):Install tools:Network plugin and version (if this is a network-related bug):Others:
/sig node
Hi, Ref. the error message, it looks like you have allocated hugepages for multiple sizes, . Can you run: to verify? If more than one of the numbers is , that is the reason. This is currently being discussed in , and the reason for implementing the check is here: Imo. this should be supported, and it would only require removing an if statement in order to implement it. I see no reason to limit the node from allocating huge pages with more than one size. Because of how it is implemented, adding support for allowing a pod to request huge pages with multiple sizes isn't trivial, and will require some potentially breaking changes, but that is not relevant to this problem. /cc
Also, can you also provide the result of , that will list the supported huge page sizes (both when is set, and when it is not.).
Thanks ! Looks like your kernel only enables huge pages with the default size. My kernel 5.0+ enable all (atleast 2MiB and 1GiB on x8664) by default. When the node booted with defaulthugepagesz=1GB it "removed support" the 2MiB one, and becuse kubelet use patch on the node status when connecting to the api server, you ended up with both and , even tho 2Mi wast supported by your machine (the sw, not the hw). Even if the error is removed, this will result in a crash when kubelet tries to write to the 2MiB hugetlb cgroup files. I think using should work, otherwise the only current solution is to remove the node from the cluster, and rejoin afterwords. Ref. it looks like you have sorted out the problem, but I still think this is a valid issue, so don't close it. The correct behaviour for kubelet is to remove unsupported sizes, and only allow use of the supported ones.
"Imo. this should be supported, and it would only require removing an if statement in order to implement it. I see no reason to limit the node from allocating huge pages with more than one size." Could you fix this one as well? We use 2M pages only for pods but we need 1G pages for system processes. So it was little surprising when k8s doesn't start because two different pages were configured no matter they are really used or not. Thanks!
I am hitting the same, I used tuned to change the default size from 2Mi to 1Gi, i hit the same problem, after the reboot node is NotReady and it cannot rejoin , giving the error: Node "test-worker-0" is invalid: [status.capacity.hugepages-2Mi: Invalid value: resource.Quantity{i:resource.int64Amount{value:, scale:0}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:"20Mi", Format:"BinarySI"}: may not have pre-allocated hugepages for multiple page sizes, status.allocatable.hugepages-2Mi: Invalid value: resource.Quantity{i:resource.int64Amount{value:, scale:0}, My system just reports information for 1Gi, not 2Mi anymore. Removing the node and rebooting fixed it, the node rejoined again reporting the right capacity. But this is not a solution for us, changing sizes should be supported.
Hitting same K8s installed with my ubuntu 18 enabled 2Mi hugepage by default. As I change my master node's hugepage to 1Gi, this happened. I have only one master, so I have to rebuild the k8s cluster.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
I'm a little confused. when i used tuned to change the hugepagesz from "defaulthugepagesz=2M hugepagesz=2M hugepages=8192" to " defaulthugepagesz=1G hugepagesz=1G hugepages=16", i hit the same problem, after the reboot node is NotReadybut when i used tuned to change the hugepagesz from " defaulthugepagesz=1G hugepagesz=1G hugepages=16" to "default_hugepagesz=2M hugepagesz=2M hugepages=8192", after the rebootkubelet status can still keep Ready. why?
Hi This will be fixed in , but it is still not merged. Hoping to get it into next release. In order to solve the issue (and use 1GiB huge pages) you have to delete the node in kubernetes via , and join the node again.
Thanks
Thanks
I think this is already fixed by can you confirm?
Hi WIth this would not cause a crash, but is still required in order to fix this properly.
I am facing the same issue. After changing the hugepage setting to . I stopped the kubelet process on the worker node and removed the node from the cluster using command and rebooted it. But after it comes back online and kubelet starts, it struggles to join the cluster . Do I have to run the join command again ?

} // TODO: Find a uniform way to deal with systemctl/initctl/service operations. #34494 <del> func restartKubelet() { </del> <ins> func findRunningKubletServiceName() string { </ins> stdout, err := exec.Command("sudo", "systemctl", "list-units", "kubelet*", "--state=running").CombinedOutput() framework.ExpectNoError(err) regex := regexp.MustCompile("(kubelet-w+)") matches := regex.FindStringSubmatch(string(stdout)) <del> framework.ExpectNotEqual(len(matches), 0) kube := matches[0] framework.Logf("Get running kubelet with systemctl: %v, %v", string(stdout), kube) stdout, err = exec.Command("sudo", "systemctl", "restart", kube).CombinedOutput() </del> <ins> framework.ExpectNotEqual(len(matches), 0, "Found more than one kubelet service running: %q", stdout) kubeletServiceName := matches[0] framework.Logf("Get running kubelet with systemctl: %v, %v", string(stdout), kubeletServiceName) return kubeletServiceName } func restartKubelet() { kubeletServiceName := findRunningKubletServiceName() stdout, err := exec.Command("sudo", "systemctl", "restart", kubeletServiceName).CombinedOutput() </ins> framework.ExpectNoError(err, "Failed to restart kubelet with systemctl: %v, %v", err, stdout) } <ins> // stopKubelet will kill the running kubelet, and returns a func that will restart the process again func stopKubelet() func() { kubeletServiceName := findRunningKubletServiceName() stdout, err := exec.Command("sudo", "systemctl", "kill", kubeletServiceName).CombinedOutput() framework.ExpectNoError(err, "Failed to stop kubelet with systemctl: %v, %v", err, stdout) return func() { stdout, err := exec.Command("sudo", "systemctl", "start", kubeletServiceName).CombinedOutput() framework.ExpectNoError(err, "Failed to restart kubelet with systemctl: %v, %v", err, stdout) } } </ins> func toCgroupFsName(cgroupName cm.CgroupName) string { if framework.TestContext.KubeletConfig.CgroupDriver == "systemd" { return cgroupName.ToSystemd()


Question:
/assign /cc /sig instrumentation
GitHub didn't allow me to assign the following users: serathius. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign /cc /sig instrumentation Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hey Looks like this test is passing on testgrid, https://k8s- Which would point to problem being related to specific environment. From your log looks like problem is specific to Macs. I will try to reproduce it to confirm it. My current ideas for solution are: Disabling test on Mac Instead of passing files as arguments, pass them in STDIN WDYT?
FWIW ARG_MAX can technically vary even on linux ...? STDIN SGTM

"//:all-srcs", ], outs = ["stable-metrics-list.yaml"], <del> cmd = "./$(locations :instrumentation) $(locations //:all-srcs) > $@", </del> <ins> cmd = "for loc in $(locations //:all-srcs); do echo $$loc; done | ./$(locations :instrumentation) - > $@", </ins> message = "Listing all stable metrics.", tools = [":instrumentation"], )


Question:
/assign /cc /sig instrumentation
GitHub didn't allow me to assign the following users: serathius. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign /cc /sig instrumentation Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hey Looks like this test is passing on testgrid, https://k8s- Which would point to problem being related to specific environment. From your log looks like problem is specific to Macs. I will try to reproduce it to confirm it. My current ideas for solution are: Disabling test on Mac Instead of passing files as arguments, pass them in STDIN WDYT?
FWIW ARG_MAX can technically vary even on linux ...? STDIN SGTM

package main import ( <ins> "bufio" </ins> "flag" "fmt" "go/ast"


Question:
/assign /cc /sig instrumentation
GitHub didn't allow me to assign the following users: serathius. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign /cc /sig instrumentation Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hey Looks like this test is passing on testgrid, https://k8s- Which would point to problem being related to specific environment. From your log looks like problem is specific to Macs. I will try to reproduce it to confirm it. My current ideas for solution are: Disabling test on Mac Instead of passing files as arguments, pass them in STDIN WDYT?
FWIW ARG_MAX can technically vary even on linux ...? STDIN SGTM

func main() { flag.Parse() if len(flag.Args()) < 1 { <del> fmt.Fprintf(os.Stderr, "USAGE: %s <DIR or FILE> [...]n", os.Args[0]) </del> <ins> fmt.Fprintf(os.Stderr, "USAGE: %s <DIR or FILE or '-'> [...]n", os.Args[0]) </ins> os.Exit(64) } stableMetrics := []metric{} errors := []error{} <ins> addStdin := false </ins> for _, arg := range flag.Args() { <ins> if arg == "-" { addStdin = true continue } </ins> ms, es := searchPathForStableMetrics(arg) stableMetrics = append(stableMetrics, ms...) errors = append(errors, es...) } <ins> if addStdin { scanner := bufio.NewScanner(os.Stdin) scanner.Split(bufio.ScanLines) for scanner.Scan() { arg := scanner.Text() ms, es := searchPathForStableMetrics(arg) stableMetrics = append(stableMetrics, ms...) errors = append(errors, es...) } } </ins> for _, err := range errors { fmt.Fprintf(os.Stderr, "%sn", err) }


Question:
<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: ci-kubernetes-e2e-gci-gce-alpha-enabled-default Which test(s) are failing: CSI hostpath-v0 tests Since when has it been failing: 8/8 Testgrid link: https://k8s- Reason for failure: Anything else we need to know: We are removing CSI 0.3 support in 1.18, which is before selflink is going to be removed in 1.20. So we should just disable hostpath-v0 from alpha tests. /assign
/milestone v1.16

Creates a replication controller to manage the created container(s). ``` <del> kubectl run NAME --image=image [--env="key=value"] [--port=port] [--replicas=replicas] [--dry-run=bool] [--overrides=inline-json] </del> <ins> kubectl run NAME --image=image [--env="key=value"] [--port=port] [--replicas=replicas] [--dry-run=bool] [--overrides=inline-json] [--command] -- [COMMAND] [args...] </ins> ``` ### Examples


Question:
<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: ci-kubernetes-e2e-gci-gce-alpha-enabled-default Which test(s) are failing: CSI hostpath-v0 tests Since when has it been failing: 8/8 Testgrid link: https://k8s- Reason for failure: Anything else we need to know: We are removing CSI 0.3 support in 1.18, which is before selflink is going to be removed in 1.20. So we should just disable hostpath-v0 from alpha tests. /assign
/milestone v1.16

* [kubectl](kubectl.md)	 - kubectl controls the Kubernetes cluster manager <del> ###### Auto generated by spf13/cobra on 6-Nov-2015 </del> <ins> ###### Auto generated by spf13/cobra on 10-Nov-2015 </ins> <!-- BEGIN MUNGE: GENERATED_ANALYTICS --> [![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/user-guide/kubectl/kubectl_run.md?pixel)]()


Question:
<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: ci-kubernetes-e2e-gci-gce-alpha-enabled-default Which test(s) are failing: CSI hostpath-v0 tests Since when has it been failing: 8/8 Testgrid link: https://k8s- Reason for failure: Anything else we need to know: We are removing CSI 0.3 support in 1.18, which is before selflink is going to be removed in 1.20. So we should just disable hostpath-v0 from alpha tests. /assign
/milestone v1.16

func NewCmdRun(f *cmdutil.Factory, cmdIn io.Reader, cmdOut, cmdErr io.Writer) *cobra.Command { cmd := &cobra.Command{ <del> Use: "run NAME --image=image [--env="key=value"] [--port=port] [--replicas=replicas] [--dry-run=bool] [--overrides=inline-json]", </del> <ins> Use: "run NAME --image=image [--env="key=value"] [--port=port] [--replicas=replicas] [--dry-run=bool] [--overrides=inline-json] [--command] -- [COMMAND] [args...]", </ins> // run-container is deprecated Aliases: []string{"run-container"}, Short:   "Run a particular image on the cluster.", Long:    run_long, Example: run_example, Run: func(cmd *cobra.Command, args []string) { <del> err := Run(f, cmdIn, cmdOut, cmdErr, cmd, args) </del> <ins> argsLenAtDash := cmd.ArgsLenAtDash() err := Run(f, cmdIn, cmdOut, cmdErr, cmd, args, argsLenAtDash) </ins> cmdutil.CheckErr(err) }, }


Question:
<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: ci-kubernetes-e2e-gci-gce-alpha-enabled-default Which test(s) are failing: CSI hostpath-v0 tests Since when has it been failing: 8/8 Testgrid link: https://k8s- Reason for failure: Anything else we need to know: We are removing CSI 0.3 support in 1.18, which is before selflink is going to be removed in 1.20. So we should just disable hostpath-v0 from alpha tests. /assign
/milestone v1.16

cmd.Flags().String("service-overrides", "", "An inline JSON override for the generated service object. If this is non-empty, it is used to override the generated object. Requires that the object supply a valid apiVersion field.  Only used if --expose is true.") } <del> func Run(f *cmdutil.Factory, cmdIn io.Reader, cmdOut, cmdErr io.Writer, cmd *cobra.Command, args []string) error { </del> <ins> func Run(f *cmdutil.Factory, cmdIn io.Reader, cmdOut, cmdErr io.Writer, cmd *cobra.Command, args []string, argsLenAtDash int) error { </ins> if len(os.Args) > 1 && os.Args[1] == "run-container" { printDeprecationWarning("run", "run-container") } <del> if len(args) == 0 { </del> <ins> // Let kubectl run follow rules for `--`, see #13004 issue if len(args) == 0 || argsLenAtDash == 0 { </ins> return cmdutil.UsageError(cmd, "NAME is required for run") }


Question:
<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: ci-kubernetes-e2e-gci-gce-alpha-enabled-default Which test(s) are failing: CSI hostpath-v0 tests Since when has it been failing: 8/8 Testgrid link: https://k8s- Reason for failure: Anything else we need to know: We are removing CSI 0.3 support in 1.18, which is before selflink is going to be removed in 1.20. So we should just disable hostpath-v0 from alpha tests. /assign
/milestone v1.16

"fmt" "io/ioutil" "net/http" <ins> "os" </ins> "reflect" "testing"


Question:
<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: ci-kubernetes-e2e-gci-gce-alpha-enabled-default Which test(s) are failing: CSI hostpath-v0 tests Since when has it been failing: 8/8 Testgrid link: https://k8s- Reason for failure: Anything else we need to know: We are removing CSI 0.3 support in 1.18, which is before selflink is going to be removed in 1.20. So we should just disable hostpath-v0 from alpha tests. /assign
/milestone v1.16

} } <ins> func TestRunArgsFollowDashRules(t *testing.T) { _, _, rc := testData() tests := []struct { args          []string argsLenAtDash int expectError   bool name          string }{ { args:          []string{}, argsLenAtDash: -1, expectError:   true, name:          "empty", }, { args:          []string{"foo"}, argsLenAtDash: -1, expectError:   false, name:          "no cmd", }, { args:          []string{"foo", "sleep"}, argsLenAtDash: -1, expectError:   false, name:          "cmd no dash", }, { args:          []string{"foo", "sleep"}, argsLenAtDash: 1, expectError:   false, name:          "cmd has dash", }, { args:          []string{"foo", "sleep"}, argsLenAtDash: 0, expectError:   true, name:          "no name", }, } for _, test := range tests { f, tf, codec := NewAPIFactory() tf.Client = &fake.RESTClient{ Codec: codec, Client: fake.CreateHTTPClient(func(req *http.Request) (*http.Response, error) { return &http.Response{StatusCode: 201, Body: objBody(codec, &rc.Items[0])}, nil }), } tf.Namespace = "test" tf.ClientConfig = &client.Config{} cmd := NewCmdRun(f, os.Stdin, os.Stdout, os.Stderr) cmd.Flags().Set("image", "nginx") err := Run(f, os.Stdin, os.Stdout, os.Stderr, cmd, test.args, test.argsLenAtDash) if test.expectError && err == nil { t.Errorf("unexpected non-error (%s)", test.name) } if !test.expectError && err != nil { t.Errorf("unexpected error: %v (%s)", err, test.name) } } } </ins> func TestGenerateService(t *testing.T) { tests := []struct {


Question:
/help /kind cleanup /area e2e-testing-framework mini guide: make sure you have signed the CNCF clone kubernetes;/kubernetes search for ginkgo.Fail calls under test/e2e/framework e.g. replace them with this that was in this PR: send a PR for kubernetes/kubernetes
/remove-kind feature /area e2e-test-framework
/sig testing
I will take this.
thank you! Please feel free to ping #testing-commons in the Kubernetes slack if you have any questions :smiley: /cc
/remove-help
/assign

} e2elog.Logf("Got an error creating template %d: %v", i, err) } <del> ginkgo.Fail("Unable to create template %d, exiting", i) </del> <ins> e2elog.Fail("Unable to create template %d, exiting", i) </ins> }) })


Question:
/help /kind cleanup /area e2e-testing-framework mini guide: make sure you have signed the CNCF clone kubernetes;/kubernetes search for ginkgo.Fail calls under test/e2e/framework e.g. replace them with this that was in this PR: send a PR for kubernetes/kubernetes
/remove-kind feature /area e2e-test-framework
/sig testing
I will take this.
thank you! Please feel free to ping #testing-commons in the Kubernetes slack if you have any questions :smiley: /cc
/remove-help
/assign

} e2elog.Logf("Got an error creating template %d: %v", i, err) } <del> ginkgo.Fail("Unable to create template %d, exiting", i) </del> <ins> e2elog.Fail("Unable to create template %d, exiting", i) </ins> }) pagedTable := &metav1beta1.Table{}


Question:
/help /kind cleanup /area e2e-testing-framework mini guide: make sure you have signed the CNCF clone kubernetes;/kubernetes search for ginkgo.Fail calls under test/e2e/framework e.g. replace them with this that was in this PR: send a PR for kubernetes/kubernetes
/remove-kind feature /area e2e-test-framework
/sig testing
I will take this.
thank you! Please feel free to ping #testing-commons in the Kubernetes slack if you have any questions :smiley: /cc
/remove-help
/assign

srcs = ["testfiles.go"], importpath = "k8s.io/kubernetes/test/e2e/framework/testfiles", visibility = ["//visibility:public"], <del> deps = ["//vendor/github.com/onsi/ginkgo:go_default_library"], </del> <ins> deps = ["//test/e2e/framework/log:go_default_library"], </ins> ) filegroup(


Question:
/help /kind cleanup /area e2e-testing-framework mini guide: make sure you have signed the CNCF clone kubernetes;/kubernetes search for ginkgo.Fail calls under test/e2e/framework e.g. replace them with this that was in this PR: send a PR for kubernetes/kubernetes
/remove-kind feature /area e2e-test-framework
/sig testing
I will take this.
thank you! Please feel free to ping #testing-commons in the Kubernetes slack if you have any questions :smiley: /cc
/remove-help
/assign

"sort" "strings" <del> "github.com/onsi/ginkgo" </del> <ins> e2elog "k8s.io/kubernetes/test/e2e/framework/log" </ins> ) var filesources []FileSource


Question:
/help /kind cleanup /area e2e-testing-framework mini guide: make sure you have signed the CNCF clone kubernetes;/kubernetes search for ginkgo.Fail calls under test/e2e/framework e.g. replace them with this that was in this PR: send a PR for kubernetes/kubernetes
/remove-kind feature /area e2e-test-framework
/sig testing
I will take this.
thank you! Please feel free to ping #testing-commons in the Kubernetes slack if you have any questions :smiley: /cc
/remove-help
/assign

func ReadOrDie(filePath string) []byte { data, err := Read(filePath) if err != nil { <del> ginkgo.Fail(err.Error(), 1) </del> <ins> e2elog.Fail(err.Error(), 1) </ins> } return data }


Question:
/help /kind cleanup /area e2e-testing-framework mini guide: make sure you have signed the CNCF clone kubernetes;/kubernetes search for ginkgo.Fail calls under test/e2e/framework e.g. replace them with this that was in this PR: send a PR for kubernetes/kubernetes
/remove-kind feature /area e2e-test-framework
/sig testing
I will take this.
thank you! Please feel free to ping #testing-commons in the Kubernetes slack if you have any questions :smiley: /cc
/remove-help
/assign

for _, filesource := range filesources { data, err := filesource.ReadTestFile(filePath) if err != nil { <del> ginkgo.Fail(fmt.Sprintf("fatal error looking for test file %s: %s", filePath, err), 1) </del> <ins> e2elog.Fail(fmt.Sprintf("fatal error looking for test file %s: %s", filePath, err), 1) </ins> } if data != nil { return true


Question:
<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: Which test(s) are failing: Several tests Eg. this run: Since when has it been failing: Unknown Testgrid link: Reason for failure: (happens to all the images). Pods in the e2e test suites use pull policy , , and it looks like the test during DiskPressure, resulting in kubelet removing some of the pre pulled images. Anything else we need to know: The runs after each test. This should also be the case in the test too.
/assign /sig node
These tests are now constantly failing due to missing images. This makes it hard to see if some of the stuff we do now, or have done in the last few weeks, breaks some of the tests. PR should hopefully fix this, but it is not in the 1.16 milestone, but has and . Should we get this merged into master so that we can see if if fixes the problem, or should we wait until code thaw? Any thoughts? /cc /cc /cc
/priority important-longterm
Thanks for all the work you've put into this ! I think we should add to the milestone
+1 to - thanks for your hard work :)
still has a lot of failing tests. What are the next steps? /reopen
Reopened this issue. <detailsIn response to : still has a lot of failing tests. What are the next steps? Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Yes, it has over the few last weeks become quite a mess... Looks like fixed some of the issues about image eviction problems, but there are still a few left. I think I have a solution, so ill try to submit a ~PR in a few hours~ edit: will do it tomorrow. A lot of new tests started failing 8-28, and that may be related to , but I am not totally sure..
Looks like fixed the flaking in ! The last 7 runs was executed after that was merged! ! We still have a lot of failing tests, but those are not flaky; they are failing (ref. comment above). Ill create a new issue about all the failing tests where we can track them each together. Closing this issue, as this was only filed for the flaking becuase of the evicted images. :smile: /close cc
Closing this issue. <detailsIn response to : Looks like fixed the flaking in ! The last 7 runs was executed after that was merged! We still have a lot of failing tests, but those are not flaky; they are failing (ref. comment above). cc Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

}) ginkgo.AfterEach(func() { <ins> defer func() { if expectedNodeCondition == v1.NodeDiskPressure && framework.TestContext.PrepullImages { // The disk eviction test may cause the prepulled images to be evicted, // prepull those images again to ensure this test not affect following tests. PrePullAllImages() } }() </ins> ginkgo.By("deleting pods") for _, spec := range testSpecs { ginkgo.By(fmt.Sprintf("deleting pod: %s", spec.pod.Name)) f.PodClient().DeleteSync(spec.pod.Name, &metav1.DeleteOptions{}, 10*time.Minute) } reduceAllocatableMemoryUsage() <del> if expectedNodeCondition == v1.NodeDiskPressure && framework.TestContext.PrepullImages { // The disk eviction test may cause the prepulled images to be evicted, // prepull those images again to ensure this test not affect following tests. PrePullAllImages() } </del> ginkgo.By("making sure we can start a new pod after the test") podName := "test-admit-pod" f.PodClient().CreateSync(&v1.Pod{


Question:
<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: Which test(s) are failing: Several tests Eg. this run: Since when has it been failing: Unknown Testgrid link: Reason for failure: (happens to all the images). Pods in the e2e test suites use pull policy , , and it looks like the test during DiskPressure, resulting in kubelet removing some of the pre pulled images. Anything else we need to know: The runs after each test. This should also be the case in the test too.
/assign /sig node
These tests are now constantly failing due to missing images. This makes it hard to see if some of the stuff we do now, or have done in the last few weeks, breaks some of the tests. PR should hopefully fix this, but it is not in the 1.16 milestone, but has and . Should we get this merged into master so that we can see if if fixes the problem, or should we wait until code thaw? Any thoughts? /cc /cc /cc
/priority important-longterm
Thanks for all the work you've put into this ! I think we should add to the milestone
+1 to - thanks for your hard work :)
still has a lot of failing tests. What are the next steps? /reopen
Reopened this issue. <detailsIn response to : still has a lot of failing tests. What are the next steps? Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Yes, it has over the few last weeks become quite a mess... Looks like fixed some of the issues about image eviction problems, but there are still a few left. I think I have a solution, so ill try to submit a ~PR in a few hours~ edit: will do it tomorrow. A lot of new tests started failing 8-28, and that may be related to , but I am not totally sure..
Looks like fixed the flaking in ! The last 7 runs was executed after that was merged! ! We still have a lot of failing tests, but those are not flaky; they are failing (ref. comment above). Ill create a new issue about all the failing tests where we can track them each together. Closing this issue, as this was only filed for the flaking becuase of the evicted images. :smile: /close cc
Closing this issue. <detailsIn response to : Looks like fixed the flaking in ! The last 7 runs was executed after that was merged! We still have a lot of failing tests, but those are not flaky; they are failing (ref. comment above). cc Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

}, time.Minute*8, time.Second*4).ShouldNot(gomega.HaveOccurred()) }) ginkgo.AfterEach(func() { <ins> defer func() { if framework.TestContext.PrepullImages { // The test may cause the prepulled images to be evicted, // prepull those images again to ensure this test not affect following tests. PrePullAllImages() } }() </ins> ginkgo.By("delete the static pod") err := deleteStaticPod(podPath, staticPodName, ns) gomega.Expect(err).ShouldNot(gomega.HaveOccurred())


Question:
<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: Which test(s) are failing: Several tests Eg. this run: Since when has it been failing: Unknown Testgrid link: Reason for failure: (happens to all the images). Pods in the e2e test suites use pull policy , , and it looks like the test during DiskPressure, resulting in kubelet removing some of the pre pulled images. Anything else we need to know: The runs after each test. This should also be the case in the test too.
/assign /sig node
These tests are now constantly failing due to missing images. This makes it hard to see if some of the stuff we do now, or have done in the last few weeks, breaks some of the tests. PR should hopefully fix this, but it is not in the 1.16 milestone, but has and . Should we get this merged into master so that we can see if if fixes the problem, or should we wait until code thaw? Any thoughts? /cc /cc /cc
/priority important-longterm
Thanks for all the work you've put into this ! I think we should add to the milestone
+1 to - thanks for your hard work :)
still has a lot of failing tests. What are the next steps? /reopen
Reopened this issue. <detailsIn response to : still has a lot of failing tests. What are the next steps? Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Yes, it has over the few last weeks become quite a mess... Looks like fixed some of the issues about image eviction problems, but there are still a few left. I think I have a solution, so ill try to submit a ~PR in a few hours~ edit: will do it tomorrow. A lot of new tests started failing 8-28, and that may be related to , but I am not totally sure..
Looks like fixed the flaking in ! The last 7 runs was executed after that was merged! ! We still have a lot of failing tests, but those are not flaky; they are failing (ref. comment above). Ill create a new issue about all the failing tests where we can track them each together. Closing this issue, as this was only filed for the flaking becuase of the evicted images. :smile: /close cc
Closing this issue. <detailsIn response to : Looks like fixed the flaking in ! The last 7 runs was executed after that was merged! We still have a lot of failing tests, but those are not flaky; they are failing (ref. comment above). cc Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

gomega.Eventually(func() error { return checkMirrorPodDisappear(f.ClientSet, mirrorPodName, ns) }, time.Minute, time.Second*2).Should(gomega.BeNil()) }) }) })


Question:
There are multiple cert controllers that use the same base struct, so when they log, you cannot differentiate which is which.
/sig auth

return nil, false, nil } if ctx.ComponentConfig.CSRSigningController.ClusterSigningCertFile == "" || ctx.ComponentConfig.CSRSigningController.ClusterSigningKeyFile == "" { <ins> klog.V(2).Info("skipping CSR signer controller because no csr cert/key was specified") </ins> return nil, false, nil }


Question:
There are multiple cert controllers that use the same base struct, so when they log, you cannot differentiate which is which.
/sig auth

// setting up the signing controller. This isn't // actually a problem since the signer is not a // required controller. <ins> klog.V(2).Info("skipping CSR signer controller because no csr cert/key was specified and the default files are missing") </ins> return nil, false, nil default: // Note that '!filesExist && !usesDefaults' is obviously


Question:
There are multiple cert controllers that use the same base struct, so when they log, you cannot differentiate which is which.
/sig auth

recognizers: recognizers(), } return certificates.NewCertificateController( <ins> "csrapproving", </ins> client, csrInformer, approver.handle,


Question:
There are multiple cert controllers that use the same base struct, so when they log, you cannot differentiate which is which.
/sig auth

) type CertificateController struct { <ins> // name is an identifier for this particular controller instance. name string </ins> kubeClient clientset.Interface csrLister  certificateslisters.CertificateSigningRequestLister


Question:
There are multiple cert controllers that use the same base struct, so when they log, you cannot differentiate which is which.
/sig auth

} func NewCertificateController( <ins> name string, </ins> kubeClient clientset.Interface, csrInformer certificatesinformers.CertificateSigningRequestInformer, handler func(*certificates.CertificateSigningRequest) error,


Question:
There are multiple cert controllers that use the same base struct, so when they log, you cannot differentiate which is which.
/sig auth

eventBroadcaster.StartRecordingToSink(&v1core.EventSinkImpl{Interface: kubeClient.CoreV1().Events("")}) cc := &CertificateController{ <ins> name:       name, </ins> kubeClient: kubeClient, queue: workqueue.NewNamedRateLimitingQueue(workqueue.NewMaxOfRateLimiter( workqueue.NewItemExponentialFailureRateLimiter(200*time.Millisecond, 1000*time.Second),


Question:
There are multiple cert controllers that use the same base struct, so when they log, you cannot differentiate which is which.
/sig auth

defer utilruntime.HandleCrash() defer cc.queue.ShutDown() <del> klog.Infof("Starting certificate controller") defer klog.Infof("Shutting down certificate controller") </del> <ins> klog.Infof("Starting certificate controller %q", cc.name) defer klog.Infof("Shutting down certificate controller %q", cc.name) </ins> <del> if !cache.WaitForNamedCacheSync("certificate", stopCh, cc.csrsSynced) { </del> <ins> if !cache.WaitForNamedCacheSync(fmt.Sprintf("certificate-%s", cc.name), stopCh, cc.csrsSynced) { </ins> return }


Question:
There are multiple cert controllers that use the same base struct, so when they log, you cannot differentiate which is which.
/sig auth

} controller := NewCertificateController( <ins> "test", </ins> client, informerFactory.Certificates().V1beta1().CertificateSigningRequests(), handler,


Question:
There are multiple cert controllers that use the same base struct, so when they log, you cannot differentiate which is which.
/sig auth

return nil, err } return certificates.NewCertificateController( <ins> "csrsigning", </ins> client, csrInformer, signer.handle,


Question:
MaxInt const is exported in scheduler api although it is not needed to be exported. It is only used by . discussed that the common practice is to have API constants in /sig scheduling
cc
The problem is that it is part of a v1 API :(
Ok, it turns out they are not actually considered part of the API, and so we can remove them:
Should I open a PR to address removing those const values from ?
Yes, we can remove those in a followup PR to the one handling sized values.
Yes, this was the plan. /assign
remember to remove as well :)

) const ( <del> // MaxUint defines the max unsigned int value. MaxUint = ^uint(0) // MaxInt defines the max signed int value. MaxInt = int(MaxUint >> 1) </del> // MaxTotalPriority defines the max total priority value. MaxTotalPriority = int64(math.MaxInt64) // MaxPriority defines the max priority value. MaxPriority = 10 // MaxWeight defines the max weight value. <del> MaxWeight = int64(math.MaxInt64 / MaxPriority) </del> <ins> MaxWeight = MaxTotalPriority / MaxPriority </ins> // DefaultPercentageOfNodesToScore defines the percentage of nodes of all nodes // that once found feasible, the scheduler stops looking for more nodes. DefaultPercentageOfNodesToScore = 50


Question:
MaxInt const is exported in scheduler api although it is not needed to be exported. It is only used by . discussed that the common practice is to have API constants in /sig scheduling
cc
The problem is that it is part of a v1 API :(
Ok, it turns out they are not actually considered part of the API, and so we can remove them:
Should I open a PR to address removing those const values from ?
Yes, we can remove those in a followup PR to the one handling sized values.
Yes, this was the plan. /assign
remember to remove as well :)

package factory import ( <del> "math" </del> "testing" "github.com/stretchr/testify/assert"


Question:
MaxInt const is exported in scheduler api although it is not needed to be exported. It is only used by . discussed that the common practice is to have API constants in /sig scheduling
cc
The problem is that it is part of a v1 API :(
Ok, it turns out they are not actually considered part of the API, and so we can remove them:
Should I open a PR to address removing those const values from ?
Yes, we can remove those in a followup PR to the one handling sized values.
Yes, this was the plan. /assign
remember to remove as well :)

expected    bool }{ { <del> description: "one of the weights is MaxInt64", configs:     []priorities.PriorityConfig{{Weight: math.MaxInt64}, {Weight: 5}}, </del> <ins> description: "one of the weights is MaxTotalPriority(MaxInt64)", configs:     []priorities.PriorityConfig{{Weight: api.MaxTotalPriority}, {Weight: 5}}, </ins> expected:    true, }, { description: "after multiplication with MaxPriority the weight is larger than MaxWeight", <del> configs:     []priorities.PriorityConfig{{Weight: math.MaxInt64/api.MaxPriority + api.MaxPriority}, {Weight: 5}}, </del> <ins> configs:     []priorities.PriorityConfig{{Weight: api.MaxWeight + api.MaxPriority}, {Weight: 5}}, </ins> expected:    true, }, {


Question:
Which jobs are failing: ci-kubernetes-node-kubelet-orphans Which test(s) are failing: [sig-storage] EmptyDir volumes pod should support shared volumes between containers Since when has it been failing: Testgrid link: Reason for failure:
/sig testing
/assign
Thank for comment
Tests are now passing, /close
Closing this issue. <detailsIn response to : Tests are now passing, Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

flags.StringVar(&TestContext.KubernetesAnywherePath, "kubernetes-anywhere-path", "/workspace/k8s.io/kubernetes-anywhere", "Which directory kubernetes-anywhere is installed to.") flags.BoolVar(&TestContext.ListImages, "list-images", false, "If true, will show list of images used for runnning tests.") <ins> flags.StringVar(&TestContext.KubectlPath, "kubectl-path", "kubectl", "The kubectl binary to use. For development, you might use 'cluster/kubectl.sh' here.") </ins> } // RegisterClusterFlags registers flags specific to the cluster e2e test suite.


Question:
Which jobs are failing: ci-kubernetes-node-kubelet-orphans Which test(s) are failing: [sig-storage] EmptyDir volumes pod should support shared volumes between containers Since when has it been failing: Testgrid link: Reason for failure:
/sig testing
/assign
Thank for comment
Tests are now passing, /close
Closing this issue. <detailsIn response to : Tests are now passing, Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

flags.StringVar(&TestContext.RepoRoot, "repo-root", "../../", "Root directory of kubernetes repository, for finding test files.") flags.StringVar(&TestContext.Provider, "provider", "", "The name of the Kubernetes provider (gce, gke, local, skeleton (the fallback if not set), etc.)") flags.StringVar(&TestContext.Tooling, "tooling", "", "The tooling in use (kops, gke, etc.)") <del> flags.StringVar(&TestContext.KubectlPath, "kubectl-path", "kubectl", "The kubectl binary to use. For development, you might use 'cluster/kubectl.sh' here.") </del> flags.StringVar(&TestContext.OutputDir, "e2e-output-dir", "/tmp", "Output directory for interesting/useful test data, like performance data, benchmarks, and other metrics.") flags.StringVar(&TestContext.Prefix, "prefix", "e2e", "A prefix to be added to cloud resources created during testing.") flags.StringVar(&TestContext.MasterOSDistro, "master-os-distro", "debian", "The OS distribution of cluster master (debian, ubuntu, gci, coreos, or custom).")


Question:
I tried to use a local volume block device in minikube. Its trying to call losetup with the -j option and its failing as the losetup in minikube does not provide this option. This may effect other distro's as well. The end result is lock block devices don't work on the platform.
/sig storage
Probably caused by this code:
Did a strace on losetup -j. looks like the same info can be had by: cat /sys/block/loop0/loop/backing_file
Just had the same problem with Ceph cluster creation in minikube, the osd-prepare doesn't start because of this:
FYI: has the same problem in minikube. Replacing the /sbin/losetup busybox link with a "real" losetup helped for now, but isn't a solution of course.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale
/help
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help </details

import ( "errors" "fmt" <ins> "io/ioutil" </ins> "os" "path/filepath" "strings"


Question:
I tried to use a local volume block device in minikube. Its trying to call losetup with the -j option and its failing as the losetup in minikube does not provide this option. This may effect other distro's as well. The end result is lock block devices don't work on the platform.
/sig storage
Probably caused by this code:
Did a strace on losetup -j. looks like the same info can be had by: cat /sys/block/loop0/loop/backing_file
Just had the same problem with Ceph cluster creation in minikube, the osd-prepare doesn't start because of this:
FYI: has the same problem in minikube. Replacing the /sbin/losetup busybox link with a "real" losetup helped for now, but isn't a solution of course.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale
/help
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help </details

// attachFileDevice takes a path to a regular file and makes it available as an // attached block device. func attachFileDevice(path string, exec utilexec.Interface) (string, error) { <del> blockDevicePath, err := getLoopDevice(path, exec) </del> <ins> blockDevicePath, err := getLoopDevice(path) </ins> if err != nil && err.Error() != ErrDeviceNotFound { return "", err }


Question:
I tried to use a local volume block device in minikube. Its trying to call losetup with the -j option and its failing as the losetup in minikube does not provide this option. This may effect other distro's as well. The end result is lock block devices don't work on the platform.
/sig storage
Probably caused by this code:
Did a strace on losetup -j. looks like the same info can be had by: cat /sys/block/loop0/loop/backing_file
Just had the same problem with Ceph cluster creation in minikube, the osd-prepare doesn't start because of this:
FYI: has the same problem in minikube. Replacing the /sbin/losetup busybox link with a "real" losetup helped for now, but isn't a solution of course.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale
/help
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help </details

} // Returns the full path to the loop device associated with the given path. <del> func getLoopDevice(path string, exec utilexec.Interface) (string, error) { </del> <ins> func getLoopDevice(path string) (string, error) { </ins> _, err := os.Stat(path) if os.IsNotExist(err) { return "", errors.New(ErrNotAvailable)