Question:
I also hit when running cri-containerd as the container runtime. The reason is that we don't populate stats like in cri-containerd integration now. is adding the containerd support in cadvisor, after that we should have full summary metrics. However, heapster should not panic in this case anyway. Can we cut a heapster patch release and update heapster in the main repo?
+1. Many new CRI integration would run into this issue.
In fact, as far as I am aware, not a single CRI implementation currently implements any of the stats calls.
I beg to differ: That's the reason why filed the issue.
Yeah does implement the CRI stats. Currently it only support core metrics, which is only part of summary api. We are working on integration with cadvisor, before that some fields in summary api such as will be nil. Current heapster version will crash when is nil, which shouldn't happen. :)
/cc
My bad, thanks for the pointer :slightlysmilingface:. I'll correct my statement: I believe no CRI implementation except for cri-containerd implements the stats calls. The ones I explicitly checked are: dockershim, cri-o, rktlet.
Something I'm wondering about though: The intention of the stats calls is in order to be able to get rid of cAdvisor. Making the stats calls depend on cAdvisor seems to cover up the issues it has. Re:
Here is the reason that we still have to integrate with cadvisor for now.
/cc
If I understand correctly this is meant as a migration mean, so that both are possible for some time, and then we drop cAdvisor once a reasonable migration period has been granted.
[MILESTONENOTIFIER] Milestone Issue Current <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
I will take a look at Heapster and try to fix this. Can you advise how to reproduce the issue? This will help a lot with debugging and verifying a fix once it's ready.
this is already fixed at head and 1.4.3 version of heapster The problem that is using kubernetes version from head and we have there 1.5.0-beta.0: I see two possible solutions: version of heapster to 1.4.3 at head of kubernetes. Piotr mentioned that 1.5.0 has still a lot of problems, so it maybe makes sense. a new release of heapster (1.5.0-beta.1) and after that bump version at head.
And thanks for taking this issue.
Not directly related to this issue: a more general discussion is at We'd like to understand which fields are required and optional in heapster, so that we won't crash heapster with partial result.
We've merged a work around for now Hopefully it works. But anyway, we should fix this in 1.9.
what needs to be done for 1.9 here?
we'll need an assignee as well
We just need to update heapster version.

- --memory={{ base_metrics_memory }} - --extra-memory={{ metrics_memory_per_node }}Mi - --threshold=5 <del> - --deployment=heapster-v1.5.0-beta.0 </del> <ins> - --deployment=heapster-v1.5.0-beta.1 </ins> - --container=heapster - --poll-period=300000 - --estimator=exponential


Question:
Is this a BUG REPORT or FEATURE REQUEST?: Bug Report /kind bug What happened: improved disk eviction testing to make it more precise. For example, it expects a pod which uses 299 Mb of disk not to be evicted if it sets its limit to 300 Mi. As shown on , this test passes for all OS distributions except CoreOS. I have ruled out an issue with the storage driver and backing filesystem since both COS and CoreOS use overlay backed by extfs. What you expected to happen: I expected the pod using 299Mb not to be evicted when its limit is 300 Mb. I expected not to see this event: How to reproduce it (as minimally and precisely as possible): The container runs this command to use 299Mb: Each iteration writes 1Mb to disk, producing a total of 299 Mb of disk usage. The pod has set to 300Mi. The test has the LocalStorageCapacityIsolation feature gate enabled. Anything else we need to know?: This may be a discrepancy in the (which we use to measure container disk usage) or , which is used by the test to write exact quantities of disk.
/sig node
cc can you take a look at this after thanksgiving?
I bumped the coreos image... going to see if it clears up the flake
It appears the root filesystem is being included in the disk space calculation . Do you think the coreos image might be slightly larger than the other images under test?
One more code snippet that is interesting:
thats the container's root filesystem
correct. the feature appears to add and compare to the quota: The reproducible steps do not take into account the rootfs space. If the quota is set to 300Mi, then the under loop count would have to take into account the rootfs space to not exceed the limit. I do not see anywhere in the tests or the reproduce steps this part of the calculation.
The refers only to the writable layer of the container, and excludes base (image) layers. With the provided command, the only files that are in the container's rootfs are the ones written by the command, and should amount to 299Mb. Additionally, the container does not write any logs (through stdout), so that value should be 0.
I ran a quick test on a vm with the current image, and found that both and correctly wrote and measured disk space correctly. I also tested as it inside a container, and verified that we are measuring the container's writable layer correctly. My guess now is that it has something to do with how we calculate disk usage by logging with journald. I am looking into it
Actually, the previous comment was incorrect. When I ran the test, I found that each file was 1028 kb instead of 1024 kb, which resulted in the command writing 300.3 Mb instead of 299 Mb. I will change the test to require slightly less precision. This appears to be because an empty file in coreos takes up 4 kb. Test this by running . On other OS we test on, empty files use 0 kb. This is what accounts for the difference.
I tracked down an issue to the dd call. The stdout and stderr need a redirect to /dev/null to suppress the following messages: Problem Line: Fix:
I PRed that fix and changed it to stderr, since the other tests seem to use stderr.
leaving this open until we get a green run
I'll continue to test locally to see if I find anything else.
opened PR with the help of and CoreOS defaults to using the actual size instead of the apprent size.

func diskConsumingPod(name string, diskConsumedMB int, volumeSource *v1.VolumeSource, resources v1.ResourceRequirements) *v1.Pod { // Each iteration writes 1 Mb, so do diskConsumedMB iterations. <del> return podWithCommand(volumeSource, resources, name, fmt.Sprintf("i=0; while [ $i -lt %d ];", diskConsumedMB)+" do dd if=/dev/urandom of=%s${i} bs=1048576 count=1; i=$(($i+1)); done; while true; do sleep 5; done") </del> <ins> return podWithCommand(volumeSource, resources, name, fmt.Sprintf("i=0; while [ $i -lt %d ];", diskConsumedMB)+" do dd if=/dev/urandom of=%s${i} bs=1048576 count=1 2>/dev/null ; i=$(($i+1)); done; while true; do sleep 5; done") </ins> } // podWithCommand returns a pod with the provided volumeSource and resourceRequirements.


Question:
Is this a BUG REPORT or FEATURE REQUEST?: Bug Report /kind bug What happened: improved disk eviction testing to make it more precise. For example, it expects a pod which uses 299 Mb of disk not to be evicted if it sets its limit to 300 Mi. As shown on , this test passes for all OS distributions except CoreOS. I have ruled out an issue with the storage driver and backing filesystem since both COS and CoreOS use overlay backed by extfs. What you expected to happen: I expected the pod using 299Mb not to be evicted when its limit is 300 Mb. I expected not to see this event: How to reproduce it (as minimally and precisely as possible): The container runs this command to use 299Mb: Each iteration writes 1Mb to disk, producing a total of 299 Mb of disk usage. The pod has set to 300Mi. The test has the LocalStorageCapacityIsolation feature gate enabled. Anything else we need to know?: This may be a discrepancy in the (which we use to measure container disk usage) or , which is used by the test to write exact quantities of disk.
/sig node
cc can you take a look at this after thanksgiving?
I bumped the coreos image... going to see if it clears up the flake
It appears the root filesystem is being included in the disk space calculation . Do you think the coreos image might be slightly larger than the other images under test?
One more code snippet that is interesting:
thats the container's root filesystem
correct. the feature appears to add and compare to the quota: The reproducible steps do not take into account the rootfs space. If the quota is set to 300Mi, then the under loop count would have to take into account the rootfs space to not exceed the limit. I do not see anywhere in the tests or the reproduce steps this part of the calculation.
The refers only to the writable layer of the container, and excludes base (image) layers. With the provided command, the only files that are in the container's rootfs are the ones written by the command, and should amount to 299Mb. Additionally, the container does not write any logs (through stdout), so that value should be 0.
I ran a quick test on a vm with the current image, and found that both and correctly wrote and measured disk space correctly. I also tested as it inside a container, and verified that we are measuring the container's writable layer correctly. My guess now is that it has something to do with how we calculate disk usage by logging with journald. I am looking into it
Actually, the previous comment was incorrect. When I ran the test, I found that each file was 1028 kb instead of 1024 kb, which resulted in the command writing 300.3 Mb instead of 299 Mb. I will change the test to require slightly less precision. This appears to be because an empty file in coreos takes up 4 kb. Test this by running . On other OS we test on, empty files use 0 kb. This is what accounts for the difference.
I tracked down an issue to the dd call. The stdout and stderr need a redirect to /dev/null to suppress the following messages: Problem Line: Fix:
I PRed that fix and changed it to stderr, since the other tests seem to use stderr.
leaving this open until we get a green run
I'll continue to test locally to see if I find anything else.
opened PR with the help of and CoreOS defaults to using the actual size instead of the apprent size.

EmptyDir: &v1.EmptyDirVolumeSource{SizeLimit: &sizeLimit}, }, v1.ResourceRequirements{}), }, <ins> { evictionPriority: 0, // This pod should not be evicted because it uses less than its limit pod:              diskConsumingPod("emptydir-disk-below-sizelimit", useUnderLimit, nil, v1.ResourceRequirements{Limits: containerLimit}), }, </ins> }) }) })


Question:
Is this a BUG REPORT or FEATURE REQUEST?: Bug Report /kind bug What happened: improved disk eviction testing to make it more precise. For example, it expects a pod which uses 299 Mb of disk not to be evicted if it sets its limit to 300 Mi. As shown on , this test passes for all OS distributions except CoreOS. I have ruled out an issue with the storage driver and backing filesystem since both COS and CoreOS use overlay backed by extfs. What you expected to happen: I expected the pod using 299Mb not to be evicted when its limit is 300 Mb. I expected not to see this event: How to reproduce it (as minimally and precisely as possible): The container runs this command to use 299Mb: Each iteration writes 1Mb to disk, producing a total of 299 Mb of disk usage. The pod has set to 300Mi. The test has the LocalStorageCapacityIsolation feature gate enabled. Anything else we need to know?: This may be a discrepancy in the (which we use to measure container disk usage) or , which is used by the test to write exact quantities of disk.
/sig node
cc can you take a look at this after thanksgiving?
I bumped the coreos image... going to see if it clears up the flake
It appears the root filesystem is being included in the disk space calculation . Do you think the coreos image might be slightly larger than the other images under test?
One more code snippet that is interesting:
thats the container's root filesystem
correct. the feature appears to add and compare to the quota: The reproducible steps do not take into account the rootfs space. If the quota is set to 300Mi, then the under loop count would have to take into account the rootfs space to not exceed the limit. I do not see anywhere in the tests or the reproduce steps this part of the calculation.
The refers only to the writable layer of the container, and excludes base (image) layers. With the provided command, the only files that are in the container's rootfs are the ones written by the command, and should amount to 299Mb. Additionally, the container does not write any logs (through stdout), so that value should be 0.
I ran a quick test on a vm with the current image, and found that both and correctly wrote and measured disk space correctly. I also tested as it inside a container, and verified that we are measuring the container's writable layer correctly. My guess now is that it has something to do with how we calculate disk usage by logging with journald. I am looking into it
Actually, the previous comment was incorrect. When I ran the test, I found that each file was 1028 kb instead of 1024 kb, which resulted in the command writing 300.3 Mb instead of 299 Mb. I will change the test to require slightly less precision. This appears to be because an empty file in coreos takes up 4 kb. Test this by running . On other OS we test on, empty files use 0 kb. This is what accounts for the difference.
I tracked down an issue to the dd call. The stdout and stderr need a redirect to /dev/null to suppress the following messages: Problem Line: Fix:
I PRed that fix and changed it to stderr, since the other tests seem to use stderr.
leaving this open until we get a green run
I'll continue to test locally to see if I find anything else.
opened PR with the help of and CoreOS defaults to using the actual size instead of the apprent size.

}, { evictionPriority: 0, <del> pod: diskConsumingPod("guaranteed-disk", 299 /* Mb */, nil, v1.ResourceRequirements{ </del> <ins> // Only require 99% accuracy (297/300 Mb) because on some OS distributions, the file itself (excluding contents), consumes disk space. pod: diskConsumingPod("guaranteed-disk", 297 /* Mb */, nil, v1.ResourceRequirements{ </ins> Requests: v1.ResourceList{ v1.ResourceEphemeralStorage: resource.MustParse("300Mi"), },


Question:
We should have e2e feature for resizing in GCE environment. Also add unit tests for node restrictions.
/sig storage /assign

"persistent_volumes-vsphere.go", "pv_reclaimpolicy.go", "pvc_label_selector.go", <ins> "volume_expand.go", </ins> "volume_io.go", "volume_metrics.go", "volume_provisioning.go",


Question:
We should have e2e feature for resizing in GCE environment. Also add unit tests for node restrictions.
/sig storage /assign

<ins> /* Copyright 2017 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package storage import ( "fmt" "time" . "github.com/onsi/ginkgo" . "github.com/onsi/gomega" "k8s.io/api/core/v1" storage "k8s.io/api/storage/v1" "k8s.io/apimachinery/pkg/api/resource" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/util/wait" clientset "k8s.io/client-go/kubernetes" "k8s.io/kubernetes/test/e2e/framework" ) const ( resizePollInterval = 2 * time.Second // total time to wait for cloudprovider or file system resize to finish totalResizeWaitPeriod = 20 * time.Minute ) var _ = SIGDescribe("Volume expand [Feature:ExpandPersistentVolumes] [Slow]", func() { var ( c           clientset.Interface ns          string err         error pvc         *v1.PersistentVolumeClaim resizableSc *storage.StorageClass ) f := framework.NewDefaultFramework("volume-expand") BeforeEach(func() { framework.SkipUnlessProviderIs("aws", "gce") c = f.ClientSet ns = f.Namespace.Name framework.ExpectNoError(framework.WaitForAllNodesSchedulable(c, framework.TestContext.NodeSchedulableTimeout)) test := storageClassTest{ name:      "default", claimSize: "2Gi", } resizableSc, err = createResizableStorageClass(test, ns, "resizing", c) Expect(err).NotTo(HaveOccurred(), "Error creating resizable storage class") Expect(*resizableSc.AllowVolumeExpansion).To(BeTrue()) pvc = newClaim(test, ns, "default") pvc.Spec.StorageClassName = &resizableSc.Name pvc, err = c.CoreV1().PersistentVolumeClaims(pvc.Namespace).Create(pvc) Expect(err).NotTo(HaveOccurred(), "Error creating pvc") }) AfterEach(func() { framework.ExpectNoError(framework.DeletePersistentVolumeClaim(c, pvc.Name, pvc.Namespace)) framework.ExpectNoError(c.StorageV1().StorageClasses().Delete(resizableSc.Name, nil)) }) It("Verify if editing PVC allows resize", func() { By("Waiting for pvc to be in bound phase") pvcClaims := []*v1.PersistentVolumeClaim{pvc} pvs, err := framework.WaitForPVClaimBoundPhase(c, pvcClaims, framework.ClaimProvisionTimeout) Expect(err).NotTo(HaveOccurred(), "Failed waiting for PVC to be bound %v", err) Expect(len(pvs)).To(Equal(1)) By("Creating a pod with dynamically provisioned volume") pod, err := framework.CreatePod(c, ns, nil, pvcClaims, false, "") Expect(err).NotTo(HaveOccurred(), "While creating pods for resizing") defer func() { err = framework.DeletePodWithWait(f, c, pod) Expect(err).NotTo(HaveOccurred(), "while cleaning up pod already deleted in resize test") }() By("Expanding current pvc") newSize := resource.MustParse("6Gi") pvc, err = expandPVCSize(pvc, newSize, c) Expect(err).NotTo(HaveOccurred(), "While updating pvc for more size") Expect(pvc).NotTo(BeNil()) pvcSize := pvc.Spec.Resources.Requests[v1.ResourceStorage] if pvcSize.Cmp(newSize) != 0 { framework.Failf("error updating pvc size %q", pvc.Name) } By("Waiting for cloudprovider resize to finish") err = waitForControllerVolumeResize(pvc, c) Expect(err).NotTo(HaveOccurred(), "While waiting for pvc resize to finish") By("Checking for conditions on pvc") pvc, err = c.CoreV1().PersistentVolumeClaims(ns).Get(pvc.Name, metav1.GetOptions{}) Expect(err).NotTo(HaveOccurred(), "While fetching pvc after controller resize") inProgressConditions := pvc.Status.Conditions Expect(len(inProgressConditions)).To(Equal(1), "pvc must have resize condition") Expect(inProgressConditions[0].Type).To(Equal(v1.PersistentVolumeClaimResizing), "pvc must have resizing condition") By("Deleting the previously created pod") err = framework.DeletePodWithWait(f, c, pod) Expect(err).NotTo(HaveOccurred(), "while deleting pod for resizing") By("Creating a new pod with same volume") pod2, err := framework.CreatePod(c, ns, nil, pvcClaims, false, "") Expect(err).NotTo(HaveOccurred(), "while recreating pod for resizing") defer func() { err = framework.DeletePodWithWait(f, c, pod2) Expect(err).NotTo(HaveOccurred(), "while cleaning up pod before exiting resizing test") }() By("Waiting for file system resize to finish") pvc, err = waitForFSResize(pvc, c) Expect(err).NotTo(HaveOccurred(), "while waiting for fs resize to finish") pvcConditions := pvc.Status.Conditions Expect(len(pvcConditions)).To(Equal(0), "pvc should not have conditions") }) }) func createResizableStorageClass(t storageClassTest, ns string, suffix string, c clientset.Interface) (*storage.StorageClass, error) { stKlass := newStorageClass(t, ns, suffix) allowExpansion := true stKlass.AllowVolumeExpansion = &allowExpansion var err error stKlass, err = c.StorageV1().StorageClasses().Create(stKlass) return stKlass, err } func expandPVCSize(origPVC *v1.PersistentVolumeClaim, size resource.Quantity, c clientset.Interface) (*v1.PersistentVolumeClaim, error) { pvcName := origPVC.Name updatedPVC := origPVC.DeepCopy() waitErr := wait.PollImmediate(resizePollInterval, 30*time.Second, func() (bool, error) { var err error updatedPVC, err = c.CoreV1().PersistentVolumeClaims(origPVC.Namespace).Get(pvcName, metav1.GetOptions{}) if err != nil { return false, fmt.Errorf("error fetching pvc %q for resizing with %v", pvcName, err) } updatedPVC.Spec.Resources.Requests[v1.ResourceStorage] = size updatedPVC, err = c.CoreV1().PersistentVolumeClaims(origPVC.Namespace).Update(updatedPVC) if err == nil { return true, nil } framework.Logf("Error updating pvc %s with %v", pvcName, err) return false, nil }) return updatedPVC, waitErr } func waitForControllerVolumeResize(pvc *v1.PersistentVolumeClaim, c clientset.Interface) error { pvName := pvc.Spec.VolumeName return wait.PollImmediate(resizePollInterval, totalResizeWaitPeriod, func() (bool, error) { pvcSize := pvc.Spec.Resources.Requests[v1.ResourceStorage] pv, err := c.CoreV1().PersistentVolumes().Get(pvName, metav1.GetOptions{}) if err != nil { return false, fmt.Errorf("error fetching pv %q for resizing %v", pvName, err) } pvSize := pv.Spec.Capacity[v1.ResourceStorage] // If pv size is greater or equal to requested size that means controller resize is finished. if pvSize.Cmp(pvcSize) >= 0 { return true, nil } return false, nil }) } func waitForFSResize(pvc *v1.PersistentVolumeClaim, c clientset.Interface) (*v1.PersistentVolumeClaim, error) { var updatedPVC *v1.PersistentVolumeClaim waitErr := wait.PollImmediate(resizePollInterval, totalResizeWaitPeriod, func() (bool, error) { var err error updatedPVC, err = c.CoreV1().PersistentVolumeClaims(pvc.Namespace).Get(pvc.Name, metav1.GetOptions{}) if err != nil { return false, fmt.Errorf("error fetching pvc %q for checking for resize status : %v", pvc.Name, err) } pvcSize := updatedPVC.Spec.Resources.Requests[v1.ResourceStorage] pvcStatusSize := updatedPVC.Status.Capacity[v1.ResourceStorage] //If pvc's status field size is greater than or equal to pvc's size then done if pvcStatusSize.Cmp(pvcSize) >= 0 { return true, nil } return false, nil }) return updatedPVC, waitErr } </ins>


Question:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: kubelet throws the exception when --cloud-provider=cloudstack is included: What you expected to happen: There shouldn't be an exception. How to reproduce it (as minimally and precisely as possible): Add the following line to /etc/systemd/system/kubelet.service.d/10- Environment="KUBELETEXTRAARGS=--cloud-provider=cloudstack --cloud-" and also configure the file. Anything else we need to know?: The condition check in the InstanceType() function in is wrong. It currently reads: but the correct code is (condition negated): Environment: Kubernetes version (use ): 1.8 Cloud provider or hardware configuration: cloudstack OS (e.g. from /etc/os-release): Ubuntu 16.04.3 LTS (Xenial Xerus) Kernel (e.g. ): 4.4.0-98-generic Install tools: kubeadm Others: /sig cloudstack
I will have a look
/sig cluster-lifecycle /area cloudprovider

// InstanceType returns the type of the specified instance. func (m *metadata) InstanceType(name types.NodeName) (string, error) { instanceType, err := m.get(metadataTypeInstanceType) <del> if err == nil { </del> <ins> if err != nil { </ins> return "", fmt.Errorf("could not get instance type: %v", err) }


Question:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: The test has been failing since Nov 22nd. Looks like it is because the test restarts Kubelet and then calls getLocalNode() which checks for node readiness. Not sure why this problem only started to show up recently. What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): 1.9 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): COS Kernel (e.g. ): Install tools: Others:
/sig node
Hi, Just FYI, . Maybe it is helpful for you.

Skip("Nvidia GPUs do not exist on the node. Skipping test.") } <ins> framework.WaitForAllNodesSchedulable(f.ClientSet, framework.TestContext.NodeSchedulableTimeout) </ins> By("Creating the Google Device Plugin pod for NVIDIA GPU in GKE") devicePluginPod = f.PodClient().CreateSync(framework.NVIDIADevicePlugin(f.Namespace.Name))


Question:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: The test has been failing since Nov 22nd. Looks like it is because the test restarts Kubelet and then calls getLocalNode() which checks for node readiness. Not sure why this problem only started to show up recently. What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): 1.9 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): COS Kernel (e.g. ): Install tools: Others:
/sig node
Hi, Just FYI, . Maybe it is helpful for you.

f.PodClient().Delete(devicePluginPod.Name, &metav1.DeleteOptions{}) By("Waiting for GPUs to become unavailable on the local node") Eventually(func() bool { <del> return framework.NumberOfNVIDIAGPUs(getLocalNode(f)) <= 0 </del> <ins> node, err := f.ClientSet.CoreV1().Nodes().Get(framework.TestContext.NodeName, metav1.GetOptions{}) framework.ExpectNoError(err) return framework.NumberOfNVIDIAGPUs(node) <= 0 </ins> }, 10*time.Minute, framework.Poll).Should(BeTrue()) By("Checking that scheduled pods can continue to run even after we delete device plugin.") count1, devIdRestart1 = getDeviceId(f, p1.Name, p1.Name, count1+1)


Question: We should not attempt resizing of file system if volume is mounted as read-only. /sig storage cc

glog.V(6).Infof("Resizing is not enabled for this volume %s", volumeToMount.VolumeName) return nil } mounter := og.volumePluginMgr.Host.GetMounter(pluginName) // Get expander, if possible expandableVolumePlugin, _ :=


Question: We should not attempt resizing of file system if volume is mounted as read-only. /sig storage cc

// File system resize was requested, proceed glog.V(4).Infof(volumeToMount.GenerateMsgDetailed("MountVolume.resizeFileSystem entering", fmt.Sprintf("DevicePath %q", volumeToMount.DevicePath))) <ins> if volumeToMount.VolumeSpec.ReadOnly { simpleMsg, detailedMsg := volumeToMount.GenerateMsg("MountVolume.resizeFileSystem failed", "requested read-only file system") glog.Warningf(detailedMsg) og.recorder.Eventf(volumeToMount.Pod, v1.EventTypeWarning, kevents.FileSystemResizeFailed, simpleMsg) return nil } </ins> diskFormatter := &mount.SafeFormatAndMount{ Interface: mounter, Exec:      og.volumePluginMgr.Host.GetExec(expandableVolumePlugin.GetPluginName()),


Question: We should not attempt resizing of file system if volume is mounted as read-only. /sig storage cc

} pvcCopy.Status.Capacity = capacity <ins> pvcCopy.Status.Conditions = []v1.PersistentVolumeClaimCondition{} </ins> newData, err := json.Marshal(pvcCopy) if err != nil {


Question:
/kind feature What happened: The DevicePlugins feature was introduced in 1.8 Kubernetes release as an alpha feature. Since then, we have seen good development work on stabling the feature as well as early adopters on using the framework to support device resources in Kubernetes. This issue is to track the work requirements of entering this feature into beta. FYI, here is list of requirements from the OSS Kubernetes feature stage doc: """ Beta level: Object Versioning: API version name contains beta (e.g. v2beta3) Availability: in official Kubernetes releases, and enabled by default Audience: users interested in providing feedback on features Completeness: all API operations, CLI commands, and UI support should be implemented; end-to-end tests complete; the API has had a thorough API review and is thought to be complete, though use during beta may frequently turn up API issues not thought of during review Upgradeability: the object schema and semantics may change in a later software release; when this happens, an upgrade path will be documented; in some cases, objects will be automatically converted to the new version; in other cases, a manual upgrade may be necessary; a manual upgrade may require downtime for anything relying on the new feature, and may require manual conversion of objects to the new version; when manual conversion is necessary, the project will provide documentation on the process Cluster Reliability: since the feature has e2e tests, enabling the feature via a flag should not create new bugs in unrelated features; because the feature is new, it may have minor bugs Support: the project commits to complete the feature, in some form, in a subsequent Stable version; typically this will happen within 3 months, but sometimes longer; releases should simultaneously support two consecutive versions (e.g. v1beta1 and v1beta2; or v1beta2 and v1) for at least one minor release cycle (typically 3 months) so that users have enough time to upgrade and migrate objects Recommended Use Cases: in short-lived testing clusters; in production clusters as part of a short-lived evaluation of the feature in order to provide feedback """ I think we met most of the requirements here, with the commit that if we need to make incompatible API change in the future, we will document the upgrade path and if necessary, may support multiple API versions in a couple of releases. Things that I think would be nice to have to enter beta (entering beta may not necessary depends on their completeness) Regarding to the thorough API review requirement, I think we had good discussions during the design proposal but there have been some recent discussions on whether we should use registration vs probe model for the general Kubelet plugin communication. I think we should have a general discussion on this topic with larger audience like people working on CSI plugins. I will create a FR issue to track this discussion soon. We would like to see multiple device plugin implementations. We know some people have started working on device plugin implementations for Solarflare, FPGA, and Infiniband, and probably want to keep closer track on their progresses. We would like to target this FR to 1.10. Please let us know your thoughts or add things that you think are critical to finish. Thanks! What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig node
I'm a fan of probe model. I do feel it unfriendly to plugin developers to let them care about the re-registration after kubelet's restart. And have received complaint from my colleague.
Hi, for cluster reliability, I think we need more test cases, including scale and soak tests, to simulate more production scenarios.
can we consolidate this with ?
Sure. Will do this with Vish after KubeCon.
Sorry to bring up this topic a bit late (feature freeze is the 22nd of Jan). Here are the features we (NVIDIA) consider important or blocker before entering beta: [BLOCKER] Per container allocation Add labels during device plugin registration (e.g: gpu-family: tesla) Add annotations to a container for beta and add an option to CRI for GA to support OCI hooks Things that are not explicitly related to the device plugin: We're also wondering if we should tie up
Thanks a lot for your input, For per container allocation, I think we should be able to implement it in 1.10. I think we should be able to support ER quota in 1.10 as well. I think allowing device plugins to export device properties is an important feature, especially when we introduce ResourceClass, but I don't think it is a beta blocker. For "Add annotations to a container for beta and add an option to CRI for GA to support OCI hooks", this seems outside the scope of device plugin. There are different ways to implement them (if they have to be implemented) on Kubernetes. We haven't seen strong use cases indicating that they have to go through device plugin.
Sorry, I meant from the device plugin. i.e: the device plugin's returns container annotations for beta Product wise this would allow us to have the nvidia-runtime be supported by CRI-O (which allows hooks through container annotations) as well as docker + nvidia as the default runtime. BTW That would allow a convergence towards a single device plugin in beta. As for GA this would be a different discussion but it would be part of converging towards using any runtime and have them support the nvidia-hooks.
Not saying it's a beta blocker but it would be useful to have labels on homogeneous nodes about what devices are present. Especially: GPU memory GPU compute capability ... It's also not a big change, it just requires consensus
As a heads up, I am going to send a PR soon to change DevicePlugins feature to beta. After introducing the feature in 1.8, we have seen good development work on improving the feature and received many valuable feedbacks from early adopters on how we may extend the framework to support various kinds of resources on Kubernetes. To us, entering beta means: a) We think we have done enough early experiments during alpha phase to get the confidence that this a useful feature that we are willing to carry on to beta and to GA. b) We think the current DevicePlugins is ready to be tried on production. It has good e2e and e2enode test coverage. Some early adopters have tried it on production and didn't report any major breakage. Very likely we will keep seeing new feature requests coming in after beta, as more people try to use DevicePlugins for their specific use cases. Some of these features may even require incompatible API changes. When this happens, the required complexity to support multiple device plugin API versions on Kubelet side should be manageable and mostly local to the devicemanager component. It should also be manageable for a device plugin to support multiple API versions. As an example, in , I extended gke gpu device plugin to support both v1alpha and v1beta1 versions. For reference, here is list of changes we made on device plugin during 1.10: Support resource quota on extended resources (Issue ). Owner: Added PreStartContainer grpc function to allow device plugins to perform per container start operations. Owner: Add annotations to the device plugin API. Owner: Track unhealthy devices in devicemanager, and have node status capacity to reflect raw capacity but allocatable to reflect healthy device capacity. Owner: Stub device plugin based e2enode test Owner: There are a few pending PRs that would be nice to get into 1.10 but we don't consider as beta blocker: Determine and implement a general Kubelet plugin communication establish model and use it in device plugin. Owner: Uses common checkpointing mechanism in device plugin (PR Issue ) Owner: Device plugin stress test. Owner: Please let us know if anyone has objections on entering the feature to beta in 1.10. Thanks!
Hi, for stress test, here is a PR :)
/cc
Should we reopen this tracing issue?
I think this issue can be closed with We can open separate issues to track future device plugin changes.

Accelerators utilfeature.Feature = "Accelerators" // owner: @jiayingz <del> // alpha: v1.8 </del> <ins> // beta: v1.10 </ins> // // Enables support for Device Plugins <del> // Only Nvidia GPUs are tested as of v1.8. </del> DevicePlugins utilfeature.Feature = "DevicePlugins" // owner: @gmarek


Question:
/kind feature What happened: The DevicePlugins feature was introduced in 1.8 Kubernetes release as an alpha feature. Since then, we have seen good development work on stabling the feature as well as early adopters on using the framework to support device resources in Kubernetes. This issue is to track the work requirements of entering this feature into beta. FYI, here is list of requirements from the OSS Kubernetes feature stage doc: """ Beta level: Object Versioning: API version name contains beta (e.g. v2beta3) Availability: in official Kubernetes releases, and enabled by default Audience: users interested in providing feedback on features Completeness: all API operations, CLI commands, and UI support should be implemented; end-to-end tests complete; the API has had a thorough API review and is thought to be complete, though use during beta may frequently turn up API issues not thought of during review Upgradeability: the object schema and semantics may change in a later software release; when this happens, an upgrade path will be documented; in some cases, objects will be automatically converted to the new version; in other cases, a manual upgrade may be necessary; a manual upgrade may require downtime for anything relying on the new feature, and may require manual conversion of objects to the new version; when manual conversion is necessary, the project will provide documentation on the process Cluster Reliability: since the feature has e2e tests, enabling the feature via a flag should not create new bugs in unrelated features; because the feature is new, it may have minor bugs Support: the project commits to complete the feature, in some form, in a subsequent Stable version; typically this will happen within 3 months, but sometimes longer; releases should simultaneously support two consecutive versions (e.g. v1beta1 and v1beta2; or v1beta2 and v1) for at least one minor release cycle (typically 3 months) so that users have enough time to upgrade and migrate objects Recommended Use Cases: in short-lived testing clusters; in production clusters as part of a short-lived evaluation of the feature in order to provide feedback """ I think we met most of the requirements here, with the commit that if we need to make incompatible API change in the future, we will document the upgrade path and if necessary, may support multiple API versions in a couple of releases. Things that I think would be nice to have to enter beta (entering beta may not necessary depends on their completeness) Regarding to the thorough API review requirement, I think we had good discussions during the design proposal but there have been some recent discussions on whether we should use registration vs probe model for the general Kubelet plugin communication. I think we should have a general discussion on this topic with larger audience like people working on CSI plugins. I will create a FR issue to track this discussion soon. We would like to see multiple device plugin implementations. We know some people have started working on device plugin implementations for Solarflare, FPGA, and Infiniband, and probably want to keep closer track on their progresses. We would like to target this FR to 1.10. Please let us know your thoughts or add things that you think are critical to finish. Thanks! What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig node
I'm a fan of probe model. I do feel it unfriendly to plugin developers to let them care about the re-registration after kubelet's restart. And have received complaint from my colleague.
Hi, for cluster reliability, I think we need more test cases, including scale and soak tests, to simulate more production scenarios.
can we consolidate this with ?
Sure. Will do this with Vish after KubeCon.
Sorry to bring up this topic a bit late (feature freeze is the 22nd of Jan). Here are the features we (NVIDIA) consider important or blocker before entering beta: [BLOCKER] Per container allocation Add labels during device plugin registration (e.g: gpu-family: tesla) Add annotations to a container for beta and add an option to CRI for GA to support OCI hooks Things that are not explicitly related to the device plugin: We're also wondering if we should tie up
Thanks a lot for your input, For per container allocation, I think we should be able to implement it in 1.10. I think we should be able to support ER quota in 1.10 as well. I think allowing device plugins to export device properties is an important feature, especially when we introduce ResourceClass, but I don't think it is a beta blocker. For "Add annotations to a container for beta and add an option to CRI for GA to support OCI hooks", this seems outside the scope of device plugin. There are different ways to implement them (if they have to be implemented) on Kubernetes. We haven't seen strong use cases indicating that they have to go through device plugin.
Sorry, I meant from the device plugin. i.e: the device plugin's returns container annotations for beta Product wise this would allow us to have the nvidia-runtime be supported by CRI-O (which allows hooks through container annotations) as well as docker + nvidia as the default runtime. BTW That would allow a convergence towards a single device plugin in beta. As for GA this would be a different discussion but it would be part of converging towards using any runtime and have them support the nvidia-hooks.
Not saying it's a beta blocker but it would be useful to have labels on homogeneous nodes about what devices are present. Especially: GPU memory GPU compute capability ... It's also not a big change, it just requires consensus
As a heads up, I am going to send a PR soon to change DevicePlugins feature to beta. After introducing the feature in 1.8, we have seen good development work on improving the feature and received many valuable feedbacks from early adopters on how we may extend the framework to support various kinds of resources on Kubernetes. To us, entering beta means: a) We think we have done enough early experiments during alpha phase to get the confidence that this a useful feature that we are willing to carry on to beta and to GA. b) We think the current DevicePlugins is ready to be tried on production. It has good e2e and e2enode test coverage. Some early adopters have tried it on production and didn't report any major breakage. Very likely we will keep seeing new feature requests coming in after beta, as more people try to use DevicePlugins for their specific use cases. Some of these features may even require incompatible API changes. When this happens, the required complexity to support multiple device plugin API versions on Kubelet side should be manageable and mostly local to the devicemanager component. It should also be manageable for a device plugin to support multiple API versions. As an example, in , I extended gke gpu device plugin to support both v1alpha and v1beta1 versions. For reference, here is list of changes we made on device plugin during 1.10: Support resource quota on extended resources (Issue ). Owner: Added PreStartContainer grpc function to allow device plugins to perform per container start operations. Owner: Add annotations to the device plugin API. Owner: Track unhealthy devices in devicemanager, and have node status capacity to reflect raw capacity but allocatable to reflect healthy device capacity. Owner: Stub device plugin based e2enode test Owner: There are a few pending PRs that would be nice to get into 1.10 but we don't consider as beta blocker: Determine and implement a general Kubelet plugin communication establish model and use it in device plugin. Owner: Uses common checkpointing mechanism in device plugin (PR Issue ) Owner: Device plugin stress test. Owner: Please let us know if anyone has objections on entering the feature to beta in 1.10. Thanks!
Hi, for stress test, here is a PR :)
/cc
Should we reopen this tracing issue?
I think this issue can be closed with We can open separate issues to track future device plugin changes.

ExperimentalHostUserNamespaceDefaultingGate: {Default: false, PreRelease: utilfeature.Beta}, ExperimentalCriticalPodAnnotation:           {Default: false, PreRelease: utilfeature.Alpha}, Accelerators:                                {Default: false, PreRelease: utilfeature.Alpha}, <del> DevicePlugins:                               {Default: false, PreRelease: utilfeature.Alpha}, </del> <ins> DevicePlugins:                               {Default: true, PreRelease: utilfeature.Beta}, </ins> TaintBasedEvictions:                         {Default: false, PreRelease: utilfeature.Alpha}, RotateKubeletServerCertificate:              {Default: false, PreRelease: utilfeature.Alpha}, RotateKubeletClientCertificate:              {Default: true, PreRelease: utilfeature.Beta},


Question:
Internal volume plugin for CSI creates VolumeAttachment with (= the plugin name). It should be CSI driver name instead, as the external attacher must know which CSI driver to call. /sig storage /kind bug please prioritize PTAL. It should be one-line change in
[MILESTONENOTIFIER] Milestone Issue Current Note: This issue is marked as , and must be updated every 1 day during code freeze. Example update: <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
ACK. In progress

}, Spec: storage.VolumeAttachmentSpec{ NodeName: node, <del> Attacher: csiPluginName, </del> <ins> Attacher: csiSource.Driver, </ins> Source: storage.VolumeAttachmentSource{ PersistentVolumeName: &pvName, },


Question:
Internal volume plugin for CSI creates VolumeAttachment with (= the plugin name). It should be CSI driver name instead, as the external attacher must know which CSI driver to call. /sig storage /kind bug please prioritize PTAL. It should be one-line change in
[MILESTONENOTIFIER] Milestone Issue Current Note: This issue is marked as , and must be updated every 1 day during code freeze. Example update: <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
ACK. In progress

}, Spec: storage.VolumeAttachmentSpec{ NodeName: nodeName, <del> Attacher: csiPluginName, </del> <ins> Attacher: "mock", </ins> Source: storage.VolumeAttachmentSource{ PersistentVolumeName: &pvName, },


Question:
BUG REPORT While configuring the kubelet with: And having the sock file on that location (default location from cri), the following error occurs: Later line: Uses the correct socket. Result: Disruptive in understanding other failures. Containers still start. Environment: Kubernetes version: 1.8.0-5, 1.9.0-beta.2 Cloud provider: gce OS: Ubuntu 16.04 Kernel (e.g. ): Linux kubeworker01 4.10.0-42-generic ~16.04.1-Ubuntu SMP Mon Dec 4 15:57:59 UTC 2017 x8664 x8664 x8664 GNU/Linux Install tools: The hard way Others: Full kubelet config (as of now): Internal log reference: syslog.v6.3
/area [kubelet] /sig [node]
/area kubelet /sig node /assign
This is because uses , while is using as default. We should make them consistent. Fixed in . PTAL.
What are the consequences? Is it more than cosmetical? Does it affect the usage of CRI-O in 1.9.0 or how does it affect users?
/assign Is this relevant for CRI-O in 1.9.0?
It (at this moment) mostly looks cosmetical, however when I add a cloud provider (--cloud-provider=gce in this case) to the kubelet config, the whole thing breaks consistently on PLEG. I am trying to figure out if there is a relation between that and the
Can you weigh in on whether this is a release-blocker for 1.9.0?
Tentatively adding milestone so we can track getting a judgement from sig-node.
-- i think this is fixed in the packaging, let me close with and
fixed in packaging yeah, from 1.8 we moved to "" so you need to start kubelet with that socket path
Discussed at sig-node earlier today, and this is not a release blocker. Also like mentioned above, the issue was fixed in the packaging.
There is a reference that needs to be fixed for the new socket location. The referenced pr is required to ensure stats summary endpoint understands the right location, and this keeps it consistent with cadvisor after its rebase in 1.9.
/priority critical-urgent /kind bug /sig node /status approved-for-milestone
/status in-progress
[MILESTONENOTIFIER] Milestone Issue Current Note: This issue is marked as , and must be updated every 1 day during code freeze. Example update: <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
Hi, I am hitting the same error in this thread on my Openshift4.2 cluster which has configured CRIO on the kubelet. I would like to hit to get inventory of the containers. Any body has any insights why I am hitting this error when I make curl calls from the k8s node which has configured for the CRIO? [root /]# curl -v --unix-socket http://localhost/info Trying Immediate connect fail for No such file or directory Closing connection 0 curl: (7) Couldn't connect to server [root /]# sshadmin kubectl version Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2020-01-14T21:04:32Z", GoVersion:"go1.13.5", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"14+", GitVersion:"v1.14.6+", GitCommit:"", GitTreeState:"clean", BuildDate:"2020-01-17T17:43:42Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"

// This is a temporary workaround to get stats for cri-o from cadvisor // and should be removed. // Related to https://github.com/kubernetes/kubernetes/issues/51798 <del> if i.runtimeEndpoint == "/var/run/crio.sock" { </del> <ins> if i.runtimeEndpoint == CrioSocket { </ins> return cadvisorfs.LabelCrioImages, nil } }


Question:
BUG REPORT While configuring the kubelet with: And having the sock file on that location (default location from cri), the following error occurs: Later line: Uses the correct socket. Result: Disruptive in understanding other failures. Containers still start. Environment: Kubernetes version: 1.8.0-5, 1.9.0-beta.2 Cloud provider: gce OS: Ubuntu 16.04 Kernel (e.g. ): Linux kubeworker01 4.10.0-42-generic ~16.04.1-Ubuntu SMP Mon Dec 4 15:57:59 UTC 2017 x8664 x8664 x8664 GNU/Linux Install tools: The hard way Others: Full kubelet config (as of now): Internal log reference: syslog.v6.3
/area [kubelet] /sig [node]
/area kubelet /sig node /assign
This is because uses , while is using as default. We should make them consistent. Fixed in . PTAL.
What are the consequences? Is it more than cosmetical? Does it affect the usage of CRI-O in 1.9.0 or how does it affect users?
/assign Is this relevant for CRI-O in 1.9.0?
It (at this moment) mostly looks cosmetical, however when I add a cloud provider (--cloud-provider=gce in this case) to the kubelet config, the whole thing breaks consistently on PLEG. I am trying to figure out if there is a relation between that and the
Can you weigh in on whether this is a release-blocker for 1.9.0?
Tentatively adding milestone so we can track getting a judgement from sig-node.
-- i think this is fixed in the packaging, let me close with and
fixed in packaging yeah, from 1.8 we moved to "" so you need to start kubelet with that socket path
Discussed at sig-node earlier today, and this is not a release blocker. Also like mentioned above, the issue was fixed in the packaging.
There is a reference that needs to be fixed for the new socket location. The referenced pr is required to ensure stats summary endpoint understands the right location, and this keeps it consistent with cadvisor after its rebase in 1.9.
/priority critical-urgent /kind bug /sig node /status approved-for-milestone
/status in-progress
[MILESTONENOTIFIER] Milestone Issue Current Note: This issue is marked as , and must be updated every 1 day during code freeze. Example update: <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
Hi, I am hitting the same error in this thread on my Openshift4.2 cluster which has configured CRIO on the kubelet. I would like to hit to get inventory of the containers. Any body has any insights why I am hitting this error when I make curl calls from the k8s node which has configured for the CRIO? [root /]# curl -v --unix-socket http://localhost/info Trying Immediate connect fail for No such file or directory Closing connection 0 curl: (7) Couldn't connect to server [root /]# sshadmin kubectl version Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2020-01-14T21:04:32Z", GoVersion:"go1.13.5", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"14+", GitVersion:"v1.14.6+", GitCommit:"", GitTreeState:"clean", BuildDate:"2020-01-17T17:43:42Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"

kubetypes "k8s.io/kubernetes/pkg/kubelet/types" ) <ins> const ( // Please keep this in sync with the one in: // github.com/google/cadvisor/container/crio/client.go CrioSocket = "/var/run/crio/crio.sock" ) </ins> func CapacityFromMachineInfo(info *cadvisorapi.MachineInfo) v1.ResourceList { c := v1.ResourceList{ v1.ResourceCPU: *resource.NewMilliQuantity(


Question:
BUG REPORT While configuring the kubelet with: And having the sock file on that location (default location from cri), the following error occurs: Later line: Uses the correct socket. Result: Disruptive in understanding other failures. Containers still start. Environment: Kubernetes version: 1.8.0-5, 1.9.0-beta.2 Cloud provider: gce OS: Ubuntu 16.04 Kernel (e.g. ): Linux kubeworker01 4.10.0-42-generic ~16.04.1-Ubuntu SMP Mon Dec 4 15:57:59 UTC 2017 x8664 x8664 x8664 GNU/Linux Install tools: The hard way Others: Full kubelet config (as of now): Internal log reference: syslog.v6.3
/area [kubelet] /sig [node]
/area kubelet /sig node /assign
This is because uses , while is using as default. We should make them consistent. Fixed in . PTAL.
What are the consequences? Is it more than cosmetical? Does it affect the usage of CRI-O in 1.9.0 or how does it affect users?
/assign Is this relevant for CRI-O in 1.9.0?
It (at this moment) mostly looks cosmetical, however when I add a cloud provider (--cloud-provider=gce in this case) to the kubelet config, the whole thing breaks consistently on PLEG. I am trying to figure out if there is a relation between that and the
Can you weigh in on whether this is a release-blocker for 1.9.0?
Tentatively adding milestone so we can track getting a judgement from sig-node.
-- i think this is fixed in the packaging, let me close with and
fixed in packaging yeah, from 1.8 we moved to "" so you need to start kubelet with that socket path
Discussed at sig-node earlier today, and this is not a release blocker. Also like mentioned above, the issue was fixed in the packaging.
There is a reference that needs to be fixed for the new socket location. The referenced pr is required to ensure stats summary endpoint understands the right location, and this keeps it consistent with cadvisor after its rebase in 1.9.
/priority critical-urgent /kind bug /sig node /status approved-for-milestone
/status in-progress
[MILESTONENOTIFIER] Milestone Issue Current Note: This issue is marked as , and must be updated every 1 day during code freeze. Example update: <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
Hi, I am hitting the same error in this thread on my Openshift4.2 cluster which has configured CRIO on the kubelet. I would like to hit to get inventory of the containers. Any body has any insights why I am hitting this error when I make curl calls from the k8s node which has configured for the CRIO? [root /]# curl -v --unix-socket http://localhost/info Trying Immediate connect fail for No such file or directory Closing connection 0 curl: (7) Couldn't connect to server [root /]# sshadmin kubectl version Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2020-01-14T21:04:32Z", GoVersion:"go1.13.5", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"14+", GitVersion:"v1.14.6+", GitCommit:"", GitTreeState:"clean", BuildDate:"2020-01-17T17:43:42Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"

func UsingLegacyCadvisorStats(runtime, runtimeEndpoint string) bool { return runtime == kubetypes.RktContainerRuntime || (runtime == kubetypes.DockerContainerRuntime && goruntime.GOOS == "linux") || <del> runtimeEndpoint == "/var/run/crio.sock" </del> <ins> runtimeEndpoint == CrioSocket </ins> }


Question:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: If a 1.8.x kubernetes master with a single controller-manager is configured with --leader-elect=false, deleting a deployment through the API with v1meta.DeletePropagationForeground doesn't work. The deployment has its deletionTimestamp set and a foregroundDeletion finalizer is , but the pod, rc and deployment are not deleted. What you expected to happen: The pod, rc and deployment should be deleted when the controller-manager uses --leader-elect=false. How to reproduce it (as minimally and precisely as possible): I used kubeadm with calico networking, and modified the controller-manager configuration via sudo sed -i /etc/kubernetes/manifests/kube-controller- -e '/leader-elect/ s/true/false/' and then ran to create and delete deployments. Anything else we need to know?: I suspect the problem is related to this change: It appears to require leader election before the GC controller will finish initialization. Looking at controller-manager logs with leader-elect=false, I saw this log entry from "GraphBuilder running" I did not see the "started %d new monitors, %d currently running" log entry at the end of the startMonitors() function. That seems consistent with the GC controller waiting for "<-gb.informersStarted" Environment: Kubernetes version (use ): tested with 1.8.2 and 1.8.4 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): vagrant virtualbox with the ubuntu/xenial64 box Kernel (e.g. ): Install tools: kubeadm Others:
cc Have you seen this in Openshift?
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
cc
I think this problem occurs because initialiuzes the stop channel top nil when leader-elect=false. This causes garbage collector graph_builder to exit early. The following release-1.8 patch seems to fix the problem. creates a dummy channel which will never be closed and allows the garbage collector graph builder to work like it does when leader-elect is true. [Edit - just noticed that my initial patch had some unexpected differences]
do you really need to remove the following code?
That change was not intended. I also noticed that after I saved my change. I redid the patch and edited my previous comment to reflect that.
Agree with the diagnosis and fix, are you going to open a PR?
Yes, I can do that. Reading your PR guidelines and ours.
Will I need to go through to get this into the release-1.8 and release-1.9 branches?

} if !s.LeaderElection.LeaderElect { <del> run(nil) </del> <ins> stopCh := make(chan struct{}) defer close(stopCh) run(stopCh) </ins> panic("unreachable") }


Question:
Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: When enable cpumanager by , but not set and , kubelet will get panic error in log What you expected to happen: Should graceful exit with error not panic. How to reproduce it (as minimally and precisely as possible): set and but not set and Anything else we need to know?:
/sig node
Great that it's not panicing anymore, but I'd also love to get the "requires + to be greater than zero" hint in this particular case. It looks like even with this doesn't seem to be the case (since it errors out with the non-actionable "unable to determine reserved CPU resources for static policy". Should I open another issue for this?

var _ Manager = &manager{} // NewManager creates new cpu manager based on provided policy <del> func NewManager( cpuPolicyName string, reconcilePeriod time.Duration, machineInfo *cadvisorapi.MachineInfo, nodeAllocatableReservation v1.ResourceList, stateFileDirecory string, ) (Manager, error) { </del> <ins> func NewManager(cpuPolicyName string, reconcilePeriod time.Duration, machineInfo *cadvisorapi.MachineInfo, nodeAllocatableReservation v1.ResourceList, stateFileDirecory string) (Manager, error) { </ins> var policy Policy switch policyName(cpuPolicyName) {


Question:
Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: When enable cpumanager by , but not set and , kubelet will get panic error in log What you expected to happen: Should graceful exit with error not panic. How to reproduce it (as minimally and precisely as possible): set and but not set and Anything else we need to know?:
/sig node
Great that it's not panicing anymore, but I'd also love to get the "requires + to be greater than zero" hint in this particular case. It looks like even with this doesn't seem to be the case (since it errors out with the non-actionable "unable to determine reserved CPU resources for static policy". Should I open another issue for this?

glog.Infof("[cpumanager] detected CPU topology: %v", topo) reservedCPUs, ok := nodeAllocatableReservation[v1.ResourceCPU] if !ok { <del> // The static policy cannot initialize without this information. Panic! panic("[cpumanager] unable to determine reserved CPU resources for static policy") </del> <ins> // The static policy cannot initialize without this information. return nil, fmt.Errorf("[cpumanager] unable to determine reserved CPU resources for static policy") </ins> } if reservedCPUs.IsZero() { <del> // Panic! // </del> // The static policy requires this to be nonzero. Zero CPU reservation // would allow the shared pool to be completely exhausted. At that point // either we would violate our guarantee of exclusivity or need to evict // any pod that has at least one container that requires zero CPUs. // See the comments in policy_static.go for more details. <del> panic("[cpumanager] the static policy requires systemreserved.cpu + kubereserved.cpu to be greater than zero") </del> <ins> return nil, fmt.Errorf("[cpumanager] the static policy requires systemreserved.cpu + kubereserved.cpu to be greater than zero") </ins> } // Take the ceiling of the reservation, since fractional CPUs cannot be


Question:
Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: When enable cpumanager by , but not set and , kubelet will get panic error in log What you expected to happen: Should graceful exit with error not panic. How to reproduce it (as minimally and precisely as possible): set and but not set and Anything else we need to know?:
/sig node
Great that it's not panicing anymore, but I'd also love to get the "requires + to be greater than zero" hint in this particular case. It looks like even with this doesn't seem to be the case (since it errors out with the non-actionable "unable to determine reserved CPU resources for static policy". Should I open another issue for this?

cpuPolicyName              string nodeAllocatableReservation v1.ResourceList isTopologyBroken           bool <del> panicMsg                   string </del> expectedPolicy             string expectedError              error skipIfPermissionsError     bool


Question:
Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: When enable cpumanager by , but not set and , kubelet will get panic error in log What you expected to happen: Should graceful exit with error not panic. How to reproduce it (as minimally and precisely as possible): set and but not set and Anything else we need to know?:
/sig node
Great that it's not panicing anymore, but I'd also love to get the "requires + to be greater than zero" hint in this particular case. It looks like even with this doesn't seem to be the case (since it errors out with the non-actionable "unable to determine reserved CPU resources for static policy". Should I open another issue for this?

description:                "static policy - broken reservation", cpuPolicyName:              "static", nodeAllocatableReservation: v1.ResourceList{}, <del> panicMsg:                   "unable to determine reserved CPU resources for static policy", </del> <ins> expectedError:              fmt.Errorf("unable to determine reserved CPU resources for static policy"), </ins> skipIfPermissionsError:     true, }, { description:                "static policy - no CPU resources", cpuPolicyName:              "static", nodeAllocatableReservation: v1.ResourceList{v1.ResourceCPU: *resource.NewQuantity(0, resource.DecimalSI)}, <del> panicMsg:                   "the static policy requires systemreserved.cpu + kubereserved.cpu to be greater than zero", </del> <ins> expectedError:              fmt.Errorf("the static policy requires systemreserved.cpu + kubereserved.cpu to be greater than zero"), </ins> skipIfPermissionsError:     true, }, }


Question:
Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: When enable cpumanager by , but not set and , kubelet will get panic error in log What you expected to happen: Should graceful exit with error not panic. How to reproduce it (as minimally and precisely as possible): set and but not set and Anything else we need to know?:
/sig node
Great that it's not panicing anymore, but I'd also love to get the "requires + to be greater than zero" hint in this particular case. It looks like even with this doesn't seem to be the case (since it errors out with the non-actionable "unable to determine reserved CPU resources for static policy". Should I open another issue for this?

t.Errorf("cannot create state file: %s", err.Error()) } defer os.RemoveAll(sDir) <del> defer func() { if err := recover(); err != nil { if testCase.panicMsg != "" { if !strings.Contains(err.(string), testCase.panicMsg) { t.Errorf("Unexpected panic message. Have: %q wants %q", err, testCase.panicMsg) } } else { t.Errorf("Unexpected panic: %q", err) } } else if testCase.panicMsg != "" { t.Error("Expected panic hasn't been raised") } }() </del> mgr, err := NewManager(testCase.cpuPolicyName, 5*time.Second, machineInfo, testCase.nodeAllocatableReservation, sDir) if testCase.expectedError != nil {


Question:
ref: and An e2e test should be , and the default should be updated. /assign cc
/assign
Thanks I can take this on.
Could someone point me to where the manifests are for the e2e test suites, I am happy to submit the patches?
Are there API server e2e tests currently?
there are not HA API server e2e tests that I'm aware of
I opened a PR for the default changes. Are there suggestions on a good E2E test to write if we don't have HA API server tests?
what is needed for this issue to complete? Do you think you'll get that by Monday, which is Code Freeze?
We need some E2E testing or add an integration test as defined . It will not be done today.
so are you planning to move it to 1.11, then? It's about to be automatically kicked out of 1.10 by automation.
I do not have permission to change the milestone. If you could kindly change it, that would be awesome!
[MILESTONENOTIFIER] Milestone Removed From Issue Important: This issue was missing the label for more than 7 days. <details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
Reopening as this was reverted in

EnableLogsHandler:      true, EventTTL:               1 * time.Hour, MasterCount:            1, <del> EndpointReconcilerType: string(reconcilers.MasterCountReconcilerType), </del> <ins> EndpointReconcilerType: string(reconcilers.LeaseEndpointReconcilerType), </ins> KubeletConfig: kubeletclient.KubeletClientConfig{ Port:         ports.KubeletPort, ReadOnlyPort: ports.KubeletReadOnlyPort,


Question:
ref: and An e2e test should be , and the default should be updated. /assign cc
/assign
Thanks I can take this on.
Could someone point me to where the manifests are for the e2e test suites, I am happy to submit the patches?
Are there API server e2e tests currently?
there are not HA API server e2e tests that I'm aware of
I opened a PR for the default changes. Are there suggestions on a good E2E test to write if we don't have HA API server tests?
what is needed for this issue to complete? Do you think you'll get that by Monday, which is Code Freeze?
We need some E2E testing or add an integration test as defined . It will not be done today.
so are you planning to move it to 1.11, then? It's about to be automatically kicked out of 1.10 by automation.
I do not have permission to change the milestone. If you could kindly change it, that would be awesome!
[MILESTONENOTIFIER] Milestone Removed From Issue Important: This issue was missing the label for more than 7 days. <details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
Reopening as this was reverted in

"Currently only applies to long-running requests.") fs.IntVar(&s.MasterCount, "apiserver-count", s.MasterCount, <del> "The number of apiservers running in the cluster, must be a positive number.") </del> <ins> "The number of apiservers running in the cluster, must be a positive number. (In use when --endpoint-reconciler-type=master-count is enabled.)") </ins> fs.StringVar(&s.EndpointReconcilerType, "endpoint-reconciler-type", string(s.EndpointReconcilerType), "Use an endpoint reconciler ("+strings.Join(reconcilers.AllTypes.Names(), ", ")+")")


Question:
ref: and An e2e test should be , and the default should be updated. /assign cc
/assign
Thanks I can take this on.
Could someone point me to where the manifests are for the e2e test suites, I am happy to submit the patches?
Are there API server e2e tests currently?
there are not HA API server e2e tests that I'm aware of
I opened a PR for the default changes. Are there suggestions on a good E2E test to write if we don't have HA API server tests?
what is needed for this issue to complete? Do you think you'll get that by Monday, which is Code Freeze?
We need some E2E testing or add an integration test as defined . It will not be done today.
so are you planning to move it to 1.11, then? It's about to be automatically kicked out of 1.10 by automation.
I do not have permission to change the milestone. If you could kindly change it, that would be awesome!
[MILESTONENOTIFIER] Milestone Removed From Issue Important: This issue was missing the label for more than 7 days. <details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
Reopening as this was reverted in

"--enable-aggregator-routing=true", "--enable-logs-handler=false", "--enable-swagger-ui=true", <del> "--endpoint-reconciler-type=" + string(reconcilers.MasterCountReconcilerType), </del> <ins> "--endpoint-reconciler-type=" + string(reconcilers.LeaseEndpointReconcilerType), </ins> "--etcd-quorum-read=false", "--etcd-keyfile=/var/run/kubernetes/etcd.key", "--etcd-certfile=/var/run/kubernetes/etcdce.crt",


Question:
ref: and An e2e test should be , and the default should be updated. /assign cc
/assign
Thanks I can take this on.
Could someone point me to where the manifests are for the e2e test suites, I am happy to submit the patches?
Are there API server e2e tests currently?
there are not HA API server e2e tests that I'm aware of
I opened a PR for the default changes. Are there suggestions on a good E2E test to write if we don't have HA API server tests?
what is needed for this issue to complete? Do you think you'll get that by Monday, which is Code Freeze?
We need some E2E testing or add an integration test as defined . It will not be done today.
so are you planning to move it to 1.11, then? It's about to be automatically kicked out of 1.10 by automation.
I do not have permission to change the milestone. If you could kindly change it, that would be awesome!
[MILESTONENOTIFIER] Milestone Removed From Issue Important: This issue was missing the label for more than 7 days. <details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
Reopening as this was reverted in

ServiceNodePortRange:   kubeoptions.DefaultServiceNodePortRange, ServiceClusterIPRange:  kubeoptions.DefaultServiceIPCIDR, MasterCount:            5, <del> EndpointReconcilerType: string(reconcilers.MasterCountReconcilerType), </del> <ins> EndpointReconcilerType: string(reconcilers.LeaseEndpointReconcilerType), </ins> AllowPrivileged:        false, GenericServerRunOptions: &apiserveroptions.ServerRunOptions{ AdvertiseAddress:            net.ParseIP("192.168.10.10"),


Question:
Is this a BUG REPORT or FEATURE REQUEST?: /kind bug for lack of better idea hack/make-rules/test-cmd-, line 1443 is failing, even when the debug output shows the expected text: from the test log: What happened: See for example https://k8s-
This should be owned by whoever owns the "test-cmd runkubectlget_tests". This doesn't look like a failure in the testing framework or the infrastructure. I believe this is
the linked test is actually failing on the next test, not the .missing test... from the test log: this is very odd:
the issue appears to be with
potential fix in
seems like the same issue for runpluginstests:
same fix will hopefully resolve the issue

local message=$1 local match=$2 <del> if echo "$message" | grep -q "$match"; then </del> <ins> if grep -q "${match}" <<< "${message}"; then </ins> echo "Successful" echo "message:$message" echo "has:$match"


Question:
Is this a BUG REPORT or FEATURE REQUEST?: /kind bug for lack of better idea hack/make-rules/test-cmd-, line 1443 is failing, even when the debug output shows the expected text: from the test log: What happened: See for example https://k8s-
This should be owned by whoever owns the "test-cmd runkubectlget_tests". This doesn't look like a failure in the testing framework or the infrastructure. I believe this is
the linked test is actually failing on the next test, not the .missing test... from the test log: this is very odd:
the issue appears to be with
potential fix in
seems like the same issue for runpluginstests:
same fix will hopefully resolve the issue

local message=$1 local match=$2 <del> if echo "$message" | grep -q "$match"; then </del> <ins> if grep -q "${match}" <<< "${message}"; then </ins> echo "FAIL!" echo "message:$message" echo "has:$match"


Question: Happy New Year! Now when I run make I get a diff like this in every zz_generated file: Not sure what a smart solution is, but we could merge something like in the meantime.

print('File %s is missing the year' % filename, file=verbose_out) return False <del> # Replace all occurrences of the regex "2017|2016|2015|2014" with "YEAR" </del> <ins> # Replace all occurrences of the regex "2014|2015|2016|2017|2018" with "YEAR" </ins> p = regexs["date"] for i, d in enumerate(data): (data[i], found) = p.subn('YEAR', d)


Question: Happy New Year! Now when I run make I get a diff like this in every zz_generated file: Not sure what a smart solution is, but we could merge something like in the meantime.

regexs = {} # Search for "YEAR" which exists in the boilerplate, but shouldn't in the real thing regexs["year"] = re.compile( 'YEAR' ) <del> # dates can be 2014, 2015, 2016, or 2017; company holder names can be anything regexs["date"] = re.compile( '(2014|2015|2016|2017)' ) </del> <ins> # dates can be 2014, 2015, 2016, 2017, or 2018; company holder names can be anything regexs["date"] = re.compile( '(2014|2015|2016|2017|2018)' ) </ins> # strip // +build nn build constraints regexs["go_build_constraints"] = re.compile(r"^(// +build.*n)+n", re.MULTILINE) # strip #!.* from shell scripts


Question: Happy New Year! Now when I run make I get a diff like this in every zz_generated file: Not sure what a smart solution is, but we could merge something like in the meantime.

BASH_TARGETS=" update-generated-protobuf update-codegen <ins> update-generated-runtime update-generated-device-plugin </ins> update-generated-docs update-generated-swagger-docs update-swagger-spec


Question: Happy New Year! Now when I run make I get a diff like this in every zz_generated file: Not sure what a smart solution is, but we could merge something like in the meantime.

// +build !ignore_autogenerated /* <del> Copyright 2017 The Kubernetes Authors. </del> <ins> Copyright 2018 The Kubernetes Authors. </ins> Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License.


Question: Happy New Year! Now when I run make I get a diff like this in every zz_generated file: Not sure what a smart solution is, but we could merge something like in the meantime.

/* <del> Copyright 2017 The Kubernetes Authors. </del> <ins> Copyright 2018 The Kubernetes Authors. </ins> Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License.


Question:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: call delete endpoint when service is not found. What you expected to happen: It is no need to do delete in controller. Because in service restStorage, endpoint is deleted when delete service. See How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/area controller-manager /sig apps
/kind bug

} service, err := e.serviceLister.Services(namespace).Get(name) if err != nil { <del> // Delete the corresponding endpoint, as the service has been deleted. // TODO: Please note that this will delete an endpoint when a // service is deleted. However, if we're down at the time when // the service is deleted, we will miss that deletion, so this // doesn't completely solve the problem. See #6877. err = e.client.CoreV1().Endpoints(namespace).Delete(name, nil) if err != nil && !errors.IsNotFound(err) { return err } </del> <ins> // Service has been deleted. So no need to do any more operations. </ins> return nil }


Question:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: The OpenStack cloud provider currently uses the name of the kubelet node to register with the API server. However the name is not guaranteed to conform to the DNS-1123 subdomain spec. Given a node name of , the following message appears in the kubelet logs: What you expected to happen: The OpenStack Cloud provider uses the instance id instead of the instance name. This will guarantee that the id conforms the DNS-1123 spec without making any assumptions on what the instance name may be. How to reproduce it (as minimally and precisely as possible): In the OpenStack console, rename a kubelet node to and attempt to register the kubelet node to the api server. Anything else we need to know?: Environment: OpenStack Newton Kubernetes version (use ):Cloud provider or hardware configuration: OpenStack OS (e.g. from /etc/os-release): Ubuntu 14.04.5 Kernel (e.g. ): Linux -cab9-44ff-9c99- 4.4.0-103-generic ~14.04.1-Ubuntu SMP Mon Dec 4 19:33:04 UTC 2017 x8664 x8664 x86_64 GNU/Linux Install tools: Others:
/sig openstack
/assign I prefer to conform hostname to DNS-1123 subdomain spec. Already submitted a PR on this.
I think the issue may be more than just the hostname. The hostname on my vm is set to Since the OpenStack cloud provider uses the instance name to register with the api server, I also get error messages like the following: It seems like for OpenStack specifically, it sets it's node name to the name returned from the OpenStack metadata server () which is not restricted to being DNS-1123 compliant.
The node name must match ([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9], not contain '/'.
I can confirm that Openstack can boot up instances with the name like . But this will break kubelet currently as stated above. For such cloud provider based nodes, the nodeName registered is retrieved from the provider, which should not be allowed when registering nodes for kubelet. Agree. +1. I've an extra check in PR . PTAL. Thanks.
I think the question here is: Why did you choose to use as the nodename? This is the display name of the server, a property made for humans by humans. Essentially it is a field that everybody should be able to fill in whatever they think their server should be called. This doesn't need to be unique and has no guarantees about format or length or character format. We've had a number of people putting in unicode characters in the display name, for example. Furthermore, it can be changed during runtime of the server, without rebooting. Other possibilities for the nodename are: instance UUID, which is a string in a defined format, guaranteed to be unique and is only separated by dashes. Found in If you would like to have some way to associate your nodename with whatever is shown in the Horizon dashboard as server display name, there is . OpenStack already takes care to sanitize any weird display name. E.g. some weird display name like gets a reasonable hostname: <img width="745" alt="screen shot 2018-01-10 at 08 52 49" src="https://user-"What do you think about changing the source for selecting the nodename?
cc
Once upon a time, the cloud provider's (only!) role was to discover a dynamic list of nodes by regex. In that world, using the nodename made perfect sense because the admin could then configure it to be something like , with the obvious use of regex patterns to group the relevant nodes. Fast forward a few years, and this style of node registration is long gone, and there is no need to make the node name human readable, and in fact numerous reasons to choose other unique identifiers. There is still to removing "human-friendly" node names, however. In the end, I didn't see a good alternative that would please everyone (and meet the required technical character set/length constraints). I stuck with because at least it was under the control of the cluster admin, and so it could be obviously contrived to match our imposed k8s node name == openstack instance name constraint (once that constraint was communicated).

// See http://docs.openstack.org/user-guide/cli_config_drive.html type Metadata struct { Uuid             string           `json:"uuid"` <del> Name             string           `json:"name"` </del> <ins> Hostname         string           `json:"hostname"` </ins> AvailabilityZone string           `json:"availability_zone"` Devices          []DeviceMetadata `json:"devices,omitempty"` // .. and other fields we don't care about.  Expand as necessary.


Question:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: The OpenStack cloud provider currently uses the name of the kubelet node to register with the API server. However the name is not guaranteed to conform to the DNS-1123 subdomain spec. Given a node name of , the following message appears in the kubelet logs: What you expected to happen: The OpenStack Cloud provider uses the instance id instead of the instance name. This will guarantee that the id conforms the DNS-1123 spec without making any assumptions on what the instance name may be. How to reproduce it (as minimally and precisely as possible): In the OpenStack console, rename a kubelet node to and attempt to register the kubelet node to the api server. Anything else we need to know?: Environment: OpenStack Newton Kubernetes version (use ):Cloud provider or hardware configuration: OpenStack OS (e.g. from /etc/os-release): Ubuntu 14.04.5 Kernel (e.g. ): Linux -cab9-44ff-9c99- 4.4.0-103-generic ~14.04.1-Ubuntu SMP Mon Dec 4 19:33:04 UTC 2017 x8664 x8664 x86_64 GNU/Linux Install tools: Others:
/sig openstack
/assign I prefer to conform hostname to DNS-1123 subdomain spec. Already submitted a PR on this.
I think the issue may be more than just the hostname. The hostname on my vm is set to Since the OpenStack cloud provider uses the instance name to register with the api server, I also get error messages like the following: It seems like for OpenStack specifically, it sets it's node name to the name returned from the OpenStack metadata server () which is not restricted to being DNS-1123 compliant.
The node name must match ([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9], not contain '/'.
I can confirm that Openstack can boot up instances with the name like . But this will break kubelet currently as stated above. For such cloud provider based nodes, the nodeName registered is retrieved from the provider, which should not be allowed when registering nodes for kubelet. Agree. +1. I've an extra check in PR . PTAL. Thanks.
I think the question here is: Why did you choose to use as the nodename? This is the display name of the server, a property made for humans by humans. Essentially it is a field that everybody should be able to fill in whatever they think their server should be called. This doesn't need to be unique and has no guarantees about format or length or character format. We've had a number of people putting in unicode characters in the display name, for example. Furthermore, it can be changed during runtime of the server, without rebooting. Other possibilities for the nodename are: instance UUID, which is a string in a defined format, guaranteed to be unique and is only separated by dashes. Found in If you would like to have some way to associate your nodename with whatever is shown in the Horizon dashboard as server display name, there is . OpenStack already takes care to sanitize any weird display name. E.g. some weird display name like gets a reasonable hostname: <img width="745" alt="screen shot 2018-01-10 at 08 52 49" src="https://user-"What do you think about changing the source for selecting the nodename?
cc
Once upon a time, the cloud provider's (only!) role was to discover a dynamic list of nodes by regex. In that world, using the nodename made perfect sense because the admin could then configure it to be something like , with the obvious use of regex patterns to group the relevant nodes. Fast forward a few years, and this style of node registration is long gone, and there is no need to make the node name human readable, and in fact numerous reasons to choose other unique identifiers. There is still to removing "human-friendly" node names, however. In the end, I didn't see a good alternative that would please everyone (and meet the required technical character set/length constraints). I stuck with because at least it was under the control of the cluster admin, and so it could be obviously contrived to match our imposed k8s node name == openstack instance name constraint (once that constraint was communicated).

var FakeMetadata = Metadata{ Uuid:             "83679162-1378-4288-a2d4-70e13ec132aa", <del> Name:             "test", </del> <ins> Hostname:         "test", </ins> AvailabilityZone: "nova", }


Question:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: The OpenStack cloud provider currently uses the name of the kubelet node to register with the API server. However the name is not guaranteed to conform to the DNS-1123 subdomain spec. Given a node name of , the following message appears in the kubelet logs: What you expected to happen: The OpenStack Cloud provider uses the instance id instead of the instance name. This will guarantee that the id conforms the DNS-1123 spec without making any assumptions on what the instance name may be. How to reproduce it (as minimally and precisely as possible): In the OpenStack console, rename a kubelet node to and attempt to register the kubelet node to the api server. Anything else we need to know?: Environment: OpenStack Newton Kubernetes version (use ):Cloud provider or hardware configuration: OpenStack OS (e.g. from /etc/os-release): Ubuntu 14.04.5 Kernel (e.g. ): Linux -cab9-44ff-9c99- 4.4.0-103-generic ~14.04.1-Ubuntu SMP Mon Dec 4 19:33:04 UTC 2017 x8664 x8664 x86_64 GNU/Linux Install tools: Others:
/sig openstack
/assign I prefer to conform hostname to DNS-1123 subdomain spec. Already submitted a PR on this.
I think the issue may be more than just the hostname. The hostname on my vm is set to Since the OpenStack cloud provider uses the instance name to register with the api server, I also get error messages like the following: It seems like for OpenStack specifically, it sets it's node name to the name returned from the OpenStack metadata server () which is not restricted to being DNS-1123 compliant.
The node name must match ([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9], not contain '/'.
I can confirm that Openstack can boot up instances with the name like . But this will break kubelet currently as stated above. For such cloud provider based nodes, the nodeName registered is retrieved from the provider, which should not be allowed when registering nodes for kubelet. Agree. +1. I've an extra check in PR . PTAL. Thanks.
I think the question here is: Why did you choose to use as the nodename? This is the display name of the server, a property made for humans by humans. Essentially it is a field that everybody should be able to fill in whatever they think their server should be called. This doesn't need to be unique and has no guarantees about format or length or character format. We've had a number of people putting in unicode characters in the display name, for example. Furthermore, it can be changed during runtime of the server, without rebooting. Other possibilities for the nodename are: instance UUID, which is a string in a defined format, guaranteed to be unique and is only separated by dashes. Found in If you would like to have some way to associate your nodename with whatever is shown in the Horizon dashboard as server display name, there is . OpenStack already takes care to sanitize any weird display name. E.g. some weird display name like gets a reasonable hostname: <img width="745" alt="screen shot 2018-01-10 at 08 52 49" src="https://user-"What do you think about changing the source for selecting the nodename?
cc
Once upon a time, the cloud provider's (only!) role was to discover a dynamic list of nodes by regex. In that world, using the nodename made perfect sense because the admin could then configure it to be something like , with the obvious use of regex patterns to group the relevant nodes. Fast forward a few years, and this style of node registration is long gone, and there is no need to make the node name human readable, and in fact numerous reasons to choose other unique identifiers. There is still to removing "human-friendly" node names, however. In the end, I didn't see a good alternative that would please everyone (and meet the required technical character set/length constraints). I stuck with because at least it was under the control of the cluster admin, and so it could be obviously contrived to match our imposed k8s node name == openstack instance name constraint (once that constraint was communicated).

t.Fatalf("Should succeed when provided with valid data: %s", err) } <del> if md.Name != "test" { t.Errorf("incorrect name: %s", md.Name) </del> <ins> if md.Hostname != "test.novalocal" { t.Errorf("incorrect hostname: %s", md.Hostname) </ins> } if md.Uuid != "83679162-1378-4288-a2d4-70e13ec132aa" {


Question:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: The OpenStack cloud provider currently uses the name of the kubelet node to register with the API server. However the name is not guaranteed to conform to the DNS-1123 subdomain spec. Given a node name of , the following message appears in the kubelet logs: What you expected to happen: The OpenStack Cloud provider uses the instance id instead of the instance name. This will guarantee that the id conforms the DNS-1123 spec without making any assumptions on what the instance name may be. How to reproduce it (as minimally and precisely as possible): In the OpenStack console, rename a kubelet node to and attempt to register the kubelet node to the api server. Anything else we need to know?: Environment: OpenStack Newton Kubernetes version (use ):Cloud provider or hardware configuration: OpenStack OS (e.g. from /etc/os-release): Ubuntu 14.04.5 Kernel (e.g. ): Linux -cab9-44ff-9c99- 4.4.0-103-generic ~14.04.1-Ubuntu SMP Mon Dec 4 19:33:04 UTC 2017 x8664 x8664 x86_64 GNU/Linux Install tools: Others:
/sig openstack
/assign I prefer to conform hostname to DNS-1123 subdomain spec. Already submitted a PR on this.
I think the issue may be more than just the hostname. The hostname on my vm is set to Since the OpenStack cloud provider uses the instance name to register with the api server, I also get error messages like the following: It seems like for OpenStack specifically, it sets it's node name to the name returned from the OpenStack metadata server () which is not restricted to being DNS-1123 compliant.
The node name must match ([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9], not contain '/'.
I can confirm that Openstack can boot up instances with the name like . But this will break kubelet currently as stated above. For such cloud provider based nodes, the nodeName registered is retrieved from the provider, which should not be allowed when registering nodes for kubelet. Agree. +1. I've an extra check in PR . PTAL. Thanks.
I think the question here is: Why did you choose to use as the nodename? This is the display name of the server, a property made for humans by humans. Essentially it is a field that everybody should be able to fill in whatever they think their server should be called. This doesn't need to be unique and has no guarantees about format or length or character format. We've had a number of people putting in unicode characters in the display name, for example. Furthermore, it can be changed during runtime of the server, without rebooting. Other possibilities for the nodename are: instance UUID, which is a string in a defined format, guaranteed to be unique and is only separated by dashes. Found in If you would like to have some way to associate your nodename with whatever is shown in the Horizon dashboard as server display name, there is . OpenStack already takes care to sanitize any weird display name. E.g. some weird display name like gets a reasonable hostname: <img width="745" alt="screen shot 2018-01-10 at 08 52 49" src="https://user-"What do you think about changing the source for selecting the nodename?
cc
Once upon a time, the cloud provider's (only!) role was to discover a dynamic list of nodes by regex. In that world, using the nodename made perfect sense because the admin could then configure it to be something like , with the obvious use of regex patterns to group the relevant nodes. Fast forward a few years, and this style of node registration is long gone, and there is no need to make the node name human readable, and in fact numerous reasons to choose other unique identifiers. There is still to removing "human-friendly" node names, however. In the end, I didn't see a good alternative that would please everyone (and meet the required technical character set/length constraints). I stuck with because at least it was under the control of the cluster admin, and so it could be obviously contrived to match our imposed k8s node name == openstack instance name constraint (once that constraint was communicated).

if err != nil { return "", err } <del> return types.NodeName(md.Name), nil </del> <ins> return types.NodeName(md.Hostname), nil </ins> } func (i *Instances) AddSSHKeyToAllInstances(user string, keyData []byte) error {


Question:
/kind feature What happened: When calling , there is no way to override default flag values for the different components of the controlplane. This is surprising, as in the config structure (and in controlplane phase codebase itself), there are explicit settings for doing that (APIServerExtraArgs, ControllerManagerExtraArgs, SchedulerExtraArgs). What you expected to happen: Those config fields should be exposed as flags on and How to reproduce it (as minimally and precisely as possible): Try to deploy K8s using Kubeadm with a custom Admission Control setting (I'll follow up on this issue with a PR)
btw, I am not sure which sig owns kubeadm.
it's sig-cluster-lifecycle.
/sig sig-cluster-lifecycle
/sig cluster-lifecycle

"k8s.io/apimachinery/pkg/runtime" "k8s.io/apimachinery/pkg/util/sets" <ins> utilflag "k8s.io/apiserver/pkg/util/flag" </ins> clientset "k8s.io/client-go/kubernetes" kubeadmapi "k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm" kubeadmapiext "k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1alpha1"