Question: // it does not have to be reimplemented. type counterRaiser struct { counter *metrics.Counter <del> reason  string </del> <ins> // programmatic id used in log and audit annotations prefixes id string // human readable explanation reason string </ins> } func (c *counterRaiser) IncreaseMetricsCounter(req *http.Request) { if req != nil && req.URL != nil { if hostname := req.URL.Hostname(); len(hostname) > 0 { <del> klog.Infof("invalid certificate detected while connecting to %q: %s", req.URL.Hostname(), c.reason) </del> <ins> prefix := fmt.Sprintf("%s.invalid-cert.kubernetes.io", c.id) klog.Infof("%s: invalid certificate detected connecting to %q: %s", prefix, hostname, c.reason) audit.AddAuditAnnotation(req.Context(), prefix+"/"+hostname, c.reason) </ins> } } c.counter.Inc()

adds the metrics. the release note on that issue describes the go change, links to go docs, describes use of the option, the new metrics, and the kube-apiserver logs what would you expect kubeadm to do with this? GODEBUG off by default and let users override GODEBUG on their own (is this possible)? GODEBUG by default on all clusters? GODEBUG by default only on upgrade? The risk of 1 is that a user upgrades without reading release notes and breaks a cluster that has SHA-1 certs in webhooks, etc The risk of 2 is that we propagate accepting insecure certificates longer, even in new clusters that we know don't have any pre-existing bad certs The risk of 3 is that it makes an upgraded 1.23 → 1.24 cluster and a new 1.24 cluster inconsistent
the majority of kubeadm users use kubeadm managed CA and certificates. there are less users that use external CA to sign their own certificates. sha1 attacks surfaced more than a before kubeadm's creation and Git hash collusion made the problem more known around 2017 - that was at a time when kubeadm did not support external CA. given these facts, an estimate on my side is that there are currently no kubeadm users that use sha1 certificates, but i would not exclude the possibility entirely. if such users exists, they are on they own and my vote goes for option 1, until they migrate their PKI to something secure. if the same users log tickets for us, our response would be "please use GODEBUG and migrate ASAP". all users should be able to see the notes in IMO, the PR needs an "action required" to appear on top of the release notes.
that's helpful, thanks agreed. weird, I set , but the label didn't get . the label manually.
and there's a way for them to set that envvar in kube-apiserver manifests when using kubeadm?
There's been a lot of intermittent github webhook delivery issues in the past few weeks or so, slash commands are going to be somewhat unreliable :/
they could use kubeadm patches to override the YAML manifest defaults - i.e. the contents of v1.Container -Env:


Question: break } } <del> if found == false { </del> <ins> if !found { </ins> return false } } for _, leftPair := range newPermission.UserIdGroupPairs { <ins> found := false </ins> for _, rightPair := range existing.UserIdGroupPairs { if isEqualUserGroupPair(leftPair, rightPair, compareGroupUserIDs) { <del> return true </del> <ins> found = true break </ins> } } <del> return false </del> <ins> if !found { return false } </ins> } return true

Per : the surrounding loop is unconditionally terminated (SA4004) It does look very suspicious.
Pertinent patches: AWS: Set up security groups, to mirror GCE firewalling Fix AWS ip permission comparison on security group
This needs to be triaged as a release-blocker or not for 1.5
all issues must be labeled either release blocker or non release blocking by end of day 18 November 2016 PST. (or please move it to 1.6) cc
Moving to 1.7 as late to happen in 1.6. Feel free to switch back if this is incorrect.
This was a bug where we had multiple userid/group pairs, but we only ever called it with 0 or 1 elements. Fix in but kicking out of the 1.7 milestone.


Question: }, shouldError: false, }, <ins> { // We don't have to test "legacy cloud provider with dual-stack // IPs" etc because we won't have gotten this far with an invalid // config like that. name:              "Dual-stack cloud, with dual-stack nodeIPs", nodeIP:            netutils.ParseIPSloppy("2600:1f14:1d4:d101::ba3d"), secondaryNodeIP:   netutils.ParseIPSloppy("10.1.1.2"), cloudProviderType: cloudProviderExternal, nodeAddresses: []v1.NodeAddress{ {Type: v1.NodeInternalIP, Address: "10.1.1.1"}, {Type: v1.NodeInternalIP, Address: "10.1.1.2"}, {Type: v1.NodeInternalIP, Address: "2600:1f14:1d4:d101::ba3d"}, {Type: v1.NodeHostName, Address: testKubeletHostname}, }, expectedAddresses: []v1.NodeAddress{ {Type: v1.NodeInternalIP, Address: "2600:1f14:1d4:d101::ba3d"}, {Type: v1.NodeInternalIP, Address: "10.1.1.2"}, {Type: v1.NodeHostName, Address: testKubeletHostname}, }, expectedAnnotations: map[string]string{ "alpha.kubernetes.io/provided-node-ip": "2600:1f14:1d4:d101::ba3d,10.1.1.2", }, shouldError: false, }, { name:              "Upgrade to cloud dual-stack nodeIPs", nodeIP:            netutils.ParseIPSloppy("10.1.1.1"), secondaryNodeIP:   netutils.ParseIPSloppy("2600:1f14:1d4:d101::ba3d"), cloudProviderType: cloudProviderExternal, nodeAddresses: []v1.NodeAddress{ {Type: v1.NodeInternalIP, Address: "10.1.1.1"}, {Type: v1.NodeInternalIP, Address: "2600:1f14:1d4:d101::ba3d"}, {Type: v1.NodeHostName, Address: testKubeletHostname}, }, expectedAddresses: []v1.NodeAddress{ {Type: v1.NodeInternalIP, Address: "10.1.1.1"}, {Type: v1.NodeInternalIP, Address: "2600:1f14:1d4:d101::ba3d"}, {Type: v1.NodeHostName, Address: testKubeletHostname}, }, existingAnnotations: map[string]string{ "alpha.kubernetes.io/provided-node-ip": "10.1.1.1", }, expectedAnnotations: map[string]string{ "alpha.kubernetes.io/provided-node-ip": "10.1.1.1,2600:1f14:1d4:d101::ba3d", }, shouldError: false, }, { name:              "Downgrade from cloud dual-stack nodeIPs", nodeIP:            netutils.ParseIPSloppy("10.1.1.1"), cloudProviderType: cloudProviderExternal, nodeAddresses: []v1.NodeAddress{ {Type: v1.NodeInternalIP, Address: "10.1.1.1"}, {Type: v1.NodeInternalIP, Address: "2600:1f14:1d4:d101::ba3d"}, {Type: v1.NodeHostName, Address: testKubeletHostname}, }, expectedAddresses: []v1.NodeAddress{ {Type: v1.NodeInternalIP, Address: "10.1.1.1"}, {Type: v1.NodeHostName, Address: testKubeletHostname}, }, existingAnnotations: map[string]string{ "alpha.kubernetes.io/provided-node-ip": "10.1.1.1,2600:1f14:1d4:d101::ba3d", }, expectedAnnotations: map[string]string{ "alpha.kubernetes.io/provided-node-ip": "10.1.1.1", }, shouldError: false, }, </ins> } for _, testCase := range cases { t.Run(testCase.name, func(t *testing.T) {

When adding to kubelet and to the OpenStack external cloud provider, which is based on kubernetes v1.27.1, the node annotation only shows one address and consequently the node addresses also only displays one address. The kubelet logs shows that node-ip contains dual-stack addresses. The annotation on the node would contain both IPv4 and IPv6 addresses and the node addresses would list them. Deploy a cluster with to kubelet and external cloud provider. Also make sure the external cloud provider is based on Kubernetes 1.27. Kubelet logs: one of CCMs Pods: one of the nodes annotation: <details</details<details</details<details</details<details</details<details</details<details</details
/sig network /sig cloud-provider /cc
/assign
/triage accepted


Question: mungedPodSpec.TerminationGracePeriodSeconds = oldPod.Spec.TerminationGracePeriodSeconds // +k8s:verify-mutation:reason=clone } <ins> // Handle validations specific to gated pods. podIsGated := len(oldPod.Spec.SchedulingGates) > 0 if opts.AllowMutableNodeSelectorAndNodeAffinity && podIsGated { // Additions to spec.nodeSelector are allowed (no deletions or mutations) for gated pods. if !apiequality.Semantic.DeepEqual(mungedPodSpec.NodeSelector, oldPod.Spec.NodeSelector) { allErrs = append(allErrs, validateNodeSelectorMutation(specPath.Child("nodeSelector"), mungedPodSpec.NodeSelector, oldPod.Spec.NodeSelector)...) mungedPodSpec.NodeSelector = oldPod.Spec.NodeSelector // +k8s:verify-mutation:reason=clone } // Validate node affinity mutations. var oldNodeAffinity *core.NodeAffinity if oldPod.Spec.Affinity != nil { oldNodeAffinity = oldPod.Spec.Affinity.NodeAffinity // +k8s:verify-mutation:reason=clone } var mungedNodeAffinity *core.NodeAffinity if mungedPodSpec.Affinity != nil { mungedNodeAffinity = mungedPodSpec.Affinity.NodeAffinity // +k8s:verify-mutation:reason=clone } if !apiequality.Semantic.DeepEqual(oldNodeAffinity, mungedNodeAffinity) { allErrs = append(allErrs, validateNodeAffinityMutation(specPath.Child("affinity").Child("nodeAffinity"), mungedNodeAffinity, oldNodeAffinity)...) switch { case mungedPodSpec.Affinity == nil && oldNodeAffinity == nil: // already effectively nil, no change needed case mungedPodSpec.Affinity == nil && oldNodeAffinity != nil: mungedPodSpec.Affinity = &core.Affinity{NodeAffinity: oldNodeAffinity} // +k8s:verify-mutation:reason=clone default: mungedPodSpec.Affinity.NodeAffinity = oldNodeAffinity // +k8s:verify-mutation:reason=clone } } } </ins> if !apiequality.Semantic.DeepEqual(mungedPodSpec, oldPod.Spec) { // This diff isn't perfect, but it's a helluva lot better an "I'm not going to tell you what the difference is". // TODO: Pinpoint the specific field that causes the invalid error after we have strategic merge diff

See for context. This issue is creating to track work on the API validation changes. Validation relaxation of certain fields in the pod spec to allow for mutable scheduling directives.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/sig scheduling
Docs PR:


Question: return &containerStatus, nil } <del> // GetDockerPodInfo returns docker info for all containers in the pod/manifest. </del> <ins> // GetDockerPodInfo returns docker info for all containers in the pod/manifest and // infrastructure container </ins> func GetDockerPodInfo(client DockerInterface, manifest api.PodSpec, podFullName string, uid types.UID) (api.PodInfo, error) { info := api.PodInfo{} expectedContainers := make(map[string]api.Container)

Kubelet now only reports ContainerStatus. PodStatus / PodCondition is observed at master side based on a set of ContainerStatus. There is no other way to get PodIP except net container for master side. This is very confusion to the end user, , , ... Now we are finalizing on v1beta3 API, including PodStatus & ContainerStatus. We can make net container completely invisible to the end users by finishing the following steps: 1) Expose network related errors at API level 2) Generate PodStaus directly from Kubelet
3) once PodStatus generated by kubelet, PodIP can be removed from ContainerStatus.
I think reporting PodStatus rather than just container status is most important. How hard is that?
Used to be hard since there are many dependencies when I first report Status back. Now it is totally feasible. I already started to write PRs for this. :-)
One more PR required to close this one.


Question: switch op { case startContainerRuntimeUnitOp: <del> conn.StartUnitContext(ctx, containerRuntimeUnitName, "replace", reschan) </del> <ins> _, err = conn.StartUnitContext(ctx, containerRuntimeUnitName, "replace", reschan) </ins> case stopContainerRuntimeUnitOp: <del> conn.StopUnitContext(ctx, containerRuntimeUnitName, "replace", reschan) </del> <ins> _, err = conn.StopUnitContext(ctx, containerRuntimeUnitName, "replace", reschan) </ins> default: framework.Failf("Unexpected container runtime op: %v", op) } <ins> framework.ExpectNoError(err, "dbus connection error") </ins> job := <-reschan framework.ExpectEqual(job, "done", "Expected job to complete with done")

E2eNode Suite.[It] [sig-node] MirrorPodWithGracePeriod when create a mirror pod and the container runtime is temporarily down during pod termination [NodeConformance] [Serial] [Disruptive] the mirror pod should terminate successfully Perma failure here: E2eNode Suite.[It] [sig-node] MirrorPodWithGracePeriod when create a mirror pod and the container runtime is temporarily down during pod termination [NodeConformance] [Serial] [Disruptive] the mirror pod should terminate successfully E2eNode Suite.[It] [sig-node] MirrorPodWithGracePeriod when create a mirror pod and the container runtime is temporarily down during pod termination [NodeConformance] [Serial] [Disruptive] the mirror pod should terminate successfully Since it was created at No response No response /sig node
/cc
/cc /cc is there anything specific to CRI-O that will make test fail
/assign
The test fails there: Hm, scraping the kubelet logs it looks like that we miss a metric: cc
/triage accepted /priority important-soon seeing this in the wild
failed for this in as well
It seems to fail on on :
Having a fix in mind in


Question: } continue } <del> if len(results) == 0 { break } </del> fullResult = append(fullResult, results) } return fullResult, nil

no longer continues parsing an item when a key is encountered that has no value. This appears to be a behavior change introduced in Kubernetes version 1.22. will continue . Run the following commands against a cluster running CoreDNS pods in the namespace: Nothing. Kubernetes version (use ): v1.22.0 Cloud provider or hardware configuration: N/A OS (e.g: ): N/A Kernel (e.g. ): N/A Install tools: N/A Network plugin and version (if this is a network-related bug): N/A Others: N/A
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Debug data for Kubernetes version 1.21 cluster:
Debug data for Kubernetes version 1.22 cluster:
/sig cli
I reproduce the issue in my 1.23 environment. I'll try to fix it. /assign
PR works on it.
/unassign
to ensure I understand your comment correct, should fix this issue?
Yes, I've tested it. can resolve it. The bug is the line:
Yes, I've tested it. can resolve it. The bug is in this line:


Question: status, found := kubelet.statusManager.GetPodStatus(pods[0].UID) assert.True(t, found, "expected to found status for pod %q", pods[0].UID) assert.Equal(t, v1.PodFailed, status.Phase) <ins> // check pod status contains ContainerStatuses, etc. assert.NotNil(t, status.ContainerStatuses) </ins> } func TestSyncPodsDoesNotSetPodsThatDidNotRunTooLongToFailed(t *testing.T) {

Was running a pod with an activeDeadlineSeconds on a 1.10 cluster. The pod reached the deadline, but the pod status did not contain the latest state of the pod containers (pod containersStatus was empty). The status only had the following fields: The statuses were there immediately prior to the deadline being reached. This broke API clients that were looking at container status.
Looks like the status is just overwritten here if returns true
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale


Question: return errors.New(log("attacher.MountDevice failed, deviceMountPath is empty")) } <del> corruptedDir := false mounted, err := isDirMounted(c.plugin, deviceMountPath) if err != nil { klog.Error(log("attacher.MountDevice failed while checking mount status for dir [%s]", deviceMountPath)) if isCorruptedDir(deviceMountPath) { corruptedDir = true // leave to CSI driver to handle corrupted mount klog.Warning(log("attacher.MountDevice detected corrupted mount for dir [%s]", deviceMountPath)) } else { return err } } if mounted && !corruptedDir { klog.V(4).Info(log("attacher.MountDevice skipping mount, dir already mounted [%s]", deviceMountPath)) return nil } </del> // Setup if spec == nil { return errors.New(log("attacher.MountDevice failed, spec is nil"))

What happened: A custom CSI driver implementing returns a failure response, indicating that it did not complete its intended steps to prepare the volume for the mounting pod. Let's say that these steps involve a filesystem at the target directory, data from a service, and that data to a set of files in the mounted filesystem. If the first filesystem mounting step succeeds, but either of the subsequent two steps fail, the volume is still not ready for use by the pod. However, if this CSI driver returns a failure response, and the CSI mounter comes back and, before invoking the RPC again, it finds a filesystem already mounted at the target directory, , and ceases further calls to to allow the CSI driver to try again. What you expected to happen: If a CSI driver returns a failure response to a RPC, Kubernetes should honor that declaration of failure and invoke the RPC again, disallowing the mounting pod from starting until this RPC succeeds. Even though it's possible to partially work around the current behavior by having the driver mount the filesystem as late as possible, and then attempting to unmount it if fails to populate the volume with its intended data, doing so makes writing drivers more complicated and forces them to do extra work. How to reproduce it (as minimally and precisely as possible): In a CSI driver (even one that serves just ephemeral, inline volumes), implement to mount a filesystem at the target path, then return a failure response. Confirm that Kubernetes does not invoke again, because there's now a filesystem mounted at the target path. Anything else we need to know?: This topic came up for discussion in the "Kubernetes" Slack team. There, helped investigate this complaint, and to discuss changing this behavior. Environment: Kubernetes version (use ):Cloud provider or hardware configuration: AWS EC2 OS (e.g: ):Kernel (e.g. ):Install tools: kubeadm Others: csi-node-driver-registrar container image tag: v1.0.2
/sig storage
While the plugin should try to clean up as much as possible, I agree that kubelet should not be assuming that if there's a mount point, the publish is successful. NodePublish is supposed to be idempotent so it should theoretically be safe to call it when it's already mounted, although there is the possibility that plugins may have written against the K8s behavior.


Question: return errors.New(log("attacher.MountDevice failed, deviceMountPath is empty")) } <del> corruptedDir := false mounted, err := isDirMounted(c.plugin, deviceMountPath) if err != nil { klog.Error(log("attacher.MountDevice failed while checking mount status for dir [%s]", deviceMountPath)) if isCorruptedDir(deviceMountPath) { corruptedDir = true // leave to CSI driver to handle corrupted mount klog.Warning(log("attacher.MountDevice detected corrupted mount for dir [%s]", deviceMountPath)) } else { return err } } if mounted && !corruptedDir { klog.V(4).Info(log("attacher.MountDevice skipping mount, dir already mounted [%s]", deviceMountPath)) return nil } </del> // Setup if spec == nil { return errors.New(log("attacher.MountDevice failed, spec is nil"))

For the mount point checking here before calling NodePublishVolume, I am wondering it is just for performance optimization, or it has other purposes. Have concern about some drivers might depend on this check if they are not implemented with idenpotency.
Those drivers would be operating in violation of the specification. I'd prefer that we don't penalize already conforming and yet-to-be written drivers in order to accommodate these existing drivers—if they even exist.
More fallout in last night's testing: I introduced a bug into my driver that caused it to panic during ("Writing that data to a set of files in the mounted filesystem"). My handler function is set up to unmount a successfully mounted volume via if it doesn't complete steps 2 and 3 above, in order to deal with this issue's principal complaint. My driver container would crash due to this panic. The kubelet would call on my driver twice, it would crash twice, but the pod still started running. I'm not yet sure whether or not the filesystem got unmounted successfully via the on the first call, but somehow the pod started despite my handler function never completing successfully. I'm going to try to get some more debugging time on this later today to see if I can figure out what the kubelet was doing there.
Studying my code more, I see that my function only unmounted the mounted directory when the containing handler function was returning an error. When unwinding due to a panic, the named return value of type I was inspecting was still nil. Revising the criteria in that function covers both failure reasons (returning an and unwinding due to a panic). Regardless, this further underscores the problem: If a driver makes it as far as mounting a filesystem at the target directory, but fails afterward—including the process crashing and never responding positively to the RPC—the mounting pod will start. That's likely to then cause the pod to fail, for lack of the files it expected to find mounted.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity.


Question: return errors.New(log("attacher.MountDevice failed, deviceMountPath is empty")) } <del> corruptedDir := false mounted, err := isDirMounted(c.plugin, deviceMountPath) if err != nil { klog.Error(log("attacher.MountDevice failed while checking mount status for dir [%s]", deviceMountPath)) if isCorruptedDir(deviceMountPath) { corruptedDir = true // leave to CSI driver to handle corrupted mount klog.Warning(log("attacher.MountDevice detected corrupted mount for dir [%s]", deviceMountPath)) } else { return err } } if mounted && !corruptedDir { klog.V(4).Info(log("attacher.MountDevice skipping mount, dir already mounted [%s]", deviceMountPath)) return nil } </del> // Setup if spec == nil { return errors.New(log("attacher.MountDevice failed, spec is nil"))

Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


Question: Namespace: namespaceName, Labels:    componentLabel, Annotations: map[string]string{ <del> util.FedDomainMapKey: fmt.Sprintf("%s=%s", federationName, dnsZoneName), </del> <ins> util.FedDomainMapKey: fmt.Sprintf("%s=%s", federationName, strings.TrimRight(dnsZoneName, ".")), </ins> }, }, Spec: v1beta1.DeploymentSpec{

Looking at our latest e2e CI , they appear to be due to the fact that the "federations value" for kube-dns install a domain with a trailing dot. The trailing period does not appear when I bring up the cluster manually with I suspect that somewhere we are reconstructing a FEDERATIONSDOMAINMAP when it is unset and a trailing period gets there.
is this blocking for 1.6?
Fixed by


Question: Expect(len(finishedJobs) == 1).To(BeTrue()) // Job should get deleted when the next job finishes the next minute <del> By("Ensuring this job does not exist anymore") err = waitForJobNotExist(f.ClientSet, f.Namespace.Name, finishedJobs[0]) </del> <ins> By("Ensuring this job and its pods does not exist anymore") err = waitForJobToDisappear(f.ClientSet, f.Namespace.Name, finishedJobs[0]) Expect(err).NotTo(HaveOccurred()) err = waitForJobsPodToDisappear(f.ClientSet, f.Namespace.Name, finishedJobs[0]) </ins> Expect(err).NotTo(HaveOccurred()) By("Ensuring there is 1 finished job by listing jobs explicitly")

What happened: There seems to be a regression after 1.13 release. A Cronjob with default settings does not clean up completed pods. The job themselves are getting deleted as expected, but the pods are left behind. This seems to be related to What you expected to happen: Expected the pods for now non-existent child jobs of a cronjob to be cleaned up. This is the way it used to work prior to 1.13/ How to reproduce it (as minimally and precisely as possible): run on this manifest: Wait, more than 3 minutes, observe, that while only 3 successful jobs are present, the pods from now deleted jobs are left behind and never get deleted. Anything else we need to know?: pointed out the following relevant places in the code base: deleting a job orphans pods by default and the cronjob controller doesn't do anything different than the default when deleting Environment: Kubernetes version (use ): 1.13.0 Cloud provider or hardware configuration: ubuntu VM in vSphere OS (e.g. from /etc/os-release): 16.04.5 LTS (Xenial Xerus) Kernel (e.g. ): 4.4.0-135-generic Install tools: kubeadm <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
/sig apps
Should include a known issues in 1.13.0 as well /priority critical-urgent
Lemme see if I can easily reproduce this.
/milestone v1.13
Fixes are in (master) and (1.13).


Question: out.TLSPrivateKeyFile = in.TLSPrivateKeyFile out.TLSCipherSuites = *(*[]string)(unsafe.Pointer(&in.TLSCipherSuites)) out.TLSMinVersion = in.TLSMinVersion <ins> out.RotateCertificates = in.RotateCertificates </ins> out.ServerTLSBootstrap = in.ServerTLSBootstrap if err := Convert_kubeletconfig_KubeletAuthentication_To_v1beta1_KubeletAuthentication(&in.Authentication, &out.Authentication, s); err != nil { return err

/kind cleanup What happened: and both require opt-in configuration knobs to the kubelet. They are kinda global settings, you probably either opt into dynamic cert rotation or you don't, so they qualify for being in the struct. The knob for enabling dynamic serving certs was (I fully support that :+1:). What you expected to happen: I'd like the same to happen to  to be consistent. (Also, we're planning to use the config file for the kubelet by default in kubeadm soon, and it'd be nice to be able to set this value then in a structured manner) /assign


Question: out.TLSPrivateKeyFile = in.TLSPrivateKeyFile out.TLSCipherSuites = *(*[]string)(unsafe.Pointer(&in.TLSCipherSuites)) out.TLSMinVersion = in.TLSMinVersion <ins> out.RotateCertificates = in.RotateCertificates </ins> out.ServerTLSBootstrap = in.ServerTLSBootstrap if err := Convert_kubeletconfig_KubeletAuthentication_To_v1beta1_KubeletAuthentication(&in.Authentication, &out.Authentication, s); err != nil { return err

Flag name: Help text: <Warning: Beta featureAuto rotate the kubelet client certificates by requesting new certificates from the kube-apiserver when the certificate expiration approaches. This is part of migrating the Kubelet command-line to a Kubernetes-style API. The flag should either be migrated to the Kubelet's API group, or simply removed from the Kubelet. If this could be considered an instance-specific flag, or a descriptor of local topology managed by the Kubelet, see: If this flag is only registered in os-specific builds, see: /assign /sig node /kind feature /priority important-soon /milestone v1.11 /status approved-for-milestone
[MILESTONENOTIFIER] Milestone Issue: Up-to-date for process <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. : New functionality. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details


Question: MinExternalEtcdVersion = "3.2.18" // DefaultEtcdVersion indicates the default etcd version that kubeadm uses <del> DefaultEtcdVersion = "3.5.0-rc.0-0" </del> <ins> DefaultEtcdVersion = "3.5.0-0" </ins> // Etcd defines variable used internally when referring to etcd component Etcd = "etcd"

/sig scalability This issue is tracking process of upgrading etcd to 3.5.0 planned for 1.22 K8s release. Etcd has recently released Beta versions and is starting the process of correctness and performance testing. To reduce change of finding performance regression late, we will start qualifying Beta Etcd releases on GKE master branch and will upgrade to 3.5.0 as soon as it is available. Steps: [x] Do preliminary scale testing of etcd v3.5.0-beta.3 release [x] Release etcd v3.5.0-beta.3 image (, ) [x] Release etcd v3.5.0-rc.0 image () [x] Upgrade etcd server to v3.5.0-rc.0 () [x] Wait for 3.5.0 release and upgrade etcd client () [x] Release etcd v3.5.0 image (, ) [x] Upgrade etcd server to v3.5.0 /milestone v1.22
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/area etcd
Does this meant that the 3.5.0 should be promoted like
Yes, created
/area release-eng release-team /sig release For tracking: cc:
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /area release-eng release-team /sig release cc: Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I do think this can be pointed in the Major Themes for the 1.22 release. I would like your opinions on this. If you do think it has a place in it, I will add it to the .
How can I be a contributor to this issue? what are the technical checklist that needs to be achieved to close out on the issue?


Question: MinExternalEtcdVersion = "3.2.18" // DefaultEtcdVersion indicates the default etcd version that kubeadm uses <del> DefaultEtcdVersion = "3.5.0-rc.0-0" </del> <ins> DefaultEtcdVersion = "3.5.0-0" </ins> // Etcd defines variable used internally when referring to etcd component Etcd = "etcd"

PR to update Etcd is ready, but we found a blocker for merging . There is one test failing I will take a look into it on Monday, still more eyes on the problem could speed up resolution.
We'll also need a new kube-cross image w/ etcd v3.5.0. I'm working on that here:
We just need to upgrade the server to use new image like in
ok, looks like is busy. Let me take a look /assign
Thanks.


Question: <del> { "id": "influx-master", "kind": "Service", "apiVersion": "v1beta1", "port": 8085, "containerPort": 8086, "provider": "kubernetes-default", "component": "influxdb", "selector": { "name": "influxdb" } } </del>

examples/monitoring/influx-grafana- and friends NEED to be replaced by a replication controller ASAP. As it stands they are bad examples. Also, they should probably move to contrib/monitoring to match contrib/logging and contrib/dns


Question: Ready            bool } <del> var _ = Describe("[FLAKY] Container runtime Conformance Test", func() { </del> <ins> var _ = Describe("Container Runtime Conformance Test", func() { </ins> var cl *client.Client BeforeEach(func() {

Slow conformance test has caused issues like We should make runtime/image conformance test run faster without breaking the test coverage, possibly by: Shorten consistently check interval . Remove redundant test cases such as ... /cc


Question: // plugin resources for the input container, issues an Allocate rpc request // for each new device resource requirement, processes their AllocateResponses, // and updates the cached containerDevices on success. <del> func (m *ManagerImpl) allocateContainerResources(pod *v1.Pod, container *v1.Container) error { </del> <ins> func (m *ManagerImpl) allocateContainerResources(pod *v1.Pod, container *v1.Container, devicesToReuse map[string]sets.String) error { </ins> podUID := string(pod.UID) contName := container.Name allocatedDevicesUpdated := false

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug /area hw-accelerators What happened: In the context of the device plugin, when using init containers in your pod spec, the current behavior will not reuse devices between init containers and regular containers. This is problematic as Pod scheduling is currently wrong: , the current predicate algorithm rightly assumes the maximum number of devices in use at one time is . However, the pod predicate ran on the node is the sum of devices used by regular and init containers. Meaning that your pod can bounce back between the scheduler and a node infinitely. Also given that many of the extended resources are typically more expensive that the built-in ones like CPU, Memory and Storage this bug has very real financial drawbacks in terms of resource usage (e.g: using 4 GPUs on a pod that only uses 2 GPUs at a time). Furthermore, this unexpected behavior was caught during implementation (but we did not catch the scheduling implications) and decision was taken to ignore it for the first implementation. We need to document it for 1.9 What you expected to happen: Devices can be reused by Init Containers Correct resource accounting In terms of the current behavior of the Allocate call to the device plugin, I'd expect the allocate call to be done before every Init containers start and once at the end of the last Init container (for regular containers). /assign
Yep, I realized the inconsistency between the predicate behavior of scheduler and that of node too. Having said that, considering we treat a pod as a whole when predicating currently, it's kind of difficult to reuse the runtime informations between init containers and regular containers. The real crux of the matter might be whether the scenario of requiring extended resources in init containers is common, if so, the issue should be addressed ASAP.
If I remembered correctly, the current design to improve DP includes a reconciliation in admit phase of Pod, it may be able to fix Allocate call for
can you point me to said design ? I'm not sure that the Allocate call for init containers is planned to be fixed as described.
Can you guys please point to the code where is happening. Looks to me that same logic is executed on both the scheduler and the node to calculate a pod's resource request and which is  On the node, function call tree looks like this: Is it possible to add few more words in explanation?


Question: // plugin resources for the input container, issues an Allocate rpc request // for each new device resource requirement, processes their AllocateResponses, // and updates the cached containerDevices on success. <del> func (m *ManagerImpl) allocateContainerResources(pod *v1.Pod, container *v1.Container) error { </del> <ins> func (m *ManagerImpl) allocateContainerResources(pod *v1.Pod, container *v1.Container, devicesToReuse map[string]sets.String) error { </ins> podUID := string(pod.UID) contName := container.Name allocatedDevicesUpdated := false

PR that the mentioned change: Code in question is here:
Above looks incorrect description. Predicate that run on the scheduler and on the node is same i.e max(init, regular). Pod admission will pass at the node. Problem might happen at allocation where allocated resources is not the max but the sum of (init, regular). And in that case, pod would hang there in fail state rather 'bounce back infinitely'.
Problem might happen at allocation where allocated resources is not the max but the sum of (init, regular). And in that case, pod would hang there in fail state rather 'bounce back infinitely'. Absolutely correct, sorry to make you confused, I think Renaud name it mainly bacause the allocate func is supposed to be called during the predicate phase, that is, passed in as a part of PredicateAdmitHandler.
The original idea was onced listed in as a sub-task: Now it's removed (I don't know why ...), while I believe the idea was: device resources allocation during pod admit phase. This comment is also related if needed during reconcile in 1 and cache these devices info cached devices later during container create phase
Handle device resources allocation during pod admit phase. This comment (comment) is related Call Allocate() if needed during reconcile in 1 and cache these devices info Use cached devices later during container create phase Current code behaviors exactly as you described, but to address the issue, the key point might be to decide which devices of init containers could be reused by which regular containers, and generate cache accordingly in step2; when creating containers, we simply fetch runtime information from cache.
Thanks a lot for filing this issue, I think for 1.9, we probably want to list this as a known issue and may advise people to not request gpu resources in init containers. We can target to fix this problem in 1.10.
Agreed We don't have a clear solution for this tough. It's pretty clear to me that device resources allocation during pod admit phase is not the right model when init containers comes into the picture. GPUs are pretty simple example. On of the things we plan to do is scrub the GPU memory during the Allocate phase. If GPUs are re-used between initContainers and regular containers the current model doesn't fit.


Question: // plugin resources for the input container, issues an Allocate rpc request // for each new device resource requirement, processes their AllocateResponses, // and updates the cached containerDevices on success. <del> func (m *ManagerImpl) allocateContainerResources(pod *v1.Pod, container *v1.Container) error { </del> <ins> func (m *ManagerImpl) allocateContainerResources(pod *v1.Pod, container *v1.Container, devicesToReuse map[string]sets.String) error { </ins> podUID := string(pod.UID) contName := container.Name allocatedDevicesUpdated := false

do you think if you have time to implement the solution you mentioned in the above comment We can try to get it into a 1.9 sub release. I think for now, lets just support device reuse among initContainers and normal containers. We can revisit this part when we need to guarantee an Allocate call per container.
Right what I am mentioning here is that it is not possible to do that. The point of the device plugin is so that we can execute vendor specific code before making the device available in the container (e.g: Scrub the GPU memory). I don't think the proposed solution addresses this problem.
Sure. But to clarify, I think if we dont't expect big shock on current implementation, we could only provide limited support on the point, that is, without re-generate runtime information for regular containers, only those whose requested resource is equal to or is a subset of that in init containers could make a resuse(e.g. if we have two init containers, each requests 2 devices, then a regular container that request 3 devices still could not reuse the devices.) Besides, if a initcontainer requests 2 devices, and we have two regular containers, each requests 1 device, then due to the lack of runtime info re-generation, the devices could only be reused by one of the containers, not both of them. In sum, since we don't re-generate runtime information, we could only reuse devices per container, not per device, like we do in scheduler. Is that what you mean?
Hmm, that is a good question. I forgot that we now return the combined runtime information in AllocResponse. Then for now, perhaps we should just return an error in device plugin manager Allocate() in case both init containers and containers request the same device plugin resource?
You mean we give up reuse and prohibit users to request same kind of device in init containers and regular containers? that seems a bit weird. And if we decide to do so, we might need to do the check at the first beginning, i.e. in API validation, not wait until it goes into DevicePlugin.
I think for 1.9, we just return an error when both init containers and normal containers of a pod request the same device resource, and document this in the device plugin user guide.


Question: // plugin resources for the input container, issues an Allocate rpc request // for each new device resource requirement, processes their AllocateResponses, // and updates the cached containerDevices on success. <del> func (m *ManagerImpl) allocateContainerResources(pod *v1.Pod, container *v1.Container) error { </del> <ins> func (m *ManagerImpl) allocateContainerResources(pod *v1.Pod, container *v1.Container, devicesToReuse map[string]sets.String) error { </ins> podUID := string(pod.UID) contName := container.Name allocatedDevicesUpdated := false

The reason is because in the current device plugin implementation, we don't have a good way to free devices used by init containers of an active pod. It is hard to do this kind of validation at API level or even scheduler because we can't know whether an extended resource is device resource or not. So for now, I think we should probably just return a clear error message during admission handler. I am not sure yet whether there will be strong use cases that need to share devices between init containers and normal containers. We can revisit this based on user feedback. If there is strong user demand to support this, something we could do is to differentiate init containers and normal containers in podDevices, and free up devices used by init containers when any normal container is about to run. That would request some code structure change, and we may wait to see whether there is strong demand to support this.
It is hard to do this at API level or even scheduler because we can't know whether an extended resource is device resource or not. Oh, you are right, I ignored the resource awareness issue. I also agree that it's hard to address the issue without big change to current implementation. But I still feel the inconsistency of behavior between API and kubelet might confusing users. However, its effect should be sustainable, since we haven't received feedback about init containers up to now.
Right. So for now, perhaps just return an error when this happens, to give clear error signals.
Oh, I'm afraid we miss something, the check may not work as expected. As I commented, because init-containers run sequentially, we did reuse between init containers in , that is, if we have two init containers, each request 1 device, then we expect 1 device in total in scheduler, but expect 2 in deviceplugin. The check could not cover the scenario.
Thinking about this problem a bit more, I wonder whether we may tackle this issue from the following approaches, ordered by their complexities: 1) Just return an error whenever an init container requests a device plugin resource, and document in the device plugin user guide that requiring device plugin resource in init containers are not supported. This may be the short-term change we can try to get into 1.9. 2) Maintain a list of devices allocated to init containers in ManagerImpl.Allocate().


Question: // plugin resources for the input container, issues an Allocate rpc request // for each new device resource requirement, processes their AllocateResponses, // and updates the cached containerDevices on success. <del> func (m *ManagerImpl) allocateContainerResources(pod *v1.Pod, container *v1.Container) error { </del> <ins> func (m *ManagerImpl) allocateContainerResources(pod *v1.Pod, container *v1.Container, devicesToReuse map[string]sets.String) error { </ins> podUID := string(pod.UID) contName := container.Name allocatedDevicesUpdated := false

Try to allocate from this list first for later init containers and normal containers. The required no. of devices we pass to ManagerImpl.devicesToAllocate() should subtract the no. of devices we have allocated to init containers. This may require some code structure change around ManagerImpl.Allocate(), ManagerImpl.allocateContainerResources(), and ManagerImpl.devicesToAllocate(). However, Allocate gRPC call is still issued during Admission all together, and it is possible that we issue an Allocate gRPC for [dev1] (say dev1 allocated to initContianer1), followed by another Allocate gRPC call for [dev1, dev2] (say dev1 and dev2 allocated to initContainer2), with none of containers started or finished. This assumes containers running in the same pod share the same security context, and device plugin doesn't need to perform per-container allocation operations. We just use Allocate gRPC call to learn container runtime settings. 3) Take a further step from 2), during Admission, we just determine what devices to allocate to containers without issuing the actual Allocate gRPC calls. I.e., we only fill deviceIds field in podDevices.deviceAllocateInfo but leaves allocResp field empty. During ManagerImpl.GetDeviceRunContainerOptions(), we check for this case and issue the actual Allocate gRPC call for the container. This ensures we can issue a gRPC Allocate call per container. However, if a pod is restarted multiple times, we don't guarantee Allocate gRPC call will be issued during subsequent pod runs. 4) Take a further step from 3), we can check for overlapping device allocations in ManagerImpl.GetDeviceRunContainerOptions() and invalidate their allocResp so that an Allocate gRPC call will be issued next time for containers using re-allocated devices. E.g., for containerA, if we find it is allocated with [dev1, dev2] and dev1 is also allocated to containerB, we set containerB's allocResp to nil so that next time before containerB is running, we will issue an Allocate gRPC call for dev1. Note that this approach does have the drawback that device plugin failure would cause previously running pods not be able to restart, so we need to carefully evaluate our use cases to see whether we need to implement this. My suggestion is to implement 1) now and gather some user feedbacks to determine whether we need to implement the following approaches.


Question: // plugin resources for the input container, issues an Allocate rpc request // for each new device resource requirement, processes their AllocateResponses, // and updates the cached containerDevices on success. <del> func (m *ManagerImpl) allocateContainerResources(pod *v1.Pod, container *v1.Container) error { </del> <ins> func (m *ManagerImpl) allocateContainerResources(pod *v1.Pod, container *v1.Container, devicesToReuse map[string]sets.String) error { </ins> podUID := string(pod.UID) contName := container.Name allocatedDevicesUpdated := false

I see this a very big limitation. Special devices/resources in init containers is a very common use case IMO. Therefore we should try to fix this. This approach perfectly makes sense to me. What is your concern in implementing this? Even today, IIUC, Allocate call is made per container. Is there any other value that this approach is adding over 1) ? I am not able to see any gains with 2) or 3) over 1). Comments please.
I feel we may not be able to implement 2) for 1.9 timeframe. Option 2) also doesn't address the case mentioned above: Option 3) covers that case for pods without restarts. Option 4) fully covers the requirement we we will scrub old container data before giving it to a new container, even if they both run in the same pod, bug it does have the mentioned drawback during device plugin failure.
Take a further step from 3), we can check for overlapping device allocations in ManagerImpl.GetDeviceRunContainerOptions() and invalidate their allocResp so that an Allocate gRPC call will be issued next time for containers using re-allocated devices. E.g., for containerA, if we find it is allocated with [dev1, dev2] and dev1 is also allocated to containerB, we set containerB's allocResp to nil so that next time before containerB is running, we will issue an Allocate gRPC call for dev1. Note that this approach does have the drawback that device plugin failure would cause previously running pods not be able to restart, so we need to carefully evaluate our use cases to see whether we need to implement this. How about making the allocation call optional? That is , if we find a contaienr using re-allocated devices, we'd try to issue an Allocate gRPC call first, and still use the cached setting if allocation failed. And, if we decide not to address the issue in v1.9, which means we have enough time to work on it later, I think approach 2) might not be a good choice. Because it has a premise that devices are stateless, recycle actions before reuse(e.g: Scrub the GPU memory mentioned above) could not be supported. And this way, as I commented above, we could only reuse devices per container, not per device, like we do in scheduler.


Question: // plugin resources for the input container, issues an Allocate rpc request // for each new device resource requirement, processes their AllocateResponses, // and updates the cached containerDevices on success. <del> func (m *ManagerImpl) allocateContainerResources(pod *v1.Pod, container *v1.Container) error { </del> <ins> func (m *ManagerImpl) allocateContainerResources(pod *v1.Pod, container *v1.Container, devicesToReuse map[string]sets.String) error { </ins> podUID := string(pod.UID) contName := container.Name allocatedDevicesUpdated := false

I noticed that you have already add the limitation into the , so I think it's already the decision that we add the limitation as a short-time solution in v1.9 So I posted a PR to add the check in code,, PTAL :)
Use Case: Let say there is a which node has two Devices of a resource type X and the device IDs are X1 and X2. User pod has one init container, lets call it cont-init and one regular container, lets call it cont-reg. Both cont-init and cont-reg requests 2 unit of resource X. cont-init will run some vendor specific code on allocated Devices and then cont-reg will use these 'initiated' devices. I think this is the use case that mentioned. And i feel 2) can handle this. 2) approach will allocate both the devices, X1 and X2, to both the containers. When the containers start, init container, cont-init, will be run first and will run to completion and here devices will get initialized. Strictly after this cont-reg will be started and now this regular container has initialized devices. Where i am wrong?
I guess in RenaudWasTaken 's case, the initialization task is considered to be managed in the plugin, not in init-container, and in init-container we only use the device, so then we might need to call plugin to do recycle action before the device reused by reguler containers. In practical use, both of the two scenarios may exist, it depends on the request of user. Besides, if we don't issue an Allocate gRPC call, we could not ensure a ”complete reuse“: say an init-container requests 2 devices, and we have two regular containers, each requests 1 device, then due to the lack of runtime info re-generation, the devices could only be reused by one of the containers, not both of them.
It is the init containers' job to initialize things and if in some special case device plugin is doing the initialization part, in that case, init containers will not be needed and serve no purpose in pod spec. Specific to example of memory scrubbing, i think this also makes more sense to be done in init container. Once Device's memory is scrubbed, it can be used by regular container. Let try to run your use case with approach 2).


Question: // plugin resources for the input container, issues an Allocate rpc request // for each new device resource requirement, processes their AllocateResponses, // and updates the cached containerDevices on success. <del> func (m *ManagerImpl) allocateContainerResources(pod *v1.Pod, container *v1.Container) error { </del> <ins> func (m *ManagerImpl) allocateContainerResources(pod *v1.Pod, container *v1.Container, devicesToReuse map[string]sets.String) error { </ins> podUID := string(pod.UID) contName := container.Name allocatedDevicesUpdated := false

There are three containers in pod, cont-init(requested 2 unit or resource X), cont-reg1(requested 1 unit of resource X) and cont-reg2(requested 1 unit of resource X). Now in approach 2), gRPC Allocate call is made for cont-init requesting X1 and X2. Next, gRPC Allocate call is made for cont-reg1. This time resource id will be send in Allocate from the previously allocated [X1, X2]. Lets say X1 is requested for cont-reg1 Next, gRPC Allocate for cont-reg2 and resource with ID X2. All this allocation info gets stored in internal cache. Now when container runs, runtimeOptions will be collected from internal cache. Where do you see a problem in this flow?
/cc
Yep, it could work if the plugin behavior the way you described, but I'm not sure if all kinds of devices could support "pre-allocation", that is, doing allocation on a same device multiple times at the same time, to prepare runtime informations for several containers that reuse the device later. Also I have no idea if all plugin developers consider their plugins "stateless", that is, managing initialization in init-containers. As I don't have enough knowledge on hardware, would like to hear opinions from other guys :)
This would defeat the purpose of a container. In the case of GPUs, when you run a container you expect the GPU to always be in the same state. Furthermore GPUs can't be shared (exclusive usage) between containers meaning that Init containers should behave in the same way as regular containers. With the only difference being that they are executed sequentially and before regular containers. Additionally, operations to setup a device before making them available in a container, whether functionally (load the driver, insert libraries, ....) or security wise (scrub the memory), should be handled by the infrastructure not the users pod spec.
I agree that it's a sane behavior to forbid the use of Device Plugin Resources in Init Containers for the 1.9 release
moving forward,
/cc I think what long-term solution we decide to implement will largely depend on use-case requirements from different device plugins. Would like to hear some thoughts from different device plugin developers on this. Can people cc-ed in this issue take a look at the proposed options mentioned in and provide some feedbacks on which one may better fit your use cases?


Question: // plugin resources for the input container, issues an Allocate rpc request // for each new device resource requirement, processes their AllocateResponses, // and updates the cached containerDevices on success. <del> func (m *ManagerImpl) allocateContainerResources(pod *v1.Pod, container *v1.Container) error { </del> <ins> func (m *ManagerImpl) allocateContainerResources(pod *v1.Pod, container *v1.Container, devicesToReuse map[string]sets.String) error { </ins> podUID := string(pod.UID) contName := container.Name allocatedDevicesUpdated := false

Answering on behalf of and Solution 1 makes sense for the moment Solution 2 defeats the purpose of the device plugin, which is setting up a device before making it available to a container. solution 4 seems to be a sane solution. As mentioned in my previous comment, as long as devices are exclusive, I expect the devices exposed in my container to always be in the same state at the start of my container. When / If we implement a mechanism for sharing a GPU / devices across containers then the scenario described by is something that we can have. It doesn't however fit the current situation.
Solution number 1 seems totally reasonable short-term. Long term, simplifying and documenting the relationship between plugin callbacks and container lifecycle would help plugin authors.
For reference, I implemented option 2 solution in PR The change itself is small. This is an alternative short-term fix we may consider to get into a 1.9 patch release. I think the main benefit of taking this fix compared to Option 1 solution is that we can change the underlying implementation to support init containers later with less user-facing impact.
No mater we take PR or PR for 1.9, I think we need to have this issue approved for 1.9 milestone. or or can any of you approve this issue for 1.9? Thanks!


Question: if l > 0 { n += 1 + l + sovApi(uint64(l)) } <ins> l = len(m.SeccompProfilePath) if l > 0 { n += 1 + l + sovApi(uint64(l)) } </ins> return n }

CRI supports AppArmor, Seccomp and Sysctls via annotations , we should move them to typed and structured fields instead. (refer ) [x] Apparmor () [x] Sysctls () [x] Seccomp () cc
/cc
/sig node
Seccomp is addressed at .


Question: if l > 0 { n += 1 + l + sovApi(uint64(l)) } <ins> l = len(m.SeccompProfilePath) if l > 0 { n += 1 + l + sovApi(uint64(l)) } </ins> return n }

Seccomp profile is passed to runtime by in CRI: is relative to node's local seccomp profile root, it is defined in kubelet. But runtime doesn't know it. We should pass full profile path in CRI. cc/ Will file a PR to fix this later.
Add support for Seccomp sounds like a new feature. The milestone should be reset to , correct?
not really. The current implementation supports seccomp already. If it's missing critical information to function properly, this would be a bug.
are we blocking releases for CRI in 1.5? i was not sure, so marking this as non-release-blocker and leaving 1.5 milestone. please correct if i am mistaken.
I don't get why this is a 1.5 targetted thing myself. Yes we want CRI to be in a good state in 1.5, but afaik we have zero intention of anyone actually using (at the very least because of the lack of exec auth regressing kubelet security). why should this be part of 1.5?
First, this is a small enough change that allows non-docker integrations to properly support seccomp. Second, if "we have zero intention of anyone actually using" is a valid reason, we should never fix any bugs in CRI. We do want people to try using it (perhaps not in production yet). However, given that seccomp is only in alpha, I am ok with dropping this and just continue the docker support for now.
Perhaps I misunderstood the meaning of code freeze, but I thought it was only bug-fixes regardless of size of change (or exceptions that have gone through some process) What I meant is we don't intend this to be used generally as part of the 1.5 release. Because of that, I don't think we should consider bugs in CRI bugs for the purpose of the 1.5 release freeze. My statement is just that we shouldn't violate code-freeze for this. We should totally fix this right after the 1.5 freeze window ends, and saying that I'm arguing we should never fix anything is disingenuous and taking it to an absurd extreme.
the only thing we don't seem to agree on is whether seccomp support was considered to be supported for CRI in 1.5.


Question: if l > 0 { n += 1 + l + sovApi(uint64(l)) } <ins> l = len(m.SeccompProfilePath) if l > 0 { n += 1 + l + sovApi(uint64(l)) } </ins> return n }

My interpretation is that we assumed seccomp would be supported until we found out that it wouldn't for other runtimes due to the lack of profile path. That's the only reason why I considered this a bug. Of course we could say that this is not really an important feature in CRI, so we can drop it, but we can be more clear on that.
If there's a bug in a feature no one uses, does fixing it make a sound? I think what we're not agreeing on is the role of CRI in 1.5. I understood it as something that is not generally usable. We can't recommend a customer use it, even as an experimental feature, due to security aspects (if nothing else). Correct me if I'm wrong. It's a developer integration point, and developers integrating with it are doing so with the kubernetes master branch, not with some release. Those reasons together result in my opinion that CRI is still not really a customer/release feature, and thus bugs or features related to it should not be exceptions to the freeze window, but rather should targeted at master as soon as the freeze lifts. If this were already post-freeze, would we be cherry-picking this over to 1.5?
This is not true. cri-o is depending on this fix for apparmor support.
is it depending on the 1.5 release branch directly, or on master?
I think it's better to make this in 1.5, so cri-o could work with first release with experimental CRI. It helps to encourage people to use CRI and find features misses or bugs.
Modifying the user-specified annotation is a little bit unsettling. I think adding a new annotation explicitly for the path would be better. Or, we should just add an explicit field for this. I spoke to and separately. I still think this is a bug fix, but we may want to be more careful about the fix. Given the time constraint, let's just list this as a known issue and work on it for 1.6.
OK, let's move this to next release and think carefully about seccomp feature besides this: kubelet may just pass the profile as string via CRI; may assume the profile locate in a file in a known format and pass path to the runtime; or maybe some other solutions.


Question: if l > 0 { n += 1 + l + sovApi(uint64(l)) } <ins> l = len(m.SeccompProfilePath) if l > 0 { n += 1 + l + sovApi(uint64(l)) } </ins> return n }

We need to clearly define it. define profile format clearly
Please document known issues for 1.5 in
(copied from ) And let's move here to discuss how to pass profiles to runtime. Related with .
This is issue is related with two level APIs: CRI, since we may also want to support profiles from ConfigMaps, I think we should pass full profile contents instead of path to runtime. issues is also related with seccomp api in kube-apiserver, it is based on annotations today. We should define seccomp profile format clearly (see ) and decide whether to add struct types to pod spec or just continue the annotations way. I think it's more clear to add specific struct types to pod spec.
Agreed - the path stuff is really a crutch (and implementation-specific) - I think is the answer here.
I don't disagree, but it's not strictly a requirement for supporting ConfigMaps (kubelet could create a file with the contents). An open question is whether to pass it as a raw string, or wait for and pass the structured data. The annotations way is deprecated (). I think the bigger question is where/how to put the data. E.g. structured data in the API, unstructured (string type) data in the API, reference to another object (e.g. ConfigMap). If the Kubelet needs to parse the spec, I think that's a good argument for a structured data type (either a standalone object or embedded in the pod API).
Yes, already opened to replace annotations with typed structs. I personally prefer a structured data type in both kubernetes API and CRI. WDYT?
A counter argument is that as the spec grows or new features are , we need to keep updating the Kubernetes API to maintain parity, or update the code in each CRI implementation to translate the Kubernetes spec. OTOH, if we use unstructured data we can use 3rd party libraries to parse out the data we need (assuming the spec evolves in a backwards compatible way).
What spec?
Whichever seccomp profile spec is being implemented by runtimes. E.g. if we decide to base the Kubernetes seccomp profile API off the OCI seccomp profile spec, and OCI adds some new features, at some point we'd probably want to add those features to Kubernetes.


Question: if l > 0 { n += 1 + l + sovApi(uint64(l)) } <ins> l = len(m.SeccompProfilePath) if l > 0 { n += 1 + l + sovApi(uint64(l)) } </ins> return n }

It also means that if different runtimes support different feature sets, we'll be stuck either tracking only the common features, or specifying features that may be ignored by the runtime. This may be a non-issue if the seccomp feature set rarely changes.
Well, yes, but that applies to everything in the CRI, and is one of the tradeoffs we ma[d|k]e by introducing this abstraction in the first place.
My $0.02: Moving the existing logic/information for seccomp into a first-class field for CRI (as in ) is productive because of the known problems with the annotation approach. My initial proposal for seccomp support included a resource similar to what has proposed in kubernetes/community and we were unable to come to an agreement on the issue refers to here: I do not necessarily expect that we will be able to come to an agreement about the seccomp resource in time for 1.7, so I think we should proceed with and do backward compatible CRI changes later if we need to inject the content of the seccomp profile instead of the path. I think we would deeply regret embedding API surface to describe seccomp profiles in the Pod API :)


Question: echo "== kube-up node config done ==" else echo "== kube-push node config starting ==" <ins> ensure-basic-networking </ins> ensure-install-dir set-kube-env create-salt-pillar

Node 2's kubelet logs: controller-manager side: I have a cluster to hand off if someone wants to debug. It's not a transient state: node 2 just failed to ever post a valid status. cc cc (who looked at it briefly with me)
I will take a look soon.
Ok, why kubelet on node 2 talks to ip address of master instead of hostname? root ps -ef | grep kubelet root 1 3 17:35 ? 00:10:45 /usr/local/bin/kubelet --apiservers=https://10.240.49.47 --config=/etc/kubernetes/manifests --allowprivileged=False --v=2 --clusterdns=10.51.240.10 -- It looks like a cluster up issue to me, not a node problem.
It looks like failed to translate the master name into the fqdn (which is done ).
Fascinating. I think we're seeing instance-DNS flakiness, because that name is supposed to get converted ASAP. But what's weirder is ... that's not even the master:
The is definitely right:
Okay, this is messed up. Here's node 1, which is happy: Here's our unhappy node .... that decided to bring itself as a master:
haha... I checked kube-env and did suspect unhappy node treat itself as master node based on those variables. But I didn't check salt state :-)
I ... just can't get there. By the time we get here: , node-2 has set: My best guess is that flaked here: . I think we're seeing a systemic issue where we're racing with GCE network configuration.
I'm going to work up a PR that hopefully improves the reliability of this section to some extent.
<a href=""<img src="" title="made at "/</a
My working theory is that we raced with the configuration done normally by the GCE NetworkManager, leaving us in a bogus state when we made that check.
Why are we relying on inferring the master/node status from the hostname. It should be obvious from the variables set in the kube-env if you are the master or not. Masters don't get passed so why not just check that? Making sure DNS is working is probably also good for reliability...
On GKE, we actually pass the same kube-env on both sides for simplicity.


Question: echo "== kube-up node config done ==" else echo "== kube-push node config starting ==" <ins> ensure-basic-networking </ins> ensure-install-dir set-kube-env create-salt-pillar

We can obviously change that, but I was going for a change that didn't involve a deployment change on our side, too.


Question: codec:       codec, versioner:   versioner, transformer: transformer, <del> pathPrefix:  prefix, watcher:     newWatcher(c, codec, versioner, transformer), </del> <ins> // for compatibility with etcd2 impl. // no-op for default prefix of '/registry'. // keeps compatibility with etcd2 impl for custom prefixes that don't start with '/' pathPrefix: path.Join("/", prefix), watcher:    newWatcher(c, codec, versioner, transformer), </ins> } if !quorumRead { // In case of non-quorum reads, we can set WithSerializable()

an API server with , populate with data, stop server etcd2 to etcd3 the API server with data is missing from the API. The etcd2 client takes the configured prefix and passes it to the etcd client as . The etcd3 client simply passes in the configured prefix. is a no-op on the default prefix (`/registry), but can modify custom prefixes that did not start with a slash. cc
I think we have to preserve the behavior from the etcd2 client setup. It's a no-op for the default prefix, but requiring a change to a custom prefix when switching from etcd2 to etcd3 is really unexpected, especially when the failure mode is that the apiserver starts up as if it's a brand new cluster (including creating default projects, kubernetes service, etc)
cc
/cc re: upgrade testing.
Throwing release blocker on here just so we don't lose it (given data loss potential)
- thank for debugging and fixing it.


Question: return coreCount } <ins> func detectThreadPerCore() int { outData, err := exec.Command("/bin/sh", "-c", "lscpu | grep "Thread(s) per core:" | cut -d ":" -f 2").Output() framework.ExpectNoError(err) threadCount, err := strconv.Atoi(strings.TrimSpace(string(outData))) framework.ExpectNoError(err) return threadCount } </ins> func makeContainers(ctnCmd string, ctnAttributes []tmCtnAttribute) (ctns []v1.Container) { for _, ctnAttr := range ctnAttributes { ctn := v1.Container{

/kind failing-test <!-- If this is a flake, please add: /kind flake --/sig node started to fail since 09-13.
/cc
/assign
/cc
/triage accepted /priority important-soon it's indeed failing too often
/reopen The test is still failing: Now timing out at 120s i.e. 2 minutes as we increased the default timeout . We need to understand the root cause and get to the bottom of this.
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Snippet from the build log () corresponding from this failure We see from the above logs that: node has two numa nodes, 8 cpus per NUMA so the root cause of the failure is not because the tests are running on a node that doesn't have multi-numa support. is started and the health check fails a couple of times but eventually succeeds. pod is created with 9 cpus (cpusperNUMA +1 i.e. 8+1) which is expected to fail with Topology Affinity Error as we cannot allocate resources for such a pod from a single NUMAnode. We fail to get matching metrics. It is not clear if the check fails because we don't get kubelet metrics at all or the expected metrics but based on point 6 below it seems like it is because of latter. We can add additional logs for more clarity here. each is executed and Kubelet is restarted and health check is done. are collected from the namespace corresponding to the test pod which shows that the pod started even though it was expected to fail with Topology Affinity Error. We need to understand why the pod got successfully created even though it requested CPUs that couldn't be allocated from a single NUMAnode.


Question: return isErrCode(err, ErrCodeResourceVersionConflicts) } <ins> // IsRequestTimeout returns true if and only if err indicates that the request has timed out. func IsRequestTimeout(err error) bool { return isErrCode(err, ErrCodeTimeout) } </ins> // IsInvalidObj returns true if and only if err is invalid error func IsInvalidObj(err error) bool { return isErrCode(err, ErrCodeInvalidObj)

It is possible to issue a list request which exceeds the duration set by the request-timeout apiserver flag. The request-timeout flag does not appear to be enforced when the apiserver is decoding the response from etcd. Specifically, listing a large number of CRDs from etcd can result in high request latency from using json encoding compared to protobuf encoding. While overall etcd latency is low, the overall listing from storage latency may be over the 1 minute timeout, particularly if the apiserver needs to fetch multiple pages from etcd. We expected the list requests to be subject to the duration specified by the request-timeout flag when this latency is coming from the listing from storage phase of the request. Create 4000 CRs and list all the objects from etcd in 2 pages of 2000 objects. If the time to list 1 page + decode takes around 50s, listing 2 pages will result in the listing from storage phase of the request taking around 1 minute and 40s before the request fails and the client is returned a 500 response code. Logs for apiserver Trace message: Audit log entry: Related issue for supporting protobuf for CRDs: <details</details<detailsAWS </details<details</details<details</details<details</details<details</details
/sig api-machinery CC:
Interesting.. - do you know where exactly we stuck? In the call to storage: we pass the context that has timeout set, so that shouldn't hang. OTOH, for the processing part: (a) I don't know if transformer honors the context - it probably depends on the implementation (b) but the decoding (appendListitem) certainly isn't honoring the context So (b) might definitely be a problem, even independently from from the transformer So I guess what I'm saying is that if listing from etcd finishes fast but the decoding is what takes most of time, then what you described can certainly be a problem. But is that the thing you're observing?
Yes, that is our understanding. Specifically, that the decoding which occurs in appendListItem doesn't respect the context: As a separate follow-up, it would be useful to include decoding time, or at least the overall listing from storage time, as a separate latency annotation, given the audit log entry is only capturing this latency as part of
+1.


Question: return isErrCode(err, ErrCodeResourceVersionConflicts) } <ins> // IsRequestTimeout returns true if and only if err indicates that the request has timed out. func IsRequestTimeout(err error) bool { return isErrCode(err, ErrCodeTimeout) } </ins> // IsInvalidObj returns true if and only if err is invalid error func IsInvalidObj(err error) bool { return isErrCode(err, ErrCodeInvalidObj)

We need more granular latency tracker especially on the latency spent on decoding. Right now, it only captures etcd round trip and transformer latency.
I would be supportive for both and would be happy to review it if someone opens up a PR for it.
/triage accepted
/assign
/reopen I think we need to reopen it to document how we resolved the unit test failures brought on by adding it
Reopened this issue. <detailsIn response to : /reopen I think we need to reopen it to document how we resolved the unit test failures brought on by adding it Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: # Use skopeo to find these values: https://github.com/containers/skopeo # # Example <del> # Manifest: skopeo inspect docker://gcr.io/k8s-staging-build-image/debian-base:v2.1.3 # Arches: skopeo inspect --raw docker://gcr.io/k8s-staging-build-image/debian-base:v2.1.3 </del> <ins> # Manifest: skopeo inspect docker://gcr.io/k8s-staging-build-image/debian-base:buster-v1.2.0 # Arches: skopeo inspect --raw docker://gcr.io/k8s-staging-build-image/debian-base:buster-v1.2.0 </ins> _DEBIAN_BASE_DIGEST = { <del> "manifest": "sha256:37cfe133a6ff3fc3aa4aa5ab9fda127861902940b8e8078fff7191c7a81be8d8", "amd64": "sha256:dc06e242160076b72bd75135fb3dd0a9e91f386b2d812ec10cbf9e65864c755d", "arm": "sha256:9c52e9b31d679102586381fb4a03bba73fc05992adacce31df3db6d2f75f010e", "arm64": "sha256:9a3ae250d59f317c9cf1f7bf0b61c0d90e70012ee82996867655401968076ee4", "ppc64le": "sha256:d7c50e06d954bedb1f362ce42380f7a059423f8cbd9e7b426a7f2d73dfb4431a", "s390x": "sha256:7e6d43baf4972f7faa33f1179fb92ff858a3e0e23f78b96a5821d13199b9da91", </del> <ins> "manifest": "sha256:ea668d3febd312e0edfbbdab6bd7d86448ddc8fddb484e20ec76b36a7aeac04c", "amd64": "sha256:2f3e61995bcd4b3a1a0aef49e4a7a6817c978031734b09df2aaaa28181898b0e", "arm": "sha256:d1073dcf8f1d55fbbd297e5b280375b6f276ea83a08a25fd59dc4f3ca43c6d50", "arm64": "sha256:dadcff1ab81177de4914f6db0e7d78a52e525daf7a1efb246cb3545de5e818d1", "ppc64le": "sha256:bfb24dc0d1e71e1deb0f04a078fadf2c94070266746b1b5acc4e739aa57d5601", "s390x": "sha256:cfe6a3508b7ee198cb5a0b3a62e0981676b1dfa4b3049f36398d03e6bd35a801", </ins> } # Use skopeo to find these values: https://github.com/containers/skopeo # # Example <del> # Manifest: skopeo inspect docker://gcr.io/k8s-staging-build-image/debian-iptables:v12.1.2 # Arches: skopeo inspect --raw docker://gcr.io/k8s-staging-build-image/debian-iptables:v12.1.2 </del> <ins> # Manifest: skopeo inspect docker://gcr.io/k8s-staging-build-image/debian-iptables:buster-v1.3.0 # Arches: skopeo inspect --raw docker://gcr.io/k8s-staging-build-image/debian-iptables:buster-v1.3.0 </ins> _DEBIAN_IPTABLES_DIGEST = { <del> "manifest": "sha256:fff1fd5ab38fefde8c9eee5470ed6ea737f30e5ab86e1c0c0d429fa6add28a84", "amd64": "sha256:448c0e019689da3f4e238922824551d02578473f7b5d11604fffd528056caafa", "arm": "sha256:08e267a6297640eb0b218a783cabaef0a039cc4734b4a2d9cb54ee41cd82656a", "arm64": "sha256:a83cf1d501ad33f5aa93e2719baa59b054939b8a819c3997f915a6acfaa8e31a", "ppc64le": "sha256:c86649ac541c35431a4df8ba44e7c61923d9e34679fd17da7723db7b6ecc5245", "s390x": "sha256:53437fe32e13bb0bd2af08ddaaf2e908add8ab576574d881ba79544f34a774b2", </del> <ins> "manifest": "sha256:4c9410a4ee555dcb0e8b7bd6fc77c65ac400f7c5bd4555df8187630efaea6ea4", "amd64": "sha256:e30919918299988b318f0208e7fd264dee21a6be9d74bbd9f7fc15e78eade9b4", "arm": "sha256:bf59578f532bfd3378c4a713eeb14cf0dbed224d5ad03f549165f8d853997ca4", "arm64": "sha256:3d7ede6013b0516f1ec3852590895d4a7b6ec8f5e15bebc1a55237bba4538da2", "ppc64le": "sha256:ebd3bb280f8da8fc6a7158b7c2fc59b4552487bffd95d0b5ac1b190aff7b0fd9", "s390x": "sha256:a73b94aace7a571f36149aa917d4c7ee13453ed24a31f26549dd13b386dae4c1", </ins> } # Use skopeo to find these values: https://github.com/containers/skopeo

What would you like to be : Have debian-iptables with iptables 1.8.5+ Why is this needed: To fix (nodelocaldns + iptables-nft) More informations Found a bug in iptables-nft, fixed in iptables 1.8.5, and now available in buster-backports https://alioth-
/sig api-machinery
ping as you did the last 2 updates
Yep, this is on my list. Will take of it next week. /sig release network /area release-eng /milestone v1.20 /assign
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
Opened to bump the debian-iptables images in kubernetes/kubernetes. /remove-triage unresolved
Here are the digests for the new image ():
Please reopen if you run into any issues with the new images. /close
Closing this issue. <detailsIn response to : Please reopen if you run into any issues with the new images. /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: }) framework.ExpectNoError(err) }) <ins> ginkgo.It("should verify that all csinodes have volume limits", func() { driverInfo := driver.GetDriverInfo() if !driverInfo.Capabilities[storageframework.CapVolumeLimits] { ginkgo.Skip(fmt.Sprintf("driver %s does not support volume limits", driverInfo.Name)) } l.ns = f.Namespace l.cs = f.ClientSet l.config, l.testCleanup = driver.PrepareTest(f) defer l.testCleanup() nodeNames := []string{} if l.config.ClientNodeSelection.Name != "" { // Some CSI drivers are deployed to a single node (e.g csi-hostpath), // so we check that node instead of checking all of them nodeNames = append(nodeNames, l.config.ClientNodeSelection.Name) } else { nodeList, err := e2enode.GetReadySchedulableNodes(f.ClientSet) framework.ExpectNoError(err) for _, node := range nodeList.Items { nodeNames = append(nodeNames, node.Name) } } for _, nodeName := range nodeNames { ginkgo.By("Checking csinode limits") _, err := getNodeLimits(l.cs, l.config, nodeName, driverInfo) if err != nil { framework.Failf("Expected volume limits to be set, error: %v", err) } } }) </ins> } func cleanupTest(cs clientset.Interface, ns string, runningPodName, unschedulablePodName string, pvcs []*v1.PersistentVolumeClaim, pvNames sets.String, timeout time.Duration) error {

Right now, we have e2e test for that executes only CSI mock driver with artificial limit of 2 volumes per node. We need a test with real CSI driver(s). Proposal: Allow individual test storage drivers to specify if they support CSI limits. Add a test that picks a random node and creates as many pods with a PV that the volume can handle (128 on GCE!). Check that the node can't handle any other volume. Make this test part of . Add similar test(s) for in-tree volume plugins. Make sure the tests work with in-tree volume plugins migrated to CSI.
/sig storage
/assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
This sounds important /remove-lifecycle rotten /lifecycle frozen
/priority backlog
To add to this I believe we should test that an in-tree plugin and its "replacement" CSI Driver share the same attach limits (both max and current usage)
/cc
I see this test has been here: But I want to add a more basic/faster version of the test that simply checks if a driver has then has been set in CSINode, similar to wdyt, would it be useful?
This is a really old issue :-D. I think the required test has been implemented, however, we do not run it in CI, because it takes too much time to provision 128 PDs that a GCP node can handle. /unassign That sounds useful!
/assign opened to close this for good :)


Question: "comparer.go", "debugger.go", "dumper.go", <ins> "signal.go", "signal_windows.go", </ins> ], importpath = "k8s.io/kubernetes/pkg/scheduler/internal/cache/debugger", visibility = ["//pkg/scheduler:__subpackages__"],

PLEASE COORDINATE WITH SIG-SCHEDULING BEFORE WORKING ON THIS ISSUE. We want to avoid the situation where two people accidentally start work on the same fix. Main tracking issue: Currently, the feature is set up at the end of in . This setup should be moved into its own helper function, preferably in the package. /sig scheduling /kind cleanup /help
/assign /remove-help
I'd like to take this one
/remove-help
the PR I sent should resolve this one. :disappointed: Are any of the others still available?


Question: } } <ins> func isContainerRunning(c clientset.Interface, namespace, podName, containerName string) wait.ConditionFunc { return func() (bool, error) { pod, err := c.CoreV1().Pods(namespace).Get(context.TODO(), podName, metav1.GetOptions{}) if err != nil { return false, err } for _, statuses := range [][]v1.ContainerStatus{pod.Status.ContainerStatuses, pod.Status.InitContainerStatuses, pod.Status.EphemeralContainerStatuses} { for _, cs := range statuses { if cs.Name == containerName { return cs.State.Running != nil, nil } } } return false, nil } } </ins> // LogPodStates logs basic info of provided pods for debugging. func LogPodStates(pods []v1.Pod) { // Find maximum widths for pod, node, and phase strings for column printing.

This issue is to discuss an e2e testing plan for Ephemeral Containers. /sig node
As a first step towards this issue, I am planning to create node-e2e tests (under ) to verify the following: With a running pod client adds a valid EphemeralContainer resource, and the ephemeral container starts from the ephemeral container, list processes of itself and other containers, and then signal other processes (if is ) from the ephemeral container, ping & curl other containers try to read/write to stdin/stdout of the ephemeral container verify resource request/limit values against calculated expectations check oom scores of processes in the ephemeral container verify volume mounts restarts according to policy, when process exits client tries to add EphemeralContainer resource with ports (should be rejected) client tries to add EphemeralContainer resource with probes (should be rejected) client tries to add EphemeralContainer resource with resources (should be rejected) client tries to add EphemeralContainer resource with securityContext (should be rejected) removes the ephemeral container, and the ephemeral container is stopped and deleted adds multiple ephemeral containers, should all startWith pod of non-running status (TBD)
Awesome, thanks! This is a great start, here are some things to look out for: Visibility should work, but I expect signaling to fail without which is not currently allowed. iirc was interested in how this would be tested for ephemeral containers. Can you point us towards any examples that would help us get started? Ephemeral containers will never restart (and the policy should always be never) These will be denied by the api server without ever making it to the kubelet, so they are better tested in the apiserver integration tests (and I think they are already). Removing ephemeral containers is not currently allowed. /assign
Validation of ephemeralContainers is well tested in Testing of removing ephemeral containers is in ,
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with .


Question: } } <ins> func isContainerRunning(c clientset.Interface, namespace, podName, containerName string) wait.ConditionFunc { return func() (bool, error) { pod, err := c.CoreV1().Pods(namespace).Get(context.TODO(), podName, metav1.GetOptions{}) if err != nil { return false, err } for _, statuses := range [][]v1.ContainerStatus{pod.Status.ContainerStatuses, pod.Status.InitContainerStatuses, pod.Status.EphemeralContainerStatuses} { for _, cs := range statuses { if cs.Name == containerName { return cs.State.Running != nil, nil } } } return false, nil } } </ins> // LogPodStates logs basic info of provided pods for debugging. func LogPodStates(pods []v1.Pod) { // Find maximum widths for pod, node, and phase strings for column printing.

Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen /assign /priority important-soon
Reopened this issue. <detailsIn response to : /reopen /assign /priority important-soon Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-contributor-experience at . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Rotten issues close after 30d of inactivity.


Question: } } <ins> func isContainerRunning(c clientset.Interface, namespace, podName, containerName string) wait.ConditionFunc { return func() (bool, error) { pod, err := c.CoreV1().Pods(namespace).Get(context.TODO(), podName, metav1.GetOptions{}) if err != nil { return false, err } for _, statuses := range [][]v1.ContainerStatus{pod.Status.ContainerStatuses, pod.Status.InitContainerStatuses, pod.Status.EphemeralContainerStatuses} { for _, cs := range statuses { if cs.Name == containerName { return cs.State.Running != nil, nil } } } return false, nil } } </ins> // LogPodStates logs basic info of provided pods for debugging. func LogPodStates(pods []v1.Pod) { // Find maximum widths for pod, node, and phase strings for column printing.

Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-contributor-experience at . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen /remove-lifecycle rotten
Reopened this issue. <detailsIn response to : /reopen /remove-lifecycle rotten Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: Image:           kubeProxyE2eImage, ImagePullPolicy: "Always", Args: []string{ <del> "net", "--serve", fmt.Sprintf("0.0.0.0:%d", testDaemonHTTPPort), </del> <ins> "net", "--serve", fmt.Sprintf(":%d", testDaemonHTTPPort), </ins> }, }, },

Which jobs are failing: kops-aws and many other jobs Which test(s) are failing: This isn't the actual test, but the text we're consistently seeing is Since when has it been failing: Testgrid links: (ref: ) Reason for failure: TBD Anything else we need to know: Very rarely tests pass when one of the tests passes: Seems related to , that was merged about the same time:
/sig cluster-lifecycle /sig aws
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig cluster-lifecycle /sig aws Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/priority critical-urgent this is merge blocking
/assign /cc /cc /cc
The PR we think we want to revert
is also failing, I don't think this is aws-specific
/cc
/assign
it failed with kind too :/ with the same message
Opened to revert
/unassign /sig network /sig testing /remove-sig cluster-lifecycle
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
This may be the actual cause PR to address But we're going with revert first to be sure we've identified the offending PR
I think I've identified the root cause
Revert brought us back to green: Awaiting root cause fix
/remove-triage unresolved
/reopen merged but we're now seeing multiple tests fail since it merged Looking at it went red between these commits:
Reopened this issue. <detailsIn response to : /reopen merged but we're now seeing multiple tests fail since it merged Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
is an attempt to fix
reverting in
is the failure cluster I suspect will go down once the revert goes in
/close Failure cluster has gone down, calling this done <img width="998" alt="Screen Shot 2020-01-10 at 9 56 17 AM" src="https://user-"
Closing this issue.


Question: Image:           kubeProxyE2eImage, ImagePullPolicy: "Always", Args: []string{ <del> "net", "--serve", fmt.Sprintf("0.0.0.0:%d", testDaemonHTTPPort), </del> <ins> "net", "--serve", fmt.Sprintf(":%d", testDaemonHTTPPort), </ins> }, }, },

<detailsIn response to : /close Failure cluster has gone down, calling this done <img width="998" alt="Screen Shot 2020-01-10 at 9 56 17 AM" src="https://user-"</details


Question: github.com/vishvananda/netlink/nl # github.com/vishvananda/netns v0.0.0-20171111001504-be1fbeda1936 => github.com/vishvananda/netns v0.0.0-20171111001504-be1fbeda1936 github.com/vishvananda/netns <del> # github.com/vmware/govmomi v0.20.0 => github.com/vmware/govmomi v0.20.0 </del> <ins> # github.com/vmware/govmomi v0.20.1 => github.com/vmware/govmomi v0.20.1 </ins> github.com/vmware/govmomi github.com/vmware/govmomi/find github.com/vmware/govmomi/list

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: The vSphere cloud provider is unable to fetch the node failure domain labels when using VCP with SAML auth because of an authentication failure. For context, the SAML token auth is described here - SAML auth support for the zones api was introduced in this PR - , which was cherry-picked to 1.14.1 here - I've tested the credentials I'm using with govc to ensure that I'm able to get a token: The credentials also work fine when using VCP without the node labels feature. Relevant kubelet logs: What you expected to happen: VCP should add failure domain labels to the node object. How to reproduce it (as minimally and precisely as possible): Setup SAML auth for VCP with zone labeling enabled. Anything else we need to know?: There was a previous issue that was raised () but that was before SAML auth support for zones was implemented. This could potentially be an issue with govc handling of SAML tokens since even with the Administrator role assigned to the user, I'm still unable to list tags. The role and keys were generated with the following: Environment: Kubernetes version (use ): 14.1 Cloud provider or hardware configuration: vSphere OS (e.g: ): Linux & Windows
/sig vmware
Would you mind taking a look at this? I can provide more info if needed.
govc's command only works with the SOAP endpoint currently. The REST endpoint has its own session manager and requires a separate login. At the moment, govc's tags commands login/logout when each command is run and don't use a persisted session (which are saved to ~/.govmomi/sessions). However, you can use a token returned by to test SAML auth with the tags commands. For example: Can you give that try? There are some TODOs around this btw:
Those commands work for me. So that would imply the solution user, role, and permissions are set up correctly? Would you expect the to also work?


Question: github.com/vishvananda/netlink/nl # github.com/vishvananda/netns v0.0.0-20171111001504-be1fbeda1936 => github.com/vishvananda/netns v0.0.0-20171111001504-be1fbeda1936 github.com/vishvananda/netns <del> # github.com/vmware/govmomi v0.20.0 => github.com/vmware/govmomi v0.20.0 </del> <ins> # github.com/vmware/govmomi v0.20.1 => github.com/vmware/govmomi v0.20.1 </ins> github.com/vmware/govmomi github.com/vmware/govmomi/find github.com/vmware/govmomi/list

I seem to be getting a 404 when using with the flag - Does seem to work when specifying a tag name: Are there any other govc commands I should check / verify?
yes, the solution user setup looks right. Not sure what the issue is with . Can you share your vCenter build number ()? I'll try to reproduce the issue.
$ govc about -u vcdo Name: VMware vCenter Server Vendor: VMware, Inc. Version: 6.7.0 Build: OS type: linux-x64 API type: VirtualCenter API version: 6.7.2 Product ID: vpx UUID: -55ed-4737-9cc4-
I can reproduce this, but don't know what the issue is yet. The version of Go and vSphere have changed since , but if it was one of those was the cause, I'd expect SAML auth to fail with the govc tags commands too. All I have to go on for now is the vCenter log message below. I'll update when I have more info.
I have it working with the change below, but need to figure out a few things first: Why this is the case when run inside kubelet and not govc (note that the port change to 443 breaks govc) A good way to determine which port to use (depends on the answer to the above) If this is behavior is specific to 6.7U2 or something triggered by a change in K8s I suppose a quick fix would change to try port 443 if 80 fails, but would prefer to avoid that as the failure generates stack traces in the
/assign
This is obvious in retrospect, but the issue is the message signature needs to use the same port as the original request URL. However, in the govc case there was no port set in the request URL. The client defaults to port when connecting (based on the scheme), but does not include any port with the header. In the case, the default is on the server side. This can be reproduced with if includes the port , rather than just as we were using. The VCP tests that use token auth for tags lookup just don't validate the signature, so this wasn't a change related to 6.7U2.


Question: github.com/vishvananda/netlink/nl # github.com/vishvananda/netns v0.0.0-20171111001504-be1fbeda1936 => github.com/vishvananda/netns v0.0.0-20171111001504-be1fbeda1936 github.com/vishvananda/netns <del> # github.com/vmware/govmomi v0.20.0 => github.com/vmware/govmomi v0.20.0 </del> <ins> # github.com/vmware/govmomi v0.20.1 => github.com/vmware/govmomi v0.20.1 </ins> github.com/vmware/govmomi github.com/vmware/govmomi/find github.com/vmware/govmomi/list

I'll open a K8s PR to pull in the fix:
Updating k/vendor to govmomi master adds almost 6K lines for a 3 line fix. I may just create a govmomi branch off of v0.20 to reduce churn in the K8s release branches for this fix, may have to wait until tomorrow.
Really appreciate the insight into this issue. Can confirm that specifying the port (443) in the GOVC_URL reproduces the issue. Thank you!
Unfortunately, I'm seeing the same issue with v1.14.3: cloud_config Works fine when I use the username and password of a vsphere admin, but using cert and key still has authentication issues.
any related errors in vCenter's ?
Seeing a lot of this exception:
whoa! that stack trace is like punch in the gut (brings back memories!) :)
indeed!! apologies, I must have had the change for PR locally, but failed to include it as I had focused on the signature issue and updating vendor/