Question:
/kind bug /priority critical-urgent What happened: In I'm removing all the reapers, unfortunately when invoking foreground deletion on a DaemonSet the delete operation hangs and controller indefinitely tries to manage a removed DS. What you expected to happen: A foreground deletion properly removes DaemonSet and all its dependents. How to reproduce it (as minimally and precisely as possible): Invoke a delete operation against kube API (without using command). Anything else we need to know?: This behavior makes it so that none of the consumers of the API is not able to remove the DS controller due to present in the DS reaper. A working fix that passed the tests with the aforementioned PR is in but I'm not 100% it's correct. This is an issue that needs fixing before 1.11 goes out. /assign /sig apps /cc
fyi
I filed to fix it. As mentioned in the PR, this is because the DaemonSet controller races with garbage collector when it tries to update DaemonSet status of the DaemonSet being deleted. Here's what happens when hangs: /something performs a foreground deletion on a DaemonSet finalizer and DeletionTimestamp are both set history objects (ControllerRevisions) and pods are being deleted by garbage collector; meanwhile, DaemonSet controller tries to update DaemonSet status. Updating DaemonSet status requires constructing DaemonSet history objects, to figure out current revision and which pods do/don't belong to current revision updating DaemonSet status, DaemonSet controller tries to create a DaemonSet history object that matches current DaemonSet spec collector then tries to delete that DaemonSet history object. And repeat. Because we can't make DaemonSet pods be deleted before DaemonSet history objects (DaemonSet history objects don't own DaemonSet pods!), we cannot reliably calculate DaemonSet status without history objects anyways. Therefore, we don't update DaemonSet status for DaemonSet being deleted. Note that the reason why the kubectl delete hack works is because it forces DaemonSet pods to be removed before history objects.
[MILESTONENOTIFIER] Milestone Issue: Up-to-date for process Note: This issue is marked as , and must be updated every 1 day during code freeze. Example update: <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details

} func (o *DeleteOptions) deleteResource(info *resource.Info, deleteOptions *metav1.DeleteOptions) error { <del> // TODO: this should be removed as soon as DaemonSet controller properly handles object deletion // see https://github.com/kubernetes/kubernetes/issues/64313 for details mapping := info.ResourceMapping() if mapping.Resource.GroupResource() == (schema.GroupResource{Group: "extensions", Resource: "daemonsets"}) || mapping.Resource.GroupResource() == (schema.GroupResource{Group: "apps", Resource: "daemonsets"}) { if err := updateDaemonSet(info.Namespace, info.Name, o.DynamicClient); err != nil { return err } } </del> if err := resource.NewHelper(info.Client, info.Mapping).DeleteWithOptions(info.Namespace, info.Name, deleteOptions); err != nil { return cmdutil.AddSourceToErr("deleting", info.Source, err) }


Question:
/kind bug /priority critical-urgent What happened: In I'm removing all the reapers, unfortunately when invoking foreground deletion on a DaemonSet the delete operation hangs and controller indefinitely tries to manage a removed DS. What you expected to happen: A foreground deletion properly removes DaemonSet and all its dependents. How to reproduce it (as minimally and precisely as possible): Invoke a delete operation against kube API (without using command). Anything else we need to know?: This behavior makes it so that none of the consumers of the API is not able to remove the DS controller due to present in the DS reaper. A working fix that passed the tests with the aforementioned PR is in but I'm not 100% it's correct. This is an issue that needs fixing before 1.11 goes out. /assign /sig apps /cc
fyi
I filed to fix it. As mentioned in the PR, this is because the DaemonSet controller races with garbage collector when it tries to update DaemonSet status of the DaemonSet being deleted. Here's what happens when hangs: /something performs a foreground deletion on a DaemonSet finalizer and DeletionTimestamp are both set history objects (ControllerRevisions) and pods are being deleted by garbage collector; meanwhile, DaemonSet controller tries to update DaemonSet status. Updating DaemonSet status requires constructing DaemonSet history objects, to figure out current revision and which pods do/don't belong to current revision updating DaemonSet status, DaemonSet controller tries to create a DaemonSet history object that matches current DaemonSet spec collector then tries to delete that DaemonSet history object. And repeat. Because we can't make DaemonSet pods be deleted before DaemonSet history objects (DaemonSet history objects don't own DaemonSet pods!), we cannot reliably calculate DaemonSet status without history objects anyways. Therefore, we don't update DaemonSet status for DaemonSet being deleted. Note that the reason why the kubectl delete hack works is because it forces DaemonSet pods to be removed before history objects.
[MILESTONENOTIFIER] Milestone Issue: Up-to-date for process Note: This issue is marked as , and must be updated every 1 day during code freeze. Example update: <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details

return nil } <del> func updateDaemonSet(namespace, name string, dynamicClient dynamic.Interface) error { dsClient := dynamicClient.Resource(schema.GroupVersionResource{Group: "apps", Version: "v1", Resource: "daemonsets"}).Namespace(namespace) obj, err := dsClient.Get(name, metav1.GetOptions{}) if err != nil { return err } ds := &appsv1.DaemonSet{} if err := runtime.DefaultUnstructuredConverter.FromUnstructured(obj.Object, ds); err != nil { return err } // We set the nodeSelector to a random label. This label is nearly guaranteed // to not be set on any node so the DameonSetController will start deleting // daemon pods. Once it's done deleting the daemon pods, it's safe to delete // the DaemonSet. ds.Spec.Template.Spec.NodeSelector = map[string]string{ string(uuid.NewUUID()): string(uuid.NewUUID()), } // force update to avoid version conflict ds.ResourceVersion = "" out, err := runtime.DefaultUnstructuredConverter.ToUnstructured(ds) if err != nil { return err } if _, err = dsClient.Update(&unstructured.Unstructured{Object: out}); err != nil { return err } // Wait for the daemon set controller to kill all the daemon pods. if err := wait.Poll(1*time.Second, 5*time.Minute, func() (bool, error) { updatedObj, err := dsClient.Get(name, metav1.GetOptions{}) if err != nil { return false, nil } updatedDS := &appsv1.DaemonSet{} if err := runtime.DefaultUnstructuredConverter.FromUnstructured(updatedObj.Object, ds); err != nil { return false, nil } return updatedDS.Status.CurrentNumberScheduled+updatedDS.Status.NumberMisscheduled == 0, nil }); err != nil { return err } return nil } </del> // deletion printing is special because we do not have an object to print. // This mirrors name printer behavior func (o *DeleteOptions) PrintObj(info *resource.Info) {


Question:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug /sig CLI What happened: Running leads to nil pointer dereference: What you expected to happen: is expected to run delete with wait in that case How to reproduce it (as minimally and precisely as possible): (with any valid resource yaml) Anything else we need to know?: Discovered in when accidentally enabled wait for deletion from other commands than and saw failing integration tests in https://k8s- Environment: Kubernetes version (use ): latest master Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
This bug was introduced in : is set in that requires to be non-nil flag is being set in is normally being set in method which is not invoked in . a result, the invocation of resource finder leads to nil pointer dereference (see stack trace in the description).
/cc

deleteFlags.AddFlags(cmd) <del> cmd.Flags().Bool("wait", true, `If true, wait for resources to be gone before returning.  This waits for finalizers.`) </del> cmdutil.AddIncludeUninitializedFlag(cmd) return cmd }


Question:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug /sig CLI What happened: Running leads to nil pointer dereference: What you expected to happen: is expected to run delete with wait in that case How to reproduce it (as minimally and precisely as possible): (with any valid resource yaml) Anything else we need to know?: Discovered in when accidentally enabled wait for deletion from other commands than and saw failing integration tests in https://k8s- Environment: Kubernetes version (use ): latest master Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
This bug was introduced in : is set in that requires to be non-nil flag is being set in is normally being set in method which is not invoked in . a result, the invocation of resource finder leads to nil pointer dereference (see stack trace in the description).
/cc

} if o.GracePeriod == 0 && !o.ForceDeletion { // To preserve backwards compatibility, but prevent accidental data loss, we convert --grace-period=0 <del> // into --grace-period=1 and wait until the object is successfully deleted. Users may provide --force // to bypass this wait. o.WaitForDeletion = true </del> <ins> // into --grace-period=1. Users may provide --force to bypass this conversion. </ins> o.GracePeriod = 1 } <del> if b, err := cmd.Flags().GetBool("wait"); err == nil { o.WaitForDeletion = b } </del> includeUninitialized := cmdutil.ShouldIncludeUninitialized(cmd, false) r := f.NewBuilder().


Question:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug /sig CLI What happened: Running leads to nil pointer dereference: What you expected to happen: is expected to run delete with wait in that case How to reproduce it (as minimally and precisely as possible): (with any valid resource yaml) Anything else we need to know?: Discovered in when accidentally enabled wait for deletion from other commands than and saw failing integration tests in https://k8s- Environment: Kubernetes version (use ): latest master Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
This bug was introduced in : is set in that requires to be non-nil flag is being set in is normally being set in method which is not invoked in . a result, the invocation of resource finder leads to nil pointer dereference (see stack trace in the description).
/cc

return fmt.Errorf("cannot set --all and --field-selector at the same time") } <ins> if o.GracePeriod == 0 && !o.ForceDeletion && !o.WaitForDeletion { // With the explicit --wait flag we need extra validation for backward compatibility return fmt.Errorf("--grace-period=0 must have either --force specified, or --wait to be set to true") } </ins> switch { case o.GracePeriod == 0 && o.ForceDeletion: fmt.Fprintf(o.ErrOut, "warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely.n")


Question:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug /sig CLI What happened: Running leads to nil pointer dereference: What you expected to happen: is expected to run delete with wait in that case How to reproduce it (as minimally and precisely as possible): (with any valid resource yaml) Anything else we need to know?: Discovered in when accidentally enabled wait for deletion from other commands than and saw failing integration tests in https://k8s- Environment: Kubernetes version (use ): latest master Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
This bug was introduced in : is set in that requires to be non-nil flag is being set in is normally being set in method which is not invoked in . a result, the invocation of resource finder leads to nil pointer dereference (see stack trace in the description).
/cc

IgnoreNotFound *bool Now            *bool Timeout        *time.Duration <ins> Wait           *bool </ins> Output         *string }


Question:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug /sig CLI What happened: Running leads to nil pointer dereference: What you expected to happen: is expected to run delete with wait in that case How to reproduce it (as minimally and precisely as possible): (with any valid resource yaml) Anything else we need to know?: Discovered in when accidentally enabled wait for deletion from other commands than and saw failing integration tests in https://k8s- Environment: Kubernetes version (use ): latest master Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
This bug was introduced in : is set in that requires to be non-nil flag is being set in is normally being set in method which is not invoked in . a result, the invocation of resource finder leads to nil pointer dereference (see stack trace in the description).
/cc

if f.Timeout != nil { options.Timeout = *f.Timeout } <ins> if f.Wait != nil { options.WaitForDeletion = *f.Wait } </ins> return options }


Question:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug /sig CLI What happened: Running leads to nil pointer dereference: What you expected to happen: is expected to run delete with wait in that case How to reproduce it (as minimally and precisely as possible): (with any valid resource yaml) Anything else we need to know?: Discovered in when accidentally enabled wait for deletion from other commands than and saw failing integration tests in https://k8s- Environment: Kubernetes version (use ): latest master Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
This bug was introduced in : is set in that requires to be non-nil flag is being set in is normally being set in method which is not invoked in . a result, the invocation of resource finder leads to nil pointer dereference (see stack trace in the description).
/cc

if f.IgnoreNotFound != nil { cmd.Flags().BoolVar(f.IgnoreNotFound, "ignore-not-found", *f.IgnoreNotFound, "Treat "resource not found" as a successful delete. Defaults to "true" when --all is specified.") } <ins> if f.Wait != nil { cmd.Flags().BoolVar(f.Wait, "wait", *f.Wait, "If true, wait for resources to be gone before returning. This waits for finalizers.") } </ins> if f.Output != nil { cmd.Flags().StringVarP(f.Output, "output", "o", *f.Output, "Output mode. Use "-o name" for shorter output (resource/name).") } } // NewDeleteCommandFlags provides default flags and values for use with the "delete" command


Question:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug /sig CLI What happened: Running leads to nil pointer dereference: What you expected to happen: is expected to run delete with wait in that case How to reproduce it (as minimally and precisely as possible): (with any valid resource yaml) Anything else we need to know?: Discovered in when accidentally enabled wait for deletion from other commands than and saw failing integration tests in https://k8s- Environment: Kubernetes version (use ): latest master Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
This bug was introduced in : is set in that requires to be non-nil flag is being set in is normally being set in method which is not invoked in . a result, the invocation of resource finder leads to nil pointer dereference (see stack trace in the description).
/cc

labelSelector := "" fieldSelector := "" timeout := time.Duration(0) <ins> wait := true </ins> filenames := []string{} recursive := false


Question:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug /sig CLI What happened: Running leads to nil pointer dereference: What you expected to happen: is expected to run delete with wait in that case How to reproduce it (as minimally and precisely as possible): (with any valid resource yaml) Anything else we need to know?: Discovered in when accidentally enabled wait for deletion from other commands than and saw failing integration tests in https://k8s- Environment: Kubernetes version (use ): latest master Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
This bug was introduced in : is set in that requires to be non-nil flag is being set in is normally being set in method which is not invoked in . a result, the invocation of resource finder leads to nil pointer dereference (see stack trace in the description).
/cc

IgnoreNotFound: &ignoreNotFound, Now:            &now, Timeout:        &timeout, <ins> Wait:           &wait, </ins> Output:         &output, } }


Question:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug /sig CLI What happened: Running leads to nil pointer dereference: What you expected to happen: is expected to run delete with wait in that case How to reproduce it (as minimally and precisely as possible): (with any valid resource yaml) Anything else we need to know?: Discovered in when accidentally enabled wait for deletion from other commands than and saw failing integration tests in https://k8s- Environment: Kubernetes version (use ): latest master Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
This bug was introduced in : is set in that requires to be non-nil flag is being set in is normally being set in method which is not invoked in . a result, the invocation of resource finder leads to nil pointer dereference (see stack trace in the description).
/cc

force := false timeout := time.Duration(0) <ins> wait := false </ins> filenames := []string{} recursive := false


Question:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug /sig CLI What happened: Running leads to nil pointer dereference: What you expected to happen: is expected to run delete with wait in that case How to reproduce it (as minimally and precisely as possible): (with any valid resource yaml) Anything else we need to know?: Discovered in when accidentally enabled wait for deletion from other commands than and saw failing integration tests in https://k8s- Environment: Kubernetes version (use ): latest master Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
This bug was introduced in : is set in that requires to be non-nil flag is being set in is normally being set in method which is not invoked in . a result, the invocation of resource finder leads to nil pointer dereference (see stack trace in the description).
/cc

// add non-defaults Force:   &force, Timeout: &timeout, <ins> Wait:    &wait, </ins> } }


Question:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug /sig CLI What happened: Running leads to nil pointer dereference: What you expected to happen: is expected to run delete with wait in that case How to reproduce it (as minimally and precisely as possible): (with any valid resource yaml) Anything else we need to know?: Discovered in when accidentally enabled wait for deletion from other commands than and saw failing integration tests in https://k8s- Environment: Kubernetes version (use ): latest master Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
This bug was introduced in : is set in that requires to be non-nil flag is being set in is normally being set in method which is not invoked in . a result, the invocation of resource finder leads to nil pointer dereference (see stack trace in the description).
/cc

effectiveTimeout = 168 * time.Hour } waitOptions := kubectlwait.WaitOptions{ <del> ResourceFinder: genericclioptions.ResourceFinderForResult(o.Result), </del> <ins> ResourceFinder: genericclioptions.ResourceFinderForResult(r), </ins> DynamicClient:  o.DynamicClient, Timeout:        effectiveTimeout,


Question:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: After joining a new ppc64le node to the k8s cluster, the kube-proxy DaemonSet fails on the new node. What you expected to happen: It was expected that the kube-proxy DS would "just work" as it does when adding an amd64 node to the cluster. How to reproduce it (as minimally and precisely as possible): Set up an amd64 based k8s cluster, then join a ppc64le node (or any non-amd64 arch) to the cluster. See the pod fail via 'kubectl -n kube-system get pods' because it's pulling the amd64 kube-proxy docker image instead of the ppc64le image. Anything else we need to know?: The yaml for the DaemonSet specifies the image: <versionIt would be preferable to have a manifest so that the image in the DS could be specified as <versionand could "just work" on any platform. See: Environment: Kubernetes version (use ): v1.10.3 Cloud provider or hardware configuration: self - cluster installed via kubeadm OS (e.g. from /etc/os-release): NAME="Ubuntu" VERSION="16.04.3 LTS (Xenial Xerus)" ID=ubuntu IDLIKE=debian PRETTYNAME="Ubuntu 16.04.3 LTS" VERSIONID="16.04" HOMEURL="" SUPPORTURL="" BUGREPORTURL="" VERSIONCODENAME=xenial UBUNTU_CODENAME=xenial Kernel (e.g. ): N/A Install tools: N/A Others: N/A
if you are running multi-architecture nodes, your daemonsets should specify a node selector to limit their arch-specific image to run on matching nodes
/close
I am well aware that a node selector can be used. That is what we are doing now. But it is a hack. The default should "just work". Adding a new non-amd64 node should not result in kube-proxy immediately failing on it because the default daemon set is amd64 arch specific. If kube-proxy is meant to "just work" clusterwide, a node selector should not be needed. Please use the manifest-tool so that the default "just works". Please see the weave-net plugin. They are using the manifest-tool and it "just works" on amd64 and ppc64le. Their daemon set does not require/use a nodeSelector.
The current kube-proxy daemon set implementation is not properly architected, IMHO. Please re-open this
Reiterating the mentions to trigger a notification: <detailsIn response to : The current kube-proxy daemon set implementation is not properly architected, IMHO. Please re-open this </details
ah, this was a comment about the kube-proxy image, not the daemonset functionality /reopen
the kube-proxy addon daemonset is probably the multi-arch kube-proxy image to support that daemonset might be
Looks like it's coming from here: /assign /sig cluster-lifecycle Lucas, Do we want to avoid the references?
A dirty workaround for this is to pull and rename to . The kube-proxy is running as daemonset, created during . That means the kube-proxy image arch keeps consistent with your master node. Currently we don't have a multi-arch kube-proxy image yet. We have to explicitly specify the right name, like .
Hi .. Yes, we just created a separate daemonset with a ppc64le node slector that refers to the ppc64le image directly. But it's a bit of a hack. That is why I opened this ticket looking for a multi-arch image. Thanks!
I don't know that mixed-ISA clusters are supported. cc
The requirement is: only use "" in DaemonSet yaml file, and the k8s will download the correct image according to the ARCH info. It's more about image registry's feature; and it seems some registries had supported that: .
Yeah, has already supported multi-arch. But actually all the images now are architecture specific, like and . We've not built any multi-arch images right now.
for the test images we have the ball rolling here - - so if you can do the same (or similar) for other images, that would be great!
I have pending PR in releases project for creating manifest for those -
The manifest images created in kubernetes/release should work for future released versions, but will they be created during CI testing?
if you see one of the jobs that pushes images say you can grep for that's basically from the same files that got changed in so i fully expect the images pushed to CI will reflect the changes made in 516.
please confirm that i am reading this right ^ :)
I was more concerned with ci jobs that do not use cross builds, such as https://k8s-
please open an issue about pushing manifests in CI as well

operator: Exists - key: {{ .MasterTaintKey }} effect: NoSchedule <del> # TODO: Remove this affinity field as soon as we are using manifest lists affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: beta.kubernetes.io/arch operator: In values: - {{ .Arch }} </del> <ins> nodeSelector: beta.kubernetes.io/arch: {{ .Arch }} </ins> ` // KubeDNSService is the kube-dns Service manifest


Question:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: After joining a new ppc64le node to the k8s cluster, the kube-proxy DaemonSet fails on the new node. What you expected to happen: It was expected that the kube-proxy DS would "just work" as it does when adding an amd64 node to the cluster. How to reproduce it (as minimally and precisely as possible): Set up an amd64 based k8s cluster, then join a ppc64le node (or any non-amd64 arch) to the cluster. See the pod fail via 'kubectl -n kube-system get pods' because it's pulling the amd64 kube-proxy docker image instead of the ppc64le image. Anything else we need to know?: The yaml for the DaemonSet specifies the image: <versionIt would be preferable to have a manifest so that the image in the DS could be specified as <versionand could "just work" on any platform. See: Environment: Kubernetes version (use ): v1.10.3 Cloud provider or hardware configuration: self - cluster installed via kubeadm OS (e.g. from /etc/os-release): NAME="Ubuntu" VERSION="16.04.3 LTS (Xenial Xerus)" ID=ubuntu IDLIKE=debian PRETTYNAME="Ubuntu 16.04.3 LTS" VERSIONID="16.04" HOMEURL="" SUPPORTURL="" BUGREPORTURL="" VERSIONCODENAME=xenial UBUNTU_CODENAME=xenial Kernel (e.g. ): N/A Install tools: N/A Others: N/A
if you are running multi-architecture nodes, your daemonsets should specify a node selector to limit their arch-specific image to run on matching nodes
/close
I am well aware that a node selector can be used. That is what we are doing now. But it is a hack. The default should "just work". Adding a new non-amd64 node should not result in kube-proxy immediately failing on it because the default daemon set is amd64 arch specific. If kube-proxy is meant to "just work" clusterwide, a node selector should not be needed. Please use the manifest-tool so that the default "just works". Please see the weave-net plugin. They are using the manifest-tool and it "just works" on amd64 and ppc64le. Their daemon set does not require/use a nodeSelector.
The current kube-proxy daemon set implementation is not properly architected, IMHO. Please re-open this
Reiterating the mentions to trigger a notification: <detailsIn response to : The current kube-proxy daemon set implementation is not properly architected, IMHO. Please re-open this </details
ah, this was a comment about the kube-proxy image, not the daemonset functionality /reopen
the kube-proxy addon daemonset is probably the multi-arch kube-proxy image to support that daemonset might be
Looks like it's coming from here: /assign /sig cluster-lifecycle Lucas, Do we want to avoid the references?
A dirty workaround for this is to pull and rename to . The kube-proxy is running as daemonset, created during . That means the kube-proxy image arch keeps consistent with your master node. Currently we don't have a multi-arch kube-proxy image yet. We have to explicitly specify the right name, like .
Hi .. Yes, we just created a separate daemonset with a ppc64le node slector that refers to the ppc64le image directly. But it's a bit of a hack. That is why I opened this ticket looking for a multi-arch image. Thanks!
I don't know that mixed-ISA clusters are supported. cc
The requirement is: only use "" in DaemonSet yaml file, and the k8s will download the correct image according to the ARCH info. It's more about image registry's feature; and it seems some registries had supported that: .
Yeah, has already supported multi-arch. But actually all the images now are architecture specific, like and . We've not built any multi-arch images right now.
for the test images we have the ball rolling here - - so if you can do the same (or similar) for other images, that would be great!
I have pending PR in releases project for creating manifest for those -
The manifest images created in kubernetes/release should work for future released versions, but will they be created during CI testing?
if you see one of the jobs that pushes images say you can grep for that's basically from the same files that got changed in so i fully expect the images pushed to CI will reflect the changes made in 516.
please confirm that i am reading this right ^ :)
I was more concerned with ci jobs that do not use cross builds, such as https://k8s-
please open an issue about pushing manifests in CI as well

importpath = "k8s.io/kubernetes/cmd/kubeadm/app/phases/addons/proxy", deps = [ "//cmd/kubeadm/app/apis/kubeadm:go_default_library", <ins> "//cmd/kubeadm/app/constants:go_default_library", </ins> "//cmd/kubeadm/app/util:go_default_library", "//cmd/kubeadm/app/util/apiclient:go_default_library", "//pkg/proxy/apis/kubeproxyconfig/scheme:go_default_library",


Question:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: After joining a new ppc64le node to the k8s cluster, the kube-proxy DaemonSet fails on the new node. What you expected to happen: It was expected that the kube-proxy DS would "just work" as it does when adding an amd64 node to the cluster. How to reproduce it (as minimally and precisely as possible): Set up an amd64 based k8s cluster, then join a ppc64le node (or any non-amd64 arch) to the cluster. See the pod fail via 'kubectl -n kube-system get pods' because it's pulling the amd64 kube-proxy docker image instead of the ppc64le image. Anything else we need to know?: The yaml for the DaemonSet specifies the image: <versionIt would be preferable to have a manifest so that the image in the DS could be specified as <versionand could "just work" on any platform. See: Environment: Kubernetes version (use ): v1.10.3 Cloud provider or hardware configuration: self - cluster installed via kubeadm OS (e.g. from /etc/os-release): NAME="Ubuntu" VERSION="16.04.3 LTS (Xenial Xerus)" ID=ubuntu IDLIKE=debian PRETTYNAME="Ubuntu 16.04.3 LTS" VERSIONID="16.04" HOMEURL="" SUPPORTURL="" BUGREPORTURL="" VERSIONCODENAME=xenial UBUNTU_CODENAME=xenial Kernel (e.g. ): N/A Install tools: N/A Others: N/A
if you are running multi-architecture nodes, your daemonsets should specify a node selector to limit their arch-specific image to run on matching nodes
/close
I am well aware that a node selector can be used. That is what we are doing now. But it is a hack. The default should "just work". Adding a new non-amd64 node should not result in kube-proxy immediately failing on it because the default daemon set is amd64 arch specific. If kube-proxy is meant to "just work" clusterwide, a node selector should not be needed. Please use the manifest-tool so that the default "just works". Please see the weave-net plugin. They are using the manifest-tool and it "just works" on amd64 and ppc64le. Their daemon set does not require/use a nodeSelector.
The current kube-proxy daemon set implementation is not properly architected, IMHO. Please re-open this
Reiterating the mentions to trigger a notification: <detailsIn response to : The current kube-proxy daemon set implementation is not properly architected, IMHO. Please re-open this </details
ah, this was a comment about the kube-proxy image, not the daemonset functionality /reopen
the kube-proxy addon daemonset is probably the multi-arch kube-proxy image to support that daemonset might be
Looks like it's coming from here: /assign /sig cluster-lifecycle Lucas, Do we want to avoid the references?
A dirty workaround for this is to pull and rename to . The kube-proxy is running as daemonset, created during . That means the kube-proxy image arch keeps consistent with your master node. Currently we don't have a multi-arch kube-proxy image yet. We have to explicitly specify the right name, like .
Hi .. Yes, we just created a separate daemonset with a ppc64le node slector that refers to the ppc64le image directly. But it's a bit of a hack. That is why I opened this ticket looking for a multi-arch image. Thanks!
I don't know that mixed-ISA clusters are supported. cc
The requirement is: only use "" in DaemonSet yaml file, and the k8s will download the correct image according to the ARCH info. It's more about image registry's feature; and it seems some registries had supported that: .
Yeah, has already supported multi-arch. But actually all the images now are architecture specific, like and . We've not built any multi-arch images right now.
for the test images we have the ball rolling here - - so if you can do the same (or similar) for other images, that would be great!
I have pending PR in releases project for creating manifest for those -
The manifest images created in kubernetes/release should work for future released versions, but will they be created during CI testing?
if you see one of the jobs that pushes images say you can grep for that's basically from the same files that got changed in so i fully expect the images pushed to CI will reflect the changes made in 516.
please confirm that i am reading this right ^ :)
I was more concerned with ci jobs that do not use cross builds, such as https://k8s-
please open an issue about pushing manifests in CI as well

readOnly: true hostNetwork: true serviceAccountName: kube-proxy <del> tolerations: - operator: Exists </del> volumes: - name: kube-proxy configMap:


Question:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: After joining a new ppc64le node to the k8s cluster, the kube-proxy DaemonSet fails on the new node. What you expected to happen: It was expected that the kube-proxy DS would "just work" as it does when adding an amd64 node to the cluster. How to reproduce it (as minimally and precisely as possible): Set up an amd64 based k8s cluster, then join a ppc64le node (or any non-amd64 arch) to the cluster. See the pod fail via 'kubectl -n kube-system get pods' because it's pulling the amd64 kube-proxy docker image instead of the ppc64le image. Anything else we need to know?: The yaml for the DaemonSet specifies the image: <versionIt would be preferable to have a manifest so that the image in the DS could be specified as <versionand could "just work" on any platform. See: Environment: Kubernetes version (use ): v1.10.3 Cloud provider or hardware configuration: self - cluster installed via kubeadm OS (e.g. from /etc/os-release): NAME="Ubuntu" VERSION="16.04.3 LTS (Xenial Xerus)" ID=ubuntu IDLIKE=debian PRETTYNAME="Ubuntu 16.04.3 LTS" VERSIONID="16.04" HOMEURL="" SUPPORTURL="" BUGREPORTURL="" VERSIONCODENAME=xenial UBUNTU_CODENAME=xenial Kernel (e.g. ): N/A Install tools: N/A Others: N/A
if you are running multi-architecture nodes, your daemonsets should specify a node selector to limit their arch-specific image to run on matching nodes
/close
I am well aware that a node selector can be used. That is what we are doing now. But it is a hack. The default should "just work". Adding a new non-amd64 node should not result in kube-proxy immediately failing on it because the default daemon set is amd64 arch specific. If kube-proxy is meant to "just work" clusterwide, a node selector should not be needed. Please use the manifest-tool so that the default "just works". Please see the weave-net plugin. They are using the manifest-tool and it "just works" on amd64 and ppc64le. Their daemon set does not require/use a nodeSelector.
The current kube-proxy daemon set implementation is not properly architected, IMHO. Please re-open this
Reiterating the mentions to trigger a notification: <detailsIn response to : The current kube-proxy daemon set implementation is not properly architected, IMHO. Please re-open this </details
ah, this was a comment about the kube-proxy image, not the daemonset functionality /reopen
the kube-proxy addon daemonset is probably the multi-arch kube-proxy image to support that daemonset might be
Looks like it's coming from here: /assign /sig cluster-lifecycle Lucas, Do we want to avoid the references?
A dirty workaround for this is to pull and rename to . The kube-proxy is running as daemonset, created during . That means the kube-proxy image arch keeps consistent with your master node. Currently we don't have a multi-arch kube-proxy image yet. We have to explicitly specify the right name, like .
Hi .. Yes, we just created a separate daemonset with a ppc64le node slector that refers to the ppc64le image directly. But it's a bit of a hack. That is why I opened this ticket looking for a multi-arch image. Thanks!
I don't know that mixed-ISA clusters are supported. cc
The requirement is: only use "" in DaemonSet yaml file, and the k8s will download the correct image according to the ARCH info. It's more about image registry's feature; and it seems some registries had supported that: .
Yeah, has already supported multi-arch. But actually all the images now are architecture specific, like and . We've not built any multi-arch images right now.
for the test images we have the ball rolling here - - so if you can do the same (or similar) for other images, that would be great!
I have pending PR in releases project for creating manifest for those -
The manifest images created in kubernetes/release should work for future released versions, but will they be created during CI testing?
if you see one of the jobs that pushes images say you can grep for that's basically from the same files that got changed in so i fully expect the images pushed to CI will reflect the changes made in 516.
please confirm that i am reading this right ^ :)
I was more concerned with ci jobs that do not use cross builds, such as https://k8s-
please open an issue about pushing manifests in CI as well

- name: lib-modules hostPath: path: /lib/modules <ins> tolerations: - key: CriticalAddonsOnly operator: Exists - key: {{ .MasterTaintKey }} effect: NoSchedule nodeSelector: beta.kubernetes.io/arch: {{ .Arch }} </ins> ` )


Question:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: After joining a new ppc64le node to the k8s cluster, the kube-proxy DaemonSet fails on the new node. What you expected to happen: It was expected that the kube-proxy DS would "just work" as it does when adding an amd64 node to the cluster. How to reproduce it (as minimally and precisely as possible): Set up an amd64 based k8s cluster, then join a ppc64le node (or any non-amd64 arch) to the cluster. See the pod fail via 'kubectl -n kube-system get pods' because it's pulling the amd64 kube-proxy docker image instead of the ppc64le image. Anything else we need to know?: The yaml for the DaemonSet specifies the image: <versionIt would be preferable to have a manifest so that the image in the DS could be specified as <versionand could "just work" on any platform. See: Environment: Kubernetes version (use ): v1.10.3 Cloud provider or hardware configuration: self - cluster installed via kubeadm OS (e.g. from /etc/os-release): NAME="Ubuntu" VERSION="16.04.3 LTS (Xenial Xerus)" ID=ubuntu IDLIKE=debian PRETTYNAME="Ubuntu 16.04.3 LTS" VERSIONID="16.04" HOMEURL="" SUPPORTURL="" BUGREPORTURL="" VERSIONCODENAME=xenial UBUNTU_CODENAME=xenial Kernel (e.g. ): N/A Install tools: N/A Others: N/A
if you are running multi-architecture nodes, your daemonsets should specify a node selector to limit their arch-specific image to run on matching nodes
/close
I am well aware that a node selector can be used. That is what we are doing now. But it is a hack. The default should "just work". Adding a new non-amd64 node should not result in kube-proxy immediately failing on it because the default daemon set is amd64 arch specific. If kube-proxy is meant to "just work" clusterwide, a node selector should not be needed. Please use the manifest-tool so that the default "just works". Please see the weave-net plugin. They are using the manifest-tool and it "just works" on amd64 and ppc64le. Their daemon set does not require/use a nodeSelector.
The current kube-proxy daemon set implementation is not properly architected, IMHO. Please re-open this
Reiterating the mentions to trigger a notification: <detailsIn response to : The current kube-proxy daemon set implementation is not properly architected, IMHO. Please re-open this </details
ah, this was a comment about the kube-proxy image, not the daemonset functionality /reopen
the kube-proxy addon daemonset is probably the multi-arch kube-proxy image to support that daemonset might be
Looks like it's coming from here: /assign /sig cluster-lifecycle Lucas, Do we want to avoid the references?
A dirty workaround for this is to pull and rename to . The kube-proxy is running as daemonset, created during . That means the kube-proxy image arch keeps consistent with your master node. Currently we don't have a multi-arch kube-proxy image yet. We have to explicitly specify the right name, like .
Hi .. Yes, we just created a separate daemonset with a ppc64le node slector that refers to the ppc64le image directly. But it's a bit of a hack. That is why I opened this ticket looking for a multi-arch image. Thanks!
I don't know that mixed-ISA clusters are supported. cc
The requirement is: only use "" in DaemonSet yaml file, and the k8s will download the correct image according to the ARCH info. It's more about image registry's feature; and it seems some registries had supported that: .
Yeah, has already supported multi-arch. But actually all the images now are architecture specific, like and . We've not built any multi-arch images right now.
for the test images we have the ball rolling here - - so if you can do the same (or similar) for other images, that would be great!
I have pending PR in releases project for creating manifest for those -
The manifest images created in kubernetes/release should work for future released versions, but will they be created during CI testing?
if you see one of the jobs that pushes images say you can grep for that's basically from the same files that got changed in so i fully expect the images pushed to CI will reflect the changes made in 516.
please confirm that i am reading this right ^ :)
I was more concerned with ci jobs that do not use cross builds, such as https://k8s-
please open an issue about pushing manifests in CI as well

clientset "k8s.io/client-go/kubernetes" clientsetscheme "k8s.io/client-go/kubernetes/scheme" kubeadmapi "k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm" <ins> kubeadmconstants "k8s.io/kubernetes/cmd/kubeadm/app/constants" </ins> kubeadmutil "k8s.io/kubernetes/cmd/kubeadm/app/util" "k8s.io/kubernetes/cmd/kubeadm/app/util/apiclient" kubeproxyconfigscheme "k8s.io/kubernetes/pkg/proxy/apis/kubeproxyconfig/scheme"


Question:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: After joining a new ppc64le node to the k8s cluster, the kube-proxy DaemonSet fails on the new node. What you expected to happen: It was expected that the kube-proxy DS would "just work" as it does when adding an amd64 node to the cluster. How to reproduce it (as minimally and precisely as possible): Set up an amd64 based k8s cluster, then join a ppc64le node (or any non-amd64 arch) to the cluster. See the pod fail via 'kubectl -n kube-system get pods' because it's pulling the amd64 kube-proxy docker image instead of the ppc64le image. Anything else we need to know?: The yaml for the DaemonSet specifies the image: <versionIt would be preferable to have a manifest so that the image in the DS could be specified as <versionand could "just work" on any platform. See: Environment: Kubernetes version (use ): v1.10.3 Cloud provider or hardware configuration: self - cluster installed via kubeadm OS (e.g. from /etc/os-release): NAME="Ubuntu" VERSION="16.04.3 LTS (Xenial Xerus)" ID=ubuntu IDLIKE=debian PRETTYNAME="Ubuntu 16.04.3 LTS" VERSIONID="16.04" HOMEURL="" SUPPORTURL="" BUGREPORTURL="" VERSIONCODENAME=xenial UBUNTU_CODENAME=xenial Kernel (e.g. ): N/A Install tools: N/A Others: N/A
if you are running multi-architecture nodes, your daemonsets should specify a node selector to limit their arch-specific image to run on matching nodes
/close
I am well aware that a node selector can be used. That is what we are doing now. But it is a hack. The default should "just work". Adding a new non-amd64 node should not result in kube-proxy immediately failing on it because the default daemon set is amd64 arch specific. If kube-proxy is meant to "just work" clusterwide, a node selector should not be needed. Please use the manifest-tool so that the default "just works". Please see the weave-net plugin. They are using the manifest-tool and it "just works" on amd64 and ppc64le. Their daemon set does not require/use a nodeSelector.
The current kube-proxy daemon set implementation is not properly architected, IMHO. Please re-open this
Reiterating the mentions to trigger a notification: <detailsIn response to : The current kube-proxy daemon set implementation is not properly architected, IMHO. Please re-open this </details
ah, this was a comment about the kube-proxy image, not the daemonset functionality /reopen
the kube-proxy addon daemonset is probably the multi-arch kube-proxy image to support that daemonset might be
Looks like it's coming from here: /assign /sig cluster-lifecycle Lucas, Do we want to avoid the references?
A dirty workaround for this is to pull and rename to . The kube-proxy is running as daemonset, created during . That means the kube-proxy image arch keeps consistent with your master node. Currently we don't have a multi-arch kube-proxy image yet. We have to explicitly specify the right name, like .
Hi .. Yes, we just created a separate daemonset with a ppc64le node slector that refers to the ppc64le image directly. But it's a bit of a hack. That is why I opened this ticket looking for a multi-arch image. Thanks!
I don't know that mixed-ISA clusters are supported. cc
The requirement is: only use "" in DaemonSet yaml file, and the k8s will download the correct image according to the ARCH info. It's more about image registry's feature; and it seems some registries had supported that: .
Yeah, has already supported multi-arch. But actually all the images now are architecture specific, like and . We've not built any multi-arch images right now.
for the test images we have the ball rolling here - - so if you can do the same (or similar) for other images, that would be great!
I have pending PR in releases project for creating manifest for those -
The manifest images created in kubernetes/release should work for future released versions, but will they be created during CI testing?
if you see one of the jobs that pushes images say you can grep for that's basically from the same files that got changed in so i fully expect the images pushed to CI will reflect the changes made in 516.
please confirm that i am reading this right ^ :)
I was more concerned with ci jobs that do not use cross builds, such as https://k8s-
please open an issue about pushing manifests in CI as well

if err != nil { return fmt.Errorf("error when parsing kube-proxy configmap template: %v", err) } <del> proxyDaemonSetBytes, err = kubeadmutil.ParseTemplate(KubeProxyDaemonSet19, struct{ ImageRepository, Arch, Version, ImageOverride string }{ </del> <ins> proxyDaemonSetBytes, err = kubeadmutil.ParseTemplate(KubeProxyDaemonSet19, struct{ ImageRepository, Arch, Version, ImageOverride, MasterTaintKey string }{ </ins> ImageRepository: cfg.GetControlPlaneImageRepository(), Arch:            runtime.GOARCH, Version:         kubeadmutil.KubernetesVersionToImageTag(cfg.KubernetesVersion), ImageOverride:   cfg.UnifiedControlPlaneImage, <ins> MasterTaintKey:  kubeadmconstants.LabelNodeRoleMaster, </ins> }) if err != nil { return fmt.Errorf("error when parsing kube-proxy daemonset template: %v", err)


Question:
BUG REPORT: Portworx volume plugin when a PVC is created using a StorageClass with empty parameters /kind bug What happened: Create a StorageClass using empty parameters and then create a PVC using that storage class. The PVC will remain in pending state. On inspecting the logs of the controller manager, we find that the portworx volume plugin is panicing. What you expected to happen: PVC should have been created successfully. How to reproduce it (as minimally and precisely as possible): Create following storage class and PVC Anything else we need to know?: Environment: N/A Kubernetes version (use ): 1.9.7, 1.10.1, 1.8.8 Cloud provider or hardware configuration: N/A OS (e.g. from /etc/os-release): N/A Kernel (e.g. ): N/A Install tools: N/A Others: N/A
/sig storage

spec = specHandler.DefaultSpec() } <del> // Pass all parameters as volume labels for Portworx server-side processing. spec.VolumeLabels = p.options.Parameters </del> <ins> // Pass all parameters as volume labels for Portworx server-side processing if len(p.options.Parameters) > 0 { spec.VolumeLabels = p.options.Parameters } else { spec.VolumeLabels = make(map[string]string, 0) } </ins> // Update the requested size in the spec spec.Size = uint64(requestGiB * volutil.GIB)


Question:
The mount code () is reasonably well tested but brings in the entire Kubernetes source tree when CSI plugins have to depend on it. With a few exceptions, most of the mount code could be refactored to be a minimal set of utilities that can be layered and make it easier to develop CSI plugins. I noticed that quite a few CSI plugins take a dependency on k/k just to get that one package. Suggestions: and split into a minimal dependency library and move it out of k/k (perhaps into staging and have it published out) stop depending on k/k and start depending on that library dependency on so the library is easily usable. In general, there is a lot of boiler plate for CSI, and pkg/util/mount is a significant portion of reuse.
If there no one was working on this, I can help do it. :-)
/assign
is this something you are still looking at? It's come up in sig-storage, and is on the planning spreadsheet for 1.12. It's something I'd like to work on as well, so want to see where you are at with it. While it didn't get any traction, I will point out that when me and some coworkers wrote several CSI drivers to the 0.1 version of the spec, we saw this problem, and factored this code out into a project that eventually landed at It looks quite a bit different now, because we found the k/k code didn't handle bind-mounts of block devices correctly, which was something new to CSI and K8s at the time.
did some work recently in pkg/util/mount to detect bind mounts better. What I would like to see is this handled in at least 3 steps: Mounter interface to separate out purely mounting helper functions and Kubernetes utilities the refactored mount library into an external repo, maybe under kubernetes-sigs mount library from Kubernetes and vendor the external repo
Awesome. I plan to start working on this now, and i agree with the refactor approach. I do wonder where the resulting library should end up? Would kubernetes/utils be the right place?, rather than kubernetes/kubernetes-sigs? k/utils seems odd to me, though, as one goal is to not have to vendor a bunch of code you don't care about. but, k/utils has the intent of holding many different projects in it. So if you want one thing out of it, you have to vendor all of k/utils. I like seeing things in separate repos. =) Location doesn't hold up the refactoring, however, so I'll work on that!
Can it just be a standalone package, like kubernetes/utils/mount?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
/remove-kind bug /kind cleanup
Hi Now that is merged, I wanted to confirm that the next step here would be to move to be a package in .
Yup!
let me know if you need a hand with git surgery
The git-foo wasn't too bad. Just slow to filter on the whole k/k repo. =) I've opened the PR at , feel free to take a look. Once that is in, we'll need to be cognizant to not letting any fixes come into k/k/pkg/util/mount, otherwise they might get lost. Once the package goes into , I can open the PR on k/k to make use of it!
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale

// HTTP Handler interface func (server *ApiServer) ServeHTTP(w http.ResponseWriter, req *http.Request) { <ins> defer func() { if x := recover(); x != nil { w.WriteHeader(http.StatusInternalServerError) fmt.Fprint(w, "apiserver panic. Look in log for details.") log.Printf("ApiServer panic'd: %#vn", x) } }() </ins> log.Printf("%s %s", req.Method, req.RequestURI) url, err := url.ParseRequestURI(req.RequestURI) if err != nil {


Question:
The mount code () is reasonably well tested but brings in the entire Kubernetes source tree when CSI plugins have to depend on it. With a few exceptions, most of the mount code could be refactored to be a minimal set of utilities that can be layered and make it easier to develop CSI plugins. I noticed that quite a few CSI plugins take a dependency on k/k just to get that one package. Suggestions: and split into a minimal dependency library and move it out of k/k (perhaps into staging and have it published out) stop depending on k/k and start depending on that library dependency on so the library is easily usable. In general, there is a lot of boiler plate for CSI, and pkg/util/mount is a significant portion of reuse.
If there no one was working on this, I can help do it. :-)
/assign
is this something you are still looking at? It's come up in sig-storage, and is on the planning spreadsheet for 1.12. It's something I'd like to work on as well, so want to see where you are at with it. While it didn't get any traction, I will point out that when me and some coworkers wrote several CSI drivers to the 0.1 version of the spec, we saw this problem, and factored this code out into a project that eventually landed at It looks quite a bit different now, because we found the k/k code didn't handle bind-mounts of block devices correctly, which was something new to CSI and K8s at the time.
did some work recently in pkg/util/mount to detect bind mounts better. What I would like to see is this handled in at least 3 steps: Mounter interface to separate out purely mounting helper functions and Kubernetes utilities the refactored mount library into an external repo, maybe under kubernetes-sigs mount library from Kubernetes and vendor the external repo
Awesome. I plan to start working on this now, and i agree with the refactor approach. I do wonder where the resulting library should end up? Would kubernetes/utils be the right place?, rather than kubernetes/kubernetes-sigs? k/utils seems odd to me, though, as one goal is to not have to vendor a bunch of code you don't care about. but, k/utils has the intent of holding many different projects in it. So if you want one thing out of it, you have to vendor all of k/utils. I like seeing things in separate repos. =) Location doesn't hold up the refactoring, however, so I'll work on that!
Can it just be a standalone package, like kubernetes/utils/mount?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
/remove-kind bug /kind cleanup
Hi Now that is merged, I wanted to confirm that the next step here would be to move to be a package in .
Yup!
let me know if you need a hand with git surgery
The git-foo wasn't too bad. Just slow to filter on the whole k/k repo. =) I've opened the PR at , feel free to take a look. Once that is in, we'll need to be cognizant to not letting any fixes come into k/k/pkg/util/mount, otherwise they might get lost. Once the package goes into , I can open the PR on k/k to make use of it!
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale

func (storage *ControllerRegistryStorage) Get(id string) (interface{}, error) { controller, err := storage.registry.GetController(id) <ins> if err != nil { return nil, err } </ins> controller.Kind = "cluster#replicationController" return controller, err }


Question:
The mount code () is reasonably well tested but brings in the entire Kubernetes source tree when CSI plugins have to depend on it. With a few exceptions, most of the mount code could be refactored to be a minimal set of utilities that can be layered and make it easier to develop CSI plugins. I noticed that quite a few CSI plugins take a dependency on k/k just to get that one package. Suggestions: and split into a minimal dependency library and move it out of k/k (perhaps into staging and have it published out) stop depending on k/k and start depending on that library dependency on so the library is easily usable. In general, there is a lot of boiler plate for CSI, and pkg/util/mount is a significant portion of reuse.
If there no one was working on this, I can help do it. :-)
/assign
is this something you are still looking at? It's come up in sig-storage, and is on the planning spreadsheet for 1.12. It's something I'd like to work on as well, so want to see where you are at with it. While it didn't get any traction, I will point out that when me and some coworkers wrote several CSI drivers to the 0.1 version of the spec, we saw this problem, and factored this code out into a project that eventually landed at It looks quite a bit different now, because we found the k/k code didn't handle bind-mounts of block devices correctly, which was something new to CSI and K8s at the time.
did some work recently in pkg/util/mount to detect bind mounts better. What I would like to see is this handled in at least 3 steps: Mounter interface to separate out purely mounting helper functions and Kubernetes utilities the refactored mount library into an external repo, maybe under kubernetes-sigs mount library from Kubernetes and vendor the external repo
Awesome. I plan to start working on this now, and i agree with the refactor approach. I do wonder where the resulting library should end up? Would kubernetes/utils be the right place?, rather than kubernetes/kubernetes-sigs? k/utils seems odd to me, though, as one goal is to not have to vendor a bunch of code you don't care about. but, k/utils has the intent of holding many different projects in it. So if you want one thing out of it, you have to vendor all of k/utils. I like seeing things in separate repos. =) Location doesn't hold up the refactoring, however, so I'll work on that!
Can it just be a standalone package, like kubernetes/utils/mount?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
/remove-kind bug /kind cleanup
Hi Now that is merged, I wanted to confirm that the next step here would be to move to be a package in .
Yup!
let me know if you need a hand with git surgery
The git-foo wasn't too bad. Just slow to filter on the whole k/k repo. =) I've opened the PR at , feel free to take a look. Once that is in, we'll need to be cognizant to not letting any fixes come into k/k/pkg/util/mount, otherwise they might get lost. Once the package goes into , I can open the PR on k/k to make use of it!
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale

func (sr *ServiceRegistryStorage) List(*url.URL) (interface{}, error) { list, err := sr.registry.ListServices() <ins> if err != nil { return nil, err } </ins> list.Kind = "cluster#serviceList" return list, err } func (sr *ServiceRegistryStorage) Get(id string) (interface{}, error) { service, err := sr.registry.GetService(id) <ins> if err != nil { return nil, err } </ins> service.Kind = "cluster#service" return service, err }


Question:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: Currently, Calico CNI spec is specified in calico-node- under cluster/addons/calico-policy-controller without MTU specified (i.e., default 1500 is used). Given this is shared across many environments, we need a way to customize its MTU, say 1460 on GCP. What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig network
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

fi } <ins> # A helper function to set up a custom yaml for a k8s addon. # # $1: addon category under /etc/kubernetes # $2: manifest source dir # $3: manifest file # $4: custom yaml function setup-addon-custom-yaml { local -r manifest_path="/etc/kubernetes/$1/$2/$3" local -r custom_yaml="$4" if [ -n "${custom_yaml:-}" ]; then # Replace with custom manifest. cat > "${manifest_path}" <<EOF $custom_yaml EOF fi } </ins> # Prepares the manifests of k8s addons, and starts the addon manager. # Vars assumed: #   CLUSTER_NAME


Question:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: Currently, Calico CNI spec is specified in calico-node- under cluster/addons/calico-policy-controller without MTU specified (i.e., default 1500 is used). Given this is shared across many environments, we need a way to customize its MTU, say 1460 on GCP. What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig network
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

if [[ "${NETWORK_POLICY_PROVIDER:-}" == "calico" ]]; then setup-addon-manifests "addons" "calico-policy-controller" <ins> setup-addon-custom-yaml "addons" "calico-policy-controller" "calico-node-daemonset.yaml" "${CUSTOM_CALICO_NODE_DAEMONSET_YAML:-}" setup-addon-custom-yaml "addons" "calico-policy-controller" "typha-deployment.yaml" "${CUSTOM_TYPHA_DEPLOYMENT_YAML:-}" </ins> # Configure Calico CNI directory. local -r ds_file="${dst_dir}/calico-policy-controller/calico-node-daemonset.yaml" sed -i -e "s@__CALICO_CNI_DIR__@/home/kubernetes/bin@g" "${ds_file}"


Question:
/kind bug /sig network (I'm not sure this is actually a bug but it is definitely a behavior that is different from iptables and some applications rely on this) What happened: From the host network namespace accessing a ClusterIP when using IPVS does not work (it only works when traffic is sent to a local pod) What you expected to happen: To be able to use ClusterIPs from the host network namespace How to reproduce it (as minimally and precisely as possible): Configure kube-proxy in IPVS mode Create a deployment and the associated service Try to access the ClusterIP from the host Anything else we need to know?: This does not work due to how routing is set up when using IPVS If we create a service with ClusterIP , kube-proxy will add this IP to the kube-ivps0 dummy interface: which will add the IP to the local route table: (where 10.0.0.5 is the main interface for the instance) So when accessing the ClusterIP from the host, the source IP will default to the ClusterIP: After IPVS, the destination will be changed to a pod IP but the source will still be ClusterIP. Since ClusterIPs are not routable the answer will be lost. A workaround would be to modify the local route to use the IP of the main interface as source: I tested this and it seems to work fine. Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Ubuntu 18.04 Kernel (e.g. ): 4.15.0-1008-gcp
/area ipvs
Don't you have MASQUERADE rule somewhere? My clusters use flannel to create MASQUERADE rule and I guess it could also be done in kube-proxy with --masquerade-all=true option.
I don't have any specific masquerade configuration: kubelet is run without so it uses the default (both my pod and service CIDRs are in this range) kube-proxy is run without So I have the following masquerading rules: (from kubelet) from kube-proxy for hairpin traffic My instances are also configured to NAT on KUBE-MARK-MASQ mark but in IPVS mode this chain is not referenced in the default configuration. I confirm that running kube-proxy with works but it feels like a workaround: if we do that we also masquerade pod-to-service traffic which seems unnecessary (and we lose the source pod IP)
Is it possible to have pod ip and service ip configured in same range? Guess I am just familiar with my own cluster setup which use cluster-cidr.... Hope others can help you.
Sorry, I meant both are in but they are different ( and )
cc I assume this bug has already been fixed?
I am going to close this issue since it should be fixed by , please re-open if you still have troubles after picking
/close
Thank you for looking into this I'll check with kube-proxy 1.11 very soon and will let you know
I just tested and it does not seem to work I updated kube-proxy to but I still have the problem Looking at the code, it looks like the KUBE-MARK-MASQ is not to the service chain because the ipset is empty (which should not be the case because I actually have some services deployed) Did I miss something?
Did you configure the --cluster-cidr?
cc any comments?
No I did not I just realized this was the source of the problem here: We usually don't configure it because we have no need for it: we don't advertise the service cidr externally we would rather have pods using loadbalancer IPs actually hit the load-balancers (easier to understand and debug if we have similar routing for hosts outside the cluster and pods for the same "host")
Yeah, seems you find the root cause. clusterIPSet should not be empty.

} // add service Cluster IP:Port to kubeServiceAccess ip set for the purpose of solving hairpin. // proxier.kubeServiceAccessSet.activeEntries.Insert(entry.String()) <del> // Install masquerade rules if 'masqueradeAll' or 'clusterCIDR' is specified. if proxier.masqueradeAll || len(proxier.clusterCIDR) > 0 { if valid := proxier.ipsetList[kubeClusterIPSet].validateEntry(entry); !valid { glog.Errorf("%s", fmt.Sprintf(EntryInvalidErr, entry, proxier.ipsetList[kubeClusterIPSet].Name)) continue } proxier.ipsetList[kubeClusterIPSet].activeEntries.Insert(entry.String()) </del> <ins> if valid := proxier.ipsetList[kubeClusterIPSet].validateEntry(entry); !valid { glog.Errorf("%s", fmt.Sprintf(EntryInvalidErr, entry, proxier.ipsetList[kubeClusterIPSet].Name)) continue </ins> } <ins> proxier.ipsetList[kubeClusterIPSet].activeEntries.Insert(entry.String()) </ins> // ipvs call serv := &utilipvs.VirtualServer{ Address:   svcInfo.ClusterIP,


Question:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: When I created a crd e.g.: Then I used . I found that the first time the result will be with an error: And as for the second time it will be normal: And it can be stably reproduced every time I created a new CRD. What you expected to happen: kubectl returns the first time I requested the new CRD instead of an error. Anything else we need to know?: I also found that requesting short name e.g. will be the same. It returns an error the first time but becomes normal the second time. Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: /kind bug
/area custom-resources
Confirmed that could show up even the crd has status /assign
/sig api-machinery Haven't looked into the problem yet but could be related:
Looks like this happens because cache file in .kube/http-cache/ or .kube/cache/ is not synced with kube-apiserver. delete all files under .kube/http-cache/ and .kube/cache/ will cause a sync and the error will not show up. This looks like a cli issue.
fyi
Tried to fix in cli:
/cc
/assign
this seems similar to
Even though this bug looks small, it might involve larger changes both in fixing and refactoring.. A typical corner case it is /shrug
Can you put this into words? What's the actual issue?
Even though this bug looks small, it might involve larger changes both in fixing and refactoring.. A typical corner case it is Can you put this into words? What's the actual issue? SRY, I should have put it clearer.. I just mean that this issue is sort of fundamental and obviously has been existing for a long time.
/remove-shrug
/unshrug looks like an important problem we want to solve (?)
¯()/¯ <detailsIn response to : /unshrug Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen /remove-lifecycle rotten ouch..
Reopened this issue. <detailsIn response to : /reopen /remove-lifecycle rotten Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc
encountered same issue, as suggested the error will not show up.
any progress?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Here is a little more info, and here is how I am able to reproduce it (kubectl version 1.18.0 and cluster version 1.19.0-beta.2): A couple interesting things i noted... If I deleted my and re-ran it, I could not reproduce it. It is only the first time the short name was EVER used that this occurs. I only tried from one client machine, so I'm not sure this is cluster-specific or client-specific. If I used a different short name, like 'fo2' and re-ran the example, then it reproduces the problem. You can also run after creating the CRD and it has the same effect as deleting your . EDIT TO ADD: This doesn't seem to be a timing thing... if I sleep for 10 seconds after creating the CRD, it still has the problem.
The root of the problem is that the is run before the discovery. So in this particular case this line continues to b/c read cached data and never reached out to server. Whereas discovery mapper in will get a not found error and will proceed invalidating cache and re-fetching discovery data and return that back. In the subsequent call the shortcut expander is already aware of the new shortcut so it works just fine. We'd need to teach shortcut expander to re-fetch discovery data, but I'm not sure that's desirable, or we could at least warn a user about this situation.
See for possible approach.
Any new progress on this ?
lists a candidate solution. This bug is causing steadily failing tests. I'll talk to a few people to see if this can get any traction. do you know who can get this moving? It sounds like you have identified the problem and have proposed a solution.

// KindFor fulfills meta.RESTMapper func (e shortcutExpander) KindFor(resource schema.GroupVersionResource) (schema.GroupVersionKind, error) { <del> return e.RESTMapper.KindFor(e.expandResourceShortcut(resource)) </del> <ins> // expandResourceShortcut works with current API resources as read from discovery cache. // In case of new CRDs this means we potentially don't have current state of discovery. // In the current wiring in k8s.io/cli-runtime/pkg/genericclioptions/config_flags.go#toRESTMapper, // we are using DeferredDiscoveryRESTMapper which on KindFor failure will clear the // cache and fetch all data from a cluster (see vendor/k8s.io/client-go/restmapper/discovery.go#KindFor). // Thus another call to expandResourceShortcut, after a NoMatchError should successfully // read Kind to the user or an error. gvk, err := e.RESTMapper.KindFor(e.expandResourceShortcut(resource)) if meta.IsNoMatchError(err) { return e.RESTMapper.KindFor(e.expandResourceShortcut(resource)) } return gvk, err </ins> } // KindsFor fulfills meta.RESTMapper


Question:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: When I created a crd e.g.: Then I used . I found that the first time the result will be with an error: And as for the second time it will be normal: And it can be stably reproduced every time I created a new CRD. What you expected to happen: kubectl returns the first time I requested the new CRD instead of an error. Anything else we need to know?: I also found that requesting short name e.g. will be the same. It returns an error the first time but becomes normal the second time. Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: /kind bug
/area custom-resources
Confirmed that could show up even the crd has status /assign
/sig api-machinery Haven't looked into the problem yet but could be related:
Looks like this happens because cache file in .kube/http-cache/ or .kube/cache/ is not synced with kube-apiserver. delete all files under .kube/http-cache/ and .kube/cache/ will cause a sync and the error will not show up. This looks like a cli issue.
fyi
Tried to fix in cli:
/cc
/assign
this seems similar to
Even though this bug looks small, it might involve larger changes both in fixing and refactoring.. A typical corner case it is /shrug
Can you put this into words? What's the actual issue?
Even though this bug looks small, it might involve larger changes both in fixing and refactoring.. A typical corner case it is Can you put this into words? What's the actual issue? SRY, I should have put it clearer.. I just mean that this issue is sort of fundamental and obviously has been existing for a long time.
/remove-shrug
/unshrug looks like an important problem we want to solve (?)
¯()/¯ <detailsIn response to : /unshrug Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen /remove-lifecycle rotten ouch..
Reopened this issue. <detailsIn response to : /reopen /remove-lifecycle rotten Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc
encountered same issue, as suggested the error will not show up.
any progress?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Here is a little more info, and here is how I am able to reproduce it (kubectl version 1.18.0 and cluster version 1.19.0-beta.2): A couple interesting things i noted... If I deleted my and re-ran it, I could not reproduce it. It is only the first time the short name was EVER used that this occurs. I only tried from one client machine, so I'm not sure this is cluster-specific or client-specific. If I used a different short name, like 'fo2' and re-ran the example, then it reproduces the problem. You can also run after creating the CRD and it has the same effect as deleting your . EDIT TO ADD: This doesn't seem to be a timing thing... if I sleep for 10 seconds after creating the CRD, it still has the problem.
The root of the problem is that the is run before the discovery. So in this particular case this line continues to b/c read cached data and never reached out to server. Whereas discovery mapper in will get a not found error and will proceed invalidating cache and re-fetching discovery data and return that back. In the subsequent call the shortcut expander is already aware of the new shortcut so it works just fine. We'd need to teach shortcut expander to re-fetch discovery data, but I'm not sure that's desirable, or we could at least warn a user about this situation.
See for possible approach.
Any new progress on this ?
lists a candidate solution. This bug is causing steadily failing tests. I'll talk to a few people to see if this can get any traction. do you know who can get this moving? It sounds like you have identified the problem and have proposed a solution.

"testing" openapi_v2 "github.com/google/gnostic/openapiv2" <ins> "github.com/google/go-cmp/cmp" </ins> "k8s.io/apimachinery/pkg/api/errors" "k8s.io/apimachinery/pkg/api/meta"


Question:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: When I created a crd e.g.: Then I used . I found that the first time the result will be with an error: And as for the second time it will be normal: And it can be stably reproduced every time I created a new CRD. What you expected to happen: kubectl returns the first time I requested the new CRD instead of an error. Anything else we need to know?: I also found that requesting short name e.g. will be the same. It returns an error the first time but becomes normal the second time. Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: /kind bug
/area custom-resources
Confirmed that could show up even the crd has status /assign
/sig api-machinery Haven't looked into the problem yet but could be related:
Looks like this happens because cache file in .kube/http-cache/ or .kube/cache/ is not synced with kube-apiserver. delete all files under .kube/http-cache/ and .kube/cache/ will cause a sync and the error will not show up. This looks like a cli issue.
fyi
Tried to fix in cli:
/cc
/assign
this seems similar to
Even though this bug looks small, it might involve larger changes both in fixing and refactoring.. A typical corner case it is /shrug
Can you put this into words? What's the actual issue?
Even though this bug looks small, it might involve larger changes both in fixing and refactoring.. A typical corner case it is Can you put this into words? What's the actual issue? SRY, I should have put it clearer.. I just mean that this issue is sort of fundamental and obviously has been existing for a long time.
/remove-shrug
/unshrug looks like an important problem we want to solve (?)
¯()/¯ <detailsIn response to : /unshrug Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen /remove-lifecycle rotten ouch..
Reopened this issue. <detailsIn response to : /reopen /remove-lifecycle rotten Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc
encountered same issue, as suggested the error will not show up.
any progress?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Here is a little more info, and here is how I am able to reproduce it (kubectl version 1.18.0 and cluster version 1.19.0-beta.2): A couple interesting things i noted... If I deleted my and re-ran it, I could not reproduce it. It is only the first time the short name was EVER used that this occurs. I only tried from one client machine, so I'm not sure this is cluster-specific or client-specific. If I used a different short name, like 'fo2' and re-ran the example, then it reproduces the problem. You can also run after creating the CRD and it has the same effect as deleting your . EDIT TO ADD: This doesn't seem to be a timing thing... if I sleep for 10 seconds after creating the CRD, it still has the problem.
The root of the problem is that the is run before the discovery. So in this particular case this line continues to b/c read cached data and never reached out to server. Whereas discovery mapper in will get a not found error and will proceed invalidating cache and re-fetching discovery data and return that back. In the subsequent call the shortcut expander is already aware of the new shortcut so it works just fine. We'd need to teach shortcut expander to re-fetch discovery data, but I'm not sure that's desirable, or we could at least warn a user about this situation.
See for possible approach.
Any new progress on this ?
lists a candidate solution. This bug is causing steadily failing tests. I'll talk to a few people to see if this can get any traction. do you know who can get this moving? It sounds like you have identified the problem and have proposed a solution.

} } <ins> func TestKindForWithNewCRDs(t *testing.T) { tests := map[string]struct { in       schema.GroupVersionResource expected schema.GroupVersionKind srvRes   []*metav1.APIResourceList }{ "": { in:       schema.GroupVersionResource{Group: "a", Version: "", Resource: "sc"}, expected: schema.GroupVersionKind{Group: "a", Version: "v1", Kind: "StorageClass"}, srvRes: []*metav1.APIResourceList{ { GroupVersion: "a/v1", APIResources: []metav1.APIResource{ { Name:       "storageclasses", ShortNames: []string{"sc"}, Kind:       "StorageClass", }, }, }, }, }, } for name, test := range tests { t.Run(name, func(t *testing.T) { invalidateCalled := false fakeDiscovery := &fakeDiscoveryClient{} fakeDiscovery.serverResourcesHandler = func() ([]*metav1.APIResourceList, error) { if invalidateCalled { return test.srvRes, nil } return []*metav1.APIResourceList{}, nil } fakeCachedDiscovery := &fakeCachedDiscoveryClient{DiscoveryInterface: fakeDiscovery} fakeCachedDiscovery.invalidateHandler = func() { invalidateCalled = true } fakeCachedDiscovery.freshHandler = func() bool { return invalidateCalled } // in real world the discovery client is fronted with a cache which // will answer the initial request, only failure to match will trigger // the cache invalidation and live discovery call delegate := NewDeferredDiscoveryRESTMapper(fakeCachedDiscovery) mapper := NewShortcutExpander(delegate, fakeCachedDiscovery) gvk, err := mapper.KindFor(test.in) if err != nil { t.Errorf("unexpected error: %v", err) } if diff := cmp.Equal(gvk, test.expected); !diff { t.Errorf("unexpected data returned %#v, expected %#v", gvk, test.expected) } }) } } </ins> type fakeRESTMapper struct { kindForInput schema.GroupVersionResource }