Question: expectNoError(ScaleRC(config.Client, config.Namespace, config.Name, newSize, true), fmt.Sprintf("scaling rc %s for the first time", config.Name)) selector := labels.SelectorFromSet(labels.Set(map[string]string{"name": config.Name})) <del> options := unversioned.ListOptions{LabelSelector: unversioned.LabelSelector{selector}} </del> <ins> options := unversioned.ListOptions{ LabelSelector:   unversioned.LabelSelector{selector}, ResourceVersion: "0", } </ins> _, err := config.Client.Pods(config.Namespace).List(options) expectNoError(err, fmt.Sprintf("listing pods from rc %v", config.Name)) }

Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: } // Check if host string is a valid IP address if ip := net.ParseIP(host); ip != nil { <ins> return ip.String() } else { </ins> glog.Errorf("invalid IP part '%s'", host) <del> return host </del> } return "" }

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: kube-proxy throws the following error message whena valid IP is used: The same error can be observed when running the UT: What you expected to happen: Errors should only be logged for invalid IP addresses. How to reproduce it (as minimally and precisely as possible): Run UT and observe or deploy kube-proxy and review logs.
/sig network /sig node


Question: <ins> apiVersion: "v1beta1" kind: "Service" id: "monitoring-grafana" port: 8087 containerPort: 80 selector: name: "influxGrafana" </ins>

examples/monitoring/influx-grafana- and friends NEED to be replaced by a replication controller ASAP. As it stands they are bad examples. Also, they should probably move to contrib/monitoring to match contrib/logging and contrib/dns


Question: "reflect" "testing" <ins> "github.com/ghodss/yaml" </ins> "github.com/GoogleCloudPlatform/kubernetes/pkg/api" "github.com/GoogleCloudPlatform/kubernetes/pkg/api/latest" "github.com/GoogleCloudPlatform/kubernetes/pkg/api/meta"

If I stuff into a file and use that after -f, it works fine. cc
What's on stdin?
Bug, I'm only accepting json because I naively reused watch partitioner and we don't have test cases. Will fix tonight.
If that's the problem, why did it work when I gave it that file not over stdin?
Different code path. Stdin supports streaming json a la watch (multiple json objects in a stream), so you can do Unfortunately I missed yaml input and didn't use a segmenting yaml decoder (returning a stream of yaml objects).
Ah! Thanks for explanation. I think it would be good for us to ask for e2e runs before merging kubectl changes, since there's not quite enough coverage in test- yet.
Yeah, I'm slacking on that for these changes.
Fix incoming.
Fix provided. More tests to test-cmd... :)


Question: See [scheduling](https://kubernetes.io/docs/concepts/scheduling-eviction/) for more information about scheduling and the kube-scheduler component.`, Run: func(cmd *cobra.Command, args []string) { <ins> if err := opts.Complete(&namedFlagSets); err != nil { fmt.Fprintf(os.Stderr, "%vn", err) os.Exit(1) } </ins> if err := runCommand(cmd, opts, registryOptions...); err != nil { fmt.Fprintf(os.Stderr, "%vn", err) os.Exit(1)

ci-kubernetes-e2e-gci-gce-alpha-features kubetest.Up Jul 02 2021, 01:26:42 UTC+0200 /sig release
based on testgrid probably cc <img width="1203" alt="Screen Shot 2021-07-02 at 4 24 25 PM" src="https://user-"
/cc working on it
/assign
the scheduler fails to start with this error: it seems the feature gates from the command line does not take effect in the defaulting phrase, do you know how to fix it?
ah, found there is a cycle here: The scheduler options , but the default config's default values depend on the feature gates options. I think this disables other feature gate controlled logic in the defaulting too:
I tried to fix it by updating the component config after the CLI flags (including the feature gates) are parsed: PTAL
fixed the flake. is back green now.
thanks!


Question: func (l *lifecycle) Admit(a admission.Attributes) (err error) { // prevent deletion of immortal namespaces <del> if a.GetOperation() == admission.Delete { if a.GetKind() == "Namespace" && l.immortalNamespaces.Has(a.GetName()) { return errors.NewForbidden(a.GetKind(), a.GetName(), fmt.Errorf("namespace can never be deleted")) } return nil </del> <ins> if a.GetOperation() == admission.Delete && a.GetKind() == "Namespace" && l.immortalNamespaces.Has(a.GetName()) { return errors.NewForbidden(a.GetKind(), a.GetName(), fmt.Errorf("namespace can never be deleted")) </ins> } defaultVersion, kind, err := api.RESTMapper.VersionAndKindForResource(a.GetResource())

We should have a single NamespaceLifecycle plugin that enforces the Namespace rules now that all providers are off NamespaceAutoProvision. We may also want to make this no longer a user choice to configure and hard-wire the server to always run this check first to simplify configuration errors. For an example


Question: configs[EchoServer] = Config{promoterE2eRegistry, "echoserver", "2.3"} configs[Etcd] = Config{gcEtcdRegistry, "etcd", "3.4.13-0"} configs[GlusterDynamicProvisioner] = Config{promoterE2eRegistry, "glusterdynamic-provisioner", "v1.0"} <del> configs[Httpd] = Config{gcHttpdRegistry, "httpd", "2.4.38-alpine"} configs[HttpdNew] = Config{gcHttpdRegistry, "httpd", "2.4.39-alpine"} </del> <ins> configs[Httpd] = Config{promoterE2eRegistry, "httpd", "2.4.38-1"} configs[HttpdNew] = Config{promoterE2eRegistry, "httpd", "2.4.39-1"} </ins> configs[InvalidRegistryImage] = Config{invalidRegistry, "alpine", "3.1"} configs[IpcUtils] = Config{promoterE2eRegistry, "ipc-utils", "1.2"} configs[JessieDnsutils] = Config{promoterE2eRegistry, "jessie-dnsutils", "1.4"} configs[Kitten] = Config{promoterE2eRegistry, "kitten", "1.4"} configs[Nautilus] = Config{promoterE2eRegistry, "nautilus", "1.4"} configs[NFSProvisioner] = Config{sigStorageRegistry, "nfs-provisioner", "v2.2.2"} <del> configs[Nginx] = Config{promoterE2eRegistry, "nginx", "1.14-alpine"} configs[NginxNew] = Config{promoterE2eRegistry, "nginx", "1.15-alpine"} </del> <ins> configs[Nginx] = Config{promoterE2eRegistry, "nginx", "1.14-1"} configs[NginxNew] = Config{promoterE2eRegistry, "nginx", "1.15-1"} </ins> configs[NodePerfNpbEp] = Config{promoterE2eRegistry, "node-perf/npb-ep", "1.1"} configs[NodePerfNpbIs] = Config{promoterE2eRegistry, "node-perf/npb-is", "1.1"} configs[NodePerfTfWideDeep] = Config{promoterE2eRegistry, "node-perf/tf-wide-deep", "1.1"}

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --/sig windows /cc After was merged I think.
I agree that looks like the issue. It started failing between the two commits: on this run:
/triage accepted
/assign


Question: configs[EchoServer] = Config{promoterE2eRegistry, "echoserver", "2.3"} configs[Etcd] = Config{gcEtcdRegistry, "etcd", "3.4.13-0"} configs[GlusterDynamicProvisioner] = Config{promoterE2eRegistry, "glusterdynamic-provisioner", "v1.0"} <del> configs[Httpd] = Config{gcHttpdRegistry, "httpd", "2.4.38-alpine"} configs[HttpdNew] = Config{gcHttpdRegistry, "httpd", "2.4.39-alpine"} </del> <ins> configs[Httpd] = Config{promoterE2eRegistry, "httpd", "2.4.38-1"} configs[HttpdNew] = Config{promoterE2eRegistry, "httpd", "2.4.39-1"} </ins> configs[InvalidRegistryImage] = Config{invalidRegistry, "alpine", "3.1"} configs[IpcUtils] = Config{promoterE2eRegistry, "ipc-utils", "1.2"} configs[JessieDnsutils] = Config{promoterE2eRegistry, "jessie-dnsutils", "1.4"} configs[Kitten] = Config{promoterE2eRegistry, "kitten", "1.4"} configs[Nautilus] = Config{promoterE2eRegistry, "nautilus", "1.4"} configs[NFSProvisioner] = Config{sigStorageRegistry, "nfs-provisioner", "v2.2.2"} <del> configs[Nginx] = Config{promoterE2eRegistry, "nginx", "1.14-alpine"} configs[NginxNew] = Config{promoterE2eRegistry, "nginx", "1.15-alpine"} </del> <ins> configs[Nginx] = Config{promoterE2eRegistry, "nginx", "1.14-1"} configs[NginxNew] = Config{promoterE2eRegistry, "nginx", "1.15-1"} </ins> configs[NodePerfNpbEp] = Config{promoterE2eRegistry, "node-perf/npb-ep", "1.1"} configs[NodePerfNpbIs] = Config{promoterE2eRegistry, "node-perf/npb-is", "1.1"} configs[NodePerfTfWideDeep] = Config{promoterE2eRegistry, "node-perf/tf-wide-deep", "1.1"}

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Kubernetes e2e suite: [sig-apps] StatefulSet [] Basic StatefulSet functionality [StatefulSetBasic] should have a working scale subresource [Conformance] Kubernetes e2e suite: [sig-apps] StatefulSet [] Basic StatefulSet functionality [StatefulSetBasic] should perform canary updates and phased rolling updates of template modifications [Conformance] and a few others.
/sig windows
/triage accepted
/cc should be resolved by
Currently blocked because there are some issues pushing the httpd test images to . Slack thread:
Latest thread:
The job failing is:
is where the fixes for publishing this image is being tracked


Question: } } <ins> func NewTimeoutError(key, msg string) *StorageError { return &StorageError{ Code:               ErrCodeTimeout, Key:                key, AdditionalErrorMsg: msg, } } </ins> func NewInvalidObjError(key, msg string) *StorageError { return &StorageError{ Code:               ErrCodeInvalidObj,

It is possible to issue a list request which exceeds the duration set by the request-timeout apiserver flag. The request-timeout flag does not appear to be enforced when the apiserver is decoding the response from etcd. Specifically, listing a large number of CRDs from etcd can result in high request latency from using json encoding compared to protobuf encoding. While overall etcd latency is low, the overall listing from storage latency may be over the 1 minute timeout, particularly if the apiserver needs to fetch multiple pages from etcd. We expected the list requests to be subject to the duration specified by the request-timeout flag when this latency is coming from the listing from storage phase of the request. Create 4000 CRs and list all the objects from etcd in 2 pages of 2000 objects. If the time to list 1 page + decode takes around 50s, listing 2 pages will result in the listing from storage phase of the request taking around 1 minute and 40s before the request fails and the client is returned a 500 response code. Logs for apiserver Trace message: Audit log entry: Related issue for supporting protobuf for CRDs: <details</details<detailsAWS </details<details</details<details</details<details</details<details</details
/sig api-machinery CC:
Interesting.. - do you know where exactly we stuck? In the call to storage: we pass the context that has timeout set, so that shouldn't hang. OTOH, for the processing part: (a) I don't know if transformer honors the context - it probably depends on the implementation (b) but the decoding (appendListitem) certainly isn't honoring the context So (b) might definitely be a problem, even independently from from the transformer So I guess what I'm saying is that if listing from etcd finishes fast but the decoding is what takes most of time, then what you described can certainly be a problem. But is that the thing you're observing?
Yes, that is our understanding. Specifically, that the decoding which occurs in appendListItem doesn't respect the context: As a separate follow-up, it would be useful to include decoding time, or at least the overall listing from storage time, as a separate latency annotation, given the audit log entry is only capturing this latency as part of
+1.


Question: } } <ins> func NewTimeoutError(key, msg string) *StorageError { return &StorageError{ Code:               ErrCodeTimeout, Key:                key, AdditionalErrorMsg: msg, } } </ins> func NewInvalidObjError(key, msg string) *StorageError { return &StorageError{ Code:               ErrCodeInvalidObj,

We need more granular latency tracker especially on the latency spent on decoding. Right now, it only captures etcd round trip and transformer latency.
I would be supportive for both and would be happy to review it if someone opens up a PR for it.
/triage accepted
/assign
/reopen I think we need to reopen it to document how we resolved the unit test failures brought on by adding it
Reopened this issue. <detailsIn response to : /reopen I think we need to reopen it to document how we resolved the unit test failures brought on by adding it Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: out.TLSPrivateKeyFile = in.TLSPrivateKeyFile out.TLSCipherSuites = *(*[]string)(unsafe.Pointer(&in.TLSCipherSuites)) out.TLSMinVersion = in.TLSMinVersion <ins> out.RotateCertificates = in.RotateCertificates </ins> out.ServerTLSBootstrap = in.ServerTLSBootstrap if err := Convert_v1beta1_KubeletAuthentication_To_kubeletconfig_KubeletAuthentication(&in.Authentication, &out.Authentication, s); err != nil { return err

/kind cleanup What happened: and both require opt-in configuration knobs to the kubelet. They are kinda global settings, you probably either opt into dynamic cert rotation or you don't, so they qualify for being in the struct. The knob for enabling dynamic serving certs was (I fully support that :+1:). What you expected to happen: I'd like the same to happen to  to be consistent. (Also, we're planning to use the config file for the kubelet by default in kubeadm soon, and it'd be nice to be able to set this value then in a structured manner) /assign


Question: out.TLSPrivateKeyFile = in.TLSPrivateKeyFile out.TLSCipherSuites = *(*[]string)(unsafe.Pointer(&in.TLSCipherSuites)) out.TLSMinVersion = in.TLSMinVersion <ins> out.RotateCertificates = in.RotateCertificates </ins> out.ServerTLSBootstrap = in.ServerTLSBootstrap if err := Convert_v1beta1_KubeletAuthentication_To_kubeletconfig_KubeletAuthentication(&in.Authentication, &out.Authentication, s); err != nil { return err

Flag name: Help text: <Warning: Beta featureAuto rotate the kubelet client certificates by requesting new certificates from the kube-apiserver when the certificate expiration approaches. This is part of migrating the Kubelet command-line to a Kubernetes-style API. The flag should either be migrated to the Kubelet's API group, or simply removed from the Kubelet. If this could be considered an instance-specific flag, or a descriptor of local topology managed by the Kubelet, see: If this flag is only registered in os-specific builds, see: /assign /sig node /kind feature /priority important-soon /milestone v1.11 /status approved-for-milestone
[MILESTONENOTIFIER] Milestone Issue: Up-to-date for process <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. : New functionality. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details


Question: 0, "10.0.0.0/8", nodeName, <del> utilnode.GetNodeIP(client, nodeName), </del> <ins> nodeIP, </ins> recorder, nil, []string{},

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Using Kubernetes 1.16.1, I started kube-proxy in iptables mode as a static pod on a master node. The master node was intentionally (and permanently) not available in the API as a Node object. kube-proxy started up and terminated soon after because it failed to (which is supposed to function as the bound address) and eventually returned an error: What you expected to happen: I expected the same behavior as in Kubernetes 1.15 where failure to look up the Node in the API would result in . How to reproduce it (as minimally and precisely as possible): Configure kube-proxy with iptables mode on Kubernetes 1.16 without the hosting Node being available in the Kubernetes API (under the OS hostname). Anything else we need to know?: PR (Fix for kube-proxy to wait for some duration for the node to be defined) introduced an additional . I believe this is the part that is causing the change in behavior: in Kubernetes 1.15, the check would be omitted and cause to set the default bind address to 127.0.0.1 (when iptables mode is set; I haven't checked other modes). The PR's primary purpose seems to be a different one (retrying node IP retrieval from the API), so my understanding is that the check was not really needed; unless I'm misunderstanding things which I don't want to rule out at all. The quick workaround was to explicitly configure kube-proxy with . However, I'm not entirely sure if that's a good solution long-term compared to letting Kubernetes do The Right Thing, which is what I'd like to clarify through this issue. Environment: Kubernetes version (use ): v1.16.1 Cloud provider or hardware configuration: Kubernetes running on DigitalOcean VMs (droplets) OS (e.g: ): Debian GNU/Linux 9 (stretch) Kernel (e.g.


Question: 0, "10.0.0.0/8", nodeName, <del> utilnode.GetNodeIP(client, nodeName), </del> <ins> nodeIP, </ins> recorder, nil, []string{},

): Linux master-k8s-1-16-test1 4.19.0-0.bpo.6-amd64 SMP Debian 4.19.67-2+deb10u1~bpo9+1 (2019-09-30) x86_64 GNU/Linux /sig network /cc who were involved in the referenced PR.
I made a similar comment in , wondering if this is a breaking change in v1.16 that we should revert? Previously if is nil, kube-proxy defaults to but now kube-proxy always exits with error instead of defaulting to local host. I think a lot of external cloud providers relied on this behavior because they needed to talk to API server prior to node addresses being set by the provider. Thoughts /assign
Opened to preserve behavior before v1.16 where node IP defaults to 127.0.01 is none is found. PTAL


Question: if len(g.predicates) == 0 { filtered = nodes } else { <del> allNodes := int32(g.cache.NodeTree().NumNodes) </del> <ins> allNodes := int32(g.cache.NodeTree().NumNodes()) </ins> numNodesToFind := g.numFeasibleNodesToFind(allNodes) // Create filtered list with enough space to avoid growing it

Description: , part of the is exporting a field which allows to modify its value externally. This also means that this field is not thread-safe. Solution: Expose a method that returns this value in a thread-safe way and do not export the field. /kind bug /sig scheduling


Question: } } } <del> if !reflect.DeepEqual(oldTaints, newTaints) { </del> <ins> oldTaints, oldErr := helper.GetTaintsFromNodeAnnotations(oldNode.GetAnnotations()) if oldErr != nil { glog.Errorf("Failed to get taints from old node annotation for equivalence cache") } newTaints, newErr := helper.GetTaintsFromNodeAnnotations(newNode.GetAnnotations()) if newErr != nil { glog.Errorf("Failed to get taints from new node annotation for equivalence cache") } if !reflect.DeepEqual(oldTaints, newTaints) || !reflect.DeepEqual(oldNode.Spec.Taints, newNode.Spec.Taints) { </ins> invalidPredicates.Insert("PodToleratesNodeTaints") } if !reflect.DeepEqual(oldNode.Status.Conditions, newNode.Status.Conditions) { oldConditions := make(map[v1.NodeConditionType]v1.ConditionStatus) newConditions := make(map[v1.NodeConditionType]v1.ConditionStatus)

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: alpha suite has been failing in the BeforeSuite since 9/22 Did we recently enable some alpha taints feature?
It may be related to cc
/assign
, do you know how to trigger this test in CI, e.g. ?
I don't think there's a way to launch a ci job from here.
This seems to be the same failure as . You can trigger the GKE GPU tests on PRs using . GKE GPU tests use alpha features.
sure, let me add some instrument code and trigger test by ci :).


Question: "spec.nodeName", "spec.restartPolicy", "spec.schedulerName", <ins> "spec.serviceAccountName", </ins> "status.phase", "status.podIP", "status.nominatedNodeName":

Is this a BUG REPORT or FEATURE REQUEST?: /kind feature What happened: When trying to list object using field spec.ServiceAccountName, we are getting the below error message. field label not supported: spec.serviceAccountName What you expected to happen: To be able to fetch pods using the list and watch based on the field 'spec.serviceAccountName' How to reproduce it (as minimally and precisely as possible): in minikube run Anything else we need to know?: Description: Unable to fetch pods based on the field 'spec.serviceAccountName'. When trying to get an overview of RBAC privileges in a cluster it is hard to find the objects attached to service accounts Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-06-27T20:17:28Z", GoVersion:"go1.10.2", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-06-27T20:08:34Z", GoVersion:"go1.10.2", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: minikube version: v0.28.0 OS (e.g. from /etc/os-release): NAME=Buildroot VERSION=2018.05 ID=buildroot VERSIONID=2018.05 PRETTYNAME="Buildroot 2018.05" Kernel (e.g. ): Linux minikube 4.16.14 SMP Mon Jun 11 16:20:15 UTC 2018 x86_64 GNU/Linux Install tools: kubeadm Others:
/sig architecture
You might do it by jsonpath filter
I intend to utilize it in the client-go list as an option
Not all fields are supported for field selector. There is a doc about field selector which is still under review: As far as I can see, supported field selectors for pods are listed here:
Didn't find that you have a pull request for it. :) Good luck!


Question: return nil, err } if mount.SubPath != "" { <ins> fileinfo, err := os.Lstat(hostPath) if err != nil { return nil, err } perm := fileinfo.Mode() </ins> hostPath = filepath.Join(hostPath, mount.SubPath) <ins> // Create the sub path now because if it's auto-created later when referenced, it may have an // incorrect ownership and mode. For example, the sub path directory must have at least g+rwx // when the pod specifies an fsGroup, and if the directory is not created here, Docker will // later auto-create it with the incorrect mode 0750 if err := os.MkdirAll(hostPath, perm); err != nil { glog.Errorf("failed to mkdir:%s", hostPath) return nil, err } // chmod the sub path because umask may have prevented us from making the sub path with the same // permissions as the mounter path if err := os.Chmod(hostPath, perm); err != nil { return nil, err } </ins> } // Docker Volume Mounts fail on Windows if it is not of the form C:/

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): no What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): subpath, emptyDir, mode, permissions Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-01-12T04:57:25Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-01-12T04:52:34Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"} Environment: Cloud provider or hardware configuration: Bare metal, SSD, but also VMs OS (e.g. from /etc/os-release): Debian GNU/Linux 8 (jessie) Kernel (e.g. ): 4.8.0-0.bpo.2-amd64 Install tools: Others: Docker v1.12.6 What happened: When I mount an volume using a subpath, I'm unable to write into it if my process is not running as . When I leave out the subpath, I can. What you expected to happen: I can write to the directory in both cases.


Question: return nil, err } if mount.SubPath != "" { <ins> fileinfo, err := os.Lstat(hostPath) if err != nil { return nil, err } perm := fileinfo.Mode() </ins> hostPath = filepath.Join(hostPath, mount.SubPath) <ins> // Create the sub path now because if it's auto-created later when referenced, it may have an // incorrect ownership and mode. For example, the sub path directory must have at least g+rwx // when the pod specifies an fsGroup, and if the directory is not created here, Docker will // later auto-create it with the incorrect mode 0750 if err := os.MkdirAll(hostPath, perm); err != nil { glog.Errorf("failed to mkdir:%s", hostPath) return nil, err } // chmod the sub path because umask may have prevented us from making the sub path with the same // permissions as the mounter path if err := os.Chmod(hostPath, perm); err != nil { return nil, err } </ins> } // Docker Volume Mounts fail on Windows if it is not of the form C:/

How to reproduce it (as minimally and precisely as possible): Output is: Anything else we need to know: This issue is related to . I know this is caused by the Docker daemon creating those directories on a bind mount. Nevertheless, I expect Kubernetes to abstract this away, since this behavior is neither documented nor IMHO intuitive.
Could someone from please take a look? Thanks! cc:
cc/
attempted fix created here
I created a cherrypick PR for 1.6 here: since this bug is against 1.6. Could someone (who agrees this could be cherrypicked) please add the cherrypick-candidate label + milestone to the original PR , thank you! edit: sorry, it's against 1.5. Should it be cherrypicked back to that as well?
I'm still facing this issue with Kubernetes version 1.12.8 on a AKS cluster when I mount a Azure Storage as azureFile share. Did anyone know if the issue is fixed for all types of mounts and not only just the emptyDir?
same issue on GCP @ k8s 1.14
k8s 1.17 via eks as well
So what's the recommendation? Not using subPath?


Question: metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" clientset "k8s.io/client-go/kubernetes" "k8s.io/kubernetes/test/e2e/framework" <ins> uexec "k8s.io/utils/exec" </ins> ) type KubeletOpt string

It would better to have e2e for iSCSI persistent volume. /sig storage
/assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close


Question: visible at runtime in the container. release: v1.9 file: test/e2e/common/downward_api.go <ins> - testname: PodTemplate, delete a collection codename: '[sig-node] PodTemplates should delete a collection of pod templates [Conformance]' description: A set of Pod Templates is created with a label selector which MUST be found when listed. The set of Pod Templates is deleted and MUST NOT show up when listed by its label selector. release: v1.19 file: test/e2e/common/podtemplates.go </ins> - testname: PodTemplate lifecycle codename: '[sig-node] PodTemplates should run the lifecycle of PodTemplates [Conformance]' description: Attempt to create a PodTemplate. Patch the created PodTemplate. Fetching

[x] Original approval issue: [x] two weeks soak end date: 2 July 2020 This issue is created to allow edit by According to this APIsnoop query, there are still an endpoint which is untested. /wip /hold a set of 3 Pod Templates with a static label within a Namespace. that all 3 Pod Templates with the label in the Namespace are created. the set of Namespaced Pod Templates with a label via DeleteCollection. that all Pod Templates with the label have been deleted. Discover useragents: List endpoints hit by the test: Display endpoint coverage change: If a test with these calls gets merged, test coverage will go up by 1 points This test is also created with the goal of conformance promotion.
/sig testing /sig architecture /area conformance
/area conformance
During Conformance Meeting: Is there a way to write this test so it is resource agnostic? If we are just changing the resource... can be parameterize the resource-type
/assign
GitHub didn't allow me to assign the following users: heyste. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
GitHub didn't allow me to assign the following users: heyste. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen Merge 17 July 2020
Reopened this issue. <detailsIn response to : /reopen Merge 17 July 2020 Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: kubectl delete secret secret-string-data --namespace=test-secrets ### Create a secret using output flags <ins> if [[ "${WAIT_FOR_DELETION:-}" == "true" ]]; then kube::test::wait_object_assert 'secrets --namespace=test-secrets' "{{range.items}}{{$id_field}}:{{end}}" '' fi </ins> # Pre-condition: no secret exists kube::test::get_object_assert 'secrets --namespace=test-secrets' "{{range.items}}{{$id_field}}:{{end}}" '' # Command

disabled them since was flaky. We need to find and fix the issue and then reenable these tests. cc Related issues: and
Probably should be higher priority so it doesn't get dropped
The issue is that we dont wait for the resource to be deleted unless we are using a reaper: This breaks for federation since we use finalizers. kubectl should wait for the resource to be deleted. We should also set OrphanDependents = false. This will have no effect in kubernetes but will enable cascading deletion in federation. Ref cc Will send a fix.
Sent
Updated to remove the wait (GET) at the end so it no longer fixes this issue. Sent to update the test to include a wait which fixes this issue.


Question: return map[string][]api.Pod{}, err } for _, scheduledPod := range pods { <del> host := scheduledPod.CurrentState.Host </del> <ins> host := scheduledPod.DesiredState.Host </ins> machineToPods[host] = append(machineToPods[host], scheduledPod) } return machineToPods, nil

When creating replication controller (e.g. frontend php-redis from guestbook example), scheduler gets stuck and never recovers. This is a race, I have to run several times to see the error (A script to delete controller and pods, then create new controller, leave enough time in between for k8s to react). I've created PR for an attempted fix.
I saw this too while following the guest book and creating the php/frontend step. In my case, I had 2 minions while the replication controller configuration had configured 3 replicas, so 2 pods got scheduled -each on another minion, the third pod had empty Host (saw that when doing 'list pods'), and the 3 of them were stuck in "waiting" status. I deleted the controller and the pods, fixed the configuration in json file to be 2 replicas and recreated the controller and everything worked. Does your fix take care of this use case? (too many replicas, too few minions/one of the pods is not scheduled)
This is different from your use case. If you have more replicas than minions, then the scheduler will get stuck for sure (with the error msg probably being 'failed to find a fit for pod'). AFAICT, it's not support yet.
If you remove the HostPorts, it should work for you. thanks for report.
, thanks for your replies. I understand why it isn't working, but I'd expect one of the 2 options to happen instead of the current situation: the creation of replication controller if there are less minions than replicas, AND there is a hostPort property. In other words - validation of the content during creation. least schedule all replicas that can get a minion (in my case 2 out of 3) and leave only one at "waiting". Is there an issue open about the use case I describe?
should be true today. is less possible than it seems like-- minions can come and go, and become over or under loaded in the time between creation and when the pod shows up at the scheduler. So even if we check up front, we still have to check again later, and sometimes the first check will pass but not the second.


Question: assert.Equal([]string{"127.0.0.2", "127.0.0.2"}, addrs) } <ins> // Because we need to be backwards compatible with release 1.1, at endpoints // that exist in release 1.1, the responses should have empty APIVersion. func TestAPIVersionOfDiscoveryEndpoints(t *testing.T) { master, etcdserver, _, assert := newMaster(t) defer etcdserver.Terminate(t) server := httptest.NewServer(master.HandlerContainer.ServeMux) // /api exists in release-1.1 resp, err := http.Get(server.URL + "/api") if err != nil { t.Errorf("unexpected error: %v", err) } apiVersions := unversioned.APIVersions{} assert.NoError(decodeResponse(resp, &apiVersions)) assert.Equal(apiVersions.APIVersion, "") // /api/v1 exists in release-1.1 resp, err = http.Get(server.URL + "/api/v1") if err != nil { t.Errorf("unexpected error: %v", err) } resourceList := unversioned.APIResourceList{} assert.NoError(decodeResponse(resp, &resourceList)) assert.Equal(resourceList.APIVersion, "") // /apis exists in release-1.1 resp, err = http.Get(server.URL + "/apis") if err != nil { t.Errorf("unexpected error: %v", err) } groupList := unversioned.APIGroupList{} assert.NoError(decodeResponse(resp, &groupList)) assert.Equal(groupList.APIVersion, "") // /apis/extensions exists in release-1.1 resp, err = http.Get(server.URL + "/apis/extensions") if err != nil { t.Errorf("unexpected error: %v", err) } group := unversioned.APIGroup{} assert.NoError(decodeResponse(resp, &group)) assert.Equal(group.APIVersion, "") // /apis/extensions/v1beta1 exists in release-1.1 resp, err = http.Get(server.URL + "/apis/extensions/v1beta1") if err != nil { t.Errorf("unexpected error: %v", err) } resourceList = unversioned.APIResourceList{} assert.NoError(decodeResponse(resp, &resourceList)) assert.Equal(resourceList.APIVersion, "") // /apis/autoscaling doesn't exist in release-1.1, so the APIVersion field // should be non-empty in the results returned by the server. resp, err = http.Get(server.URL + "/apis/autoscaling") if err != nil { t.Errorf("unexpected error: %v", err) } group = unversioned.APIGroup{} assert.NoError(decodeResponse(resp, &group)) assert.Equal(group.APIVersion, "v1") // apis/autoscaling/v1 doesn't exist in release-1.1, so the APIVersion field // should be non-empty in the results returned by the server. resp, err = http.Get(server.URL + "/apis/autoscaling/v1") if err != nil { t.Errorf("unexpected error: %v", err) } resourceList = unversioned.APIResourceList{} assert.NoError(decodeResponse(resp, &resourceList)) assert.Equal(resourceList.APIVersion, "v1") } </ins> func TestDiscoveryAtAPIS(t *testing.T) { master, etcdserver, config, assert := newMaster(t) defer etcdserver.Terminate(t)

Kubectl client Kubectl api-versions should check if v1 is in available api versions [Conformance] Expected error: <_errors.errorString | 0xc208641770: { s: "Error running &{/jenkins-master-data/jobs/kubernetes-upgrade-gke-1.1-master-step1-deploy/workspace/kubernetes/platforms/linux/amd64/kubectl [kubectl --server=https://130.211.136.232 --kubeconfig=/var/lib/jenkins/jobs/kubernetes-upgrade-gke-1.1-master-step1- api-versions] [] error: Couldn't get available api versions from server: no kind "APIVersions" is registered for version "v1"n [] 0xc208382fa0 exit status 1 true [0xc208627148 0xc208627168 0xc208627188] [0xc208627148 0xc208627168 0xc208627188] [0xc208627160 0xc208627180] [0x6d81a0 0x6d81a0] 0xc2082fb2c0}:nCommand stdout:nnstderr:nerror: Couldn't get available api versions from server: no kind "APIVersions" is registered for version "v1"nn", } Error running &{/jenkins-master-data/jobs/kubernetes-upgrade-gke-1.1-master-step1-deploy/workspace/kubernetes/platforms/linux/amd64/kubectl [kubectl --server=https://130.211.136.232 --kubeconfig=/var/lib/jenkins/jobs/kubernetes-upgrade-gke-1.


Question: assert.Equal([]string{"127.0.0.2", "127.0.0.2"}, addrs) } <ins> // Because we need to be backwards compatible with release 1.1, at endpoints // that exist in release 1.1, the responses should have empty APIVersion. func TestAPIVersionOfDiscoveryEndpoints(t *testing.T) { master, etcdserver, _, assert := newMaster(t) defer etcdserver.Terminate(t) server := httptest.NewServer(master.HandlerContainer.ServeMux) // /api exists in release-1.1 resp, err := http.Get(server.URL + "/api") if err != nil { t.Errorf("unexpected error: %v", err) } apiVersions := unversioned.APIVersions{} assert.NoError(decodeResponse(resp, &apiVersions)) assert.Equal(apiVersions.APIVersion, "") // /api/v1 exists in release-1.1 resp, err = http.Get(server.URL + "/api/v1") if err != nil { t.Errorf("unexpected error: %v", err) } resourceList := unversioned.APIResourceList{} assert.NoError(decodeResponse(resp, &resourceList)) assert.Equal(resourceList.APIVersion, "") // /apis exists in release-1.1 resp, err = http.Get(server.URL + "/apis") if err != nil { t.Errorf("unexpected error: %v", err) } groupList := unversioned.APIGroupList{} assert.NoError(decodeResponse(resp, &groupList)) assert.Equal(groupList.APIVersion, "") // /apis/extensions exists in release-1.1 resp, err = http.Get(server.URL + "/apis/extensions") if err != nil { t.Errorf("unexpected error: %v", err) } group := unversioned.APIGroup{} assert.NoError(decodeResponse(resp, &group)) assert.Equal(group.APIVersion, "") // /apis/extensions/v1beta1 exists in release-1.1 resp, err = http.Get(server.URL + "/apis/extensions/v1beta1") if err != nil { t.Errorf("unexpected error: %v", err) } resourceList = unversioned.APIResourceList{} assert.NoError(decodeResponse(resp, &resourceList)) assert.Equal(resourceList.APIVersion, "") // /apis/autoscaling doesn't exist in release-1.1, so the APIVersion field // should be non-empty in the results returned by the server. resp, err = http.Get(server.URL + "/apis/autoscaling") if err != nil { t.Errorf("unexpected error: %v", err) } group = unversioned.APIGroup{} assert.NoError(decodeResponse(resp, &group)) assert.Equal(group.APIVersion, "v1") // apis/autoscaling/v1 doesn't exist in release-1.1, so the APIVersion field // should be non-empty in the results returned by the server. resp, err = http.Get(server.URL + "/apis/autoscaling/v1") if err != nil { t.Errorf("unexpected error: %v", err) } resourceList = unversioned.APIResourceList{} assert.NoError(decodeResponse(resp, &resourceList)) assert.Equal(resourceList.APIVersion, "v1") } </ins> func TestDiscoveryAtAPIS(t *testing.T) { master, etcdserver, config, assert := newMaster(t) defer etcdserver.Terminate(t)

1-master-step1- api-versions] [] error: Couldn't get available api versions from server: no kind "APIVersions" is registered for version "v1" [] 0xc208382fa0 exit status 1 true [0xc208627148 0xc208627168 0xc208627188] [0xc208627148 0xc208627168 0xc208627188] [0xc208627160 0xc208627180] [0x6d81a0 0x6d81a0] 0xc2082fb2c0}: Command stdout: stderr: error: Couldn't get available api versions from server: no kind "APIVersions" is registered for version "v1" not to have occurred* See and Investigating. cc
Okiedokie. Looks like the v1.1 client is getting a response from the v1.2 master, but the v1.1 client doesn't know how to handle the v1 APIVersions object, because in v1.1, that object was never expected to have an . From a 1.1 server, we get which doesn't include a . We should expect to have plenty of pre-v1.1.7 clients floating around, and releasing a fixed v1.1.8 probably isn't going to solve this; if we want to be backward compatible, we need to not return an string with unversioned kinds like . Looking into more details about how we got here... cc cc
I have a maybe feeling that this is caused by ? In particular, we went from: to
Probably it's caused by this:
Let me double check the code. cc
What about that line? I've never dealt with the version negotiation; should the server and client be negotiating in a way that the server can return an object without ?
More precisely it's this line: This line sets the version to "v1". In , during encoding, we set unversioned object's APIVersion field to "v1".
I think in the long run, because we expect to move many unversioned types to group "common", so they should be versioned, so making APIVersion=v1 is the right move. Back to the backwards compatibility issue, how's the situation at other url, like http://127.0.0.1:8080/apis/extensions?


Question: assert.Equal([]string{"127.0.0.2", "127.0.0.2"}, addrs) } <ins> // Because we need to be backwards compatible with release 1.1, at endpoints // that exist in release 1.1, the responses should have empty APIVersion. func TestAPIVersionOfDiscoveryEndpoints(t *testing.T) { master, etcdserver, _, assert := newMaster(t) defer etcdserver.Terminate(t) server := httptest.NewServer(master.HandlerContainer.ServeMux) // /api exists in release-1.1 resp, err := http.Get(server.URL + "/api") if err != nil { t.Errorf("unexpected error: %v", err) } apiVersions := unversioned.APIVersions{} assert.NoError(decodeResponse(resp, &apiVersions)) assert.Equal(apiVersions.APIVersion, "") // /api/v1 exists in release-1.1 resp, err = http.Get(server.URL + "/api/v1") if err != nil { t.Errorf("unexpected error: %v", err) } resourceList := unversioned.APIResourceList{} assert.NoError(decodeResponse(resp, &resourceList)) assert.Equal(resourceList.APIVersion, "") // /apis exists in release-1.1 resp, err = http.Get(server.URL + "/apis") if err != nil { t.Errorf("unexpected error: %v", err) } groupList := unversioned.APIGroupList{} assert.NoError(decodeResponse(resp, &groupList)) assert.Equal(groupList.APIVersion, "") // /apis/extensions exists in release-1.1 resp, err = http.Get(server.URL + "/apis/extensions") if err != nil { t.Errorf("unexpected error: %v", err) } group := unversioned.APIGroup{} assert.NoError(decodeResponse(resp, &group)) assert.Equal(group.APIVersion, "") // /apis/extensions/v1beta1 exists in release-1.1 resp, err = http.Get(server.URL + "/apis/extensions/v1beta1") if err != nil { t.Errorf("unexpected error: %v", err) } resourceList = unversioned.APIResourceList{} assert.NoError(decodeResponse(resp, &resourceList)) assert.Equal(resourceList.APIVersion, "") // /apis/autoscaling doesn't exist in release-1.1, so the APIVersion field // should be non-empty in the results returned by the server. resp, err = http.Get(server.URL + "/apis/autoscaling") if err != nil { t.Errorf("unexpected error: %v", err) } group = unversioned.APIGroup{} assert.NoError(decodeResponse(resp, &group)) assert.Equal(group.APIVersion, "v1") // apis/autoscaling/v1 doesn't exist in release-1.1, so the APIVersion field // should be non-empty in the results returned by the server. resp, err = http.Get(server.URL + "/apis/autoscaling/v1") if err != nil { t.Errorf("unexpected error: %v", err) } resourceList = unversioned.APIResourceList{} assert.NoError(decodeResponse(resp, &resourceList)) assert.Equal(resourceList.APIVersion, "v1") } </ins> func TestDiscoveryAtAPIS(t *testing.T) { master, etcdserver, config, assert := newMaster(t) defer etcdserver.Terminate(t)

That address also returns a versioned object, which at release 1.1 is unversioned: Do we need to keep all unversioned objects unversioned, or do we only need to ensure the server returns unversioned object at api/?
cc
Per our versioning rules, we should expect a v1.2 server to remain backward compatible with v1.1 clients. I don't have much experience with the api machinery, but if clients weren't expecting an unversioned object, it seems like we can't up and switch to versioned objects.
I think we have to maintain compatibility somehow. any ideas for a fix other than the obvious one?
Yes, we have to maintain backward compatibility with 1.1 (and 1.0) clients. New paths that old clients didn't know about could behave differently than paths that existed in previous releases.
Nope, let's make sure we leave a test and comment.
We can't make the change here though, this would break "versioned unversioned" objects (like )
The other implication of course is now we "versioned unversioned" objects (v1 status) and "unversioned unversioned", which means we shouldn't ever add new objects that don't return a version. I don't think anything in 1.2 qualifies as a new object without a version, just 1.1 objects that didn't have correct versions. Having objects from extensions group be returned without versions terrifies me - I don't see any way of describing that except as a bug.
We may need per API group version rules registered on scheme about whether "unversioned" objects are returned in their scheme version or as empty. That's probably the backward compatible fix that gives us a hope of reasoning about behavior correctly (status in 1.0 is v1, status in extensions might be "", apiversions in any new group is "v1"). The unversioned marker on the objects should ensure 1.2 clients can receive objects in both "" and "v1" form, we should add a test to verify that. I'm against letting empty API versions spread, but agree backcompat for existing groups versions should match previous behavior.
Can we make (or use) a different serializer/codec just for the endpoints that 1.1 clients used?


Question: assert.Equal([]string{"127.0.0.2", "127.0.0.2"}, addrs) } <ins> // Because we need to be backwards compatible with release 1.1, at endpoints // that exist in release 1.1, the responses should have empty APIVersion. func TestAPIVersionOfDiscoveryEndpoints(t *testing.T) { master, etcdserver, _, assert := newMaster(t) defer etcdserver.Terminate(t) server := httptest.NewServer(master.HandlerContainer.ServeMux) // /api exists in release-1.1 resp, err := http.Get(server.URL + "/api") if err != nil { t.Errorf("unexpected error: %v", err) } apiVersions := unversioned.APIVersions{} assert.NoError(decodeResponse(resp, &apiVersions)) assert.Equal(apiVersions.APIVersion, "") // /api/v1 exists in release-1.1 resp, err = http.Get(server.URL + "/api/v1") if err != nil { t.Errorf("unexpected error: %v", err) } resourceList := unversioned.APIResourceList{} assert.NoError(decodeResponse(resp, &resourceList)) assert.Equal(resourceList.APIVersion, "") // /apis exists in release-1.1 resp, err = http.Get(server.URL + "/apis") if err != nil { t.Errorf("unexpected error: %v", err) } groupList := unversioned.APIGroupList{} assert.NoError(decodeResponse(resp, &groupList)) assert.Equal(groupList.APIVersion, "") // /apis/extensions exists in release-1.1 resp, err = http.Get(server.URL + "/apis/extensions") if err != nil { t.Errorf("unexpected error: %v", err) } group := unversioned.APIGroup{} assert.NoError(decodeResponse(resp, &group)) assert.Equal(group.APIVersion, "") // /apis/extensions/v1beta1 exists in release-1.1 resp, err = http.Get(server.URL + "/apis/extensions/v1beta1") if err != nil { t.Errorf("unexpected error: %v", err) } resourceList = unversioned.APIResourceList{} assert.NoError(decodeResponse(resp, &resourceList)) assert.Equal(resourceList.APIVersion, "") // /apis/autoscaling doesn't exist in release-1.1, so the APIVersion field // should be non-empty in the results returned by the server. resp, err = http.Get(server.URL + "/apis/autoscaling") if err != nil { t.Errorf("unexpected error: %v", err) } group = unversioned.APIGroup{} assert.NoError(decodeResponse(resp, &group)) assert.Equal(group.APIVersion, "v1") // apis/autoscaling/v1 doesn't exist in release-1.1, so the APIVersion field // should be non-empty in the results returned by the server. resp, err = http.Get(server.URL + "/apis/autoscaling/v1") if err != nil { t.Errorf("unexpected error: %v", err) } resourceList = unversioned.APIResourceList{} assert.NoError(decodeResponse(resp, &resourceList)) assert.Equal(resourceList.APIVersion, "v1") } </ins> func TestDiscoveryAtAPIS(t *testing.T) { master, etcdserver, config, assert := newMaster(t) defer etcdserver.Terminate(t)

Probably we just need to change the version registered here:
That won't work like you expect - the type registry then has no knowledge in v1 of what schema the object is. That's not what we want - we want to not serialize the version of the object for specific API groups, which is a much more targeted change. This is a special, exceptional case (v1 group should return v1 Status. v1beta1 should not). , Chao Xu wrote:
Changing v1 breaks 1.0 backwards compat (we always returned apiVersion=v1 for 1.0) , Clayton Coleman wrote:
I mean this: Will this work?
I'm not sure without looking at the outcome (which is why unversioned is a bit of a disaster in our codebase). I would much prefer to explicitly identify which objects should have an unset apiversion via a method on the scheme, and do the explicit check at the time we do convert to version. That's an extremely controlled change, is obvious, and doesn't make the unversioned problem worse.
I remember somewhere in asserts version cannot be empty, so I guess my proposed solution won't work without extra work. I'll take the approach suggested by


Question: if err != nil { return nil, err } <ins> ams, err := getVolumeAccessModes(spec) if err != nil { return nil, err } </ins> secretName, secretNs, err := getSecretNameAndNamespace(spec, pod.Namespace) if err != nil {

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: When use ceph as storageclass, i set accessModes: ReadOnlyMany, and create pods using the claim, only one pod would be running and the others on different nodes would get stuck on ContainerCreating. But if the pods are all on same node, it works fine. What you expected to happen: Pods on different nodes can mount ceph rbd volume. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): v1.9.2 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): 3.10.0-327.18.2.el7.x86_64 Kernel (e.g. ): CentOS Linux release 7.2.1511 (Core) Install tools: kubeadm Others:
/sig storage
/assign
, how would this work? For example, how do you put the initial contents of the RBD volume on it if it is in the ReadOnlyMany access mode? How would you use this feature? As a cheap way of sharing files across a Kubernetes cluster? I'm just interested. -ben england
Do we really want to support ? I have the same questions as mentioned above. If we set the PVC AccessMode to , then the field in will need to be set true ? If so, for the first time, in , we will mount the device without formatting, is that intended ? We need to format the device manually before we use it ? /cc FYI: also there is an issue related to this:
We have this same problem for GCE PD, which supports ReadOnlyMany. You need to mount the volume first as RWO, in order to format the filesystem and populate it with data (mounting a read only volume with no data on it seems pointless). Then you have to fully unmount/detach it, and then you can deploy pods using it as ROX. Basically, you can dynamically provision it in RWO to format + populate data. But afterwards, you need to manually create a different PV/PVC for ROX
continuation


Question: <del> // +build !windows /* Copyright 2018 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package factory import "syscall" // compareSignal is the signal to trigger cache compare. For non-windows // environment it's SIGUSR2. var compareSignal = syscall.SIGUSR2 </del>

PLEASE COORDINATE WITH SIG-SCHEDULING BEFORE WORKING ON THIS ISSUE. We want to avoid the situation where two people accidentally start work on the same fix. Main tracking issue: Currently, the feature is set up at the end of in . This setup should be moved into its own helper function, preferably in the package. /sig scheduling /kind cleanup /help
/assign /remove-help
I'd like to take this one
/remove-help
the PR I sent should resolve this one. :disappointed: Are any of the others still available?


Question: } // Ensure that the if the APIService OpenAPI is updated, the // aggregated OpenAPI is also updated. <ins> expectPath(t, swagger, "/apis/apiservicegroup/v1/path2") expectNoPath(t, swagger, "/apis/apiservicegroup/v1/path1") expectPath(t, swagger, "/apis/apiregistration.k8s.io/v1") } // Tests that an APIService that registers OpenAPI will only have the OpenAPI // for its specific group version served registered. // See https://github.com/kubernetes/kubernetes/pull/123570 for full context. func TestAPIServiceOpenAPIServiceMismatch(t *testing.T) { mux := http.NewServeMux() var delegationHandlers []http.Handler delegate1 := &openAPIHandler{openapi: &spec.Swagger{ SwaggerProps: spec.SwaggerProps{ Paths: &spec.Paths{ Paths: map[string]spec.PathItem{ "/apis/foo/v1": {}, }, }, }, }} delegationHandlers = append(delegationHandlers, delegate1) s := buildAndRegisterSpecAggregator(delegationHandlers, mux) apiService := &v1.APIService{ Spec: v1.APIServiceSpec{ Group:   "apiservicegroup", Version: "v1", Service: &v1.ServiceReference{Name: "dummy"}, }, } apiService.Name = "apiservice" apiService2 := &v1.APIService{ Spec: v1.APIServiceSpec{ Group:   "apiservicegroup", Version: "v2", Service: &v1.ServiceReference{Name: "dummy2"}, }, } apiService2.Name = "apiservice2" handler := &openAPIHandler{openapi: &spec.Swagger{ SwaggerProps: spec.SwaggerProps{ Paths: &spec.Paths{ Paths: map[string]spec.PathItem{ "/apis/apiservicegroup/v1": {}, }, }, }, }} handler2 := &openAPIHandler{openapi: &spec.Swagger{ SwaggerProps: spec.SwaggerProps{ Paths: &spec.Paths{ Paths: map[string]spec.PathItem{ "/apis/a":                  {}, "/apis/apiservicegroup/v1": {}, "/apis/apiservicegroup/v2": {}, }, }, }, }} if err := s.AddUpdateAPIService(apiService, handler); err != nil { t.Error(err) } if err := s.UpdateAPIServiceSpec(apiService.Name); err != nil { t.Error(err) } if err := s.AddUpdateAPIService(apiService2, handler2); err != nil { t.Error(err) } if err := s.UpdateAPIServiceSpec(apiService2.Name); err != nil { t.Error(err) } swagger, err := fetchOpenAPI(mux) if err != nil { t.Error(err) } expectPath(t, swagger, "/apis/apiservicegroup/v1") </ins> expectPath(t, swagger, "/apis/apiservicegroup/v2") <ins> expectPath(t, swagger, "/apis/apiregistration.k8s.io/v1") t.Logf("Remove APIService %s", apiService.Name) s.RemoveAPIService(apiService.Name) swagger, err = fetchOpenAPI(mux) if err != nil { t.Error(err) } // Ensure that the if the APIService is added then removed, the OpenAPI disappears from the aggregated OpenAPI as well. </ins> expectNoPath(t, swagger, "/apis/apiservicegroup/v1") expectPath(t, swagger, "/apis/apiregistration.k8s.io/v1") }

When installing the metrics-server to implement the API and the prometheus-adapter to implement the API, it is possible to make the OpenAPI handler fail to initialize with this error log in apiserver: This will result in kubectl discovery calls to /openapi/v2 to fail with a 503: The OpenAPI handler should not fail to initialize if both and are serving the same path for different API groups/versions. I can provide exact installation steps if needed, however, a high-level repro is: install metrics-server to serve install prometheus-adapter to serve ensure resourceRules is populated in the prometheus-adapter config so that resource metrics are also served by restarting apiserver will result in the OpenAPI failure and 503 responses to /openapi/v2 path No response <details</details<detailsAWS </details<details</details<details</details<details</details<details</details
Please let me know if I should reopen the issue in
/sig api-machinery
/sig instrumentation [metrics apis]
Would you mind taking a look when have time? Thank you^^ /triage accepted
I have a question. should be serving under . Why does an apiservice serving another gvk register paths under ? cc we've made the assumption that an APIService (such as metrics-server or prometheus-adapter) serves only gvks under its own registered group version. I'll do some more investigation but just wondering if prometheus doesn't follow that assumption?
the prometheus-adapter can also be configured to serve and replace the metrics-server by enabling these metrics, However, if the prometheus-adapter fronts and metrics-server fronts , it does seem like the correct behavior would be for prometheus-adapter to serve until an operator updates the APIService to reference prometheus-adapter rather than metrics-server. We're seeing this issue occur for upgrades between 1.27 and 1.28 so it does look this path duplication did not cause the OpenAPI handler to fail on 1.27.
I just upgraded from 1.27 to 1.28 and running into this problem. Some kubectl apply operations fail, and ArgoCD cannot load its custom resources. Any suggested workaround? Would removing the from the prometheus configuration resolve it? Or perhaps disable metrics-server?
Simply looking at the code diff between 1.27 and 1.28 I can't pinpoint the cause.


Question: } // Ensure that the if the APIService OpenAPI is updated, the // aggregated OpenAPI is also updated. <ins> expectPath(t, swagger, "/apis/apiservicegroup/v1/path2") expectNoPath(t, swagger, "/apis/apiservicegroup/v1/path1") expectPath(t, swagger, "/apis/apiregistration.k8s.io/v1") } // Tests that an APIService that registers OpenAPI will only have the OpenAPI // for its specific group version served registered. // See https://github.com/kubernetes/kubernetes/pull/123570 for full context. func TestAPIServiceOpenAPIServiceMismatch(t *testing.T) { mux := http.NewServeMux() var delegationHandlers []http.Handler delegate1 := &openAPIHandler{openapi: &spec.Swagger{ SwaggerProps: spec.SwaggerProps{ Paths: &spec.Paths{ Paths: map[string]spec.PathItem{ "/apis/foo/v1": {}, }, }, }, }} delegationHandlers = append(delegationHandlers, delegate1) s := buildAndRegisterSpecAggregator(delegationHandlers, mux) apiService := &v1.APIService{ Spec: v1.APIServiceSpec{ Group:   "apiservicegroup", Version: "v1", Service: &v1.ServiceReference{Name: "dummy"}, }, } apiService.Name = "apiservice" apiService2 := &v1.APIService{ Spec: v1.APIServiceSpec{ Group:   "apiservicegroup", Version: "v2", Service: &v1.ServiceReference{Name: "dummy2"}, }, } apiService2.Name = "apiservice2" handler := &openAPIHandler{openapi: &spec.Swagger{ SwaggerProps: spec.SwaggerProps{ Paths: &spec.Paths{ Paths: map[string]spec.PathItem{ "/apis/apiservicegroup/v1": {}, }, }, }, }} handler2 := &openAPIHandler{openapi: &spec.Swagger{ SwaggerProps: spec.SwaggerProps{ Paths: &spec.Paths{ Paths: map[string]spec.PathItem{ "/apis/a":                  {}, "/apis/apiservicegroup/v1": {}, "/apis/apiservicegroup/v2": {}, }, }, }, }} if err := s.AddUpdateAPIService(apiService, handler); err != nil { t.Error(err) } if err := s.UpdateAPIServiceSpec(apiService.Name); err != nil { t.Error(err) } if err := s.AddUpdateAPIService(apiService2, handler2); err != nil { t.Error(err) } if err := s.UpdateAPIServiceSpec(apiService2.Name); err != nil { t.Error(err) } swagger, err := fetchOpenAPI(mux) if err != nil { t.Error(err) } expectPath(t, swagger, "/apis/apiservicegroup/v1") </ins> expectPath(t, swagger, "/apis/apiservicegroup/v2") <ins> expectPath(t, swagger, "/apis/apiregistration.k8s.io/v1") t.Logf("Remove APIService %s", apiService.Name) s.RemoveAPIService(apiService.Name) swagger, err = fetchOpenAPI(mux) if err != nil { t.Error(err) } // Ensure that the if the APIService is added then removed, the OpenAPI disappears from the aggregated OpenAPI as well. </ins> expectNoPath(t, swagger, "/apis/apiservicegroup/v1") expectPath(t, swagger, "/apis/apiregistration.k8s.io/v1") }

Can I get your exact repro steps? suggests that the conflicting paths should be removed before the merge. We always call this function with
One other question is if this is reproducible by just applying a set of yaml/installations or does it require a cluster upgrade to trigger? Thanks!
hit same problem and we cannot install any other new helm plugins. maybe we need to move forward
Here are verbose repro instructions. An apiserver restart on 1.28 will induce the issue.
Hello. I'm still having some trouble reproducing although I'm not 100% sure if the configuration is correct. I'm trying to repro via kind. afaik installing via helm has prometheus adapter serving . seems to suggest that prometheus adapter is serving under . Applying the update to the configmap per the instructions didn't change the apiservice on my end. Not sure if this is my misconfiguration or intended. I've also tested via integration test that having two openapis serving the same path doesn't cause errors and one will get filtered out. A simple workaround is probably to delete the metrics-server if it is already superseded by the prometheus-adapter.
The is not set up in an resource for us, we only created an APIService with . When I go fetch the openAPI schema served up by the endpoint, it has schema information for in it if that is enabled using . If I remove then the schema fetched from the server doesn't have stuff in it, and the error is resolved. We have been keeping around because I think something uses it directly, or used to. Maybe ? That could have changed But in any case, this problem is quite severe when it happens so it would be nice to fix it as there may be others who do the same update and have to do some deep diving into the logs and all kinds of crazy diagnostics and guesswork to figure it out.
How does the get reached without an APIService? The resources would be unavailable for the apiserver to query and it'll be through something else? I see, adding resources for me doesn't populate the OpenAPI. I wonder if this could be cloud provider specific. I'm totally supportive of fixing, it's just quite difficult to reproduce on my part .
Hmm good question.


Question: } // Ensure that the if the APIService OpenAPI is updated, the // aggregated OpenAPI is also updated. <ins> expectPath(t, swagger, "/apis/apiservicegroup/v1/path2") expectNoPath(t, swagger, "/apis/apiservicegroup/v1/path1") expectPath(t, swagger, "/apis/apiregistration.k8s.io/v1") } // Tests that an APIService that registers OpenAPI will only have the OpenAPI // for its specific group version served registered. // See https://github.com/kubernetes/kubernetes/pull/123570 for full context. func TestAPIServiceOpenAPIServiceMismatch(t *testing.T) { mux := http.NewServeMux() var delegationHandlers []http.Handler delegate1 := &openAPIHandler{openapi: &spec.Swagger{ SwaggerProps: spec.SwaggerProps{ Paths: &spec.Paths{ Paths: map[string]spec.PathItem{ "/apis/foo/v1": {}, }, }, }, }} delegationHandlers = append(delegationHandlers, delegate1) s := buildAndRegisterSpecAggregator(delegationHandlers, mux) apiService := &v1.APIService{ Spec: v1.APIServiceSpec{ Group:   "apiservicegroup", Version: "v1", Service: &v1.ServiceReference{Name: "dummy"}, }, } apiService.Name = "apiservice" apiService2 := &v1.APIService{ Spec: v1.APIServiceSpec{ Group:   "apiservicegroup", Version: "v2", Service: &v1.ServiceReference{Name: "dummy2"}, }, } apiService2.Name = "apiservice2" handler := &openAPIHandler{openapi: &spec.Swagger{ SwaggerProps: spec.SwaggerProps{ Paths: &spec.Paths{ Paths: map[string]spec.PathItem{ "/apis/apiservicegroup/v1": {}, }, }, }, }} handler2 := &openAPIHandler{openapi: &spec.Swagger{ SwaggerProps: spec.SwaggerProps{ Paths: &spec.Paths{ Paths: map[string]spec.PathItem{ "/apis/a":                  {}, "/apis/apiservicegroup/v1": {}, "/apis/apiservicegroup/v2": {}, }, }, }, }} if err := s.AddUpdateAPIService(apiService, handler); err != nil { t.Error(err) } if err := s.UpdateAPIServiceSpec(apiService.Name); err != nil { t.Error(err) } if err := s.AddUpdateAPIService(apiService2, handler2); err != nil { t.Error(err) } if err := s.UpdateAPIServiceSpec(apiService2.Name); err != nil { t.Error(err) } swagger, err := fetchOpenAPI(mux) if err != nil { t.Error(err) } expectPath(t, swagger, "/apis/apiservicegroup/v1") </ins> expectPath(t, swagger, "/apis/apiservicegroup/v2") <ins> expectPath(t, swagger, "/apis/apiregistration.k8s.io/v1") t.Logf("Remove APIService %s", apiService.Name) s.RemoveAPIService(apiService.Name) swagger, err = fetchOpenAPI(mux) if err != nil { t.Error(err) } // Ensure that the if the APIService is added then removed, the OpenAPI disappears from the aggregated OpenAPI as well. </ins> expectNoPath(t, swagger, "/apis/apiservicegroup/v1") expectPath(t, swagger, "/apis/apiregistration.k8s.io/v1") }

In our staging cluster I tried bringing back the in the custom metrics server configuration and restarting all the masters, and the problem did not come back, everything seems OK now. So it maybe isn't that easy to reproduce
After updating the configmap, restarting the prometheus-adapter pod to pickup the changes and restarting the apiserver are both required to induce the issue. I have been able to deterministically reproduce this following those steps, please follow up here if you are still having issues.
That's good to hear. I'll need some more help with reproducing it. I've been able to verify that restarting the prometheus adapter pod picks up the configmap changes but restarting the apiserver hasn't caused any crashes for me. I've tested this both on the latest master branch and 1.28.
While I'm yet to fully understand these changes, the 2 PRs below (both of which went into 1.28) seem likely candidates for a regression here: - introduced lazy loading in openAPI v2 handler to load/compute the aggregator spec when requested by client - switched the spec merging function from to any clue if one of the above changes could be causing mischief?
Also from example request at the top, the impact seems to be to the v2 API: Nathan - when you get a chance, could you also confirm if v1 API doesn't run into this?
OpenAPI only supports v2 and v3.
hi, any clue about this issue?
It's possible but I haven't been able to reproduce it. Can I ask if this is happening on cloud providers outside AWS?
Here are the apiservices and relevant pod objects with a working repro from these steps If it's useful, I can provide yamls for these objects in addition to the settings applied in the prometheus-adapter configmap. Can you confirm that the pods backing your prometheus-adapter and metrics-server are running?
I meet this problem
We the problem in GCP as well.
ACK in Alibaba Cloud also meets this problem
We were able to fix the v2 availability by and re-installing the kube-prometheus helm chart. Unfortunately, after restarting the k8s api server, v2 availability is broken again.
yes, I'm running on kind. I'll do some more digging, assuming it's related to those PRs linked by but still haven't pinpointed anything.


Question: } // Ensure that the if the APIService OpenAPI is updated, the // aggregated OpenAPI is also updated. <ins> expectPath(t, swagger, "/apis/apiservicegroup/v1/path2") expectNoPath(t, swagger, "/apis/apiservicegroup/v1/path1") expectPath(t, swagger, "/apis/apiregistration.k8s.io/v1") } // Tests that an APIService that registers OpenAPI will only have the OpenAPI // for its specific group version served registered. // See https://github.com/kubernetes/kubernetes/pull/123570 for full context. func TestAPIServiceOpenAPIServiceMismatch(t *testing.T) { mux := http.NewServeMux() var delegationHandlers []http.Handler delegate1 := &openAPIHandler{openapi: &spec.Swagger{ SwaggerProps: spec.SwaggerProps{ Paths: &spec.Paths{ Paths: map[string]spec.PathItem{ "/apis/foo/v1": {}, }, }, }, }} delegationHandlers = append(delegationHandlers, delegate1) s := buildAndRegisterSpecAggregator(delegationHandlers, mux) apiService := &v1.APIService{ Spec: v1.APIServiceSpec{ Group:   "apiservicegroup", Version: "v1", Service: &v1.ServiceReference{Name: "dummy"}, }, } apiService.Name = "apiservice" apiService2 := &v1.APIService{ Spec: v1.APIServiceSpec{ Group:   "apiservicegroup", Version: "v2", Service: &v1.ServiceReference{Name: "dummy2"}, }, } apiService2.Name = "apiservice2" handler := &openAPIHandler{openapi: &spec.Swagger{ SwaggerProps: spec.SwaggerProps{ Paths: &spec.Paths{ Paths: map[string]spec.PathItem{ "/apis/apiservicegroup/v1": {}, }, }, }, }} handler2 := &openAPIHandler{openapi: &spec.Swagger{ SwaggerProps: spec.SwaggerProps{ Paths: &spec.Paths{ Paths: map[string]spec.PathItem{ "/apis/a":                  {}, "/apis/apiservicegroup/v1": {}, "/apis/apiservicegroup/v2": {}, }, }, }, }} if err := s.AddUpdateAPIService(apiService, handler); err != nil { t.Error(err) } if err := s.UpdateAPIServiceSpec(apiService.Name); err != nil { t.Error(err) } if err := s.AddUpdateAPIService(apiService2, handler2); err != nil { t.Error(err) } if err := s.UpdateAPIServiceSpec(apiService2.Name); err != nil { t.Error(err) } swagger, err := fetchOpenAPI(mux) if err != nil { t.Error(err) } expectPath(t, swagger, "/apis/apiservicegroup/v1") </ins> expectPath(t, swagger, "/apis/apiservicegroup/v2") <ins> expectPath(t, swagger, "/apis/apiregistration.k8s.io/v1") t.Logf("Remove APIService %s", apiService.Name) s.RemoveAPIService(apiService.Name) swagger, err = fetchOpenAPI(mux) if err != nil { t.Error(err) } // Ensure that the if the APIService is added then removed, the OpenAPI disappears from the aggregated OpenAPI as well. </ins> expectNoPath(t, swagger, "/apis/apiservicegroup/v1") expectPath(t, swagger, "/apis/apiregistration.k8s.io/v1") }

I tried to reproduce on my local setup with based on steps + , but I could still get a successful response when I executed . Then I reverted one of the step mentioned in to and waited until the new pod came alive and then did a apiserver restart and I was able to reproduce the same thing on as well i.e; I get the 503 response when I execute following command after the fact. After this point, even if i change it back to and restart the apiserver , the error is persistent. Let me know if this works for you to reproduce ? PS: Also, i want to call out that there is no issue with , its .
Thanks, alright I found a way to reproduce it consistently without restarting the apiserver. Still investigating the root cause. Thanks for the detailed reproduction and
Quick update: This issue was a combination of bugs both in k/k and prometheus adapter. I opened to mitigate on k/k side which should fix the issue. The issue was difficult to reproduce because it requires a very specific ordering of OpenAPI merging to trigger, which is why I suspect many people were able to reproduce after some restarts to kube-apiserver. Prometheus and metrics server serve differing OpenAPIs for the endpoint. Prometheus also doesn't seem to be registering discovery's OpenAPI properly (missing OpenAPI for but having the OpenAPI for whereas it is all present in metrics server. This inconsistency in how prometheus and metrics server handle the OpenAPI surfaced the merging issue.
a little wired question: why do we need to restart kube-apiserver to reproduce this problem?
Tried to reproduce the issue after cherry picking the fix and didn't run into below error. Fix is working as expected.
It can look like restarting prometheus-adapter "fixes" the issue? I am also curious about suggestion of removing the section from the config map works as a temporary workaround until released upstream (we use and cannot delete the apiregistration)? As I understand the fix for this issue is currently in


Question: }, }, } <del> testCount := int32(3) testStatuses := []testStatus{ {"terminate-cmd-rpa", api.RestartPolicyAlways, api.PodRunning, ContainerStateWaiting | ContainerStateRunning | ContainerStateTerminated, ">", testCount, false}, {"terminate-cmd-rpof", api.RestartPolicyOnFailure, api.PodSucceeded, ContainerStateTerminated, "==", testCount, false}, {"terminate-cmd-rpn", api.RestartPolicyNever, api.PodSucceeded, ContainerStateTerminated, "==", 0, false}, </del> <ins> testCases := []testCase{ {"terminate-cmd-rpa", api.RestartPolicyAlways, api.PodRunning, ContainerStateRunning, "==", 2, true}, {"terminate-cmd-rpof", api.RestartPolicyOnFailure, api.PodSucceeded, ContainerStateTerminated, "==", 1, false}, {"terminate-cmd-rpn", api.RestartPolicyNever, api.PodFailed, ContainerStateTerminated, "==", 0, false}, </ins> } <del> for _, testStatus := range testStatuses { </del> <ins> for _, testCase := range testCases { </ins> tmpFile, err := ioutil.TempFile("", "restartCount") Expect(err).NotTo(HaveOccurred()) defer os.Remove(tmpFile.Name()) <del> // It fails in the first three runs and succeeds after that. tmpCmd := fmt.Sprintf("echo 'hello' >> /restart-count/%s ; test $(wc -l /restart-count/%s| awk {'print $1'}) -ge %d", path.Base(tmpFile.Name()), path.Base(tmpFile.Name()), testCount+1) testContainer.Name = testStatus.Name </del> <ins> // It failed at the 1st run, then succeeded at 2nd run, then run forever cmdScripts := ` f=%s count=$(echo 'hello' >> $f ; wc -l $f | awk {'print $1'}) if [ $count -eq 1 ]; then exit 1 fi if [ $count -eq 2 ]; then exit 0 fi while true; do sleep 1; done ` tmpCmd := fmt.Sprintf(cmdScripts, path.Join(restartCountVolumePath, path.Base(tmpFile.Name()))) testContainer.Name = testCase.Name </ins> testContainer.Command = []string{"sh", "-c", tmpCmd} terminateContainer := ConformanceContainer{ Container:     testContainer, Client:        cl, <del> RestartPolicy: testStatus.RestartPolicy, </del> <ins> RestartPolicy: testCase.RestartPolicy, </ins> Volumes:       testVolumes, NodeName:      *nodeName, Namespace:     namespace, } <del> err = terminateContainer.Create() Expect(err).NotTo(HaveOccurred()) testCContainers = append(testCContainers, terminateContainer) </del> <ins> Expect(terminateContainer.Create()).To(Succeed()) defer terminateContainer.Delete() </ins> <del> Eventually(func() api.PodPhase { _, phase, _ := terminateContainer.GetStatus() return phase }, retryTimeout, pollInterval).ShouldNot(Equal(api.PodPending)) var status api.ContainerStatus </del> By("it should get the expected 'RestartCount'") <del> Eventually(func() int32 { status, _, _ = terminateContainer.GetStatus() return status.RestartCount }, retryTimeout, pollInterval).Should(BeNumerically(testStatus.RestartCountOper, testStatus.RestartCount)) </del> <ins> Eventually(func() (int32, error) { status, err := terminateContainer.GetStatus() return status.RestartCount, err }, retryTimeout, pollInterval).Should(BeNumerically(testCase.RestartCountOper, testCase.RestartCount)) </ins> <del> By("it should get the expected 'Ready' status") Expect(status.Ready).To(Equal(testStatus.Ready)) </del> <ins> By("it should get the expected 'Phase'") Eventually(terminateContainer.GetPhase, retryTimeout, pollInterval).Should(Equal(testCase.Phase)) </ins> <del> By("it should get the expected 'State'") Expect(GetContainerState(status.State) & testStatus.State).NotTo(Equal(0)) </del> <ins> By("it should get the expected 'Ready' condition") Expect(terminateContainer.IsReady()).Should(Equal(testCase.Ready)) </ins> <del> By("it should be possible to delete [Conformance]") err = terminateContainer.Delete() Expect(err).NotTo(HaveOccurred()) Eventually(func() bool { isPresent, err := terminateContainer.Present() return err == nil && !isPresent }, retryTimeout, pollInterval).Should(BeTrue()) } }) }) Context("when start a container that keeps running", func() { It("it should run with the expected status [Conformance]", func() { testContainer := api.Container{ Image:           ImageRegistry[busyBoxImage], Command:         []string{"sh", "-c", "while true; do echo hello; sleep 1; done"}, ImagePullPolicy: api.PullIfNotPresent, } testStatuses := []testStatus{ {"loop-cmd-rpa", api.RestartPolicyAlways, api.PodRunning, ContainerStateRunning, "==", 0, true}, {"loop-cmd-rpof", api.RestartPolicyOnFailure, api.PodRunning, ContainerStateRunning, "==", 0, true}, {"loop-cmd-rpn", api.RestartPolicyNever, api.PodRunning, ContainerStateRunning, "==", 0, true}, } for _, testStatus := range testStatuses { testContainer.Name = testStatus.Name runningContainer := ConformanceContainer{ Container:     testContainer, Client:        cl, RestartPolicy: testStatus.RestartPolicy, NodeName:      *nodeName, Namespace:     namespace, } err := runningContainer.Create() Expect(err).NotTo(HaveOccurred()) testCContainers = append(testCContainers, runningContainer) Eventually(func() api.PodPhase { _, phase, _ := runningContainer.GetStatus() return phase }, retryTimeout, pollInterval).Should(Equal(api.PodRunning)) var status api.ContainerStatus var phase api.PodPhase Consistently(func() api.PodPhase { status, phase, err = runningContainer.GetStatus() return phase }, consistentCheckTimeout, pollInterval).Should(Equal(testStatus.Phase)) Expect(err).NotTo(HaveOccurred()) By("it should get the expected 'RestartCount'") Expect(status.RestartCount).To(BeNumerically(testStatus.RestartCountOper, testStatus.RestartCount)) By("it should get the expected 'Ready' status") Expect(status.Ready).To(Equal(testStatus.Ready)) </del> <ins> status, err := terminateContainer.GetStatus() Expect(err).ShouldNot(HaveOccurred()) </ins> By("it should get the expected 'State'") <del> Expect(GetContainerState(status.State) & testStatus.State).NotTo(Equal(0)) </del> <ins> Expect(GetContainerState(status.State)).To(Equal(testCase.State)) </ins> By("it should be possible to delete [Conformance]") <del> err = runningContainer.Delete() Expect(err).NotTo(HaveOccurred()) Eventually(func() bool { isPresent, err := runningContainer.Present() return err == nil && !isPresent }, retryTimeout, pollInterval).Should(BeTrue()) } }) }) Context("when start a container that exits failure", func() { It("it should run with the expected status [Conformance]", func() { testContainer := api.Container{ Image:           ImageRegistry[busyBoxImage], Command:         []string{"false"}, ImagePullPolicy: api.PullIfNotPresent, } testStatuses := []testStatus{ {"fail-cmd-rpa", api.RestartPolicyAlways, api.PodRunning, ContainerStateWaiting | ContainerStateRunning | ContainerStateTerminated, ">", 0, false}, {"fail-cmd-rpof", api.RestartPolicyOnFailure, api.PodRunning, ContainerStateTerminated, ">", 0, false}, {"fail-cmd-rpn", api.RestartPolicyNever, api.PodFailed, ContainerStateTerminated, "==", 0, false}, } for _, testStatus := range testStatuses { testContainer.Name = testStatus.Name failureContainer := ConformanceContainer{ Container:     testContainer, Client:        cl, RestartPolicy: testStatus.RestartPolicy, NodeName:      *nodeName, Namespace:     namespace, } err := failureContainer.Create() Expect(err).NotTo(HaveOccurred()) testCContainers = append(testCContainers, failureContainer) Eventually(func() api.PodPhase { _, phase, _ := failureContainer.GetStatus() return phase }, retryTimeout, pollInterval).ShouldNot(Equal(api.PodPending)) var status api.ContainerStatus By("it should get the expected 'RestartCount'") Eventually(func() int32 { status, _, _ = failureContainer.GetStatus() return status.RestartCount }, retryTimeout, pollInterval).Should(BeNumerically(testStatus.RestartCountOper, testStatus.RestartCount)) By("it should get the expected 'Ready' status") Expect(status.Ready).To(Equal(testStatus.Ready)) By("it should get the expected 'State'") Expect(GetContainerState(status.State) & testStatus.State).NotTo(Equal(0)) By("it should be possible to delete [Conformance]") err = failureContainer.Delete() Expect(err).NotTo(HaveOccurred()) Eventually(func() bool { isPresent, err := failureContainer.Present() return err == nil && !isPresent }, retryTimeout, pollInterval).Should(BeTrue()) </del> <ins> Expect(terminateContainer.Delete()).To(Succeed()) Eventually(terminateContainer.Present, retryTimeout, pollInterval).Should(BeFalse()) </ins> } }) })

Slow conformance test has caused issues like We should make runtime/image conformance test run faster without breaking the test coverage, possibly by: Shorten consistently check interval . Remove redundant test cases such as ... /cc


Question: import ( "errors" "fmt" <ins> "net" </ins> "os" "path/filepath" "strconv"

Kubelet was not able to attach + mount an iSCSI PV when it has IPv6 server address. Volume is mounted + attached. Set up iSCSI target that's reachable over IPv6 and create a PV for it. For example: Kubelet logs: Notice around the IP address. Various iSCSI files in /sys and /dev/disk/by-path do not have these square brackets and kubelet is then not able to find the iSCSI host in /sys. That's where comes from. No response <detailsA recent master: console NAME="Fedora Linux" VERSION="36 (Workstation Edition)" </details<detailshack/local-up- </details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage /priority important-soon
/assign


Question: return nil } if effectiveSc.RunAsUser != nil { <del> klog.Warningf("Windows container does not support SecurityContext.RunAsUser, please use SecurityContext.WindowsOptions") </del> <ins> klog.Warningf("Windows container does not support SecurityContext.RunAsUser, please use SecurityContext.WindowsOptions (pod: %q, container: %s)", format.Pod(pod), container.Name) </ins> } if effectiveSc.SELinuxOptions != nil { <del> klog.Warningf("Windows container does not support SecurityContext.SELinuxOptions, please use SecurityContext.WindowsOptions") </del> <ins> klog.Warningf("Windows container does not support SecurityContext.SELinuxOptions, please use SecurityContext.WindowsOptions (pod: %q, container: %s)", format.Pod(pod), container.Name) </ins> } if effectiveSc.RunAsGroup != nil { <del> klog.Warningf("Windows container does not support SecurityContext.RunAsGroup") </del> <ins> klog.Warningf("Windows container does not support SecurityContext.RunAsGroup (pod: %q, container: %s)", format.Pod(pod), container.Name) </ins> } if effectiveSc.WindowsOptions != nil { if effectiveSc.WindowsOptions.RunAsUserName != nil { if *effectiveSc.WindowsOptions.RunAsUserName == windowsRootUserName { <del> return fmt.Errorf("container's runAsUser (%s) which will be regarded as root identity and will break non-root policy", username) </del> <ins> return fmt.Errorf("container's runAsUser (%s) which will be regarded as root identity and will break non-root policy (pod: %q, container: %s)", username, format.Pod(pod), container.Name) </ins> } return nil } } if len(username) > 0 && username == windowsRootUserName { <del> return fmt.Errorf("container's runAsUser (%s) which will be regarded as root identity and will break non-root policy", username) </del> <ins> return fmt.Errorf("container's runAsUser (%s) which will be regarded as root identity and will break non-root policy (pod: %q, container: %s)", username, format.Pod(pod), container.Name) </ins> } return nil }

when deploying a container that has runAsNonRoot but runs as root this error appears, but for pods with lots of containers it can be confusing which container failed, especially if there are 3-rd party webhooks that inject containers please add the container name to the message, making it
/sig api-machinery
/assign I can try this fix
I just submitted a pr, can you review it for me, is it as you expected?
thx, looks good!
/remove-sig api-machiner /sig node /area kubelet
Those labels are not set on the issue: <detailsIn response to : /remove-sig api-machiner /sig node /area kubelet Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-sig api-machinery


Question: } return key, setOutput } <ins> func TestPrefix(t *testing.T) { codec := apitesting.TestCodec(codecs, examplev1.SchemeGroupVersion) cluster := integration.NewClusterV3(t, &integration.ClusterConfig{Size: 1}) transformer := prefixTransformer{prefix: []byte("test!")} testcases := map[string]string{ "custom/prefix":     "/custom/prefix", "/custom//prefix//": "/custom/prefix", "/registry":         "/registry", } for configuredPrefix, effectivePrefix := range testcases { store := newStore(cluster.RandClient(), false, codec, configuredPrefix, transformer) if store.pathPrefix != effectivePrefix { t.Errorf("configured prefix of %s, expected effective prefix of %s, got %s", configuredPrefix, effectivePrefix, store.pathPrefix) } } } </ins>

an API server with , populate with data, stop server etcd2 to etcd3 the API server with data is missing from the API. The etcd2 client takes the configured prefix and passes it to the etcd client as . The etcd3 client simply passes in the configured prefix. is a no-op on the default prefix (`/registry), but can modify custom prefixes that did not start with a slash. cc
I think we have to preserve the behavior from the etcd2 client setup. It's a no-op for the default prefix, but requiring a change to a custom prefix when switching from etcd2 to etcd3 is really unexpected, especially when the failure mode is that the apiserver starts up as if it's a brand new cluster (including creating default projects, kubernetes service, etc)
cc
/cc re: upgrade testing.
Throwing release blocker on here just so we don't lose it (given data loss potential)
- thank for debugging and fixing it.


Question: } // IntValue returns the IntVal if type Int, or if <del> // it is a String, will attempt a conversion to int. </del> <ins> // it is a String, will attempt a conversion to int, // returning 0 if a parsing error occurs. </ins> func (intstr *IntOrString) IntValue() int { if intstr.Type == String { i, _ := strconv.Atoi(intstr.StrVal)

There is no value by that conversion, even worse it can be misused.
should we use in such case?
I do not think this is a bug. 25% == .25 and int(.25) == 0. ;) Even in the case where the user puts in something like 100% I do not think it's a bug. Users should not put strings like "25%" into intOrString and expect to get the integer 25 out. Likely a user doing that is confused about what format the field takes in. I am not sure why the function GetValueFromIntOrPercent exists, and I bet I am not going to like the answer :)
No. First, this depends on how you do the rounding. If you are rounding up (MaxSurge in Deployments), this is 1. Then, the replica size of the deployment also plays a part. 25% of a controller with 1 replica is different from a controller with 4 replicas, etc.
Also, the fact that IntValue returns 0 for every percentage has nothing to do with any logic. It's purely because the conversion errors out:
the first line there was supposed to be a joke, sorry. :) I know that the actual reason for returning 0 is the % sign means it is not an int. More seriously: I actually agree with the parsing function that percentages are not ints. Users who want to do this should not use IntOrString. Is there a place in the API that wants to do this?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/lifecycle frozen /remove-lifecycle stale
Opened to ensure all in-tree callers only call this when the type is Int (or verify it is a parseable number first), and to update the method documentation


Question: "github.com/onsi/gomega" // TODO: Remove the following imports (ref: https://github.com/kubernetes/kubernetes/issues/81245) <ins> "k8s.io/kubernetes/pkg/kubelet/util/format" </ins> e2epod "k8s.io/kubernetes/test/e2e/framework/pod" )

This issue is to discuss an e2e testing plan for Ephemeral Containers. /sig node
As a first step towards this issue, I am planning to create node-e2e tests (under ) to verify the following: With a running pod client adds a valid EphemeralContainer resource, and the ephemeral container starts from the ephemeral container, list processes of itself and other containers, and then signal other processes (if is ) from the ephemeral container, ping & curl other containers try to read/write to stdin/stdout of the ephemeral container verify resource request/limit values against calculated expectations check oom scores of processes in the ephemeral container verify volume mounts restarts according to policy, when process exits client tries to add EphemeralContainer resource with ports (should be rejected) client tries to add EphemeralContainer resource with probes (should be rejected) client tries to add EphemeralContainer resource with resources (should be rejected) client tries to add EphemeralContainer resource with securityContext (should be rejected) removes the ephemeral container, and the ephemeral container is stopped and deleted adds multiple ephemeral containers, should all startWith pod of non-running status (TBD)
Awesome, thanks! This is a great start, here are some things to look out for: Visibility should work, but I expect signaling to fail without which is not currently allowed. iirc was interested in how this would be tested for ephemeral containers. Can you point us towards any examples that would help us get started? Ephemeral containers will never restart (and the policy should always be never) These will be denied by the api server without ever making it to the kubelet, so they are better tested in the apiserver integration tests (and I think they are already). Removing ephemeral containers is not currently allowed. /assign
Validation of ephemeralContainers is well tested in Testing of removing ephemeral containers is in ,
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with .


Question: "github.com/onsi/gomega" // TODO: Remove the following imports (ref: https://github.com/kubernetes/kubernetes/issues/81245) <ins> "k8s.io/kubernetes/pkg/kubelet/util/format" </ins> e2epod "k8s.io/kubernetes/test/e2e/framework/pod" )

Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen /assign /priority important-soon
Reopened this issue. <detailsIn response to : /reopen /assign /priority important-soon Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-contributor-experience at . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Rotten issues close after 30d of inactivity.


Question: "github.com/onsi/gomega" // TODO: Remove the following imports (ref: https://github.com/kubernetes/kubernetes/issues/81245) <ins> "k8s.io/kubernetes/pkg/kubelet/util/format" </ins> e2epod "k8s.io/kubernetes/test/e2e/framework/pod" )

Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-contributor-experience at . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen /remove-lifecycle rotten
Reopened this issue. <detailsIn response to : /reopen /remove-lifecycle rotten Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: } errorCase := []struct { <del> name         string requirements v1.ResourceRequirements </del> <ins> name                  string requirements          v1.ResourceRequirements skipLimitValueCheck   bool skipRequestValueCheck bool </ins> }{ { name: "Resources with Requests Larger Than Limits",

Created limit range object with this parameters then tried to create this pod got message: Error from server (Invalid): error when creating "pod-to-be-": Pod "nginx-to-be-removed-4" is invalid: spec.containers[0].resources.requests: Invalid value: "700m": must be less than or equal to cpu limit Pod should be created with Requests & Limits of cpu equal to 700m (since it is in range of min/max cpu limitrange object) create limitrange object and pod as in description No response <details</details<detailson premise/AKS </details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
You have a default CPU limit of 10m, and no explicit CPU limit on the container, but a request larger than the limit. The request must always be less than or equal to the limit. Your pod needs to also set a higher limit, or the limit range needs to not set a default limit.
/remove-kind bug /kind support
/cc
but this is not obvious behavior since there is min & max set in limit range. This is limits (not min/max) that should be overwritten by pod. limits should be set to requests in this case imho
limit range limits what the user can set, it doesn't perform any mutations other than the defaulting and doesn't claim to. Min/max enforce that pods cannot set higher / lower. The default ensures that if the pod doesn't set a limit it will have one. personally I would remove the default limit, especially at 10m, this will aggressively limit pods. it would be, if the pod set a limit that would be a substantial breaking change, I don't think we could do that short of a v2 API and I'm not sure it would be correct behavior even then. we could make it opt-in, but then, why not just set them to match on the pod to begin with?