Question: return selector.PodSelectorMatches } <del> // NodeMatchesNodeSelectorTerms checks if a node's labels satisfy a list of node selector terms, </del> <ins> // nodeMatchesNodeSelectorTerms checks if a node's labels satisfy a list of node selector terms, </ins> // terms are ORed, and an emtpy a list of terms will match nothing. <del> func NodeMatchesNodeSelectorTerms(node *api.Node, nodeSelectorTerms []api.NodeSelectorTerm) bool { </del> <ins> func nodeMatchesNodeSelectorTerms(node *api.Node, nodeSelectorTerms []api.NodeSelectorTerm) bool { </ins> for _, req := range nodeSelectorTerms { nodeSelector, err := api.NodeSelectorRequirementsAsSelector(req.MatchExpressions) if err != nil {

I just noticed you forgot to register the image locality priority function in in I also missed it when I reviewed your PR. I have fixed this in . But it would be great if you could figure out a mechanism that would ensure that all predicate/priority functions that are tested (you can assume they will always live under ) are registered, and fail the test if not. That will allow us to prevent this mistake in the future.
Thanks! Will fix this soon!
Seems need to wait for in before I send PR? Otherwise this test will always fail ...


Question: if want == "" { want = "<nil>" } <del> if got := fmt.Sprintf("%v", err); got != want { </del> <ins> if got := fmt.Sprintf("%v", err); !strings.Contains(got, want) { </ins> t.Errorf("wrong error for message %q: want=%q, got=%q", test.message, want, got) } }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Below k8s UT fail when run with master-golang. TestCheckpointStateRestore/RestorecheckpointwithinvalidJSON error:=== RUN TestCheckpointStateRestore/RestorecheckpointwithinvalidJSON I1124 05:38:14. ] [cpumanager] initializing new in-memory state store unexpected error while creatng checkpointState: could not restore state from checkpoint: json: unexpected end of JSON input, please drain this node and delete the CPU manager checkpoint file "/tmp/cpumanagerstatetest479660663/cpumanagercheckpointtest" before restarting Kubelet --- FAIL: TestCheckpointStateRestore/RestorecheckpointwithinvalidJSON (0.00s) TestStrategicMergePatchInvalid error:=== RUN TestStrategicMergePatchInvalid expected "invalid character 'b' looking for beginning of value", got "json: invalid character 'b' looking for beginning of value" --- FAIL: TestStrategicMergePatchInvalid (0.00s) TestJSONPatch error:=== RUN TestJSONPatch invalid-syntax: expected error invalid character 'i' looking for beginning of value, but got json: invalid character 'i' looking for beginning of value --- FAIL: TestJSONPatch (0.00s) TestV4ErrorDecoder expandless error:=== RUN TestV4ErrorDecoder wrong error for message "{": want="error stream protocol error: unexpected end of JSON input in "{"", got="error stream protocol error: json: unexpected end of JSON input in "{"" --- FAIL: TestV4ErrorDecoder (0.00s What you expected to happen: The tests should pass with no error. How to reproduce it (as minimally and precisely as possible): a stable version of golang installed master go from upstream git clone cd go/src cd ../.. cp -r ./go /usr/local export PATH=/usr/local/go/bin:$PATH kubernetes repo and run the unit tests. () Anything else we need to know?: These failures are occurring due to mismatch of error messages between upstream go and the k8s tests.


Question: if want == "" { want = "<nil>" } <del> if got := fmt.Sprintf("%v", err); got != want { </del> <ins> if got := fmt.Sprintf("%v", err); !strings.Contains(got, want) { </ins> t.Errorf("wrong error for message %q: want=%q, got=%q", test.message, want, got) } }

There is an addition "json:" prefix to SyntaxError messages after this below commit: Environment: Kubernetes version (use ): latest source code from upstream Cloud provider or hardware configuration: OS (e.g: ): "CentOS Linux 8 (Core)" Kernel (e.g. ): Linux rajalakshmi-workspace 4.18.0-193.19.1.el82.ppc64le SMP Mon Sep 14 14:48:04 UTC 2020 ppc64le ppc64le ppc64le GNU/Linux Install tools: [root go]# go version go version devel + Fri Nov 20 00:54:43 2020 +0000 linux/ppc64le [root go]# Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig testing
will fix it in Go 1.16 as release-blocker.
I would fix the test here to be more tolerant. exact string matches on error messages aren't really an API… I would not expect that to block the golang release
Agree too, we can think about having a substring match to avoid these issues instead!
/assign Lets see if we can fix this in the k8s tests


Question: apierrors "k8s.io/apimachinery/pkg/api/errors" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/util/intstr" <ins> "k8s.io/apimachinery/pkg/util/uuid" </ins> "k8s.io/apimachinery/pkg/util/wait" "k8s.io/kubernetes/test/e2e/framework" e2epod "k8s.io/kubernetes/test/e2e/framework/pod" imageutils "k8s.io/kubernetes/test/utils/image" "github.com/onsi/ginkgo" <ins> "k8s.io/apimachinery/pkg/types" </ins> ) func extinguish(f *framework.Framework, totalNS int, maxAllowedAfterDel int, maxSeconds int) {

This issue proposes the writing of a new test for the API endpoint with the intention of promoting it to Conformance. This issue contains a very masic mock test in order to start the conversation about what a good e2e test would look like for this endpoint. The code below is verified to be hitting the intended endpoint, show as per the queries in APIsnoop's live view of the cluster's audit logs. According to this query, there is only one API endpoint which is for Namespace that isn't being covered. We selected patchCoreV1Namespace to write a test for. Dependencies: The mock test: If a test with these calls is build and merged, conformance will go up by 1 endpoint.
/sig testing /sig architecture /area conformance


Question: cloud: gce EOF <ins> DOCKER_OPTS="" if [[ -n "${EXTRA_DOCKER_OPTS-}" ]]; then DOCKER_OPTS="${EXTRA_DOCKER_OPTS}" fi </ins> # Decide if enable the cache <del> if [[ "${ENABLE_DOCKER_REGISTRY_CACHE}" == "true" ]]; then </del> <ins> if [[ "${ENABLE_DOCKER_REGISTRY_CACHE}" == "true" ]]; then </ins> REGION=$(echo "${ZONE}" | cut -f 1,2 -d -) echo "Enable docker registry cache at region: " $REGION <del> DOCKER_OPTS="--registry-mirror="https://${REGION}.docker-cache.clustermaster.net"" </del> <ins> DOCKER_OPTS="${DOCKER_OPTS} --registry-mirror='https://${REGION}.docker-cache.clustermaster.net'" fi </ins> <del> cat <<EOF >>/etc/salt/minion.d/grains.conf </del> <ins> if [[ -n "{DOCKER_OPTS}" ]]; then cat <<EOF >>/etc/salt/minion.d/grains.conf </ins> docker_opts: $DOCKER_OPTS EOF fi

0/8 is with a company's intranet, so it can be insecure. This is similiar to "For GCE, allow insecure registries anywhere in 10.0.0.0/8" Thanks!
Linking to and .
This was


Question: } func TestValidateNamespaceStatusUpdate(t *testing.T) { <ins> now := util.Now() </ins> tests := []struct { oldNamespace api.Namespace namespace    api.Namespace valid        bool }{ <del> {api.Namespace{}, api.Namespace{}, true}, </del> <ins> {api.Namespace{}, api.Namespace{ Status: api.NamespaceStatus{ Phase: api.NamespaceActive, }, }, true}, </ins> {api.Namespace{ ObjectMeta: api.ObjectMeta{ Name: "foo"}}, api.Namespace{ ObjectMeta: api.ObjectMeta{ <del> Name: "foo"}, </del> <ins> Name:              "foo", DeletionTimestamp: &now}, </ins> Status: api.NamespaceStatus{ Phase: api.NamespaceTerminating, },

There is a funny subtle behvaiour in get namespaces, in which if i call more than one time, it seems that the removal from Terminating state is accelarated. It seems this might indicate a bug, or maybe its not a bug and the eventual semantics of is a little funny sometimes. in any case, here's what we are seeing : 1) when i delete a namespace the first time usning kubectl, we get a state change to . that is correct, of course. 2) but ... if make the call a second time, then the same status line is immediately removed from . Expected behaviour : A second call to "kubectl delete ..." shouldn't have any effect on the timing of removal of a namespace from 'Terminating' state*... So i guess there is some "eventual consistency" subtle bug that is resolved by repeated calls to
suggests this may not be e bug, maybe its just the etcd eventual deleter thingy.
Looks to definitely be a bug. The 2nd delete seems to have interrupted the delete process and artifacts (like events) from the namespace still exist in etcd.
investigating...
oops, see the error, will submit a fix.
I have a fix, but need a test case, will submit PR tomorrow AM


Question: nodePorts[key] = true } <ins> // Check for duplicate TargetPort portsPath = specPath.Child("ports") targetPorts := make(map[api.ServicePort]bool) for i, port := range service.Spec.Ports { if (port.TargetPort.Type == intstr.Int && port.TargetPort.IntVal == 0) || (port.TargetPort.Type == intstr.String && port.TargetPort.StrVal == "") { continue } portPath := portsPath.Index(i) key := api.ServicePort{Protocol: port.Protocol, TargetPort: port.TargetPort} _, found := targetPorts[key] if found { allErrs = append(allErrs, field.Duplicate(portPath.Child("targetPort"), port.TargetPort)) } targetPorts[key] = true } </ins> // Validate SourceRange field and annotation _, ok := service.Annotations[api.AnnotationLoadBalancerSourceRangesKey] if len(service.Spec.LoadBalancerSourceRanges) > 0 || ok {

It is possible to express a Service where multiple front-side ports target the same back-side port. This is not very useful and may cause trouble for implementations of Services. `
just for the purpose of clarity, are you requesting that duplicate targetPorts should be rejected or the opposite? PS: I am new to Kubernetes so please pardon the dumb question :)
I think that should probably be rejected at validation unless we have a good reason WHY we need that.
Aye aye, thanks for the clarificiation!
/assign
GitHub didn't allow me to assign the following users: xiangpengzhao. Note that only can be assigned. <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
what problems did this cause with services? This change broke backwards compatibility for services that depended on this feature. For example, lets take the load balancer service The load balancer here terminates SSL so the backend is always port 80; however, you want to make sure you support both protocols so having the same targetPort here is a valid use case.
This is an interesting (ab)use of Service. Not really what I had in mind - the LB in a Service is generally assumed to be L4. I'll try to dredge up the issues that this was causing.
FYI, the does non layer4 things as well.
One issue is that this REQUIRES connection tracking when we otherwise might not need it. Consider two ServicePorts: my-service:80 mys-service:443 Let's say we map both services to a backend on port 8080, but we pass-through the VIP, so the pod sees packets as dest = my-service:8080. We MUST conntrack this VIP because we don't know whether to map the src port to 80 or 443. If service backends have unique TargetPorts, then we wouldn't have this issue. Port rewrites are at worst stateless, and at best can engage direct-return mode (no rewrites). It's worth noting that the PR which fixes this, which is proposed for revert, doesn't quite go far enough. It prevents a Service from using a given number twice and from using a given name twice, but not from using a number and a name that resolves to that number.


Question: nodePorts[key] = true } <ins> // Check for duplicate TargetPort portsPath = specPath.Child("ports") targetPorts := make(map[api.ServicePort]bool) for i, port := range service.Spec.Ports { if (port.TargetPort.Type == intstr.Int && port.TargetPort.IntVal == 0) || (port.TargetPort.Type == intstr.String && port.TargetPort.StrVal == "") { continue } portPath := portsPath.Index(i) key := api.ServicePort{Protocol: port.Protocol, TargetPort: port.TargetPort} _, found := targetPorts[key] if found { allErrs = append(allErrs, field.Duplicate(portPath.Child("targetPort"), port.TargetPort)) } targetPorts[key] = true } </ins> // Validate SourceRange field and annotation _, ok := service.Annotations[api.AnnotationLoadBalancerSourceRangesKey] if len(service.Spec.LoadBalancerSourceRanges) > 0 || ok {

That is a much trickier (impossible to do statically) proposition. :( Kind of a mess for getting high-perf Service networking.
was this an issue that came up during performance testing or IRL?
Anyone has a workaround to deal with this? I think there is a lot of people using AWS ELBs on this way
if your use-case is Ingress, then you can set . This should avoid generating the invalid Service spec. Obviously, you will lose the redirect behavior.
This is not an abuse of Service, but a quite common and wide-adopted pattern, especially in the AWS world.
you can't re-open an issue/PR unless you authored it or you are assigned to it. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
you can't re-open an issue/PR unless you authored it or you are assigned to it. <detailsIn response to : /assign /reopen for further discussion Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen for further discussion.
I suspect we should probably revert the original PR. While I understand it might not have been the intention for Services, if I understand this correctly then it's a breaking change to a v1 API. Can we at least point to any documentation that indicates this was a code bug rather than a supported behavior?
Sentiment on today's sig call was that we should revert the original commit, since it broke lots of users, and then revisit to see if there's another solution.
FYI: the revert PR is
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with .


Question: nodePorts[key] = true } <ins> // Check for duplicate TargetPort portsPath = specPath.Child("ports") targetPorts := make(map[api.ServicePort]bool) for i, port := range service.Spec.Ports { if (port.TargetPort.Type == intstr.Int && port.TargetPort.IntVal == 0) || (port.TargetPort.Type == intstr.String && port.TargetPort.StrVal == "") { continue } portPath := portsPath.Index(i) key := api.ServicePort{Protocol: port.Protocol, TargetPort: port.TargetPort} _, found := targetPorts[key] if found { allErrs = append(allErrs, field.Duplicate(portPath.Child("targetPort"), port.TargetPort)) } targetPorts[key] = true } </ins> // Validate SourceRange field and annotation _, ok := service.Annotations[api.AnnotationLoadBalancerSourceRangesKey] if len(service.Spec.LoadBalancerSourceRanges) > 0 || ok {

Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close


Question: "name":        "The name of this port. All ports in an EndpointSlice must have a unique name. If the EndpointSlice is dervied from a Kubernetes service, this corresponds to the Service.ports[].name. Name must either be an empty string or pass DNS_LABEL validation: * must be no more than 63 characters long. * must consist of lower case alphanumeric characters or '-'. * must start and end with an alphanumeric character. Default is empty string.", "protocol":    "The IP protocol for this port. Must be UDP, TCP, or SCTP. Default is TCP.", "port":        "The port number of the endpoint. If this is not specified, ports are not restricted and must be interpreted in the context of the specific consumer.", <del> "appProtocol": "The application protocol for this port. This field follows standard Kubernetes label syntax. Un-prefixed names are reserved for IANA standard service names (as per RFC-6335 and http://www.iana.org/assignments/service-names). Non-standard protocols should use prefixed names such as mycompany.com/my-custom-protocol.", </del> <ins> "appProtocol": "The application protocol for this port. This field follows standard Kubernetes label syntax. Un-prefixed names are reserved for IANA standard service names (as per RFC-6335 and https://www.iana.org/assignments/service-names). Non-standard protocols should use prefixed names such as mycompany.com/my-custom-protocol.", </ins> } func (EndpointPort) SwaggerDoc() map[string]string {

We should make the change in PR API reference links to should use HTTPS, not HTTP Visit h and look at the IANA hyperlinks Once this fix is made, a PR to is needed to update the docs (by autogenerating them). This typically would be deferred until the next Kubernetes release. v1.23 not applicable not applicable not applicable not applicable not applicable
/sig docs
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I opened this issue so that someone else can open a new PR that is equivalent to .
/retitle Update links to IANA for Service Name and Transport Protocol Port Number Registry
I would like to work on this issue. May I assign it to myself?
Anyone - including you - is welcome to work on this issue! So long as you've you can open a pull request to propose a fix.
Thanks
/assign
I have made the PR. I hope this works. If not then will be happy to do the necessary changes.
Had to close the previous PR because of issues, here is the new one I made.


Question: "reflect" "sort" "testing" <ins> "time" </ins> "github.com/stretchr/testify/assert" "github.com/stretchr/testify/require"

What happened: One pod stuck at CreateContainerError and can't get recovered automatically. Kubelet log shows that the requested name (let's say Container2) is already in use. shows the Container2 Running, and it's created before Container1 This should never happen. How to reproduce it (as minimally and precisely as possible): I wish i knew. Anything else we need to know?: I know this can be caused by time shifting, but I didn't find any time shifting sign in system log and chrony log, and there is no rebooting between the create timestamp of Container1 and Container2, so this should be caused by something else. This happens intermittently in our env, like 3 times in half a year, and I didn't find any ime shifting sign each time. Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.6", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-11-13T11:20:18Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.6", GitCommit:"", GitTreeState:"clean", BuildDate:"2020-08-28T11:18:26Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: OS (e.g: ): CentOS Linux release 7.5.1804 (Core) Kernel (e.g. ): Linux node-1 3.10.0-957.21.3.el7.x8664 SMP Wed Jul 17 06:57:50 UTC 2019 x8664 x8664 x86_64 GNU/Linux Install tools: kubeadm Network plugin and version (if this is a network-related bug): Others:
/sig node
Any update on this issue, we are also facing the same issue. EKS version : 1.14.9 Docker version: 19.03.6-ce AWS CNI: 1.6.2 Intermittently we are also facing this , what might the issue, could please guide us here? Below are the kubelet logs. )"


Question: "reflect" "sort" "testing" <ins> "time" </ins> "github.com/stretchr/testify/assert" "github.com/stretchr/testify/require"

with CreatePodSandboxError: "CreatePodSandbox for pod "-f0fe-11ea-9872-)" failed: rpc error: code = Unknown desc = failed to create a sandbox for pod "aiqx-server-dep--b9lmv": operation timeout: context deadline exceeded" Sep 07 11:40:40 kubelet[2899]: W0907 11:40:40. 2899 ] No ref for container "docker://7b9c86b529530f15d941f8e68973a202c8f348ab5db864a8371e8c82ddbf9dc4" (r(-e845-11ea-bd96-) Sep 07 11:40:43 kubelet[2899]: W0907 11:40:43. 2899 ] Unable to create pod sandbox due to conflict. Attempting to remove sandbox "a73bd9a03d56bc7855e4e89e163e13c5e7173397f10edcc200fb686e1b6885c8" Sep 07 11:40:43 kubelet[2899]: E0907 11:40:43. 2899 ] Failed to remove the conflicting container "a73bd9a03d56bc7855e4e89e163e13c5e7173397f10edcc200fb686e1b6885c8": Error: No such container: a73bd9a03d56bc7855e4e89e163e13c5e7173397f10edcc200fb686e1b6885c8
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or .


Question: "reflect" "sort" "testing" <ins> "time" </ins> "github.com/stretchr/testify/assert" "github.com/stretchr/testify/require"

/lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-contributor-experience at . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: <del> { "apiVersion": "apidiscovery.k8s.io/v2beta1", "items": [ { "metadata": { "creationTimestamp": null, "name": "apiregistration.k8s.io" }, "versions": [ { "freshness": "Current", "resources": [ { "categories": [ "api-extensions" ], "resource": "apiservices", "responseKind": { "group": "", "kind": "APIService", "version": "" }, "scope": "Cluster", "singularResource": "apiservice", "subresources": [ { "responseKind": { "group": "", "kind": "APIService", "version": "" }, "subresource": "status", "verbs": [ "get", "patch", "update" ] } ], "verbs": [ "create", "delete", "deletecollection", "get", "list", "patch", "update", "watch" ] } ], "version": "v1" } ] }, { "metadata": { "creationTimestamp": null, "name": "apps" }, "versions": [ { "freshness": "Current", "resources": [ { "resource": "controllerrevisions", "responseKind": { "group": "", "kind": "ControllerRevision", "version": "" }, "scope": "Namespaced", "singularResource": "controllerrevision", "verbs": [ "create", "delete", "deletecollection", "get", "list", "patch", "update", "watch" ] }, { "categories": [ "all" ], "resource": "daemonsets", "responseKind": { "group": "", "kind": "DaemonSet", "version": "" }, "scope": "Namespaced", "shortNames": [ "ds" ], "singularResource": "daemonset", "subresources": [ { "responseKind": { "group": "", "kind": "DaemonSet", "version": "" }, "subresource": "status", "verbs": [ "get", "patch", "update" ] } ], "verbs": [ "create", "delete", "deletecollection", "get", "list", "patch", "update", "watch" ] }, { "categories": [ "all" ], "resource": "deployments", "responseKind": { "group": "", "kind": "Deployment", "version": "" }, "scope": "Namespaced", "shortNames": [ "deploy" ], "singularResource": "deployment", "subresources": [ { "responseKind": { "group": "autoscaling", "kind": "Scale", "version": "v1" }, "subresource": "scale", "verbs": [ "get", "patch", "update" ] }, { "responseKind": { "group": "", "kind": "Deployment", "version": "" }, "subresource": "status", "verbs": [ "get", "patch", "update" ] } ], "verbs": [ "create", "delete", "deletecollection", "get", "list", "patch", "update", "watch" ] }, { "categories": [ "all" ], "resource": "replicasets", "responseKind": { "group": "", "kind": "ReplicaSet", "version": "" }, "scope": "Namespaced", "shortNames": [ "rs" ], "singularResource": "replicaset", "subresources": [ { "responseKind": { "group": "autoscaling", "kind": "Scale", "version": "v1" }, "subresource": "scale", "verbs": [ "get", "patch", "update" ] }, { "responseKind": { "group": "", "kind": "ReplicaSet", "version": "" }, "subresource": "status", "verbs": [ "get", "patch", "update" ] } ], "verbs": [ "create", "delete", "deletecollection", "get", "list", "patch", "update", "watch" ] }, { "categories": [ "all" ], "resource": "statefulsets", "responseKind": { "group": "", "kind": "StatefulSet", "version": "" }, "scope": "Namespaced", "shortNames": [ "sts" ], "singularResource": "statefulset", "subresources": [ { "responseKind": { "group": "autoscaling", "kind": "Scale", "version": "v1" }, "subresource": "scale", "verbs": [ "get", "patch", "update" ] }, { "responseKind": { "group": "", "kind": "StatefulSet", "version": "" }, "subresource": "status", "verbs": [ "get", "patch", "update" ] } ], "verbs": [ "create", "delete", "deletecollection", "get", "list", "patch", "update", "watch" ] } ], "version": "v1" } ] }, { "metadata": { "creationTimestamp": null, "name": "events.k8s.io" }, "versions": [ { "freshness": "Current", "resources": [ { "resource": "events", "responseKind": { "group": "", "kind": "Event", "version": "" }, "scope": "Namespaced", "shortNames": [ "ev" ], "singularResource": "event", "verbs": [ "create", "delete", "deletecollection", "get", "list", "patch", "update", "watch" ] } ], "version": "v1" } ] }, { "metadata": { "creationTimestamp": null, "name": "authentication.k8s.io" }, "versions": [ { "freshness": "Current", "resources": [ { "resource": "selfsubjectreviews", "responseKind": { "group": "", "kind": "SelfSubjectReview", "version": "" }, "scope": "Cluster", "singularResource": "selfsubjectreview", "verbs": [ "create" ] }, { "resource": "tokenreviews", "responseKind": { "group": "", "kind": "TokenReview", "version": "" }, "scope": "Cluster", "singularResource": "tokenreview", "verbs": [ "create" ] } ], "version": "v1" }, { "freshness": "Current", "resources": [ { "resource": "selfsubjectreviews", "responseKind": { "group": "", "kind": "SelfSubjectReview", "version": "" }, "scope": "Cluster", "singularResource": "selfsubjectreview", "verbs": [ "create" ] } ], "version": "v1beta1" }, { "freshness": "Current", "resources": [ { "resource": "selfsubjectreviews", "responseKind": { "group": "", "kind": "SelfSubjectReview", "version": "" }, "scope": "Cluster", "singularResource": "selfsubjectreview", "verbs": [ "create" ] } ], "version": "v1alpha1" } ] }, { "metadata": { "creationTimestamp": null, "name": "authorization.k8s.io" }, "versions": [ { "freshness": "Current", "resources": [ { "resource": "localsubjectaccessreviews", "responseKind": { "group": "", "kind": "LocalSubjectAccessReview", "version": "" }, "scope": "Namespaced", "singularResource": "localsubjectaccessreview", "verbs": [ "create" ] }, { "resource": "selfsubjectaccessreviews", "responseKind": { "group": "", "kind": "SelfSubjectAccessReview", "version": "" }, "scope": "Cluster", "singularResource": "selfsubjectaccessreview", "verbs": [ "create" ] }, { "resource": "selfsubjectrulesreviews", "responseKind": { "group": "", "kind": "SelfSubjectRulesReview", "version": "" }, "scope": "Cluster", "singularResource": "selfsubjectrulesreview", "verbs": [ "create" ] }, { "resource": "subjectaccessreviews", "responseKind": { "group": "", "kind": "SubjectAccessReview", "version": "" }, "scope": "Cluster", "singularResource": "subjectaccessreview", "verbs": [ "create" ] } ], "version": "v1" } ] }, { "metadata": { "creationTimestamp": null, "name": "autoscaling" }, "versions": [ { "freshness": "Current", "resources": [ { "categories": [ "all" ], "resource": "horizontalpodautoscalers", "responseKind": { "group": "", "kind": "HorizontalPodAutoscaler", "version": "" }, "scope": "Namespaced", "shortNames": [ "hpa" ], "singularResource": "horizontalpodautoscaler", "subresources": [ { "responseKind": { "group": "", "kind": "HorizontalPodAutoscaler", "version": "" }, "subresource": "status", "verbs": [ "get", "patch", "update" ] } ], "verbs": [ "create", "delete", "deletecollection", "get", "list", "patch", "update", "watch" ] } ], "version": "v2" }, { "freshness": "Current", "resources": [ { "categories": [ "all" ], "resource": "horizontalpodautoscalers", "responseKind": { "group": "", "kind": "HorizontalPodAutoscaler", "version": "" }, "scope": "Namespaced", "shortNames": [ "hpa" ], "singularResource": "horizontalpodautoscaler", "subresources": [ { "responseKind": { "group": "", "kind": "HorizontalPodAutoscaler", "version": "" }, "subresource": "status", "verbs": [ "get", "patch", "update" ] } ], "verbs": [ "create", "delete", "deletecollection", "get", "list", "patch", "update", "watch" ] } ], "version": "v1" } ] }, { "metadata": { "creationTimestamp": null, "name": "batch" }, "versions": [ { "freshness": "Current", "resources": [ { "categories": [ "all" ], "resource": "cronjobs", "responseKind": { "group": "", "kind": "CronJob", "version": "" }, "scope": "Namespaced", "shortNames": [ "cj" ], "singularResource": "cronjob", "subresources": [ { "responseKind": { "group": "", "kind": "CronJob", "version": "" }, "subresource": "status", "verbs": [ "get", "patch", "update" ] } ], "verbs": [ "create", "delete", "deletecollection", "get", "list", "patch", "update", "watch" ] }, { "categories": [ "all" ], "resource": "jobs", "responseKind": { "group": "", "kind": "Job", "version": "" }, "scope": "Namespaced", "singularResource": "job", "subresources": [ { "responseKind": { "group": "", "kind": "Job", "version": "" }, "subresource": "status", "verbs": [ "get", "patch", "update" ] } ], "verbs": [ "create", "delete", "deletecollection", "get", "list", "patch", "update", "watch" ] } ], "version": "v1" } ] }, { "metadata": { "creationTimestamp": null, "name": "certificates.k8s.io" }, "versions": [ { "freshness": "Current", "resources": [ { "resource": "certificatesigningrequests", "responseKind": { "group": "", "kind": "CertificateSigningRequest", "version": "" }, "scope": "Cluster", "shortNames": [ "csr" ], "singularResource": "certificatesigningrequest", "subresources": [ { "responseKind": { "group": "", "kind": "CertificateSigningRequest", "version": "" }, "subresource": "approval", "verbs": [ "get", "patch", "update" ] }, { "responseKind": { "group": "", "kind": "CertificateSigningRequest", "version": "" }, "subresource": "status", "verbs": [ "get", "patch", "update" ] } ], "verbs": [ "create", "delete", "deletecollection", "get", "list", "patch", "update", "watch" ] } ], "version": "v1" }, { "freshness": "Current", "resources": [ { "resource": "clustertrustbundles", "responseKind": { "group": "", "kind": "ClusterTrustBundle", "version": "" }, "scope": "Cluster", "singularResource": "clustertrustbundle", "verbs": [ "create", "delete", "deletecollection", "get", "list", "patch", "update", "watch" ] } ], "version": "v1alpha1" } ] }, { "metadata": { "creationTimestamp": null, "name": "networking.k8s.io" }, "versions": [ { "freshness": "Current", "resources": [ { "resource": "ingressclasses", "responseKind": { "group": "", "kind": "IngressClass", "version": "" }, "scope": "Cluster", "singularResource": "ingressclass", "verbs": [ "create", "delete", "deletecollection", "get", "list", "patch", "update", "watch" ] }, { "resource": "ingresses", "responseKind": { "group": "", "kind": "Ingress", "version": "" }, "scope": "Namespaced", "shortNames": [ "ing" ], "singularResource": "ingress", "subresources": [ { "responseKind": { "group": "", "kind": "Ingress", "version": "" }, "subresource": "status", "verbs": [ "get", "patch", "update" ] } ], "verbs": [ "create", "delete", "deletecollection", "get", "list", "patch", "update", "watch" ] }, { "resource": "networkpolicies", "responseKind": { "group": "", "kind": "NetworkPolicy", "version": "" }, "scope": "Namespaced", "shortNames": [ "netpol" ], "singularResource": "networkpolicy", "verbs": [ "create", "delete", "deletecollection", "get", "list", "patch", "update", "watch" ] } ], "version": "v1" }, { "freshness": "Current", "resources": [ { "resource": "ipaddresses", "responseKind": { "group": "", "kind": "IPAddress", "version": "" }, "scope": "Cluster", "shortNames": [ "ip" ], "singularResource": "ipaddress", "verbs": [ "create", "delete", "deletecollection", "get", "list", "patch", "update", "watch" ] }, { "resource": "servicecidrs", "responseKind": { "group": "", "kind": "ServiceCIDR", "version": "" }, "scope": "Cluster", "singularResource": "servicecidr", "subresources": [ { "responseKind": { "group": "", "kind": "ServiceCIDR", "version": "" }, "subresource": "status", "verbs": [ "get", "patch", "update" ] } ], "verbs": [ "create", "delete", "deletecollection", "get", "list", "patch", "update", "watch" ] } ], "version": "v1beta1" } ] }, { "metadata": { "creationTimestamp": null, "name": "policy" }, "versions": [ { "freshness": "Current", "resources": [ { "resource": "poddisruptionbudgets", "responseKind": { "group": "", "kind": "PodDisruptionBudget", "version": "" }, "scope": "Namespaced", "shortNames": [ "pdb" ], "singularResource": "poddisruptionbudget", "subresources": [ { "responseKind": { "group": "", "kind": "PodDisruptionBudget", "version": "" }, "subresource": "status", "verbs": [ "get", "patch", "update" ] } ], "verbs": [ "create", "delete", "deletecollection", "get", "list", "patch", "update", "watch" ] } ], "version": "v1" } ] }, { "metadata": { "creationTimestamp": null, "name": "rbac.authorization.k8s.io" }, "versions": [ { "freshness": "Current", "resources": [ { "resource": "clusterrolebindings", "responseKind": { "group": "", "kind": "ClusterRoleBinding", "version": "" }, "scope": "Cluster", "singularResource": "clusterrolebinding", "verbs": [ "create", "delete", "deletecollection", "get", "list", "patch", "update", "watch" ] }, { "resource": "clusterroles", "responseKind": { "group": "", "kind": "ClusterRole", "version": "" }, "scope": "Cluster", "singularResource": "clusterrole", "verbs": [ "create", "delete", "deletecollection", "get", "list", "patch", "update", "watch" ] }, { "resource": "rolebindings", "responseKind": { "group": "", "kind": "RoleBinding", "version": "" }, "scope": "Namespaced", "singularResource": "rolebinding", "verbs": [ "create", "delete", "deletecollection", "get", "list", "patch", "update", "watch" ] }, { "resource": "roles", "responseKind": { "group": "", "kind": "Role", "version": "" }, "scope": "Namespaced", "singularResource": "role", "verbs": [ "create", "delete", "deletecollection", "get", "list", "patch", "update", "watch" ] } ], "version": "v1" } ] }, { "metadata": { "creationTimestamp": null, "name": "storage.k8s.io" }, "versions": [ { "freshness": "Current", "resources": [ { "resource": "csidrivers", "responseKind": { "group": "", "kind": "CSIDriver", "version": "" }, "scope": "Cluster", "singularResource": "csidriver", "verbs": [ "create", "delete", "deletecollection", "get", "list", "patch", "update", "watch" ] }, { "resource": "csinodes", "responseKind": { "group": "", "kind": "CSINode", "version": "" }, "scope": "Cluster", "singularResource": "csinode", "verbs": [ "create", "delete", "deletecollection", "get", "list", "patch", "update", "watch" ] }, { "resource": "csistoragecapacities", "responseKind": { "group": "", "kind": "CSIStorageCapacity", "version": "" }, "scope": "Namespaced", "singularResource": "csistoragecapacity", "verbs": [ "create", "delete", "deletecollection", "get", "list", "patch", "update", "watch" ] }, { "resource": "storageclasses", "responseKind": { "group": "", "kind": "StorageClass", "version": "" }, "scope": "Cluster", "shortNames": [ "sc" ], "singularResource": "storageclass", "verbs": [ "create", "delete", "deletecollection", "get", "list", "patch", "update", "watch" ] }, { "resource": "volumeattachments", "responseKind": { "group": "", "kind": "VolumeAttachment", "version": "" }, "scope": "Cluster", "singularResource": "volumeattachment", "subresources": [ { "responseKind": { "group": "", "kind": "VolumeAttachment", "version": "" }, "subresource": "status", "verbs": [ "get", "patch", "update" ] } ], "verbs": [ "create", "delete", "deletecollection", "get", "list", "patch", "update", "watch" ] } ], "version": "v1" }, { "freshness": "Current", "resources": [ { "resource": "volumeattributesclasses", "responseKind": { "group": "", "kind": "VolumeAttributesClass", "version": "" }, "scope": "Cluster", "shortNames": [ "vac" ], "singularResource": "volumeattributesclass", "verbs": [ "create", "delete", "deletecollection", "get", "list", "patch", "update", "watch" ] } ], "version": "v1beta1" }, { "freshness": "Current", "resources": [ { "resource": "volumeattributesclasses", "responseKind": { "group": "", "kind": "VolumeAttributesClass", "version": "" }, "scope": "Cluster", "shortNames": [ "vac" ], "singularResource": "volumeattributesclass", "verbs": [ "create", "delete", "deletecollection", "get", "list", "patch", "update", "watch" ] } ], "version": "v1alpha1" } ] }, { "metadata": { "creationTimestamp": null, "name": "admissionregistration.k8s.io" }, "versions": [ { "freshness": "Current", "resources": [ { "categories": [ "api-extensions" ], "resource": "mutatingwebhookconfigurations", "responseKind": { "group": "", "kind": "MutatingWebhookConfiguration", "version": "" }, "scope": "Cluster", "singularResource": "mutatingwebhookconfiguration", "verbs": [ "create", "delete", "deletecollection", "get", "list", "patch", "update", "watch" ] }, { "categories": [ "api-extensions" ], "resource": "validatingadmissionpolicies", "responseKind": { "group": "", "kind": "ValidatingAdmissionPolicy", "version": "" }, "scope": "Cluster", "singularResource": "validatingadmissionpolicy", "subresources": [ { "responseKind": { "group": "", "kind": "ValidatingAdmissionPolicy", "version": "" }, "subresource": "status", "verbs": [ "get", "patch", "update" ] } ], "verbs": [ "create", "delete", "deletecollection", "get", "list", "patch", "update", "watch" ] }, { "categories": [ "api-extensions" ], "resource": "validatingadmissionpolicybindings", "responseKind": { "group": "", "kind": "ValidatingAdmissionPolicyBinding", "version": "" }, "scope": "Cluster", "singularResource": "validatingadmissionpolicybinding", "verbs": [ "create", "delete", "deletecollection", "get", "list", "patch", "update", "watch" ] }, { "categories": [ "api-extensions" ], "resource": "validatingwebhookconfigurations", "responseKind": { "group": "", "kind": "ValidatingWebhookConfiguration", "version": "" }, "scope": "Cluster", "singularResource": "validatingwebhookconfiguration", "verbs": [ "create", "delete", "deletecollection", "get", "list", "patch", "update", "watch" ] } ], "version": "v1" }, { "freshness": "Current", "resources": [ { "categories": [ "api-extensions" ], "resource": "validatingadmissionpolicies", "responseKind": { "group": "", "kind": "ValidatingAdmissionPolicy", "version": "" }, "scope": "Cluster", "singularResource": "validatingadmissionpolicy", "subresources": [ { "responseKind": { "group": "", "kind": "ValidatingAdmissionPolicy", "version": "" }, "subresource": "status", "verbs": [ "get", "patch", "update" ] } ], "verbs": [ "create", "delete", "deletecollection", "get", "list", "patch", "update", "watch" ] }, { "categories": [ "api-extensions" ], "resource": "validatingadmissionpolicybindings", "responseKind": { "group": "", "kind": "ValidatingAdmissionPolicyBinding", "version": "" }, "scope": "Cluster", "singularResource": "validatingadmissionpolicybinding", "verbs": [ "create", "delete", "deletecollection", "get", "list", "patch", "update", "watch" ] } ], "version": "v1beta1" }, { "freshness": "Current", "resources": [ { "categories": [ "api-extensions" ], "resource": "validatingadmissionpolicies", "responseKind": { "group": "", "kind": "ValidatingAdmissionPolicy", "version": "" }, "scope": "Cluster", "singularResource": "validatingadmissionpolicy", "subresources": [ { "responseKind": { "group": "", "kind": "ValidatingAdmissionPolicy", "version": "" }, "subresource": "status", "verbs": [ "get", "patch", "update" ] } ], "verbs": [ "create", "delete", "deletecollection", "get", "list", "patch", "update", "watch" ] }, { "categories": [ "api-extensions" ], "resource": "validatingadmissionpolicybindings", "responseKind": { "group": "", "kind": "ValidatingAdmissionPolicyBinding", "version": "" }, "scope": "Cluster", "singularResource": "validatingadmissionpolicybinding", "verbs": [ "create", "delete", "deletecollection", "get", "list", "patch", "update", "watch" ] } ], "version": "v1alpha1" } ] }, { "metadata": { "creationTimestamp": null, "name": "apiextensions.k8s.io" }, "versions": [ { "freshness": "Current", "resources": [ { "categories": [ "api-extensions" ], "resource": "customresourcedefinitions", "responseKind": { "group": "", "kind": "CustomResourceDefinition", "version": "" }, "scope": "Cluster", "shortNames": [ "crd", "crds" ], "singularResource": "customresourcedefinition", "subresources": [ { "responseKind": { "group": "", "kind": "CustomResourceDefinition", "version": "" }, "subresource": "status", "verbs": [ "get", "patch", "update" ] } ], "verbs": [ "create", "delete", "deletecollection", "get", "list", "patch", "update", "watch" ] } ], "version": "v1" } ] }, { "metadata": { "creationTimestamp": null, "name": "scheduling.k8s.io" }, "versions": [ { "freshness": "Current", "resources": [ { "resource": "priorityclasses", "responseKind": { "group": "", "kind": "PriorityClass", "version": "" }, "scope": "Cluster", "shortNames": [ "pc" ], "singularResource": "priorityclass", "verbs": [ "create", "delete", "deletecollection", "get", "list", "patch", "update", "watch" ] } ], "version": "v1" } ] }, { "metadata": { "creationTimestamp": null, "name": "coordination.k8s.io" }, "versions": [ { "freshness": "Current", "resources": [ { "resource": "leases", "responseKind": { "group": "", "kind": "Lease", "version": "" }, "scope": "Namespaced", "singularResource": "lease", "verbs": [ "create", "delete", "deletecollection", "get", "list", "patch", "update", "watch" ] } ], "version": "v1" }, { "freshness": "Current", "resources": [ { "resource": "leasecandidates", "responseKind": { "group": "", "kind": "LeaseCandidate", "version": "" }, "scope": "Namespaced", "singularResource": "leasecandidate", "verbs": [ "create", "delete", "deletecollection", "get", "list", "patch", "update", "watch" ] } ], "version": "v1alpha1" } ] }, { "metadata": { "creationTimestamp": null, "name": "node.k8s.io" }, "versions": [ { "freshness": "Current", "resources": [ { "resource": "runtimeclasses", "responseKind": { "group": "", "kind": "RuntimeClass", "version": "" }, "scope": "Cluster", "singularResource": "runtimeclass", "verbs": [ "create", "delete", "deletecollection", "get", "list", "patch", "update", "watch" ] } ], "version": "v1" } ] }, { "metadata": { "creationTimestamp": null, "name": "discovery.k8s.io" }, "versions": [ { "freshness": "Current", "resources": [ { "resource": "endpointslices", "responseKind": { "group": "", "kind": "EndpointSlice", "version": "" }, "scope": "Namespaced", "singularResource": "endpointslice", "verbs": [ "create", "delete", "deletecollection", "get", "list", "patch", "update", "watch" ] } ], "version": "v1" } ] }, { "metadata": { "creationTimestamp": null, "name": "flowcontrol.apiserver.k8s.io" }, "versions": [ { "freshness": "Current", "resources": [ { "resource": "flowschemas", "responseKind": { "group": "", "kind": "FlowSchema", "version": "" }, "scope": "Cluster", "singularResource": "flowschema", "subresources": [ { "responseKind": { "group": "", "kind": "FlowSchema", "version": "" }, "subresource": "status", "verbs": [ "get", "patch", "update" ] } ], "verbs": [ "create", "delete", "deletecollection", "get", "list", "patch", "update", "watch" ] }, { "resource": "prioritylevelconfigurations", "responseKind": { "group": "", "kind": "PriorityLevelConfiguration", "version": "" }, "scope": "Cluster", "singularResource": "prioritylevelconfiguration", "subresources": [ { "responseKind": { "group": "", "kind": "PriorityLevelConfiguration", "version": "" }, "subresource": "status", "verbs": [ "get", "patch", "update" ] } ], "verbs": [ "create", "delete", "deletecollection", "get", "list", "patch", "update", "watch" ] } ], "version": "v1" }, { "freshness": "Current", "resources": [ { "resource": "flowschemas", "responseKind": { "group": "", "kind": "FlowSchema", "version": "" }, "scope": "Cluster", "singularResource": "flowschema", "subresources": [ { "responseKind": { "group": "", "kind": "FlowSchema", "version": "" }, "subresource": "status", "verbs": [ "get", "patch", "update" ] } ], "verbs": [ "create", "delete", "deletecollection", "get", "list", "patch", "update", "watch" ] }, { "resource": "prioritylevelconfigurations", "responseKind": { "group": "", "kind": "PriorityLevelConfiguration", "version": "" }, "scope": "Cluster", "singularResource": "prioritylevelconfiguration", "subresources": [ { "responseKind": { "group": "", "kind": "PriorityLevelConfiguration", "version": "" }, "subresource": "status", "verbs": [ "get", "patch", "update" ] } ], "verbs": [ "create", "delete", "deletecollection", "get", "list", "patch", "update", "watch" ] } ], "version": "v1beta3" } ] }, { "metadata": { "creationTimestamp": null, "name": "internal.apiserver.k8s.io" }, "versions": [ { "freshness": "Current", "resources": [ { "resource": "storageversions", "responseKind": { "group": "", "kind": "StorageVersion", "version": "" }, "scope": "Cluster", "singularResource": "storageversion", "subresources": [ { "responseKind": { "group": "", "kind": "StorageVersion", "version": "" }, "subresource": "status", "verbs": [ "get", "patch", "update" ] } ], "verbs": [ "create", "delete", "deletecollection", "get", "list", "patch", "update", "watch" ] } ], "version": "v1alpha1" } ] }, { "metadata": { "creationTimestamp": null, "name": "resource.k8s.io" }, "versions": [ { "freshness": "Current", "resources": [ { "resource": "deviceclasses", "responseKind": { "group": "", "kind": "DeviceClass", "version": "" }, "scope": "Cluster", "singularResource": "deviceclass", "verbs": [ "create", "delete", "deletecollection", "get", "list", "patch", "update", "watch" ] }, { "resource": "podschedulingcontexts", "responseKind": { "group": "", "kind": "PodSchedulingContext", "version": "" }, "scope": "Namespaced", "singularResource": "podschedulingcontext", "subresources": [ { "responseKind": { "group": "", "kind": "PodSchedulingContext", "version": "" }, "subresource": "status", "verbs": [ "get", "patch", "update" ] } ], "verbs": [ "create", "delete", "deletecollection", "get", "list", "patch", "update", "watch" ] }, { "resource": "resourceclaims", "responseKind": { "group": "", "kind": "ResourceClaim", "version": "" }, "scope": "Namespaced", "singularResource": "resourceclaim", "subresources": [ { "responseKind": { "group": "", "kind": "ResourceClaim", "version": "" }, "subresource": "status", "verbs": [ "get", "patch", "update" ] } ], "verbs": [ "create", "delete", "deletecollection", "get", "list", "patch", "update", "watch" ] }, { "resource": "resourceclaimtemplates", "responseKind": { "group": "", "kind": "ResourceClaimTemplate", "version": "" }, "scope": "Namespaced", "singularResource": "resourceclaimtemplate", "verbs": [ "create", "delete", "deletecollection", "get", "list", "patch", "update", "watch" ] }, { "resource": "resourceslices", "responseKind": { "group": "", "kind": "ResourceSlice", "version": "" }, "scope": "Cluster", "singularResource": "resourceslice", "verbs": [ "create", "delete", "deletecollection", "get", "list", "patch", "update", "watch" ] } ], "version": "v1alpha3" } ] }, { "metadata": { "creationTimestamp": null, "name": "storagemigration.k8s.io" }, "versions": [ { "freshness": "Current", "resources": [ { "resource": "storageversionmigrations", "responseKind": { "group": "", "kind": "StorageVersionMigration", "version": "" }, "scope": "Cluster", "singularResource": "storageversionmigration", "subresources": [ { "responseKind": { "group": "", "kind": "StorageVersionMigration", "version": "" }, "subresource": "status", "verbs": [ "get", "patch", "update" ] } ], "verbs": [ "create", "delete", "deletecollection", "get", "list", "patch", "update", "watch" ] } ], "version": "v1alpha1" } ] } ], "kind": "APIGroupDiscoveryList", "metadata": {} } </del>

Aggregated discovery v2beta1 fixtures should be removed. /sig api-machinery /triage accepted


Question: if !pluginStatus.IsUnschedulable() { // Filter plugins are not supposed to return any status other than // Success or Unschedulable. <del> errStatus := framework.AsStatus(fmt.Errorf("running %q filter plugin: %w", pl.Name(), pluginStatus.AsError())).WithFailedPlugin(pl.Name()) return map[string]*framework.Status{pl.Name(): errStatus} </del> <ins> pluginStatus = framework.AsStatus(fmt.Errorf("running %q filter plugin: %w", pl.Name(), pluginStatus.AsError())) </ins> } pluginStatus.SetFailedPlugin(pl.Name()) <del> statuses[pl.Name()] = pluginStatus </del> <ins> return map[string]*framework.Status{pl.Name(): pluginStatus} </ins> } }

changed the default behavior of scheduler framework's - from stopping at the first failed filter to always running all filters. This change breaks Cluster Autoscaler at scale. Cluster Autoscaler imports the scheduler framework to simulate filters on artificial nodes. In a (pretty common) one-pod-per-node scenario, the number of calls is ~quadratic wrt. the number of nodes needed for unschedulable pods - it can be a lot of calls. Before , in such scenario the vast majority of calls would stop at failing Fit or NodeAffinity filters, and not run the rest of them (e.g. PV-related filters). After the change, every call to runs all filters, which completely breaks Cluster Autoscaler at scale (an example scenario of 300 1-pod-per-node pods and 200 node groups takes 10min in 1.24, and ~50s in <1.22). Can we get the old behavior back? Or re-introduce the config option so that Cluster Autoscaler can set it accordingly? Ideally, we'd also like to get this backported as far as possible (the change was introduced in 1.24).
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scheduling
This seems to be a regression. I would be interested in adding a toggle flag that allows one to toggle on that behavior if this was done on purpose.
That PR wasn't supposed to change the behavior actually. Scheduler is always defaulted to stop upon the first Filter failure, and it's just we offered an option to continue go through other Filter plugins upon failures, and that option was removed recently. Let me check the code to see if there is a regression.
When removed the following changes: It should have kept . I will submit a fix along with test, and backport to 1.24, 1.25, 1.26. Thanks for reporting!


Question: if !pluginStatus.IsUnschedulable() { // Filter plugins are not supposed to return any status other than // Success or Unschedulable. <del> errStatus := framework.AsStatus(fmt.Errorf("running %q filter plugin: %w", pl.Name(), pluginStatus.AsError())).WithFailedPlugin(pl.Name()) return map[string]*framework.Status{pl.Name(): errStatus} </del> <ins> pluginStatus = framework.AsStatus(fmt.Errorf("running %q filter plugin: %w", pl.Name(), pluginStatus.AsError())) </ins> } pluginStatus.SetFailedPlugin(pl.Name()) <del> statuses[pl.Name()] = pluginStatus </del> <ins> return map[string]*framework.Status{pl.Name(): pluginStatus} </ins> } }

cc /assign /kind bug /kind regression /priority critical-urgent /triage/accepted
Thanks Kuba, this is indeed a regression on the scheduler end.
/open for patches
/re-open /reopen
Reopened this issue. <detailsIn response to : /re-open /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Thanks for the reporting, and thanks for the fix.
Really sorry about this regression, and hope not to happen again. I think there's no e2e tests between kubernetes scheduler and cluster-autoscaler, so can we also have testcases of pods unschedulable in scheduling perf-test, maybe this can also help to prevent this happening again. UT benchmark test also makes sense, but it's not graphical.
Thanks for the quick fix! Unfortunately I'm afraid this kind of issue would only be caught in a large-scale test scenario, which is not cheap to run continuously.
For this particular issue, UT ( in the PR) is able to catch it. And hence doesn't need to be that complicated to test with cluster-autoscaler. However, standing from a broader angle, why we didn't catch the regression in perf test is b/c the triggering actor is unschedulable Pods, instead of schedulable Pods (which are the happy path most perf tests covered). So going forward, I'd like to see more perf tests to mixin unschedulable Pods, and hence more likely to surface this kind of issue. (we have preemptio preemption test suite but doesn't seem adequate)
This is not for the particular issue, but keeping an eye on whether there's any performance degradation between two kubernetes components, like we do for scheduling performance. but it's actually out of the scope of scheduler. I'll add it as one of the goals of
Actually, the schedulerperf has a test case . And, the schedulerperf collects the metric (), which has the label in it (). But, the problem is currently we don't distinguish the metric values by label on each data; we just collect/aggregate it regardless the label.


Question: if !pluginStatus.IsUnschedulable() { // Filter plugins are not supposed to return any status other than // Success or Unschedulable. <del> errStatus := framework.AsStatus(fmt.Errorf("running %q filter plugin: %w", pl.Name(), pluginStatus.AsError())).WithFailedPlugin(pl.Name()) return map[string]*framework.Status{pl.Name(): errStatus} </del> <ins> pluginStatus = framework.AsStatus(fmt.Errorf("running %q filter plugin: %w", pl.Name(), pluginStatus.AsError())) </ins> } pluginStatus.SetFailedPlugin(pl.Name()) <del> statuses[pl.Name()] = pluginStatus </del> <ins> return map[string]*framework.Status{pl.Name(): pluginStatus} </ins> } }

(We can distinguish the metric values by any labels in the schedulerperf as we do in the metric ()) I'd say just distinguishing the happy path and unhappy path with label in metric should be enough to observe the particular performance degradation in the unhappy path.
could you help share the perf-dash link where this metric () is exposed?
Hm, cannot see any scheduler metrics result from the current perf-dash I see some configuration that seems to be to parse the schedulerperf result file though. https://perf- TBH, I haven't visited the perf-dash so much, so not sure whether it's not working for a long time, or got broken recently. Also, another thing, I noticed the perf-dash doesn't show all test cases. All test cases after haven't been executed.
So... probably, something wrong happens in  job (probably in the test case) and the schedulerperf is failing. The schedulerperf doesn't make the result file when fails, so I'm guessing that's why we cannot see any scheduler metrics results from the perf-dash. Okay, I found it. Let me open another issue for this.
cc is this actually a regression, and is it resolved yet?
It's resolved by , and also cherry-picked to 1.24.x ~ 1.26.x. /close
Closing this issue. <detailsIn response to : It's resolved by , and also cherry-picked to 1.24.x ~ 1.26.x. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: } } <ins> func TestDescribePodSecurityPolicy(t *testing.T) { expected := []string{ "Name:t*mypsp", "Allow Privileged:t*false", "Default Add Capabilities:t*<none>", "Required Drop Capabilities:t*<none>", "Allowed Capabilities:t*<none>", "Allowed Volume Types:t*<none>", "Allow Host Network:t*false", "Allow Host Ports:t*<none>", "Allow Host PID:t*false", "Allow Host IPC:t*false", "Read Only Root Filesystem:t*false", "SELinux Context Strategy: RunAsAny", "User:t*<none>", "Role:t*<none>", "Type:t*<none>", "Level:t*<none>", "Run As User Strategy: RunAsAny", "FSGroup Strategy: RunAsAny", "Supplemental Groups Strategy: RunAsAny", } fake := fake.NewSimpleClientset(&extensions.PodSecurityPolicy{ ObjectMeta: metav1.ObjectMeta{ Name: "mypsp", }, Spec: extensions.PodSecurityPolicySpec{ SELinux: extensions.SELinuxStrategyOptions{ Rule: extensions.SELinuxStrategyRunAsAny, }, RunAsUser: extensions.RunAsUserStrategyOptions{ Rule: extensions.RunAsUserStrategyRunAsAny, }, FSGroup: extensions.FSGroupStrategyOptions{ Rule: extensions.FSGroupStrategyRunAsAny, }, SupplementalGroups: extensions.SupplementalGroupsStrategyOptions{ Rule: extensions.SupplementalGroupsStrategyRunAsAny, }, }, }) c := &describeClient{T: t, Namespace: "", Interface: fake} d := PodSecurityPolicyDescriber{c} out, err := d.Describe("", "mypsp", printers.DescriberSettings{}) if err != nil { t.Fatalf("unexpected error: %v", err) } for _, item := range expected { if matched, _ := regexp.MatchString(item, out); !matched { t.Errorf("Expected to find %q in: %q", item, out) } } } </ins> func TestDescribeResourceQuota(t *testing.T) { fake := fake.NewSimpleClientset(&api.ResourceQuota{ ObjectMeta: metav1.ObjectMeta{

Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): No What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): describe psp Is this a BUG REPORT or FEATURE REQUEST? (choose one): FEATURE REQUEST Kubernetes version (use ): Environment: Doesn't matter. What happened: doesn't show PSP's fields: What you expected to happen: All PSP fields should be shown. How to reproduce it (as minimally and precisely as possible):
If you're agree with that than I (or :) ) could implement it.
yes, improve the describer. Look at the scc describer for formatting.
any ideas which SIG should own PSP?
In this case it should be sig-cli /sig cli
yeah, this is cli related. I'd go with sig-auth for actual behavior.


Question: createdConts.Insert(p[:n-8]) } } <del> // We expect 9: 2 pod infra containers + 2 pods from the replication controller + //              1 pod infra container + 2 pods from the URL + //              1 pod infra container + 1 pod from the service test. // In addition, runStaticPodTest creates 1 pod infra containers + //                                       1 pod container from the mainfest file // The total number of container created is 11 </del> <ins> // We expect 9: 2 pod infra containers + 2 containers from the replication controller + //              1 pod infra container + 2 containers from the URL on first Kubelet + //              1 pod infra container + 2 containers from the URL on second Kubelet + //              1 pod infra container + 1 container from the service test. // In addition, runStaticPodTest creates 2 pod infra containers + //                                       2 pod containers from the file (1 for manifest and 1 for spec) // The total number of container created is 13 </ins> <del> if len(createdConts) != 11 { glog.Fatalf("Expected 11 containers; got %vnnlist of created containers:nn%#vnnDocker 1 Created:nn%#vnnDocker 2 Created:nn%#vnn", len(createdConts), createdConts.List(), fakeDocker1.Created, fakeDocker2.Created) </del> <ins> if len(createdConts) != 16 { glog.Fatalf("Expected 16 containers; got %vnnlist of created containers:nn%#vnnDocker 1 Created:nn%#vnnDocker 2 Created:nn%#vnn", len(createdConts), createdConts.List(), fakeDocker1.Created, fakeDocker2.Created) </ins> } glog.Infof("OK - found created containers: %#v", createdConts.List()) } // ServeCachedManifestFile serves a file for kubelet to read. <del> func ServeCachedManifestFile() (servingAddress string) { </del> <ins> func ServeCachedManifestFile(contents string) (servingAddress string) { </ins> server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { if r.URL.Path == "/manifest" { <del> w.Write([]byte(testManifestFile)) </del> <ins> w.Write([]byte(contents)) </ins> return } glog.Fatalf("Got request: %#vn", r)

Currently it assumes it can read ContainerManifest - it should try both
Why not hold on this until we have Kubelet-read-Pod in?
Also the HTTP source has the same problem.
It should read Pod. Changed the title.
Ideally, I'd like all Pods created directly via Kubelet to be to etcd and surfaced via the API. If updates are made via the API, I'd like those to propagate back to the node. We can discuss whether the config file should be updated or ignored. At some point, I'd like Kubelet to persistently cache pods received from the apiserver, perhaps by just writing them all out as config files.
is the issue to sync local pods back to apiserver.
this is somewhat related to other issues we talked about.
Detail: We want locally specified pods to be able to use YAML config files, but we also want to rip YAML out of the API server . The config-file behavior should be as similar to the kubectl experience. Kubelet could use parts of the kubectl library to translate YAML to JSON. Vision: Eventually, I'd like Kubelet to use the apiserver library and support exactly the same Pod API as the master, at which time I'd like kubectl be able to create pods directly on Kubelets, as well as reading from config files. I also expect to have a per-node controller . I'd like to be able to use the per-node controller to update pods started via config files (or direct kubectl-kubelet pods), overriding those original files. For availability and self-hosting, we'll eventually need local checkpointing , ideally just using serialized JSON for each object. If Kubelet used apiserver, this could be done using a custom registry implementation.
I think that is a prerequisite for it.
If noone picks it up, I'm going to start working on it tomorrow.
thanks, wojtek-t
fixed it for http channel


Question: command: - /heapster - --source=kubernetes.summary_api:'' <del> - image: gcr.io/google_containers/addon-resizer:1.5 </del> <ins> - image: gcr.io/google_containers/addon-resizer:1.6 </ins> name: heapster-nanny resources: limits:

HPA tests are currently failing in gke-large cluster suite: I did some debugging and the problem seems to be related to pod-nanny. Looking into logs from the last run: was first scheduled on the large node dedicated to it pod-nany updated heapster deployment then heapster was scheduled on the normal node (when it actually shouldn't fit) that point it was never updated by the pod nanny. that point, heapster started crashlooping (OOMing) [which in fact was expected in too small machine] So it's pretty clear that that pod-nanny changed the amount of resources to something smaller, which in fact it shouldn't.


Question: testAPIVersion = "testgroup/testversion" ) <del> var scheme *runtime.Scheme func init() { scheme = runtime.NewScheme() metav1.AddMetaToScheme(scheme) } </del> func newPartialObjectMetadata(apiVersion, kind, namespace, name string) *metav1.PartialObjectMetadata { return &metav1.PartialObjectMetadata{ TypeMeta: metav1.TypeMeta{

Because of this scheme mutation, this leads to concurrent use of the same scheme to cause the program to panic. It would be ideal to do this in some concurrent safe manner or add a warning or similar
/sig api-machinery
/assign
/assign /triage accepted


Question: <del> apiVersion: v1 kind: Service metadata: name: elasticsearch labels: component: elasticsearch role: client spec: type: LoadBalancer selector: component: elasticsearch role: client ports: - name: http port: 9200 protocol: TCP </del>

Presently, the elasticsearch example Kubernetes configurations target elasticsearch v1.7.1, which is quite old. As of this writing, the latest elasticsearch release is v5.6.2, with many deployment changes since then. Could the elasticsearch examples be updated to target the v5 series of elasticsearch? See for a more up to date reference configuration.
cc
hey it's been a long while since I last contributed to the docs. i can do it soon but i need pointers on how to properly do it.
hey, so we need to update the stuff under examples/elasticsearch/ *&repos=kubernetes So basically update the images, yaml files etc and test that everything works (and submit a PR)
/sig testing
Ah, so there's no documentation updates anymore. OK I'll take care of it, assign it to me.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
GitHub didn't allow me to assign the following users: me. Note that only and repo collaborators can be assigned. <detailsIn response to : /assign me Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign pires
/remove-lifecycle stale
it seems that the example moved to I am now reading that repo guidelines and will update this there. I'll also open a PR that cleans this folder in the current repo and closes this issue.


Question: // More details here: https://github.com/kubernetes/kubernetes/issues/50986 AutoDetectCloudProvider = "auto-detect" <del> defaultIPTablesMasqueradeBit = 14 defaultIPTablesDropBit       = 15 </del> <ins> DefaultIPTablesMasqueradeBit = 14 DefaultIPTablesDropBit       = 15 </ins> ) var ( zeroDuration = metav1.Duration{} // Refer to [Node Allocatable](https://git.k8s.io/community/contributors/design-proposals/node/node-allocatable.md) doc for more information. <del> defaultNodeAllocatableEnforcement = []string{"pods"} </del> <ins> DefaultNodeAllocatableEnforcement = []string{"pods"} </ins> ) func addDefaultingFuncs(scheme *kruntime.Scheme) error {

Solution: I will add one.
/cc
xref
Thanks for sending a PR so quickly!
[MILESTONENOTIFIER] Milestone Issue Needs Attention Action required: During code slush, issues in the milestone should be in progress. If this issue is not being actively worked on, please remove it from the milestone. If it is being worked on, please add the label so it can be tracked with other in-flight issues. Note: If this issue is not resolved or labeled as by Wed, Nov 22 it will be moved out of the v1.9 milestone. <details<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details


Question: ExpectedEtcdPath: "/registry/configmaps/" + namespace + "/cm1", }, gvr("", "v1", "services"): { <del> Stub:             `{"metadata": {"name": "service1"}, "spec": {"externalName": "service1name", "ports": [{"port": 10000, "targetPort": 11000}], "selector": {"test": "data"}}}`, </del> <ins> Stub:             `{"metadata": {"name": "service1"}, "spec": {"type": "LoadBalancer", "ports": [{"port": 10000, "targetPort": 11000}], "selector": {"test": "data"}}}`, </ins> ExpectedEtcdPath: "/registry/services/specs/" + namespace + "/service1", }, gvr("", "v1", "podtemplates"): {

xref I think this may be one of the few cases where we can actually change the behavior in a "breaking" way. User creates service with , is empty Controller or user updates the status subresource, sets (not wiped or validated, should be) User updates in any way, is wiped It's clear that the is not meant to be set when is not "LoadBalancer". The question is: 1) Fail validation? 2) Silently discard the value? 3) Warn and then discard the value? 4) Just warn and leave it. Because we allowed it in the past, but it is so volatile, I lean towards 3. and - the people most familair with this type. Edit: PrepareForUpdate() runs before warnings. So: (1) is easy (2) is easy (4) is slightly less easy than (2) (3) is slightly less easy than (4)
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
OMG change it to clusterIP as expected Status is wiped update status as ClusterIP ... and if I edit the service back to loadbalancer it keeps the status??
/kind bug
/retitle Service can be set when !=
We talked about this one at the SIG Network meeting today. suggested we use the feature gate for deprecation option to phase the current (problematic) functionality out.
/triage accept
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /triage accept Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: defer close(stopCh) manager, informers := testNewReplicaSetControllerFromClient(client, stopCh, BurstReplicas) <del> // A controller with 2 replicas and no pods in the store, 2 creates expected </del> <ins> // A controller with 2 replicas and no active pods in the store. // Inactive pods should be ignored. 2 creates expected. </ins> labelMap := map[string]string{"foo": "bar"} rs := newReplicaSet(2, labelMap) informers.Extensions().V1beta1().ReplicaSets().Informer().GetIndexer().Add(rs) <ins> failedPod := newPod("failed-pod", rs, v1.PodFailed, nil, true) deletedPod := newPod("deleted-pod", rs, v1.PodRunning, nil, true) deletedPod.DeletionTimestamp = &metav1.Time{Time: time.Now()} informers.Core().V1().Pods().Informer().GetIndexer().Add(failedPod) informers.Core().V1().Pods().Informer().GetIndexer().Add(deletedPod) </ins> fakePodControl := controller.FakePodControl{} manager.podControl = &fakePodControl

https://k8s- Failed: [] Network Partition [Disruptive] [Slow] [] [ReplicationController] should eagerly create replacement pod during network partition when termination grace is non-zero {Kubernetes e2e suite}
https://k8s- seems to be the first failure here. It seems to have gone in with a bunch of RC/RS changes () cc/


Question: // Represents the requirement on the container exit codes. // +optional <del> OnExitCodes *PodFailurePolicyOnExitCodesRequirement `json:"onExitCodes" protobuf:"bytes,2,opt,name=onExitCodes"` </del> <ins> OnExitCodes *PodFailurePolicyOnExitCodesRequirement `json:"onExitCodes,omitempty" protobuf:"bytes,2,opt,name=onExitCodes"` </ins> // Represents the requirement on the pod conditions. The requirement is represented // as a list of pod condition patterns. The requirement is satisfied if at // least one pattern matches an actual pod condition. At most 20 elements are allowed. // +listType=atomic // +optional <del> OnPodConditions []PodFailurePolicyOnPodConditionsPattern `json:"onPodConditions" protobuf:"bytes,3,opt,name=onPodConditions"` </del> <ins> OnPodConditions []PodFailurePolicyOnPodConditionsPattern `json:"onPodConditions,omitempty" protobuf:"bytes,3,opt,name=onPodConditions"` </ins> } // PodFailurePolicy describes how failed pods influence the backoffLimit.

for a PodFailurePolicyRule. However, when trying to create a JobSet programmatically with a JobTemplate that defines a PodFailurePolicy rule for failing a job immediately if the failed pod had condition DisruptionTarget, I get a validation error indicating that cannot be null. I've tried using both and (with matching versions for package as well). Error: JobSet spec: JobTemplate passes validation. JobSet: applying example JobSet in the description of this issue. JobSet: and applying example JobSet in the description of this issue. No response 30 (tests on a kind cluster using kindest/node:v1.30.0) <details</details<details</details<details</details<details</details<details</details
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
cc
/assign
I confirmed the issue, moreover, I see that when the JobSet webhook is disabled then it works ok, I think it must be the JobSet webhook which adds the "onExitCodes" field during deserializion /serialization step. I can also see that when I rebuild JobSet using kubernetes field "onExitCodes" with "omitempty", then it the JobSet creates ok. I will open a PR.


Question: }) ginkgo.It("Kubelet's oom-score-adj should be -999", func() { kubeletPids, err := getPidsForProcess(kubeletProcessName, "") <del> gomega.Expect(err).To(gomega.BeNil(), "failed to get list of kubelet pids") </del> <ins> framework.ExpectNoError(err, "failed to get list of kubelet pids") </ins> framework.ExpectEqual(len(kubeletPids), 1, "expected only one kubelet process; found %d", len(kubeletPids)) gomega.Eventually(func() error { return validateOOMScoreAdjSetting(kubeletPids[0], -999)

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: kubernetes/node-problem-detector/node-problem-detector-ci: ci-npd-e2e-node See the Which test(s) are failing: [] NodeProblemDetector [NodeFeature:NodeProblemDetector] [] SystemLogMonitor should generate node condition and events for corresponding errors Since when has it been failing: Dec 03. was at 19:50 PST, was at 21:51 PST. Testgrid link: https://k8s- Reason for failure: Apparently is the culprit. Anything else we need to know: This is likely caused by some library upgrade. Nothing big.
/sig node
/assign
Seems like the failure is caused by . /cc /cc /cc /cc
/priority critical-urgent
The particular bug I care about has been fixed in . I'm gonna still work on submitting , as the same error should also be affecting a few other tests.


Question: } sgList := []string{securityGroupID} <del> for _, extraSG := range strings.Split(annotation, ",") { </del> <ins> for _, extraSG := range strings.Split(annotations[ServiceAnnotationLoadBalancerExtraSecurityGroups], ",") { </ins> extraSG = strings.TrimSpace(extraSG) if len(extraSG) > 0 { sgList = append(sgList, extraSG)

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind feature /sig aws What happened: In k8s 1.7, when creating a service of type LoadBalancer, K8S creates the ELB and a Security Group for it. For these resources, it adds a tag, constructed dynamically, containing the cluster id: "${cluster_id}" with the value "true" What you expected to happen: We'd like to have also a statically named tag , as it was in 1.5 ("KubernetesCluster") with the value of the cluster id. How to reproduce it (as minimally and precisely as possible): Running on AWS cloud provider, create a service of type LoadBalancer. Anything else we need to know?: The reason for what we expect is that we have tight security constraints that require us to fine tune the IAM policies for the Instance profiles that the controllers use. With a statically named tag we were able, in 1.5, to restrict the permissions in the IAM policy to act only on the k8s managed resources (ELBs, security groups). Here is the relevant part of the IAM policy: With a dynamically named tag we are not able to do this any more as AWS IAM doesn't support something like: We are sharing the Instance Profiles among different k8s clusters, hence the "" in the value of the tag. Environment: Kubernetes version (use ):Cloud provider or hardware configuration*: AWS OS (e.g. from /etc/os-release):Kernel (e.g. ):Install tools: kube-aws Others:
Reiterating the mentions to trigger a notification: <detailsIn response to : <!-- This form is for bug reports and feature requests ONLY! -- /sig aws In k8s 1.7, when creating a service of type LoadBalancer, K8S creates the ELB and a Security Group for it. For these resources, it adds a tag, constructed dynamically, containing the cluster id: "${cluster_id}" with the value "true" We'd like to have also a statically named tag , as it was in 1.5 ("KubernetesCluster") with the value of the cluster id. Running on AWS cloud provider, create a service of type LoadBalancer.


Question: } sgList := []string{securityGroupID} <del> for _, extraSG := range strings.Split(annotation, ",") { </del> <ins> for _, extraSG := range strings.Split(annotations[ServiceAnnotationLoadBalancerExtraSecurityGroups], ",") { </ins> extraSG = strings.TrimSpace(extraSG) if len(extraSG) > 0 { sgList = append(sgList, extraSG)

The reason for what we expect is that we have tight security constraints that require us to fine tune the IAM policies for the Instance profiles that the controllers use. With a statically named tag we were able, in 1.5, to restrict the permissions in the IAM policy to act only on the k8s managed resources (ELBs, security groups). Here is the relevant part of the IAM policy: Environment: Kubernetes version (use ):OS (e.g. from /etc/os-release):Others: </details
I believe they are moving away from the tag since there are many resources that can be shared by multiple clusters. could you use an annotation?
We removed the condition in roles in the end. Thank you for your suggestion.
That annotation affects the ELB but not any security groups that are created. can you please reopen this? You reported a real issue, and even if it no longer affects you, it needs to be tracked and fixed.
Indeed, this might be a feature that someone else would need.
In fact, this is a feature I need!
Food for thought: should we propagate the tags on the ELB to the SG, or should we introduce an annotation for the SG specifically?


Question: expectMetrics(t, expectedMetrics{desiredSlices: 1, actualSlices: 1, desiredEndpoints: 10, addedPerSync: 20, removedPerSync: 10, numCreated: 1, numUpdated: 1, numDeleted: 0}) } <ins> // When a Service has a non-nil deletionTimestamp we want to avoid creating any // new EndpointSlices but continue to allow updates and deletes through. This // test uses 3 EndpointSlices, 1 "to-create", 1 "to-update", and 1 "to-delete". // Each test case exercises different combinations of calls to finalize with // those resources. func TestReconcilerFinalizeSvcDeletionTimestamp(t *testing.T) { now := metav1.Now() testCases := []struct { name               string deletionTimestamp  *metav1.Time attemptCreate      bool attemptUpdate      bool attemptDelete      bool expectCreatedSlice bool expectUpdatedSlice bool expectDeletedSlice bool }{{ name:               "Attempt create and update, nil deletion timestamp", deletionTimestamp:  nil, attemptCreate:      true, attemptUpdate:      true, expectCreatedSlice: true, expectUpdatedSlice: true, expectDeletedSlice: true, }, { name:               "Attempt create and update, deletion timestamp set", deletionTimestamp:  &now, attemptCreate:      true, attemptUpdate:      true, expectCreatedSlice: false, expectUpdatedSlice: true, expectDeletedSlice: true, }, { // Slice scheduled for creation is transitioned to update of Slice // scheduled for deletion. name:               "Attempt create, update, and delete, nil deletion timestamp, recycling in action", deletionTimestamp:  nil, attemptCreate:      true, attemptUpdate:      true, attemptDelete:      true, expectCreatedSlice: false, expectUpdatedSlice: true, expectDeletedSlice: true, }, { // Slice scheduled for creation is transitioned to update of Slice // scheduled for deletion. name:               "Attempt create, update, and delete, deletion timestamp set, recycling in action", deletionTimestamp:  &now, attemptCreate:      true, attemptUpdate:      true, attemptDelete:      true, expectCreatedSlice: false, expectUpdatedSlice: true, expectDeletedSlice: true, }, { // Update and delete continue to work when deletionTimestamp is set. name:               "Attempt update delete, deletion timestamp set", deletionTimestamp:  &now, attemptCreate:      false, attemptUpdate:      true, attemptDelete:      true, expectCreatedSlice: false, expectUpdatedSlice: true, expectDeletedSlice: false, }} for _, tc := range testCases { t.Run(tc.name, func(t *testing.T) { client := newClientset() setupMetrics() r := newReconciler(client, []*corev1.Node{{ObjectMeta: metav1.ObjectMeta{Name: "node-1"}}}, defaultMaxEndpointsPerSlice) namespace := "test" svc, endpointMeta := newServiceAndEndpointMeta("foo", namespace) svc.DeletionTimestamp = tc.deletionTimestamp esToCreate := &discovery.EndpointSlice{ ObjectMeta:  metav1.ObjectMeta{Name: "to-create"}, AddressType: endpointMeta.AddressType, Ports:       endpointMeta.Ports, } // Add EndpointSlice that can be updated. esToUpdate, err := client.DiscoveryV1beta1().EndpointSlices(namespace).Create(context.TODO(), &discovery.EndpointSlice{ ObjectMeta:  metav1.ObjectMeta{Name: "to-update"}, AddressType: endpointMeta.AddressType, Ports:       endpointMeta.Ports, }, metav1.CreateOptions{}) if err != nil { t.Fatalf("Expected no error creating EndpointSlice during test setup, got %v", err) } // Add an endpoint so we can see if this has actually been updated by // finalize func. esToUpdate.Endpoints = []discovery.Endpoint{{Addresses: []string{"10.2.3.4"}}} // Add EndpointSlice that can be deleted. esToDelete, err := client.DiscoveryV1beta1().EndpointSlices(namespace).Create(context.TODO(), &discovery.EndpointSlice{ ObjectMeta:  metav1.ObjectMeta{Name: "to-delete"}, AddressType: endpointMeta.AddressType, Ports:       endpointMeta.Ports, }, metav1.CreateOptions{}) if err != nil { t.Fatalf("Expected no error creating EndpointSlice during test setup, got %v", err) } slicesToCreate := []*discovery.EndpointSlice{} if tc.attemptCreate { slicesToCreate = append(slicesToCreate, esToCreate.DeepCopy()) } slicesToUpdate := []*discovery.EndpointSlice{} if tc.attemptUpdate { slicesToUpdate = append(slicesToUpdate, esToUpdate.DeepCopy()) } slicesToDelete := []*discovery.EndpointSlice{} if tc.attemptDelete { slicesToDelete = append(slicesToDelete, esToDelete.DeepCopy()) } err = r.finalize(&svc, slicesToCreate, slicesToUpdate, slicesToDelete, time.Now()) if err != nil { t.Errorf("Error calling r.finalize(): %v", err) } fetchedSlices := fetchEndpointSlices(t, client, namespace) createdSliceFound := false updatedSliceFound := false deletedSliceFound := false for _, epSlice := range fetchedSlices { if epSlice.Name == esToCreate.Name { createdSliceFound = true } if epSlice.Name == esToUpdate.Name { updatedSliceFound = true if tc.attemptUpdate && len(epSlice.Endpoints) != len(esToUpdate.Endpoints) { t.Errorf("Expected EndpointSlice to be updated with %d endpoints, got %d endpoints", len(esToUpdate.Endpoints), len(epSlice.Endpoints)) } } if epSlice.Name == esToDelete.Name { deletedSliceFound = true } } if createdSliceFound != tc.expectCreatedSlice { t.Errorf("Expected created EndpointSlice existence to be %t, got %t", tc.expectCreatedSlice, createdSliceFound) } if updatedSliceFound != tc.expectUpdatedSlice { t.Errorf("Expected updated EndpointSlice existence to be %t, got %t", tc.expectUpdatedSlice, updatedSliceFound) } if deletedSliceFound != tc.expectDeletedSlice { t.Errorf("Expected deleted EndpointSlice existence to be %t, got %t", tc.expectDeletedSlice, deletedSliceFound) } }) } } </ins> // Test Helpers func newReconciler(client *fake.Clientset, nodes []*corev1.Node, maxEndpointsPerSlice int32) *reconciler {

What happened: In the PMEM-CSI CI, we rapidly deploy and remove a custom resources which then owns all objects created for that resource. Deletion of those objects is handled through ownership references, i.e. deleting the CR is expected to cascade to all objects. This worked fine up to Kubernetes 1.17. When starting to test with 1.18.2, we pretty reliably get into a state where the CR and the service objects owned by it are marked for deletion, but don't go away. Debugging with the garbage collector's dependency graph (see ) showed that endpointslices were blocking the deletion of the services and thus the CR. In the graph dump (= a snaphot in time), they were not even marked for deletion. A closer look showed that endpointslices for the services were getting created and deleted continuously for over 15 minutes. Eventually, the garbage collector won and managed to delete endpointslices and services before new endpointslices were created. What you expected to happen: There seems to be a missing check in if the service is marked for deletion (i.e. has a deletionTimestamp), then the controller should not create new endpointslices. How to reproduce it (as minimally and precisely as possible): I tried to boil it down to a script, but it just ran fine. The "easiest" way to reproduce it is to run PMEM-CSI's E2E testing, which is probably not easy enough :disappointed: Anything else we need to know?: Disabling the EndpointSlice feature makes the problem go away. Environment: Kubernetes version (use ): 1.18.2 Cloud provider or hardware configuration: QEMU virtual machines OS (e.g: ): Fedora Install tools: kubeadm /sig network
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
cc for endpointslice
Thanks for the detailed bug report! Will work on getting a fix in soon. /assign
/remove-triage unresolved
Can you describe the general process to reproduce this bug? Would it involve hundreds of EndpointSlices for a Service? I've tried to recreate this in a , but unfortunately I haven't been able to get anything to break. Any ideas for what I could try?
There's one endpointslice per service. Our operator creates two services.


Question: expectMetrics(t, expectedMetrics{desiredSlices: 1, actualSlices: 1, desiredEndpoints: 10, addedPerSync: 20, removedPerSync: 10, numCreated: 1, numUpdated: 1, numDeleted: 0}) } <ins> // When a Service has a non-nil deletionTimestamp we want to avoid creating any // new EndpointSlices but continue to allow updates and deletes through. This // test uses 3 EndpointSlices, 1 "to-create", 1 "to-update", and 1 "to-delete". // Each test case exercises different combinations of calls to finalize with // those resources. func TestReconcilerFinalizeSvcDeletionTimestamp(t *testing.T) { now := metav1.Now() testCases := []struct { name               string deletionTimestamp  *metav1.Time attemptCreate      bool attemptUpdate      bool attemptDelete      bool expectCreatedSlice bool expectUpdatedSlice bool expectDeletedSlice bool }{{ name:               "Attempt create and update, nil deletion timestamp", deletionTimestamp:  nil, attemptCreate:      true, attemptUpdate:      true, expectCreatedSlice: true, expectUpdatedSlice: true, expectDeletedSlice: true, }, { name:               "Attempt create and update, deletion timestamp set", deletionTimestamp:  &now, attemptCreate:      true, attemptUpdate:      true, expectCreatedSlice: false, expectUpdatedSlice: true, expectDeletedSlice: true, }, { // Slice scheduled for creation is transitioned to update of Slice // scheduled for deletion. name:               "Attempt create, update, and delete, nil deletion timestamp, recycling in action", deletionTimestamp:  nil, attemptCreate:      true, attemptUpdate:      true, attemptDelete:      true, expectCreatedSlice: false, expectUpdatedSlice: true, expectDeletedSlice: true, }, { // Slice scheduled for creation is transitioned to update of Slice // scheduled for deletion. name:               "Attempt create, update, and delete, deletion timestamp set, recycling in action", deletionTimestamp:  &now, attemptCreate:      true, attemptUpdate:      true, attemptDelete:      true, expectCreatedSlice: false, expectUpdatedSlice: true, expectDeletedSlice: true, }, { // Update and delete continue to work when deletionTimestamp is set. name:               "Attempt update delete, deletion timestamp set", deletionTimestamp:  &now, attemptCreate:      false, attemptUpdate:      true, attemptDelete:      true, expectCreatedSlice: false, expectUpdatedSlice: true, expectDeletedSlice: false, }} for _, tc := range testCases { t.Run(tc.name, func(t *testing.T) { client := newClientset() setupMetrics() r := newReconciler(client, []*corev1.Node{{ObjectMeta: metav1.ObjectMeta{Name: "node-1"}}}, defaultMaxEndpointsPerSlice) namespace := "test" svc, endpointMeta := newServiceAndEndpointMeta("foo", namespace) svc.DeletionTimestamp = tc.deletionTimestamp esToCreate := &discovery.EndpointSlice{ ObjectMeta:  metav1.ObjectMeta{Name: "to-create"}, AddressType: endpointMeta.AddressType, Ports:       endpointMeta.Ports, } // Add EndpointSlice that can be updated. esToUpdate, err := client.DiscoveryV1beta1().EndpointSlices(namespace).Create(context.TODO(), &discovery.EndpointSlice{ ObjectMeta:  metav1.ObjectMeta{Name: "to-update"}, AddressType: endpointMeta.AddressType, Ports:       endpointMeta.Ports, }, metav1.CreateOptions{}) if err != nil { t.Fatalf("Expected no error creating EndpointSlice during test setup, got %v", err) } // Add an endpoint so we can see if this has actually been updated by // finalize func. esToUpdate.Endpoints = []discovery.Endpoint{{Addresses: []string{"10.2.3.4"}}} // Add EndpointSlice that can be deleted. esToDelete, err := client.DiscoveryV1beta1().EndpointSlices(namespace).Create(context.TODO(), &discovery.EndpointSlice{ ObjectMeta:  metav1.ObjectMeta{Name: "to-delete"}, AddressType: endpointMeta.AddressType, Ports:       endpointMeta.Ports, }, metav1.CreateOptions{}) if err != nil { t.Fatalf("Expected no error creating EndpointSlice during test setup, got %v", err) } slicesToCreate := []*discovery.EndpointSlice{} if tc.attemptCreate { slicesToCreate = append(slicesToCreate, esToCreate.DeepCopy()) } slicesToUpdate := []*discovery.EndpointSlice{} if tc.attemptUpdate { slicesToUpdate = append(slicesToUpdate, esToUpdate.DeepCopy()) } slicesToDelete := []*discovery.EndpointSlice{} if tc.attemptDelete { slicesToDelete = append(slicesToDelete, esToDelete.DeepCopy()) } err = r.finalize(&svc, slicesToCreate, slicesToUpdate, slicesToDelete, time.Now()) if err != nil { t.Errorf("Error calling r.finalize(): %v", err) } fetchedSlices := fetchEndpointSlices(t, client, namespace) createdSliceFound := false updatedSliceFound := false deletedSliceFound := false for _, epSlice := range fetchedSlices { if epSlice.Name == esToCreate.Name { createdSliceFound = true } if epSlice.Name == esToUpdate.Name { updatedSliceFound = true if tc.attemptUpdate && len(epSlice.Endpoints) != len(esToUpdate.Endpoints) { t.Errorf("Expected EndpointSlice to be updated with %d endpoints, got %d endpoints", len(esToUpdate.Endpoints), len(epSlice.Endpoints)) } } if epSlice.Name == esToDelete.Name { deletedSliceFound = true } } if createdSliceFound != tc.expectCreatedSlice { t.Errorf("Expected created EndpointSlice existence to be %t, got %t", tc.expectCreatedSlice, createdSliceFound) } if updatedSliceFound != tc.expectUpdatedSlice { t.Errorf("Expected updated EndpointSlice existence to be %t, got %t", tc.expectUpdatedSlice, updatedSliceFound) } if deletedSliceFound != tc.expectDeletedSlice { t.Errorf("Expected deleted EndpointSlice existence to be %t, got %t", tc.expectDeletedSlice, deletedSliceFound) } }) } } </ins> // Test Helpers func newReconciler(client *fake.Clientset, nodes []*corev1.Node, maxEndpointsPerSlice int32) *reconciler {

One peculiarity may be that we don't actually want the pods to start during these tests because we only want to verify object creation, not the usage of the result, and therefore all pods reference non-existent images. All objects immediately get deleted once the test has verified that they exist, so the timing is probably very different from "normal" usage patterns. I suspect that it is more important to create and delete services quickly than it is to have many endpointslices per service.
I had tried to come up with a script that triggers the problem, without success. That was before I realized that endpointslices are involved. On Monday I can have another look.
That was because I thought that (the default) enables foreground deletion. It doesn't, and there's also no option to select it. After explicitly selecting it via curl, the following script failed for me eventually: Output: Note that for the endpointslice is greater than the of its service.


[x] Original approval issue: - Reverted due to flakes [x] Two weeks soak end date: 20 November 2020 This issue is created to allow edit by /wip /hold According to this APIsnoop query, there are still some remaining Pod endpoints which are untested. a Pod with a static label the Pod with a new Label and updated data the Pod to ensure it's patched the Pod's status the Pod's status Ready condition to False the Pod's status to check if Ready is False all Pods in all Namespaces find the Pod(1) ensure that the Pod is found and is patched Namespaced Pod(1) via a Collection with a LabelSelector the Pod to check that it's deleted Discover useragents: List endpoints hit by the test: Display endpoint coverage change: If a test with these calls gets merged, test coverage will go up by 4 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/reopen Test PR Merge Promotion PR to be created next
Reopened this issue. <detailsIn response to : /reopen Test PR Merge Promotion PR to be created next Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
reverted by due to flaky test
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
/reopen PR on test grid for 2 weeks before promotion to conformance
Reopened this issue. <detailsIn response to : /reopen PR on test grid for 2 weeks before promotion to conformance Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/triage accepted


Question: <del> apiVersion: v1 kind: ReplicationController metadata: name: es labels: component: elasticsearch spec: replicas: 1 template: metadata: labels: component: elasticsearch spec: serviceAccount: elasticsearch containers: - name: es securityContext: capabilities: add: - IPC_LOCK image: quay.io/pires/docker-elasticsearch-kubernetes:1.7.1-4 env: - name: KUBERNETES_CA_CERTIFICATE_FILE value: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt - name: NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace - name: "CLUSTER_NAME" value: "myesdb" - name: "DISCOVERY_SERVICE" value: "elasticsearch" - name: NODE_MASTER value: "true" - name: NODE_DATA value: "true" - name: HTTP_ENABLE value: "true" ports: - containerPort: 9200 name: http protocol: TCP - containerPort: 9300 name: transport protocol: TCP volumeMounts: - mountPath: /data name: storage volumes: - name: storage emptyDir: {} </del>