Question: pod := config.endpointPods[0] config.getPodClient().Delete(pod.Name, nil) config.endpointPods = config.endpointPods[1:] <del> time.Sleep(5 * time.Second) // wait for kube-proxy to catch up with the pod being deleted. </del> <ins> // wait for pod being deleted. err := waitForPodToDisappear(config.f.Client, config.f.Namespace.Name, pod.Name, labels.Everything(), time.Second, util.ForeverTestTimeout) if err != nil { Failf("Failed to delete %s pod: %v", pod.Name, err) } // wait for endpoint being removed. err = waitForServiceEndpointsNum(config.f.Client, config.f.Namespace.Name, nodePortServiceName, len(config.endpointPods), time.Second, util.ForeverTestTimeout) if err != nil { Failf("Failed to remove endpoint from service: %s", nodePortServiceName) } // wait for kube-proxy to catch up with the pod being deleted. time.Sleep(5 * time.Second) </ins> } func (config *KubeProxyTestConfig) createPod(pod *api.Pod) *api.Pod {

More detail can be seen in e2e test
cc
, Marek Grabowski wrote:
you are running an older version of the test. can you get the latest and try again ? This test had not failed at all yesterday in our automated test runs yesterday. Closing issue for now.
This failure depends on if the pod is really deleted and kube-proxy (the test container on) catch that the endpoint is deleted in 5s . After sending "delete pod" request for 5s, if the pod was still there and kube-proxy still contains this endpoints(this may happen under pressure), hit test can still detect 5 endpoints. At least we should ensure the pod was deleted. Anyway, I think will be helpful to this test. What do you think?
It doesn't matter if the pod is actually deleted or not. We just care that kube-proxy thinks that it should not be part of the service endpoint. Having said that, it still doesn't hurt to do the check that you are doing. I will assign your PR to myself and review it. In the meantime, can you rebase your PR ?
Thanks for your review. The PR is up to date.
Failed twice today in kubernetes-e2e-gce-parallel
One more today. The error message is not very helpful.
18:28:37 STEP: dialing(http) endpointPodIP:endpointHttpPort from node1 18:28:37 STEP: Dialing from node. command:for i in ' wc -l This is the point where the test fails. Am looking into it. We need to have more tries, we just do seq (1 1), instead of seq (6 6). this is causing flakiness.
Fixed by PR


Question: package e2e_node import ( <del> "fmt" </del> "time" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/util/intstr" <del> "k8s.io/apimachinery/pkg/util/uuid" </del> "k8s.io/kubernetes/pkg/api/v1" "k8s.io/kubernetes/test/e2e/framework"

The prestop and poststart exec tests volume mount /tmp from the host and attempt to create a file there from inside the pod. This is blocked by selinux. Failure:
we need to have a mechanism to say that a test should not be run if selinux is enabled (if legitimately required) or fix each test (where appropriate)


Question: "k8s.io/kubernetes/pkg/api" "k8s.io/kubernetes/pkg/api/resource" "k8s.io/kubernetes/pkg/apis/extensions" <ins> "k8s.io/kubernetes/pkg/util/codeinspector" </ins> "k8s.io/kubernetes/plugin/pkg/scheduler" "k8s.io/kubernetes/plugin/pkg/scheduler/algorithm" priorityutil "k8s.io/kubernetes/plugin/pkg/scheduler/algorithm/priorities/util"

I just noticed you forgot to register the image locality priority function in in I also missed it when I reviewed your PR. I have fixed this in . But it would be great if you could figure out a mechanism that would ensure that all predicate/priority functions that are tested (you can assume they will always live under ) are registered, and fail the test if not. That will allow us to prevent this mistake in the future.
Thanks! Will fix this soon!
Seems need to wait for in before I send PR? Otherwise this test will always fail ...


Question: psps, err := kubeClient.PolicyV1beta1().PodSecurityPolicies().List(context.TODO(), metav1.ListOptions{}) if err != nil { Logf("Error listing PodSecurityPolicies; assuming PodSecurityPolicy is disabled: %v", err) <del> isPSPEnabled = false } else if psps == nil || len(psps.Items) == 0 { </del> <ins> return } if psps == nil || len(psps.Items) == 0 { </ins> Logf("No PodSecurityPolicies found; assuming PodSecurityPolicy is disabled.") <del> isPSPEnabled = false } else { Logf("Found PodSecurityPolicies; assuming PodSecurityPolicy is enabled.") isPSPEnabled = true </del> <ins> return } Logf("Found PodSecurityPolicies; testing pod creation to see if PodSecurityPolicy is enabled") testPod := &v1.Pod{ ObjectMeta: metav1.ObjectMeta{GenerateName: "psp-test-pod-"}, Spec:       v1.PodSpec{Containers: []v1.Container{{Name: "test", Image: imageutils.GetPauseImageName()}}}, } dryRunPod, err := kubeClient.CoreV1().Pods("kube-system").Create(context.TODO(), testPod, metav1.CreateOptions{DryRun: []string{metav1.DryRunAll}}) if err != nil { if strings.Contains(err.Error(), "PodSecurityPolicy") { Logf("PodSecurityPolicy error creating dryrun pod; assuming PodSecurityPolicy is enabled: %v", err) isPSPEnabled = true } else { Logf("Error creating dryrun pod; assuming PodSecurityPolicy is disabled: %v", err) } return } pspAnnotation, pspAnnotationExists := dryRunPod.Annotations["kubernetes.io/psp"] if !pspAnnotationExists { Logf("No PSP annotation exists on dry run pod; assuming PodSecurityPolicy is disabled") return </ins> } <ins> Logf("PSP annotation exists on dry run pod: %q; assuming PodSecurityPolicy is enabled", pspAnnotation) isPSPEnabled = true </ins> }) return isPSPEnabled }

What happened: e2e tests detect whether PSP is enabled based on presence of PSP API objects. An addon that proactively includes a PSP in a manifest can trigger that detection and enable PSP-specific e2e tests that should not be run. What you expected to happen: In addition to detecting presence of PSP objects, consider dry-run creating a pod and verifying the annotation indicating the selected PSP policy is , indicating the admission plugin is active. /sig auth /assign


Question: } } <ins> func TestDiscoveryContentTypeVersion(t *testing.T) { tests := []struct { contentType string isV2Beta1   bool }{ { contentType: "application/json; g=apidiscovery.k8s.io;v=v2beta1;as=APIGroupDiscoveryList", isV2Beta1:   true, }, { // content-type parameters are not in correct order, but comparison ignores order. contentType: "application/json; v=v2beta1;as=APIGroupDiscoveryList;g=apidiscovery.k8s.io", isV2Beta1:   true, }, { // content-type parameters are not in correct order, but comparison ignores order. contentType: "application/json; as=APIGroupDiscoveryList;g=apidiscovery.k8s.io;v=v2beta1", isV2Beta1:   true, }, { // Ignores extra parameter "charset=utf-8" contentType: "application/json; g=apidiscovery.k8s.io;v=v2beta1;as=APIGroupDiscoveryList;charset=utf-8", isV2Beta1:   true, }, { contentType: "application/json", isV2Beta1:   false, }, { contentType: "application/json; charset=UTF-8", isV2Beta1:   false, }, { contentType: "text/json", isV2Beta1:   false, }, { contentType: "text/html", isV2Beta1:   false, }, { contentType: "", isV2Beta1:   false, }, } for _, test := range tests { isV2Beta1 := isV2Beta1ContentType(test.contentType) assert.Equal(t, test.isV2Beta1, isV2Beta1) } } </ins> func TestUseLegacyDiscovery(t *testing.T) { // Default client sends aggregated discovery accept format (first) as well as legacy format. server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, req *http.Request) {

We are using (and ) to interact with Kubernetes clusters behind a couple of proxy layers. One of these proxies completes the headers to include character encoding details. This was working fine until recently, when client-go/kubectl have started to fail when interacting with the discovery API. It seems that a recent change has made kubectl more restrictive of the content-type that it expects. I suspect that it is related to this commit: For example, this is an error from a kubectl command: We can of course try to fix it on our end, removing that charset information. However, it is a valid content-type and the way kubectl is handling the content-type there seems like it is relying on implementation details of kube-apiserver (that it does not add metadata to the content-type). I propose to fix this issue in , by properly parsing and handling the content-type.
cc /kind regression /kind bug /sig api-machinery /priority important-soon looks like the switch statement in downloadLegacy() needs to be more tolerant of the variety of content-types it can get from a server (including text/plain or no content-type if old servers could have returned that...)
/transfer kubernetes/kubernetes
Something went wrong or the destination repo kubernetes/kubernetes/kubernetes does not exist. <detailsIn response to : /transfer kubernetes/kubernetes Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/transfer kubernetes
Yep, the string comparison for the content-type is flimsy, let's address that. Thanks for the report!
/assign
/triage accepted
/cc
The fix is being cherry-picked back to: : :


Question: package main import ( <del> "fmt" </del> "math/rand" "os" "time"

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: kubectl version --client=yes command is giving the following error message. Error message is repeated at the start and the end and also containing programming jargon Error: invalid argument "yes" for "--client" flag: strconv.ParseBool: parsing "yes": invalid syntax Examples: Print the client and server versions for the current context kubectl version Options: --client=false: Client version only (no server required). -o, --output='': One of 'yaml' or 'json'. --short=false: Print just the version number. Usage: kubectl version [flags] [options] Use "kubectl options" for a list of global command-line options (applies to all commands). invalid argument "yes" for "--client" flag: strconv.ParseBool: parsing "yes": invalid syntax What you expected to happen: The error message can be improved with one of the two variants invalid argument "yes" for "--client" flag Examples: Print the client and server versions for the current context kubectl version Options: --client=false: Client version only (no server required). -o, --output='': One of 'yaml' or 'json'. --short=false: Print just the version number. Usage: kubectl version [flags] [options] Use "kubectl options" for a list of global command-line options (applies to all commands). (or) Invalid argument "yes" for "--client" flag How to reproduce it (as minimally and precisely as possible): Run the command kubectl version --client=yes Anything else we need to know?: Environment: Kubernetes version (use ): 1.15 Cloud provider or hardware configuration: AWS OS (e.g: ): Amazon Linux Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig cli
Hi Thanks for posting this issue! It turns out that this duplicate output has been in kubectl, and all the other cli cmds, since . This should be fixed by , and hopefully we can cherry pick it back to at least .


Question: // our target metric should be updated by now if err := wait.PollUntilContextTimeout(ctx, 10*time.Second, 2*time.Minute, true, func(_ context.Context) (bool, error) { metrics, err := metricsGrabber.GrabFromKubeProxy(ctx, nodeName) <del> framework.ExpectNoError(err) </del> <ins> if err != nil { return false, fmt.Errorf("failed to fetch metrics: %w", err) } </ins> targetMetricAfter, err := metrics.GetCounterMetricValue(metricName) <del> framework.ExpectNoError(err) </del> <ins> if err != nil { return false, fmt.Errorf("failed to fetch metric: %w", err) } </ins> return targetMetricAfter > targetMetricBefore, nil }); err != nil { <del> framework.Failf("expected %s metric to be updated after accessing endpoints via localhost nodeports", metricName) </del> <ins> if wait.Interrupted(err) { framework.Failf("expected %s metric to be updated after accessing endpoints via localhost nodeports", metricName) } framework.ExpectNoError(err) </ins> } }) })

seen in this inside a like loop we execute a function that return a bool and an error. If we want to fail fast we return the error directly and assert on the output of the wait function. If we want to retry we log the error and return false nil /kind flake /kind bug /sig testing /sig network
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc since I think you this test but most probable the code is inherited and is a copy paste thing, but besides that, the change is to remove those there
/assign


Question: NodeDiskPressure NodeConditionType = "DiskPressure" // NodeNetworkUnavailable means that network for the node is not correctly configured. NodeNetworkUnavailable NodeConditionType = "NetworkUnavailable" <del> // NodeInodePressure means the kubelet is under pressure due to insufficient available inodes. NodeInodePressure NodeConditionType = "InodePressure" </del> ) // NodeCondition contains condition information for a node.

Is this a BUG REPORT or FEATURE REQUEST?: /kind cleanup What happened: It seems kubelet did not update inode pressure; and only test codes are using . So prefer to deprecate or remove it directly. (seems no node-misc group :). )


Question: go_test( name = "go_default_test", srcs = [ <ins> "codec_test.go", </ins> "conversion_test.go", "converter_test.go", "embedded_test.go",

What happened: Set up a HA cluster with 1.9 and 1.10 API servers (occurs on any apiserver skew boundary where a cohabitating resource's storage version changed groups) Observed errors when modifying workload objects: when modified via the 1.10 server, saw this error in the 1.9 server: when modified via the 1.9 server, saw this error in the 1.10 server: When this error was encountered, all watchers for the resource in question would be dropped and have to re-establish, and the internal watchers feeding the watch cache would drop and re-establish What you expected to happen: Watch events from all supported storage versions are processed correctly How to reproduce it (as minimally and precisely as possible): Start a server from master (for inspectability, store in json). With Create a deployment (e.g. kube-dns in kube-system) Establish a watch via extensions (kubectl get deployments.extensions --all-namespaces -w) Establish a watch via apps (kubectl get --all-namespaces -w) Tail the kube-apiserver log Change the apps/v1 deployment in etcd to extensions/v1beta1: Observer client watches drop and server log errors: Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: /sig api-machinery /cc /priority critical-urgent
error is triggered by this returning false: constructed here: the info about expected cohabitating groups is not included in the list of acceptable inputs


Question: fi proxy_mem_per_node=50 proxy_mem=$((100 * 1024 + proxy_mem_per_node*NUM_NODES)) <ins> hollow_kubelet_params=$(eval "for param in ${HOLLOW_KUBELET_TEST_ARGS:-}; do echo -n "$param",; done") hollow_kubelet_params=${hollow_kubelet_params%?} hollow_proxy_params=$(eval "for param in ${HOLLOW_PROXY_TEST_ARGS:-}; do echo -n "$param",; done") hollow_proxy_params=${hollow_proxy_params%?} </ins> sed -i'' -e "s/{{HOLLOW_PROXY_CPU}}/${proxy_cpu}/g" "${RESOURCE_DIRECTORY}/hollow-node.yaml" sed -i'' -e "s/{{HOLLOW_PROXY_MEM}}/${proxy_mem}/g" "${RESOURCE_DIRECTORY}/hollow-node.yaml" sed -i'' -e "s'{{kubemark_image_registry}}'${KUBEMARK_IMAGE_REGISTRY}${KUBE_NAMESPACE}'g" "${RESOURCE_DIRECTORY}/hollow-node.yaml" sed -i'' -e "s/{{kubemark_image_tag}}/${KUBEMARK_IMAGE_TAG}/g" "${RESOURCE_DIRECTORY}/hollow-node.yaml" sed -i'' -e "s/{{master_ip}}/${MASTER_IP}/g" "${RESOURCE_DIRECTORY}/hollow-node.yaml" <del> sed -i'' -e "s/{{hollow_kubelet_params}}/${HOLLOW_KUBELET_TEST_ARGS:-}/g" "${RESOURCE_DIRECTORY}/hollow-node.yaml" sed -i'' -e "s/{{hollow_proxy_params}}/${HOLLOW_PROXY_TEST_ARGS:-}/g" "${RESOURCE_DIRECTORY}/hollow-node.yaml" </del> <ins> sed -i'' -e "s/{{hollow_kubelet_params}}/${hollow_kubelet_params}/g" "${RESOURCE_DIRECTORY}/hollow-node.yaml" sed -i'' -e "s/{{hollow_proxy_params}}/${hollow_proxy_params}/g" "${RESOURCE_DIRECTORY}/hollow-node.yaml" </ins> sed -i'' -e "s'{{kubemark_mig_config}}'${KUBEMARK_MIG_CONFIG:-}'g" "${RESOURCE_DIRECTORY}/hollow-node.yaml" "${KUBECTL}" create -f "${RESOURCE_DIRECTORY}/hollow-node.yaml" --namespace="kubemark"

There are CVEs happening in different images - we already migrated Kubmeark from busybox in , but we should go one step further and migrate the base image to distroless. There are some things that would need to be changed mentioned here: /sig scalability
Can I work on this? /assign
/remove-help
- sure that would be great.
Hi based on the discussion in , the kubemark image has dependency on and maybe more other utilities. So it seems impossible to migrate the base image to distroless? Or do we need something like debian-hyperkube-base?
The mount one we may get rid of - basically we should be able to switch to FakeMounter:


Question: var resourcesByGV map[schema.GroupVersion]*metav1.APIResourceList // Switch on content-type server responded with: aggregated or unaggregated. <del> switch responseContentType { case AcceptV1: </del> <ins> switch { case isV2Beta1ContentType(responseContentType): var aggregatedDiscovery apidiscovery.APIGroupDiscoveryList err = json.Unmarshal(body, &aggregatedDiscovery) if err != nil { return nil, nil, nil, err } apiGroupList, resourcesByGV, failedGVs = SplitGroupsAndResources(aggregatedDiscovery) default: // Default is unaggregated discovery v1. </ins> var v metav1.APIVersions err = json.Unmarshal(body, &v) if err != nil {


Question: } } <ins> func TestInClusterClientConfigPrecedence(t *testing.T) { tt := []struct { overrides *ConfigOverrides }{ { overrides: &ConfigOverrides{ ClusterInfo: clientcmdapi.Cluster{ Server: "https://host-from-overrides.com", }, }, }, { overrides: &ConfigOverrides{ AuthInfo: clientcmdapi.AuthInfo{ Token: "https://host-from-overrides.com", }, }, }, { overrides: &ConfigOverrides{ ClusterInfo: clientcmdapi.Cluster{ CertificateAuthority: "/path/to/ca-from-overrides.crt", }, }, }, { overrides: &ConfigOverrides{ ClusterInfo: clientcmdapi.Cluster{ Server: "https://host-from-overrides.com", }, AuthInfo: clientcmdapi.AuthInfo{ Token: "https://host-from-overrides.com", }, }, }, { overrides: &ConfigOverrides{ ClusterInfo: clientcmdapi.Cluster{ Server:               "https://host-from-overrides.com", CertificateAuthority: "/path/to/ca-from-overrides.crt", }, }, }, { overrides: &ConfigOverrides{ ClusterInfo: clientcmdapi.Cluster{ CertificateAuthority: "/path/to/ca-from-overrides.crt", }, AuthInfo: clientcmdapi.AuthInfo{ Token: "https://host-from-overrides.com", }, }, }, { overrides: &ConfigOverrides{ ClusterInfo: clientcmdapi.Cluster{ Server:               "https://host-from-overrides.com", CertificateAuthority: "/path/to/ca-from-overrides.crt", }, AuthInfo: clientcmdapi.AuthInfo{ Token: "https://host-from-overrides.com", }, }, }, { overrides: &ConfigOverrides{}, }, } for _, tc := range tt { expectedServer := "https://host-from-cluster.com" expectedToken := "token-from-cluster" expectedCAFile := "/path/to/ca-from-cluster.crt" icc := &inClusterClientConfig{ inClusterConfigProvider: func() (*restclient.Config, error) { return &restclient.Config{ Host:        expectedServer, BearerToken: expectedToken, TLSClientConfig: restclient.TLSClientConfig{ CAFile: expectedCAFile, }, }, nil }, overrides: tc.overrides, } clientConfig, err := icc.ClientConfig() if err != nil { t.Fatalf("Unxpected error: %v", err) } if overridenServer := tc.overrides.ClusterInfo.Server; len(overridenServer) > 0 { expectedServer = overridenServer } if overridenToken := tc.overrides.AuthInfo.Token; len(overridenToken) > 0 { expectedToken = overridenToken } if overridenCAFile := tc.overrides.ClusterInfo.CertificateAuthority; len(overridenCAFile) > 0 { expectedCAFile = overridenCAFile } if clientConfig.Host != expectedServer { t.Errorf("Expected server %v, got %v", expectedServer, clientConfig.Host) } if clientConfig.BearerToken != expectedToken { t.Errorf("Expected token %v, got %v", expectedToken, clientConfig.BearerToken) } if clientConfig.TLSClientConfig.CAFile != expectedCAFile { t.Errorf("Expected Certificate Authority %v, got %v", expectedCAFile, clientConfig.TLSClientConfig.CAFile) } } } </ins> func matchBoolArg(expected, got bool, t *testing.T) { if expected != got { t.Errorf("Expected %v, got %v", expected, got)

Found in openshift, but I'm pretty sure the codepaths are the same. That means the token is getting ignored and in-cluster config is being used in preference to flags.
Is a similar issue?
Yep, kind of regression.
Sigh.
Yes, most likely the same cause.
I'm on it.
Unit test :)
Probably will want to cherrypick this into 1.5.
WIP in


Question: // The name of this port. All ports in an EndpointSlice must have a unique // name. If the EndpointSlice is dervied from a Kubernetes service, this // corresponds to the Service.ports[].name. <del> // Name must either be an empty string or pass IANA_SVC_NAME validation: // * must be no more than 15 characters long // * may contain only [-a-z0-9] // * must contain at least one letter [a-z] // * it must not start or end with a hyphen, nor contain adjacent hyphens </del> <ins> // Name must either be an empty string or pass DNS_LABEL validation: // * must be no more than 63 characters long. // * must consist of lower case alphanumeric characters or '-'. // * must start and end with an alphanumeric character. </ins> Name *string // The IP protocol for this port. // Must be UDP, TCP, or SCTP.

What happened: No endpoints created in the EndpointSlice for my service due to the port on the service being too long. Logs show: EndpointSlice created has no endpoint: What you expected to happen: Service that works with Endpoints should work with EndpointSlice. If this is intentional, it should be clearly documented How to reproduce it (as minimally and precisely as possible): Create a service with a long port name Environment: Kubernetes version (use ): 1.16 Cloud provider or hardware configuration: kind
/sig networking
The label(s) cannot be applied. These labels are supported: <detailsIn response to : /sig networking Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig network These SIGs are my best guesses for this issue. Please comment if I am incorrect about one. I am a bot run by vllry.
For a rough measure of impact, Istio had 5 ports that were over this limit (and many more that were right at 15 characters), so I suspect it may not be terribly rare.
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
This needs to be resolved before endpointslices graduate from alpha
/assign
/remove-triage unresolved


Question: docs/man/man1/kubectl-alpha-diff.1 docs/man/man1/kubectl-alpha.1 docs/man/man1/kubectl-annotate.1 <ins> docs/man/man1/kubectl-api-resources.1 </ins> docs/man/man1/kubectl-api-versions.1 docs/man/man1/kubectl-apply-edit-last-applied.1 docs/man/man1/kubectl-apply-set-last-applied.1

As the RBAC role need to be related to resources. I think we could use a command to get the supported resources. ref to
This would be cool.
I thought we a command that listed resources in the discover doc a long time ago.
is just the versions, but principle is the same. We want to list all the types, their resources, and tie that back to "explain".
What's the best way to determine this information currently? I'm having difficulty figuring out the exact set of permissions (and how they would be written in an RBAC role) for any given kubectl command. The documentation for this right now is split across multiple places and certain things are not explained at all. Specifically, it's difficult to determine which "API verb" is used for which operations. Take as an example. It's unclear if the way to allow this is to use the "proxy" subresource of a pod or service with "get"/"create"/etc. verbs, or to use the "proxy" verb with the resource (pod, service) directly. is a similar case. The almost works for this purpose, but it's still difficult. Operations for a given resource are grouped into categories like write, read, status, and proxy, but these things don't obviously correlate with API verbs. Again, proxy is especially confusing as there is no mention of a proxy "verb". The API operations are only shown using their HTTP verbs. One thing that could help on this document would be a section on the right column under "kubectl command" that shows exactly the rule needed in an RBAC role to allow this command via kubectl. And even then, that assumes an exact 1-to-1 mapping between API endpoints and kubectl commands, which I doubt is always the case. (For example, port fowarding to a pod seems to require both a GET on the pod resource and a CREATE on a port forward subresource.) There also needs to be better documentation about how this information could have been determined in the first place. I've resorted to manually looking through API requests made with but even then it's difficult to figure out. Amusingly, I was the one who wrote which is referenced in the original issue description (as well as the table explaining verbs in ).


Question: docs/man/man1/kubectl-alpha-diff.1 docs/man/man1/kubectl-alpha.1 docs/man/man1/kubectl-annotate.1 <ins> docs/man/man1/kubectl-api-resources.1 </ins> docs/man/man1/kubectl-api-versions.1 docs/man/man1/kubectl-apply-edit-last-applied.1 docs/man/man1/kubectl-apply-set-last-applied.1

tl;dr: It's too difficult right now to go from "I want a user to be able to run " to "here is the RBAC rule to use for that" for any case where there isn't a specific example shown in the docs.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
It seem that relative PR stop updated. If this issue still valuable, i will help fix it
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close


Question: maxStorageAccounts                     = 100 // max # is 200 (250 with special request). this allows 100 for everything else including stand alone disks maxDisksPerStorageAccounts             = 60 storageAccountUtilizationBeforeGrowing = 0.5 <ins> // Disk Caching is not supported for disks 4 TiB and larger // https://docs.microsoft.com/en-us/azure/virtual-machines/premium-storage-performance#disk-caching diskCachingLimit = 4096 // GiB </ins> maxLUN               = 64 // max number of LUNs per VM errLeaseFailed       = "AcquireDiskLeaseFailed"

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: there is a disk resize scenario: In the beginning, it’s 3TiB with cachingMode=ReadOnly, and then disk detached, resized to 4TiB, then user find they could not attach that disk back to any agent node, I think that’s a bug we need to fix. What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /kind bug /assign /priority important-soon /sig cloud-provider /area provider/azure
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: // list all pods to include the pods that don't match the rs`s selector // anymore but has the stale controller ref. // TODO: Do the List and Filter in a single pass, or use an index. <del> pods, err := rsc.podLister.Pods(rs.Namespace).List(labels.Everything()) </del> <ins> allPods, err := rsc.podLister.Pods(rs.Namespace).List(labels.Everything()) </ins> if err != nil { return err } // Ignore inactive pods. var filteredPods []*v1.Pod <del> for _, pod := range pods { </del> <ins> for _, pod := range allPods { </ins> if controller.IsPodActive(pod) { filteredPods = append(filteredPods, pod) }

https://k8s- Failed: [] Network Partition [Disruptive] [Slow] [] [ReplicationController] should eagerly create replacement pod during network partition when termination grace is non-zero {Kubernetes e2e suite}
https://k8s- seems to be the first failure here. It seems to have gone in with a bunch of RC/RS changes () cc/


Question: statuses[container.Name] = status } <ins> // Copy the slice before sorting it containerStatusesCopy := make([]*kubecontainer.Status, len(podStatus.ContainerStatuses)) copy(containerStatusesCopy, podStatus.ContainerStatuses) </ins> // Make the latest container status comes first. <del> sort.Sort(sort.Reverse(kubecontainer.SortContainerStatusesByCreationTime(podStatus.ContainerStatuses))) </del> <ins> sort.Sort(sort.Reverse(kubecontainer.SortContainerStatusesByCreationTime(containerStatusesCopy))) </ins> // Set container statuses according to the statuses seen in pod status containerSeen := map[string]int{} <del> for _, cStatus := range podStatus.ContainerStatuses { </del> <ins> for _, cStatus := range containerStatusesCopy { </ins> cName := cStatus.Name if _, ok := statuses[cName]; !ok { // This would also ignore the infra container.

What happened: One pod stuck at CreateContainerError and can't get recovered automatically. Kubelet log shows that the requested name (let's say Container2) is already in use. shows the Container2 Running, and it's created before Container1 This should never happen. How to reproduce it (as minimally and precisely as possible): I wish i knew. Anything else we need to know?: I know this can be caused by time shifting, but I didn't find any time shifting sign in system log and chrony log, and there is no rebooting between the create timestamp of Container1 and Container2, so this should be caused by something else. This happens intermittently in our env, like 3 times in half a year, and I didn't find any ime shifting sign each time. Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.6", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-11-13T11:20:18Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.6", GitCommit:"", GitTreeState:"clean", BuildDate:"2020-08-28T11:18:26Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: OS (e.g: ): CentOS Linux release 7.5.1804 (Core) Kernel (e.g. ): Linux node-1 3.10.0-957.21.3.el7.x8664 SMP Wed Jul 17 06:57:50 UTC 2019 x8664 x8664 x86_64 GNU/Linux Install tools: kubeadm Network plugin and version (if this is a network-related bug): Others:
/sig node
Any update on this issue, we are also facing the same issue. EKS version : 1.14.9 Docker version: 19.03.6-ce AWS CNI: 1.6.2 Intermittently we are also facing this , what might the issue, could please guide us here? Below are the kubelet logs. )"


Question: statuses[container.Name] = status } <ins> // Copy the slice before sorting it containerStatusesCopy := make([]*kubecontainer.Status, len(podStatus.ContainerStatuses)) copy(containerStatusesCopy, podStatus.ContainerStatuses) </ins> // Make the latest container status comes first. <del> sort.Sort(sort.Reverse(kubecontainer.SortContainerStatusesByCreationTime(podStatus.ContainerStatuses))) </del> <ins> sort.Sort(sort.Reverse(kubecontainer.SortContainerStatusesByCreationTime(containerStatusesCopy))) </ins> // Set container statuses according to the statuses seen in pod status containerSeen := map[string]int{} <del> for _, cStatus := range podStatus.ContainerStatuses { </del> <ins> for _, cStatus := range containerStatusesCopy { </ins> cName := cStatus.Name if _, ok := statuses[cName]; !ok { // This would also ignore the infra container.

with CreatePodSandboxError: "CreatePodSandbox for pod "-f0fe-11ea-9872-)" failed: rpc error: code = Unknown desc = failed to create a sandbox for pod "aiqx-server-dep--b9lmv": operation timeout: context deadline exceeded" Sep 07 11:40:40 kubelet[2899]: W0907 11:40:40. 2899 ] No ref for container "docker://7b9c86b529530f15d941f8e68973a202c8f348ab5db864a8371e8c82ddbf9dc4" (r(-e845-11ea-bd96-) Sep 07 11:40:43 kubelet[2899]: W0907 11:40:43. 2899 ] Unable to create pod sandbox due to conflict. Attempting to remove sandbox "a73bd9a03d56bc7855e4e89e163e13c5e7173397f10edcc200fb686e1b6885c8" Sep 07 11:40:43 kubelet[2899]: E0907 11:40:43. 2899 ] Failed to remove the conflicting container "a73bd9a03d56bc7855e4e89e163e13c5e7173397f10edcc200fb686e1b6885c8": Error: No such container: a73bd9a03d56bc7855e4e89e163e13c5e7173397f10edcc200fb686e1b6885c8
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or .


Question: statuses[container.Name] = status } <ins> // Copy the slice before sorting it containerStatusesCopy := make([]*kubecontainer.Status, len(podStatus.ContainerStatuses)) copy(containerStatusesCopy, podStatus.ContainerStatuses) </ins> // Make the latest container status comes first. <del> sort.Sort(sort.Reverse(kubecontainer.SortContainerStatusesByCreationTime(podStatus.ContainerStatuses))) </del> <ins> sort.Sort(sort.Reverse(kubecontainer.SortContainerStatusesByCreationTime(containerStatusesCopy))) </ins> // Set container statuses according to the statuses seen in pod status containerSeen := map[string]int{} <del> for _, cStatus := range podStatus.ContainerStatuses { </del> <ins> for _, cStatus := range containerStatusesCopy { </ins> cName := cStatus.Name if _, ok := statuses[cName]; !ok { // This would also ignore the infra container.

/lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-contributor-experience at . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: "k8s.io/apimachinery/pkg/util/diff" "k8s.io/apimachinery/pkg/util/intstr" "k8s.io/apimachinery/pkg/util/sets" <ins> "k8s.io/kubernetes/federation/apis/federation" </ins> "k8s.io/kubernetes/pkg/api" "k8s.io/kubernetes/pkg/api/testapi" "k8s.io/kubernetes/pkg/apis/apps"

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: After labeling clusters in a Federation, when you type the field is empty. What you expected to happen: The fiels should contain information about the labels in the cluster How to reproduce it (as minimally and precisely as possible): a federation control pane in any Cloud Provider and any DNS provider your cluster: info aboyut the cluster with the flag: Note: if you describe it, you see the labels being : Environment: Kubernetes version (use ): 1.8.0 Cloud provider or hardware configuration**: anyone OS (e.g. from /etc/os-release): Ubuntu 16.04 Kernel (e.g. ): Linux Install tools: curl Others:
/sig multicluster
I'm working on it.


Question: }, } if strings.Contains(err.Error(), "0/3 nodes are available") { <del> if strings.Contains(err.Error(), "2 NodeUnderDiskPressure") && strings.Contains(err.Error(), "1 NodeUnderMemoryPressure") { </del> <ins> if strings.Contains(err.Error(), "2 node(s) had disk pressure") && strings.Contains(err.Error(), "1 node(s) had memory pressure") { </ins> return } }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind feature The is not user friendly. There has been to make it better, but we still print in the pending message. We should change these errors to a more human readable message. For example, should be changed to something like . /sig scheduling


Question: } } <ins> if utilfeature.DefaultFeatureGate.Enabled(features.HonorPVReclaimPolicy) { if metav1.HasAnnotation(volume.ObjectMeta, storagehelpers.AnnMigratedTo) { // CSI migration scenario - do not depend on in-tree plugin return nil, nil } if volume.Spec.CSI != nil { // CSI volume source scenario - external provisioner is requested return nil, nil } } </ins> // The plugin that provisioned the volume was not found or the volume // was not dynamically provisioned. Try to find a plugin by spec. spec := vol.NewSpecFromPersistentVolume(volume, false)

In my CSI controller I have enabled the feature gate HonorPVReclaimPolicy for csi-provisioner. Based on the documentation Finalizers can be on a PersistentVolume to ensure that PersistentVolumes having Delete reclaim policy are deleted only after the backing storage are deleted. The newly introduced finalizers and external- are only to dynamically provisioned volumes. I can see the finalizer set on the PV while creating static Persistent Volume. PV deletion is stuck due to this finalizer apiVersion: v1 kind: PersistentVolume metadata: creationTimestamp: "2023-11-08T11:39:24Z" deletionGracePeriodSeconds: 0 deletionTimestamp: "2023-11-08T11:54:31Z" finalizers: external- name: csi-infoscale-pv PV should be deleted when PVC is deleted. Persistent Volume yaml apiVersion: v1 kind: PersistentVolume metadata: name: csi-infoscale-pv spec: storageClassName: csi-infoscale-sc-prod persistentVolumeReclaimPolicy: Delete capacity: storage: 2Gi accessModes: ReadWriteOnce csi: driver: org.veritas.infoscale # Please provide pre-provisioned Infoscale volume name. volumeHandle: clust21432/vrtskube_dg-/vol1 fsType: vxfs Persistent Volume Claim apiVersion: v1 kind: PersistentVolumeClaim metadata: name: csi-infoscale-pvc-static spec: accessModes: ReadWriteOnce resources: requests: storage: 2Gi storageClassName: csi-infoscale-sc-prod volumeName: csi-infoscale-pv Same thing works fine on OCP. WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short. Use --output=yaml|json to get the full version.


Question: } } <ins> if utilfeature.DefaultFeatureGate.Enabled(features.HonorPVReclaimPolicy) { if metav1.HasAnnotation(volume.ObjectMeta, storagehelpers.AnnMigratedTo) { // CSI migration scenario - do not depend on in-tree plugin return nil, nil } if volume.Spec.CSI != nil { // CSI volume source scenario - external provisioner is requested return nil, nil } } </ins> // The plugin that provisioned the volume was not found or the volume // was not dynamically provisioned. Try to find a plugin by spec. spec := vol.NewSpecFromPersistentVolume(volume, false)

Client Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.5", GitCommit:"", GitTreeState:"clean", BuildDate:"2023-08-24T00:48:26Z", GoVersion:"go1.20.7", Compiler:"gc", Platform:"linux/amd64"} Kustomize Version: v5.0.1 Server Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.5", GitCommit:"", GitTreeState:"clean", BuildDate:"2023-08-24T00:42:11Z", GoVersion:"go1.20.7", Compiler:"gc", Platform:"linux/amd64"} Vanilla K8s setup <details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage maybe similar issues should go in
Opened similar issue in external-provisioner repo also There is no reply on that as well. Can you please help understand the behavior of the feature HonorPVReclaimPolicy.
If you want Persistent Volume (PV) to be deleted when the associated Persistent Volume Claim (PVC) is deleted, you donot need to use finalizers explicitly. Kubernetes handles this behavior by default. It is ok that if you use to get the desire output using Finalizer we can do it.
As per my understanding Please follow the steps : Add finalizer in metadata of as you did above need to make change in the controller which is handling PV deletion in .go file. See whether the controller administrator is working or not. I think it will work.
Finalizer is because the feature gate HonorPVReclaimPolicy for csi-provisioner is enabled. We are not adding the finalizer explicitly. From the documentation, "The newly introduced finalizers and external- are only to dynamically provisioned volumes." Not sure why is the finalizer getting set.


Question: } } <ins> if utilfeature.DefaultFeatureGate.Enabled(features.HonorPVReclaimPolicy) { if metav1.HasAnnotation(volume.ObjectMeta, storagehelpers.AnnMigratedTo) { // CSI migration scenario - do not depend on in-tree plugin return nil, nil } if volume.Spec.CSI != nil { // CSI volume source scenario - external provisioner is requested return nil, nil } } </ins> // The plugin that provisioned the volume was not found or the volume // was not dynamically provisioned. Try to find a plugin by spec. spec := vol.NewSpecFromPersistentVolume(volume, false)

Also, for deleting PV it requires the annotation "" but this annotation is set in case of dynamic PVs. Should it be set explicitly for static PV?
This feature is alpha so the feature gate is not enabled by default. Did you enable the feature gate on kube-controller-manager?
Yes, i have enabled this feature.
external- is also applied to statically provisioned volumes. FYI
Hi Which version of the external provisioner is the driver using? Does the driver meet the requirements of the CSI spec on the DeleteVolume PRC? FYI: Is this problem occasional or permanently reproducible? Can you provide the provisioner logs at that time?
/triage needs-information
cc
cc
External provisioner used is : "csi-provisioner:v3.5.0" This problem is always reproducible Provisioner logs: I1123 12:22:38. 1 ] handleProtectionFinalizer Volume : &PersistentVolume{ObjectMeta:{csi-infoscale-pv -420a-4e9e-be82- 0 2023-11-23 12:21:18 +0000 UTC <nil<nilmap[] map[{"apiVersion":"v1","kind":"PersistentVolume","metadata":{"annotations":{},"name":"csi-infoscale-pv"},"spec":{"accessModes":["ReadWriteOnce"],"capacity":{"storage":"2Gi"},"csi":{"driver":"org.veritas.infoscale","fsType":"vxfs","volumeHandle":"clust40559/vrtskubedg-/vol1"},"persistentVolumeReclaimPolicy":"Delete","storageClassName":"csi-infoscale-sc-prod"}} ] [] [ external-] [{kubectl-client-side-apply Update v1 2023-11-23 12:21:18 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"":{}}},"f:spec":{"f:accessModes":{},"f:capacity":{".":{},"f:storage":{}},"f:csi":{".


Question: } } <ins> if utilfeature.DefaultFeatureGate.Enabled(features.HonorPVReclaimPolicy) { if metav1.HasAnnotation(volume.ObjectMeta, storagehelpers.AnnMigratedTo) { // CSI migration scenario - do not depend on in-tree plugin return nil, nil } if volume.Spec.CSI != nil { // CSI volume source scenario - external provisioner is requested return nil, nil } } </ins> // The plugin that provisioned the volume was not found or the volume // was not dynamically provisioned. Try to find a plugin by spec. spec := vol.NewSpecFromPersistentVolume(volume, false)

":{},"f:driver":{},"f:fsType":{},"f:volumeHandle":{}},"f:persistentVolumeReclaimPolicy":{},"f:storageClassName":{},"f:volumeMode":{}}} } {csi-provisioner Update v1 2023-11-23 12:21:22 +0000 UTC FieldsV1 {"f:metadata":{"f:finalizers":{"v:"external-"":{}}}} } {kube-controller-manager Update v1 2023-11-23 12:21:22 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"":{}}},"f:spec":{"f:claimRef":{".


Question: } } <ins> if utilfeature.DefaultFeatureGate.Enabled(features.HonorPVReclaimPolicy) { if metav1.HasAnnotation(volume.ObjectMeta, storagehelpers.AnnMigratedTo) { // CSI migration scenario - do not depend on in-tree plugin return nil, nil } if volume.Spec.CSI != nil { // CSI volume source scenario - external provisioner is requested return nil, nil } } </ins> // The plugin that provisioned the volume was not found or the volume // was not dynamically provisioned. Try to find a plugin by spec. spec := vol.NewSpecFromPersistentVolume(volume, false)

":{},"f:apiVersion":{},"f:kind":{},"f:name":{},"f:namespace":{},"f:resourceVersion":{},"f:uid":{}}}} } {kube-controller-manager Update v1 2023-11-23 12:22:38 +0000 UTC FieldsV1 {"f:status":{"f:phase":{}}} status}]},Spec:PersistentVolumeSpec{Capacity:ResourceList{storage: {{ 0} {<nil} 2Gi BinarySI},},PersistentVolumeSource:PersistentVolumeSource{GCEPersistentDisk:nil,AWSElasticBlockStore:nil,HostPath:nil,Glusterfs:nil,NFS:nil,RBD:nil,ISCSI:nil,Cinder:nil,CephFS:nil,FC:nil,Flocker:nil,FlexVolume:nil,AzureFile:nil,VsphereVolume:nil,Quobyte:nil,AzureDisk:nil,PhotonPersistentDisk:nil,PortworxVolume:nil,ScaleIO:nil,Local:nil,StorageOS:nil,CSI:&CSIPersistentVolumeSource{Driver:org.veritas.infoscale,VolumeHandle:clust40559/vrtskubedg-/vol1,ReadOnly:false,FSType:vxfs,VolumeAttributes:map[string]string{},ControllerPublishSecretRef:nil,NodeStageSecretRef:nil,NodePublishSecretRef:nil,ControllerExpandSecretRef:nil,NodeExpandSecretRef:nil,},},AccessModes:[ReadWriteOnce],ClaimRef:&ObjectReference{Kind:PersistentVolumeClaim,Namespace:default,Name:csi-infoscale-pvc-static,UID:-59a5-4f02-83ac-,APIVersion:v1,ResourceVersion:,FieldPath:,},PersistentVolumeReclaimPolicy:Delete,StorageClassName:csi-infoscale-sc-prod,MountOptions:[],VolumeMode:Filesystem,NodeAffinity:nil,},Status:PersistentVolumeStatus{Phase:Released,Message:,Reason:,},} I1123 12:22:38. 1 ] delete "csi-infoscale-pv": started I1123 12:22:38. 1 ] GRPC call: /csi.v1.Controller/DeleteVolume I1123 12:22:38. 1 ] GRPC request: {"volumeid":"clust40559/vrtskubedg-/vol1"} I1123 12:22:39.


Question: } } <ins> if utilfeature.DefaultFeatureGate.Enabled(features.HonorPVReclaimPolicy) { if metav1.HasAnnotation(volume.ObjectMeta, storagehelpers.AnnMigratedTo) { // CSI migration scenario - do not depend on in-tree plugin return nil, nil } if volume.Spec.CSI != nil { // CSI volume source scenario - external provisioner is requested return nil, nil } } </ins> // The plugin that provisioned the volume was not found or the volume // was not dynamically provisioned. Try to find a plugin by spec. spec := vol.NewSpecFromPersistentVolume(volume, false)

1 ] GRPC response: {} I1123 12:22:39. 1 ] GRPC error: <nilI1123 12:22:39. 1 ] delete "csi-infoscale-pv": volume deleted I1123 12:22:39. 1 ] delete "csi-infoscale-pv": failed to remove finalizer for persistentvolume: Operation cannot be fulfilled on persistentvolumes "csi-infoscale-pv": the object has been modified; please apply your changes to the latest version and try again W1123 12:22:39. 1 ] Retrying syncing volume "csi-infoscale-pv", failure 0 E1123 12:22:39. 1 ] error syncing volume "csi-infoscale-pv": Operation cannot be fulfilled on persistentvolumes "csi-infoscale-pv": the object has been modified; please apply your changes to the latest version and try again I1123 12:22:39. 1 ] handleProtectionFinalizer Volume : &PersistentVolume{ObjectMeta:{csi-infoscale-pv -420a-4e9e-be82- 0 2023-11-23 12:21:18 +0000 UTC 2023-11-23 12:22:38 +0000 UTC 0xc0007f63d0 map[] map[{"apiVersion":"v1","kind":"PersistentVolume","metadata":{"annotations":{},"name":"csi-infoscale-pv"},"spec":{"accessModes":["ReadWriteOnce"],"capacity":{"storage":"2Gi"},"csi":{"driver":"org.veritas.infoscale","fsType":"vxfs","volumeHandle":"clust40559/vrtskubedg-/vol1"},"persistentVolumeReclaimPolicy":"Delete","storageClassName":"csi-infoscale-sc-prod"}} ] [] [external-] [{kubectl-client-side-apply Update v1 2023-11-23 12:21:18 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"":{}}},"f:spec":{"f:accessModes":{},"f:capacity":{".":{},"f:storage":{}},"f:csi":{".


Question: } } <ins> if utilfeature.DefaultFeatureGate.Enabled(features.HonorPVReclaimPolicy) { if metav1.HasAnnotation(volume.ObjectMeta, storagehelpers.AnnMigratedTo) { // CSI migration scenario - do not depend on in-tree plugin return nil, nil } if volume.Spec.CSI != nil { // CSI volume source scenario - external provisioner is requested return nil, nil } } </ins> // The plugin that provisioned the volume was not found or the volume // was not dynamically provisioned. Try to find a plugin by spec. spec := vol.NewSpecFromPersistentVolume(volume, false)


Question: } } <ins> if utilfeature.DefaultFeatureGate.Enabled(features.HonorPVReclaimPolicy) { if metav1.HasAnnotation(volume.ObjectMeta, storagehelpers.AnnMigratedTo) { // CSI migration scenario - do not depend on in-tree plugin return nil, nil } if volume.Spec.CSI != nil { // CSI volume source scenario - external provisioner is requested return nil, nil } } </ins> // The plugin that provisioned the volume was not found or the volume // was not dynamically provisioned. Try to find a plugin by spec. spec := vol.NewSpecFromPersistentVolume(volume, false)

":{},"f:apiVersion":{},"f:kind":{},"f:name":{},"f:namespace":{},"f:resourceVersion":{},"f:uid":{}}}} } {kube-controller-manager Update v1 2023-11-23 12:22:38 +0000 UTC FieldsV1 {"f:status":{"f:message":{},"f:phase":{}}} status}]},Spec:PersistentVolumeSpec{Capacity:ResourceList{storage: {{ 0} {<nil} 2Gi BinarySI},},PersistentVolumeSource:PersistentVolumeSource{GCEPersistentDisk:nil,AWSElasticBlockStore:nil,HostPath:nil,Glusterfs:nil,NFS:nil,RBD:nil,ISCSI:nil,Cinder:nil,CephFS:nil,FC:nil,Flocker:nil,FlexVolume:nil,AzureFile:nil,VsphereVolume:nil,Quobyte:nil,AzureDisk:nil,PhotonPersistentDisk:nil,PortworxVolume:nil,ScaleIO:nil,Local:nil,StorageOS:nil,CSI:&CSIPersistentVolumeSource{Driver:org.veritas.infoscale,VolumeHandle:clust40559/vrtskubedg-/vol1,ReadOnly:false,FSType:vxfs,VolumeAttributes:map[string]string{},ControllerPublishSecretRef:nil,NodeStageSecretRef:nil,NodePublishSecretRef:nil,ControllerExpandSecretRef:nil,NodeExpandSecretRef:nil,},},AccessModes:[ReadWriteOnce],ClaimRef:&ObjectReference{Kind:PersistentVolumeClaim,Namespace:default,Name:csi-infoscale-pvc-static,UID:-59a5-4f02-83ac-,APIVersion:v1,ResourceVersion:,FieldPath:,},PersistentVolumeReclaimPolicy:Delete,StorageClassName:csi-infoscale-sc-prod,MountOptions:[],VolumeMode:Filesystem,NodeAffinity:nil,},Status:PersistentVolumeStatus{Phase:Failed,Message:error getting deleter volume plugin for volume "csi-infoscale-pv": no deletable volume plugin matched,Reason:,},} I1123 12:22:40.


Question: } } <ins> if utilfeature.DefaultFeatureGate.Enabled(features.HonorPVReclaimPolicy) { if metav1.HasAnnotation(volume.ObjectMeta, storagehelpers.AnnMigratedTo) { // CSI migration scenario - do not depend on in-tree plugin return nil, nil } if volume.Spec.CSI != nil { // CSI volume source scenario - external provisioner is requested return nil, nil } } </ins> // The plugin that provisioned the volume was not found or the volume // was not dynamically provisioned. Try to find a plugin by spec. spec := vol.NewSpecFromPersistentVolume(volume, false)

1 ] handleProtectionFinalizer Volume : &PersistentVolume{ObjectMeta:{csi-infoscale-pv -420a-4e9e-be82- 0 2023-11-23 12:21:18 +0000 UTC 2023-11-23 12:22:38 +0000 UTC 0xc0007f63d0 map[] map[{"apiVersion":"v1","kind":"PersistentVolume","metadata":{"annotations":{},"name":"csi-infoscale-pv"},"spec":{"accessModes":["ReadWriteOnce"],"capacity":{"storage":"2Gi"},"csi":{"driver":"org.veritas.infoscale","fsType":"vxfs","volumeHandle":"clust40559/vrtskubedg-/vol1"},"persistentVolumeReclaimPolicy":"Delete","storageClassName":"csi-infoscale-sc-prod"}} ] [] [external-] [{kubectl-client-side-apply Update v1 2023-11-23 12:21:18 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"":{}}},"f:spec":{"f:accessModes":{},"f:capacity":{".":{},"f:storage":{}},"f:csi":{".":{},"f:driver":{},"f:fsType":{},"f:volumeHandle":{}},"f:persistentVolumeReclaimPolicy":{},"f:storageClassName":{},"f:volumeMode":{}}} } {csi-provisioner Update v1 2023-11-23 12:21:22 +0000 UTC FieldsV1 {"f:metadata":{"f:finalizers":{"v:"external-"":{}}}} } {kube-controller-manager Update v1 2023-11-23 12:21:22 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"":{}}},"f:spec":{"f:claimRef":{".


Question: } } <ins> if utilfeature.DefaultFeatureGate.Enabled(features.HonorPVReclaimPolicy) { if metav1.HasAnnotation(volume.ObjectMeta, storagehelpers.AnnMigratedTo) { // CSI migration scenario - do not depend on in-tree plugin return nil, nil } if volume.Spec.CSI != nil { // CSI volume source scenario - external provisioner is requested return nil, nil } } </ins> // The plugin that provisioned the volume was not found or the volume // was not dynamically provisioned. Try to find a plugin by spec. spec := vol.NewSpecFromPersistentVolume(volume, false)

":{},"f:apiVersion":{},"f:kind":{},"f:name":{},"f:namespace":{},"f:resourceVersion":{},"f:uid":{}}}} } {kube-controller-manager Update v1 2023-11-23 12:22:38 +0000 UTC FieldsV1 {"f:status":{"f:message":{},"f:phase":{}}} status}]},Spec:PersistentVolumeSpec{Capacity:ResourceList{storage: {{ 0} {<nil} 2Gi BinarySI},},PersistentVolumeSource:PersistentVolumeSource{GCEPersistentDisk:nil,AWSElasticBlockStore:nil,HostPath:nil,Glusterfs:nil,NFS:nil,RBD:nil,ISCSI:nil,Cinder:nil,CephFS:nil,FC:nil,Flocker:nil,FlexVolume:nil,AzureFile:nil,VsphereVolume:nil,Quobyte:nil,AzureDisk:nil,PhotonPersistentDisk:nil,PortworxVolume:nil,ScaleIO:nil,Local:nil,StorageOS:nil,CSI:&CSIPersistentVolumeSource{Driver:org.veritas.infoscale,VolumeHandle:clust40559/vrtskubedg-/vol1,ReadOnly:false,FSType:vxfs,VolumeAttributes:map[string]string{},ControllerPublishSecretRef:nil,NodeStageSecretRef:nil,NodePublishSecretRef:nil,ControllerExpandSecretRef:nil,NodeExpandSecretRef:nil,},},AccessModes:[ReadWriteOnce],ClaimRef:&ObjectReference{Kind:PersistentVolumeClaim,Namespace:default,Name:csi-infoscale-pvc-static,UID:-59a5-4f02-83ac-,APIVersion:v1,ResourceVersion:,FieldPath:,},PersistentVolumeReclaimPolicy:Delete,StorageClassName:csi-infoscale-sc-prod,MountOptions:[],VolumeMode:*Filesystem,NodeAffinity:nil,},Status:PersistentVolumeStatus{Phase:Failed,Message:error getting deleter volume plugin for volume "csi-infoscale-pv": no deletable volume plugin matched,Reason:,},}
Thank you I will try to fix it. /remove-triage needs-information /assign


Question: } func (s *containerScope) Admit(pod *v1.Pod) lifecycle.PodAdmitResult { <del> // Exception - Policy : none if s.policy.Name() == PolicyNone { return s.admitPolicyNone(pod) } </del> for _, container := range append(pod.Spec.InitContainers, pod.Spec.Containers...) { bestHint, admit := s.calculateAffinity(pod, &container) klog.InfoS("Best TopologyHint", "bestHint", bestHint, "pod", klog.KObj(pod), "containerName", container.Name)

Follow up from: See details there. When is , there is no need to discover the NUMA topology. /sig node /priority backlog /kind feature /help-wanted
/cc
/triage accepted
I would like to take this up also can I get more brief information about it? code review comments from asking that inside the NewManager() implementation, we need to skip NewPolicyOptions and NewNUMAInfo but when the NumaTopology comes kubelet issue also comes. I need some more information to start with this!
/cc
When there is no topology manger policy specified, NewManager should be a no-op. Just skip all checks.
Exactly. Just to add: we may need few iterations over the change to get a clean flow in the call.
/assign


Question: Name: "foo"}}, api.Namespace{ ObjectMeta: api.ObjectMeta{ <ins> Name: "foo"}, Status: api.NamespaceStatus{ Phase: api.NamespaceTerminating, }, }, false}, {api.Namespace{ ObjectMeta: api.ObjectMeta{ Name: "foo"}}, api.Namespace{ ObjectMeta: api.ObjectMeta{ </ins> Name: "bar"}, Status: api.NamespaceStatus{ Phase: api.NamespaceTerminating,

There is a funny subtle behvaiour in get namespaces, in which if i call more than one time, it seems that the removal from Terminating state is accelarated. It seems this might indicate a bug, or maybe its not a bug and the eventual semantics of is a little funny sometimes. in any case, here's what we are seeing : 1) when i delete a namespace the first time usning kubectl, we get a state change to . that is correct, of course. 2) but ... if make the call a second time, then the same status line is immediately removed from . Expected behaviour : A second call to "kubectl delete ..." shouldn't have any effect on the timing of removal of a namespace from 'Terminating' state*... So i guess there is some "eventual consistency" subtle bug that is resolved by repeated calls to
suggests this may not be e bug, maybe its just the etcd eventual deleter thingy.
Looks to definitely be a bug. The 2nd delete seems to have interrupted the delete process and artifacts (like events) from the namespace still exist in etcd.
investigating...
oops, see the error, will submit a fix.
I have a fix, but need a test case, will submit PR tomorrow AM


Question: Rule:       admissionv1beta1.Rule{APIGroups: []string{""}, APIVersions: []string{"v1"}, Resources: []string{"pods"}}, }}, ObjectSelector:          webhook.objectSelector, <ins> NamespaceSelector:       &metav1.LabelSelector{MatchLabels: nsLabels}, </ins> FailurePolicy:           &fail, ReinvocationPolicy:      webhook.policy, AdmissionReviewVersions: []string{"v1beta1"}, }) } <ins> // Register a marker checking webhook with each set of webhook configurations markerEndpoint := webhookServer.URL + "/marker" webhooks = append(webhooks, admissionv1beta1.MutatingWebhook{ Name: "admission.integration.test.marker", ClientConfig: admissionv1beta1.WebhookClientConfig{ URL:      &markerEndpoint, CABundle: localhostCert, }, Rules: []admissionv1beta1.RuleWithOperations{{ Operations: []admissionv1beta1.OperationType{admissionv1beta1.OperationAll}, Rule:       admissionv1beta1.Rule{APIGroups: []string{""}, APIVersions: []string{"v1"}, Resources: []string{"pods"}}, }}, NamespaceSelector:       &metav1.LabelSelector{MatchLabels: markerNsLabels}, ObjectSelector:          &metav1.LabelSelector{MatchLabels: map[string]string{"marker": "true"}}, AdmissionReviewVersions: []string{"v1beta1"}, }) </ins> cfg, err := client.AdmissionregistrationV1beta1().MutatingWebhookConfigurations().Create(&admissionv1beta1.MutatingWebhookConfiguration{ ObjectMeta: metav1.ObjectMeta{Name: fmt.Sprintf("admission.integration.test-%d", i)},

Which jobs are failing: Which test(s) are failing: Since when has it been failing: Testgrid link: Reason for failure: Anything else we need to know: /sig api-machinery /assign
/cc
want to take a look at the audit event issue first? I can dig in after that if issues remain.
yes, I'm looking at the audit event issue now
actually the failure message suggests that the missing audit log is caused by unexpected admission invocation: I think it's unrelated to auditing Note that there are two tests named "no reinvocation of webhook B when in-tree or prior webhook mutations" :( and from the audit log (), it also suggests that the test is hitting a wrong endpoint
Thanks I'll dig in.


Question: } func (az *Cloud) isCurrentInstance(name types.NodeName, metadataVMName string) (bool, error) { <ins> var err error </ins> nodeName := mapNodeNameToVMName(name) <ins> // VMSS vmName is not same with hostname, use hostname instead. </ins> if az.VMType == vmTypeVMSS { <del> // VMSS vmName is not same with hostname, construct the node name "{computer-name-prefix}{base-36-instance-id}". // Refer https://docs.microsoft.com/en-us/azure/virtual-machine-scale-sets/virtual-machine-scale-sets-instance-ids#scale-set-vm-computer-name. if ssName, instanceID, err := extractVmssVMName(metadataVMName); err == nil { instance, err := strconv.ParseInt(instanceID, 10, 64) if err != nil { return false, fmt.Errorf("failed to parse VMSS instanceID %q: %v", instanceID, err) } metadataVMName = fmt.Sprintf("%s%06s", ssName, strconv.FormatInt(instance, 36)) </del> <ins> metadataVMName, err = os.Hostname() if err != nil { return false, err } // Use name from env variable "NODE_NAME" if it is set. nodeNameEnv := os.Getenv(nodeNameEnvironmentName) if nodeNameEnv != "" { metadataVMName = nodeNameEnv </ins> } }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: If computer name prefix does not match VMSS name, then Azure cloud provider will use VMSS API instead of using metadata service data. This leads to throttling, because every affected kubelet is making API requests. What you expected to happen: Azure cloud provider should use metadata service data instead of calling VMSS API when instance is current. How to reproduce it (as minimally and precisely as possible): Set computer name prefix of the VMSS to something different than default. You should observe Azure provider making requests to the VMSS API rather than using metadata service, which leads to throttling if you have multiple kubelets with this bug. Anything else we need to know?: This is bug was introduced by - returns false even if instance is current, because it expects node name to match VMSS name + instance id, however node name is derived from the computer-name-prefix which can be different than VMSS name. Node name should be matched against , not against . Environment: Kubernetes version (use ): 1.18.x and higher Cloud provider or hardware configuration: azure /sig cloud-provider
/sig cloud-provider
/area provider/azure
CC
/remove-sig cloud-provider /sig cloud-provider (trying to remove the label)
/sig cloud-provider
Thanks for the reporting. Unfortunately, computer name prefix is not included in IMDS, so it can't be used without VMSS API for constructing the NodeName. Would check what could we do to workaround the issue.
Can you memorize the computer name prefix in the first request to the Azure API and then use it for the isCurrentInstance check?
opened to fix the issue. Would you like to have a look?
I can confirm that your change should fix the problem I reported. Do you know if your fix will be to 1.18.x?
is working on the cherry pick
Thank you for quick fix!


Question: Expect(lineCount).To(Equal(expectedLineCount), "Line count of the written file should be %d.", expectedLineCount) <del> // Verify the pod is scheduled in the other zone. By("verifying the pod is scheduled in a different zone.") var otherZone string if cloudZones[0] == podZone { otherZone = cloudZones[1] } else { otherZone = cloudZones[0] </del> <ins> } func addTaint(c clientset.Interface, ns string, nodes []v1.Node, podZone string) (removeTaint func()) { reversePatches := make(map[string][]byte) for _, node := range nodes { oldData, err := json.Marshal(node) Expect(err).NotTo(HaveOccurred()) node.Spec.Taints = append(node.Spec.Taints, v1.Taint{ Key:    taintKeyPrefix + ns, Value:  podZone, Effect: v1.TaintEffectNoSchedule, }) newData, err := json.Marshal(node) Expect(err).NotTo(HaveOccurred()) patchBytes, err := strategicpatch.CreateTwoWayMergePatch(oldData, newData, v1.Node{}) Expect(err).NotTo(HaveOccurred()) reversePatchBytes, err := strategicpatch.CreateTwoWayMergePatch(newData, oldData, v1.Node{}) Expect(err).NotTo(HaveOccurred()) reversePatches[node.Name] = reversePatchBytes _, err = c.CoreV1().Nodes().Patch(node.Name, types.StrategicMergePatchType, patchBytes) Expect(err).ToNot(HaveOccurred()) </ins> } <del> nodeName = pod.Spec.NodeName node, err = c.CoreV1().Nodes().Get(nodeName, metav1.GetOptions{}) Expect(err).ToNot(HaveOccurred()) newPodZone := node.Labels[apis.LabelZoneFailureDomain] Expect(newPodZone).To(Equal(otherZone), "The pod should be scheduled in zone %s after all nodes in zone %s have been deleted", otherZone, podZone) </del> <ins> return func() { for nodeName, reversePatch := range reversePatches { _, err := c.CoreV1().Nodes().Patch(nodeName, types.StrategicMergePatchType, reversePatch) Expect(err).ToNot(HaveOccurred()) } } </ins> } func testRegionalDelayedBinding(c clientset.Interface, ns string) {

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: Testgrid: https://k8s-
/assign
The test's assumption about test parameter is wrong. Previously it assumed the group name is the same across all zones, but they are different in regional clusters. Looking into using node taints as the permanent fix, as suggested by the TODO comment in the test.
Hm I think taint based evictions is still an alpha feature.
OK. I propose the following solution: Get the zone of Schedule the StatefulSet pod to that zone using node affinity; wait for the pod to run Remove the node affinity from StatefulSet Delete
Can it be simplified? Schedule the pod Figure out which zone that pod got scheduled in Delete the instance group of that zone
Unfortunately I don't know of a good way to look up the instance group given the zone, since there could be instance groups from multiple clusters in the same project
How about: Schedule the pod Figure out the zone Taint all the nodes in the zone Kill the pod
Taint with NoSchedule? yeah that sounds good


Question: ginkgo.BeforeEach(func() { c = f.ClientSet ns = f.Namespace.Name <ins> dc = f.DynamicClient </ins> }) ginkgo.It("deployment reaping should cascade to its replica sets and pods", func() {

[x] Original approval issue: [x] Test pr: [x] Two weeks soak end date: 19 November 2020 This issue is created to allow edit by /wip /hold According to this APIsnoop query, there are still some remaining Deployment endpoints which are untested. a Deployment with a static label the Deployment with a new Label and updated data the Deployment to ensure it's patched all Deployments in all Namespaces find the Deployment(1) ensure that the Deployment is found and is patched Namespaced Deployment(1) via a Collection with a LabelSelector Discover useragents: List endpoints hit by the test: Display endpoint coverage change: If a test with these calls gets merged, test coverage will go up by 5 points* This test is also created with the goal of conformance promotion.
/sig testing /sig architecture /area conformance
BobyMcbobs commented in original Issue: Unsure if the endpoint replaceAppsV1NamespacedDeploymentStatus is able to be hit. When this endpoint is attempted to be updated the data after retrieving the update resource doesn't reflect the values in the update
/assign
When using the appv1.Deployment structure instead of maps of interfaces to build the structure in a patch of the Deployment, this error comes up. Still in progress yment-static":"true"}: does not match template , spec.selector: Invalid value: "null": field is immutable]
replaceAppsV1NamespacedDeploymentStatus to test. Increase cover by 1 Endpoint
/reopen PR reverted by due to flaky test
Reopened this issue. <detailsIn response to : /reopen PR reverted by due to flaky test Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/reopen Soaking, ready for promotion 19 November 2020
Reopened this issue. <detailsIn response to : /reopen Soaking, ready for promotion 19 November 2020 Instructions for interacting with me using PR comments are available .


Question: ginkgo.BeforeEach(func() { c = f.ClientSet ns = f.Namespace.Name <ins> dc = f.DynamicClient </ins> }) ginkgo.It("deployment reaping should cascade to its replica sets and pods", func() {

If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-lifecycle stale
/triage accepted
/merged


Question: func (i *Instances) NodeAddresses(name string) ([]api.NodeAddress, error) { glog.V(4).Infof("NodeAddresses(%v) called", name) <del> srv, err := getServerByName(i.compute, name) </del> <ins> addrs, err := getAddressesByName(i.compute, name) </ins> if err != nil { return nil, err } <del> addrs := []api.NodeAddress{} for _, addr := range findAddrs(srv.Addresses["private"]) { addrs = append(addrs, api.NodeAddress{ Type:    api.NodeInternalIP, Address: addr, }) } for _, addr := range findAddrs(srv.Addresses["public"]) { addrs = append(addrs, api.NodeAddress{ Type:    api.NodeExternalIP, Address: addr, }) } // AccessIPs are usually duplicates of "public" addresses. api.AddToNodeAddresses(&addrs, api.NodeAddress{ Type:    api.NodeExternalIP, Address: srv.AccessIPv6, }, api.NodeAddress{ Type:    api.NodeExternalIP, Address: srv.AccessIPv4, }, ) </del> glog.V(4).Infof("NodeAddresses(%v) => %v", name, addrs) return addrs, nil }

From we can find that, hardcode "private" and "public", seems only support that network with such name. , it fetch the first ip. usually, it can include both v6 and v4 ip. what's the meaning to just fetch the first one ? is it v4 or v6 ?
ping and
Yes. Zoom out a level or two in the code, and you'll see we're trying to either a) setup an openstack loadbalancer to point to the minion node (a single address) using , or b) report to kubernetes all the addresses we know about the node and whether they are "internal" or "external" in . All we have at the starting point is the server name (from kubernetes), and we need to make up the rest of this information somehow. (I agree (a) should reuse the NodeAddresses() function from (b), but the latter is a newer function and the code hasn't yet been refactored in that way. This doesn't really change the substance of this discussion.) Re (1): The logic as currently implemented is: Assume any network named "private" is "internal" Assume any network named "public" is "external" Assume the AccessIPs (IPv4 and IPv6) are "external" Any other network addresses are ignored. I implemented the first two ("private" and "public") following the Rackspace public cloud precedent, and on the basis that users in other OpenStack clouds can use these names if they want to encourage an address other than the Access addresses. For the (a) use case, we just take the first address we find in order of the above; for (b) we report that full list. I preferred "private" to "public" since (again, at least on Rackspace public cloud), the traffic on "private" network is sometimes cheaper/free. I agree this isn't great, and please suggest something better if you have a use case in mind. Perhaps we should pull a list of network names that should be considered "internal" and "external" from the provider config? Perhaps I could assume that provider networks are public and non-provider networks are internal? I don't want to add too much configuration here if there's a good algorithm that will Just Work.


Question: func (i *Instances) NodeAddresses(name string) ([]api.NodeAddress, error) { glog.V(4).Infof("NodeAddresses(%v) called", name) <del> srv, err := getServerByName(i.compute, name) </del> <ins> addrs, err := getAddressesByName(i.compute, name) </ins> if err != nil { return nil, err } <del> addrs := []api.NodeAddress{} for _, addr := range findAddrs(srv.Addresses["private"]) { addrs = append(addrs, api.NodeAddress{ Type:    api.NodeInternalIP, Address: addr, }) } for _, addr := range findAddrs(srv.Addresses["public"]) { addrs = append(addrs, api.NodeAddress{ Type:    api.NodeExternalIP, Address: addr, }) } // AccessIPs are usually duplicates of "public" addresses. api.AddToNodeAddresses(&addrs, api.NodeAddress{ Type:    api.NodeExternalIP, Address: srv.AccessIPv6, }, api.NodeAddress{ Type:    api.NodeExternalIP, Address: srv.AccessIPv4, }, ) </del> glog.V(4).Infof("NodeAddresses(%v) => %v", name, addrs) return addrs, nil }

I think in practice it's probably ok to continue to prefer "internal" to "external" for loadbalancer setup, but I can contrive hypothetical cases that need more flexibility here too. Again, if you have a concrete need for such a thing, please file a bug so we have some idea of the flexibility required. Complicating all this is that Kubernetes never actually defined what an "internal" address was. just says "Generally the IP address of the node that is routable only within the cluster" which isn't a sufficiently strict definition. Looking at where Kubernetes actually uses it: just uses the first address reported (preferring InternalIPs to ExternalIPs) and the Kubernetes master configures an "SSH Tunnel Check" which just uses the first ExternalIP ... which kind of makes it pointless to ever even bother reporting anything beyond the first address. In both cases, it's assumed that this address is reachable by the Kubernetes master nodes, which makes the Internal vs External distinction a bit pointless. Re (2): is meant to return more than one IP, and the loadbalancer setup can only use a single IP for each node (or would unnecessarily track/healthceck multiple paths to the same backend for multiple addresses). If you're seeing single addresses returned by then you should file a bug. All the above treats IPv4 and IPv6 as equally important, if the OpenStack cloud has both, and the "first" address used for the loadbalancer will be an arbitrary choice between the two protocols (not quite true: will use AccessIPv4 before AccessIPv6). I think that's how it should be (the rest of the code should be protocol agnostic), but I wouldn't be surprised if there's some part of Kubernetes that doesn't support IPv6 properly. There certainly used to be lots of places that assumed IPv4 in Kubernetes, and GCE doesn't support IPv6 iirc so it gets less testing - if you've found something that breaks with IPv6, please file bugs. Now this bug: what would you like done here? Would you like the current situation to be documented further? Is the "private"/"public" naming thing too restrictive for your usecase? Something else?


Question: func (i *Instances) NodeAddresses(name string) ([]api.NodeAddress, error) { glog.V(4).Infof("NodeAddresses(%v) called", name) <del> srv, err := getServerByName(i.compute, name) </del> <ins> addrs, err := getAddressesByName(i.compute, name) </ins> if err != nil { return nil, err } <del> addrs := []api.NodeAddress{} for _, addr := range findAddrs(srv.Addresses["private"]) { addrs = append(addrs, api.NodeAddress{ Type:    api.NodeInternalIP, Address: addr, }) } for _, addr := range findAddrs(srv.Addresses["public"]) { addrs = append(addrs, api.NodeAddress{ Type:    api.NodeExternalIP, Address: addr, }) } // AccessIPs are usually duplicates of "public" addresses. api.AddToNodeAddresses(&addrs, api.NodeAddress{ Type:    api.NodeExternalIP, Address: srv.AccessIPv6, }, api.NodeAddress{ Type:    api.NodeExternalIP, Address: srv.AccessIPv4, }, ) </del> glog.V(4).Infof("NodeAddresses(%v) => %v", name, addrs) return addrs, nil }

Hi, if we talked about openstack, I expect it not bind to rackspace, although, the kubernetes called RK SDK, But from my understanding, seems RK itself SDK not bind network to "private" and "pubic". User can define public network with their names, like site-network, for example. or "testnet", And for private, it should be seem. For nova list, the addresses, it can have "OS-EXT-IPS:type":"fixed" or "OS-EXT-IPS:type":"floating" for floating, I think what you means "Public", for fixed, I think its what you means "Private". In other words, we'd better to find related information from other fields in nova or neutron infromation. Instead of bind to "private" and "public" name.
I this issue and wrote a patch that implements what suggests. I will send a PR soon
Glad to see some improvements here. And wish to use it if implemented


Question: "fmt" "net/http" "net/http/httptest" <ins> "net/url" </ins> "reflect" "testing" <ins> "k8s.io/apimachinery/pkg/util/sets" </ins> ) func TestInstallHandler(t *testing.T) {

What would you like to be : Add support to the kube-apiserver so that we can disable health-checks via query params. Why is this needed: Currently, the healthz endpoint is inflexible in that any and all health checks are currently executed when the endpoint is hit. Since the kube-apiserver only currently supports a liveness check, this means all persistent health check failures will eventually result in a kubelet driven restart. However, this may not be desirable behavior if you have additional monitoring services layered on top of your control plane. For instance, take the etcd failure from the perspective of the kube-apiserver. Currently, persistent failures during the etcd health check from the api-server healthz endpoint will cause the kube-apiserver to be restarted. This sometimes helps (in the case of a faulty etcd-client connection) but also can be detrimental (the kube-apiserver restarts until the etcd cluster is functioning, then the kube-apiserver bombards the etcd cluster with requests to update local caches, saturating the etcd cluster with requests, causing it to timeout on its own liveness probes and trigger a kubelet-driven restart). Let's say that we could now disable etcd from our liveness health check. Then if we had an external monitoring service, we could ask independently ask etcd if it was healthy and we could also ask the kube-apiserver if it thought etcd was unhealthy. Based on the combination of those answers, we could enable more intelligent responses to certain situation, like restarting the kube-apiserver when it thought etcd was unhealthy but etcd was actually reporting as healthy but not doing so when etcd was actually unhealthy. This feature is backwards-compatible and opt-in. There would be no impact to existing behavior unless these query params are actually used. /sig api-machinery /kind feature
/cc
SGTM
/assign


Question: # Get the compute zone zone=${ZONE:-"$(gcloud info --format='value(config.properties.compute.zone.value)')"} <ins> zone=${zone// /} </ins> if [[ ${zone} == "" ]]; then echo "Could not find gcloud compute/zone when running: `gcloud info --format='value(config.properties.compute.zone.value)'`" exit 1

The compute project and zone in the prerequisites check are incorrectly calculated: !!! Error in hack/make-rules/test-e2e- Error in hack/make-rules/test-e2e- 'tee -i "${artifacts}/build-"' exited with status 1 0 Call stack: 1: hack/make-rules/test-e2e- main(...) Exiting with status 1 blocked by the following errors in each case: 1- Could not find gcloud compute/zone when running: .... 2- Could not find gcloud project when running: ..... 1- gcloud config unset compute/zone 2- gcloud config unset project 3- make test-e2e-node REMOTE=true LIST_IMAGES=true gcloud --version Google Cloud SDK 453.0.0 alpha 2023.10.27 beta 2023.10.27 bq 2.0.98 core 2023.10.27 gcloud-crc32c 1.0.0 <details</details<details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node /assign


Question: Searches: []string{testHostDomain}, }, nil } <del> // getResolvConf returns the hostResolvConf string, which will be used to get the Host's DNS configuration. func getResolvConf(t *testing.T) (string, func()) { return hostResolvConf, func() {} } </del>

When a --resolv-conf (DNS resolver config) is passed to kubelet on a Windows node, the config content is used as part of sandbox initialization (through getHostDNSConfig()). Now with the recent changes (), it seems now to fail with an error like: E0315 10:22:16. 4764 ] "Error syncing pod, skipping" err="failed to "CreatePodSandbox" for "img-pull-(-b40a-459d-9033-)" with CreatePodSandboxError: "Failed to generate sandbox config for pod "img-pull-(-b40a-459d-9033-)": Unexpected resolver config value: "C:etckubernetescni". Expected "" or "Host"."" pod="img-puller-3842/img-pull-" podUID=-b40a-459d-9033- This is showing on all the runs of GCE testgrid (latest): Reading/using the passed resolv config on Windows Run a Windows pool with DNS resolv config being set as part of kubelet params. No response_ 27 alpha.3 1.27 beta.0 master <detailsGCE </details<details</details<details</details<details</details<details</details
/triage accepted /milestone v1.27 /sig node /sig network
/sig windows /cc
/priority criticial-urgent as this is likely a release blocking
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /priority criticial-urgent Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/priority critical-urgent
Hmm, so you're setting the [1]. This PR introduced the capability to read the DNS config options from the Windows host registries while removing the capability to read from a file, introducing a regression. I've sent a PR addressing this issue: [1]
Is the file you are pointing to a UNIX-style or something else?
Was the file changing the behavior of the node?


Question: Searches: []string{testHostDomain}, }, nil } <del> // getResolvConf returns the hostResolvConf string, which will be used to get the Host's DNS configuration. func getResolvConf(t *testing.T) (string, func()) { return hostResolvConf, func() {} } </del>

Yeah, what used on GCE/GKE, seems to be a Linux-like conf, eg: nameserver 10.128.0.1 search c.<project_name.internal google.internal ` We haven't tried to remove from kubelet to see the behavior.
This Issue/PR has been marked as release blocker. Please have the fix merged by eod today otherwise the feature has to be reverted. For the Issue/PR owner: please have a revert/feature disable PR open today in case the fix failed to get in. Thank you! See the -- Kubernetes 1.27 release manager
I opened up a revert PR in case this PR does not merge by code freeze.
Thanks for the resolution! both 2019 and 2022 grids seems to be back to green now!
Thanks for confirming! I opened to ensure gets updated too


Question: } if h.nodeCacheCapable && result.NodeNames != nil { <del> nodeResult = make([]*v1.Node, 0, len(*result.NodeNames)) for i := range *result.NodeNames { nodeResult = append(nodeResult, nodeNameToInfo[(*result.NodeNames)[i]].Node()) </del> <ins> nodeResult = make([]*v1.Node, len(*result.NodeNames)) for i, nodeName := range *result.NodeNames { if node, ok := nodeNameToInfo[nodeName]; ok { nodeResult[i] = node.Node() } else { return nil, nil, fmt.Errorf( "extender %q claims a filtered node %q which is not found in nodeNameToInfo map", h.extenderURL, nodeName) } </ins> } } else if result.Nodes != nil { <del> nodeResult = make([]*v1.Node, 0, len(result.Nodes.Items)) </del> <ins> nodeResult = make([]*v1.Node, len(result.Nodes.Items)) </ins> for i := range result.Nodes.Items { <del> nodeResult = append(nodeResult, &result.Nodes.Items[i]) </del> <ins> nodeResult[i] = &result.Nodes.Items[i] </ins> } }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Default scheduler crash if scheduler extender filter returns a not found node What you expected to happen: Default scheduler just should consider the not found node as not an candidate, instead of crash itself (panic). How to reproduce it (as minimally and precisely as possible): cluster only have 1 node: 0.0.0.1 scheduler extender (with nodeCacheCapable enabled) filter accidentally returns a candidate node: 0.0.0.2 scheduler recieved the candidate nodes: 0.0.0.2 from scheduler extender, but got below error: The is SuggestedHost: filteredNodes[0].Name: Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig scheduling


Question: return nil } <del> policyGroupVersion, err := CheckEvictionSupport(d.Client) if err != nil { return err } </del> // TODO(justinsb): unnecessary? getPodFn := func(namespace, name string) (*corev1.Pod, error) { return d.Client.CoreV1().Pods(namespace).Get(name, metav1.GetOptions{}) } <del> if len(policyGroupVersion) > 0 { return d.evictPods(pods, policyGroupVersion, getPodFn) </del> <ins> if !d.DisableEviction { policyGroupVersion, err := CheckEvictionSupport(d.Client) if err != nil { return err } if len(policyGroupVersion) > 0 { return d.evictPods(pods, policyGroupVersion, getPodFn) } </ins> } return d.deletePods(pods, getPodFn)

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Allow to have an additional flag (e.g. or ) to allow a to drain pods off a node, even if they violate PodDisruptionBudgets Why is this needed: There are times where you want to drain pods off a node even if violating a PDB. For example, if you are running a one-worker node kubernetes cluster (not minikube), and you want to upgrade it. Or, like me, you are running a test cluster and want to scale down to zero nodes when you're not going to be using the cluster for a while in order to save costs. If that worker node has a deployment with a PDB, then will get stuck as it cannot get rid of the pod while respecting the PDB, which is expected; but in this case, it is a hindrance. Adding the switch will allow users to explicitly acknowledge that they want to ignore the PDBs (and accept the corresponding downtime for an application/service)
/sig node
+1
+1 Our company offers Kubernetes as PaaS. We manage Kubernetes, including IaaS OS updates and if a customer does not specify the PDB correctly, we are not able to drain the node for the OS updates. A flag would be werry helpfull for our use case.
+1
/active
I have created to address this. This is a pretty simple implementation, but I think it achieves the desired results. A suitable workflow would be: 1) Configure you component to use drain as normal 2) After some timeout, run drain with --disable-eviction option.
I'll wait until the PR is merged and released before testing, but if I understand right, it would require me do this, right? If so, I think it would be better if we did it a different way and have one call, something like this: Where implies that if the drain is still running after this time, switch to the mode without having to call kubectl again
I think that would require a sizeable refactor or add a great deal of complexity to the code, the same could be achieved with In practical terms, I'm focused on automatic node remediation via destroying the node and getting a new one. The timeout would be very large, the drain library is blocking, so I want it to fail quickly and requeue.


Question: return nil } <del> policyGroupVersion, err := CheckEvictionSupport(d.Client) if err != nil { return err } </del> // TODO(justinsb): unnecessary? getPodFn := func(namespace, name string) (*corev1.Pod, error) { return d.Client.CoreV1().Pods(namespace).Get(name, metav1.GetOptions{}) } <del> if len(policyGroupVersion) > 0 { return d.evictPods(pods, policyGroupVersion, getPodFn) </del> <ins> if !d.DisableEviction { policyGroupVersion, err := CheckEvictionSupport(d.Client) if err != nil { return err } if len(policyGroupVersion) > 0 { return d.evictPods(pods, policyGroupVersion, getPodFn) } </ins> } return d.deletePods(pods, getPodFn)

I'm going to be performing the calculation of when it's appropriate to evict/delete outside of the drain library, and I suspect most will be as well, so there's not really a great incentive to do it in the drain library itself IMO.
That's all good, I have no issue using the flag as long as does error out and return a non-zero code as expected (there's been a few times where it seems to error out, but still return 0)


Question: // shouldRetryHTTPRequest determines if the request is retriable. func shouldRetryHTTPRequest(resp *http.Response, err error) bool { if resp != nil { <del> // HTTP 412 (StatusPreconditionFailed) means etag mismatch, hence we shouldn't retry. if resp.StatusCode == http.StatusPreconditionFailed { </del> <ins> // HTTP 412 (StatusPreconditionFailed) means etag mismatch // HTTP 400 (BadRequest) means the request cannot be accepted, hence we shouldn't retry. if resp.StatusCode == http.StatusPreconditionFailed || resp.StatusCode == http.StatusBadRequest { </ins> return false }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: If you create a service that reference a Basic IP in a standard load balancer cluster, the service will stuck at "Pending" state for days without detailed error from Azure response. When you describe the service, it only has following: What you expected to happen: In this case Azure responded 400 with detailed error message that the referenced IP is a basic IP and cannot be used on standard LB. I would expect the detailed error message show up as events for easier diagnostic. How to reproduce it (as minimally and precisely as possible): Create a service that reference a Basic IP in a standard load balancer cluster Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/assign /area provider/azure /sig cloud-provider
/area cloudprovider


Question: return expected.ret } <del> func (testcase *testcase) DiskIsAttached(diskName string, nodeName types.NodeName) (bool, error) { </del> <ins> func (testcase *testcase) DiskIsAttached(diskName string, nodeName types.NodeName) (bool, string, error) { </ins> expected := &testcase.diskIsAttached if expected.diskName == "" && expected.nodeName == "" { // testcase.diskIsAttached looks uninitialized, test did not expect to // call DiskIsAttached testcase.t.Errorf("Unexpected DiskIsAttached call!") <del> return false, errors.New("Unexpected DiskIsAttached call!") </del> <ins> return false, diskName, errors.New("Unexpected DiskIsAttached call!") </ins> } if expected.diskName != diskName { testcase.t.Errorf("Unexpected DiskIsAttached call: expected diskName %s, got %s", expected.diskName, diskName) <del> return false, errors.New("Unexpected DiskIsAttached call: wrong diskName") </del> <ins> return false, diskName, errors.New("Unexpected DiskIsAttached call: wrong diskName") </ins> } if expected.nodeName != nodeName { testcase.t.Errorf("Unexpected DiskIsAttached call: expected nodeName %s, got %s", expected.nodeName, nodeName) <del> return false, errors.New("Unexpected DiskIsAttached call: wrong nodeName") </del> <ins> return false, diskName, errors.New("Unexpected DiskIsAttached call: wrong nodeName") </ins> } klog.V(4).Infof("DiskIsAttached call: %s, %s, returning %v, %v", diskName, nodeName, expected.isAttached, expected.ret) <del> return expected.isAttached, expected.ret </del> <ins> return expected.isAttached, diskName, expected.ret </ins> } func (testcase *testcase) DisksAreAttached(nodeVolumes map[types.NodeName][]string) (map[types.NodeName]map[string]bool, error) {

When user creates a static PV using folder name in path: on vSAN backed datastore then detach disk does not work because check fails - The code expects folder name to be UUID but it does not and hence it determines disk is not attached to the node and hence no detach is ever performed. /sig storage cc


Question: package dns <del> import ( "fmt" "os" runtimeapi "k8s.io/cri-api/pkg/apis/runtime/v1" "k8s.io/klog/v2" ) func getHostDNSConfig(resolverConfig string) (*runtimeapi.DNSConfig, error) { var hostDNS, hostSearch, hostOptions []string // Get host DNS settings if resolverConfig != "" { f, err := os.Open(resolverConfig) if err != nil { klog.ErrorS(err, "Could not open resolv conf file.") return nil, err } defer f.Close() hostDNS, hostSearch, hostOptions, err = parseResolvConf(f) if err != nil { err := fmt.Errorf("Encountered error while parsing resolv conf file. Error: %w", err) klog.ErrorS(err, "Could not parse resolv conf file.") return nil, err } } return &runtimeapi.DNSConfig{ Servers:  hostDNS, Searches: hostSearch, Options:  hostOptions, }, nil } </del> <ins> // Read the DNS configuration from a resolv.conf file. var getHostDNSConfig = getDNSConfig </ins>


Question: package dns <del> import ( "fmt" "os" runtimeapi "k8s.io/cri-api/pkg/apis/runtime/v1" "k8s.io/klog/v2" ) func getHostDNSConfig(resolverConfig string) (*runtimeapi.DNSConfig, error) { var hostDNS, hostSearch, hostOptions []string // Get host DNS settings if resolverConfig != "" { f, err := os.Open(resolverConfig) if err != nil { klog.ErrorS(err, "Could not open resolv conf file.") return nil, err } defer f.Close() hostDNS, hostSearch, hostOptions, err = parseResolvConf(f) if err != nil { err := fmt.Errorf("Encountered error while parsing resolv conf file. Error: %w", err) klog.ErrorS(err, "Could not parse resolv conf file.") return nil, err } } return &runtimeapi.DNSConfig{ Servers:  hostDNS, Searches: hostSearch, Options:  hostOptions, }, nil } </del> <ins> // Read the DNS configuration from a resolv.conf file. var getHostDNSConfig = getDNSConfig </ins>


Question: return nil } <del> // Ensures that the load balancer associated with the given service is deleted, // doing the deletion if necessary. Should always be retried upon failure. func (s *ServiceController) ensureLBDeleted(service *api.Service) error { // This is only needed because not all delete load balancer implementations // are currently idempotent to the LB not existing. if _, exists, err := s.balancer.GetTCPLoadBalancer(s.loadBalancerName(service), s.zone.Region); err != nil { return err } else if !exists { return nil } if err := s.balancer.DeleteTCPLoadBalancer(s.loadBalancerName(service), s.zone.Region); err != nil { return err } return nil } </del> // ListKeys implements the interface required by DeltaFIFO to list the keys we // already know about. func (s *serviceCache) ListKeys() []string {

All of our external load balancer tests failed for the better part of a day due to the GCE project that our tests run in hitting its quota for number of target pools. While the target pools have been deleted, the fact that we hit our quota indicates that there must be a leak somewhere, and we need to plug it.
For reference, the previous issue was
Could the leakage be related to ? The quota exhaustion occurred about 22 hrs after that.
I'd guess that this is due to GetTCPLoadBalancer returning that the load balancer doesn't exist if the forwarding rule is gone, ignoring the target pool, while the service controller considers its deleting done if the cloud provider says that the load balancer doesn't exist. I that logic to the cloud provider because the other implementations of DeleteTCPLoadBalancer weren't/won't be idempotent in the sense of returning ok if the LB doesn't exist. To fix it, we'll either need to make sure all Delete implementations are idempotent and remove the Get call in the controller, or add a method along the lines of IsTCPLoadBalancerDeleted to ask the cloud provider whether all pieces of it have been completely cleaned up. We can't make GetTCPLoadBalancer check that because it's relied on to tell us whether or not the balancer is fully created.
Copying over my commentary from on why the GKE e2e tests are leaking so many forwarding rules and target pools vs the GCE tests:
Is there a way for the services test to wait for the service controller to clean up the external resources before finishing (without just doing a sleep)?
Not without polling GCE directly for whether the resources still exist, although that may be worth it as a workaround. What would be perfect is that clean-up-all-cloud-provider-resources API method that we talked about a couple months ago.
Can we not perhaps use GCE namespaces or projects as a cheap way of getting this here?
I don't believe so. We can't use projects due to the fact that projects can't be programatically created (they're intended to be long-lived). I'm not aware of anything along the lines of a GCE namespace other than networks, which won't work because target pools and forwarding rules unfortunately don't seem to be scoped to networks.