Question:
Cluster api provides a framework to run the e2e tests which uses the conformance image: The gmsa tests use bash to install the webhook: The test will fail when run in the conformance image because of the bash: we should install the component another way run this e2e conformance image with /sig windows fyi <details</details<details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Conformance image:
Hi does the "work e2e" test case pass? Deploying GMSA webhook is the same for both "work e2e" and this test case.
no, they both fail due to bash not being present in the conformance image. The gmsa tests themselves pass if I run locally
/sig testing

} return strings.Contains(output, "This is a test file.") }, 1*time.Minute, 1*time.Second).Should(gomega.BeTrue()) }) }) })


Question:
Cluster api provides a framework to run the e2e tests which uses the conformance image: The gmsa tests use bash to install the webhook: The test will fail when run in the conformance image because of the bash: we should install the component another way run this e2e conformance image with /sig windows fyi <details</details<details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Conformance image:
Hi does the "work e2e" test case pass? Deploying GMSA webhook is the same for both "work e2e" and this test case.
no, they both fail due to bash not being present in the conformance image. The gmsa tests themselves pass if I run locally
/sig testing

// deployGmsaWebhook deploys the GMSA webhook, and returns a cleanup function // to be called when done with testing, that removes the temp files it's created // on disks as well as the API resources it's created. <del> func deployGmsaWebhook(f *framework.Framework, deployScriptPath string) (func(), error) { cleanUpFunc := func() {} </del> <ins> func deployGmsaWebhook(f *framework.Framework) (func(), error) { deployerName := "webhook-deployer" deployerNamespace := f.Namespace.Name webHookName := "gmsa-webhook" webHookNamespace := deployerNamespace + "-webhook" </ins> <del> tempDir, err := os.MkdirTemp("", "") if err != nil { return cleanUpFunc, fmt.Errorf("unable to create temp dir: %w", err) } </del> <ins> // regardless of whether the deployment succeeded, let's do a best effort at cleanup cleanUpFunc := func() { framework.Logf("Best effort clean up of the webhook:n") stdout, err := framework.RunKubectl("", "delete", "CustomResourceDefinition", "gmsacredentialspecs.windows.k8s.io") framework.Logf("stdout:%snerror:%s", stdout, err) </ins> <del> manifestsFile := path.Join(tempDir, "manifests.yml") name := "gmsa-webhook" namespace := f.Namespace.Name + "-webhook" certsDir := path.Join(tempDir, "certs") </del> <ins> stdout, err = framework.RunKubectl("", "delete", "CertificateSigningRequest", fmt.Sprintf("%s.%s", webHookName, webHookNamespace)) framework.Logf("stdout:%snerror:%s", stdout, err) </ins> <del> // regardless of whether the deployment succeeded, let's do a best effort at cleanup cleanUpFunc = func() { framework.RunKubectl(f.Namespace.Name, "delete", "--filename", manifestsFile) framework.RunKubectl(f.Namespace.Name, "delete", "CustomResourceDefinition", "gmsacredentialspecs.windows.k8s.io") framework.RunKubectl(f.Namespace.Name, "delete", "CertificateSigningRequest", fmt.Sprintf("%s.%s", name, namespace)) os.RemoveAll(tempDir) </del> <ins> stdout, err = runKubectlExecInNamespace(deployerNamespace, deployerName, "--", "kubectl", "delete", "-f", "/manifests.yml") framework.Logf("stdout:%snerror:%s", stdout, err) } // ensure the deployer has ability to approve certificatesigningrequests to install the webhook s := createServiceAccount(f) bindClusterRBACRoleToServiceAccount(f, s, "cluster-admin") installSteps := []string{ "echo "@testing http://dl-cdn.alpinelinux.org/alpine/edge/testing/" >> /etc/apk/repositories", "&& apk add kubectl@testing gettext openssl", "&& apk add --update coreutils", fmt.Sprintf("&& curl %s > gmsa.sh", gmsaWebhookDeployScriptURL), "&& chmod +x gmsa.sh", fmt.Sprintf("&& ./gmsa.sh --file %s --name %s --namespace %s --certs-dir %s --tolerate-master", "/manifests.yml", webHookName, webHookNamespace, "certs"), "&& /agnhost pause", </ins> } <ins> installCommand := strings.Join(installSteps, " ") </ins> <del> cmd := exec.Command("bash", deployScriptPath, "--file", manifestsFile, "--name", name, "--namespace", namespace, "--certs-dir", certsDir, "--tolerate-master") </del> <ins> pod := &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name:      deployerName, Namespace: deployerNamespace, }, Spec: v1.PodSpec{ ServiceAccountName: s, NodeSelector: map[string]string{ "kubernetes.io/os": "linux", }, Containers: []v1.Container{ { Name:    deployerName, Image:   imageutils.GetE2EImage(imageutils.Agnhost), Command: []string{"bash", "-c"}, Args:    []string{installCommand}, }, }, Tolerations: []v1.Toleration{ { Operator: v1.TolerationOpExists, Effect:   v1.TaintEffectNoSchedule, }, }, }, } f.PodClient().CreateSync(pod) </ins> <del> output, err := cmd.CombinedOutput() </del> <ins> // Wait for the Webhook deployment to become ready. The deployer pod takes a few seconds to initialize and create resources err := waitForDeployment(func() (*appsv1.Deployment, error) { return f.ClientSet.AppsV1().Deployments(webHookNamespace).Get(context.TODO(), webHookName, metav1.GetOptions{}) }, 10*time.Second, f.Timeouts.PodStart) </ins> if err == nil { <del> framework.Logf("GMSA webhook successfully deployed, output:n%s", string(output)) </del> <ins> framework.Logf("GMSA webhook successfully deployed") </ins> } else { <del> err = fmt.Errorf("unable to deploy GMSA webhook, output:n%s: %w", string(output), err) </del> <ins> err = fmt.Errorf("GMSA webhook did not become ready: %w", err) </ins> } <ins> // Dump deployer logs logs, _ := e2epod.GetPodLogs(f.ClientSet, deployerNamespace, deployerName, deployerName) framework.Logf("GMSA deployment logs:n%s", logs) </ins> return cleanUpFunc, err }


Question:
Cluster api provides a framework to run the e2e tests which uses the conformance image: The gmsa tests use bash to install the webhook: The test will fail when run in the conformance image because of the bash: we should install the component another way run this e2e conformance image with /sig windows fyi <details</details<details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Conformance image:
Hi does the "work e2e" test case pass? Deploying GMSA webhook is the same for both "work e2e" and this test case.
no, they both fail due to bash not being present in the conformance image. The gmsa tests themselves pass if I run locally
/sig testing

// createServiceAccount creates a service account, and returns its name. func createServiceAccount(f *framework.Framework) string { <del> accountName := f.Namespace.Name + "-sa" </del> <ins> accountName := f.Namespace.Name + "-sa-" + string(uuid.NewUUID()) </ins> account := &v1.ServiceAccount{ ObjectMeta: metav1.ObjectMeta{ Name:      accountName,


Question:
Cluster api provides a framework to run the e2e tests which uses the conformance image: The gmsa tests use bash to install the webhook: The test will fail when run in the conformance image because of the bash: we should install the component another way run this e2e conformance image with /sig windows fyi <details</details<details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Conformance image:
Hi does the "work e2e" test case pass? Deploying GMSA webhook is the same for both "work e2e" and this test case.
no, they both fail due to bash not being present in the conformance image. The gmsa tests themselves pass if I run locally
/sig testing

f.ClientSet.RbacV1().RoleBindings(f.Namespace.Name).Create(context.TODO(), binding, metav1.CreateOptions{}) } <ins> func bindClusterRBACRoleToServiceAccount(f *framework.Framework, serviceAccountName, rbacRoleName string) { binding := &rbacv1.ClusterRoleBinding{ ObjectMeta: metav1.ObjectMeta{ Name:      f.Namespace.Name + "-rbac-binding", Namespace: f.Namespace.Name, }, Subjects: []rbacv1.Subject{ { Kind:      "ServiceAccount", Name:      serviceAccountName, Namespace: f.Namespace.Name, }, }, RoleRef: rbacv1.RoleRef{ APIGroup: "rbac.authorization.k8s.io", Kind:     "ClusterRole", Name:     rbacRoleName, }, } f.ClientSet.RbacV1().ClusterRoleBindings().Create(context.TODO(), binding, metav1.CreateOptions{}) } </ins> // createPodWithGmsa creates a pod using the test GMSA cred spec, and returns its name. func createPodWithGmsa(f *framework.Framework, serviceAccountName string) string { podName := "pod-with-gmsa"


Question:
Cluster api provides a framework to run the e2e tests which uses the conformance image: The gmsa tests use bash to install the webhook: The test will fail when run in the conformance image because of the bash: we should install the component another way run this e2e conformance image with /sig windows fyi <details</details<details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Conformance image:
Hi does the "work e2e" test case pass? Deploying GMSA webhook is the same for both "work e2e" and this test case.
no, they both fail due to bash not being present in the conformance image. The gmsa tests themselves pass if I run locally
/sig testing

package windows import ( <del> "fmt" "io" "net/http" "os" </del> <ins> "time" appsv1 "k8s.io/api/apps/v1" apierrors "k8s.io/apimachinery/pkg/api/errors" "k8s.io/apimachinery/pkg/util/wait" "k8s.io/kubernetes/pkg/controller/deployment/util" "k8s.io/kubernetes/test/e2e/framework" </ins> ) <del> // downloadFile saves a remote URL to a local temp file, and returns its path. // It's the caller's responsibility to clean up the temp file when done. func downloadFile(url string) (string, error) { response, err := http.Get(url) if err != nil { return "", fmt.Errorf("unable to download from %q: %w", url, err) } defer response.Body.Close() tempFile, err := os.CreateTemp("", "") if err != nil { return "", fmt.Errorf("unable to create temp file: %w", err) } defer tempFile.Close() _, err = io.Copy(tempFile, response.Body) return tempFile.Name(), err </del> <ins> // waits for a deployment to be created and the desired replicas // are updated and available, and no old pods are running. func waitForDeployment(getDeploymentFunc func() (*appsv1.Deployment, error), interval, timeout time.Duration) error { return wait.PollImmediate(interval, timeout, func() (bool, error) { deployment, err := getDeploymentFunc() if err != nil { if apierrors.IsNotFound(err) { framework.Logf("deployment not found, continue waiting: %s", err) return false, nil } framework.Logf("error while deploying, error %s", err) return false, err } framework.Logf("deployment status %s", &deployment.Status) return util.DeploymentComplete(deployment, &deployment.Status), nil }) </ins> }


Question:
I create a ResourceQuota with two pods limit in my namespace. Then, I create a statefuleSet with 3 replicas. As expected, two pods are created successfully, and the third pod failed to create. But, the ReadyReplicas is not 2 ,but 1. The ReadyReplicas is correct 1Create a ResourceQuota with two pods limit in a new namespace. 2Create a statefulSet with 3 replicas 3Check if the ReadyReplicas of statefulSet matchs the number of stateful running pods. 1According to the logic of the statefulSet controller, new pod created in current sync is not be immediately counted into the ReadyReplicas. 2Failure to create pod will cause to return error, then skip updating status. <details</details<details</details<detailskind v1.23.3 </details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig apps
/assign

if err != nil { return currentRevision, updateRevision, currentStatus, err } <del> // update the set's status err = ssc.updateStatefulSetStatus(ctx, set, currentStatus) if err != nil { return currentRevision, updateRevision, currentStatus, err } klog.V(4).InfoS("StatefulSet pod status", "statefulSet", klog.KObj(set), "replicas", currentStatus.Replicas, "readyReplicas", currentStatus.ReadyReplicas, "currentReplicas", currentStatus.CurrentReplicas, "updatedReplicas", currentStatus.UpdatedReplicas) </del> klog.V(4).InfoS("StatefulSet revisions", "statefulSet", klog.KObj(set), "currentRevision", currentStatus.CurrentRevision,


Question:
I create a ResourceQuota with two pods limit in my namespace. Then, I create a statefuleSet with 3 replicas. As expected, two pods are created successfully, and the third pod failed to create. But, the ReadyReplicas is not 2 ,but 1. The ReadyReplicas is correct 1Create a ResourceQuota with two pods limit in a new namespace. 2Create a statefulSet with 3 replicas 3Check if the ReadyReplicas of statefulSet matchs the number of stateful running pods. 1According to the logic of the statefulSet controller, new pod created in current sync is not be immediately counted into the ReadyReplicas. 2Failure to create pod will cause to return error, then skip updating status. <details</details<details</details<detailskind v1.23.3 </details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig apps
/assign

currentRevision *apps.ControllerRevision, updateRevision *apps.ControllerRevision, collisionCount int32, <del> pods []*v1.Pod) (*apps.StatefulSetStatus, error) { </del> <ins> pods []*v1.Pod) (statefulSetStatus *apps.StatefulSetStatus, updateErr error) { </ins> // get the current and update revisions of the set. currentSet, err := ApplyRevision(set, currentRevision) if err != nil {


Question:
I create a ResourceQuota with two pods limit in my namespace. Then, I create a statefuleSet with 3 replicas. As expected, two pods are created successfully, and the third pod failed to create. But, the ReadyReplicas is not 2 ,but 1. The ReadyReplicas is correct 1Create a ResourceQuota with two pods limit in a new namespace. 2Create a statefulSet with 3 replicas 3Check if the ReadyReplicas of statefulSet matchs the number of stateful running pods. 1According to the logic of the statefulSet controller, new pod created in current sync is not be immediately counted into the ReadyReplicas. 2Failure to create pod will cause to return error, then skip updating status. <details</details<details</details<detailskind v1.23.3 </details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig apps
/assign

// If the ordinal could not be parsed (ord < 0), ignore the Pod. } <ins> // make sure to update the latest status even if there is an error later defer func() { // update the set's status statusErr := ssc.updateStatefulSetStatus(ctx, set, &status) if statusErr == nil { klog.V(4).InfoS("Updated status", "statefulSet", klog.KObj(set), "replicas", status.Replicas, "readyReplicas", status.ReadyReplicas, "currentReplicas", status.CurrentReplicas, "updatedReplicas", status.UpdatedReplicas) } else if updateErr == nil { updateErr = statusErr } else { klog.V(4).InfoS("Could not update status", "statefulSet", klog.KObj(set), "err", statusErr) } }() </ins> // for any empty indices in the sequence [0,set.Spec.Replicas) create a new Pod at the correct revision for ord := 0; ord < replicaCount; ord++ { if replicas[ord] == nil {


Question:
I create a ResourceQuota with two pods limit in my namespace. Then, I create a statefuleSet with 3 replicas. As expected, two pods are created successfully, and the third pod failed to create. But, the ReadyReplicas is not 2 ,but 1. The ReadyReplicas is correct 1Create a ResourceQuota with two pods limit in a new namespace. 2Create a statefulSet with 3 replicas 3Check if the ReadyReplicas of statefulSet matchs the number of stateful running pods. 1According to the logic of the statefulSet controller, new pod created in current sync is not be immediately counted into the ReadyReplicas. 2Failure to create pod will cause to return error, then skip updating status. <details</details<details</details<detailskind v1.23.3 </details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig apps
/assign

"k8s.io/apimachinery/pkg/util/wait" utilfeature "k8s.io/apiserver/pkg/util/feature" "k8s.io/client-go/dynamic" <ins> "k8s.io/client-go/informers" </ins> clientset "k8s.io/client-go/kubernetes" <ins> restclient "k8s.io/client-go/rest" </ins> featuregatetesting "k8s.io/component-base/featuregate/testing" apiservertesting "k8s.io/kubernetes/cmd/kube-apiserver/app/testing" podutil "k8s.io/kubernetes/pkg/api/v1/pod" <ins> "k8s.io/kubernetes/pkg/controller/statefulset" </ins> "k8s.io/kubernetes/pkg/features" "k8s.io/kubernetes/test/integration/framework" )


Question:
I create a ResourceQuota with two pods limit in my namespace. Then, I create a statefuleSet with 3 replicas. As expected, two pods are created successfully, and the third pod failed to create. But, the ReadyReplicas is not 2 ,but 1. The ReadyReplicas is correct 1Create a ResourceQuota with two pods limit in a new namespace. 2Create a statefulSet with 3 replicas 3Check if the ReadyReplicas of statefulSet matchs the number of stateful running pods. 1According to the logic of the statefulSet controller, new pod created in current sync is not be immediately counted into the ReadyReplicas. 2Failure to create pod will cause to return error, then skip updating status. <details</details<details</details<detailskind v1.23.3 </details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig apps
/assign

t.Fatalf("failed to mark all StatefulSet pods to ready: %v", err) } } <ins> // add for issue: https://github.com/kubernetes/kubernetes/issues/108837 func TestStatefulSetStatusWithPodFail(t *testing.T) { limitedPodNumber := 2 controlPlaneConfig := framework.NewIntegrationTestControlPlaneConfig() controlPlaneConfig.GenericConfig.AdmissionControl = &fakePodFailAdmission{ limitedPodNumber: limitedPodNumber, } _, s, closeFn := framework.RunAnAPIServer(controlPlaneConfig) defer closeFn() config := restclient.Config{Host: s.URL} c, err := clientset.NewForConfig(&config) if err != nil { t.Fatalf("Could not create clientset: %v", err) } resyncPeriod := 12 * time.Hour informers := informers.NewSharedInformerFactory(clientset.NewForConfigOrDie(restclient.AddUserAgent(&config, "statefulset-informers")), resyncPeriod) ssc := statefulset.NewStatefulSetController( informers.Core().V1().Pods(), informers.Apps().V1().StatefulSets(), informers.Core().V1().PersistentVolumeClaims(), informers.Apps().V1().ControllerRevisions(), clientset.NewForConfigOrDie(restclient.AddUserAgent(&config, "statefulset-controller")), ) ns := framework.CreateTestingNamespace("test-pod-fail", s, t) defer framework.DeleteTestingNamespace(ns, s, t) ctx, cancel := context.WithCancel(context.Background()) defer cancel() informers.Start(ctx.Done()) go ssc.Run(ctx, 5) sts := newSTS("sts", ns.Name, 4) _, err = c.AppsV1().StatefulSets(sts.Namespace).Create(context.TODO(), sts, metav1.CreateOptions{}) if err != nil { t.Fatalf("Could not create statefuleSet %s: %v", sts.Name, err) } wantReplicas := limitedPodNumber var gotReplicas int32 if err := wait.PollImmediate(pollInterval, pollTimeout, func() (bool, error) { newSTS, err := c.AppsV1().StatefulSets(sts.Namespace).Get(context.TODO(), sts.Name, metav1.GetOptions{}) if err != nil { return false, err } gotReplicas = newSTS.Status.Replicas return gotReplicas == int32(wantReplicas), nil }); err != nil { t.Fatalf("StatefulSet %s status has %d replicas, want replicas %d: %v", sts.Name, gotReplicas, wantReplicas, err) } } </ins>


Question:
I create a ResourceQuota with two pods limit in my namespace. Then, I create a statefuleSet with 3 replicas. As expected, two pods are created successfully, and the third pod failed to create. But, the ReadyReplicas is not 2 ,but 1. The ReadyReplicas is correct 1Create a ResourceQuota with two pods limit in a new namespace. 2Create a statefulSet with 3 replicas 3Check if the ReadyReplicas of statefulSet matchs the number of stateful running pods. 1According to the logic of the statefulSet controller, new pod created in current sync is not be immediately counted into the ReadyReplicas. 2Failure to create pod will cause to return error, then skip updating status. <details</details<details</details<detailskind v1.23.3 </details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig apps
/assign

metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/labels" "k8s.io/apimachinery/pkg/util/wait" <ins> "k8s.io/apiserver/pkg/admission" </ins> "k8s.io/client-go/informers" clientset "k8s.io/client-go/kubernetes" typedappsv1 "k8s.io/client-go/kubernetes/typed/apps/v1" typedv1 "k8s.io/client-go/kubernetes/typed/core/v1" restclient "k8s.io/client-go/rest" "k8s.io/client-go/util/retry" <ins> api "k8s.io/kubernetes/pkg/apis/core" </ins> //svc "k8s.io/kubernetes/pkg/api/v1/service" "k8s.io/kubernetes/pkg/controller/statefulset"


Question:
I create a ResourceQuota with two pods limit in my namespace. Then, I create a statefuleSet with 3 replicas. As expected, two pods are created successfully, and the third pod failed to create. But, the ReadyReplicas is not 2 ,but 1. The ReadyReplicas is correct 1Create a ResourceQuota with two pods limit in a new namespace. 2Create a statefulSet with 3 replicas 3Check if the ReadyReplicas of statefulSet matchs the number of stateful running pods. 1According to the logic of the statefulSet controller, new pod created in current sync is not be immediately counted into the ReadyReplicas. 2Failure to create pod will cause to return error, then skip updating status. <details</details<details</details<detailskind v1.23.3 </details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig apps
/assign

} waitSTSStable(t, c, sts) } <ins> var _ admission.ValidationInterface = &fakePodFailAdmission{} type fakePodFailAdmission struct { limitedPodNumber int succeedPodsCount int } func (f *fakePodFailAdmission) Handles(operation admission.Operation) bool { return operation == admission.Create } func (f *fakePodFailAdmission) Validate(ctx context.Context, attr admission.Attributes, o admission.ObjectInterfaces) (err error) { if attr.GetKind().GroupKind() != api.Kind("Pod") { return nil } if f.succeedPodsCount >= f.limitedPodNumber { return fmt.Errorf("fakePodFailAdmission error") } f.succeedPodsCount++ return nil } </ins>


Question:
A lot of different routines failing just when setting up an apiserver. Example: <details</detailsNo errors The test still passes, but those errors seem very worrisome <detailsmaster </details<details</details<details</details<details</details<details</details<details</details
/sig api-machinery /sig testing cc
yeah, the framework has some long standing tech debt /cc
all the panics I see are:
looks like again, due to
Looks like some integration tests must not be enabling SSA.
yeah, a lot don't, because it requires openapi, which also isn't universally enabled in integration. it is really expensive at API server startup, and when we tried enabling it universally in integration it blew up runtimes and integration timed out super frequently
that's what is still open for
/triage accepted

"context" "crypto/sha256" "encoding/binary" <ins> "encoding/json" </ins> "errors" "fmt" "math"


Question:
A lot of different routines failing just when setting up an apiserver. Example: <details</detailsNo errors The test still passes, but those errors seem very worrisome <detailsmaster </details<details</details<details</details<details</details<details</details<details</details
/sig api-machinery /sig testing cc
yeah, the framework has some long standing tech debt /cc
all the panics I see are:
looks like again, due to
Looks like some integration tests must not be enabling SSA.
yeah, a lot don't, because it requires openapi, which also isn't universally enabled in integration. it is really expensive at API server startup, and when we tried enabling it universally in integration it blew up runtimes and integration timed out super frequently
that's what is still open for
/triage accepted

apierrors "k8s.io/apimachinery/pkg/api/errors" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/labels" <ins> apitypes "k8s.io/apimachinery/pkg/types" </ins> utilerrors "k8s.io/apimachinery/pkg/util/errors" utilruntime "k8s.io/apimachinery/pkg/util/runtime" "k8s.io/apimachinery/pkg/util/sets"


Question:
A lot of different routines failing just when setting up an apiserver. Example: <details</detailsNo errors The test still passes, but those errors seem very worrisome <detailsmaster </details<details</details<details</details<details</details<details</details<details</details
/sig api-machinery /sig testing cc
yeah, the framework has some long standing tech debt /cc
all the panics I see are:
looks like again, due to
Looks like some integration tests must not be enabling SSA.
yeah, a lot don't, because it requires openapi, which also isn't universally enabled in integration. it is really expensive at API server startup, and when we tried enabling it universally in integration it blew up runtimes and integration timed out super frequently
that's what is still open for
/triage accepted

"k8s.io/utils/clock" flowcontrol "k8s.io/api/flowcontrol/v1beta2" <del> flowcontrolapplyconfiguration "k8s.io/client-go/applyconfigurations/flowcontrol/v1beta2" </del> flowcontrolclient "k8s.io/client-go/kubernetes/typed/flowcontrol/v1beta2" flowcontrollister "k8s.io/client-go/listers/flowcontrol/v1beta2" )


Question:
A lot of different routines failing just when setting up an apiserver. Example: <details</detailsNo errors The test still passes, but those errors seem very worrisome <detailsmaster </details<details</details<details</details<details</details<details</details<details</details
/sig api-machinery /sig testing cc
yeah, the framework has some long standing tech debt /cc
all the panics I see are:
looks like again, due to
Looks like some integration tests must not be enabling SSA.
yeah, a lot don't, because it requires openapi, which also isn't universally enabled in integration. it is really expensive at API server startup, and when we tried enabling it universally in integration it blew up runtimes and integration timed out super frequently
that's what is still open for
/triage accepted

// if we are going to issue an update, be sure we track every name we update so we know if we update it too often. currResult.updatedItems.Insert(fsu.flowSchema.Name) <ins> patchBytes, err := makeFlowSchemaConditionPatch(fsu.condition) if err != nil { // should never happen because these conditions are created here and well formed panic(fmt.Sprintf("Failed to json.Marshall(%#+v): %s", fsu.condition, err.Error())) } </ins> if klogV := klog.V(4); klogV.Enabled() { klogV.Infof("%s writing Condition %s to FlowSchema %s, which had ResourceVersion=%s, because its previous value was %s, diff: %s", cfgCtlr.name, fsu.condition, fsu.flowSchema.Name, fsu.flowSchema.ResourceVersion, fcfmt.Fmt(fsu.oldValue), cmp.Diff(fsu.oldValue, fsu.condition)) } fsIfc := cfgCtlr.flowcontrolClient.FlowSchemas() <del> applyOptions := metav1.ApplyOptions{FieldManager: cfgCtlr.asFieldManager, Force: true} // the condition field in fsStatusUpdate holds the new condition we want to update. // TODO: this will break when we have multiple conditions for a flowschema _, err := fsIfc.ApplyStatus(context.TODO(), toFlowSchemaApplyConfiguration(fsu), applyOptions) </del> <ins> patchOptions := metav1.PatchOptions{FieldManager: cfgCtlr.asFieldManager} _, err = fsIfc.Patch(context.TODO(), fsu.flowSchema.Name, apitypes.StrategicMergePatchType, patchBytes, patchOptions, "status") </ins> if err != nil { if apierrors.IsNotFound(err) { // This object has been deleted.  A notification is coming


Question:
A lot of different routines failing just when setting up an apiserver. Example: <details</detailsNo errors The test still passes, but those errors seem very worrisome <detailsmaster </details<details</details<details</details<details</details<details</details<details</details
/sig api-machinery /sig testing cc
yeah, the framework has some long standing tech debt /cc
all the panics I see are:
looks like again, due to
Looks like some integration tests must not be enabling SSA.
yeah, a lot don't, because it requires openapi, which also isn't universally enabled in integration. it is really expensive at API server startup, and when we tried enabling it universally in integration it blew up runtimes and integration timed out super frequently
that's what is still open for
/triage accepted

return suggestedDelay, utilerrors.NewAggregate(errs) } <del> func toFlowSchemaApplyConfiguration(fsUpdate fsStatusUpdate) *flowcontrolapplyconfiguration.FlowSchemaApplyConfiguration { condition := flowcontrolapplyconfiguration.FlowSchemaCondition(). WithType(fsUpdate.condition.Type). WithStatus(fsUpdate.condition.Status). WithReason(fsUpdate.condition.Reason). WithLastTransitionTime(fsUpdate.condition.LastTransitionTime). WithMessage(fsUpdate.condition.Message) return flowcontrolapplyconfiguration.FlowSchema(fsUpdate.flowSchema.Name). WithStatus(flowcontrolapplyconfiguration.FlowSchemaStatus(). WithConditions(condition), ) </del> <ins> // makeFlowSchemaConditionPatch takes in a condition and returns the patch status as a json. func makeFlowSchemaConditionPatch(condition flowcontrol.FlowSchemaCondition) ([]byte, error) { o := struct { Status flowcontrol.FlowSchemaStatus `json:"status"` }{ Status: flowcontrol.FlowSchemaStatus{ Conditions: []flowcontrol.FlowSchemaCondition{ condition, }, }, } return json.Marshal(o) </ins> } // shouldDelayUpdate checks to see if a flowschema has been updated too often and returns true if a delay is needed.


Question:
A lot of different routines failing just when setting up an apiserver. Example: <details</detailsNo errors The test still passes, but those errors seem very worrisome <detailsmaster </details<details</details<details</details<details</details<details</details<details</details
/sig api-machinery /sig testing cc
yeah, the framework has some long standing tech debt /cc
all the panics I see are:
looks like again, due to
Looks like some integration tests must not be enabling SSA.
yeah, a lot don't, because it requires openapi, which also isn't universally enabled in integration. it is really expensive at API server startup, and when we tried enabling it universally in integration it blew up runtimes and integration timed out super frequently
that's what is still open for
/triage accepted

<ins> /* Copyright 2021 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package flowcontrol import ( "fmt" "reflect" "testing" "time" "github.com/google/go-cmp/cmp" flowcontrol "k8s.io/api/flowcontrol/v1beta2" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" ) func Test_configController_generatePatchBytes(t *testing.T) { now := time.Now().UTC() tests := []struct { name      string condition flowcontrol.FlowSchemaCondition want      []byte }{ { name: "check if only condition is parsed", condition: flowcontrol.FlowSchemaCondition{ Type:               flowcontrol.FlowSchemaConditionDangling, Status:             flowcontrol.ConditionTrue, Reason:             "test reason", Message:            "test none", LastTransitionTime: metav1.NewTime(now), }, want: []byte(fmt.Sprintf(`{"status":{"conditions":[{"type":"Dangling","status":"True","lastTransitionTime":"%s","reason":"test reason","message":"test none"}]}}`, now.Format(time.RFC3339))), }, { name: "check when message has double quotes", condition: flowcontrol.FlowSchemaCondition{ Type:               flowcontrol.FlowSchemaConditionDangling, Status:             flowcontrol.ConditionTrue, Reason:             "test reason", Message:            `test ""none`, LastTransitionTime: metav1.NewTime(now), }, want: []byte(fmt.Sprintf(`{"status":{"conditions":[{"type":"Dangling","status":"True","lastTransitionTime":"%s","reason":"test reason","message":"test ""none"}]}}`, now.Format(time.RFC3339))), }, { name: "check when message has a whitespace character that can be escaped", condition: flowcontrol.FlowSchemaCondition{ Type:   flowcontrol.FlowSchemaConditionDangling, Status: flowcontrol.ConditionTrue, Reason: "test reason", Message: `test 		none`, LastTransitionTime: metav1.NewTime(now), }, want: []byte(fmt.Sprintf(`{"status":{"conditions":[{"type":"Dangling","status":"True","lastTransitionTime":"%s","reason":"test reason","message":"test ttnone"}]}}`, now.Format(time.RFC3339))), }, { name: "check when a few fields (message & lastTransitionTime) are missing", condition: flowcontrol.FlowSchemaCondition{ Type:   flowcontrol.FlowSchemaConditionDangling, Status: flowcontrol.ConditionTrue, Reason: "test reason", }, want: []byte(`{"status":{"conditions":[{"type":"Dangling","status":"True","lastTransitionTime":null,"reason":"test reason"}]}}`), }, } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { got, _ := makeFlowSchemaConditionPatch(tt.condition) if !reflect.DeepEqual(got, tt.want) { t.Errorf("makeFlowSchemaConditionPatch() got = %s, want %s; diff is %s", got, tt.want, cmp.Diff(tt.want, got)) } }) } } </ins>


Question:
A lot of different routines failing just when setting up an apiserver. Example: <details</detailsNo errors The test still passes, but those errors seem very worrisome <detailsmaster </details<details</details<details</details<details</details<details</details<details</details
/sig api-machinery /sig testing cc
yeah, the framework has some long standing tech debt /cc
all the panics I see are:
looks like again, due to
Looks like some integration tests must not be enabling SSA.
yeah, a lot don't, because it requires openapi, which also isn't universally enabled in integration. it is really expensive at API server startup, and when we tried enabling it universally in integration it blew up runtimes and integration timed out super frequently
that's what is still open for
/triage accepted

// resetFieldsStatusDefault conflicts with statusDefault const resetFieldsStatusDefault = `{"status": {"conditions": [{"type": "MyStatus", "status":"False"}]}}` <del> var resetFieldsSkippedResources = map[string]struct{}{} </del> <ins> var resetFieldsSkippedResources = map[string]struct{}{ // TODO: flowschemas is flaking, // possible bug in the flowschemas controller. "flowschemas": {}, } </ins> // noConflicts is the set of reources for which // a conflict cannot occur.


Question:
A pod uses a PVC with . The block volume is backed by a block device on the node (which is created by some CSI storage providers). If the underlying block device corresponding to the volume is accidentally disconnected from the node (due to network, hardware problem, etc...), the user cannot delete the pod. The pod would be stuck in terminating forever since Kubelet refuses to unmount the block volume. This prevents the user from cleaning up the pod and spinning up a new replacement pod thus leading to a long service degradation. For example, if the pod is part of a statefulset, the replacement pod cannot come up due to the old pod being stuck terminating. Kubelet should be able to finish terminating a pod when the corresponding block device is already removed from the node. a CSI storage provider that supports . For example, a statefulset that has volumeClaimTemplates with . For example: <details close<summaryclick to open yaml manifest</summary</detailsa network/hardware problem in which the backing block device is removed from the host unexpectedly. In our example, we delete a Longhorn engine instance manager which leads to the underlying block device being removed from the host. that the pod cannot access the block volume, try to delete the pod. pod is stuck in terminating state forever. Logs from kubelet: <details close<summaryClick to open logs</summary<br</detailsPlease pay attention to the log . Kubelet is refusing to terminate the pod because the underlying block device is removed from the host. If we run on the host, we can see that the backing file of the corresponding loop device is marked as : In our humble opinion, we think that Kubelet should process to remove the loop device corresponding to the deleted block device so that pod can finish terminating as detailed in this PR: We have tested that PR and can confirm that it fixes the issue. We have prepared a PR at : We have tested that PR and can confirm that it fixes the issue. <details</details<detailsRancher RKE2 </details<details</details<detailsRancher RKE2 </details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage
Ref:
/cc
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
/remove-lifecycle rotten
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle rotten
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
kubelet log 13:31:57 wanguo-72 kubelet[1131]: E0428 13:31:57. 1131 ] Operation for "{^-54fa-4068-bbad- podName:-9521-4aed-9bfe- nodeName:}" failed. No retries permitted until 2023-04-28 13:33:59. +0800 CST m=+8838. (durationBeforeRetry 2m2s). Error: UnmapVolume.UnmapBlockVolume failed for volume "pvc--a4e4-48a7-9057-" (UniqueName: "^-54fa-4068-bbad-") pod "-9521-4aed-9bfe-" (UID: "-9521-4aed-9bfe-") : blkUtil.DetachFileDevice failed. globalUnmapPath:, podUID: -9521-4aed-9bfe-, bindMount: true: failed to unmap device from map path. mapPath is empty node poweroff restart , pod use block mode volume, and pod stuck in terminating
so it is a new issue, and I encounter it as well with XSKY CSI after node restart. My kubernetes version is 1.25.8
which Kubernetes version is it?
Hi, I also see the issue reported by I'm using v1.25.10. I assume it happens when using containerized kubelet. In my environment, losetup reports weird backing file paths. This can be reproduced as follows. Until this moment, the correct path is visible from the host. But, after terminating the container, I get the wrong results. I consider this behavior causes failing unmap. Any thoughts?

} // Return the first match. <del> backingFilePath := strings.TrimSpace(string(data)) </del> <ins> backingFilePath := cleanBackingFilePath(string(data)) </ins> if backingFilePath == path || backingFilePath == realPath { return fmt.Sprintf("/dev/%s", filepath.Base(device)), nil }


Question:
A pod uses a PVC with . The block volume is backed by a block device on the node (which is created by some CSI storage providers). If the underlying block device corresponding to the volume is accidentally disconnected from the node (due to network, hardware problem, etc...), the user cannot delete the pod. The pod would be stuck in terminating forever since Kubelet refuses to unmount the block volume. This prevents the user from cleaning up the pod and spinning up a new replacement pod thus leading to a long service degradation. For example, if the pod is part of a statefulset, the replacement pod cannot come up due to the old pod being stuck terminating. Kubelet should be able to finish terminating a pod when the corresponding block device is already removed from the node. a CSI storage provider that supports . For example, a statefulset that has volumeClaimTemplates with . For example: <details close<summaryclick to open yaml manifest</summary</detailsa network/hardware problem in which the backing block device is removed from the host unexpectedly. In our example, we delete a Longhorn engine instance manager which leads to the underlying block device being removed from the host. that the pod cannot access the block volume, try to delete the pod. pod is stuck in terminating state forever. Logs from kubelet: <details close<summaryClick to open logs</summary<br</detailsPlease pay attention to the log . Kubelet is refusing to terminate the pod because the underlying block device is removed from the host. If we run on the host, we can see that the backing file of the corresponding loop device is marked as : In our humble opinion, we think that Kubelet should process to remove the loop device corresponding to the deleted block device so that pod can finish terminating as detailed in this PR: We have tested that PR and can confirm that it fixes the issue. We have prepared a PR at : We have tested that PR and can confirm that it fixes the issue. <details</details<detailsRancher RKE2 </details<details</details<detailsRancher RKE2 </details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage
Ref:
/cc
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
/remove-lifecycle rotten
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle rotten
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
kubelet log 13:31:57 wanguo-72 kubelet[1131]: E0428 13:31:57. 1131 ] Operation for "{^-54fa-4068-bbad- podName:-9521-4aed-9bfe- nodeName:}" failed. No retries permitted until 2023-04-28 13:33:59. +0800 CST m=+8838. (durationBeforeRetry 2m2s). Error: UnmapVolume.UnmapBlockVolume failed for volume "pvc--a4e4-48a7-9057-" (UniqueName: "^-54fa-4068-bbad-") pod "-9521-4aed-9bfe-" (UID: "-9521-4aed-9bfe-") : blkUtil.DetachFileDevice failed. globalUnmapPath:, podUID: -9521-4aed-9bfe-, bindMount: true: failed to unmap device from map path. mapPath is empty node poweroff restart , pod use block mode volume, and pod stuck in terminating
so it is a new issue, and I encounter it as well with XSKY CSI after node restart. My kubernetes version is 1.25.8
which Kubernetes version is it?
Hi, I also see the issue reported by I'm using v1.25.10. I assume it happens when using containerized kubelet. In my environment, losetup reports weird backing file paths. This can be reproduced as follows. Until this moment, the correct path is visible from the host. But, after terminating the container, I get the wrong results. I consider this behavior causes failing unmap. Any thoughts?

return "", errors.New(ErrDeviceNotFound) } <ins> // cleanPath remove any trailing substrings that are not part of the backing file path. func cleanBackingFilePath(path string) string { // If the block device was deleted, the path will contain a "(deleted)" suffix path = strings.TrimSpace(path) path = strings.TrimSuffix(path, "(deleted)") return strings.TrimSpace(path) } </ins> // FindGlobalMapPathUUIDFromPod finds {pod uuid} bind mount under globalMapPath // corresponding to map path symlink, and then return global map path with pod uuid. // (See pkg/volume/volume.go for details on a global map path and a pod device map path.)


Question:
A pod uses a PVC with . The block volume is backed by a block device on the node (which is created by some CSI storage providers). If the underlying block device corresponding to the volume is accidentally disconnected from the node (due to network, hardware problem, etc...), the user cannot delete the pod. The pod would be stuck in terminating forever since Kubelet refuses to unmount the block volume. This prevents the user from cleaning up the pod and spinning up a new replacement pod thus leading to a long service degradation. For example, if the pod is part of a statefulset, the replacement pod cannot come up due to the old pod being stuck terminating. Kubelet should be able to finish terminating a pod when the corresponding block device is already removed from the node. a CSI storage provider that supports . For example, a statefulset that has volumeClaimTemplates with . For example: <details close<summaryclick to open yaml manifest</summary</detailsa network/hardware problem in which the backing block device is removed from the host unexpectedly. In our example, we delete a Longhorn engine instance manager which leads to the underlying block device being removed from the host. that the pod cannot access the block volume, try to delete the pod. pod is stuck in terminating state forever. Logs from kubelet: <details close<summaryClick to open logs</summary<br</detailsPlease pay attention to the log . Kubelet is refusing to terminate the pod because the underlying block device is removed from the host. If we run on the host, we can see that the backing file of the corresponding loop device is marked as : In our humble opinion, we think that Kubelet should process to remove the loop device corresponding to the deleted block device so that pod can finish terminating as detailed in this PR: We have tested that PR and can confirm that it fixes the issue. We have prepared a PR at : We have tested that PR and can confirm that it fixes the issue. <details</details<detailsRancher RKE2 </details<details</details<detailsRancher RKE2 </details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage
Ref:
/cc
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
/remove-lifecycle rotten
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle rotten
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
kubelet log 13:31:57 wanguo-72 kubelet[1131]: E0428 13:31:57. 1131 ] Operation for "{^-54fa-4068-bbad- podName:-9521-4aed-9bfe- nodeName:}" failed. No retries permitted until 2023-04-28 13:33:59. +0800 CST m=+8838. (durationBeforeRetry 2m2s). Error: UnmapVolume.UnmapBlockVolume failed for volume "pvc--a4e4-48a7-9057-" (UniqueName: "^-54fa-4068-bbad-") pod "-9521-4aed-9bfe-" (UID: "-9521-4aed-9bfe-") : blkUtil.DetachFileDevice failed. globalUnmapPath:, podUID: -9521-4aed-9bfe-, bindMount: true: failed to unmap device from map path. mapPath is empty node poweroff restart , pod use block mode volume, and pod stuck in terminating
so it is a new issue, and I encounter it as well with XSKY CSI after node restart. My kubernetes version is 1.25.8
which Kubernetes version is it?
Hi, I also see the issue reported by I'm using v1.25.10. I assume it happens when using containerized kubelet. In my environment, losetup reports weird backing file paths. This can be reproduced as follows. Until this moment, the correct path is visible from the host. But, after terminating the container, I get the wrong results. I consider this behavior causes failing unmap. Any thoughts?

<ins> /* Copyright 2023 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package volumepathhandler import ( "fmt" "testing" ) func pathWithSuffix(suffix string) string { return fmt.Sprintf("%s%s", "/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/pvc-1d205234-06cd-4fe4-a7ea-0e8f3e2faf5f/dev/e196ebd3-2ab1-4185-bed4-b997ba38d1dc", suffix) } func TestCleanBackingFilePath(t *testing.T) { const defaultPath = "/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/pvc-1d205234-06cd-4fe4-a7ea-0e8f3e2faf5f/dev/e196ebd3-2ab1-4185-bed4-b997ba38d1dc" testCases := []struct { name          string input         string expectedOuput string }{ { name:          "regular path", input:         defaultPath, expectedOuput: defaultPath, }, { name:          "path is suffixed with whitespaces", input:         fmt.Sprintf("%srtn ", defaultPath), expectedOuput: defaultPath, }, { name:          "path is suffixed with "(deleted)"", input:         pathWithSuffix("(deleted)"), expectedOuput: defaultPath, }, { name:          "path is suffixed with "(deleted)" and whitespaces", input:         pathWithSuffix(" (deleted)t"), expectedOuput: defaultPath, }, } for _, tc := range testCases { t.Run(tc.name, func(t *testing.T) { output := cleanBackingFilePath(tc.input) if output != tc.expectedOuput { t.Fatalf("expected %q, got %q", tc.expectedOuput, output) } }) } } </ins>


Question:
17:42 PDT 3/29 Failed with the same error for last 4 runs. Beta release cut is scheduled for tomorrow and this is currently a blocker. /sig testing
Looks like this is also being discussed and x-ref this test was merged 4 hours ago
/sig api-machinery
More x-ref: SIG api machinery is tracking it themselves in There is a PR to increase the timeout in
More x-ref: Same post in the .
Per this should be resolved and not be a blocker, testgrid is still red but I think the expectation is that it should clear up. Leaving this open for now but currently expecting this not to block release cut tomorrow.

var r result client := newClient(true) for i := 0; i < 5; i++ { <del> r = doer.Do(client, func(httptrace.GotConnInfo) {}, fmt.Sprintf("/echo?message=attempt-%d", i), 100*time.Millisecond) </del> <ins> r = doer.Do(client, func(httptrace.GotConnInfo) {}, fmt.Sprintf("/echo?message=attempt-%d", i), 1*time.Second) </ins> if r.err != nil { break }


Question:
Schedulerperf sometimes causes "json: unsupported value: NaN" panic . Not panic. in my PC, when running schedulerperf with the below config, there is a high probability of the panic. No response_ latest master <details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scheduling /assign
/kind failing-test
No. This problem is expected to exist for quite some time. Also, this test is just for measuring performance and does not block merging. Correct labeling requires background knowledge of the issue. For example, if, hypothetically, this test is built into ci and blocks the merge when it fails, then you are right, it would be a high priority.

} if err := vec.Validate(); err != nil { <del> klog.Error(err) </del> <ins> klog.ErrorS(err, "the validation for HistogramVec is failed. The data for this metric won't be stored in a benchmark result file", "metric", metric, "labels", labels) return nil } if vec.GetAggregatedSampleCount() == 0 { klog.InfoS("It is expected that this metric wasn't recorded. The data for this metric won't be stored in a benchmark result file", "metric", metric, "labels", labels) </ins> return nil }


Question:
As described in , When the pod start or stop, negative metrics generated. generate the correct metrics pod restart No response <details</details<detailsnone </details<details</details<details</details<details</details<details</details
/sig node /sig instrumentation
can you please give a little bit more information on steps to repro. Is it happens all the time? Is the root cause known (I see a linked PR). /triage needs-information
This is a feedback from contributors in kubernetes-sigs/metrics-server. It is riggered on specific conditions. From the description of several issues, the way to reproduce should be pod start or stop. such as:
Do you know if this is specific to ?
I'm not sure. However, obtains data from the kubelet endpoint. If there is a negative value, it will cause statistical problems.
/triage accepted /assign
FYI I encounter this with . seems to be an uninitialized time type. The in is create from the response of cri RPC, this maybe cause either by cri-o or the metrics collect part of Kubernetes. I have not dig into it yet.

} func (rc *resourceMetricsCollector) collectContainerStartTime(ch chan<- metrics.Metric, pod summary.PodStats, s summary.ContainerStats) { <del> if s.StartTime.Unix() == 0 { </del> <ins> if s.StartTime.Unix() <= 0 { </ins> return }


Question:
As described in , When the pod start or stop, negative metrics generated. generate the correct metrics pod restart No response <details</details<detailsnone </details<details</details<details</details<details</details<details</details
/sig node /sig instrumentation
can you please give a little bit more information on steps to repro. Is it happens all the time? Is the root cause known (I see a linked PR). /triage needs-information
This is a feedback from contributors in kubernetes-sigs/metrics-server. It is riggered on specific conditions. From the description of several issues, the way to reproduce should be pod start or stop. such as:
Do you know if this is specific to ?
I'm not sure. However, obtains data from the kubelet endpoint. If there is a negative value, it will cause statistical problems.
/triage accepted /assign
FYI I encounter this with . seems to be an uninitialized time type. The in is create from the response of cri RPC, this maybe cause either by cri-o or the metrics collect part of Kubernetes. I have not dig into it yet.

`, }, { <ins> name: "arbitrary container metrics for negative StartTime", summary: &statsapi.Summary{ Pods: []statsapi.PodStats{ { PodRef: statsapi.PodReference{ Name:      "pod_a", Namespace: "namespace_a", }, Containers: []statsapi.ContainerStats{ { Name:      "container_a", StartTime: metav1.NewTime(time.Unix(0, -1624396278302091597)), CPU: &statsapi.CPUStats{ Time:                 testTime, UsageCoreNanoSeconds: uint64Ptr(10000000000), }, Memory: &statsapi.MemoryStats{ Time:            testTime, WorkingSetBytes: uint64Ptr(1000), }, }, }, }, }, }, summaryErr: nil, expectedMetrics: ` # HELP scrape_error [ALPHA] 1 if there was an error while getting container metrics, 0 otherwise # TYPE scrape_error gauge scrape_error 0 # HELP container_cpu_usage_seconds_total [ALPHA] Cumulative cpu time consumed by the container in core-seconds # TYPE container_cpu_usage_seconds_total counter container_cpu_usage_seconds_total{container="container_a",namespace="namespace_a",pod="pod_a"} 10 1624396278302 # HELP container_memory_working_set_bytes [ALPHA] Current working set of the container in bytes # TYPE container_memory_working_set_bytes gauge container_memory_working_set_bytes{container="container_a",namespace="namespace_a",pod="pod_a"} 1000 1624396278302 `, }, { </ins> name: "nil container metrics", summary: &statsapi.Summary{ Pods: []statsapi.PodStats{


Question:
once the we hit a beta release, we can reset the default value for strict handling to false to prep for 1.25. It will likely be introduced in /kind bug /priority important-soon /milestone v1.25 ignore template No response this time <details</details<details</details<details</details<details</details
/sig api-machinery
/assign /triage accepted
Hello my name is Justin Chizer and I am one of the k8s 1.25 bug triage shadow assigned to track this body of work. Is this still on track for 1.25?
This needs to be done now that master is on 1.26 until 1.25.0 is cut, since verify jobs are failing on removal of 1.26 deprecated APIs /priority critical-urgent

} func NewResourceExpirationEvaluator(currentVersion apimachineryversion.Info) (ResourceExpirationEvaluator, error) { <del> ret := &resourceExpirationEvaluator{ // TODO https://github.com/kubernetes/kubernetes/issues/109799 set this back to false after beta is tagged. strictRemovedHandlingInAlpha: true, } </del> <ins> ret := &resourceExpirationEvaluator{} </ins> if len(currentVersion.Major) > 0 { currentMajor64, err := strconv.ParseInt(currentVersion.Major, 10, 32) if err != nil {


Question:
Follow-up to Discussed in As part of kubelet tracing beta, we should add tracing support to the kubelet's read-only endpoint, which currently does not have tracing. It should use WithPublicEndpoint to prevent callers from influencing sampling decisions. Part of
/sig node /sig instrumentation
/triage accepted /assign
can i help you with this issue?
I don't think we'll be able to start with this until merges
Ok, I'm waiting to see if I can help you. Thank you
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale is merged now, so this can be worked on.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
Could I help with this issue?
Sure. See We pass a NoOp Tracer provider to disable tracing for the endpoint. Instead, we need to pass the tracerprovider, but make sure that the eventual uses withPublicEndpoint. Looking closer, i'm not sure its currently possible to use WithPublicEndpoint with the otelrestful package... doesn't contain that option (unlike otelhttp: ), so we may need to add that upstream first. Another thing to keep in mind is that updating otel libraries is currently blocked by (see my attempt in )
I gave the whole thing another try. At least locally, the dependencies seem now to be able to be updated. :raccoon: Could you maybe have a look at it :)
He's on leave for the time being.
oh, can you provide some feedback or point me to the right people?
I'm back now. Feel free to assign me to PRs related to this, and I'll review
Welcome back :) Could you re-run the CI ? Its just the dependencies. The PR itself shouldn't be that much afterwards.
otel is now up-to-date enough. here:
Hi, I would like to ask about one trouble that caused by WithPublicEndpoint. Aggregated APIServer that I customized uses kubernetes's RecommendedConfig. Due to the fact that WithPublicEndpoint is enabled, it can't be tracked by the full trace for its span-link model. And I think child-parent span model is more appropriate. So, could you please give me some advice? Looking forward to your reply.
thanks. We know about this, and are tracking it in

go k.ListenAndServe(kubeCfg, kubeDeps.TLSOptions, kubeDeps.Auth, kubeDeps.TracerProvider) } if kubeCfg.ReadOnlyPort > 0 { <del> go k.ListenAndServeReadOnly(netutils.ParseIPSloppy(kubeCfg.Address), uint(kubeCfg.ReadOnlyPort)) </del> <ins> go k.ListenAndServeReadOnly(netutils.ParseIPSloppy(kubeCfg.Address), uint(kubeCfg.ReadOnlyPort), kubeDeps.TracerProvider) </ins> } go k.ListenAndServePodResources() }


Question:
Follow-up to Discussed in As part of kubelet tracing beta, we should add tracing support to the kubelet's read-only endpoint, which currently does not have tracing. It should use WithPublicEndpoint to prevent callers from influencing sampling decisions. Part of
/sig node /sig instrumentation
/triage accepted /assign
can i help you with this issue?
I don't think we'll be able to start with this until merges
Ok, I'm waiting to see if I can help you. Thank you
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale is merged now, so this can be worked on.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
Could I help with this issue?
Sure. See We pass a NoOp Tracer provider to disable tracing for the endpoint. Instead, we need to pass the tracerprovider, but make sure that the eventual uses withPublicEndpoint. Looking closer, i'm not sure its currently possible to use WithPublicEndpoint with the otelrestful package... doesn't contain that option (unlike otelhttp: ), so we may need to add that upstream first. Another thing to keep in mind is that updating otel libraries is currently blocked by (see my attempt in )
I gave the whole thing another try. At least locally, the dependencies seem now to be able to be updated. :raccoon: Could you maybe have a look at it :)
He's on leave for the time being.
oh, can you provide some feedback or point me to the right people?
I'm back now. Feel free to assign me to PRs related to this, and I'll review
Welcome back :) Could you re-run the CI ? Its just the dependencies. The PR itself shouldn't be that much afterwards.
otel is now up-to-date enough. here:
Hi, I would like to ask about one trouble that caused by WithPublicEndpoint. Aggregated APIServer that I customized uses kubernetes's RecommendedConfig. Due to the fact that WithPublicEndpoint is enabled, it can't be tracked by the full trace for its span-link model. And I think child-parent span model is more appropriate. So, could you please give me some advice? Looking forward to your reply.
thanks. We know about this, and are tracking it in

BirthCry() StartGarbageCollection() ListenAndServe(kubeCfg *kubeletconfiginternal.KubeletConfiguration, tlsOptions *server.TLSOptions, auth server.AuthInterface, tp trace.TracerProvider) <del> ListenAndServeReadOnly(address net.IP, port uint) </del> <ins> ListenAndServeReadOnly(address net.IP, port uint, tp trace.TracerProvider) </ins> ListenAndServePodResources() Run(<-chan kubetypes.PodUpdate) RunOnce(<-chan kubetypes.PodUpdate) ([]RunPodResult, error)


Question:
Follow-up to Discussed in As part of kubelet tracing beta, we should add tracing support to the kubelet's read-only endpoint, which currently does not have tracing. It should use WithPublicEndpoint to prevent callers from influencing sampling decisions. Part of
/sig node /sig instrumentation
/triage accepted /assign
can i help you with this issue?
I don't think we'll be able to start with this until merges
Ok, I'm waiting to see if I can help you. Thank you
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale is merged now, so this can be worked on.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
Could I help with this issue?
Sure. See We pass a NoOp Tracer provider to disable tracing for the endpoint. Instead, we need to pass the tracerprovider, but make sure that the eventual uses withPublicEndpoint. Looking closer, i'm not sure its currently possible to use WithPublicEndpoint with the otelrestful package... doesn't contain that option (unlike otelhttp: ), so we may need to add that upstream first. Another thing to keep in mind is that updating otel libraries is currently blocked by (see my attempt in )
I gave the whole thing another try. At least locally, the dependencies seem now to be able to be updated. :raccoon: Could you maybe have a look at it :)
He's on leave for the time being.
oh, can you provide some feedback or point me to the right people?
I'm back now. Feel free to assign me to PRs related to this, and I'll review
Welcome back :) Could you re-run the CI ? Its just the dependencies. The PR itself shouldn't be that much afterwards.
otel is now up-to-date enough. here:
Hi, I would like to ask about one trouble that caused by WithPublicEndpoint. Aggregated APIServer that I customized uses kubernetes's RecommendedConfig. Due to the fact that WithPublicEndpoint is enabled, it can't be tracked by the full trace for its span-link model. And I think child-parent span model is more appropriate. So, could you please give me some advice? Looking forward to your reply.
thanks. We know about this, and are tracking it in

} // ListenAndServeReadOnly runs the kubelet HTTP server in read-only mode. <del> func (kl *Kubelet) ListenAndServeReadOnly(address net.IP, port uint) { server.ListenAndServeKubeletReadOnlyServer(kl, kl.resourceAnalyzer, address, port) </del> <ins> func (kl *Kubelet) ListenAndServeReadOnly(address net.IP, port uint, tp trace.TracerProvider) { server.ListenAndServeKubeletReadOnlyServer(kl, kl.resourceAnalyzer, address, port, tp) </ins> } // ListenAndServePodResources runs the kubelet podresources grpc service


Question:
Follow-up to Discussed in As part of kubelet tracing beta, we should add tracing support to the kubelet's read-only endpoint, which currently does not have tracing. It should use WithPublicEndpoint to prevent callers from influencing sampling decisions. Part of
/sig node /sig instrumentation
/triage accepted /assign
can i help you with this issue?
I don't think we'll be able to start with this until merges
Ok, I'm waiting to see if I can help you. Thank you
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale is merged now, so this can be worked on.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
Could I help with this issue?
Sure. See We pass a NoOp Tracer provider to disable tracing for the endpoint. Instead, we need to pass the tracerprovider, but make sure that the eventual uses withPublicEndpoint. Looking closer, i'm not sure its currently possible to use WithPublicEndpoint with the otelrestful package... doesn't contain that option (unlike otelhttp: ), so we may need to add that upstream first. Another thing to keep in mind is that updating otel libraries is currently blocked by (see my attempt in )
I gave the whole thing another try. At least locally, the dependencies seem now to be able to be updated. :raccoon: Could you maybe have a look at it :)
He's on leave for the time being.
oh, can you provide some feedback or point me to the right people?
I'm back now. Feel free to assign me to PRs related to this, and I'll review
Welcome back :) Could you re-run the CI ? Its just the dependencies. The PR itself shouldn't be that much afterwards.
otel is now up-to-date enough. here:
Hi, I would like to ask about one trouble that caused by WithPublicEndpoint. Aggregated APIServer that I customized uses kubernetes's RecommendedConfig. Due to the fact that WithPublicEndpoint is enabled, it can't be tracked by the full trace for its span-link model. And I think child-parent span model is more appropriate. So, could you please give me some advice? Looking forward to your reply.
thanks. We know about this, and are tracking it in

address := netutils.ParseIPSloppy(kubeCfg.Address) port := uint(kubeCfg.Port) klog.InfoS("Starting to listen", "address", address, "port", port) <del> handler := NewServer(host, resourceAnalyzer, auth, tp, kubeCfg) </del> <ins> handler := NewServer(host, resourceAnalyzer, auth, kubeCfg) if utilfeature.DefaultFeatureGate.Enabled(features.KubeletTracing) { handler.InstallTracingFilter(tp) } </ins> s := &http.Server{ Addr:           net.JoinHostPort(address.String(), strconv.FormatUint(uint64(port), 10)), Handler:        &handler,


Question:
Follow-up to Discussed in As part of kubelet tracing beta, we should add tracing support to the kubelet's read-only endpoint, which currently does not have tracing. It should use WithPublicEndpoint to prevent callers from influencing sampling decisions. Part of
/sig node /sig instrumentation
/triage accepted /assign
can i help you with this issue?
I don't think we'll be able to start with this until merges
Ok, I'm waiting to see if I can help you. Thank you
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale is merged now, so this can be worked on.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
Could I help with this issue?
Sure. See We pass a NoOp Tracer provider to disable tracing for the endpoint. Instead, we need to pass the tracerprovider, but make sure that the eventual uses withPublicEndpoint. Looking closer, i'm not sure its currently possible to use WithPublicEndpoint with the otelrestful package... doesn't contain that option (unlike otelhttp: ), so we may need to add that upstream first. Another thing to keep in mind is that updating otel libraries is currently blocked by (see my attempt in )
I gave the whole thing another try. At least locally, the dependencies seem now to be able to be updated. :raccoon: Could you maybe have a look at it :)
He's on leave for the time being.
oh, can you provide some feedback or point me to the right people?
I'm back now. Feel free to assign me to PRs related to this, and I'll review
Welcome back :) Could you re-run the CI ? Its just the dependencies. The PR itself shouldn't be that much afterwards.
otel is now up-to-date enough. here:
Hi, I would like to ask about one trouble that caused by WithPublicEndpoint. Aggregated APIServer that I customized uses kubernetes's RecommendedConfig. Due to the fact that WithPublicEndpoint is enabled, it can't be tracked by the full trace for its span-link model. And I think child-parent span model is more appropriate. So, could you please give me some advice? Looking forward to your reply.
thanks. We know about this, and are tracking it in

host HostInterface, resourceAnalyzer stats.ResourceAnalyzer, address net.IP, <del> port uint) { </del> <ins> port uint, tp oteltrace.TracerProvider) { </ins> klog.InfoS("Starting to listen read-only", "address", address, "port", port) <del> // TODO: https://github.com/kubernetes/kubernetes/issues/109829 tracer should use WithPublicEndpoint s := NewServer(host, resourceAnalyzer, nil, oteltrace.NewNoopTracerProvider(), nil) </del> <ins> s := NewServer(host, resourceAnalyzer, nil, nil) if utilfeature.DefaultFeatureGate.Enabled(features.KubeletTracing) { s.InstallTracingFilter(tp, otelrestful.WithPublicEndpoint()) } </ins> server := &http.Server{ Addr:           net.JoinHostPort(address.String(), strconv.FormatUint(uint64(port), 10)),


Question:
Follow-up to Discussed in As part of kubelet tracing beta, we should add tracing support to the kubelet's read-only endpoint, which currently does not have tracing. It should use WithPublicEndpoint to prevent callers from influencing sampling decisions. Part of
/sig node /sig instrumentation
/triage accepted /assign
can i help you with this issue?
I don't think we'll be able to start with this until merges
Ok, I'm waiting to see if I can help you. Thank you
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale is merged now, so this can be worked on.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
Could I help with this issue?
Sure. See We pass a NoOp Tracer provider to disable tracing for the endpoint. Instead, we need to pass the tracerprovider, but make sure that the eventual uses withPublicEndpoint. Looking closer, i'm not sure its currently possible to use WithPublicEndpoint with the otelrestful package... doesn't contain that option (unlike otelhttp: ), so we may need to add that upstream first. Another thing to keep in mind is that updating otel libraries is currently blocked by (see my attempt in )
I gave the whole thing another try. At least locally, the dependencies seem now to be able to be updated. :raccoon: Could you maybe have a look at it :)
He's on leave for the time being.
oh, can you provide some feedback or point me to the right people?
I'm back now. Feel free to assign me to PRs related to this, and I'll review
Welcome back :) Could you re-run the CI ? Its just the dependencies. The PR itself shouldn't be that much afterwards.
otel is now up-to-date enough. here:
Hi, I would like to ask about one trouble that caused by WithPublicEndpoint. Aggregated APIServer that I customized uses kubernetes's RecommendedConfig. Due to the fact that WithPublicEndpoint is enabled, it can't be tracked by the full trace for its span-link model. And I think child-parent span model is more appropriate. So, could you please give me some advice? Looking forward to your reply.
thanks. We know about this, and are tracking it in

host HostInterface, resourceAnalyzer stats.ResourceAnalyzer, auth AuthInterface, <del> tp oteltrace.TracerProvider, </del> kubeCfg *kubeletconfiginternal.KubeletConfiguration) Server { server := Server{


Question:
Follow-up to Discussed in As part of kubelet tracing beta, we should add tracing support to the kubelet's read-only endpoint, which currently does not have tracing. It should use WithPublicEndpoint to prevent callers from influencing sampling decisions. Part of
/sig node /sig instrumentation
/triage accepted /assign
can i help you with this issue?
I don't think we'll be able to start with this until merges
Ok, I'm waiting to see if I can help you. Thank you
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale is merged now, so this can be worked on.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
Could I help with this issue?
Sure. See We pass a NoOp Tracer provider to disable tracing for the endpoint. Instead, we need to pass the tracerprovider, but make sure that the eventual uses withPublicEndpoint. Looking closer, i'm not sure its currently possible to use WithPublicEndpoint with the otelrestful package... doesn't contain that option (unlike otelhttp: ), so we may need to add that upstream first. Another thing to keep in mind is that updating otel libraries is currently blocked by (see my attempt in )
I gave the whole thing another try. At least locally, the dependencies seem now to be able to be updated. :raccoon: Could you maybe have a look at it :)
He's on leave for the time being.
oh, can you provide some feedback or point me to the right people?
I'm back now. Feel free to assign me to PRs related to this, and I'll review
Welcome back :) Could you re-run the CI ? Its just the dependencies. The PR itself shouldn't be that much afterwards.
otel is now up-to-date enough. here:
Hi, I would like to ask about one trouble that caused by WithPublicEndpoint. Aggregated APIServer that I customized uses kubernetes's RecommendedConfig. Due to the fact that WithPublicEndpoint is enabled, it can't be tracked by the full trace for its span-link model. And I think child-parent span model is more appropriate. So, could you please give me some advice? Looking forward to your reply.
thanks. We know about this, and are tracking it in

if auth != nil { server.InstallAuthFilter() } <del> if utilfeature.DefaultFeatureGate.Enabled(features.KubeletTracing) { server.InstallTracingFilter(tp) } </del> server.InstallDefaultHandlers() if kubeCfg != nil && kubeCfg.EnableDebuggingHandlers { server.InstallDebuggingHandlers()


Question:
Follow-up to Discussed in As part of kubelet tracing beta, we should add tracing support to the kubelet's read-only endpoint, which currently does not have tracing. It should use WithPublicEndpoint to prevent callers from influencing sampling decisions. Part of
/sig node /sig instrumentation
/triage accepted /assign
can i help you with this issue?
I don't think we'll be able to start with this until merges
Ok, I'm waiting to see if I can help you. Thank you
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale is merged now, so this can be worked on.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
Could I help with this issue?
Sure. See We pass a NoOp Tracer provider to disable tracing for the endpoint. Instead, we need to pass the tracerprovider, but make sure that the eventual uses withPublicEndpoint. Looking closer, i'm not sure its currently possible to use WithPublicEndpoint with the otelrestful package... doesn't contain that option (unlike otelhttp: ), so we may need to add that upstream first. Another thing to keep in mind is that updating otel libraries is currently blocked by (see my attempt in )
I gave the whole thing another try. At least locally, the dependencies seem now to be able to be updated. :raccoon: Could you maybe have a look at it :)
He's on leave for the time being.
oh, can you provide some feedback or point me to the right people?
I'm back now. Feel free to assign me to PRs related to this, and I'll review
Welcome back :) Could you re-run the CI ? Its just the dependencies. The PR itself shouldn't be that much afterwards.
otel is now up-to-date enough. here:
Hi, I would like to ask about one trouble that caused by WithPublicEndpoint. Aggregated APIServer that I customized uses kubernetes's RecommendedConfig. Due to the fact that WithPublicEndpoint is enabled, it can't be tracked by the full trace for its span-link model. And I think child-parent span model is more appropriate. So, could you please give me some advice? Looking forward to your reply.
thanks. We know about this, and are tracking it in

} // InstallTracingFilter installs OpenTelemetry tracing filter with the restful Container. <del> func (s *Server) InstallTracingFilter(tp oteltrace.TracerProvider) { s.restfulCont.Filter(otelrestful.OTelFilter("kubelet", otelrestful.WithTracerProvider(tp))) </del> <ins> func (s *Server) InstallTracingFilter(tp oteltrace.TracerProvider, opts ...otelrestful.Option) { s.restfulCont.Filter(otelrestful.OTelFilter("kubelet", append(opts, otelrestful.WithTracerProvider(tp))...)) </ins> } // addMetricsBucketMatcher adds a regexp matcher and the relevant bucket to use when