Question:
we just log that the user should install etcd, which is impractical in an ephemeral container we actually already call installing etcd in the verify script, but we don't export it into the current PATH for the process so it doesn't work ... ref: /sig api-machinery /assign /kind bug
/triage accepted
for one possible fix
/sig etcd

} kube::etcd::install() { <del> ( local os local arch os=$(kube::util::host_os) arch=$(kube::util::host_arch) cd "${KUBE_ROOT}/third_party" || return 1 if [[ $(readlink etcd) == etcd-v${ETCD_VERSION}-${os}-* ]]; then kube::log::info "etcd v${ETCD_VERSION} already installed. To use:" kube::log::info "export PATH="$(pwd)/etcd:${PATH}"" return  #already installed fi if [[ ${os} == "darwin" ]]; then download_file="etcd-v${ETCD_VERSION}-${os}-${arch}.zip" url="https://github.com/etcd-io/etcd/releases/download/v${ETCD_VERSION}/${download_file}" kube::util::download_file "${url}" "${download_file}" unzip -o "${download_file}" ln -fns "etcd-v${ETCD_VERSION}-${os}-${arch}" etcd rm "${download_file}" elif [[ ${os} == "linux" ]]; then url="https://github.com/coreos/etcd/releases/download/v${ETCD_VERSION}/etcd-v${ETCD_VERSION}-${os}-${arch}.tar.gz" download_file="etcd-v${ETCD_VERSION}-${os}-${arch}.tar.gz" kube::util::download_file "${url}" "${download_file}" tar xzf "${download_file}" ln -fns "etcd-v${ETCD_VERSION}-${os}-${arch}" etcd rm "${download_file}" else kube::log::info "${os} is NOT supported." fi kube::log::info "etcd v${ETCD_VERSION} installed. To use:" </del> <ins> local os local arch os=$(kube::util::host_os) arch=$(kube::util::host_arch) cd "${KUBE_ROOT}/third_party" || return 1 if [[ $(readlink etcd) == etcd-v${ETCD_VERSION}-${os}-* ]]; then kube::log::info "etcd v${ETCD_VERSION} already installed. To use:" </ins> kube::log::info "export PATH="$(pwd)/etcd:${PATH}"" <del> ) </del> <ins> # export into current process PATH="$(pwd)/etcd:${PATH}" export PATH return  #already installed fi if [[ ${os} == "darwin" ]]; then download_file="etcd-v${ETCD_VERSION}-${os}-${arch}.zip" url="https://github.com/etcd-io/etcd/releases/download/v${ETCD_VERSION}/${download_file}" kube::util::download_file "${url}" "${download_file}" unzip -o "${download_file}" ln -fns "etcd-v${ETCD_VERSION}-${os}-${arch}" etcd rm "${download_file}" elif [[ ${os} == "linux" ]]; then url="https://github.com/coreos/etcd/releases/download/v${ETCD_VERSION}/etcd-v${ETCD_VERSION}-${os}-${arch}.tar.gz" download_file="etcd-v${ETCD_VERSION}-${os}-${arch}.tar.gz" kube::util::download_file "${url}" "${download_file}" tar xzf "${download_file}" ln -fns "etcd-v${ETCD_VERSION}-${os}-${arch}" etcd rm "${download_file}" else kube::log::info "${os} is NOT supported." fi kube::log::info "etcd v${ETCD_VERSION} installed. To use:" kube::log::info "export PATH="$(pwd)/etcd:${PATH}"" # export into current process PATH="$(pwd)/etcd:${PATH}" export PATH </ins> }


Question:
we just log that the user should install etcd, which is impractical in an ephemeral container we actually already call installing etcd in the verify script, but we don't export it into the current PATH for the process so it doesn't work ... ref: /sig api-machinery /assign /kind bug
/triage accepted
for one possible fix
/sig etcd

kube::util::require-jq kube::golang::setup_env <ins> kube::etcd::install </ins> make -C "${KUBE_ROOT}" WHAT=cmd/kube-apiserver


Question:
we just log that the user should install etcd, which is impractical in an ephemeral container we actually already call installing etcd in the verify script, but we don't export it into the current PATH for the process so it doesn't work ... ref: /sig api-machinery /assign /kind bug
/triage accepted
for one possible fix
/sig etcd

trap cleanup EXIT SIGINT <del> kube::golang::setup_env </del> TMP_DIR=${TMP_DIR:-$(kube::realpath "$(mktemp -d -t "$(basename "$0").XXXXXX")")} ETCD_HOST=${ETCD_HOST:-127.0.0.1} ETCD_PORT=${ETCD_PORT:-2379}


Question:
Runs k8s with master golang: test Failure was observed on Aug 25th, it is passing after that. However it can be reproduced with stress tool. No response No response It is failing on both ppc and x86 architectures. Steps to reproduce: up golang tip version. clone kubernetes master /sig testing
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
it is a sig node test, sig testing charter is not about doing or maintain tests, we need to remove that old mindset of having a QA department, that is not how software development works or should work :) Besides, thanks for spotting it and submittn a patch /sig node /remove-sig testing
Flaky in
merging this PR will fix the issue

if err != nil { assert.NoErrorf(t, err, "unable to create temp file") } stdoutBuf := &bytes.Buffer{} stderrBuf := &bytes.Buffer{} containerID := "fake-container-id"


Question:
Runs k8s with master golang: test Failure was observed on Aug 25th, it is passing after that. However it can be reproduced with stress tool. No response No response It is failing on both ppc and x86 architectures. Steps to reproduce: up golang tip version. clone kubernetes master /sig testing
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
it is a sig node test, sig testing charter is not about doing or maintain tests, we need to remove that old mindset of having a QA department, that is not how software development works or should work :) Besides, thanks for spotting it and submittn a patch /sig node /remove-sig testing
Flaky in
merging this PR will fix the issue

ctx, cancel := context.WithCancel(context.Background()) defer cancel() // Start to follow the container's log. <ins> fileName := file.Name() </ins> go func(ctx context.Context) { podLogOptions := v1.PodLogOptions{ Follow: true, } opts := NewLogOptions(&podLogOptions, time.Now()) <del> ReadLogs(ctx, file.Name(), containerID, opts, fakeRuntimeService, stdoutBuf, stderrBuf) </del> <ins> _ = ReadLogs(ctx, fileName, containerID, opts, fakeRuntimeService, stdoutBuf, stderrBuf) </ins> }(ctx) // log in stdout


Question:
Runs k8s with master golang: test Failure was observed on Aug 25th, it is passing after that. However it can be reproduced with stress tool. No response No response It is failing on both ppc and x86 architectures. Steps to reproduce: up golang tip version. clone kubernetes master /sig testing
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
it is a sig node test, sig testing charter is not about doing or maintain tests, we need to remove that old mindset of having a QA department, that is not how software development works or should work :) Besides, thanks for spotting it and submittn a patch /sig node /remove-sig testing
Flaky in
merging this PR will fix the issue

// Write 10 lines to log file. // Let ReadLogs start. time.Sleep(50 * time.Millisecond) for line := 0; line < 10; line++ { // Write the first three lines to log file now := time.Now().Format(types.RFC3339NanoLenient)


Question:
The swap fedora job is failing consistently for OOM. For a while. It looks like OOM is not actually killing these pods when swap is enabled. I did not see any examples of this failing without swap enabled so I think this is a real issue. There is a failure in OOMKilled as part of the serial fedora job but that is not related to this. This is part of and should be required for promotion to Beta2. /sig node
/triage accepted /priority important-longterm

v1 "k8s.io/api/core/v1" "k8s.io/apimachinery/pkg/api/resource" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" <ins> utilfeature "k8s.io/apiserver/pkg/util/feature" "k8s.io/kubernetes/pkg/features" </ins> kubeletconfig "k8s.io/kubernetes/pkg/kubelet/apis/config" "k8s.io/kubernetes/test/e2e/framework" e2epod "k8s.io/kubernetes/test/e2e/framework/pod"


Question:
The swap fedora job is failing consistently for OOM. For a while. It looks like OOM is not actually killing these pods when swap is enabled. I did not see any examples of this failing without swap enabled so I think this is a real issue. There is a failure in OOMKilled as part of the serial fedora job but that is not related to this. This is part of and should be required for promotion to Beta2. /sig node
/triage accepted /priority important-longterm

}) ginkgo.It("The containers terminated by OOM killer should have the reason set to OOMKilled", func() { <ins> cfg, configErr := getCurrentKubeletConfig(context.TODO()) framework.ExpectNoError(configErr) if utilfeature.DefaultFeatureGate.Enabled(features.NodeSwap) { // If Swap is enabled, we should test OOM with LimitedSwap. // UnlimitedSwap allows for workloads to use unbounded swap which // makes testing OOM challenging. // We are not able to change the default for these conformance tests, // so we will skip these tests if swap is enabled. if cfg.MemorySwap.SwapBehavior == "" || cfg.MemorySwap.SwapBehavior == "UnlimitedSwap" { ginkgo.Skip("OOMKiller should not run with UnlimitedSwap") } } </ins> ginkgo.By("Waiting for the pod to be failed") err := e2epod.WaitForPodTerminatedInNamespace(context.TODO(), f.ClientSet, testCase.podSpec.Name, "", f.Namespace.Name) framework.ExpectNoError(err, "Failed waiting for pod to terminate, %s/%s", f.Namespace.Name, testCase.podSpec.Name)


Question:
pkg/registry/core/persistentvolume/storage TestUpdateStatus unknown No response /sig storage
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/priority important-soon as it failed in master blocking

package storage import ( <del> utilfeature "k8s.io/apiserver/pkg/util/feature" featuregatetesting "k8s.io/component-base/featuregate/testing" "k8s.io/kubernetes/pkg/features" </del> <ins> "context" </ins> "testing" <ins> "time" </ins> <del> "context" </del> "github.com/google/go-cmp/cmp" apiequality "k8s.io/apimachinery/pkg/api/equality" "k8s.io/apimachinery/pkg/api/resource"


Question:
pkg/registry/core/persistentvolume/storage TestUpdateStatus unknown No response /sig storage
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/priority important-soon as it failed in master blocking

genericregistrytest "k8s.io/apiserver/pkg/registry/generic/testing" "k8s.io/apiserver/pkg/registry/rest" etcd3testing "k8s.io/apiserver/pkg/storage/etcd3/testing" <ins> utilfeature "k8s.io/apiserver/pkg/util/feature" featuregatetesting "k8s.io/component-base/featuregate/testing" </ins> api "k8s.io/kubernetes/pkg/apis/core" <ins> "k8s.io/kubernetes/pkg/features" </ins> "k8s.io/kubernetes/pkg/registry/core/persistentvolume" "k8s.io/kubernetes/pkg/registry/registrytest" )


Question:
pkg/registry/core/persistentvolume/storage TestUpdateStatus unknown No response /sig storage
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/priority important-soon as it failed in master blocking

t.Errorf("unexpected error: %v", err) } <ins> // We need to set custom timestamp which is not the same as the one on existing PV // - doing so will prevent timestamp update on phase change and custom one is used instead. pvStartTimestamp = &metav1.Time{Time: pvStartTimestamp.Time.Add(time.Second)} </ins> pvIn := &api.PersistentVolume{ ObjectMeta: metav1.ObjectMeta{ Name: "foo", }, Status: api.PersistentVolumeStatus{ <del> Phase: api.VolumeBound, // Set the same timestamp as original PV so this won't get updated on phase change breaking DeepEqual() later in test. </del> <ins> Phase:                   api.VolumeBound, </ins> LastPhaseTransitionTime: pvStartTimestamp, }, }


Question:
When unit tests in are invoked with 1, it fails Unit tests should work for any . This is important for troubleshooting flaking tests. Discovered when I tried to add parallel execution as described in master N/A N/A N/A N/A N/A
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig network /area kube-proxy
/cc

"k8s.io/apimachinery/pkg/util/sets" utilfeature "k8s.io/apiserver/pkg/util/feature" featuregatetesting "k8s.io/component-base/featuregate/testing" <ins> "k8s.io/component-base/metrics/legacyregistry" </ins> "k8s.io/component-base/metrics/testutil" "k8s.io/klog/v2" "k8s.io/kubernetes/pkg/features"


Question:
When unit tests in are invoked with 1, it fails Unit tests should work for any . This is important for troubleshooting flaking tests. Discovered when I tried to add parallel execution as described in master N/A N/A N/A N/A N/A
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig network /area kube-proxy
/cc

ipt := iptablestest.NewFake() fp := NewFakeProxier(ipt) metrics.RegisterMetrics() <ins> defer legacyregistry.Reset() </ins> // Create initial state var svc2 *v1.Service


Question:
When unit tests in are invoked with 1, it fails Unit tests should work for any . This is important for troubleshooting flaking tests. Discovered when I tried to add parallel execution as described in master N/A N/A N/A N/A N/A
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig network /area kube-proxy
/cc

if fp.needFullSync { t.Fatalf("Proxier unexpectedly already needs a full sync?") } <del> prFailures, err := testutil.GetCounterMetricValue(metrics.IptablesPartialRestoreFailuresTotal) </del> <ins> partialRestoreFailures, err := testutil.GetCounterMetricValue(metrics.IptablesPartialRestoreFailuresTotal) </ins> if err != nil { t.Fatalf("Could not get partial restore failures metric: %v", err) } <del> if prFailures != 0.0 { </del> <ins> if partialRestoreFailures != 0.0 { </ins> t.Errorf("Already did a partial resync? Something failed earlier!") }


Question:
When unit tests in are invoked with 1, it fails Unit tests should work for any . This is important for troubleshooting flaking tests. Discovered when I tried to add parallel execution as described in master N/A N/A N/A N/A N/A
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig network /area kube-proxy
/cc

if !fp.needFullSync { t.Errorf("Proxier did not fail on previous partial resync?") } <del> updatedPRFailures, err := testutil.GetCounterMetricValue(metrics.IptablesPartialRestoreFailuresTotal) </del> <ins> updatedPartialRestoreFailures, err := testutil.GetCounterMetricValue(metrics.IptablesPartialRestoreFailuresTotal) </ins> if err != nil { t.Errorf("Could not get partial restore failures metric: %v", err) } <del> if updatedPRFailures != prFailures+1.0 { t.Errorf("Partial restore failures metric was not incremented after failed partial resync (expected %.02f, got %.02f)", prFailures+1.0, updatedPRFailures) </del> <ins> if updatedPartialRestoreFailures != partialRestoreFailures+1.0 { t.Errorf("Partial restore failures metric was not incremented after failed partial resync (expected %.02f, got %.02f)", partialRestoreFailures+1.0, updatedPartialRestoreFailures) </ins> } // On retry we should do a full resync, which should succeed (and delete svc4)


Question:
Created a RWX volume and mounted it on a pod having readOnly flag set to true. While the volume provisioning happened successfully, we did observe some warning message on describing the pod: Warning FileSystemResizeFailed 27m kubelet MountVolume.NodeExpandVolume failed for volume "pvc--a6ba-49f4-a560-" requested read-only file system It seems like this message is being triggered from here: But we never did an expand volume call, so we should not be seeing warning message. It can be confusing to see this warning message. The warning message should not be displayed when expand volume was not even called. Create a RWX volume and mount it on a pod with readOnly flag set to true. Wait for a few minutes and describe the pod. No response_ 28 vSphere <details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage
/assign

return false, fmt.Errorf("mountVolume.NodeExpandVolume get PVC failed : %v", err) } <del> if volumeToMount.VolumeSpec.ReadOnly { simpleMsg, detailedMsg := volumeToMount.GenerateMsg("MountVolume.NodeExpandVolume failed", "requested read-only file system") klog.Warningf(detailedMsg) og.recorder.Eventf(volumeToMount.Pod, v1.EventTypeWarning, kevents.FileSystemResizeFailed, simpleMsg) og.recorder.Eventf(pvc, v1.EventTypeWarning, kevents.FileSystemResizeFailed, simpleMsg) return true, nil } </del> pvcStatusCap := pvc.Status.Capacity[v1.ResourceStorage] pvSpecCap := pv.Spec.Capacity[v1.ResourceStorage] if pvcStatusCap.Cmp(pvSpecCap) < 0 { <ins> if volumeToMount.VolumeSpec.ReadOnly { simpleMsg, detailedMsg := volumeToMount.GenerateMsg("MountVolume.NodeExpandVolume failed", "requested read-only file system") klog.Warningf(detailedMsg) og.recorder.Eventf(volumeToMount.Pod, v1.EventTypeWarning, kevents.FileSystemResizeFailed, simpleMsg) og.recorder.Eventf(pvc, v1.EventTypeWarning, kevents.FileSystemResizeFailed, simpleMsg) return true, nil } </ins> rsOpts.NewSize = pvSpecCap rsOpts.OldSize = pvcStatusCap resizeOp := nodeResizeOperationOpts{


Question:
When running subcommand like this: a warning message will display: which is incorrect in mode. It is only correct in mode. Warning message to only display when flag is passed in. Run No response <details</details<details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cli

} switch { <del> case o.Watch || o.WatchOnly: </del> <ins> case o.Watch: </ins> if len(o.SortBy) > 0 { <del> fmt.Fprintf(o.IOStreams.ErrOut, "warning: --watch or --watch-only requested, --sort-by will be ignoredn") </del> <ins> fmt.Fprintf(o.IOStreams.ErrOut, "warning: --watch requested, --sort-by will be ignored for watch events receivedn") } case o.WatchOnly: if len(o.SortBy) > 0 { fmt.Fprintf(o.IOStreams.ErrOut, "warning: --watch-only requested, --sort-by will be ignoredn") </ins> } default: if len(args) == 0 && cmdutil.IsFilenameSliceEmpty(o.Filenames, o.Kustomize) {


Question:
! Please also see /kind failing-test <!-- If this is a flake, please add: /kind flake --<!-- Please assign a SIG using: /sig SIG-NAME --/sig testing
cc
slack thread:
/triage accepted
got one good run!

scheduledJobTimeout = 5 * time.Minute ) <del> var _ = framework.KubeDescribe("[Feature:ScheduledJob]", func() { </del> <ins> var _ = framework.KubeDescribe("ScheduledJob", func() { </ins> options := framework.FrameworkOptions{ ClientQPS:    20, ClientBurst:  50,


Question:
/kind failing-test /sig node
/assign
/triage accepted
Seems to me that problem is related with test-infra config ? I noticed Checking kubelet logs there are complains about kubelet not finding configuration file. Even if this job run in AWS it is defined under ( ) and not under aws subfolder. Performing vimdiff periodic- ./aws/periodic-eks- i notice some variables are missing, but i don't have permissions to re-run the job to be able to debug more on this. Test code for InPlacePodVerticalScaling doesn't seem to be the root cause here, if we could increase the kubelet log level and perform a rerun might be helpful to debug more. Checking containerd logs , it seems that a container failed to stop, this might explain why pod resize failed due to lack of resources)
These two tests are permanent failing on this job
True, but looking deeper it seems for the first test it fails ( unable to resize there are no available resources ) because a previous pod has not been deleted according to containerd logs . For second error it seems to be a config error , it complains that it could not login. I believe something is wrong with the setup ( repeated kubelet config errors in kubelet logs )
/sig storage
More information for the container delete error, it belongs to mainpod-with-poststart-http-hookrestartable-init-container-lifecycle-hook. I will try to check if InPlacePodVerticalScaling test can be improved to cover this corner case. Please note, Garbage collect error happens before the InPlacePodVerticalScaling test failure, perhaps this is the root cause of the failure in InPlacePodVerticalScaling error. Perhaps we should update Issue title and include the garbage collect error for this issue? ! Source: / . Please search for

nodeAllocatableMilliCPU2, nodeAvailableMilliCPU2 := getNodeAllocatableAndAvailableMilliCPUValues(&node) framework.Logf("TEST2: Node '%s': NodeAllocatable MilliCPUs = %dm. MilliCPUs currently available to allocate = %dm.", node.Name, nodeAllocatableMilliCPU2, nodeAvailableMilliCPU2) <del> testPod3CPUQuantity := resource.NewMilliQuantity(nodeAvailableMilliCPU2+testPod1CPUQuantity.MilliValue()/2, resource.DecimalSI) </del> <ins> testPod3CPUQuantity := resource.NewMilliQuantity(nodeAvailableMilliCPU2+testPod1CPUQuantity.MilliValue()/4, resource.DecimalSI) </ins> testPod1CPUQuantityResized := resource.NewMilliQuantity(testPod1CPUQuantity.MilliValue()/3, resource.DecimalSI) framework.Logf("TEST2: testPod1 MilliCPUs after resize '%dm'", testPod1CPUQuantityResized.MilliValue())


Question:
,%20google-gce all I can't say , but at least a few weeks ,%20google-gce No response No response /sig network
/assign
/triage accepted

<ins> # See the OWNERS docs at https://go.k8s.io/owners approvers: - aojea - danwinship reviewers: - aojea - danwinship </ins>


Question:
,%20google-gce all I can't say , but at least a few weeks ,%20google-gce No response No response /sig network
/assign
/triage accepted

<ins> kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: system:network-policies namespace: kube-system labels: addonmanager.kubernetes.io/mode: Reconcile rules: - apiGroups: [""] resources: - pods - nodes - namespaces verbs: - get - watch - list # Watch for changes to Kubernetes NetworkPolicies. - apiGroups: ["networking.k8s.io"] resources: - networkpolicies verbs: - watch - list --- kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: kube-network-policies labels: addonmanager.kubernetes.io/mode: Reconcile roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: system:network-policies subjects: - kind: ServiceAccount name: kube-network-policies namespace: kube-system --- apiVersion: v1 kind: ServiceAccount metadata: name: kube-network-policies namespace: kube-system labels: k8s-app: kube-network-policies kubernetes.io/cluster-service: "true" addonmanager.kubernetes.io/mode: Reconcile </ins>  No newline at end of file


Question:
,%20google-gce all I can't say , but at least a few weeks ,%20google-gce No response No response /sig network
/assign
/triage accepted

<ins> --- apiVersion: apps/v1 kind: DaemonSet metadata: name: kube-network-policies namespace: kube-system labels: tier: node app: kube-network-policies k8s-app: kube-network-policies addonmanager.kubernetes.io/mode: Reconcile spec: selector: matchLabels: app: kube-network-policies template: metadata: labels: tier: node app: kube-network-policies k8s-app: kube-network-policies spec: hostNetwork: true tolerations: - operator: Exists effect: NoSchedule serviceAccountName: kube-network-policies containers: - name: kube-network-policies image: registry.k8s.io/networking/kube-network-policies:v0.1.0 command: - /bin/sh - -c - /bin/netpol -v 4 1>>/var/log/kube-network-policies.log 2>&1 resources: requests: cpu: "100m" memory: "50Mi" securityContext: privileged: true volumeMounts: - mountPath: /var/log name: varlog readOnly: false - mountPath: /lib/modules name: lib-modules readOnly: true volumes: - name: varlog hostPath: path: /var/log - name: lib-modules hostPath: path: /lib/modules </ins>


Question:
,%20google-gce all I can't say , but at least a few weeks ,%20google-gce No response No response /sig network
/assign
/triage accepted

local -r ds_file="${dst_dir}/calico-policy-controller/calico-node-daemonset.yaml" sed -i -e "s@__CALICO_CNI_DIR__@/home/kubernetes/bin@g" "${ds_file}" fi <ins> if [[ "${NETWORK_POLICY_PROVIDER:-}" == "kube-network-policies" ]]; then setup-addon-manifests "addons" "kube-network-policies" fi </ins> if [[ "${ENABLE_DEFAULT_STORAGE_CLASS:-}" == "true" ]]; then setup-addon-manifests "addons" "storage-class/gce" fi


Question:
,%20google-gce all I can't say , but at least a few weeks ,%20google-gce No response No response /sig network
/assign
/triage accepted

} EOF if [[ "${KUBERNETES_MASTER:-}" != "true" ]]; then <del> if [[ "${NETWORK_POLICY_PROVIDER:-"none"}" != "none" || "${ENABLE_NETD:-}" == "true" ]]; then # Use Kubernetes cni daemonset on node if network policy provider is specified </del> <ins> if [[ "${NETWORK_POLICY_PROVIDER:-"none"}" == "calico" || "${ENABLE_NETD:-}" == "true" ]]; then # Use Kubernetes cni daemonset on node if network policy provider calico is specified </ins> # or netd is enabled. cni_template_path="" fi


Question:
,%20google-gce all I can't say , but at least a few weeks ,%20google-gce No response No response /sig network
/assign
/triage accepted

readonly gcloud_supported_providers="gce gke" readonly master_logfiles="kube-apiserver.log kube-apiserver-audit.log kube-scheduler.log kube-controller-manager.log cloud-controller-manager.log etcd.log etcd-events.log glbc.log cluster-autoscaler.log kube-addon-manager.log konnectivity-server.log fluentd.log kubelet.cov" <del> readonly node_logfiles="kube-proxy.log containers/konnectivity-agent-*.log fluentd.log node-problem-detector.log kubelet.cov" </del> <ins> readonly node_logfiles="kube-proxy.log containers/konnectivity-agent-*.log fluentd.log node-problem-detector.log kubelet.cov kube-network-policies.log" </ins> readonly node_systemd_services="node-problem-detector" readonly hollow_node_logfiles="kubelet-hollow-node-*.log kubeproxy-hollow-node-*.log npd-hollow-node-*.log" readonly aws_logfiles="cloud-init-output.log"


Question:
The following api endpoints are currently in - As each group has no stable endpoints they are reported in as promoted to stable without tests. The current logic for endpoints will address this issue.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig architecture /area test

- endpoint: createAuthenticationV1SelfSubjectReview reason: Cluster providers are allowed to choose to not serve this API, and the whoami command handles unavailability gracefully. link: https://github.com/kubernetes/enhancements/blob/master/keps/sig-auth/3325-self-subject-attributes-review-api/README.md#ga <ins> - endpoint: getInternalApiserverAPIGroup reason: The endpoint is part of a group that has no stable endpoints yet, reporting as a stable endpoint in apisnoop. link: https://github.com/kubernetes/issues/124248 - endpoint: getResourceAPIGroup reason: The endpoint is part of a group that has no stable endpoints yet, reporting as a stable endpoint in apisnoop. link: https://github.com/kubernetes/issues/124248 - endpoint: getStoragemigrationAPIGroup reason: The endpoint is part of a group that has no stable endpoints yet, reporting as a stable endpoint in apisnoop. link: https://github.com/kubernetes/issues/124248 </ins>


Question:
The following api endpoints are currently in - As each group has no stable endpoints they are reported in as promoted to stable without tests. The current logic for endpoints will address this issue.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig architecture /area test

- deleteCoreV1Node - deleteStorageV1CollectionCSINode - deleteStorageV1CSINode <del> - getInternalApiserverAPIGroup - getResourceAPIGroup - getStoragemigrationAPIGroup </del> - listStorageV1CSINode - patchStorageV1CSINode - patchStorageV1VolumeAttachmentStatus


Question:
After restart Kubelet, node will become notReady in first kubelet update period. node condition is : in kubelet's logs: when go in to next period (default 10s), node status resume to Ready. Restart Kubelet should not make node status change to notReady. Restart Kubelet, and watch node status for second The root case maybe here: When kubelet start, it will start two go routine, one detect runtime status, one check node status and update to APIServer; detect runtime status: in it will update : in it will check runtimeStatus include : is one of the notReady setter func: So if nodestatus execute before , kubelet will update a node status with notReady condition: In kubelet starting, both go routine and will be called before , so the first node status is notReady. In our cluster is v1.22.17, but I belive it can reproduce in latest version. <details</details<details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /sig node Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I can reproduce this issue in v1.30 When the container runtime is healthy, the kubelet should not report with the reason We called in the function to update the status of the container runtime, but we did not do this in , We only need to execute function once before and , just like we did in the function, than this problem can be avoided. But in , seems to have some different opinions...
Yeah, in our env, we modify kubelet the same way as you said, to let it execute once before , it worked. I can submit a PR later.
I may be missing something but what happens if something has changed that caused the kubelet to not be ready ... imagine, the Network plugin is not working ... during the initial period until the check changes the state pods will be scheduled on the node and will fail .... I think that we are operating the assumption that this is a restart and during that time nothing changed, but is this a safe assumption or can we guarantee somehow that nothing has changed that could impact the node readiness state?
In this issue, what I found is that nothing has changed, every time I restart kubelet, node will become notReady only in first sync period, in next period it will be ready; It's not an assumption. or other problem of course will cause node notReady, but they are not in this issue's scope, in this scope, notReady reason is message, have not relationship with other problem;
ok, I misread this issue sorry, so it is not to blindly set the node to ready, is just to perform the runtime check before the other checks ... I think you both are right ... actually it seems that if wins the race then there is no problem, righ ?
Yes, but and run in different goroutines, so we cannot guarantee that will complete faster. So it seems like a good choice to execute before running , or do you have better suggestions?
We need to put the goroutines in the right order to make the startup more predictable
Yes, if execute very fast, may there is no problem, because in it will execute . I have test in v1.28 (which code in fastStatusUpdateOnce almost same as 1.30 ) , it will not always reproduce, but still occurr some times. In 1.22, will sleep for about 100ms before everything, it can be reproduced everytimes.
Arranging the goroutines in order can solve our problems in most cases, but in the method, we need to call the containerRuntime api to get the status. I am worried that its response will not be timely enough in special scenarios, there may be a risk that will complete faster.
moving will execute first
I think I must not have expressed clearly. What I'm worried about is that under special circumstances, calls the API of the container runtime and cannot return immediately (not sure if this situation really exists). Even if the method is executed first, it will still complete later than :)
Add some detail: in it call runtime api to check runtime status, then set variable . in , it will check variable, if is nil, this issue's problem will occurr. Even if goroutine call firstly, call container runtime check will still need some time, it can't guarantee that when firstly call, is setted.
are you suggesting to lock syncNodeStatus on updateRuntimeUp? I'm afraid of some corner cases we can hit ... if the container runtime does not return do we block forever?
Container runtime API call have timeout.
Can you try this on a newer version? 1.22 is out of support and we have made changes to node readiness since. /triage needs-information
I have tried in 1.28, it can reproduce, can reproduce in 1.30.
/remove-triage needs-information

// Introduce some small jittering to ensure that over time the requests won't start // accumulating at approximately the same time from the set of nodes due to priority and // fairness effect. <del> go wait.JitterUntil(kl.syncNodeStatus, kl.nodeStatusUpdateFrequency, 0.04, true, wait.NeverStop) </del> <ins> go func() { // Call updateRuntimeUp once before syncNodeStatus to make sure kubelet had already checked runtime state // otherwise when restart kubelet, syncNodeStatus will report node notReady in first report period kl.updateRuntimeUp() wait.JitterUntil(kl.syncNodeStatus, kl.nodeStatusUpdateFrequency, 0.04, true, wait.NeverStop) }() </ins> go kl.fastStatusUpdateOnce() // start syncing lease


Question:
What happened: If you use to test whether a user can approve certificate signing requests, you get a response indicating that is not a valid verb An example of the verb being used can be seen What you expected to happen: should be recognized as a valid verb on objects. How to reproduce it (as minimally and precisely as possible): Run Anything else we need to know?: I think that needs to be to the kubectl auth can-i code Environment: Kubernetes client and server versions (use ): Cloud provider or hardware configuration: OS (e.g: ):
This issue is currently awaiting triage. SIG CLI takes a lead on issue triage for this repo, but any Kubernetes member can accept issues by applying the label. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I don't think is a valid verb for kubectl auth command. Valid verbs are in
so in that line of code there are a number of verbs which are only usuable on specific objects for example which works on users. AFAIK the point of that line is to inform kubectl what the valid verbs are on any object in Kubernetes, so that it knows what's possible when users use the command. the verb is valid on certificate objects as shown in the docs example I listed. Without being present in that line of code, will incorrectly state that there is no verb in Kubernetes.
I tested it, seems working when we add in that list. But the question is should we add it or not?
There have been cases where Kubectl has had additions to handle non-standard resources/verbs in the past (e.g. I raised a couple of years back). AFAIK the explicit list of verbs here is designed to address this point, it's just not got approve in it. You could ask SIG-Auth about it, I'm sure they'd have more info.
/transfer kubenertes /sig auth Decision can be made by sig-auth
Something went wrong or the destination repo kubernetes/kubenertes does not exist. <detailsIn response to : /transfer kubenertes /sig auth Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/transfer kubernetes/kubernetes
Something went wrong or the destination repo kubernetes/kubernetes/kubernetes does not exist. <detailsIn response to : /transfer kubernetes/kubernetes Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/transfer kubernetes
/triage accepted
This will work even without adding verb there, adding verb just removes the warning. Submitted a PR to add that. /assign

# Check to see if I can access the URL /logs/ kubectl auth can-i get /logs/ <ins> # Check to see if I can approve certificates.k8s.io kubectl auth can-i approve certificates.k8s.io </ins> # List all allowed actions in namespace "foo" kubectl auth can-i --list --namespace=foo`) <del> resourceVerbs       = sets.NewString("get", "list", "watch", "create", "update", "patch", "delete", "deletecollection", "use", "bind", "impersonate", "*") </del> <ins> resourceVerbs       = sets.NewString("get", "list", "watch", "create", "update", "patch", "delete", "deletecollection", "use", "bind", "impersonate", "*", "approve") </ins> nonResourceURLVerbs = sets.NewString("get", "put", "post", "head", "options", "delete", "patch", "*") // holds all the server-supported resources that cannot be discovered by clients. i.e. users and groups for the impersonate verb nonStandardResourceNames = sets.NewString("users", "groups")


Question:
What happened: If you use to test whether a user can approve certificate signing requests, you get a response indicating that is not a valid verb An example of the verb being used can be seen What you expected to happen: should be recognized as a valid verb on objects. How to reproduce it (as minimally and precisely as possible): Run Anything else we need to know?: I think that needs to be to the kubectl auth can-i code Environment: Kubernetes client and server versions (use ): Cloud provider or hardware configuration: OS (e.g: ):
This issue is currently awaiting triage. SIG CLI takes a lead on issue triage for this repo, but any Kubernetes member can accept issues by applying the label. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I don't think is a valid verb for kubectl auth command. Valid verbs are in
so in that line of code there are a number of verbs which are only usuable on specific objects for example which works on users. AFAIK the point of that line is to inform kubectl what the valid verbs are on any object in Kubernetes, so that it knows what's possible when users use the command. the verb is valid on certificate objects as shown in the docs example I listed. Without being present in that line of code, will incorrectly state that there is no verb in Kubernetes.
I tested it, seems working when we add in that list. But the question is should we add it or not?
There have been cases where Kubectl has had additions to handle non-standard resources/verbs in the past (e.g. I raised a couple of years back). AFAIK the explicit list of verbs here is designed to address this point, it's just not got approve in it. You could ask SIG-Auth about it, I'm sure they'd have more info.
/transfer kubenertes /sig auth Decision can be made by sig-auth
Something went wrong or the destination repo kubernetes/kubenertes does not exist. <detailsIn response to : /transfer kubenertes /sig auth Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/transfer kubernetes/kubernetes
Something went wrong or the destination repo kubernetes/kubernetes/kubernetes does not exist. <detailsIn response to : /transfer kubernetes/kubernetes Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/transfer kubernetes
/triage accepted
This will work even without adding verb there, adding verb just removes the warning. Submitted a PR to add that. /assign

default: errString := "you must specify two arguments: verb resource or verb resource/resourceName." usageString := "See 'kubectl auth can-i -h' for help and examples." <del> return errors.New(fmt.Sprintf("%sn%s", errString, usageString)) </del> <ins> return fmt.Errorf("%sn%s", errString, usageString) </ins> } }


Question:
Everything 2024-05-24 No response /sig testing
/sig node
kubelet logs dmesg:
Seems a regression in
/cc
/triage accepted /priority important-longterm

"k8s.io/apimachinery/pkg/util/version" "k8s.io/klog/v2" <ins> "k8s.io/kubernetes/pkg/kubelet/userns/inuserns" </ins> utilkernel "k8s.io/kubernetes/pkg/util/kernel" "k8s.io/mount-utils" )


Question:
Everything 2024-05-24 No response /sig testing
/sig node
kubelet logs dmesg:
Seems a regression in
/cc
/triage accepted /priority important-longterm

return false } <ins> if inuserns.RunningInUserNS() { // Turning off swap in unprivileged tmpfs mounts unsupported // https://github.com/torvalds/linux/blob/v6.8/mm/shmem.c#L4004-L4011 // https://github.com/kubernetes/kubernetes/issues/125137 klog.InfoS("Running under a user namespace - tmpfs noswap is not supported") return false } </ins> kernelVersion, err := utilkernel.GetVersion() if err != nil { klog.ErrorS(err, "cannot determine kernel version, unable to determine is tmpfs noswap is supported")


Question:
master-blocking: ci-kubernetes-unit It starts flaking today 30-05-2024 The test fail is This failure may be caused by the error message's order sensitivity. The error message's order of Resource Classes versus is different. N/A /sig node /sig scheduling
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-sig testing /sig scheduling (sig testing doesn't own all of the unit tests, just test frameworks, tooling, CI infra, etc) This appears to be a flake in scheduler unit tests which would be sig scheduling ( )
.. and it appears to be DRA specific subset of scheduling so /sig node as well
FYI - looks like this is happening every 3-6 hours give or take. Yesterday it was every six hours and today it seems to be every 3-4 hours. Looking into it a bit more, it looks like it may just be an issue with a connection via the test while it's trying to hit the server. I don't think the test itself is failing, I think the connection to the server is failing. ! CC
Hey All - hope you're well. Michael here from the Release Signal team. Just wanted to touch base and make sure we're good to go for the cut on the 4th. CC /sig-node

} return parameters, nil default: <ins> sort.Slice(objs, func(i, j int) bool { obj1, obj2 := objs[i].(*resourcev1alpha2.ResourceClassParameters), objs[j].(*resourcev1alpha2.ResourceClassParameters) if obj1 == nil || obj2 == nil { return false } return obj1.Name < obj2.Name }) </ins> return nil, statusError(logger, fmt.Errorf("multiple generated class parameters for %s.%s %s found: %s", class.ParametersRef.Kind, class.ParametersRef.APIGroup, klog.KRef(class.Namespace, class.ParametersRef.Name), klog.KObjSlice(objs))) } }


Question:
master-blocking: ci-kubernetes-unit It starts flaking today 30-05-2024 The test fail is This failure may be caused by the error message's order sensitivity. The error message's order of Resource Classes versus is different. N/A /sig node /sig scheduling
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-sig testing /sig scheduling (sig testing doesn't own all of the unit tests, just test frameworks, tooling, CI infra, etc) This appears to be a flake in scheduler unit tests which would be sig scheduling ( )
.. and it appears to be DRA specific subset of scheduling so /sig node as well
FYI - looks like this is happening every 3-6 hours give or take. Yesterday it was every six hours and today it seems to be every 3-4 hours. Looking into it a bit more, it looks like it may just be an issue with a connection via the test while it's trying to hit the server. I don't think the test itself is failing, I think the connection to the server is failing. ! CC
Hey All - hope you're well. Michael here from the Release Signal team. Just wanted to touch base and make sure we're good to go for the cut on the 4th. CC /sig-node

} return parameters, nil default: <ins> sort.Slice(objs, func(i, j int) bool { obj1, obj2 := objs[i].(*resourcev1alpha2.ResourceClaimParameters), objs[j].(*resourcev1alpha2.ResourceClaimParameters) if obj1 == nil || obj2 == nil { return false } return obj1.Name < obj2.Name }) </ins> return nil, statusError(logger, fmt.Errorf("multiple generated claim parameters for %s.%s %s found: %s", claim.Spec.ParametersRef.Kind, claim.Spec.ParametersRef.APIGroup, klog.KRef(claim.Namespace, claim.Spec.ParametersRef.Name), klog.KObjSlice(objs))) } }


Question:
Kubelet soft admission handlers were created as a way for the Kubelet to block pods that cannot be run for reasons that can be resolved. In practice, this mechanism was only ever used by AppArmor (). Problems with soft-admission rejection: is only used by AppArmor, and only for situations that cannot realistically be resolved (host or build does not support apparmor) held in the blocked state must be manually cleaned up. While there they hold reserved capacity from the scheduler and continually retry syncpod. other features work this way - usually Kubelet hard rejects pods that it knows it cannot run, or the runtime fails to start the pod/container. This was originally to prevent controllers from continuously recreating pods that cannot run, but that is a separate problem that should be approached globally, rather than with this 1-off AppArmor solution. I propose we remove Kubelet soft admission entirely, and move the AppArmor admission handler to the hard-admission list. /sig node /milestone v1.31 /kind bug
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
We have a different, alpha mechanism: . We could change AppArmor, seccomp and other mechanisms to work either with PodSchedulingContext or its eventual replacement. IMO we only need to sketch out an approach and see that it looks viable; we don't have to block graduation for AppArmor support on a fleshed-out design.
From my admittedly limited PoV I'm not against removing soft admission, and I got no opinions about AppArmor specifically. However this seems to connected to the recent conversation we're having in WG serving and SIG-node about retriable vs unretriable errors so I think we would need some coordination cc
+1
My understanding is that after this PR: soft admission just doesn't make sense any longer.
Hello ! Just a reminder, the code freeze is starting (about from now), and while there is still plenty of time, we want to ensure that each PR has a chance to be merged on time. As the issue is tagged for , is it still planned for this release?
Discussed in sig-node, decided to move forward with this (move AppArmor admission to the hard admission set, remove soft-admission code)
/assign
- do you want me to incorporate this change into

if sysruntime.GOOS == "linux" { // AppArmor is a Linux kernel security module and it does not support other operating systems. klet.appArmorValidator = apparmor.NewValidator() <del> klet.softAdmitHandlers.AddPodAdmitHandler(lifecycle.NewAppArmorAdmitHandler(klet.appArmorValidator)) </del> <ins> klet.admitHandlers.AddPodAdmitHandler(lifecycle.NewAppArmorAdmitHandler(klet.appArmorValidator)) </ins> } leaseDuration := time.Duration(kubeCfg.NodeLeaseDurationSeconds) * time.Second


Question:
Kubelet soft admission handlers were created as a way for the Kubelet to block pods that cannot be run for reasons that can be resolved. In practice, this mechanism was only ever used by AppArmor (). Problems with soft-admission rejection: is only used by AppArmor, and only for situations that cannot realistically be resolved (host or build does not support apparmor) held in the blocked state must be manually cleaned up. While there they hold reserved capacity from the scheduler and continually retry syncpod. other features work this way - usually Kubelet hard rejects pods that it knows it cannot run, or the runtime fails to start the pod/container. This was originally to prevent controllers from continuously recreating pods that cannot run, but that is a separate problem that should be approached globally, rather than with this 1-off AppArmor solution. I propose we remove Kubelet soft admission entirely, and move the AppArmor admission handler to the hard-admission list. /sig node /milestone v1.31 /kind bug
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
We have a different, alpha mechanism: . We could change AppArmor, seccomp and other mechanisms to work either with PodSchedulingContext or its eventual replacement. IMO we only need to sketch out an approach and see that it looks viable; we don't have to block graduation for AppArmor support on a fleshed-out design.
From my admittedly limited PoV I'm not against removing soft admission, and I got no opinions about AppArmor specifically. However this seems to connected to the recent conversation we're having in WG serving and SIG-node about retriable vs unretriable errors so I think we would need some coordination cc
+1
My understanding is that after this PR: soft admission just doesn't make sense any longer.
Hello ! Just a reminder, the code freeze is starting (about from now), and while there is still plenty of time, we want to ensure that each PR has a chance to be merged on time. As the issue is tagged for , is it still planned for this release?
Discussed in sig-node, decided to move forward with this (move AppArmor admission to the hard admission set, remove soft-admission code)
/assign
- do you want me to incorporate this change into

// the list of handlers to call during pod admission. admitHandlers lifecycle.PodAdmitHandlers <del> // softAdmithandlers are applied to the pod after it is admitted by the Kubelet, but before it is // run. A pod rejected by a softAdmitHandler will be left in a Pending state indefinitely. If a // rejected pod should not be recreated, or the scheduler is not aware of the rejection rule, the // admission rule should be applied by a softAdmitHandler. softAdmitHandlers lifecycle.PodAdmitHandlers </del> // the list of handlers to call during pod sync loop. lifecycle.PodSyncLoopHandlers


Question:
Kubelet soft admission handlers were created as a way for the Kubelet to block pods that cannot be run for reasons that can be resolved. In practice, this mechanism was only ever used by AppArmor (). Problems with soft-admission rejection: is only used by AppArmor, and only for situations that cannot realistically be resolved (host or build does not support apparmor) held in the blocked state must be manually cleaned up. While there they hold reserved capacity from the scheduler and continually retry syncpod. other features work this way - usually Kubelet hard rejects pods that it knows it cannot run, or the runtime fails to start the pod/container. This was originally to prevent controllers from continuously recreating pods that cannot run, but that is a separate problem that should be approached globally, rather than with this 1-off AppArmor solution. I propose we remove Kubelet soft admission entirely, and move the AppArmor admission handler to the hard-admission list. /sig node /milestone v1.31 /kind bug
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
We have a different, alpha mechanism: . We could change AppArmor, seccomp and other mechanisms to work either with PodSchedulingContext or its eventual replacement. IMO we only need to sketch out an approach and see that it looks viable; we don't have to block graduation for AppArmor support on a fleshed-out design.
From my admittedly limited PoV I'm not against removing soft admission, and I got no opinions about AppArmor specifically. However this seems to connected to the recent conversation we're having in WG serving and SIG-node about retriable vs unretriable errors so I think we would need some coordination cc
+1
My understanding is that after this PR: soft admission just doesn't make sense any longer.
Hello ! Just a reminder, the code freeze is starting (about from now), and while there is still plenty of time, we want to ensure that each PR has a chance to be merged on time. As the issue is tagged for , is it still planned for this release?
Discussed in sig-node, decided to move forward with this (move AppArmor admission to the hard admission set, remove soft-admission code)
/assign
- do you want me to incorporate this change into

return isTerminal, nil } <del> // If the pod should not be running, we request the pod's containers be stopped. This is not the same // as termination (we want to stop the pod, but potentially restart it later if soft admission allows // it later). Set the status and phase appropriately runnable := kl.canRunPod(pod) if !runnable.Admit { // Pod is not runnable; and update the Pod and Container statuses to why. if apiPodStatus.Phase != v1.PodFailed && apiPodStatus.Phase != v1.PodSucceeded { apiPodStatus.Phase = v1.PodPending } apiPodStatus.Reason = runnable.Reason apiPodStatus.Message = runnable.Message // Waiting containers are not creating. const waitingReason = "Blocked" for _, cs := range apiPodStatus.InitContainerStatuses { if cs.State.Waiting != nil { cs.State.Waiting.Reason = waitingReason } } for _, cs := range apiPodStatus.ContainerStatuses { if cs.State.Waiting != nil { cs.State.Waiting.Reason = waitingReason } } } </del> // Record the time it takes for the pod to become running // since kubelet first saw the pod if firstSeenTime is set. existingStatus, ok := kl.statusManager.GetPodStatus(pod.UID)


Question:
Kubelet soft admission handlers were created as a way for the Kubelet to block pods that cannot be run for reasons that can be resolved. In practice, this mechanism was only ever used by AppArmor (). Problems with soft-admission rejection: is only used by AppArmor, and only for situations that cannot realistically be resolved (host or build does not support apparmor) held in the blocked state must be manually cleaned up. While there they hold reserved capacity from the scheduler and continually retry syncpod. other features work this way - usually Kubelet hard rejects pods that it knows it cannot run, or the runtime fails to start the pod/container. This was originally to prevent controllers from continuously recreating pods that cannot run, but that is a separate problem that should be approached globally, rather than with this 1-off AppArmor solution. I propose we remove Kubelet soft admission entirely, and move the AppArmor admission handler to the hard-admission list. /sig node /milestone v1.31 /kind bug
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
We have a different, alpha mechanism: . We could change AppArmor, seccomp and other mechanisms to work either with PodSchedulingContext or its eventual replacement. IMO we only need to sketch out an approach and see that it looks viable; we don't have to block graduation for AppArmor support on a fleshed-out design.
From my admittedly limited PoV I'm not against removing soft admission, and I got no opinions about AppArmor specifically. However this seems to connected to the recent conversation we're having in WG serving and SIG-node about retriable vs unretriable errors so I think we would need some coordination cc
+1
My understanding is that after this PR: soft admission just doesn't make sense any longer.
Hello ! Just a reminder, the code freeze is starting (about from now), and while there is still plenty of time, we want to ensure that each PR has a chance to be merged on time. As the issue is tagged for , is it still planned for this release?
Discussed in sig-node, decided to move forward with this (move AppArmor admission to the hard admission set, remove soft-admission code)
/assign
- do you want me to incorporate this change into

kl.statusManager.SetPodStatus(pod, apiPodStatus) <del> // Pods that are not runnable must be stopped - return a typed error to the pod worker if !runnable.Admit { klog.V(2).InfoS("Pod is not runnable and must have running containers stopped", "pod", klog.KObj(pod), "podUID", pod.UID, "message", runnable.Message) var syncErr error p := kubecontainer.ConvertPodStatusToRunningPod(kl.getRuntime().Type(), podStatus) if err := kl.killPod(ctx, pod, p, nil); err != nil { if !wait.Interrupted(err) { kl.recorder.Eventf(pod, v1.EventTypeWarning, events.FailedToKillPod, "error killing pod: %v", err) syncErr = fmt.Errorf("error killing pod: %w", err) utilruntime.HandleError(syncErr) } } else { // There was no error killing the pod, but the pod cannot be run. // Return an error to signal that the sync loop should back off. syncErr = fmt.Errorf("pod cannot be run: %v", runnable.Message) } return false, syncErr } </del> // If the network plugin is not ready, only start the pod if it uses the host network if err := kl.runtimeState.networkErrors(); err != nil && !kubecontainer.IsHostNetworkPod(pod) { kl.recorder.Eventf(pod, v1.EventTypeWarning, events.NetworkNotReady, "%s: %v", NetworkNotReadyErrorMsg, err)


Question:
Kubelet soft admission handlers were created as a way for the Kubelet to block pods that cannot be run for reasons that can be resolved. In practice, this mechanism was only ever used by AppArmor (). Problems with soft-admission rejection: is only used by AppArmor, and only for situations that cannot realistically be resolved (host or build does not support apparmor) held in the blocked state must be manually cleaned up. While there they hold reserved capacity from the scheduler and continually retry syncpod. other features work this way - usually Kubelet hard rejects pods that it knows it cannot run, or the runtime fails to start the pod/container. This was originally to prevent controllers from continuously recreating pods that cannot run, but that is a separate problem that should be approached globally, rather than with this 1-off AppArmor solution. I propose we remove Kubelet soft admission entirely, and move the AppArmor admission handler to the hard-admission list. /sig node /milestone v1.31 /kind bug
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
We have a different, alpha mechanism: . We could change AppArmor, seccomp and other mechanisms to work either with PodSchedulingContext or its eventual replacement. IMO we only need to sketch out an approach and see that it looks viable; we don't have to block graduation for AppArmor support on a fleshed-out design.
From my admittedly limited PoV I'm not against removing soft admission, and I got no opinions about AppArmor specifically. However this seems to connected to the recent conversation we're having in WG serving and SIG-node about retriable vs unretriable errors so I think we would need some coordination cc
+1
My understanding is that after this PR: soft admission just doesn't make sense any longer.
Hello ! Just a reminder, the code freeze is starting (about from now), and while there is still plenty of time, we want to ensure that each PR has a chance to be merged on time. As the issue is tagged for , is it still planned for this release?
Discussed in sig-node, decided to move forward with this (move AppArmor admission to the hard admission set, remove soft-admission code)
/assign
- do you want me to incorporate this change into

return true, "", "" } <del> func (kl *Kubelet) canRunPod(pod *v1.Pod) lifecycle.PodAdmitResult { attrs := &lifecycle.PodAdmitAttributes{Pod: pod} // Get "OtherPods". Rejected pods are failed, so only include admitted pods that are alive. attrs.OtherPods = kl.GetActivePods() for _, handler := range kl.softAdmitHandlers { if result := handler.Admit(attrs); !result.Admit { return result } } return lifecycle.PodAdmitResult{Admit: true} } </del> // syncLoop is the main loop for processing changes. It watches for changes from // three channels (file, apiserver, and http) and creates a union of them. For // any new change seen, will run a sync against desired state and running state. If


Question:
pull-kubernetes-unit TestStoreListResourceVersion in seems like July 7th 2024: ! Don't know, here is the log: No response /sig
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig api-machinery
See also
/sig etcd
/assign /cc
Bisected this to cc
I'm not sure if this is the proper way to fix it, but setting a limit in the ListOptions deflakes the test. The flakiness comes from the fact that the cache is sometimes not initialized in time which causes it to return a "too many requests". Perhaps in tests we could wait till the cache is ready in order to proceed?
Right, setting a limit when a resource version is set would delegate the list to etcd when cache is not initialized based on the KEP description. It is due to this type of request is not that costly to etcd. Looks like it is exactly how the wait cache ready is in See below So I think we should implement this same thing in when initializing the .
Agree - waiting for cacher to be initialized when creating it here is a better fix for it:
I thought about this as well, but waiting on is not possible outside of the cacher module, since it is a private field. I'm thinking to either add an option to cacherstorage.Config and make wait or add a new public method to . Do you guys have a better idea? Thanks.
Exposing Wait(context) method sounds fine
I have pushed the changes we discussed. Thank you all for your guidance.

if err != nil { t.Fatalf("Couldn't create cacher: %v", err) } <ins> if utilfeature.DefaultFeatureGate.Enabled(features.ResilientWatchCacheInitialization) { // The tests assume that Get/GetList/Watch calls shouldn't fail. // However, 429 error can now be returned if watchcache is under initialization. // To avoid rewriting all tests, we wait for watchcache to initialize. if err := cacher.Wait(context.Background()); err != nil { t.Fatal(err) } } </ins> d := destroyFunc s = cacher destroyFunc = func() {


Question:
pull-kubernetes-unit TestStoreListResourceVersion in seems like July 7th 2024: ! Don't know, here is the log: No response /sig
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig api-machinery
See also
/sig etcd
/assign /cc
Bisected this to cc
I'm not sure if this is the proper way to fix it, but setting a limit in the ListOptions deflakes the test. The flakiness comes from the fact that the cache is sometimes not initialized in time which causes it to return a "too many requests". Perhaps in tests we could wait till the cache is ready in order to proceed?
Right, setting a limit when a resource version is set would delegate the list to etcd when cache is not initialized based on the KEP description. It is due to this type of request is not that costly to etcd. Looks like it is exactly how the wait cache ready is in See below So I think we should implement this same thing in when initializing the .
Agree - waiting for cacher to be initialized when creating it here is a better fix for it:
I thought about this as well, but waiting on is not possible outside of the cacher module, since it is a private field. I'm thinking to either add an option to cacherstorage.Config and make wait or add a new public method to . Do you guys have a better idea? Thanks.
Exposing Wait(context) method sounds fine
I have pushed the changes we discussed. Thank you all for your guidance.

return nil } <ins> // Wait blocks until the cacher is Ready or Stopped, it returns an error if Stopped. func (c *Cacher) Wait(ctx context.Context) error { return c.ready.wait(ctx) } </ins> // errWatcher implements watch.Interface to return a single error type errWatcher struct { result chan watch.Event


Question:
pull-kubernetes-unit TestStoreListResourceVersion in seems like July 7th 2024: ! Don't know, here is the log: No response /sig
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig api-machinery
See also
/sig etcd
/assign /cc
Bisected this to cc
I'm not sure if this is the proper way to fix it, but setting a limit in the ListOptions deflakes the test. The flakiness comes from the fact that the cache is sometimes not initialized in time which causes it to return a "too many requests". Perhaps in tests we could wait till the cache is ready in order to proceed?
Right, setting a limit when a resource version is set would delegate the list to etcd when cache is not initialized based on the KEP description. It is due to this type of request is not that costly to etcd. Looks like it is exactly how the wait cache ready is in See below So I think we should implement this same thing in when initializing the .
Agree - waiting for cacher to be initialized when creating it here is a better fix for it:
I thought about this as well, but waiting on is not possible outside of the cacher module, since it is a private field. I'm thinking to either add an option to cacherstorage.Config and make wait or add a new public method to . Do you guys have a better idea? Thanks.
Exposing Wait(context) method sounds fine
I have pushed the changes we discussed. Thank you all for your guidance.

if utilfeature.DefaultFeatureGate.Enabled(features.ResilientWatchCacheInitialization) { // The tests assume that Get/GetList/Watch calls shouldn't fail. // However, 429 error can now be returned if watchcache is under initialization. <del> // To avoid rewriting all tests, we wait for watcache to initialize. if err := cacher.ready.wait(ctx); err != nil { </del> <ins> // To avoid rewriting all tests, we wait for watchcache to initialize. if err := cacher.Wait(ctx); err != nil { </ins> t.Fatal(err) } }


Question:
pull-kubernetes-unit TestStoreListResourceVersion in seems like July 7th 2024: ! Don't know, here is the log: No response /sig
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig api-machinery
See also
/sig etcd
/assign /cc
Bisected this to cc
I'm not sure if this is the proper way to fix it, but setting a limit in the ListOptions deflakes the test. The flakiness comes from the fact that the cache is sometimes not initialized in time which causes it to return a "too many requests". Perhaps in tests we could wait till the cache is ready in order to proceed?
Right, setting a limit when a resource version is set would delegate the list to etcd when cache is not initialized based on the KEP description. It is due to this type of request is not that costly to etcd. Looks like it is exactly how the wait cache ready is in See below So I think we should implement this same thing in when initializing the .
Agree - waiting for cacher to be initialized when creating it here is a better fix for it:
I thought about this as well, but waiting on is not possible outside of the cacher module, since it is a private field. I'm thinking to either add an option to cacherstorage.Config and make wait or add a new public method to . Do you guys have a better idea? Thanks.
Exposing Wait(context) method sounds fine
I have pushed the changes we discussed. Thank you all for your guidance.

// The tests assume that Get/GetList/Watch calls shouldn't fail. // However, 429 error can now be returned if watchcache is under initialization. // To avoid rewriting all tests, we wait for watcache to initialize. <del> if err := cacher.ready.wait(context.Background()); err != nil { </del> <ins> if err := cacher.Wait(context.Background()); err != nil { </ins> return nil, storage.APIObjectVersioner{}, err } }


Question:
The PR above moved everything related to into because that was where most of the code already was, but didn't renamed the files to avoid a larger diff and merge conflicts. Once is merged, we can do the renaming. Code clarity. /sig node
/triage accepted /priority important-soon
/assign
a PR can be created for this now.
will do

<del> /* Copyright 2022 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package plugin import ( "context" "errors" "fmt" "net" "sync" "time" "google.golang.org/grpc" "google.golang.org/grpc/connectivity" "google.golang.org/grpc/credentials/insecure" utilversion "k8s.io/apimachinery/pkg/util/version" "k8s.io/klog/v2" drapb "k8s.io/kubelet/pkg/apis/dra/v1alpha4" ) // NewDRAPluginClient returns a wrapper around those gRPC methods of a DRA // driver kubelet plugin which need to be called by kubelet. The wrapper // handles gRPC connection management and logging. Connections are reused // across different NewDRAPluginClient calls. func NewDRAPluginClient(pluginName string) (*Plugin, error) { if pluginName == "" { return nil, fmt.Errorf("plugin name is empty") } existingPlugin := draPlugins.get(pluginName) if existingPlugin == nil { return nil, fmt.Errorf("plugin name %s not found in the list of registered DRA plugins", pluginName) } return existingPlugin, nil } type Plugin struct { backgroundCtx context.Context cancel        func(cause error) mutex                   sync.Mutex conn                    *grpc.ClientConn endpoint                string highestSupportedVersion *utilversion.Version clientCallTimeout       time.Duration } func (p *Plugin) getOrCreateGRPCConn() (*grpc.ClientConn, error) { p.mutex.Lock() defer p.mutex.Unlock() if p.conn != nil { return p.conn, nil } ctx := p.backgroundCtx logger := klog.FromContext(ctx) network := "unix" logger.V(4).Info("Creating new gRPC connection", "protocol", network, "endpoint", p.endpoint) // grpc.Dial is deprecated. grpc.NewClient should be used instead. // For now this gets ignored because this function is meant to establish // the connection, with the one second timeout below. Perhaps that // approach should be reconsidered? //nolint:staticcheck conn, err := grpc.Dial( p.endpoint, grpc.WithTransportCredentials(insecure.NewCredentials()), grpc.WithContextDialer(func(ctx context.Context, target string) (net.Conn, error) { return (&net.Dialer{}).DialContext(ctx, network, target) }), ) if err != nil { return nil, err } ctx, cancel := context.WithTimeout(context.Background(), time.Second) defer cancel() if ok := conn.WaitForStateChange(ctx, connectivity.Connecting); !ok { return nil, errors.New("timed out waiting for gRPC connection to be ready") } p.conn = conn return p.conn, nil } func (p *Plugin) NodePrepareResources( ctx context.Context, req *drapb.NodePrepareResourcesRequest, opts ...grpc.CallOption, ) (*drapb.NodePrepareResourcesResponse, error) { logger := klog.FromContext(ctx) logger.V(4).Info("Calling NodePrepareResources rpc", "request", req) conn, err := p.getOrCreateGRPCConn() if err != nil { return nil, err } ctx, cancel := context.WithTimeout(ctx, p.clientCallTimeout) defer cancel() nodeClient := drapb.NewNodeClient(conn) response, err := nodeClient.NodePrepareResources(ctx, req) logger.V(4).Info("Done calling NodePrepareResources rpc", "response", response, "err", err) return response, err } func (p *Plugin) NodeUnprepareResources( ctx context.Context, req *drapb.NodeUnprepareResourcesRequest, opts ...grpc.CallOption, ) (*drapb.NodeUnprepareResourcesResponse, error) { logger := klog.FromContext(ctx) logger.V(4).Info("Calling NodeUnprepareResource rpc", "request", req) conn, err := p.getOrCreateGRPCConn() if err != nil { return nil, err } ctx, cancel := context.WithTimeout(ctx, p.clientCallTimeout) defer cancel() nodeClient := drapb.NewNodeClient(conn) response, err := nodeClient.NodeUnprepareResources(ctx, req) logger.V(4).Info("Done calling NodeUnprepareResources rpc", "response", response, "err", err) return response, err } </del>


Question:
The PR above moved everything related to into because that was where most of the code already was, but didn't renamed the files to avoid a larger diff and merge conflicts. Once is merged, we can do the renaming. Code clarity. /sig node
/triage accepted /priority important-soon
/assign
a PR can be created for this now.
will do

<del> /* Copyright 2023 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package plugin import ( "context" "fmt" "net" "os" "path/filepath" "sync" "testing" "github.com/stretchr/testify/assert" "google.golang.org/grpc" drapb "k8s.io/kubelet/pkg/apis/dra/v1alpha4" "k8s.io/kubernetes/test/utils/ktesting" ) const ( v1alpha4Version = "v1alpha4" ) type fakeV1alpha4GRPCServer struct { drapb.UnimplementedNodeServer } var _ drapb.NodeServer = &fakeV1alpha4GRPCServer{} func (f *fakeV1alpha4GRPCServer) NodePrepareResources(ctx context.Context, in *drapb.NodePrepareResourcesRequest) (*drapb.NodePrepareResourcesResponse, error) { return &drapb.NodePrepareResourcesResponse{Claims: map[string]*drapb.NodePrepareResourceResponse{"claim-uid": { Devices: []*drapb.Device{ { RequestNames: []string{"test-request"}, CDIDeviceIDs: []string{"test-cdi-id"}, }, }, }}}, nil } func (f *fakeV1alpha4GRPCServer) NodeUnprepareResources(ctx context.Context, in *drapb.NodeUnprepareResourcesRequest) (*drapb.NodeUnprepareResourcesResponse, error) { return &drapb.NodeUnprepareResourcesResponse{}, nil } type tearDown func() func setupFakeGRPCServer(version string) (string, tearDown, error) { p, err := os.MkdirTemp("", "dra_plugin") if err != nil { return "", nil, err } closeCh := make(chan struct{}) addr := filepath.Join(p, "server.sock") teardown := func() { close(closeCh) os.RemoveAll(addr) } listener, err := net.Listen("unix", addr) if err != nil { teardown() return "", nil, err } s := grpc.NewServer() switch version { case v1alpha4Version: fakeGRPCServer := &fakeV1alpha4GRPCServer{} drapb.RegisterNodeServer(s, fakeGRPCServer) default: return "", nil, fmt.Errorf("unsupported version: %s", version) } go func() { go s.Serve(listener) <-closeCh s.GracefulStop() }() return addr, teardown, nil } func TestGRPCConnIsReused(t *testing.T) { tCtx := ktesting.Init(t) addr, teardown, err := setupFakeGRPCServer(v1alpha4Version) if err != nil { t.Fatal(err) } defer teardown() reusedConns := make(map[*grpc.ClientConn]int) wg := sync.WaitGroup{} m := sync.Mutex{} p := &Plugin{ backgroundCtx:     tCtx, endpoint:          addr, clientCallTimeout: defaultClientCallTimeout, } conn, err := p.getOrCreateGRPCConn() defer func() { err := conn.Close() if err != nil { t.Error(err) } }() if err != nil { t.Fatal(err) } // ensure the plugin we are using is registered draPlugins.add("dummy-plugin", p) defer draPlugins.delete("dummy-plugin") // we call `NodePrepareResource` 2 times and check whether a new connection is created or the same is reused for i := 0; i < 2; i++ { wg.Add(1) go func() { defer wg.Done() client, err := NewDRAPluginClient("dummy-plugin") if err != nil { t.Error(err) return } req := &drapb.NodePrepareResourcesRequest{ Claims: []*drapb.Claim{ { Namespace: "dummy-namespace", UID:       "dummy-uid", Name:      "dummy-claim", }, }, } _, err = client.NodePrepareResources(tCtx, req) assert.NoError(t, err) client.mutex.Lock() conn := client.conn client.mutex.Unlock() m.Lock() defer m.Unlock() reusedConns[conn]++ }() } wg.Wait() // We should have only one entry otherwise it means another gRPC connection has been created if len(reusedConns) != 1 { t.Errorf("expected length to be 1 but got %d", len(reusedConns)) } if counter, ok := reusedConns[conn]; ok && counter != 2 { t.Errorf("expected counter to be 2 but got %d", counter) } } func TestNewDRAPluginClient(t *testing.T) { for _, test := range []struct { description string setup       func(string) tearDown pluginName  string shouldError bool }{ { description: "plugin name is empty", setup: func(_ string) tearDown { return func() {} }, pluginName:  "", shouldError: true, }, { description: "plugin name not found in the list", setup: func(_ string) tearDown { return func() {} }, pluginName:  "plugin-name-not-found-in-the-list", shouldError: true, }, { description: "plugin exists", setup: func(name string) tearDown { draPlugins.add(name, &Plugin{}) return func() { draPlugins.delete(name) } }, pluginName: "dummy-plugin", }, } { t.Run(test.description, func(t *testing.T) { teardown := test.setup(test.pluginName) defer teardown() client, err := NewDRAPluginClient(test.pluginName) if test.shouldError { assert.Nil(t, client) assert.Error(t, err) } else { assert.NotNil(t, client) assert.Nil(t, err) } }) } } func TestNodeUnprepareResources(t *testing.T) { for _, test := range []struct { description   string serverSetup   func(string) (string, tearDown, error) serverVersion string request       *drapb.NodeUnprepareResourcesRequest }{ { description:   "server supports v1alpha4", serverSetup:   setupFakeGRPCServer, serverVersion: v1alpha4Version, request:       &drapb.NodeUnprepareResourcesRequest{}, }, } { t.Run(test.description, func(t *testing.T) { tCtx := ktesting.Init(t) addr, teardown, err := setupFakeGRPCServer(test.serverVersion) if err != nil { t.Fatal(err) } defer teardown() p := &Plugin{ backgroundCtx:     tCtx, endpoint:          addr, clientCallTimeout: defaultClientCallTimeout, } conn, err := p.getOrCreateGRPCConn() defer func() { err := conn.Close() if err != nil { t.Error(err) } }() if err != nil { t.Fatal(err) } draPlugins.add("dummy-plugin", p) defer draPlugins.delete("dummy-plugin") client, err := NewDRAPluginClient("dummy-plugin") if err != nil { t.Fatal(err) } _, err = client.NodeUnprepareResources(tCtx, test.request) if err != nil { t.Fatal(err) } }) } } </del>


Question:
The PR above moved everything related to into because that was where most of the code already was, but didn't renamed the files to avoid a larger diff and merge conflicts. Once is merged, we can do the renaming. Code clarity. /sig node
/triage accepted /priority important-soon
/assign
a PR can be created for this now.
will do

"context" "errors" "fmt" <ins> "net" "sync" </ins> "time" <del> v1 "k8s.io/api/core/v1" resourceapi "k8s.io/api/resource/v1alpha3" apierrors "k8s.io/apimachinery/pkg/api/errors" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/fields" </del> <ins> "google.golang.org/grpc" "google.golang.org/grpc/connectivity" "google.golang.org/grpc/credentials/insecure" </ins> utilversion "k8s.io/apimachinery/pkg/util/version" <del> "k8s.io/apimachinery/pkg/util/wait" "k8s.io/client-go/kubernetes" </del> "k8s.io/klog/v2" <del> "k8s.io/kubernetes/pkg/kubelet/pluginmanager/cache" </del> <ins> drapb "k8s.io/kubelet/pkg/apis/dra/v1alpha4" </ins> ) <del> // defaultClientCallTimeout is the default amount of time that a DRA driver has // to respond to any of the gRPC calls. kubelet uses this value by passing nil // to RegisterPlugin. Some tests use a different, usually shorter timeout to // speed up testing. // // This is half of the kubelet retry period (according to // https://github.com/kubernetes/kubernetes/commit/0449cef8fd5217d394c5cd331d852bd50983e6b3). const defaultClientCallTimeout = 45 * time.Second // RegistrationHandler is the handler which is fed to the pluginwatcher API. type RegistrationHandler struct { // backgroundCtx is used for all future activities of the handler. // This is necessary because it implements APIs which don't // provide a context. backgroundCtx context.Context kubeClient    kubernetes.Interface getNode       func() (*v1.Node, error) } </del> <ins> // NewDRAPluginClient returns a wrapper around those gRPC methods of a DRA // driver kubelet plugin which need to be called by kubelet. The wrapper // handles gRPC connection management and logging. Connections are reused // across different NewDRAPluginClient calls. func NewDRAPluginClient(pluginName string) (*Plugin, error) { if pluginName == "" { return nil, fmt.Errorf("plugin name is empty") } </ins> <del> var _ cache.PluginHandler = &RegistrationHandler{} // NewPluginHandler returns new registration handler. // // Must only be called once per process because it manages global state. // If a kubeClient is provided, then it synchronizes ResourceSlices // with the resource information provided by plugins. func NewRegistrationHandler(kubeClient kubernetes.Interface, getNode func() (*v1.Node, error)) *RegistrationHandler { handler := &RegistrationHandler{ // The context and thus logger should come from the caller. backgroundCtx: klog.NewContext(context.TODO(), klog.LoggerWithName(klog.TODO(), "DRA registration handler")), kubeClient:    kubeClient, getNode:       getNode, </del> <ins> existingPlugin := draPlugins.get(pluginName) if existingPlugin == nil { return nil, fmt.Errorf("plugin name %s not found in the list of registered DRA plugins", pluginName) </ins> } <del> // When kubelet starts up, no DRA driver has registered yet. None of // the drivers are usable until they come back, which might not happen // at all. Therefore it is better to not advertise any local resources // because pods could get stuck on the node waiting for the driver // to start up. // // This has to run in the background. go handler.wipeResourceSlices("") </del> <ins> return existingPlugin, nil } type Plugin struct { backgroundCtx context.Context cancel        func(cause error) </ins> <del> return handler </del> <ins> mutex                   sync.Mutex conn                    *grpc.ClientConn endpoint                string highestSupportedVersion *utilversion.Version clientCallTimeout       time.Duration </ins> } <del> // wipeResourceSlices deletes ResourceSlices of the node, optionally just for a specific driver. func (h *RegistrationHandler) wipeResourceSlices(driver string) { if h.kubeClient == nil { return } ctx := h.backgroundCtx logger := klog.FromContext(ctx) </del> <ins> func (p *Plugin) getOrCreateGRPCConn() (*grpc.ClientConn, error) { p.mutex.Lock() defer p.mutex.Unlock() </ins> <del> backoff := wait.Backoff{ Duration: time.Second, Factor:   2, Jitter:   0.2, Cap:      5 * time.Minute, Steps:    100, </del> <ins> if p.conn != nil { return p.conn, nil </ins> } <del> // Error logging is done inside the loop. Context cancellation doesn't get logged. _ = wait.ExponentialBackoffWithContext(ctx, backoff, func(ctx context.Context) (bool, error) { node, err := h.getNode() if apierrors.IsNotFound(err) { return false, nil } if err != nil { logger.Error(err, "Unexpected error checking for node") return false, nil } fieldSelector := fields.Set{resourceapi.ResourceSliceSelectorNodeName: node.Name} if driver != "" { fieldSelector[resourceapi.ResourceSliceSelectorDriver] = driver } err = h.kubeClient.ResourceV1alpha3().ResourceSlices().DeleteCollection(ctx, metav1.DeleteOptions{}, metav1.ListOptions{FieldSelector: fieldSelector.String()}) switch { case err == nil: logger.V(3).Info("Deleted ResourceSlices", "fieldSelector", fieldSelector) return true, nil case apierrors.IsUnauthorized(err): // This can happen while kubelet is still figuring out // its credentials. logger.V(5).Info("Deleting ResourceSlice failed, retrying", "fieldSelector", fieldSelector, "err", err) return false, nil default: // Log and retry for other errors. logger.V(3).Info("Deleting ResourceSlice failed, retrying", "fieldSelector", fieldSelector, "err", err) return false, nil } }) } // RegisterPlugin is called when a plugin can be registered. func (h *RegistrationHandler) RegisterPlugin(pluginName string, endpoint string, versions []string, pluginClientTimeout *time.Duration) error { // Prepare a context with its own logger for the plugin. // // The lifecycle of the plugin's background activities is tied to our // root context, so canceling that will also cancel the plugin. // // The logger injects the plugin name as additional value // into all log output related to the plugin. ctx := h.backgroundCtx </del> <ins> ctx := p.backgroundCtx </ins> logger := klog.FromContext(ctx) <del> logger = klog.LoggerWithValues(logger, "pluginName", pluginName) ctx = klog.NewContext(ctx, logger) logger.V(3).Info("Register new DRA plugin", "endpoint", endpoint) highestSupportedVersion, err := h.validateVersions(pluginName, versions) </del> <ins> network := "unix" logger.V(4).Info("Creating new gRPC connection", "protocol", network, "endpoint", p.endpoint) // grpc.Dial is deprecated. grpc.NewClient should be used instead. // For now this gets ignored because this function is meant to establish // the connection, with the one second timeout below. Perhaps that // approach should be reconsidered? //nolint:staticcheck conn, err := grpc.Dial( p.endpoint, grpc.WithTransportCredentials(insecure.NewCredentials()), grpc.WithContextDialer(func(ctx context.Context, target string) (net.Conn, error) { return (&net.Dialer{}).DialContext(ctx, network, target) }), ) </ins> if err != nil { <del> return fmt.Errorf("version check of plugin %s failed: %w", pluginName, err) } var timeout time.Duration if pluginClientTimeout == nil { timeout = defaultClientCallTimeout } else { timeout = *pluginClientTimeout </del> <ins> return nil, err </ins> } <del> ctx, cancel := context.WithCancelCause(ctx) </del> <ins> ctx, cancel := context.WithTimeout(context.Background(), time.Second) defer cancel() </ins> <del> pluginInstance := &Plugin{ backgroundCtx:           ctx, cancel:                  cancel, conn:                    nil, endpoint:                endpoint, highestSupportedVersion: highestSupportedVersion, clientCallTimeout:       timeout, </del> <ins> if ok := conn.WaitForStateChange(ctx, connectivity.Connecting); !ok { return nil, errors.New("timed out waiting for gRPC connection to be ready") </ins> } <del> // Storing endpoint of newly registered DRA Plugin into the map, where plugin name will be the key // all other DRA components will be able to get the actual socket of DRA plugins by its name. if draPlugins.add(pluginName, pluginInstance) { logger.V(1).Info("Already registered, previous plugin was replaced") } return nil </del> <ins> p.conn = conn return p.conn, nil </ins> } <del> func (h *RegistrationHandler) validateVersions( pluginName string, versions []string, ) (*utilversion.Version, error) { if len(versions) == 0 { return nil, errors.New("empty list for supported versions") } </del> <ins> func (p *Plugin) NodePrepareResources( ctx context.Context, req *drapb.NodePrepareResourcesRequest, opts ...grpc.CallOption, ) (*drapb.NodePrepareResourcesResponse, error) { logger := klog.FromContext(ctx) logger.V(4).Info("Calling NodePrepareResources rpc", "request", req) </ins> <del> // Validate version newPluginHighestVersion, err := utilversion.HighestSupportedVersion(versions) </del> <ins> conn, err := p.getOrCreateGRPCConn() </ins> if err != nil { <del> // HighestSupportedVersion includes the list of versions in its error // if relevant, no need to repeat it here. return nil, fmt.Errorf("none of the versions are supported: %w", err) } existingPlugin := draPlugins.get(pluginName) if existingPlugin == nil { return newPluginHighestVersion, nil </del> <ins> return nil, err </ins> } <del> if existingPlugin.highestSupportedVersion.LessThan(newPluginHighestVersion) { return newPluginHighestVersion, nil } return nil, fmt.Errorf("another plugin instance is already registered with a higher supported version: %q < %q", newPluginHighestVersion, existingPlugin.highestSupportedVersion) } // DeRegisterPlugin is called when a plugin has removed its socket, // signaling it is no longer available. func (h *RegistrationHandler) DeRegisterPlugin(pluginName string) { if p := draPlugins.delete(pluginName); p != nil { logger := klog.FromContext(p.backgroundCtx) logger.V(3).Info("Deregister DRA plugin", "endpoint", p.endpoint) // Clean up the ResourceSlices for the deleted Plugin since it // may have died without doing so itself and might never come // back. go h.wipeResourceSlices(pluginName) return } </del> <ins> ctx, cancel := context.WithTimeout(ctx, p.clientCallTimeout) defer cancel() </ins> <del> logger := klog.FromContext(h.backgroundCtx) logger.V(3).Info("Deregister DRA plugin not necessary, was already removed") </del> <ins> nodeClient := drapb.NewNodeClient(conn) response, err := nodeClient.NodePrepareResources(ctx, req) logger.V(4).Info("Done calling NodePrepareResources rpc", "response", response, "err", err) return response, err </ins> } <del> // ValidatePlugin is called by kubelet's plugin watcher upon detection // of a new registration socket opened by DRA plugin. func (h *RegistrationHandler) ValidatePlugin(pluginName string, endpoint string, versions []string) error { _, err := h.validateVersions(pluginName, versions) </del> <ins> func (p *Plugin) NodeUnprepareResources( ctx context.Context, req *drapb.NodeUnprepareResourcesRequest, opts ...grpc.CallOption, ) (*drapb.NodeUnprepareResourcesResponse, error) { logger := klog.FromContext(ctx) logger.V(4).Info("Calling NodeUnprepareResource rpc", "request", req) conn, err := p.getOrCreateGRPCConn() </ins> if err != nil { <del> return fmt.Errorf("invalid versions of plugin %s: %w", pluginName, err) </del> <ins> return nil, err </ins> } <del> return err </del> <ins> ctx, cancel := context.WithTimeout(ctx, p.clientCallTimeout) defer cancel() nodeClient := drapb.NewNodeClient(conn) response, err := nodeClient.NodeUnprepareResources(ctx, req) logger.V(4).Info("Done calling NodeUnprepareResources rpc", "response", response, "err", err) return response, err </ins> }