Question:
The following log line is often (~30times) logged during our tests in : . This suggests a possible bug as it was assumed to do not happen. No log line saying . Look at artifacts of all (or almost all) builds of: No response <details</details<detailsReproducible in tests </details<details</details<details</details<details</details<details</details
/sig node FYI
/triage accepted since it's a new error, it is a regression. So adding a corresponding priority /priority important-soon
are you working on this? Should I assign bug to you?
Yes, I'm taking a look at this. /assign
/cc
This is a metrics only error (we don’t report a fraction of metrics) and has no functional impact.
That makes sense, but is it expected that this is a valid state in first place? . My assumption was that if static is false, (i.e. API pod) then we should have a config?
I took a closer look at the logs, it seems that this is happening when we enter while a static pod is being synced which has UID reuse and another pod which has the same full name did not terminate yet. In this case, we block the static pod from starting i.e. () returns false: Because the pod can't start, we don't set : which means that when we log metrics there is no set and thus or are both defaulted to false:
Based on the above, it looks like this case is expected. As a result, maybe the simplest option is to simply drop the log statement since by design we only want to into account a subset of the states for the metrics?

}, }, { <ins> name:    "pod that could not start still has a pending update and is tracked in metrics", wantErr: false, pods: []*v1.Pod{ staticPod(), }, prepareWorker: func(t *testing.T, w *podWorkers, records map[types.UID][]syncPodRecord) { // send a create of a static pod pod := staticPod() // block startup of the static pod due to full name collision w.startedStaticPodsByFullname[kubecontainer.GetPodFullName(pod)] = types.UID("2") w.UpdatePod(UpdatePodOptions{ UpdateType: kubetypes.SyncPodCreate, StartTime:  time.Unix(1, 0).UTC(), Pod:        pod, }) drainAllWorkers(w) if _, ok := records[pod.UID]; ok { t.Fatalf("unexpected records: %#v", records) } // pod worker is unaware of pod1 yet }, wantWorker: func(t *testing.T, w *podWorkers, records map[types.UID][]syncPodRecord) { uid := types.UID("1") if len(w.podSyncStatuses) != 1 { t.Fatalf("unexpected sync statuses: %#v", w.podSyncStatuses) } s, ok := w.podSyncStatuses[uid] if !ok || s.IsTerminationRequested() || s.IsTerminationStarted() || s.IsFinished() || s.IsWorking() || s.IsDeleted() || s.restartRequested || s.activeUpdate != nil || s.pendingUpdate == nil { t.Errorf("unexpected requested pod termination: %#v", s) } // expect that no sync calls are made, since the pod doesn't ever start if actual, expected := records[uid], []syncPodRecord(nil); !reflect.DeepEqual(expected, actual) { t.Fatalf("unexpected pod sync records: %s", cmp.Diff(expected, actual, cmp.AllowUnexported(syncPodRecord{}))) } }, expectMetrics: map[string]string{ metrics.DesiredPodCount.FQName(): `# HELP kubelet_desired_pods [ALPHA] The number of pods the kubelet is being instructed to run. static is true if the pod is not from the apiserver. # TYPE kubelet_desired_pods gauge kubelet_desired_pods{static=""} 0 kubelet_desired_pods{static="true"} 1 `, metrics.WorkingPodCount.FQName(): `# HELP kubelet_working_pods [ALPHA] Number of pods the kubelet is actually running, broken down by lifecycle phase, whether the pod is desired, orphaned, or runtime only (also orphaned), and whether the pod is static. An orphaned pod has been removed from local configuration or force deleted in the API and consumes resources that are not otherwise visible. # TYPE kubelet_working_pods gauge kubelet_working_pods{config="desired",lifecycle="sync",static=""} 0 kubelet_working_pods{config="desired",lifecycle="sync",static="true"} 1 kubelet_working_pods{config="desired",lifecycle="terminated",static=""} 0 kubelet_working_pods{config="desired",lifecycle="terminated",static="true"} 0 kubelet_working_pods{config="desired",lifecycle="terminating",static=""} 0 kubelet_working_pods{config="desired",lifecycle="terminating",static="true"} 0 kubelet_working_pods{config="orphan",lifecycle="sync",static=""} 0 kubelet_working_pods{config="orphan",lifecycle="sync",static="true"} 0 kubelet_working_pods{config="orphan",lifecycle="terminated",static=""} 0 kubelet_working_pods{config="orphan",lifecycle="terminated",static="true"} 0 kubelet_working_pods{config="orphan",lifecycle="terminating",static=""} 0 kubelet_working_pods{config="orphan",lifecycle="terminating",static="true"} 0 kubelet_working_pods{config="runtime_only",lifecycle="sync",static="unknown"} 0 kubelet_working_pods{config="runtime_only",lifecycle="terminated",static="unknown"} 0 kubelet_working_pods{config="runtime_only",lifecycle="terminating",static="unknown"} 0 `, }, }, { </ins> name:           "pod that could not start and is not in config is force terminated without runtime during pod cleanup", wantErr:        false, terminatingErr: errors.New("unable to terminate"),


Question:
The following log line is often (~30times) logged during our tests in : . This suggests a possible bug as it was assumed to do not happen. No log line saying . Look at artifacts of all (or almost all) builds of: No response <details</details<detailsReproducible in tests </details<details</details<details</details<details</details<details</details
/sig node FYI
/triage accepted since it's a new error, it is a regression. So adding a corresponding priority /priority important-soon
are you working on this? Should I assign bug to you?
Yes, I'm taking a look at this. /assign
/cc
This is a metrics only error (we don’t report a fraction of metrics) and has no functional impact.
That makes sense, but is it expected that this is a valid state in first place? . My assumption was that if static is false, (i.e. API pod) then we should have a config?
I took a closer look at the logs, it seems that this is happening when we enter while a static pod is being synced which has UID reuse and another pod which has the same full name did not terminate yet. In this case, we block the static pod from starting i.e. () returns false: Because the pod can't start, we don't set : which means that when we log metrics there is no set and thus or are both defaulted to false:
Based on the above, it looks like this case is expected. As a result, maybe the simplest option is to simply drop the log statement since by design we only want to into account a subset of the states for the metrics?

// or can be started, and updates the cached pod state so that downstream components can observe what the // pod worker goroutine is currently attempting to do. If ok is false, there is no available event. If any // of the boolean values is false, ensure the appropriate cleanup happens before returning. <ins> // // This method should ensure that either status.pendingUpdate is cleared and merged into status.activeUpdate, // or when a pod cannot be started status.pendingUpdate remains the same. Pods that have not been started // should never have an activeUpdate because that is exposed to downstream components on started pods. </ins> func (p *podWorkers) startPodSync(podUID types.UID) (ctx context.Context, update podWork, canStart, canEverStart, ok bool) { p.podLock.Lock() defer p.podLock.Unlock()


Question:
The following log line is often (~30times) logged during our tests in : . This suggests a possible bug as it was assumed to do not happen. No log line saying . Look at artifacts of all (or almost all) builds of: No response <details</details<detailsReproducible in tests </details<details</details<details</details<details</details<details</details
/sig node FYI
/triage accepted since it's a new error, it is a regression. So adding a corresponding priority /priority important-soon
are you working on this? Should I assign bug to you?
Yes, I'm taking a look at this. /assign
/cc
This is a metrics only error (we don’t report a fraction of metrics) and has no functional impact.
That makes sense, but is it expected that this is a valid state in first place? . My assumption was that if static is false, (i.e. API pod) then we should have a config?
I took a closer look at the logs, it seems that this is happening when we enter while a static pod is being synced which has UID reuse and another pod which has the same full name did not terminate yet. In this case, we block the static pod from starting i.e. () returns false: Because the pod can't start, we don't set : which means that when we log metrics there is no set and thus or are both defaulted to false:
Based on the above, it looks like this case is expected. As a result, maybe the simplest option is to simply drop the log statement since by design we only want to into account a subset of the states for the metrics?

klog.V(4).InfoS("Pod cannot start ever", "pod", klog.KObj(update.Options.Pod), "podUID", podUID, "updateType", update.WorkType) return ctx, update, canStart, canEverStart, true case !canStart: <ins> // this is the only path we don't start the pod, so we need to put the change back in pendingUpdate status.pendingUpdate = &update.Options </ins> status.working = false klog.V(4).InfoS("Pod cannot start yet", "pod", klog.KObj(update.Options.Pod), "podUID", podUID) return ctx, update, canStart, canEverStart, true


Question:
The following log line is often (~30times) logged during our tests in : . This suggests a possible bug as it was assumed to do not happen. No log line saying . Look at artifacts of all (or almost all) builds of: No response <details</details<detailsReproducible in tests </details<details</details<details</details<details</details<details</details
/sig node FYI
/triage accepted since it's a new error, it is a regression. So adding a corresponding priority /priority important-soon
are you working on this? Should I assign bug to you?
Yes, I'm taking a look at this. /assign
/cc
This is a metrics only error (we don’t report a fraction of metrics) and has no functional impact.
That makes sense, but is it expected that this is a valid state in first place? . My assumption was that if static is false, (i.e. API pod) then we should have a config?
I took a closer look at the logs, it seems that this is happening when we enter while a static pod is being synced which has UID reuse and another pod which has the same full name did not terminate yet. In this case, we block the static pod from starting i.e. () returns false: Because the pod can't start, we don't set : which means that when we log metrics there is no set and thus or are both defaulted to false:
Based on the above, it looks like this case is expected. As a result, maybe the simplest option is to simply drop the log statement since by design we only want to into account a subset of the states for the metrics?

State:  status.WorkType(), Orphan: orphan, } <del> if status.activeUpdate != nil && status.activeUpdate.Pod != nil { sync.HasConfig = true sync.Static = kubetypes.IsStaticPod(status.activeUpdate.Pod) </del> <ins> switch { case status.activeUpdate != nil: if status.activeUpdate.Pod != nil { sync.HasConfig = true sync.Static = kubetypes.IsStaticPod(status.activeUpdate.Pod) } case status.pendingUpdate != nil: if status.pendingUpdate.Pod != nil { sync.HasConfig = true sync.Static = kubetypes.IsStaticPod(status.pendingUpdate.Pod) } </ins> } workers[uid] = sync }


Question:
If a pod is created and then deleted an entry in processedPods map remains forever (sometimes, it's racy). The problem comes from the fact that calls volume_manager asynchronously. If the last call happens after populator cleaned up resources, an entry is to processedPods and will never be seen. An entry in processedPods map must be deleted if pod doesn't exist and pod runtime should removed according to . To verify that the issue really exists I applaied a simple printing resources in map. Here is a reproducer: brand-new fresh k8s cluster with entries in : a pod, check its uid: that this uid is accurately reported: the pod, wait for a few mins, check for again: The entry will be kept forever because kubelet doesn't know anything about this pod anymore: so no more events will ever come to populater for this uid, and the loop in won't find it because it was already deleted from . n/a <details</details<detailsany </details<details</details<details</details<details</details<details</details
/sig storage
/triage accepted
/priority impoerant-soon
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /priority impoerant-soon Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/priority important-soon
/assign
PR () fixed this issue. It is ready to merge since Mar 21.
/assign

// Iterate through all pods in desired state of world, and remove if they no // longer exist func (dswp *desiredStateOfWorldPopulator) findAndRemoveDeletedPods() { <ins> podsFromCache := make(map[volumetypes.UniquePodName]struct{}) </ins> for _, volumeToMount := range dswp.desiredStateOfWorld.GetVolumesToMount() { <ins> podsFromCache[volumetypes.UniquePodName(volumeToMount.Pod.UID)] = struct{}{} </ins> pod, podExists := dswp.podManager.GetPodByUID(volumeToMount.Pod.UID) if podExists {


Question:
If a pod is created and then deleted an entry in processedPods map remains forever (sometimes, it's racy). The problem comes from the fact that calls volume_manager asynchronously. If the last call happens after populator cleaned up resources, an entry is to processedPods and will never be seen. An entry in processedPods map must be deleted if pod doesn't exist and pod runtime should removed according to . To verify that the issue really exists I applaied a simple printing resources in map. Here is a reproducer: brand-new fresh k8s cluster with entries in : a pod, check its uid: that this uid is accurately reported: the pod, wait for a few mins, check for again: The entry will be kept forever because kubelet doesn't know anything about this pod anymore: so no more events will ever come to populater for this uid, and the loop in won't find it because it was already deleted from . n/a <details</details<detailsany </details<details</details<details</details<details</details<details</details
/sig storage
/triage accepted
/priority impoerant-soon
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /priority impoerant-soon Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/priority important-soon
/assign
PR () fixed this issue. It is ready to merge since Mar 21.
/assign

dswp.deleteProcessedPod(volumeToMount.PodName) } <ins> // Cleanup orphanded entries from processedPods dswp.pods.Lock() orphanedPods := make([]volumetypes.UniquePodName, 0, len(dswp.pods.processedPods)) for k := range dswp.pods.processedPods { if _, ok := podsFromCache[k]; !ok { orphanedPods = append(orphanedPods, k) } } dswp.pods.Unlock() for _, orphanedPod := range orphanedPods { uid := types.UID(orphanedPod) _, podExists := dswp.podManager.GetPodByUID(uid) if !podExists && dswp.podStateProvider.ShouldPodRuntimeBeRemoved(uid) { dswp.deleteProcessedPod(orphanedPod) } } </ins> podsWithError := dswp.desiredStateOfWorld.GetPodsWithErrors() for _, podName := range podsWithError { if _, podExists := dswp.podManager.GetPodByUID(types.UID(podName)); !podExists {


Question:
If a pod is created and then deleted an entry in processedPods map remains forever (sometimes, it's racy). The problem comes from the fact that calls volume_manager asynchronously. If the last call happens after populator cleaned up resources, an entry is to processedPods and will never be seen. An entry in processedPods map must be deleted if pod doesn't exist and pod runtime should removed according to . To verify that the issue really exists I applaied a simple printing resources in map. Here is a reproducer: brand-new fresh k8s cluster with entries in : a pod, check its uid: that this uid is accurately reported: the pod, wait for a few mins, check for again: The entry will be kept forever because kubelet doesn't know anything about this pod anymore: so no more events will ever come to populater for this uid, and the loop in won't find it because it was already deleted from . n/a <details</details<detailsany </details<details</details<details</details<details</details<details</details
/sig storage
/triage accepted
/priority impoerant-soon
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /priority impoerant-soon Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/priority important-soon
/assign
PR () fixed this issue. It is ready to merge since Mar 21.
/assign

t.Fatalf("Failed to remove pods from desired state of world since they no longer exist") } <ins> // podWorker may call volume_manager WaitForUnmount() after we processed the pod in findAndRemoveDeletedPods() dswp.ReprocessPod(podName) dswp.findAndRemoveDeletedPods() // findAndRemoveDeletedPods() above must detect orphaned pod and delete it from the map if _, ok := dswp.pods.processedPods[podName]; ok { t.Fatalf("Failed to remove orphanded pods from internal map") } </ins> volumeExists := dswp.desiredStateOfWorld.VolumeExists(expectedVolumeName, "" /* SELinuxContext */) if volumeExists { t.Fatalf(


Question:
Not , but removed: Looking at this comment: this should be removed once the migration to lease in leader-election was done. Can this be removed already? I mean, the following 2 lines: Security folks internally were asking if we could get rid of this upstream if it is not needed anymore (if this "migration to lease in leader-election" was fully done)
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig sig-scheduling
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /sig sig-scheduling Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scheduling
/cc
/assign
Support for endpoints and configmap based leader election has been removed with this PR: Validation to error out leader elections configuration for resource lock type other than "lease" has been implemented in With this, it looks safe to remove the two lines mentioned in this issue.

// TODO: scope this to the kube-system namespace rbacv1helpers.NewRule("create").Groups(coordinationGroup).Resources("leases").RuleOrDie(), rbacv1helpers.NewRule("get", "update").Groups(coordinationGroup).Resources("leases").Names("kube-scheduler").RuleOrDie(), <del> // TODO: Remove once we fully migrate to lease in leader-election. rbacv1helpers.NewRule("create").Groups(legacyGroup).Resources("endpoints").RuleOrDie(), rbacv1helpers.NewRule("get", "update").Groups(legacyGroup).Resources("endpoints").Names("kube-scheduler").RuleOrDie(), </del> // Fundamental resources rbacv1helpers.NewRule(Read...).Groups(legacyGroup).Resources("nodes").RuleOrDie(),


Question:
Not , but removed: Looking at this comment: this should be removed once the migration to lease in leader-election was done. Can this be removed already? I mean, the following 2 lines: Security folks internally were asking if we could get rid of this upstream if it is not needed anymore (if this "migration to lease in leader-election" was fully done)
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig sig-scheduling
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /sig sig-scheduling Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scheduling
/cc
/assign
Support for endpoints and configmap based leader election has been removed with this PR: Validation to error out leader elections configuration for resource lock type other than "lease" has been implemented in With this, it looks safe to remove the two lines mentioned in this issue.

- apiGroups: - "" resources: <del> - endpoints verbs: - create - apiGroups: - "" resourceNames: - kube-scheduler resources: - endpoints verbs: - get - update - apiGroups: - "" resources: </del> - nodes verbs: - get


Question:
Followup feedback from In a few e2e node tests (e.g. tests of deletion grace period) we have some busybox pods that wait for SIGTERM and perform some action like a small sleep to simulate SIGTERM taking some time. These tests follow the pattern of -- see The will block the SIGTERM handler from being run until after the time is elapsed. We should consider to remove the and instead use to avoid blocking the SIGTERM handler, see pattern followed in n/a n/a No response n/a n/a <details</details<details</details<details</details<details</details
/sig node /cc
/assign
/triage accepted

Image:   imageutils.GetE2EImage(imageutils.BusyBox), Command: []string{"sh", "-c"}, Args: []string{` <ins> sleep 9999999 & PID=$! </ins> _term() { <ins> kill $PID </ins> echo "Caught SIGTERM signal!" <del> exit 0 </del> } trap _term SIGTERM <del> while true; do sleep 5; done </del> <ins> wait $PID exit 0 </ins> `, }, },


Question:
Followup feedback from In a few e2e node tests (e.g. tests of deletion grace period) we have some busybox pods that wait for SIGTERM and perform some action like a small sleep to simulate SIGTERM taking some time. These tests follow the pattern of -- see The will block the SIGTERM handler from being run until after the time is elapsed. We should consider to remove the and instead use to avoid blocking the SIGTERM handler, see pattern followed in n/a n/a No response n/a n/a <details</details<details</details<details</details<details</details
/sig node /cc
/assign
/triage accepted

} return priority } <ins> // getGracePeriodOverrideTestPod returns a new Pod object containing a container // runs a shell script, hangs the process until a SIGTERM signal is received. // The script waits for $PID to ensure that the process does not exist. // If priorityClassName is scheduling.SystemNodeCritical, the Pod is marked as critical and a comment is added. </ins> func getGracePeriodOverrideTestPod(name string, node string, gracePeriod int64, priorityClassName string) *v1.Pod { pod := &v1.Pod{ TypeMeta: metav1.TypeMeta{


Question:
Followup feedback from In a few e2e node tests (e.g. tests of deletion grace period) we have some busybox pods that wait for SIGTERM and perform some action like a small sleep to simulate SIGTERM taking some time. These tests follow the pattern of -- see The will block the SIGTERM handler from being run until after the time is elapsed. We should consider to remove the and instead use to avoid blocking the SIGTERM handler, see pattern followed in n/a n/a No response n/a n/a <details</details<details</details<details</details<details</details
/sig node /cc
/assign
/triage accepted

Image:   busyboxImage, Command: []string{"sh", "-c"}, Args: []string{` <del> _term() { echo "Caught SIGTERM signal!" while true; do sleep 5; done } trap _term SIGTERM while true; do sleep 5; done `}, </del> <ins> sleep 9999999 & PID=$! _term() { echo "Caught SIGTERM signal!" wait $PID } trap _term SIGTERM wait $PID `}, </ins> }, }, TerminationGracePeriodSeconds: &gracePeriod,


Question:
No response No response /sig windows /kind failing-test
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/sig windows
i think this problem can be fixed when it imports os/net in Am i right?
I also found some other unit test bugs in CI, such as running linux-related tests on windows. I will try to fix them. /assign
/cc
Thanks for opening the issue! I didn't notice the failure through the testgrid dashboard. I've sent a PR here: . While fixing this issue, I noticed there's one unit test that was failing, so that PR also addresses that as well.
Yep, the imports you mentioned are needed. Keep in mind that imports are in alphabetical order. But in addition to that, there was another failing unit test which was masked by this import errors.

"fmt" "net" "net/url" <ins> "os" </ins> "path/filepath" "strings" "syscall"


Question:
No response No response /sig windows /kind failing-test
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/sig windows
i think this problem can be fixed when it imports os/net in Am i right?
I also found some other unit test bugs in CI, such as running linux-related tests on windows. I will try to fix them. /assign
/cc
Thanks for opening the issue! I didn't notice the failure through the testgrid dashboard. I've sent a PR here: . While fixing this issue, I noticed there's one unit test that was failing, so that PR also addresses that as well.
Yep, the imports you mentioned are needed. Keep in mind that imports are in alphabetical order. But in addition to that, there was another failing unit test which was masked by this import errors.

// does NOT work in 1809 if the socket file is created within a bind mounted directory by a container // and the FSCTL is issued in the host by the kubelet. <ins> // If the file does not exist, it cannot be a Unix domain socket. if _, err := os.Stat(filePath); os.IsNotExist(err) { return false, fmt.Errorf("File %s not found. Err: %v", filePath, err) } </ins> klog.V(6).InfoS("Function IsUnixDomainSocket starts", "filePath", filePath) // As detailed in https://github.com/kubernetes/kubernetes/issues/104584 we cannot rely // on the Unix Domain socket working on the very first try, hence the potential need to


Question:
No response No response /sig windows /kind failing-test
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/sig windows
i think this problem can be fixed when it imports os/net in Am i right?
I also found some other unit test bugs in CI, such as running linux-related tests on windows. I will try to fix them. /assign
/cc
Thanks for opening the issue! I didn't notice the failure through the testgrid dashboard. I've sent a PR here: . While fixing this issue, I noticed there's one unit test that was failing, so that PR also addresses that as well.
Yep, the imports you mentioned are needed. Keep in mind that imports are in alphabetical order. But in addition to that, there was another failing unit test which was masked by this import errors.

import ( "fmt" "math/rand" <ins> "net" "os" </ins> "reflect" "runtime" "sync"


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Following the example here: Everything here works when you use , but when using 1.27.1, I see the following panic: What you expected to happen: When downgrading, this works as expected: How to reproduce it (as minimally and precisely as possible): <!-- Please make sure you are able to reproduce the bug using a supported version and version skew of Kubernetes. See for which versions and are currently supported. --You will need a Kubernetes cluster with Agones installed (I'm testing on 1.25.x, as you can see above), then follow this quickstart: Anything else we need to know?: The is implemented as an API Extension, not a CRD -- which is probably where the issue lies. Looking at the line: There seems to be an assuming that there will be a under and , in - there should be some nil checks in there. Environment: Kubernetes client and server versions (use ): See above Cloud provider or hardware configuration: GKE OS (e.g: ): Linux
seems like an another edge case; Due to the server version is old and openapiv3 is returning empty?
Thanks for opening this issue /triage accepted
I'm digging into the Agones code, but I'm 99.4% sure we don't even implement openapiv3 at all for our api endpoint
Yep, confirmed -- I see a bunch of 404's in my API Extension logs for requests for
/transfer kubernetes /kind regression
/sig cli /assign
I can also work on tomorrow if Sean can not get time to look at it
If nothing else, this does also force me to publish an openapi spec for our API extension
not sure if needed to handle nils inside the schema better, or if didn't fall back to v2 correctly in the absence of v3 data
It appears to be the former, where an OpenAPI V3 document is returned without , while the code assumes this field is non-nil.
seems like that should still fall back to v2
Can we get a kubectl -v8 to see what actually happens with downloading the OpenAPI?
As requested!
Thanks for the quick response. It looks like the OpenAPI V3 document is almost completely empty The code is expecting an Open API document with . I will have a fix out shortly.
I was digging through that - I found that really interesting. We only expose an basically empty spec for the api extension. It's always "just worked", so never needed to update it. I am curious where K8s is even getting the something that it is outputting for , but that's a side note.

// the PATCH end-point. Returns true if the query param is supported by the // spec for the passed GVK; false otherwise. func supportsQueryParamV3(doc *spec3.OpenAPI, gvk schema.GroupVersionKind, queryParam VerifiableQueryParam) bool { <ins> if doc == nil || doc.Paths == nil { return false } </ins> for _, path := range doc.Paths.Paths { // If operation is not PATCH, then continue. op := path.PathProps.Patch


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Following the example here: Everything here works when you use , but when using 1.27.1, I see the following panic: What you expected to happen: When downgrading, this works as expected: How to reproduce it (as minimally and precisely as possible): <!-- Please make sure you are able to reproduce the bug using a supported version and version skew of Kubernetes. See for which versions and are currently supported. --You will need a Kubernetes cluster with Agones installed (I'm testing on 1.25.x, as you can see above), then follow this quickstart: Anything else we need to know?: The is implemented as an API Extension, not a CRD -- which is probably where the issue lies. Looking at the line: There seems to be an assuming that there will be a under and , in - there should be some nil checks in there. Environment: Kubernetes client and server versions (use ): See above Cloud provider or hardware configuration: GKE OS (e.g: ): Linux
seems like an another edge case; Due to the server version is old and openapiv3 is returning empty?
Thanks for opening this issue /triage accepted
I'm digging into the Agones code, but I'm 99.4% sure we don't even implement openapiv3 at all for our api endpoint
Yep, confirmed -- I see a bunch of 404's in my API Extension logs for requests for
/transfer kubernetes /kind regression
/sig cli /assign
I can also work on tomorrow if Sean can not get time to look at it
If nothing else, this does also force me to publish an openapi spec for our API extension
not sure if needed to handle nils inside the schema better, or if didn't fall back to v2 correctly in the absence of v3 data
It appears to be the former, where an OpenAPI V3 document is returned without , while the code assumes this field is non-nil.
seems like that should still fall back to v2
Can we get a kubectl -v8 to see what actually happens with downloading the OpenAPI?
As requested!
Thanks for the quick response. It looks like the OpenAPI V3 document is almost completely empty The code is expecting an Open API document with . I will have a fix out shortly.
I was digging through that - I found that really interesting. We only expose an basically empty spec for the api extension. It's always "just worked", so never needed to update it. I am curious where K8s is even getting the something that it is outputting for , but that's a side note.

"k8s.io/client-go/openapi/cached" "k8s.io/client-go/openapi/openapitest" "k8s.io/client-go/openapi3" <ins> "k8s.io/kube-openapi/pkg/spec3" </ins> ) func TestV3SupportsQueryParamBatchV1(t *testing.T) {


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Following the example here: Everything here works when you use , but when using 1.27.1, I see the following panic: What you expected to happen: When downgrading, this works as expected: How to reproduce it (as minimally and precisely as possible): <!-- Please make sure you are able to reproduce the bug using a supported version and version skew of Kubernetes. See for which versions and are currently supported. --You will need a Kubernetes cluster with Agones installed (I'm testing on 1.25.x, as you can see above), then follow this quickstart: Anything else we need to know?: The is implemented as an API Extension, not a CRD -- which is probably where the issue lies. Looking at the line: There seems to be an assuming that there will be a under and , in - there should be some nil checks in there. Environment: Kubernetes client and server versions (use ): See above Cloud provider or hardware configuration: GKE OS (e.g: ): Linux
seems like an another edge case; Due to the server version is old and openapiv3 is returning empty?
Thanks for opening this issue /triage accepted
I'm digging into the Agones code, but I'm 99.4% sure we don't even implement openapiv3 at all for our api endpoint
Yep, confirmed -- I see a bunch of 404's in my API Extension logs for requests for
/transfer kubernetes /kind regression
/sig cli /assign
I can also work on tomorrow if Sean can not get time to look at it
If nothing else, this does also force me to publish an openapi spec for our API extension
not sure if needed to handle nils inside the schema better, or if didn't fall back to v2 correctly in the absence of v3 data
It appears to be the former, where an OpenAPI V3 document is returned without , while the code assumes this field is non-nil.
seems like that should still fall back to v2
Can we get a kubectl -v8 to see what actually happens with downloading the OpenAPI?
As requested!
Thanks for the quick response. It looks like the OpenAPI V3 document is almost completely empty The code is expecting an Open API document with . I will have a fix out shortly.
I was digging through that - I found that really interesting. We only expose an basically empty spec for the api extension. It's always "just worked", so never needed to update it. I am curious where K8s is even getting the something that it is outputting for , but that's a side note.

}) } } <ins> func TestInvalidOpenAPIV3Document(t *testing.T) { tests := map[string]struct { spec *spec3.OpenAPI }{ "nil document correctly returns Unsupported error": { spec: nil, }, "empty document correctly returns Unsupported error": { spec: &spec3.OpenAPI{}, }, "minimal document correctly returns Unsupported error": { spec: &spec3.OpenAPI{ Version: "openapi 3.0.0", Paths:   nil, }, }, "document with empty Paths correctly returns Unsupported error": { spec: &spec3.OpenAPI{ Version: "openapi 3.0.0", Paths:   &spec3.Paths{}, }, }, } gvk := schema.GroupVersionKind{ Group:   "batch", Version: "v1", Kind:    "Job", } for tn, tc := range tests { t.Run(tn, func(t *testing.T) { verifier := &queryParamVerifierV3{ finder: NewCRDFinder(func() ([]schema.GroupKind, error) { return []schema.GroupKind{}, nil }), root:       &fakeRoot{tc.spec}, queryParam: QueryParamFieldValidation, } err := verifier.HasSupport(gvk) if err == nil { t.Errorf("Expected not supports error, but none received.") } }) } } // fakeRoot implements Root interface; manually specifies the returned OpenAPI V3 document. type fakeRoot struct { spec *spec3.OpenAPI } func (f *fakeRoot) GroupVersions() ([]schema.GroupVersion, error) { // Unused return nil, nil } // GVSpec returns hard-coded OpenAPI V3 document. func (f *fakeRoot) GVSpec(gv schema.GroupVersion) (*spec3.OpenAPI, error) { return f.spec, nil } func (f *fakeRoot) GVSpecAsMap(gv schema.GroupVersion) (map[string]interface{}, error) { // Unused return nil, nil } </ins>


Question:
is recording the time that a Pod is gated. We use the timestamp when the scheduler inserts the pod into the queue: The period of time when a Pod fails PreEnque (like being gated) shouldn't be accounted in the podschedulingdurationseconds. Create a Pod with scheduling gates. Wait some time before removing the gate. Observe the metric No response <details</details<detailsAny </details<details</details<details</details<details</details<details</details
/sig scheduling
/help /triage accepted
This request has been marked as needing help from a contributor. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help /triage accepted Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
would you want to take a stab on this issue?
Yup, I can give it a shot. /assign helayoty
, as we discussed, we can address this issue with different solutions: the pod timestamp to the current time , so the gated time is substituted. timestamp . Use a pointer to the InitialAttemptTimestamp and interpret as the newest timestamp so the pod will go to the back of the queue. I'll go ahead and implement solution no. 3, wdyt?
sgtm. Any concerns ?
I think it should be option 1 + 3, and pInfo.Timestamp should be pInfo.InitialAttemptTimestamp, right? as records the time based on :
I think we should revisit this issue, in terms of its assumption: So is it reasonable to always exclude time spent in PreEnqueue from ? Note that PreEnqueue != SchedulingGates, is one of the PreEnqueue's usecases. In other words, some user of PreEnqueue may want to include the time spent in PreEnqueue.
is a signal that cluster providers can use to build SLOs. The time spent before passing PreEnqueue is not under control of kube-scheduler, so considering it would not allow to build an SLO. Are you aware of other users of PreEnqueue?
True, but comparing to state that metric X can or cannot be used to build SLO (which is hard IMO), I'm in favor of giving suggestion on SLI/SLO's semantic definition, for example: the official SLI definition of has a footnote clearly explaining its definition: So if a pod is gated, it's not "immediately schedulable", and hence doesn't fall into the definition of podstartup_latency.
A viable solution might be providing an extra label like to explicitly exclude the time spent on preenqueue.
That is true, but there is no way today to exclude those pods from the metric, or separate the time. Maybe we need an extra metric, but I think the new metric could be the one that includes the time guarded in PreEnqueue. It should probably be part of the KEP. The old metric is already stable, so we can't add labels. And it might already be in use for SLOs.
cc
I think we should be working on that. And treat it as part of GA requirement for PodSchedulingReadiness.
An edge(maybe not edge) case is a pod's scheduling attempts is 0 but the is a long time, so prefer not to account for the waiting time when gated. For users not using PreEnqueue, they will not be impacted, but for users using PreEnqueue, it's a break change for it increases the duration time. And we can add a new metric like to measure how long a pod stays in a scheduler.
I thought of "attempts", it's just I'm not sure it's a good idea to rely on "attempts" to define a SLI/SLO, b/c "attempts" is obtained from the in-memory object and in a HA config env, it's only updated by the primary scheduler instance. In other words, the value is inaccurate when the lease owner switches.
attempt=0 is good enough for most scenarios
do we have a conclusion? Should we move on with the solution or remove this issue and the linked PR from the current release?
I would like to move forward with this proposal. ?
To move it forward, I think we should add new metric(s) to explicitly tell it's including or excluding the time spent on PreEnqueue. Given is in stable level: We will need a new metric like "podschedulingexcludingpreenqueuedurationseconds".
28 Bug Triage Shadow here. Just checking-in, if this is still on track for the 1.28 release?
We don't have a conclusion yet, but as a bug, it should qualify for changes after the freezes.

// back to the queue multiple times before it's successfully scheduled. // It shouldn't be updated once initialized. It's used to record the e2e scheduling // latency for a pod. <del> InitialAttemptTimestamp time.Time </del> <ins> InitialAttemptTimestamp *time.Time </ins> // If a Pod failed in a scheduling cycle, record the plugin names it failed by. UnschedulablePlugins sets.Set[string] // Whether the Pod is scheduling gated (by PreEnqueuePlugins) or not.


Question:
is recording the time that a Pod is gated. We use the timestamp when the scheduler inserts the pod into the queue: The period of time when a Pod fails PreEnque (like being gated) shouldn't be accounted in the podschedulingdurationseconds. Create a Pod with scheduling gates. Wait some time before removing the gate. Observe the metric No response <details</details<detailsAny </details<details</details<details</details<details</details<details</details
/sig scheduling
/help /triage accepted
This request has been marked as needing help from a contributor. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help /triage accepted Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
would you want to take a stab on this issue?
Yup, I can give it a shot. /assign helayoty
, as we discussed, we can address this issue with different solutions: the pod timestamp to the current time , so the gated time is substituted. timestamp . Use a pointer to the InitialAttemptTimestamp and interpret as the newest timestamp so the pod will go to the back of the queue. I'll go ahead and implement solution no. 3, wdyt?
sgtm. Any concerns ?
I think it should be option 1 + 3, and pInfo.Timestamp should be pInfo.InitialAttemptTimestamp, right? as records the time based on :
I think we should revisit this issue, in terms of its assumption: So is it reasonable to always exclude time spent in PreEnqueue from ? Note that PreEnqueue != SchedulingGates, is one of the PreEnqueue's usecases. In other words, some user of PreEnqueue may want to include the time spent in PreEnqueue.
is a signal that cluster providers can use to build SLOs. The time spent before passing PreEnqueue is not under control of kube-scheduler, so considering it would not allow to build an SLO. Are you aware of other users of PreEnqueue?
True, but comparing to state that metric X can or cannot be used to build SLO (which is hard IMO), I'm in favor of giving suggestion on SLI/SLO's semantic definition, for example: the official SLI definition of has a footnote clearly explaining its definition: So if a pod is gated, it's not "immediately schedulable", and hence doesn't fall into the definition of podstartup_latency.
A viable solution might be providing an extra label like to explicitly exclude the time spent on preenqueue.
That is true, but there is no way today to exclude those pods from the metric, or separate the time. Maybe we need an extra metric, but I think the new metric could be the one that includes the time guarded in PreEnqueue. It should probably be part of the KEP. The old metric is already stable, so we can't add labels. And it might already be in use for SLOs.
cc
I think we should be working on that. And treat it as part of GA requirement for PodSchedulingReadiness.
An edge(maybe not edge) case is a pod's scheduling attempts is 0 but the is a long time, so prefer not to account for the waiting time when gated. For users not using PreEnqueue, they will not be impacted, but for users using PreEnqueue, it's a break change for it increases the duration time. And we can add a new metric like to measure how long a pod stays in a scheduler.
I thought of "attempts", it's just I'm not sure it's a good idea to rely on "attempts" to define a SLI/SLO, b/c "attempts" is obtained from the in-memory object and in a HA config env, it's only updated by the primary scheduler instance. In other words, the value is inaccurate when the lease owner switches.
attempt=0 is good enough for most scenarios
do we have a conclusion? Should we move on with the solution or remove this issue and the linked PR from the current release?
I would like to move forward with this proposal. ?
To move it forward, I think we should add new metric(s) to explicitly tell it's including or excluding the time spent on PreEnqueue. Given is in stable level: We will need a new metric like "podschedulingexcludingpreenqueuedurationseconds".
28 Bug Triage Shadow here. Just checking-in, if this is still on track for the 1.28 release?
We don't have a conclusion yet, but as a bug, it should qualify for changes after the freezes.

p.unschedulablePods.addOrUpdate(pInfo) return false, nil } <ins> if pInfo.InitialAttemptTimestamp == nil { now := p.clock.Now() pInfo.InitialAttemptTimestamp = &now } </ins> if err := p.activeQ.Add(pInfo); err != nil { logger.Error(err, "Error adding pod to the active queue", "pod", klog.KObj(pInfo.Pod)) return false, err


Question:
is recording the time that a Pod is gated. We use the timestamp when the scheduler inserts the pod into the queue: The period of time when a Pod fails PreEnque (like being gated) shouldn't be accounted in the podschedulingdurationseconds. Create a Pod with scheduling gates. Wait some time before removing the gate. Observe the metric No response <details</details<detailsAny </details<details</details<details</details<details</details<details</details
/sig scheduling
/help /triage accepted
This request has been marked as needing help from a contributor. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help /triage accepted Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
would you want to take a stab on this issue?
Yup, I can give it a shot. /assign helayoty
, as we discussed, we can address this issue with different solutions: the pod timestamp to the current time , so the gated time is substituted. timestamp . Use a pointer to the InitialAttemptTimestamp and interpret as the newest timestamp so the pod will go to the back of the queue. I'll go ahead and implement solution no. 3, wdyt?
sgtm. Any concerns ?
I think it should be option 1 + 3, and pInfo.Timestamp should be pInfo.InitialAttemptTimestamp, right? as records the time based on :
I think we should revisit this issue, in terms of its assumption: So is it reasonable to always exclude time spent in PreEnqueue from ? Note that PreEnqueue != SchedulingGates, is one of the PreEnqueue's usecases. In other words, some user of PreEnqueue may want to include the time spent in PreEnqueue.
is a signal that cluster providers can use to build SLOs. The time spent before passing PreEnqueue is not under control of kube-scheduler, so considering it would not allow to build an SLO. Are you aware of other users of PreEnqueue?
True, but comparing to state that metric X can or cannot be used to build SLO (which is hard IMO), I'm in favor of giving suggestion on SLI/SLO's semantic definition, for example: the official SLI definition of has a footnote clearly explaining its definition: So if a pod is gated, it's not "immediately schedulable", and hence doesn't fall into the definition of podstartup_latency.
A viable solution might be providing an extra label like to explicitly exclude the time spent on preenqueue.
That is true, but there is no way today to exclude those pods from the metric, or separate the time. Maybe we need an extra metric, but I think the new metric could be the one that includes the time guarded in PreEnqueue. It should probably be part of the KEP. The old metric is already stable, so we can't add labels. And it might already be in use for SLOs.
cc
I think we should be working on that. And treat it as part of GA requirement for PodSchedulingReadiness.
An edge(maybe not edge) case is a pod's scheduling attempts is 0 but the is a long time, so prefer not to account for the waiting time when gated. For users not using PreEnqueue, they will not be impacted, but for users using PreEnqueue, it's a break change for it increases the duration time. And we can add a new metric like to measure how long a pod stays in a scheduler.
I thought of "attempts", it's just I'm not sure it's a good idea to rely on "attempts" to define a SLI/SLO, b/c "attempts" is obtained from the in-memory object and in a HA config env, it's only updated by the primary scheduler instance. In other words, the value is inaccurate when the lease owner switches.
attempt=0 is good enough for most scenarios
do we have a conclusion? Should we move on with the solution or remove this issue and the linked PR from the current release?
I would like to move forward with this proposal. ?
To move it forward, I think we should add new metric(s) to explicitly tell it's including or excluding the time spent on PreEnqueue. Given is in stable level: We will need a new metric like "podschedulingexcludingpreenqueuedurationseconds".
28 Bug Triage Shadow here. Just checking-in, if this is still on track for the 1.28 release?
We don't have a conclusion yet, but as a bug, it should qualify for changes after the freezes.

return &framework.QueuedPodInfo{ PodInfo:                 podInfo, Timestamp:               now, <del> InitialAttemptTimestamp: now, </del> <ins> InitialAttemptTimestamp: nil, </ins> UnschedulablePlugins:    sets.New(plugins...), } }


Question:
is recording the time that a Pod is gated. We use the timestamp when the scheduler inserts the pod into the queue: The period of time when a Pod fails PreEnque (like being gated) shouldn't be accounted in the podschedulingdurationseconds. Create a Pod with scheduling gates. Wait some time before removing the gate. Observe the metric No response <details</details<detailsAny </details<details</details<details</details<details</details<details</details
/sig scheduling
/help /triage accepted
This request has been marked as needing help from a contributor. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help /triage accepted Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
would you want to take a stab on this issue?
Yup, I can give it a shot. /assign helayoty
, as we discussed, we can address this issue with different solutions: the pod timestamp to the current time , so the gated time is substituted. timestamp . Use a pointer to the InitialAttemptTimestamp and interpret as the newest timestamp so the pod will go to the back of the queue. I'll go ahead and implement solution no. 3, wdyt?
sgtm. Any concerns ?
I think it should be option 1 + 3, and pInfo.Timestamp should be pInfo.InitialAttemptTimestamp, right? as records the time based on :
I think we should revisit this issue, in terms of its assumption: So is it reasonable to always exclude time spent in PreEnqueue from ? Note that PreEnqueue != SchedulingGates, is one of the PreEnqueue's usecases. In other words, some user of PreEnqueue may want to include the time spent in PreEnqueue.
is a signal that cluster providers can use to build SLOs. The time spent before passing PreEnqueue is not under control of kube-scheduler, so considering it would not allow to build an SLO. Are you aware of other users of PreEnqueue?
True, but comparing to state that metric X can or cannot be used to build SLO (which is hard IMO), I'm in favor of giving suggestion on SLI/SLO's semantic definition, for example: the official SLI definition of has a footnote clearly explaining its definition: So if a pod is gated, it's not "immediately schedulable", and hence doesn't fall into the definition of podstartup_latency.
A viable solution might be providing an extra label like to explicitly exclude the time spent on preenqueue.
That is true, but there is no way today to exclude those pods from the metric, or separate the time. Maybe we need an extra metric, but I think the new metric could be the one that includes the time guarded in PreEnqueue. It should probably be part of the KEP. The old metric is already stable, so we can't add labels. And it might already be in use for SLOs.
cc
I think we should be working on that. And treat it as part of GA requirement for PodSchedulingReadiness.
An edge(maybe not edge) case is a pod's scheduling attempts is 0 but the is a long time, so prefer not to account for the waiting time when gated. For users not using PreEnqueue, they will not be impacted, but for users using PreEnqueue, it's a break change for it increases the duration time. And we can add a new metric like to measure how long a pod stays in a scheduler.
I thought of "attempts", it's just I'm not sure it's a good idea to rely on "attempts" to define a SLI/SLO, b/c "attempts" is obtained from the in-memory object and in a HA config env, it's only updated by the primary scheduler instance. In other words, the value is inaccurate when the lease owner switches.
attempt=0 is good enough for most scenarios
do we have a conclusion? Should we move on with the solution or remove this issue and the linked PR from the current release?
I would like to move forward with this proposal. ?
To move it forward, I think we should add new metric(s) to explicitly tell it's including or excluding the time spent on PreEnqueue. Given is in stable level: We will need a new metric like "podschedulingexcludingpreenqueuedurationseconds".
28 Bug Triage Shadow here. Just checking-in, if this is still on track for the 1.28 release?
We don't have a conclusion yet, but as a bug, it should qualify for changes after the freezes.

t.Fatalf("Failed to pop a pod %v", err) } checkPerPodSchedulingMetrics("Attempt twice with update", t, pInfo, 2, timestamp) <ins> // Case 4: A gated pod is created and scheduled after lifting gate. The queue operations are //  Add gated pod -> check unschedulablePods -> lift gate & update pod -> Pop. c = testingclock.NewFakeClock(timestamp) // Create a queue with PreEnqueuePlugin m := map[string][]framework.PreEnqueuePlugin{"": {&preEnqueuePlugin{allowlists: []string{"foo"}}}} queue = NewTestQueue(ctx, newDefaultQueueSort(), WithClock(c), WithPreEnqueuePluginMap(m), WithPluginMetricsSamplePercent(0)) // Create a pod without PreEnqueuePlugin label. gatedPod := st.MakePod().Name("gated-test-pod").Namespace("test-ns").UID("test-uid").Obj() err = queue.Add(logger, gatedPod) if err != nil { t.Fatalf("Failed to add a pod %v", err) } // Check pod is added to the unschedulablePods queue. if getUnschedulablePod(queue, gatedPod) != gatedPod { t.Errorf("Pod %v was not found in the unschedulablePods.", gatedPod.Name) } // Override clock to get different InitialAttemptTimestamp c.Step(1 * time.Minute) // Update pod with the required label to get it out of unschedulablePods queue. updateGatedPod := gatedPod.DeepCopy() updateGatedPod.Labels = map[string]string{"foo": ""} queue.Update(logger, gatedPod, updateGatedPod) pInfo, err = queue.Pop() if err != nil { t.Fatalf("Failed to pop a pod %v", err) } checkPerPodSchedulingMetrics("Attempt once/gated", t, pInfo, 1, timestamp.Add(1*time.Minute)) </ins> } func TestIncomingPodsMetrics(t *testing.T) {


Question:
is recording the time that a Pod is gated. We use the timestamp when the scheduler inserts the pod into the queue: The period of time when a Pod fails PreEnque (like being gated) shouldn't be accounted in the podschedulingdurationseconds. Create a Pod with scheduling gates. Wait some time before removing the gate. Observe the metric No response <details</details<detailsAny </details<details</details<details</details<details</details<details</details
/sig scheduling
/help /triage accepted
This request has been marked as needing help from a contributor. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help /triage accepted Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
would you want to take a stab on this issue?
Yup, I can give it a shot. /assign helayoty
, as we discussed, we can address this issue with different solutions: the pod timestamp to the current time , so the gated time is substituted. timestamp . Use a pointer to the InitialAttemptTimestamp and interpret as the newest timestamp so the pod will go to the back of the queue. I'll go ahead and implement solution no. 3, wdyt?
sgtm. Any concerns ?
I think it should be option 1 + 3, and pInfo.Timestamp should be pInfo.InitialAttemptTimestamp, right? as records the time based on :
I think we should revisit this issue, in terms of its assumption: So is it reasonable to always exclude time spent in PreEnqueue from ? Note that PreEnqueue != SchedulingGates, is one of the PreEnqueue's usecases. In other words, some user of PreEnqueue may want to include the time spent in PreEnqueue.
is a signal that cluster providers can use to build SLOs. The time spent before passing PreEnqueue is not under control of kube-scheduler, so considering it would not allow to build an SLO. Are you aware of other users of PreEnqueue?
True, but comparing to state that metric X can or cannot be used to build SLO (which is hard IMO), I'm in favor of giving suggestion on SLI/SLO's semantic definition, for example: the official SLI definition of has a footnote clearly explaining its definition: So if a pod is gated, it's not "immediately schedulable", and hence doesn't fall into the definition of podstartup_latency.
A viable solution might be providing an extra label like to explicitly exclude the time spent on preenqueue.
That is true, but there is no way today to exclude those pods from the metric, or separate the time. Maybe we need an extra metric, but I think the new metric could be the one that includes the time guarded in PreEnqueue. It should probably be part of the KEP. The old metric is already stable, so we can't add labels. And it might already be in use for SLOs.
cc
I think we should be working on that. And treat it as part of GA requirement for PodSchedulingReadiness.
An edge(maybe not edge) case is a pod's scheduling attempts is 0 but the is a long time, so prefer not to account for the waiting time when gated. For users not using PreEnqueue, they will not be impacted, but for users using PreEnqueue, it's a break change for it increases the duration time. And we can add a new metric like to measure how long a pod stays in a scheduler.
I thought of "attempts", it's just I'm not sure it's a good idea to rely on "attempts" to define a SLI/SLO, b/c "attempts" is obtained from the in-memory object and in a HA config env, it's only updated by the primary scheduler instance. In other words, the value is inaccurate when the lease owner switches.
attempt=0 is good enough for most scenarios
do we have a conclusion? Should we move on with the solution or remove this issue and the linked PR from the current release?
I would like to move forward with this proposal. ?
To move it forward, I think we should add new metric(s) to explicitly tell it's including or excluding the time spent on PreEnqueue. Given is in stable level: We will need a new metric like "podschedulingexcludingpreenqueuedurationseconds".
28 Bug Triage Shadow here. Just checking-in, if this is still on track for the 1.28 release?
We don't have a conclusion yet, but as a bug, it should qualify for changes after the freezes.

if pInfo.Attempts != wantAttempts { t.Errorf("[%s] Pod schedule attempt unexpected, got %v, want %v", name, pInfo.Attempts, wantAttempts) } <del> if pInfo.InitialAttemptTimestamp != wantInitialAttemptTs { t.Errorf("[%s] Pod initial schedule attempt timestamp unexpected, got %v, want %v", name, pInfo.InitialAttemptTimestamp, wantInitialAttemptTs) </del> <ins> if *pInfo.InitialAttemptTimestamp != wantInitialAttemptTs { t.Errorf("[%s] Pod initial schedule attempt timestamp unexpected, got %v, want %v", name, *pInfo.InitialAttemptTimestamp, wantInitialAttemptTs) </ins> } }


Question:
is recording the time that a Pod is gated. We use the timestamp when the scheduler inserts the pod into the queue: The period of time when a Pod fails PreEnque (like being gated) shouldn't be accounted in the podschedulingdurationseconds. Create a Pod with scheduling gates. Wait some time before removing the gate. Observe the metric No response <details</details<detailsAny </details<details</details<details</details<details</details<details</details
/sig scheduling
/help /triage accepted
This request has been marked as needing help from a contributor. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help /triage accepted Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
would you want to take a stab on this issue?
Yup, I can give it a shot. /assign helayoty
, as we discussed, we can address this issue with different solutions: the pod timestamp to the current time , so the gated time is substituted. timestamp . Use a pointer to the InitialAttemptTimestamp and interpret as the newest timestamp so the pod will go to the back of the queue. I'll go ahead and implement solution no. 3, wdyt?
sgtm. Any concerns ?
I think it should be option 1 + 3, and pInfo.Timestamp should be pInfo.InitialAttemptTimestamp, right? as records the time based on :
I think we should revisit this issue, in terms of its assumption: So is it reasonable to always exclude time spent in PreEnqueue from ? Note that PreEnqueue != SchedulingGates, is one of the PreEnqueue's usecases. In other words, some user of PreEnqueue may want to include the time spent in PreEnqueue.
is a signal that cluster providers can use to build SLOs. The time spent before passing PreEnqueue is not under control of kube-scheduler, so considering it would not allow to build an SLO. Are you aware of other users of PreEnqueue?
True, but comparing to state that metric X can or cannot be used to build SLO (which is hard IMO), I'm in favor of giving suggestion on SLI/SLO's semantic definition, for example: the official SLI definition of has a footnote clearly explaining its definition: So if a pod is gated, it's not "immediately schedulable", and hence doesn't fall into the definition of podstartup_latency.
A viable solution might be providing an extra label like to explicitly exclude the time spent on preenqueue.
That is true, but there is no way today to exclude those pods from the metric, or separate the time. Maybe we need an extra metric, but I think the new metric could be the one that includes the time guarded in PreEnqueue. It should probably be part of the KEP. The old metric is already stable, so we can't add labels. And it might already be in use for SLOs.
cc
I think we should be working on that. And treat it as part of GA requirement for PodSchedulingReadiness.
An edge(maybe not edge) case is a pod's scheduling attempts is 0 but the is a long time, so prefer not to account for the waiting time when gated. For users not using PreEnqueue, they will not be impacted, but for users using PreEnqueue, it's a break change for it increases the duration time. And we can add a new metric like to measure how long a pod stays in a scheduler.
I thought of "attempts", it's just I'm not sure it's a good idea to rely on "attempts" to define a SLI/SLO, b/c "attempts" is obtained from the in-memory object and in a HA config env, it's only updated by the primary scheduler instance. In other words, the value is inaccurate when the lease owner switches.
attempt=0 is good enough for most scenarios
do we have a conclusion? Should we move on with the solution or remove this issue and the linked PR from the current release?
I would like to move forward with this proposal. ?
To move it forward, I think we should add new metric(s) to explicitly tell it's including or excluding the time spent on PreEnqueue. Given is in stable level: We will need a new metric like "podschedulingexcludingpreenqueuedurationseconds".
28 Bug Triage Shadow here. Just checking-in, if this is still on track for the 1.28 release?
We don't have a conclusion yet, but as a bug, it should qualify for changes after the freezes.

logger.V(2).Info("Successfully bound pod to node", "pod", klog.KObj(assumedPod), "node", scheduleResult.SuggestedHost, "evaluatedNodes", scheduleResult.EvaluatedNodes, "feasibleNodes", scheduleResult.FeasibleNodes) metrics.PodScheduled(fwk.ProfileName(), metrics.SinceInSeconds(start)) metrics.PodSchedulingAttempts.Observe(float64(assumedPodInfo.Attempts)) <del> metrics.PodSchedulingDuration.WithLabelValues(getAttemptsLabel(assumedPodInfo)).Observe(metrics.SinceInSeconds(assumedPodInfo.InitialAttemptTimestamp)) </del> <ins> if assumedPodInfo.InitialAttemptTimestamp != nil { metrics.PodSchedulingDuration.WithLabelValues(getAttemptsLabel(assumedPodInfo)).Observe(metrics.SinceInSeconds(*assumedPodInfo.InitialAttemptTimestamp)) } </ins> // Run "postbind" plugins. fwk.RunPostBindPlugins(ctx, state, assumedPod, scheduleResult.SuggestedHost)


Question:
Data as of June 3rd, 9:00 PM Eastern. Latest data can be found here: Additional Context:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig release
cc Do we want to keep this job and fix it, or do something different considering the debs/rpms efforts?
I think we should keep this job as long as we build packages for Google infra (those are using different specs then packages for OBS).
fixed it:

<ins> #!/bin/bash # Copyright 2014 Google Inc. All rights reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # #     http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # Bring up a Kubernetes cluster. # Usage: #   wget -q -O - https://get.k8s.io | sh # or #   curl -sS https://get.k8s.io | sh # # Advanced options #  Set KUBERNETES_PROVIDER to choose between different providers: #  Google Compute Engine [default] #   * export KUBERNETES_PROVIDER=gce; wget -q -O - https://get.k8s.io | sh #  Amazon EC2 #   * export KUBERNETES_PROVIDER=aws; wget -q -O - https://get.k8s.io | sh #  Microsoft Azure #   * export KUBERNETES_PROVIDER=azure; wget -q -O - https://get.k8s.io | sh #  Vagrant (local virtual machines) #   * export KUBERNETES_PROVIDER=vagrant; wget -q -O - https://get.k8s.io | sh #  VMWare VSphere #   * export KUBERNETES_PROVIDER=vsphere; wget -q -O - https://get.k8s.io | sh #  Rackspace #   * export KUBERNETES_PROVIDER=rackspace; wget -q -O - https://get.k8s.io | sh # #  Set KUBERNETES_SKIP_DOWNLOAD to non-empty to skip downloading a release. #  Set KUBERNETES_SKIP_CONFIRM to skip the installation confirmation prompt. set -o errexit set -o nounset set -o pipefail function create-cluster { echo "Creating a kubernetes on ${KUBERNETES_PROVIDER:-gce}..." ( cd kubernetes ./cluster/kube-up.sh echo "Kubernetes binaries at ${PWD}/kubernetes/cluster/" echo "You may want to add this directory to your PATH in $HOME/.profile" echo "Installation successful!" ) } if [[ "${KUBERNETES_SKIP_DOWNLOAD-}" ]]; then create-cluster exit 0 fi release=v0.7.2 release_url=https://storage.googleapis.com/kubernetes-release/release/${release}/kubernetes.tar.gz uname=$(uname) if [[ "${uname}" == "Darwin" ]]; then platform="darwin" elif [[ "${uname}" == "Linux" ]]; then platform="linux" else echo "Unknown, unsupported platform: (${uname})." echo "Supported platforms: Linux, Darwin." echo "Bailing out." exit 2 fi machine=$(uname -m) if [[ "${machine}" == "x86_64" ]]; then arch="amd64" elif [[ "${machine}" == "i686" ]]; then arch="386" elif [[ "${machine}" == "arm*" ]]; then arch="arm" else echo "Unknown, unsupported architecture (${machine})." echo "Supported architectures x86_64, i686, arm*" echo "Bailing out." exit 3 fi file=kubernetes.tar.gz echo "Downloading kubernetes release ${release} to ${PWD}/kubernetes.tar.gz" if [[ -n "${KUBERNETES_SKIP_CONFIRM-}" ]]; then echo "Is this ok? [Y]/n" read confirm if [[ "$confirm" == "n" ]]; then echo "Aborting." exit 0 fi fi if [[ $(which wget) ]]; then wget -O ${file} ${release_url} elif [[ $(which curl) ]]; then curl -L -o ${file} ${release_url} else echo "Couldn't find curl or wget.  Bailing out." exit 1 fi echo "Unpacking kubernetes release ${release}" tar -xzf ${file} rm ${file} create-cluster </ins>


Question:
This pertains more for clusters running in GCP but not necessarily in a GKE environment. Clusters that using a CNI supporting overlay such as Calico, do not need explicit routes to be created on the infrastructure level, meaning that the route controller should not be necessary. The following snippet sets the condition for new GCE nodes when a new node is picked up by the node controller. This condition is supposed to be removed later by the route controller in However, as stated above there are cases where the route controller is not needed and should not be running. When Calico is deployed in such a way that the calico-node pods are ignoring any and all taints including the cloudprovider's taint (which we consider reasonable for a system pod such as calico), the above check creates the following race condition: a Calico pod may start and set the node network conditions ( to false) before the node controller reconciles the node. After that the node controller picks up the node and sets the condition to an otherwise healthy node. Leaving aside the race condition itself, I have to ask whether the check is meaningful in the first place. AFAIK, the kubelet in Kubelet only sets the condition for provider, but clusters should already be using kubelet with cloudprovider= For the node controller to not set the condition unnecessarily. An easy way to do it with a custom CCM image is to introduce some delay in the node sync loop, enough for pods such as calico-node to start on that node. No response <details</details<detailsGCP </details<details</details<details</details<details</details<detailsCalico </details
This should be fixed by /cc
/sig cloud-provider /sig network /assign
/triage accepted

}) err = clientretry.RetryOnConflict(UpdateNodeSpecBackoff, func() error { <del> var curNode *v1.Node if cnc.cloud.ProviderName() == "gce" { // TODO(wlan0): Move this logic to the route controller using the node taint instead of condition // Since there are node taints, do we still need this? // This condition marks the node as unusable until routes are initialized in the cloud provider if err := nodeutil.SetNodeCondition(cnc.kubeClient, types.NodeName(nodeName), v1.NodeCondition{ Type:               v1.NodeNetworkUnavailable, Status:             v1.ConditionTrue, Reason:             "NoRouteCreated", Message:            "Node created without a route", LastTransitionTime: metav1.Now(), }); err != nil { return err } // fetch latest node from API server since GCE-specific condition was set and informer cache may be stale curNode, err = cnc.kubeClient.CoreV1().Nodes().Get(context.TODO(), nodeName, metav1.GetOptions{}) if err != nil { return err } } else { curNode, err = cnc.nodeInformer.Lister().Get(nodeName) if err != nil { return err } </del> <ins> curNode, err = cnc.nodeInformer.Lister().Get(nodeName) if err != nil { return err </ins> } newNode := curNode.DeepCopy()


Question:
This pertains more for clusters running in GCP but not necessarily in a GKE environment. Clusters that using a CNI supporting overlay such as Calico, do not need explicit routes to be created on the infrastructure level, meaning that the route controller should not be necessary. The following snippet sets the condition for new GCE nodes when a new node is picked up by the node controller. This condition is supposed to be removed later by the route controller in However, as stated above there are cases where the route controller is not needed and should not be running. When Calico is deployed in such a way that the calico-node pods are ignoring any and all taints including the cloudprovider's taint (which we consider reasonable for a system pod such as calico), the above check creates the following race condition: a Calico pod may start and set the node network conditions ( to false) before the node controller reconciles the node. After that the node controller picks up the node and sets the condition to an otherwise healthy node. Leaving aside the race condition itself, I have to ask whether the check is meaningful in the first place. AFAIK, the kubelet in Kubelet only sets the condition for provider, but clusters should already be using kubelet with cloudprovider= For the node controller to not set the condition unnecessarily. An easy way to do it with a custom CCM image is to introduce some delay in the node sync loop, enough for pods such as calico-node to start on that node. No response <details</details<detailsGCP </details<details</details<details</details<details</details<detailsCalico </details
This should be fixed by /cc
/sig cloud-provider /sig network /assign
/triage accepted

} } <del> // test syncNode with instanceV2, same test case with TestGCECondition. func TestGCEConditionV2(t *testing.T) { existingNode := &v1.Node{ ObjectMeta: metav1.ObjectMeta{ Name:              "node0", CreationTimestamp: metav1.Date(2012, 1, 1, 0, 0, 0, 0, time.UTC), }, Status: v1.NodeStatus{ Conditions: []v1.NodeCondition{ { Type:               v1.NodeReady, Status:             v1.ConditionUnknown, LastHeartbeatTime:  metav1.Date(2015, 1, 1, 12, 0, 0, 0, time.UTC), LastTransitionTime: metav1.Date(2015, 1, 1, 12, 0, 0, 0, time.UTC), }, }, }, Spec: v1.NodeSpec{ Taints: []v1.Taint{ { Key:    cloudproviderapi.TaintExternalCloudProvider, Value:  "true", Effect: v1.TaintEffectNoSchedule, }, }, }, } fakeCloud := &fakecloud.Cloud{ EnableInstancesV2: true, InstanceTypes: map[types.NodeName]string{ types.NodeName("node0"): "t1.micro", }, ProviderID: map[types.NodeName]string{ types.NodeName("node0"): "fake://12334", }, Addresses: []v1.NodeAddress{ { Type:    v1.NodeHostName, Address: "node0.cloud.internal", }, { Type:    v1.NodeInternalIP, Address: "10.0.0.1", }, { Type:    v1.NodeExternalIP, Address: "132.143.154.163", }, }, Provider: "gce", Err:      nil, } clientset := fake.NewSimpleClientset(existingNode) factory := informers.NewSharedInformerFactory(clientset, 0) eventBroadcaster := record.NewBroadcaster() cloudNodeController := &CloudNodeController{ kubeClient:                clientset, nodeInformer:              factory.Core().V1().Nodes(), nodesLister:               factory.Core().V1().Nodes().Lister(), cloud:                     fakeCloud, recorder:                  eventBroadcaster.NewRecorder(scheme.Scheme, v1.EventSource{Component: "cloud-node-controller"}), nodeStatusUpdateFrequency: 1 * time.Second, } stopCh := make(chan struct{}) defer close(stopCh) factory.Start(stopCh) factory.WaitForCacheSync(stopCh) w := eventBroadcaster.StartLogging(klog.Infof) defer w.Stop() cloudNodeController.syncNode(context.TODO(), existingNode.Name) updatedNode, err := clientset.CoreV1().Nodes().Get(context.TODO(), existingNode.Name, metav1.GetOptions{}) if err != nil { t.Fatalf("error getting updated nodes: %v", err) } conditionAdded := false for _, cond := range updatedNode.Status.Conditions { if cond.Status == "True" && cond.Type == "NetworkUnavailable" && cond.Reason == "NoRouteCreated" { conditionAdded = true } } assert.True(t, conditionAdded, "Network Route Condition for GCE not added by external cloud initializer") } // This test checks that a node with the external cloud provider taint is cloudprovider initialized and // the GCE route condition is added if cloudprovider is GCE func TestGCECondition(t *testing.T) { existingNode := &v1.Node{ ObjectMeta: metav1.ObjectMeta{ Name:              "node0", CreationTimestamp: metav1.Date(2012, 1, 1, 0, 0, 0, 0, time.UTC), }, Status: v1.NodeStatus{ Conditions: []v1.NodeCondition{ { Type:               v1.NodeReady, Status:             v1.ConditionUnknown, LastHeartbeatTime:  metav1.Date(2015, 1, 1, 12, 0, 0, 0, time.UTC), LastTransitionTime: metav1.Date(2015, 1, 1, 12, 0, 0, 0, time.UTC), }, }, }, Spec: v1.NodeSpec{ Taints: []v1.Taint{ { Key:    cloudproviderapi.TaintExternalCloudProvider, Value:  "true", Effect: v1.TaintEffectNoSchedule, }, }, }, } fakeCloud := &fakecloud.Cloud{ EnableInstancesV2: false, InstanceTypes: map[types.NodeName]string{ types.NodeName("node0"): "t1.micro", }, Addresses: []v1.NodeAddress{ { Type:    v1.NodeHostName, Address: "node0.cloud.internal", }, { Type:    v1.NodeInternalIP, Address: "10.0.0.1", }, { Type:    v1.NodeExternalIP, Address: "132.143.154.163", }, }, Provider: "gce", Err:      nil, } clientset := fake.NewSimpleClientset(existingNode) factory := informers.NewSharedInformerFactory(clientset, 0) eventBroadcaster := record.NewBroadcaster() cloudNodeController := &CloudNodeController{ kubeClient:                clientset, nodeInformer:              factory.Core().V1().Nodes(), nodesLister:               factory.Core().V1().Nodes().Lister(), cloud:                     fakeCloud, recorder:                  eventBroadcaster.NewRecorder(scheme.Scheme, v1.EventSource{Component: "cloud-node-controller"}), nodeStatusUpdateFrequency: 1 * time.Second, } stopCh := make(chan struct{}) defer close(stopCh) factory.Start(stopCh) factory.WaitForCacheSync(stopCh) w := eventBroadcaster.StartLogging(klog.Infof) defer w.Stop() cloudNodeController.syncNode(context.TODO(), existingNode.Name) updatedNode, err := clientset.CoreV1().Nodes().Get(context.TODO(), existingNode.Name, metav1.GetOptions{}) if err != nil { t.Fatalf("error getting updated nodes: %v", err) } conditionAdded := false for _, cond := range updatedNode.Status.Conditions { if cond.Status == "True" && cond.Type == "NetworkUnavailable" && cond.Reason == "NoRouteCreated" { conditionAdded = true } } assert.True(t, conditionAdded, "Network Route Condition for GCE not added by external cloud initializer") } </del> func Test_reconcileNodeLabels(t *testing.T) { testcases := []struct { name           string


Question:
Add the following to be safe sysctls: net.ipv4.tcpfintimeout net.ipv4.tcpkeepaliveintvl net.ipv4.tcpkeepaliveprobes These are related settings to net.ipv4.tcpkeepalivetime that was enabled in These are related ipv4 TCP settings for tuning network connections between a client and a pod.
/sig network
May you help provide some tests to ensure that setting this sysctl can't affect any other pods or the node.
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : May you help provide some tests to ensure that setting this sysctl can't affect any other pods or the node. /sig nodes Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
I adjusted it with the plugin.
Would this change apply at the node level or the pod level? I assumed it would be at the pod level, and should not affect any other pods--however Kubernetes does not natively have any limits on network connections that I am aware of--but only the pod with the changed net settings would be affected.
To verify that a sysctl has an effect on other pods, we need to verify that it's a namespace sysctl.
Here's a simple test Use net.ipv4.tcpfintimeout sysctl to create a container check if it set on the host =It is the default value on host run another container attach to the network namespace of the one where the sysctl is set =results in sysctl being set Verify in another container under a different network namespace that it is not set =It is the default value According to the test results, net.ipv4.tcpfintimeout is a namespaced sysctl
I likewise verified and , which are also namespaced sysctl, and I believe we can add them to the safe sysctl /assign
Probably need to wait for to finish before start working on it
/triage accepted

name: "net.ipv4.tcp_keepalive_time", // refer to https://github.com/torvalds/linux/commit/13b287e8d1cad951634389f85b8c9b816bd3bb1e. kernel: "4.5", <ins> }, { // refer to https://github.com/torvalds/linux/commit/1e579caa18b96f9eb18f4f5416658cd15f37c062. name:   "net.ipv4.tcp_fin_timeout", kernel: "4.6", }, { // refer to https://github.com/torvalds/linux/commit/b840d15d39128d08ed4486085e5507d2617b9ae1. name:   "net.ipv4.tcp_keepalive_intvl", kernel: "4.5", }, { // refer to https://github.com/torvalds/linux/commit/9bd6861bd4326e3afd3f14a9ec8a723771fb20bb. name:   "net.ipv4.tcp_keepalive_probes", kernel: "4.5", </ins> }, }


Question:
Add the following to be safe sysctls: net.ipv4.tcpfintimeout net.ipv4.tcpkeepaliveintvl net.ipv4.tcpkeepaliveprobes These are related settings to net.ipv4.tcpkeepalivetime that was enabled in These are related ipv4 TCP settings for tuning network connections between a client and a pod.
/sig network
May you help provide some tests to ensure that setting this sysctl can't affect any other pods or the node.
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : May you help provide some tests to ensure that setting this sysctl can't affect any other pods or the node. /sig nodes Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
I adjusted it with the plugin.
Would this change apply at the node level or the pod level? I assumed it would be at the pod level, and should not affect any other pods--however Kubernetes does not natively have any limits on network connections that I am aware of--but only the pod with the changed net settings would be affected.
To verify that a sysctl has an effect on other pods, we need to verify that it's a namespace sysctl.
Here's a simple test Use net.ipv4.tcpfintimeout sysctl to create a container check if it set on the host =It is the default value on host run another container attach to the network namespace of the one where the sysctl is set =results in sysctl being set Verify in another container under a different network namespace that it is not set =It is the default value According to the test results, net.ipv4.tcpfintimeout is a namespaced sysctl
I likewise verified and , which are also namespaced sysctl, and I believe we can add them to the safe sysctl /assign
Probably need to wait for to finish before start working on it
/triage accepted

}, }, { <del> name: "kernelVersion is 5.15.0, return safeSysctls with no kernelVersion limit and net.ipv4.ip_local_reserved_ports and net.ipv4.tcp_keepalive_time", </del> <ins> name: "kernelVersion is 5.15.0, return safeSysctls with no kernelVersion limit and kernelVersion below 5.15.0", </ins> getVersion: func() (*version.Version, error) { kernelVersionStr := "5.15.0-75-generic" return version.ParseGeneric(kernelVersionStr)


Question:
Add the following to be safe sysctls: net.ipv4.tcpfintimeout net.ipv4.tcpkeepaliveintvl net.ipv4.tcpkeepaliveprobes These are related settings to net.ipv4.tcpkeepalivetime that was enabled in These are related ipv4 TCP settings for tuning network connections between a client and a pod.
/sig network
May you help provide some tests to ensure that setting this sysctl can't affect any other pods or the node.
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : May you help provide some tests to ensure that setting this sysctl can't affect any other pods or the node. /sig nodes Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
I adjusted it with the plugin.
Would this change apply at the node level or the pod level? I assumed it would be at the pod level, and should not affect any other pods--however Kubernetes does not natively have any limits on network connections that I am aware of--but only the pod with the changed net settings would be affected.
To verify that a sysctl has an effect on other pods, we need to verify that it's a namespace sysctl.
Here's a simple test Use net.ipv4.tcpfintimeout sysctl to create a container check if it set on the host =It is the default value on host run another container attach to the network namespace of the one where the sysctl is set =results in sysctl being set Verify in another container under a different network namespace that it is not set =It is the default value According to the test results, net.ipv4.tcpfintimeout is a namespaced sysctl
I likewise verified and , which are also namespaced sysctl, and I believe we can add them to the safe sysctl /assign
Probably need to wait for to finish before start working on it
/triage accepted

"net.ipv4.ip_unprivileged_port_start", "net.ipv4.ip_local_reserved_ports", "net.ipv4.tcp_keepalive_time", <ins> "net.ipv4.tcp_fin_timeout", "net.ipv4.tcp_keepalive_intvl", "net.ipv4.tcp_keepalive_probes", </ins> }, }, }


Question:
Add the following to be safe sysctls: net.ipv4.tcpfintimeout net.ipv4.tcpkeepaliveintvl net.ipv4.tcpkeepaliveprobes These are related settings to net.ipv4.tcpkeepalivetime that was enabled in These are related ipv4 TCP settings for tuning network connections between a client and a pod.
/sig network
May you help provide some tests to ensure that setting this sysctl can't affect any other pods or the node.
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : May you help provide some tests to ensure that setting this sysctl can't affect any other pods or the node. /sig nodes Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
I adjusted it with the plugin.
Would this change apply at the node level or the pod level? I assumed it would be at the pod level, and should not affect any other pods--however Kubernetes does not natively have any limits on network connections that I am aware of--but only the pod with the changed net settings would be affected.
To verify that a sysctl has an effect on other pods, we need to verify that it's a namespace sysctl.
Here's a simple test Use net.ipv4.tcpfintimeout sysctl to create a container check if it set on the host =It is the default value on host run another container attach to the network namespace of the one where the sysctl is set =results in sysctl being set Verify in another container under a different network namespace that it is not set =It is the default value According to the test results, net.ipv4.tcpfintimeout is a namespaced sysctl
I likewise verified and , which are also namespaced sysctl, and I believe we can add them to the safe sysctl /assign
Probably need to wait for to finish before start working on it
/triage accepted

'net.ipv4.ip_unprivileged_port_start' 'net.ipv4.ip_local_reserved_ports' 'net.ipv4.tcp_keepalive_time' <ins> 'net.ipv4.tcp_fin_timeout' 'net.ipv4.tcp_keepalive_intvl' 'net.ipv4.tcp_keepalive_probes' </ins> */


Question:
Add the following to be safe sysctls: net.ipv4.tcpfintimeout net.ipv4.tcpkeepaliveintvl net.ipv4.tcpkeepaliveprobes These are related settings to net.ipv4.tcpkeepalivetime that was enabled in These are related ipv4 TCP settings for tuning network connections between a client and a pod.
/sig network
May you help provide some tests to ensure that setting this sysctl can't affect any other pods or the node.
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : May you help provide some tests to ensure that setting this sysctl can't affect any other pods or the node. /sig nodes Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
I adjusted it with the plugin.
Would this change apply at the node level or the pod level? I assumed it would be at the pod level, and should not affect any other pods--however Kubernetes does not natively have any limits on network connections that I am aware of--but only the pod with the changed net settings would be affected.
To verify that a sysctl has an effect on other pods, we need to verify that it's a namespace sysctl.
Here's a simple test Use net.ipv4.tcpfintimeout sysctl to create a container check if it set on the host =It is the default value on host run another container attach to the network namespace of the one where the sysctl is set =results in sysctl being set Verify in another container under a different network namespace that it is not set =It is the default value According to the test results, net.ipv4.tcpfintimeout is a namespaced sysctl
I likewise verified and , which are also namespaced sysctl, and I believe we can add them to the safe sysctl /assign
Probably need to wait for to finish before start working on it
/triage accepted

"net.ipv4.ip_unprivileged_port_start", "net.ipv4.ip_local_reserved_ports", "net.ipv4.tcp_keepalive_time", <ins> "net.ipv4.tcp_fin_timeout", "net.ipv4.tcp_keepalive_intvl", "net.ipv4.tcp_keepalive_probes", </ins> ) )


Question:
Add the following to be safe sysctls: net.ipv4.tcpfintimeout net.ipv4.tcpkeepaliveintvl net.ipv4.tcpkeepaliveprobes These are related settings to net.ipv4.tcpkeepalivetime that was enabled in These are related ipv4 TCP settings for tuning network connections between a client and a pod.
/sig network
May you help provide some tests to ensure that setting this sysctl can't affect any other pods or the node.
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : May you help provide some tests to ensure that setting this sysctl can't affect any other pods or the node. /sig nodes Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
I adjusted it with the plugin.
Would this change apply at the node level or the pod level? I assumed it would be at the pod level, and should not affect any other pods--however Kubernetes does not natively have any limits on network connections that I am aware of--but only the pod with the changed net settings would be affected.
To verify that a sysctl has an effect on other pods, we need to verify that it's a namespace sysctl.
Here's a simple test Use net.ipv4.tcpfintimeout sysctl to create a container check if it set on the host =It is the default value on host run another container attach to the network namespace of the one where the sysctl is set =results in sysctl being set Verify in another container under a different network namespace that it is not set =It is the default value According to the test results, net.ipv4.tcpfintimeout is a namespaced sysctl
I likewise verified and , which are also namespaced sysctl, and I believe we can add them to the safe sysctl /assign
Probably need to wait for to finish before start working on it
/triage accepted

expectReason: `forbidden sysctls`, expectDetail: `net.ipv4.tcp_keepalive_time`, }, <ins> { name: "new supported sysctls not supported: net.ipv4.tcp_fin_timeout", pod: &corev1.Pod{Spec: corev1.PodSpec{ SecurityContext: &corev1.PodSecurityContext{ Sysctls: []corev1.Sysctl{{Name: "net.ipv4.tcp_fin_timeout", Value: "60"}}, }, }}, allowed:      false, expectReason: `forbidden sysctls`, expectDetail: `net.ipv4.tcp_fin_timeout`, }, { name: "new supported sysctls not supported: net.ipv4.tcp_keepalive_intvl", pod: &corev1.Pod{Spec: corev1.PodSpec{ SecurityContext: &corev1.PodSecurityContext{ Sysctls: []corev1.Sysctl{{Name: "net.ipv4.tcp_keepalive_intvl", Value: "75"}}, }, }}, allowed:      false, expectReason: `forbidden sysctls`, expectDetail: `net.ipv4.tcp_keepalive_intvl`, }, { name: "new supported sysctls not supported: net.ipv4.tcp_keepalive_probes", pod: &corev1.Pod{Spec: corev1.PodSpec{ SecurityContext: &corev1.PodSecurityContext{ Sysctls: []corev1.Sysctl{{Name: "net.ipv4.tcp_keepalive_probes", Value: "9"}}, }, }}, allowed:      false, expectReason: `forbidden sysctls`, expectDetail: `net.ipv4.tcp_keepalive_probes`, }, </ins> } for _, tc := range tests {