Question: // GetActivePods returns pods that may have a running container (a // terminated pod is one that is known to have no running containers and // will not get any more). <ins> // // TODO: This method must include pods that have been force deleted from // the config source (and thus removed from the pod manager) but are still // terminating. </ins> func (kl *Kubelet) GetActivePods() []*v1.Pod { allPods := kl.podManager.GetPods() activePods := kl.filterOutTerminatedPods(allPods)

</details
/sig node
/cc
Looking at the kubelet log in 1.22 () here's what I see -- First pod starts up... starts dies after ~10 seconds since the job is completed (the container simply runs a ) Pod completes, status is updated to API server Next pod comes in but is rejected with despite the earlier pod already being completed/terminated
did the kubelet pod lifecycle rework result in a reordering of kubelet resource bookeeping w.r.t. pod status? it seems reasonable for the scheduler to send another pod to a kubelet once it reports the pod complete and the use of the resources is done from an API perspective
I am also seeing this same behavior, but with "OutOfPods" rather than "OutOfMemory", when running a job with parallelism=100, completions=400 on GKE (which has a 110 pod per node limit).
this is easily reproducible on a local cluster as well start with this: then create this job: result on 1.22 is the job hangs with 2 completions and a bunch of rejections by the kubelet: on 1.21, the same job completes all pods successfully in a couple minutes with no OutOfpods errors
verified introduced the OutOfPods issue
Shame we didn’t have an e2e which caught this given how easy the reproducer is. This looks to be the opposite of the previous cpu admission bug (ignoring things that are terminating but not terminated). We are weak on admission testing in general. This potentially implies admission is looking at the wrong set of pods (admitted + terminating). Note there is already a bug in 1.21 and before where force deleted pods are incorrectly ignored from the set of allocated resources while they are terminating, but this doesn’t immediately seem to be that issue (since i doubt those jobs are being force deleted)
yeah, there was no force deletion involved in (or any deletion at all, actually... the completed pods just sat there, completed)
The set of admitted pods considered for pod count is everything in the pod manager filtered by filterOutTerminatingPods, which calls CouldHaveRunningContainers which returns true once a pod is seen by the workers (shortly after admission, and until we’ve confirmed via syncTerminatingPod that no containers are running). Whatever the problem is it’s likely in the interaction of that.


Question: // GetActivePods returns pods that may have a running container (a // terminated pod is one that is known to have no running containers and // will not get any more). <ins> // // TODO: This method must include pods that have been force deleted from // the config source (and thus removed from the pod manager) but are still // terminating. </ins> func (kl *Kubelet) GetActivePods() []*v1.Pod { allPods := kl.podManager.GetPods() activePods := kl.filterOutTerminatedPods(allPods)

A pod the pod worker hasn’t seen yet is also filtered out (that means bursts of pods can get in, which seems the opposite of this problem, and is the bug i mentioned before).
Edit: one wrong thing spotted while looking - I'm not sure it fixes the issue you're seeing, but it's definitely wrong for admission (admission is filtering the known pod set down to remove pods that are terminated, but shouldn't filter pods that aren't admitted yet). This needs a node e2e reproducer that can run in serial (resource consumption is really hard to test from parallel, is slow in serial, and node e2e is back in a good state now) - / is that something either of you could help me with? I believe the reason the existing tests don't catch this is the job controller is creating pods fast enough that we see races in admission (multiple pods are admitted in one call to HandlePodAdditions) that are not caught in the current serial node e2e admission tests which rarely create more than one at a time.
Job pods still go through the scheduler to get assigned to a node, so even though the job controller can create pods in parallel, I'd still expect them to arrive at the node in an orderly fashion. It's still possible multiple arrive in a single HandlePodAdditions call, of course. It seems like kubelet admission is using laggy information that doesn't reflect pod completion the kubelet itself wrote to the pod API.
The bug turned out to be much simpler. Pods that are completed never go to the pod worker, so once a pod has reached terminal state it was being back to the list of active pods. The set of e2es that would have prevented this regression: node e2e that creates multiple pods at the same time and verifies that admission works correctly when HandlePodAdditions takes two pods at once node e2e that verifies that completed pods don't consume resources for admission purposes in the kubelet job controller e2e that attempts to verify some part of this reliably (ideally in a way that can be run in parallel e2e)
another related issue:
Linking here for completeness - a similar symptom might get exposed after this fix as described in The kubelet considers resources for terminating pods to be in use (they are!), but the scheduler ignores terminating pods and schedules new pods.


Question: // GetActivePods returns pods that may have a running container (a // terminated pod is one that is known to have no running containers and // will not get any more). <ins> // // TODO: This method must include pods that have been force deleted from // the config source (and thus removed from the pod manager) but are still // terminating. </ins> func (kl *Kubelet) GetActivePods() []*v1.Pod { allPods := kl.podManager.GetPods() activePods := kl.filterOutTerminatedPods(allPods)

Because the kubelet now considers terminating pods, it rejects those rapidly rescheduled pods. The follow up is more involved but requires the kubelet to allow pods to be potentially admitted that can't be run currently due to those terminating pods. See the linked issue.


Question: // Visit implements Visitor over a stream. func (v *StreamVisitor) Visit(fn VisitorFunc) error { <del> d := json.NewDecoder(v.Reader) </del> <ins> d := yaml.NewYAMLOrJSONDecoder(v.Reader, 4096) </ins> for { ext := runtime.RawExtension{} if err := d.Decode(&ext); err != nil {

If I stuff into a file and use that after -f, it works fine. cc
What's on stdin?
Bug, I'm only accepting json because I naively reused watch partitioner and we don't have test cases. Will fix tonight.
If that's the problem, why did it work when I gave it that file not over stdin?
Different code path. Stdin supports streaming json a la watch (multiple json objects in a stream), so you can do Unfortunately I missed yaml input and didn't use a segmenting yaml decoder (returning a stream of yaml objects).
Ah! Thanks for explanation. I think it would be good for us to ask for e2e runs before merging kubectl changes, since there's not quite enough coverage in test- yet.
Yeah, I'm slacking on that for these changes.
Fix incoming.
Fix provided. More tests to test-cmd... :)


Question: "context" "errors" "fmt" <del> "net" </del> "net/url" "strings" "time"

Tests in are using image for gRPC probe tests: should not be restarted with a GRPC liveness probe should be restarted with a GRPC liveness probe We should use image for these tests as it is much smaller size. /sig node /kind cleanup /priority important-soon /area tests /help-wanted /good-first-issue KEP:
This request has been marked as suitable for new contributors. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : Tests in are using image for gRPC probe tests: /kind cleanup /priority important-soon /area tests /help-wanted /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : Tests in are using image for gRPC probe tests: /kind cleanup /priority important-soon /area tests /help-wanted /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hi can I work on this issue? I'm new to Kubernetes and want to contribute!
Should I change to in this file?
Yes.


Question: "context" "errors" "fmt" <del> "net" </del> "net/url" "strings" "time"

Also pass proper parameters to agnhost to enable gRPC endpoint, change ports, and run tests to ensure that this worked.
Thanks for your reply! Will let you know if I have more questions as I work
/assign
/assign


Question: // are found. workqueue.ParallelizeUntil(ctx, 16, allNodes, checkNode) processedNodes := int(filteredLen) + len(filteredNodesStatuses) + len(failedPredicateMap) <del> g.lastProcessedNodeIndex = (g.lastProcessedNodeIndex + processedNodes) % allNodes </del> <ins> g.nextStartNodeIndex = (g.nextStartNodeIndex + processedNodes) % allNodes </ins> filtered = filtered[:filteredLen] if err := errCh.ReceiveError(); err != nil {

What would you like to be : A unit test that verifies the evaluation order of the nodes in the scheduler. Why is this needed: The scheduler evaluates the nodes in a specific order until it finds a pre-defined minimum number of feasible nodes. To give all nodes a fair chance of being evaluated, in each scheduling cycle, the scheduler starts evaluating from where it left off in the previous cycle. This is captured by the parameter in  We need to add a unit test that captures regressions related to this behavior to avoid bugs like /sig scheduling /priority important-soon /help
/assign
/remove-help


Question: if activeQueues == 0 { return 0 } <del> return math.Min(float64(reqs), float64(qs.dCfg.ConcurrencyLimit)) / float64(activeQueues) </del> <ins> return math.Min(float64(seatsInUse), float64(qs.dCfg.ConcurrencyLimit)) / float64(activeQueues) </ins> } // timeoutOldRequestsAndRejectOrEnqueueLocked encapsulates the logic required

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --I noticed that the the code in for calculating the dR/dt, the derivative of R(t) with respect to t, called in the code , was not updated when the generalization for width was introduced. The should be between C and the sum of seats not the number of requests. See , where it says Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
yes, i noticed it, I am going to open a PR. /assign
/triage accepted


Question: volumeMounts: - name: www-data mountPath: /srv/www <del> readOnly </del> <ins> readOnly: true </ins> containers: - name: git-monitor image: kubernetes/git-monitor

Which jobs are failing: Since when has it been failing: Testgrid link: Reason for failure: Anything else we need to know: /priority important-soon /cc /cc /milestone v1.18 /sig release
/assign
This is failing because a 1.18 release for CRI tools doesn't exist yet. I can tweak the test job arguments to skip this later.
(As a note, this job is important, but not ) /remove-priority critical-urgent /priority important-soon
Do we need to remove the job from dashboard?
-- No. I'll work on fixing the job later today.
I've started to investigate, but I haven't yet reproduced the new error locally: This is likely related to a recent refactor of the package that handles interpreting deb/rpm package versions. As is not currently in use, I'm moving this to the milestone. /milestone v1.19
the version issue is fixed via The test is now failing because it tries to download cri-tools v1.18.0: which I think is related to this code part:
Updating testgrid link here: Any updates here /retitle [Flaky Test] build-packages-debs (ci-release-build-packages-debs) /kind flaky-test
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : Updating testgrid link here: /kind flaky-test Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/kind flake
I was assuming that is working on this issue. I can jump in if you want and do not find the time to work on it. :)
/assign /unassign Sascha -- Can you take this? I thought I was going to have time to look at this, but it's been a while.
Sure, the reason for the failing test seems to have changed, I'm on it.
Fix is incoming in


Question: ResourceName string Command      []string <del> FullCmdName       string SuggestedCmdUsage string </del> <ins> ParentCommandName       string EnableSuggestedCmdUsage bool </ins> Builder          func() *resource.Builder ExecutablePodFn  polymorphichelpers.AttachablePodForObjectFunc

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: The pod name is missing in suggested cmd usage when executing : What you expected to happen: The suggested cmd usage should be a complete cmd like the one got from How to reproduce it (as minimally and precisely as possible): Use to attach any pod/deployment/service, the PodName will be missing in the suggested cmd usage like above. Anything else we need to know?: It works fine for kubectl < v1.12 Environment: Kubernetes version (use ): v1.13.1 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig cli /assign


Question: opts.Tail = strconv.FormatInt(*logOptions.TailLines, 10) } <ins> if logOptions.LimitBytes != nil { // stdout and stderr share the total write limit max := *logOptions.LimitBytes stderr = sharedLimitWriter(stderr, &max) stdout = sharedLimitWriter(stdout, &max) } </ins> sopts := libdocker.StreamOptions{ OutputStream: stdout, ErrorStream:  stderr, RawTerminal:  container.Config.Tty, } <del> return d.client.Logs(containerID.ID, opts, sopts) </del> <ins> err = d.client.Logs(containerID.ID, opts, sopts) if errors.Is(err, errMaximumWrite) { klog.V(2).Infof("finished logs, hit byte limit %d", *logOptions.LimitBytes) err = nil } return err </ins> } // GetContainerLogTail attempts to read up to MaxContainerTerminationMessageLogLength

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Conformance test failed: xref What you expected to happen: I expected this conformance test to pass. How to reproduce it (as minimally and precisely as possible): We have configured docker's logging driver to use journald, and then ran sonobouy tests against a k8s 1.16.3 cluster. Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: OS (e.g: ): Red Hat Enterprise Linux Server release 7.7 (Maipo) Kernel (e.g. ): 5.3.11- SMP Tue Nov 12 14:45:59 EST 2019 x8664 x8664 x8664 GNU/Linux Install tools: Network plugin and version (if this is a network-related bug): Others:
There are no sig labels on this issue. Please add a sig label by either: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. See the . The in method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/sig cli
/assign
I have a working patch that fixes this, will push a PR shortly.


Question: go_library( name = "go_default_library", <del> srcs = ["logs.go"], </del> <ins> srcs = [ "logs.go", "options.go", "registry.go", ], </ins> importmap = "k8s.io/kubernetes/vendor/k8s.io/component-base/logs", importpath = "k8s.io/component-base/logs", deps = [ "//staging/src/k8s.io/apimachinery/pkg/util/wait:go_default_library", <ins> "//vendor/github.com/go-logr/logr:go_default_library", </ins> "//vendor/github.com/spf13/pflag:go_default_library", "//vendor/k8s.io/klog/v2:go_default_library", ],

What would you like to be : As part of should add logging configuration allowing to change logging implementation. Things to implement: --logging-format - flag to pick between available implementations. Defaults to "text" using unchanged klog. LoggingConfig - structure storing component configuration for logging LogFormatRegistry - Used for registration of additional log formats. New formats should be registered by providing unique name and implementation of interface LoggingConfig.Complete method should configure klog by passing logr interface implementation into klog.SetLogger method Why is this needed: Subtask of /help /wg component-standard
The klog 1.0.0 version looks like no SetLogger method, should update klog version in vendor?
yes, forgot about that. I remember saying that it will require some changes, but shouldn't be to hard. I will help with it if needed.
/assign /remove-help
Klog was upgraded to v2.


<!-- Please only use this template for submitting enhancement requests --What would you like to be : Why is this needed: Because of is not recommended to create cronjob (with flags) anymore. But there is no imperative command available for to create cronjob. /sig CLI <!-- DO NOT EDIT BELOW THIS LINE --/kind feature
/assign
GitHub didn't allow me to assign the following users: Pingan2017. Note that only and repo collaborators can be assigned. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
i wanna work on this. give me several days
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale


Question: } func gRPCServerPodSpec(readinessProbe, livenessProbe *v1.Probe, containerName string) *v1.Pod { <del> etcdLocalhostAddress := "127.0.0.1" if framework.TestContext.ClusterIsIPv6() { etcdLocalhostAddress = "::1" } etcdURL := fmt.Sprintf("http://%s", net.JoinHostPort(etcdLocalhostAddress, "2379")) </del> return &v1.Pod{ ObjectMeta: metav1.ObjectMeta{Name: "test-grpc-" + string(uuid.NewUUID())}, Spec: v1.PodSpec{ Containers: []v1.Container{ { Name:  containerName, <del> Image: imageutils.GetE2EImage(imageutils.Etcd), </del> <ins> Image: imageutils.GetE2EImage(imageutils.Agnhost), </ins> Command: []string{ <del> "/usr/local/bin/etcd", "--listen-client-urls", "http://0.0.0.0:2379", //should listen on all addresses "--advertise-client-urls", etcdURL, </del> <ins> "/agnhost", "grpc-health-checking", </ins> }, <del> // 2380 is an automatic peer URL Ports:          []v1.ContainerPort{{ContainerPort: int32(2379)}, {ContainerPort: int32(2380)}}, </del> <ins> Ports:          []v1.ContainerPort{{ContainerPort: int32(5000)}, {ContainerPort: int32(8080)}}, </ins> LivenessProbe:  livenessProbe, ReadinessProbe: readinessProbe, },


Question: } func gRPCServerPodSpec(readinessProbe, livenessProbe *v1.Probe, containerName string) *v1.Pod { <del> etcdLocalhostAddress := "127.0.0.1" if framework.TestContext.ClusterIsIPv6() { etcdLocalhostAddress = "::1" } etcdURL := fmt.Sprintf("http://%s", net.JoinHostPort(etcdLocalhostAddress, "2379")) </del> return &v1.Pod{ ObjectMeta: metav1.ObjectMeta{Name: "test-grpc-" + string(uuid.NewUUID())}, Spec: v1.PodSpec{ Containers: []v1.Container{ { Name:  containerName, <del> Image: imageutils.GetE2EImage(imageutils.Etcd), </del> <ins> Image: imageutils.GetE2EImage(imageutils.Agnhost), </ins> Command: []string{ <del> "/usr/local/bin/etcd", "--listen-client-urls", "http://0.0.0.0:2379", //should listen on all addresses "--advertise-client-urls", etcdURL, </del> <ins> "/agnhost", "grpc-health-checking", </ins> }, <del> // 2380 is an automatic peer URL Ports:          []v1.ContainerPort{{ContainerPort: int32(2379)}, {ContainerPort: int32(2380)}}, </del> <ins> Ports:          []v1.ContainerPort{{ContainerPort: int32(5000)}, {ContainerPort: int32(8080)}}, </ins> LivenessProbe:  livenessProbe, ReadinessProbe: readinessProbe, },


Question: expectedCondition       *batch.JobConditionType expectedConditionReason string }{ <ins> "backoffLimit 0 should have 1 pod active": { 1, 1, 0, true, []int32{0}, 1, 0, 0, nil, "", }, "backoffLimit 1 with restartCount 0 should have 1 pod active": { 1, 1, 1, true, []int32{0}, 1, 0, 0, nil, "", }, "backoffLimit 1 with restartCount 1 should have 0 pod active": { 1, 1, 1, true, []int32{1}, 0, 0, 1, &jobConditionFailed, "BackoffLimitExceeded", }, </ins> "too many job failures - single pod": { 1, 5, 2, true, []int32{2},

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: If you have a JobSpec and set with in 1.9.x, then the job will start a pod and never restart it regardless of success of failure. In 1.10.x, a job spec with and , k8s does not start any pods at all. We had to set in order to get the pod to run. The wording in the docs suggest that should only take effect when the pod is finished/or failed. It shouldn't cap starting the initial pod. How to reproduce it (as minimally and precisely as possible): Run the following job spec in 1.9.x and 1.10.x: Anything else we need to know?: The docs links this existing ticket saying is ineffective with , but the ticket talks about restarting too many times, not never starting a pod. Given this issue was slightly different, I originally did not think it was a backoffPolicy bug. Environment: Kubernetes version (use ): Tested on 1.9.7 and 1.10.6 Cloud provider or hardware configuration: AWS OS (e.g. from /etc/os-release): Fedora 28
/sig apps
We've been hit by this too, in v1.10.6. It seems to me that with the current behavior, a value of doesn't really make any semantic sense. I think the problem comes from the function in : called here: if is , then will end with .


Question: f.BindFlags(cmds.PersistentFlags()) f.BindExternalFlags(cmds.PersistentFlags()) <ins> cmds.SetHelpCommand(NewCmdHelp(f, out)) </ins> // From this point and forward we get warnings on flags that contain "_" separators cmds.SetGlobalNormalizationFunc(flag.WarnWordSepNormalizeFunc)

It'd be similar to Linux , a command to search the man page files
Why not just make do a search if it doesn't have an exact match?
sounds good.
SGTM
- Can be useful in this issue
I will work on this issue, if nobody is already working on it ( is not working on it).
thanks!
I focused currently on a bit different stuff, sorry. So I will start/finish it later. If anyone want to take over this one, feel free for now.


Question: } // markAsDeleting sets the obj's DeletionGracePeriodSeconds to 0, and sets the <del> // DeletionTimestamp to "now". Finalizers are watching for such updates and will </del> <ins> // DeletionTimestamp to "now" if there is no existing deletionTimestamp or if the existing // deletionTimestamp is further in future. Finalizers are watching for such updates and will </ins> // finalize the object if their IDs are present in the object's Finalizers list. <del> func markAsDeleting(obj runtime.Object) (err error) { </del> <ins> func markAsDeleting(obj runtime.Object, now time.Time) (err error) { </ins> objectMeta, kerr := meta.Accessor(obj) if kerr != nil { return kerr } <del> now := metav1.NewTime(time.Now()) </del> // This handles Generation bump for resources that don't support graceful // deletion. For resources that support graceful deletion is handle in // pkg/api/rest/delete.go if objectMeta.GetDeletionTimestamp() == nil && objectMeta.GetGeneration() > 0 { objectMeta.SetGeneration(objectMeta.GetGeneration() + 1) } <del> objectMeta.SetDeletionTimestamp(&now) </del> <ins> existingDeletionTimestamp := objectMeta.GetDeletionTimestamp() if existingDeletionTimestamp == nil || existingDeletionTimestamp.After(now) { metaNow := metav1.NewTime(now) objectMeta.SetDeletionTimestamp(&metaNow) } </ins> var zero int64 = 0 objectMeta.SetDeletionGracePeriodSeconds(&zero) return nil

What happened: a custom resource defintion a namespace a custom resource in the namespace with a finalizer attached the namespace custom resource in the namespace will get its deletionTimestamp field updated about every 8 seconds. What you expected to happen: The resource would have the deletionTimestamp set once then not be updated again. This becomes a hassle if a controller needs to perform an operation that takes longer than the update period as it will need to perform a read-modify-write with retry loop when removing the finalizer. How to reproduce it (as minimally and precisely as possible): Environment: Kubernetes version (use ):Cloud provider or hardware configuration: GKE 1.11
/sig api-machinery
/area custom-resources deletionTimestamps should not update even if subsequent deletion requests are made
/assign
as this description: if object has pending finalizers, the deletionTimestamps should be updated. Do we need to change this logic ?
deletionTimestamp is supposed to be immutable: storage modifying that timestamp is unexpected, at least to me
we should at least ensure that this documentation is honored:
opened to do that


Question: </g> <g> <path style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" d="M 934.594 747.007 C 928,762 804,738 797.63,754.926"/> <del> <polygon style="fill: #000000" points="939.171,749.02 938.62,737.853 930.017,744.994 "/> <polygon style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" points="939.171,749.02 938.62,737.853 930.017,744.994 "/> </del> <ins> <polygon style="fill: #000000" points="939.171,749.02 938.619,737.853 930.017,744.994 "/> <polygon style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" points="939.171,749.02 938.619,737.853 930.017,744.994 "/> </ins> <polygon style="fill: #000000" points="794.988,761.945 793.831,750.825 797.63,754.926 803.19,754.347 "/> <polygon style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" points="794.988,761.945 793.831,750.825 797.63,754.926 803.19,754.347 "/> </g> <g> <path style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" d="M 974.333 745.616 C 998,776 1131,714 1154.77,755.585"/> <del> <polygon style="fill: #000000" points="978.277,742.543 968.188,737.727 970.388,748.689 "/> <polygon style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" points="978.277,742.543 968.188,737.727 970.388,748.689 "/> </del> <ins> <polygon style="fill: #000000" points="978.278,742.544 968.188,737.727 970.389,748.689 "/> <polygon style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" points="978.278,742.544 968.188,737.727 970.389,748.689 "/> </ins> <polygon style="fill: #000000" points="1158.49,762.097 1149.19,755.896 1154.77,755.585 1157.87,750.934 "/> <polygon style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" points="1158.49,762.097 1149.19,755.896 1154.77,755.585 1157.87,750.934 "/> </g>

Architecture picture: says kubecfg is deprecated. The architecture picture should be updated to show kubectl instead of kubecfg.
ref /cc


Question: Verb:     "get", Resource: schema.GroupVersionResource{Resource: "resource"}, } <del> c.Invokes(action, nil) </del> <ins> if _, err = c.Invokes(action, nil); err != nil { return resultGroups, c.Resources, err } </ins> return resultGroups, c.Resources, nil }

I'm writing a UT for the following function: The code branch where the server returns an error other than could not be tested using 's . This is because does not handle the error returned by and therefore could not return errors other than : Ref: Add a to so simulate errors other than . Run the following UT (should fail):
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig api-machinery
Same issue for some other methods of : -
I'll submit a PR shortly to address this issue. /assign


Question: } csiAttacher := attacher.(*csiAttacher) <del> go func(spec *volume.Spec, id, nodename string, fail bool) { </del> <ins> go func(spec *volume.Spec, nodename string, fail bool) { </ins> attachID, err := csiAttacher.Attach(spec, types.NodeName(nodename)) if fail != (err != nil) { t.Errorf("expecting no failure, but got err: %v", err) } <del> if attachID != id && !fail { t.Errorf("expecting attachID %v, got %v", id, attachID) </del> <ins> if attachID != "" { t.Errorf("expecting empty attachID, got %v", attachID) </ins> } <del> }(tc.spec, tc.attachID, tc.nodeName, tc.shouldFail) </del> <ins> }(tc.spec, tc.nodeName, tc.shouldFail) </ins> var status storage.VolumeAttachmentStatus if tc.injectAttacherError {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: Followup to In 1.15, we can merge this commit: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
/assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


Question: Names: []string{"/k8s_POD_foo.new.test_.deadbeef_42"}, ID:    "1876", }, <ins> }, }, // maxContainerCount applies per container.. { containers: []docker.APIContainers{ </ins> { // pod infra container <del> Names: []string{"/k8s_POD_foo.new.test_.deadbeef_42"}, ID:    "2876", </del> <ins> Names: []string{"/k8s_POD_foo2.new.test_.beefbeef_40"}, ID:    "1706", </ins> }, { // pod infra container <del> Names: []string{"/k8s_POD_foo.new.test_.deadbeef_42"}, ID:    "3876", </del> <ins> Names: []string{"/k8s_POD_foo2.new.test_.beefbeef_40"}, ID:    "2706", </ins> }, { // pod infra container <del> Names: []string{"/k8s_POD_foo.new.test_.deadbeef_42"}, ID:    "4876", </del> <ins> Names: []string{"/k8s_POD_foo2.new.test_.beefbeef_40"}, ID:    "3706", </ins> }, { // pod infra container Names: []string{"/k8s_POD_foo.new.test_.deadbeef_42"}, <del> ID:    "5876", </del> <ins> ID:    "1876", </ins> }, { // pod infra container Names: []string{"/k8s_POD_foo.new.test_.deadbeef_42"}, <del> ID:    "6876", </del> <ins> ID:    "2876", </ins> }, { // pod infra container Names: []string{"/k8s_POD_foo.new.test_.deadbeef_42"}, <del> ID:    "7876", </del> <ins> ID:    "3876", </ins> }, }, containerDetails: map[string]*docker.Container{ <del> "1876": { </del> <ins> "1706": { </ins> State: docker.State{ <del> Running: true, </del> <ins> Running: false, </ins> }, <del> ID:      "1876", </del> <ins> ID:      "1706", </ins> Created: time.Now(), }, <del> "2876": { </del> <ins> "1876": { </ins> State: docker.State{ Running: false, }, <del> ID:      "2876", </del> <ins> ID:      "1876", </ins> Created: time.Now(), }, }, <del> expectedRemoved: []string{"2876"}, }, { containers: []docker.APIContainers{ { // pod infra container Names: []string{"/k8s_POD_foo.new.test_.deadbeef_42"}, ID:    "1876", }, }, </del> <ins> expectedRemoved: []string{"1706", "1876"}, </ins> }, } for _, test := range tests { kubelet, fakeDocker := newTestKubelet(t) <del> kubelet.maxContainerCount = 5 </del> <ins> kubelet.maxContainerCount = 2 </ins> fakeDocker.ContainerList = test.containers fakeDocker.ContainerMap = test.containerDetails fakeDocker.Container = &docker.Container{ID: "error", Created: time.Now()}

Successfully ran containers with "onFailure" restartPolicy are restarting after purging oldest docker containers. Kubelet purges/removes oldest containers periodically from docker. After purging, we loose the exit code and history of previously ran containers and restart containers as if they were never run.
Thanks & I can help fix this. Let me know. Thanks
Thanks for reporting Just to make sure I understand the issue: You ran some containers with an "onFailure" restartPolicy. When the Kubelet purged old containers the container was restarted (unexpectedly).
Ah, after reading again I think the issue is that after the containers are purged, when the container exits successfully it get restarted although it shouldn't. Is that correct?
After reading through the code I found a bug in the implementation that is most likely causing this error. Should have a PR soon.
Yes. Correct. After initial container run, it should never be run again, since it already ran to completion. Will wait for your PR.
PR is out. This should fix the issue. Feel free to test if you can :)


Question: github.com/mitchellh/go-homedir => github.com/mitchellh/go-homedir v1.1.0 github.com/mitchellh/go-wordwrap => github.com/mitchellh/go-wordwrap v1.0.0 github.com/mitchellh/mapstructure => github.com/mitchellh/mapstructure v1.1.2 <del> github.com/moby/ipvs => github.com/moby/ipvs v1.0.0 </del> <ins> github.com/moby/ipvs => github.com/moby/ipvs v1.0.1 </ins> github.com/moby/term => github.com/moby/term v0.0.0-20200312100748-672ec06f55cd github.com/modern-go/concurrent => github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd github.com/modern-go/reflect2 => github.com/modern-go/reflect2 v1.0.1

Hi, Since my upgrade to 1.18 version, I have errors in kube-proxy: I have ipv4/ipv6 dualstack enable. No problem with cluster and IPVS works despite errors. Do you have also this issue ? OS: RHEL7 Kubernetes version (use ):
/sig network
Related to cc
/cc
Taking one of the outputs it gives me the following addresses? can you share more details about your environment, Is it possible that the CoreDNS pod has the address ? what are your cluster-cidr and service-cidr? Indeed that new net.Utils change looks suspicious , a quick grep in the repo gives me some suspects, maybe the service/ipallocator? It's important to mention there were similar issues with IPVS before, interpreting IPv4 addresses as IPv6, has some experience with this kind of issues, maybe he can see something:) /cc
A bit tricky to trace since libnetwork/ipvs is moving around at the moment. Problem seems to be in; The "family" must be wrong. It is called from;
Please do a and attach the output.
Hi, Some informations: cluster-cidr and service-cidr: fd00:10:245:0:0:0:0:0/64,10.245.0.0/16 fd00:10:96:0:0:0:0:0/112,10.96.0.0/12 ipvsadm output:
Ref:
The commit in the ref above should be in v1.18.0 which is the version used in this issue. So the commit does not fix the problem.
Maybe something wrong around here Before , it looks like this: Now it looks like this: However is not initializedit is zero :(.
I do not get these faults. Has the netlink API changed? What kernel version are you using?
d.AddressFamily is supposed to be set from the "NetlinkRouteAttr" slice; It should definitely not be hard-coded to ipv4.
Made a trying to reproduce this in my environment but didn't got this error.


Question: github.com/mitchellh/go-homedir => github.com/mitchellh/go-homedir v1.1.0 github.com/mitchellh/go-wordwrap => github.com/mitchellh/go-wordwrap v1.0.0 github.com/mitchellh/mapstructure => github.com/mitchellh/mapstructure v1.1.2 <del> github.com/moby/ipvs => github.com/moby/ipvs v1.0.0 </del> <ins> github.com/moby/ipvs => github.com/moby/ipvs v1.0.1 </ins> github.com/moby/term => github.com/moby/term v0.0.0-20200312100748-672ec06f55cd github.com/modern-go/concurrent => github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd github.com/modern-go/reflect2 => github.com/modern-go/reflect2 v1.0.1

I may try this on some older kernel like from CentOS 7 Added exactly the same IPVS configuration here: And when running this 'gist/program' pointing to the IPVS from the DNS (fd00:10:96::a port 53 UDP) got the expected result: Maybe trying to run this also in some other scenarios could clarify if this is something related to Netlink API changed between Kernels, something with the used library, etc.
After some study, I find that this issue most likely is caused by the low linux kernel version. I have reproduced this problem on . Accroding to the code, netlink will try to get attribute, but I find the kernel just does not support it :( This is the defined in (kernel 3.10) No attribute is defined! But in new kernel version, the is defined like : Obviously, the kernel has some attributes (IPVSDESTATTRADDRFAMILY, IPVSDESTATTRSTATS64...). That is why kube-proxy works well on systems with a higher version of the kernel. So we can address this issue by upgrading our linux kernel. Not sure which kernel version is the minimum requirement, maybe we should document something about this.
Here is the defined in :
Good finding. So, we can conclude that this really is kernel version dependent. It is hard to know what to do about it. We must check the family. I would propose that we make it work for ipv4 with some fallback and put a requirement on kernel version for ipv6-only/dual-stack. But the decision is others to make. /cc
/cc
The symbol in the kernel tree The for service family seems very old. Since the service must have the same family as the real-servers perhaps it can be used for compatibility with old kernels?
The commit that adds the symbol; Seem to go into linux-3.18
I did not know the minimum required for the kernel version, I was using the stable and official version of RHEL7. My kernel version is: Currently the max version in the repository is: According to the documentation (), I should switch to RHEL8 if I want a recent kernel :-
There is no minimum kernel version requirement afaik. So I think this must be fixed for ipv4-only clusters IMO.


Question: github.com/mitchellh/go-homedir => github.com/mitchellh/go-homedir v1.1.0 github.com/mitchellh/go-wordwrap => github.com/mitchellh/go-wordwrap v1.0.0 github.com/mitchellh/mapstructure => github.com/mitchellh/mapstructure v1.1.2 <del> github.com/moby/ipvs => github.com/moby/ipvs v1.0.0 </del> <ins> github.com/moby/ipvs => github.com/moby/ipvs v1.0.1 </ins> github.com/moby/term => github.com/moby/term v0.0.0-20200312100748-672ec06f55cd github.com/modern-go/concurrent => github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd github.com/modern-go/reflect2 => github.com/modern-go/reflect2 v1.0.1

However ipv6-only and dual-stack are in "alpha" so a requirement on kernel version might be ok, but I can't say myself. A complication is that the problem is not in k8s itself but in a 3rd party lib.
ipv6-only is beta since 1.18 :smile:
/assign
I opened a PR to fix this in moby/ipvs If someone w/ a cluster on the older kernel version can test it that would be great, otherwise I will setup a cluster to validate this some time this week.
Instead of bootstraping a whole cluster right now (pretty busy here) I can use the Gist I've mentioned to check in a CentOS 7.1 if it faces the problem before and after the patch. At least to have some more test right now
with a small test, it worked fine. Environment: CentOS 7.1 - Kernel 3.10.0-229.el7.x86_64 + ipvsadm 1.2.1 Pre-loaded some IPv6 rules into ipvs: I've used code, searching for the following structure: I then call ` to get the IPVS object, and then try to fetch the destinations. The following are the results: Before patching the ipvs module: After patching the ipvs module (pinning to your repo/branch) Also I've made those tests with the structure but searching for TCP, and for kube-apiserver (that have only one destination) and it worked fine in all those scenarios. If wanted I can improve the gist to pass the parameters as command arguments instead of hardcoded (sorry, did this in a rush to test)
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/remove-triage unresolved
Thanks just waiting on now
same issue,sloved by update kernel to 5.X
I didn't want to wait, I updated to kernel-lt in the version 4.4 . The problem is solved on my side :)
Thanks for confirming. We'll still try to get in since I'm sure other folks with older kernels will run into this issue.
/retitle parseIP error with IPVS proxy
upgrade your system kernel to 4.xx edition
FYI: just merged, need to run some validation & cut a new release (v1.0.1).


Question: github.com/mitchellh/go-homedir => github.com/mitchellh/go-homedir v1.1.0 github.com/mitchellh/go-wordwrap => github.com/mitchellh/go-wordwrap v1.0.0 github.com/mitchellh/mapstructure => github.com/mitchellh/mapstructure v1.1.2 <del> github.com/moby/ipvs => github.com/moby/ipvs v1.0.0 </del> <ins> github.com/moby/ipvs => github.com/moby/ipvs v1.0.1 </ins> github.com/moby/term => github.com/moby/term v0.0.0-20200312100748-672ec06f55cd github.com/modern-go/concurrent => github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd github.com/modern-go/reflect2 => github.com/modern-go/reflect2 v1.0.1

FYI
/retitle DualStack / IPv6-only: parseIP error with IPVS proxy on CentOS 7
v1.18 cherry-pick
Hi, it seems that you're using v1.18.0 and this has been corrected in v1.18.3, can you please update and check? Tks


Question: calls:        []string{"inspect_image", "pull"}, err:          true, }, <del> "should pull private image using dockerauth if image doesn't exist": { injectImage:  true, imgNeedsAuth: true, injectErr:    libdocker.ImageNotFoundError{ID: "image_id"}, calls:        []string{"inspect_image", "pull"}, configJSON:   authJSON, err:          false, }, </del> } { t.Logf("TestCase: %q", desc) _, fakeDocker, _ := newTestDockerService()

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: We have a private registry configured (where all images including the sandbox one), are hosted. The registry credentials are being pulled by one of k8s credentials providers () With Kubernetes 1.7 sandbox image failed to be pulled with the following error message: Looks like with this only creds from docker config are being evaluated on the sandbox image pull What you expected to happen: Sandbox image pull should be attempted using credentials returned by not just the basic docker credential provider, but other credentials providers registered in the system. How to reproduce it (as minimally and precisely as possible): 1) Register private repo credentials with one of the k8s credential providers (we've tried with rancher). Configure the to use the image from the the private repo. 2) Make sure you are not logged in to this private repo on the kubelet host 3) Try to create a pod. Sandbox image pull will fail with an error mentioned above. Anything else we need to know?: Non-sandbox images from the same private repo were able to download correctly using private repo credentials provider. Environment: Kubernetes version (use ): 1.7.2 Cloud provider or hardware configuration**: OS (e.g. from /etc/os-release): Ubuntu 16.04 Kernel (e.g. ): Install tools: Others:
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig node Just tested the same use case against k8s 1.5.x, it worked fine there and sandbox image was pulled using the credentials passed by the credential provider, not from docker config file. So it might have regressed in 1.7.x (haven't tested it on 1.6, so can't tell for sure)
Confirmed a bug introduced in dockershim. Will file a PR to fix this.


Question: calls:        []string{"inspect_image", "pull"}, err:          true, }, <del> "should pull private image using dockerauth if image doesn't exist": { injectImage:  true, imgNeedsAuth: true, injectErr:    libdocker.ImageNotFoundError{ID: "image_id"}, calls:        []string{"inspect_image", "pull"}, configJSON:   authJSON, err:          false, }, </del> } { t.Logf("TestCase: %q", desc) _, fakeDocker, _ := newTestDockerService()

Could you also verify whether fixed your problem?
tested against your commits cherry-picked on top of 1.8 beta, the fix seems to work
thanks.
[MILESTONENOTIFIER] Milestone Labels Complete Issue label settings: sig/node: Issue will be escalated to these SIGs if needed. priority/important-soon: Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. kind/bug: Fixes a bug discovered during the current release. <detailsAdditional instructions available <a href=""here</aThe commands available for adding these labels are documented <a href=""here</a</details


Question: st.MakePod().Name("p1").UID("p1").Node("node1").Priority(midPriority).Obj(), st.MakePod().Name("p2").UID("p2").Node("node2").Priority(midPriority).Obj(), }, <del> expected: [][]Candidate{ { &candidate{victims: &extenderv1.Victims{}, name: "node1"}, }, }, expectedNumFilterCalled: []int32{3}, </del> <ins> expected:                [][]Candidate{{}}, fakeFilterRC:            framework.Unschedulable, expectedNumFilterCalled: []int32{2}, </ins> }, { name: "a pod that fits on both nodes when lower priority pods are preempted",

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --Kubernetes 1.19 (confirmed with 1.19.7 and 1.19.10) kube-scheduler crashes and restarts with the following errors in . panic: runtime error: index out of range [0] with length 0 The problem code is line 389. When is nil or is empty, the error happens. If we skip when it's nil or empty, the error is gone. panic: runtime error: invalid memory address or nil poster dereference The problem is does not exist and returns nil sometimes. Simply skipping it won't solve the problem. The scheduler will reach either line 456 or 341. The scheduler works without failures. When there are preemptions in a cluster. Kubernetes version (use ): v1.19.10 or v1.19.7 Cloud provider or hardware configuration: bare metal OS (e.g: ): centos 7 Kernel (e.g. ): Linux 5.4.77-7.el7pie SMP Sat Nov 21 01:16:27 UTC 2020 x8664 x8664 x8664 GNU/Linux Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scheduling
/cc
/assign
Are you running the vanilla scheduler? or a customized one that leverages the utilities in
It's a custom scheduler with some internal out-of-tree plugins (just like ), but it doesn't have any custom postFilter plugins. Which utility functions are you referring to? Thanks.
are those custom plugins filter plugins? do they maintain state?


Question: st.MakePod().Name("p1").UID("p1").Node("node1").Priority(midPriority).Obj(), st.MakePod().Name("p2").UID("p2").Node("node2").Priority(midPriority).Obj(), }, <del> expected: [][]Candidate{ { &candidate{victims: &extenderv1.Victims{}, name: "node1"}, }, }, expectedNumFilterCalled: []int32{3}, </del> <ins> expected:                [][]Candidate{{}}, fakeFilterRC:            framework.Unschedulable, expectedNumFilterCalled: []int32{2}, </ins> }, { name: "a pod that fits on both nodes when lower priority pods are preempted",

If yes, then one hypothesis is that those custom filters filter the node in the filter phase, but not when executed in the preemption phase, and so this will result in a candidate node with no victims?
Yes, it includes filter plugins and uses cycle state.
you need to make sure that those filter plugins play nicely when executed again in the preemption phase in the same cycle: i.e., produce the same result.
Thanks What do you mean by performing filter in the preemption phase? would you mind elaborating it a little?
What additional logic is needed to handle the preemption case?
in the preemption phase we run the filters again to check if removing lower priority pods make the node schedulable: we later add them one by one to reduce the set of victim pods to the absolute minimum: My hypothesis is that somehow your filter returns success when adding all the pods back, and so you end up with zero victims and a candidate node. This shouldn't happen because if the pod fits the node without removing any victims, then we shouldn't be running preemption in the first place. So the theory is that the custom filter is returning false when run in the filter phase, but returns true when executed in the preemption phase perhaps because it makes some assumptions about cyclestate that makes it behave this way (e.g. that a the filter will be executed once per node in a scheduling cycle).
Disabled the filter plugin, but still see the same issues.
are you able to change the scheduler code and run the test again? I can send a patch tomorrow to add some debugging messages to help us root cause the issue.
yes, thanks a lot! really appreciate it!
You are right. We still use a deprecated Predicate extender, which causes the problem. After disabling it, it's working fine. It used to work with 1.18 though. We are retiring it. Thank you so much!!!
If the test cases can help debug scheduler extender Webhook (Predicate) too, I'd like to try it. Thanks again!
How can we improve the preemption logic to prevent bugs and issues in custom Filters or Predicates crashing the scheduler?
for starter, we should check the list len, the scheduler should not panic in all cases.


Question: st.MakePod().Name("p1").UID("p1").Node("node1").Priority(midPriority).Obj(), st.MakePod().Name("p2").UID("p2").Node("node2").Priority(midPriority).Obj(), }, <del> expected: [][]Candidate{ { &candidate{victims: &extenderv1.Victims{}, name: "node1"}, }, }, expectedNumFilterCalled: []int32{3}, </del> <ins> expected:                [][]Candidate{{}}, fakeFilterRC:            framework.Unschedulable, expectedNumFilterCalled: []int32{2}, </ins> }, { name: "a pod that fits on both nodes when lower priority pods are preempted",

Other than that, I think we need to clearly document what the preemption logic does (that it executes the prefilter extensions points and the filter plugins multiple times) and the expectations from filter plugins (that they may get executed more than once in the same cycle)
Is it a predicate extender or preemption extender? If it's a predicate extender, I don't quite think we found the root cause - b/c predicate extenders are not invoked during preemption (). It makes more sense if it's a preemption extender - if the preemption extender mutates the to make it invalid. If it's the case, could you verify it by adding debug info before and after L157 to check the validness of ?
That's always true :)
filed a PR to mitigate it.
It's a predicate extender. I'll debug it more.
Added some debug info. after and . Here's the result. returned 4 candidates, but one of candidates () has zero pod. The candidate happened to be the best candidate and hence caused the "" error. It should have nothing to do with since our extender doesn't support Preemption. Disabling it may change pod placements/candidates selection and bypassed the error. It's like a coincidence. To remove the panic, a simple fix is to check if is empty. If it is, do not add the node to candidates. Alternatively, a change to may work too. Here is the log with debug info.
This looks more promising. Back to digging into the root cause, it seems returns an empty but with . So that implies firstly we have a non-nil , and then in , the is separated into two list: violatingVictims and nonViolatingVictims. Finally the two victims list are tried one by one to run . It looks like returns {fits=true, nil error} for every victim, which is quite abnormal. I think it's problematic that every pod is retrievable and in which we can debug into.
Yes, should not return fit=true with empty pods in the first place, but it's hard to locate the problematic code in this function.
The following code in looks questionable to me. If is empty, should the function return nil, nil? Also, if is true for all extenders, the loop won't do anything and candidates will remain the same.


Question: st.MakePod().Name("p1").UID("p1").Node("node1").Priority(midPriority).Obj(), st.MakePod().Name("p2").UID("p2").Node("node2").Priority(midPriority).Obj(), }, <del> expected: [][]Candidate{ { &candidate{victims: &extenderv1.Victims{}, name: "node1"}, }, }, expectedNumFilterCalled: []int32{3}, </del> <ins> expected:                [][]Candidate{{}}, fakeFilterRC:            framework.Unschedulable, expectedNumFilterCalled: []int32{2}, </ins> }, { name: "a pod that fits on both nodes when lower priority pods are preempted",

Why do we bother creating a new candidates later? Can it just return candidates?
It can prevent from returning candidates with empty pods, but what about ? It's possible to add nodes without victim pods to candidates if returned by contain such nodes. How about adding the following check to CallExtender?
Updated the PR based on the finding and discussions.
Yes, we can use a flag to mark if all extenders don't support preemption. If yes, simply return the candidates.
Technically this can exist due to faulty extender implementation. If we really want to guard it, instead of logging the error and continue, I'm more inclined to return the error immediately as this is a fatal error - the victimsMap cannot be used either for latter extender or further preemptor nominating.
Is my understanding correct? In , if victimsMap is empty, should it return instead of ?
The only case victimsMap can be empty is empty candidates, and so return either doesn't quite matter, right?
You are right.
As long as one extender returns an invalid victimsMap (with empty pods), s returns an error immediately? What about ? Should it return an error immediately if it returnswith empty pods?
I think so. Because the result it returned is faulty, and we don't want to continue based on it The same. We may don't want to continue the scheduling cycle based on faulty result.


Question: "//staging/src/k8s.io/client-go/kubernetes/fake:go_default_library", "//staging/src/k8s.io/client-go/testing:go_default_library", "//staging/src/k8s.io/client-go/tools/clientcmd:go_default_library", <del> "//staging/src/k8s.io/cluster-bootstrap/token/api:go_default_library", </del> "//vendor/github.com/ghodss/yaml:go_default_library", "//vendor/github.com/pkg/errors:go_default_library", "//vendor/github.com/renstrom/dedent:go_default_library",

Just got this while running the tests:
let's please move this to /close /sig cluster-lifecycle


Question: return &d, nil } <ins> // getIPFamily parses the IP family based on raw data from netlink. // For AF_INET, netlink will set the first 4 bytes with trailing zeros //   10.0.0.1 -> [10 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0] // For AF_INET6, the full 16 byte array is used: //   2001:db8:3c4d:15::1a00 -> [32 1 13 184 60 77 0 21 0 0 0 0 0 0 26 0] func getIPFamily(address []byte) (uint16, error) { if len(address) == 4 { return syscall.AF_INET, nil } if isZeros(address) { return 0, errors.New("could not parse IP family from address data") } // assume IPv4 if first 4 bytes are non-zero but rest of the data is trailing zeros if !isZeros(address[:4]) && isZeros(address[4:]) { return syscall.AF_INET, nil } return syscall.AF_INET6, nil } func isZeros(b []byte) bool { for i := 0; i < len(b); i++ { if b[i] != 0 { return false } } return true } </ins> // parseDestination given a ipvs netlink response this function will respond with a valid destination entry, an error otherwise func (i *Handle) parseDestination(msg []byte) (*Destination, error) { var dst *Destination


Question: return &d, nil } <ins> // getIPFamily parses the IP family based on raw data from netlink. // For AF_INET, netlink will set the first 4 bytes with trailing zeros //   10.0.0.1 -> [10 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0] // For AF_INET6, the full 16 byte array is used: //   2001:db8:3c4d:15::1a00 -> [32 1 13 184 60 77 0 21 0 0 0 0 0 0 26 0] func getIPFamily(address []byte) (uint16, error) { if len(address) == 4 { return syscall.AF_INET, nil } if isZeros(address) { return 0, errors.New("could not parse IP family from address data") } // assume IPv4 if first 4 bytes are non-zero but rest of the data is trailing zeros if !isZeros(address[:4]) && isZeros(address[4:]) { return syscall.AF_INET, nil } return syscall.AF_INET6, nil } func isZeros(b []byte) bool { for i := 0; i < len(b); i++ { if b[i] != 0 { return false } } return true } </ins> // parseDestination given a ipvs netlink response this function will respond with a valid destination entry, an error otherwise func (i *Handle) parseDestination(msg []byte) (*Destination, error) { var dst *Destination


Question: return &d, nil } <ins> // getIPFamily parses the IP family based on raw data from netlink. // For AF_INET, netlink will set the first 4 bytes with trailing zeros //   10.0.0.1 -> [10 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0] // For AF_INET6, the full 16 byte array is used: //   2001:db8:3c4d:15::1a00 -> [32 1 13 184 60 77 0 21 0 0 0 0 0 0 26 0] func getIPFamily(address []byte) (uint16, error) { if len(address) == 4 { return syscall.AF_INET, nil } if isZeros(address) { return 0, errors.New("could not parse IP family from address data") } // assume IPv4 if first 4 bytes are non-zero but rest of the data is trailing zeros if !isZeros(address[:4]) && isZeros(address[4:]) { return syscall.AF_INET, nil } return syscall.AF_INET6, nil } func isZeros(b []byte) bool { for i := 0; i < len(b); i++ { if b[i] != 0 { return false } } return true } </ins> // parseDestination given a ipvs netlink response this function will respond with a valid destination entry, an error otherwise func (i *Handle) parseDestination(msg []byte) (*Destination, error) { var dst *Destination


Question: return &d, nil } <ins> // getIPFamily parses the IP family based on raw data from netlink. // For AF_INET, netlink will set the first 4 bytes with trailing zeros //   10.0.0.1 -> [10 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0] // For AF_INET6, the full 16 byte array is used: //   2001:db8:3c4d:15::1a00 -> [32 1 13 184 60 77 0 21 0 0 0 0 0 0 26 0] func getIPFamily(address []byte) (uint16, error) { if len(address) == 4 { return syscall.AF_INET, nil } if isZeros(address) { return 0, errors.New("could not parse IP family from address data") } // assume IPv4 if first 4 bytes are non-zero but rest of the data is trailing zeros if !isZeros(address[:4]) && isZeros(address[4:]) { return syscall.AF_INET, nil } return syscall.AF_INET6, nil } func isZeros(b []byte) bool { for i := 0; i < len(b); i++ { if b[i] != 0 { return false } } return true } </ins> // parseDestination given a ipvs netlink response this function will respond with a valid destination entry, an error otherwise func (i *Handle) parseDestination(msg []byte) (*Destination, error) { var dst *Destination


Question: } } <ins> func TestFakeTimer(t *testing.T) { tc := NewFakeClock(time.Now()) if tc.HasWaiters() { t.Errorf("unexpected waiter?") } oneSec := tc.NewTimer(time.Second) twoSec := tc.NewTimer(time.Second * 2) treSec := tc.NewTimer(time.Second * 3) if !tc.HasWaiters() { t.Errorf("unexpected lack of waiter?") } select { case <-oneSec.C(): t.Errorf("unexpected channel read") case <-twoSec.C(): t.Errorf("unexpected channel read") case <-treSec.C(): t.Errorf("unexpected channel read") default: } tc.Step(999999999 * time.Nanosecond) // t=.999,999,999 select { case <-oneSec.C(): t.Errorf("unexpected channel read") case <-twoSec.C(): t.Errorf("unexpected channel read") case <-treSec.C(): t.Errorf("unexpected channel read") default: } tc.Step(time.Nanosecond) // t=1 select { case <-twoSec.C(): t.Errorf("unexpected channel read") case <-treSec.C(): t.Errorf("unexpected channel read") default: } select { case <-oneSec.C(): // Expected! default: t.Errorf("unexpected channel non-read") } tc.Step(time.Nanosecond) // t=1.000,000,001 select { case <-oneSec.C(): t.Errorf("unexpected channel read") case <-twoSec.C(): t.Errorf("unexpected channel read") case <-treSec.C(): t.Errorf("unexpected channel read") default: } if oneSec.Stop() { t.Errorf("Expected oneSec.Stop() to return false") } if !twoSec.Stop() { t.Errorf("Expected twoSec.Stop() to return true") } tc.Step(time.Second) // t=2.000,000,001 select { case <-oneSec.C(): t.Errorf("unexpected channel read") case <-twoSec.C(): t.Errorf("unexpected channel read") case <-treSec.C(): t.Errorf("unexpected channel read") default: } if twoSec.Reset(time.Second) { t.Errorf("Expected twoSec.Reset() to return false") } if !treSec.Reset(time.Second) { t.Errorf("Expected treSec.Reset() to return true") } tc.Step(time.Nanosecond * 999999999) // t=3.0 select { case <-oneSec.C(): t.Errorf("unexpected channel read") case <-twoSec.C(): t.Errorf("unexpected channel read") case <-treSec.C(): t.Errorf("unexpected channel read") default: } tc.Step(time.Nanosecond) // t=3.000,000,001 select { case <-oneSec.C(): t.Errorf("unexpected channel read") case <-twoSec.C(): t.Errorf("unexpected channel read") default: } select { case <-treSec.C(): // Expected! default: t.Errorf("unexpected channel non-read") } } </ins> func TestFakeTick(t *testing.T) { tc := NewFakeClock(time.Now()) if tc.HasWaiters() {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: I was writing a variant of clock.FakeClock, and noticed a couple of bugs in . Look at the pointer comparison in --- the two pointers will never be equal, but the author clearly expected equality when is the index of the fakeClockWaiter corresponding to the fakeTimer. The two pointers will never be equal because one is a pointer into the fakeClock's slice while the other pointer is to a field of the . The consequence of this bug is that the Stop function is ineffective. Also, determines the returned value based on the original copy of the waiter, which --- being a copy --- did not get updated if/when the waiter fired. Note also that does not exercise FakeClock::NewTimer at all. What you expected to happen: I expected FakeClock Timers to work properly. How to reproduce it (as minimally and precisely as possible): Use the TestFakeTimer func introduced in Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
The fake timer's Reset() method also does not work correctly.


Question: stopCh := make(chan struct{}) fakeClientset := fake.NewSimpleClientset(&v1.PodList{Items: item.pods}) holder := &podHolder{} <del> controller := NewNoExecuteTaintManager(fakeClientset, holder.getPod, (&nodeHolder{item.newNode}).getNode, getPodsAssignedToNode(fakeClientset)) </del> <ins> controller := NewNoExecuteTaintManager(fakeClientset, holder.getPod, (&nodeHolder{node: item.newNode}).getNode, getPodsAssignedToNode(fakeClientset)) </ins> controller.recorder = testutil.NewFakeRecorder() go controller.Run(stopCh)

What happened: A Pod was scheduled for deletion because a taint w/ effect was observed in the Node the Pod was assigned to, and after the taint is removed from the Node, the Pod scheduled deletion isn't canceled. After scheduling pod deletion, the only cancels pod deletion if all taints are removed from the node, including any taints that the user relies on for their use-cases (example below). What you expected to happen: After the taint that triggered scheduling pod deletion is removed, cancels the scheduled pod deletion. How to reproduce it (as minimally and precisely as possible): A node is tainted w/ , so any workloads already assigned to it which don't tolerate this taint are evicted. A pod is created w/ tolerations: w/ unspecified and w/ . is assigned to . At some point in time after the previous steps took place, is tainted as . gets an update event for and observes two taints. It proceeds to calculate the minimum time (in seconds) that tolerates for the two taints which at this point is 300 seconds, and marks for deletion in ~300 seconds - this happens in-memory, in a timed-queue. At some point in time after the previous steps took place, and before the 300 seconds trigger happen, taint is removed from . gets an update event for and observes only one taint, . It proceeds to calculate the minimum time (in seconds) that tolerates for the this taint which is infinity, and returns, not canceling the previous deletion. It completely ignores the fact that the taint that triggered deletion is no longer observed. Anything else we need to know?: Worked w/ to figure this out after all pods in a node got evicted without apparent reason. Environment: Kubernetes version (use ): 1.17.3 (but looking at the code seems to affect all of 1.17.x and 1.18.x and master. Cloud provider or hardware configuration: N/A OS (e.g: ): N/A Kernel (e.g. ): 5.x Install tools: N/A Network plugin and version (if this is a network-related bug): N/A Others: N/A cc (owners of this code)
There seems to be at least .


Question: stopCh := make(chan struct{}) fakeClientset := fake.NewSimpleClientset(&v1.PodList{Items: item.pods}) holder := &podHolder{} <del> controller := NewNoExecuteTaintManager(fakeClientset, holder.getPod, (&nodeHolder{item.newNode}).getNode, getPodsAssignedToNode(fakeClientset)) </del> <ins> controller := NewNoExecuteTaintManager(fakeClientset, holder.getPod, (&nodeHolder{node: item.newNode}).getNode, getPodsAssignedToNode(fakeClientset)) </ins> controller.recorder = testutil.NewFakeRecorder() go controller.Run(stopCh)

You can sort of get a feel for what's going on here by looking at these two code snippets: This is what happens in when the node does not have any NoExecute taints: If you have any NoExecute taints on the node processPodOnNode gets called and this routine is really, really eager to evict pods. So : We are currently working around this issue by changing over our NoExecute taints to NoSchedule.
/sig scheduling node
/cc
/remove-sig scheduling scheduler isn't involved in this case.
Which SIG owns the controller-manager or is the node lifecycle manager (and its NoExecuteTaintManager) owned by sig-node alone?
I think node lifecycle is jointly owned by sig-node and sig-cloud-provider
API machinery owns the mechanics of the controller manager (controller loop setup and management), but the individual controllers are SIG-specific
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Looks like we lost some momentum here and I'd like to avoid this falling into the abyss of bugs that never get fixed. When this bug is triggered the outcome is pretty terrible and once it does happen it is very difficult to dig into and understand what actually happened. Did we ever get this routed to the right owner?
adding sig-node leads for triage/routing. this seems like a clearly reproducible bug with significant correctness issues. /assign
thanks for the detailed report and reproducer. to help accelerate this, do you have a form of your reproducer we could add in the e2e suite (even if its failing initially)?
custom taints are considered disruptive, so it couldn't run in the main suite
It's been a while since I wrote tests for the e2e suite but if you are available over slack in case guidance is needed, I'm willing to put the effort.
Do all suites need to pass for a fix to be merged? I'd assume it is the case, otherwise how would a regression be prevented after the fix lands?


Question: stopCh := make(chan struct{}) fakeClientset := fake.NewSimpleClientset(&v1.PodList{Items: item.pods}) holder := &podHolder{} <del> controller := NewNoExecuteTaintManager(fakeClientset, holder.getPod, (&nodeHolder{item.newNode}).getNode, getPodsAssignedToNode(fakeClientset)) </del> <ins> controller := NewNoExecuteTaintManager(fakeClientset, holder.getPod, (&nodeHolder{node: item.newNode}).getNode, getPodsAssignedToNode(fakeClientset)) </ins> controller.recorder = testutil.NewFakeRecorder() go controller.Run(stopCh)

i was more inquiring if there was a test that could automate the reproduction, so even just a linked commit may be helpful for someone that wants to pick this up. i am happy to help if you want to reach out over slack later this week.
looks like a unit test could reproduce the issue, opened a WIP fix in
our code is not publicly accessible, I'm sorry. PR linked by seems to cover both the fix and the test that proves it.
/remove-lifecycle frozen
frozen keeps the bot from auto-closing
Ah, sorry then.


Question: return "" } <ins> func (m *LinuxContainerSecurityContext) GetSeccompProfilePath() string { if m != nil { return m.SeccompProfilePath } return "" } </ins> // LinuxContainerConfig contains platform-specific configuration for // Linux-based containers. type LinuxContainerConfig struct {

CRI supports AppArmor, Seccomp and Sysctls via annotations , we should move them to typed and structured fields instead. (refer ) [x] Apparmor () [x] Sysctls () [x] Seccomp () cc
/cc
/sig node
Seccomp is addressed at .


Question: return "" } <ins> func (m *LinuxContainerSecurityContext) GetSeccompProfilePath() string { if m != nil { return m.SeccompProfilePath } return "" } </ins> // LinuxContainerConfig contains platform-specific configuration for // Linux-based containers. type LinuxContainerConfig struct {

Seccomp profile is passed to runtime by in CRI: is relative to node's local seccomp profile root, it is defined in kubelet. But runtime doesn't know it. We should pass full profile path in CRI. cc/ Will file a PR to fix this later.
Add support for Seccomp sounds like a new feature. The milestone should be reset to , correct?
not really. The current implementation supports seccomp already. If it's missing critical information to function properly, this would be a bug.
are we blocking releases for CRI in 1.5? i was not sure, so marking this as non-release-blocker and leaving 1.5 milestone. please correct if i am mistaken.
I don't get why this is a 1.5 targetted thing myself. Yes we want CRI to be in a good state in 1.5, but afaik we have zero intention of anyone actually using (at the very least because of the lack of exec auth regressing kubelet security). why should this be part of 1.5?
First, this is a small enough change that allows non-docker integrations to properly support seccomp. Second, if "we have zero intention of anyone actually using" is a valid reason, we should never fix any bugs in CRI. We do want people to try using it (perhaps not in production yet). However, given that seccomp is only in alpha, I am ok with dropping this and just continue the docker support for now.
Perhaps I misunderstood the meaning of code freeze, but I thought it was only bug-fixes regardless of size of change (or exceptions that have gone through some process) What I meant is we don't intend this to be used generally as part of the 1.5 release. Because of that, I don't think we should consider bugs in CRI bugs for the purpose of the 1.5 release freeze. My statement is just that we shouldn't violate code-freeze for this. We should totally fix this right after the 1.5 freeze window ends, and saying that I'm arguing we should never fix anything is disingenuous and taking it to an absurd extreme.
the only thing we don't seem to agree on is whether seccomp support was considered to be supported for CRI in 1.5.


Question: return "" } <ins> func (m *LinuxContainerSecurityContext) GetSeccompProfilePath() string { if m != nil { return m.SeccompProfilePath } return "" } </ins> // LinuxContainerConfig contains platform-specific configuration for // Linux-based containers. type LinuxContainerConfig struct {

My interpretation is that we assumed seccomp would be supported until we found out that it wouldn't for other runtimes due to the lack of profile path. That's the only reason why I considered this a bug. Of course we could say that this is not really an important feature in CRI, so we can drop it, but we can be more clear on that.
If there's a bug in a feature no one uses, does fixing it make a sound? I think what we're not agreeing on is the role of CRI in 1.5. I understood it as something that is not generally usable. We can't recommend a customer use it, even as an experimental feature, due to security aspects (if nothing else). Correct me if I'm wrong. It's a developer integration point, and developers integrating with it are doing so with the kubernetes master branch, not with some release. Those reasons together result in my opinion that CRI is still not really a customer/release feature, and thus bugs or features related to it should not be exceptions to the freeze window, but rather should targeted at master as soon as the freeze lifts. If this were already post-freeze, would we be cherry-picking this over to 1.5?
This is not true. cri-o is depending on this fix for apparmor support.
is it depending on the 1.5 release branch directly, or on master?
I think it's better to make this in 1.5, so cri-o could work with first release with experimental CRI. It helps to encourage people to use CRI and find features misses or bugs.
Modifying the user-specified annotation is a little bit unsettling. I think adding a new annotation explicitly for the path would be better. Or, we should just add an explicit field for this. I spoke to and separately. I still think this is a bug fix, but we may want to be more careful about the fix. Given the time constraint, let's just list this as a known issue and work on it for 1.6.
OK, let's move this to next release and think carefully about seccomp feature besides this: kubelet may just pass the profile as string via CRI; may assume the profile locate in a file in a known format and pass path to the runtime; or maybe some other solutions.


Question: return "" } <ins> func (m *LinuxContainerSecurityContext) GetSeccompProfilePath() string { if m != nil { return m.SeccompProfilePath } return "" } </ins> // LinuxContainerConfig contains platform-specific configuration for // Linux-based containers. type LinuxContainerConfig struct {

We need to clearly define it. define profile format clearly
Please document known issues for 1.5 in
(copied from ) And let's move here to discuss how to pass profiles to runtime. Related with .
This is issue is related with two level APIs: CRI, since we may also want to support profiles from ConfigMaps, I think we should pass full profile contents instead of path to runtime. issues is also related with seccomp api in kube-apiserver, it is based on annotations today. We should define seccomp profile format clearly (see ) and decide whether to add struct types to pod spec or just continue the annotations way. I think it's more clear to add specific struct types to pod spec.
Agreed - the path stuff is really a crutch (and implementation-specific) - I think is the answer here.
I don't disagree, but it's not strictly a requirement for supporting ConfigMaps (kubelet could create a file with the contents). An open question is whether to pass it as a raw string, or wait for and pass the structured data. The annotations way is deprecated (). I think the bigger question is where/how to put the data. E.g. structured data in the API, unstructured (string type) data in the API, reference to another object (e.g. ConfigMap). If the Kubelet needs to parse the spec, I think that's a good argument for a structured data type (either a standalone object or embedded in the pod API).
Yes, already opened to replace annotations with typed structs. I personally prefer a structured data type in both kubernetes API and CRI. WDYT?
A counter argument is that as the spec grows or new features are , we need to keep updating the Kubernetes API to maintain parity, or update the code in each CRI implementation to translate the Kubernetes spec. OTOH, if we use unstructured data we can use 3rd party libraries to parse out the data we need (assuming the spec evolves in a backwards compatible way).
What spec?
Whichever seccomp profile spec is being implemented by runtimes. E.g. if we decide to base the Kubernetes seccomp profile API off the OCI seccomp profile spec, and OCI adds some new features, at some point we'd probably want to add those features to Kubernetes.


Question: return "" } <ins> func (m *LinuxContainerSecurityContext) GetSeccompProfilePath() string { if m != nil { return m.SeccompProfilePath } return "" } </ins> // LinuxContainerConfig contains platform-specific configuration for // Linux-based containers. type LinuxContainerConfig struct {

It also means that if different runtimes support different feature sets, we'll be stuck either tracking only the common features, or specifying features that may be ignored by the runtime. This may be a non-issue if the seccomp feature set rarely changes.
Well, yes, but that applies to everything in the CRI, and is one of the tradeoffs we ma[d|k]e by introducing this abstraction in the first place.
My $0.02: Moving the existing logic/information for seccomp into a first-class field for CRI (as in ) is productive because of the known problems with the annotation approach. My initial proposal for seccomp support included a resource similar to what has proposed in kubernetes/community and we were unable to come to an agreement on the issue refers to here: I do not necessarily expect that we will be able to come to an agreement about the seccomp resource in time for 1.7, so I think we should proceed with and do backward compatible CRI changes later if we need to inject the content of the seccomp profile instead of the path. I think we would deeply regret embedding API surface to describe seccomp profiles in the Pod API :)


Question: expectedList: []HostPriority{{"machine1", 37 /* int(75% / 2) */}, {"machine2", 62 /* int( 75% + 50% / 2) */}}, test:         "no resources requested", pods: []api.Pod{ <del> {DesiredState: cpuOnly, CurrentState: machine1State}, {DesiredState: cpuAndMemory, CurrentState: machine2State}, </del> <ins> {DesiredState: cpuOnly}, {DesiredState: cpuAndMemory}, </ins> }, }, {

When creating replication controller (e.g. frontend php-redis from guestbook example), scheduler gets stuck and never recovers. This is a race, I have to run several times to see the error (A script to delete controller and pods, then create new controller, leave enough time in between for k8s to react). I've created PR for an attempted fix.
I saw this too while following the guest book and creating the php/frontend step. In my case, I had 2 minions while the replication controller configuration had configured 3 replicas, so 2 pods got scheduled -each on another minion, the third pod had empty Host (saw that when doing 'list pods'), and the 3 of them were stuck in "waiting" status. I deleted the controller and the pods, fixed the configuration in json file to be 2 replicas and recreated the controller and everything worked. Does your fix take care of this use case? (too many replicas, too few minions/one of the pods is not scheduled)
This is different from your use case. If you have more replicas than minions, then the scheduler will get stuck for sure (with the error msg probably being 'failed to find a fit for pod'). AFAICT, it's not support yet.
If you remove the HostPorts, it should work for you. thanks for report.
, thanks for your replies. I understand why it isn't working, but I'd expect one of the 2 options to happen instead of the current situation: the creation of replication controller if there are less minions than replicas, AND there is a hostPort property. In other words - validation of the content during creation. least schedule all replicas that can get a minion (in my case 2 out of 3) and leave only one at "waiting". Is there an issue open about the use case I describe?
should be true today. is less possible than it seems like-- minions can come and go, and become over or under loaded in the time between creation and when the pod shows up at the scheduler. So even if we check up front, we still have to check again later, and sometimes the first check will pass but not the second.


Question: if err != nil { return nil, "", err } <del> return aggregator.FilterSpecByPathsWithoutSideEffects(result, []string{"/apis/"}), etag, nil </del> <ins> group := specInfo.apiService.Spec.Group version := specInfo.apiService.Spec.Version return aggregator.FilterSpecByPathsWithoutSideEffects(result, []string{"/apis/" + group + "/" + version}), etag, nil </ins> }, cached.Result[*spec.Swagger]{Value: result, Etag: etag, Err: err}) specInfo.spec.Store(filteredResult) return err

When installing the metrics-server to implement the API and the prometheus-adapter to implement the API, it is possible to make the OpenAPI handler fail to initialize with this error log in apiserver: This will result in kubectl discovery calls to /openapi/v2 to fail with a 503: The OpenAPI handler should not fail to initialize if both and are serving the same path for different API groups/versions. I can provide exact installation steps if needed, however, a high-level repro is: install metrics-server to serve install prometheus-adapter to serve ensure resourceRules is populated in the prometheus-adapter config so that resource metrics are also served by restarting apiserver will result in the OpenAPI failure and 503 responses to /openapi/v2 path No response <details</details<detailsAWS </details<details</details<details</details<details</details<details</details
Please let me know if I should reopen the issue in
/sig api-machinery
/sig instrumentation [metrics apis]
Would you mind taking a look when have time? Thank you^^ /triage accepted
I have a question. should be serving under . Why does an apiservice serving another gvk register paths under ? cc we've made the assumption that an APIService (such as metrics-server or prometheus-adapter) serves only gvks under its own registered group version. I'll do some more investigation but just wondering if prometheus doesn't follow that assumption?
the prometheus-adapter can also be configured to serve and replace the metrics-server by enabling these metrics, However, if the prometheus-adapter fronts and metrics-server fronts , it does seem like the correct behavior would be for prometheus-adapter to serve until an operator updates the APIService to reference prometheus-adapter rather than metrics-server. We're seeing this issue occur for upgrades between 1.27 and 1.28 so it does look this path duplication did not cause the OpenAPI handler to fail on 1.27.
I just upgraded from 1.27 to 1.28 and running into this problem. Some kubectl apply operations fail, and ArgoCD cannot load its custom resources. Any suggested workaround? Would removing the from the prometheus configuration resolve it? Or perhaps disable metrics-server?
Simply looking at the code diff between 1.27 and 1.28 I can't pinpoint the cause.


Question: if err != nil { return nil, "", err } <del> return aggregator.FilterSpecByPathsWithoutSideEffects(result, []string{"/apis/"}), etag, nil </del> <ins> group := specInfo.apiService.Spec.Group version := specInfo.apiService.Spec.Version return aggregator.FilterSpecByPathsWithoutSideEffects(result, []string{"/apis/" + group + "/" + version}), etag, nil </ins> }, cached.Result[*spec.Swagger]{Value: result, Etag: etag, Err: err}) specInfo.spec.Store(filteredResult) return err

Can I get your exact repro steps? suggests that the conflicting paths should be removed before the merge. We always call this function with
One other question is if this is reproducible by just applying a set of yaml/installations or does it require a cluster upgrade to trigger? Thanks!
hit same problem and we cannot install any other new helm plugins. maybe we need to move forward
Here are verbose repro instructions. An apiserver restart on 1.28 will induce the issue.
Hello. I'm still having some trouble reproducing although I'm not 100% sure if the configuration is correct. I'm trying to repro via kind. afaik installing via helm has prometheus adapter serving . seems to suggest that prometheus adapter is serving under . Applying the update to the configmap per the instructions didn't change the apiservice on my end. Not sure if this is my misconfiguration or intended. I've also tested via integration test that having two openapis serving the same path doesn't cause errors and one will get filtered out. A simple workaround is probably to delete the metrics-server if it is already superseded by the prometheus-adapter.
The is not set up in an resource for us, we only created an APIService with . When I go fetch the openAPI schema served up by the endpoint, it has schema information for in it if that is enabled using . If I remove then the schema fetched from the server doesn't have stuff in it, and the error is resolved. We have been keeping around because I think something uses it directly, or used to. Maybe ? That could have changed But in any case, this problem is quite severe when it happens so it would be nice to fix it as there may be others who do the same update and have to do some deep diving into the logs and all kinds of crazy diagnostics and guesswork to figure it out.
How does the get reached without an APIService? The resources would be unavailable for the apiserver to query and it'll be through something else? I see, adding resources for me doesn't populate the OpenAPI. I wonder if this could be cloud provider specific. I'm totally supportive of fixing, it's just quite difficult to reproduce on my part .
Hmm good question.


Question: if err != nil { return nil, "", err } <del> return aggregator.FilterSpecByPathsWithoutSideEffects(result, []string{"/apis/"}), etag, nil </del> <ins> group := specInfo.apiService.Spec.Group version := specInfo.apiService.Spec.Version return aggregator.FilterSpecByPathsWithoutSideEffects(result, []string{"/apis/" + group + "/" + version}), etag, nil </ins> }, cached.Result[*spec.Swagger]{Value: result, Etag: etag, Err: err}) specInfo.spec.Store(filteredResult) return err

In our staging cluster I tried bringing back the in the custom metrics server configuration and restarting all the masters, and the problem did not come back, everything seems OK now. So it maybe isn't that easy to reproduce
After updating the configmap, restarting the prometheus-adapter pod to pickup the changes and restarting the apiserver are both required to induce the issue. I have been able to deterministically reproduce this following those steps, please follow up here if you are still having issues.
That's good to hear. I'll need some more help with reproducing it. I've been able to verify that restarting the prometheus adapter pod picks up the configmap changes but restarting the apiserver hasn't caused any crashes for me. I've tested this both on the latest master branch and 1.28.
While I'm yet to fully understand these changes, the 2 PRs below (both of which went into 1.28) seem likely candidates for a regression here: - introduced lazy loading in openAPI v2 handler to load/compute the aggregator spec when requested by client - switched the spec merging function from to any clue if one of the above changes could be causing mischief?
Also from example request at the top, the impact seems to be to the v2 API: Nathan - when you get a chance, could you also confirm if v1 API doesn't run into this?
OpenAPI only supports v2 and v3.
hi, any clue about this issue?
It's possible but I haven't been able to reproduce it. Can I ask if this is happening on cloud providers outside AWS?
Here are the apiservices and relevant pod objects with a working repro from these steps If it's useful, I can provide yamls for these objects in addition to the settings applied in the prometheus-adapter configmap. Can you confirm that the pods backing your prometheus-adapter and metrics-server are running?
I meet this problem
We the problem in GCP as well.
ACK in Alibaba Cloud also meets this problem
We were able to fix the v2 availability by and re-installing the kube-prometheus helm chart. Unfortunately, after restarting the k8s api server, v2 availability is broken again.
yes, I'm running on kind. I'll do some more digging, assuming it's related to those PRs linked by but still haven't pinpointed anything.


Question: if err != nil { return nil, "", err } <del> return aggregator.FilterSpecByPathsWithoutSideEffects(result, []string{"/apis/"}), etag, nil </del> <ins> group := specInfo.apiService.Spec.Group version := specInfo.apiService.Spec.Version return aggregator.FilterSpecByPathsWithoutSideEffects(result, []string{"/apis/" + group + "/" + version}), etag, nil </ins> }, cached.Result[*spec.Swagger]{Value: result, Etag: etag, Err: err}) specInfo.spec.Store(filteredResult) return err

I tried to reproduce on my local setup with based on steps + , but I could still get a successful response when I executed . Then I reverted one of the step mentioned in to and waited until the new pod came alive and then did a apiserver restart and I was able to reproduce the same thing on as well i.e; I get the 503 response when I execute following command after the fact. After this point, even if i change it back to and restart the apiserver , the error is persistent. Let me know if this works for you to reproduce ? PS: Also, i want to call out that there is no issue with , its .
Thanks, alright I found a way to reproduce it consistently without restarting the apiserver. Still investigating the root cause. Thanks for the detailed reproduction and
Quick update: This issue was a combination of bugs both in k/k and prometheus adapter. I opened to mitigate on k/k side which should fix the issue. The issue was difficult to reproduce because it requires a very specific ordering of OpenAPI merging to trigger, which is why I suspect many people were able to reproduce after some restarts to kube-apiserver. Prometheus and metrics server serve differing OpenAPIs for the endpoint. Prometheus also doesn't seem to be registering discovery's OpenAPI properly (missing OpenAPI for but having the OpenAPI for whereas it is all present in metrics server. This inconsistency in how prometheus and metrics server handle the OpenAPI surfaced the merging issue.
a little wired question: why do we need to restart kube-apiserver to reproduce this problem?
Tried to reproduce the issue after cherry picking the fix and didn't run into below error. Fix is working as expected.
It can look like restarting prometheus-adapter "fixes" the issue? I am also curious about suggestion of removing the section from the config map works as a temporary workaround until released upstream (we use and cannot delete the apiregistration)? As I understand the fix for this issue is currently in