Question: m.updateAllocatedDevices(m.activePods()) allocatedDevicesUpdated = true } <del> allocDevices, err := m.devicesToAllocate(podUID, contName, resource, needed) </del> <ins> allocDevices, err := m.devicesToAllocate(podUID, contName, resource, needed, devicesToReuse[resource]) </ins> if err != nil { return err }

do you think if you have time to implement the solution you mentioned in the above comment We can try to get it into a 1.9 sub release. I think for now, lets just support device reuse among initContainers and normal containers. We can revisit this part when we need to guarantee an Allocate call per container.
Right what I am mentioning here is that it is not possible to do that. The point of the device plugin is so that we can execute vendor specific code before making the device available in the container (e.g: Scrub the GPU memory). I don't think the proposed solution addresses this problem.
Sure. But to clarify, I think if we dont't expect big shock on current implementation, we could only provide limited support on the point, that is, without re-generate runtime information for regular containers, only those whose requested resource is equal to or is a subset of that in init containers could make a resuse(e.g. if we have two init containers, each requests 2 devices, then a regular container that request 3 devices still could not reuse the devices.) Besides, if a initcontainer requests 2 devices, and we have two regular containers, each requests 1 device, then due to the lack of runtime info re-generation, the devices could only be reused by one of the containers, not both of them. In sum, since we don't re-generate runtime information, we could only reuse devices per container, not per device, like we do in scheduler. Is that what you mean?
Hmm, that is a good question. I forgot that we now return the combined runtime information in AllocResponse. Then for now, perhaps we should just return an error in device plugin manager Allocate() in case both init containers and containers request the same device plugin resource?
You mean we give up reuse and prohibit users to request same kind of device in init containers and regular containers? that seems a bit weird. And if we decide to do so, we might need to do the check at the first beginning, i.e. in API validation, not wait until it goes into DevicePlugin.
I think for 1.9, we just return an error when both init containers and normal containers of a pod request the same device resource, and document this in the device plugin user guide.


Question: m.updateAllocatedDevices(m.activePods()) allocatedDevicesUpdated = true } <del> allocDevices, err := m.devicesToAllocate(podUID, contName, resource, needed) </del> <ins> allocDevices, err := m.devicesToAllocate(podUID, contName, resource, needed, devicesToReuse[resource]) </ins> if err != nil { return err }

The reason is because in the current device plugin implementation, we don't have a good way to free devices used by init containers of an active pod. It is hard to do this kind of validation at API level or even scheduler because we can't know whether an extended resource is device resource or not. So for now, I think we should probably just return a clear error message during admission handler. I am not sure yet whether there will be strong use cases that need to share devices between init containers and normal containers. We can revisit this based on user feedback. If there is strong user demand to support this, something we could do is to differentiate init containers and normal containers in podDevices, and free up devices used by init containers when any normal container is about to run. That would request some code structure change, and we may wait to see whether there is strong demand to support this.
It is hard to do this at API level or even scheduler because we can't know whether an extended resource is device resource or not. Oh, you are right, I ignored the resource awareness issue. I also agree that it's hard to address the issue without big change to current implementation. But I still feel the inconsistency of behavior between API and kubelet might confusing users. However, its effect should be sustainable, since we haven't received feedback about init containers up to now.
Right. So for now, perhaps just return an error when this happens, to give clear error signals.
Oh, I'm afraid we miss something, the check may not work as expected. As I commented, because init-containers run sequentially, we did reuse between init containers in , that is, if we have two init containers, each request 1 device, then we expect 1 device in total in scheduler, but expect 2 in deviceplugin. The check could not cover the scenario.
Thinking about this problem a bit more, I wonder whether we may tackle this issue from the following approaches, ordered by their complexities: 1) Just return an error whenever an init container requests a device plugin resource, and document in the device plugin user guide that requiring device plugin resource in init containers are not supported. This may be the short-term change we can try to get into 1.9. 2) Maintain a list of devices allocated to init containers in ManagerImpl.Allocate().


Question: m.updateAllocatedDevices(m.activePods()) allocatedDevicesUpdated = true } <del> allocDevices, err := m.devicesToAllocate(podUID, contName, resource, needed) </del> <ins> allocDevices, err := m.devicesToAllocate(podUID, contName, resource, needed, devicesToReuse[resource]) </ins> if err != nil { return err }

Try to allocate from this list first for later init containers and normal containers. The required no. of devices we pass to ManagerImpl.devicesToAllocate() should subtract the no. of devices we have allocated to init containers. This may require some code structure change around ManagerImpl.Allocate(), ManagerImpl.allocateContainerResources(), and ManagerImpl.devicesToAllocate(). However, Allocate gRPC call is still issued during Admission all together, and it is possible that we issue an Allocate gRPC for [dev1] (say dev1 allocated to initContianer1), followed by another Allocate gRPC call for [dev1, dev2] (say dev1 and dev2 allocated to initContainer2), with none of containers started or finished. This assumes containers running in the same pod share the same security context, and device plugin doesn't need to perform per-container allocation operations. We just use Allocate gRPC call to learn container runtime settings. 3) Take a further step from 2), during Admission, we just determine what devices to allocate to containers without issuing the actual Allocate gRPC calls. I.e., we only fill deviceIds field in podDevices.deviceAllocateInfo but leaves allocResp field empty. During ManagerImpl.GetDeviceRunContainerOptions(), we check for this case and issue the actual Allocate gRPC call for the container. This ensures we can issue a gRPC Allocate call per container. However, if a pod is restarted multiple times, we don't guarantee Allocate gRPC call will be issued during subsequent pod runs. 4) Take a further step from 3), we can check for overlapping device allocations in ManagerImpl.GetDeviceRunContainerOptions() and invalidate their allocResp so that an Allocate gRPC call will be issued next time for containers using re-allocated devices. E.g., for containerA, if we find it is allocated with [dev1, dev2] and dev1 is also allocated to containerB, we set containerB's allocResp to nil so that next time before containerB is running, we will issue an Allocate gRPC call for dev1. Note that this approach does have the drawback that device plugin failure would cause previously running pods not be able to restart, so we need to carefully evaluate our use cases to see whether we need to implement this. My suggestion is to implement 1) now and gather some user feedbacks to determine whether we need to implement the following approaches.


Question: m.updateAllocatedDevices(m.activePods()) allocatedDevicesUpdated = true } <del> allocDevices, err := m.devicesToAllocate(podUID, contName, resource, needed) </del> <ins> allocDevices, err := m.devicesToAllocate(podUID, contName, resource, needed, devicesToReuse[resource]) </ins> if err != nil { return err }

I see this a very big limitation. Special devices/resources in init containers is a very common use case IMO. Therefore we should try to fix this. This approach perfectly makes sense to me. What is your concern in implementing this? Even today, IIUC, Allocate call is made per container. Is there any other value that this approach is adding over 1) ? I am not able to see any gains with 2) or 3) over 1). Comments please.
I feel we may not be able to implement 2) for 1.9 timeframe. Option 2) also doesn't address the case mentioned above: Option 3) covers that case for pods without restarts. Option 4) fully covers the requirement we we will scrub old container data before giving it to a new container, even if they both run in the same pod, bug it does have the mentioned drawback during device plugin failure.
Take a further step from 3), we can check for overlapping device allocations in ManagerImpl.GetDeviceRunContainerOptions() and invalidate their allocResp so that an Allocate gRPC call will be issued next time for containers using re-allocated devices. E.g., for containerA, if we find it is allocated with [dev1, dev2] and dev1 is also allocated to containerB, we set containerB's allocResp to nil so that next time before containerB is running, we will issue an Allocate gRPC call for dev1. Note that this approach does have the drawback that device plugin failure would cause previously running pods not be able to restart, so we need to carefully evaluate our use cases to see whether we need to implement this. How about making the allocation call optional? That is , if we find a contaienr using re-allocated devices, we'd try to issue an Allocate gRPC call first, and still use the cached setting if allocation failed. And, if we decide not to address the issue in v1.9, which means we have enough time to work on it later, I think approach 2) might not be a good choice. Because it has a premise that devices are stateless, recycle actions before reuse(e.g: Scrub the GPU memory mentioned above) could not be supported. And this way, as I commented above, we could only reuse devices per container, not per device, like we do in scheduler.


Question: m.updateAllocatedDevices(m.activePods()) allocatedDevicesUpdated = true } <del> allocDevices, err := m.devicesToAllocate(podUID, contName, resource, needed) </del> <ins> allocDevices, err := m.devicesToAllocate(podUID, contName, resource, needed, devicesToReuse[resource]) </ins> if err != nil { return err }

I noticed that you have already add the limitation into the , so I think it's already the decision that we add the limitation as a short-time solution in v1.9 So I posted a PR to add the check in code,, PTAL :)
Use Case: Let say there is a which node has two Devices of a resource type X and the device IDs are X1 and X2. User pod has one init container, lets call it cont-init and one regular container, lets call it cont-reg. Both cont-init and cont-reg requests 2 unit of resource X. cont-init will run some vendor specific code on allocated Devices and then cont-reg will use these 'initiated' devices. I think this is the use case that mentioned. And i feel 2) can handle this. 2) approach will allocate both the devices, X1 and X2, to both the containers. When the containers start, init container, cont-init, will be run first and will run to completion and here devices will get initialized. Strictly after this cont-reg will be started and now this regular container has initialized devices. Where i am wrong?
I guess in RenaudWasTaken 's case, the initialization task is considered to be managed in the plugin, not in init-container, and in init-container we only use the device, so then we might need to call plugin to do recycle action before the device reused by reguler containers. In practical use, both of the two scenarios may exist, it depends on the request of user. Besides, if we don't issue an Allocate gRPC call, we could not ensure a ”complete reuse“: say an init-container requests 2 devices, and we have two regular containers, each requests 1 device, then due to the lack of runtime info re-generation, the devices could only be reused by one of the containers, not both of them.
It is the init containers' job to initialize things and if in some special case device plugin is doing the initialization part, in that case, init containers will not be needed and serve no purpose in pod spec. Specific to example of memory scrubbing, i think this also makes more sense to be done in init container. Once Device's memory is scrubbed, it can be used by regular container. Let try to run your use case with approach 2).


Question: m.updateAllocatedDevices(m.activePods()) allocatedDevicesUpdated = true } <del> allocDevices, err := m.devicesToAllocate(podUID, contName, resource, needed) </del> <ins> allocDevices, err := m.devicesToAllocate(podUID, contName, resource, needed, devicesToReuse[resource]) </ins> if err != nil { return err }

There are three containers in pod, cont-init(requested 2 unit or resource X), cont-reg1(requested 1 unit of resource X) and cont-reg2(requested 1 unit of resource X). Now in approach 2), gRPC Allocate call is made for cont-init requesting X1 and X2. Next, gRPC Allocate call is made for cont-reg1. This time resource id will be send in Allocate from the previously allocated [X1, X2]. Lets say X1 is requested for cont-reg1 Next, gRPC Allocate for cont-reg2 and resource with ID X2. All this allocation info gets stored in internal cache. Now when container runs, runtimeOptions will be collected from internal cache. Where do you see a problem in this flow?
/cc
Yep, it could work if the plugin behavior the way you described, but I'm not sure if all kinds of devices could support "pre-allocation", that is, doing allocation on a same device multiple times at the same time, to prepare runtime informations for several containers that reuse the device later. Also I have no idea if all plugin developers consider their plugins "stateless", that is, managing initialization in init-containers. As I don't have enough knowledge on hardware, would like to hear opinions from other guys :)
This would defeat the purpose of a container. In the case of GPUs, when you run a container you expect the GPU to always be in the same state. Furthermore GPUs can't be shared (exclusive usage) between containers meaning that Init containers should behave in the same way as regular containers. With the only difference being that they are executed sequentially and before regular containers. Additionally, operations to setup a device before making them available in a container, whether functionally (load the driver, insert libraries, ....) or security wise (scrub the memory), should be handled by the infrastructure not the users pod spec.
I agree that it's a sane behavior to forbid the use of Device Plugin Resources in Init Containers for the 1.9 release
moving forward,
/cc I think what long-term solution we decide to implement will largely depend on use-case requirements from different device plugins. Would like to hear some thoughts from different device plugin developers on this. Can people cc-ed in this issue take a look at the proposed options mentioned in and provide some feedbacks on which one may better fit your use cases?


Question: m.updateAllocatedDevices(m.activePods()) allocatedDevicesUpdated = true } <del> allocDevices, err := m.devicesToAllocate(podUID, contName, resource, needed) </del> <ins> allocDevices, err := m.devicesToAllocate(podUID, contName, resource, needed, devicesToReuse[resource]) </ins> if err != nil { return err }

Answering on behalf of and Solution 1 makes sense for the moment Solution 2 defeats the purpose of the device plugin, which is setting up a device before making it available to a container. solution 4 seems to be a sane solution. As mentioned in my previous comment, as long as devices are exclusive, I expect the devices exposed in my container to always be in the same state at the start of my container. When / If we implement a mechanism for sharing a GPU / devices across containers then the scenario described by is something that we can have. It doesn't however fit the current situation.
Solution number 1 seems totally reasonable short-term. Long term, simplifying and documenting the relationship between plugin callbacks and container lifecycle would help plugin authors.
For reference, I implemented option 2 solution in PR The change itself is small. This is an alternative short-term fix we may consider to get into a 1.9 patch release. I think the main benefit of taking this fix compared to Option 1 solution is that we can change the underlying implementation to support init containers later with less user-facing impact.
No mater we take PR or PR for 1.9, I think we need to have this issue approved for 1.9 milestone. or or can any of you approve this issue for 1.9? Thanks!


Question: func logFileHandler(req *restful.Request, resp *restful.Response) { logdir := "/var/log" actual := path.Join(logdir, req.PathParameter("logpath")) <ins> // check filename length first, return 404 if it's oversize. if logFileNameIsTooLong(actual) { http.Error(resp, "file not found", http.StatusNotFound) return } </ins> http.ServeFile(resp.ResponseWriter, req.Request, actual) }

Hello and I were fuzzing the k8s (via ) and found out following bug. Getting logs from unicode logpath via API causes internal server error. The API endpoint is . Response with non-500 HTTP status code We were fuzzing k8s locally via minikube. minukube ca certificates k8s IP address the IP to as access token and export it as request Kubernetes version (use ):Cloud provider or hardware configuration: minikube with virtual-box driver OS (e.g: ): Ubuntu 18.04.5 LTS (bionic) Kernel (e.g. ): Linux pine 4.15.0-140-generic -Ubuntu SMP Fri Mar 19 14:12:35 UTC 2021 x8664 x8664 x86_64 GNU/Linux Install tools: downloaded precompiled binary minikube version: v1.19.0 (commit: )
/sig api-machinery
/assign I will look into it.
thank you , please keep us posted! /triage accepted
For clarity are you saying that the fuzzed URL is valid or invalid UTF-8 (URL-encoded)?
It's valid UTF-8
Actually, according to my test,this bug comes from http.ServeFile which replies to the request with the contents of the named file or directory. When giving too long(On Linux: The maximum length for a file name is 255 bytes. The maximum combined length of both the file name and path name is 4096 bytes.) filename or pathname to http.ServeFile,it will return code 500(file name too long) rather than 404(no such file or directory). And when filename is not that long, unicode path work correctly too. Here is my simple test below: And since /logs is the only place use http.ServeFile, there is no other api has the same problem. ! So I will add a length check before calling http.ServeFile to solve this. Does that sound good to you?
Maybe 400 - bad request would be more suitable response code. But I'm not a maintainer so it's not up to me to decide.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with .


Question: func logFileHandler(req *restful.Request, resp *restful.Response) { logdir := "/var/log" actual := path.Join(logdir, req.PathParameter("logpath")) <ins> // check filename length first, return 404 if it's oversize. if logFileNameIsTooLong(actual) { http.Error(resp, "file not found", http.StatusNotFound) return } </ins> http.ServeFile(resp.ResponseWriter, req.Request, actual) }

Send feedback to sig-contributor-experience at . /lifecycle stale
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: TerminationMessagePolicy: "File", }, }, <ins> "Invalid env from": { { Name:                     "env-from-source", Image:                    "image", ImagePullPolicy:          "IfNotPresent", TerminationMessagePolicy: "File", EnvFrom: []api.EnvFromSource{ { ConfigMapRef: &api.ConfigMapEnvSource{ LocalObjectReference: api.LocalObjectReference{ Name: "$%^&*#", }, }, }, }, }, }, </ins> } for k, v := range errorCases { if errs := validateContainers(v, volumes, field.NewPath("field")); len(errs) == 0 {

Scenario: Expected results: pod would not be created as [0] is invalid Actual results: pod is created. Turns out we are not doing any validation of envFrom in container spec today.
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list .
I don't know that we can tighten without failing pods.
The validation code exists, but there was no wiring done from validateContainers.
Pods fail on kubelet already. So those pods were not doing much already. We had a BZ on this.
Fail how - fail to even start, or fail to get the input?


Question: }, } } <ins> // ListenForSignal starts a goroutine that will trigger the CacheDebugger's // behavior when the process receives SIGINT (Windows) or SIGUSER2 (non-Windows). func (d *CacheDebugger) ListenForSignal(stopCh <-chan struct{}) { ch := make(chan os.Signal, 1) signal.Notify(ch, compareSignal) go func() { for { select { case <-stopCh: return case <-ch: d.Comparer.Compare() d.Dumper.DumpAll() } } }() } </ins>

PLEASE COORDINATE WITH SIG-SCHEDULING BEFORE WORKING ON THIS ISSUE. We want to avoid the situation where two people accidentally start work on the same fix. Main tracking issue: Currently, the feature is set up at the end of in . This setup should be moved into its own helper function, preferably in the package. /sig scheduling /kind cleanup /help
/assign /remove-help
I'd like to take this one
/remove-help
the PR I sent should resolve this one. :disappointed: Are any of the others still available?


Question: "k8s.io/api/core/v1" "k8s.io/apimachinery/pkg/types" <ins> "k8s.io/apimachinery/pkg/util/intstr" </ins> "k8s.io/apimachinery/pkg/util/sets" "k8s.io/apimachinery/pkg/util/wait" genericfeatures "k8s.io/apiserver/pkg/features"

There is no value by that conversion, even worse it can be misused.
should we use in such case?
I do not think this is a bug. 25% == .25 and int(.25) == 0. ;) Even in the case where the user puts in something like 100% I do not think it's a bug. Users should not put strings like "25%" into intOrString and expect to get the integer 25 out. Likely a user doing that is confused about what format the field takes in. I am not sure why the function GetValueFromIntOrPercent exists, and I bet I am not going to like the answer :)
No. First, this depends on how you do the rounding. If you are rounding up (MaxSurge in Deployments), this is 1. Then, the replica size of the deployment also plays a part. 25% of a controller with 1 replica is different from a controller with 4 replicas, etc.
Also, the fact that IntValue returns 0 for every percentage has nothing to do with any logic. It's purely because the conversion errors out:
the first line there was supposed to be a joke, sorry. :) I know that the actual reason for returning 0 is the % sign means it is not an int. More seriously: I actually agree with the parsing function that percentages are not ints. Users who want to do this should not use IntOrString. Is there a place in the API that wants to do this?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/lifecycle frozen /remove-lifecycle stale
Opened to ensure all in-tree callers only call this when the type is Int (or verify it is a parseable number first), and to update the method documentation


Question: "/pod0-i":  getTestContainerInfo(seedPod0Infra, pName0, namespace, leaky.PodInfraContainerName), "/pod0-c0": getTestContainerInfo(seedPod0Container0, pName0, namespace, cName00), } <del> output := removeTerminatedContainerInfo(infos) assert.Len(t, output, 2) </del> <ins> filteredInfos, allInfos := filterTerminatedContainerInfoAndAssembleByPodCgroupKey(infos) assert.Len(t, filteredInfos, 2) assert.Len(t, allInfos, 6) </ins> for _, c := range []string{"/pod0-i", "/pod0-c0"} { <del> if _, found := output[c]; !found { </del> <ins> if _, found := filteredInfos[c]; !found { t.Errorf("%q is expected to be in the outputn", c) } } for _, c := range []string{"pod0-i-terminated-1", "pod0-c0-terminated-1", "pod0-i-terminated-2", "pod0-c0-terminated-2", "pod0-i", "pod0-c0"} { if _, found := allInfos[c]; !found { </ins> t.Errorf("%q is expected to be in the outputn", c) } }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --For single node with hundreds and thousands of pods and containers, the cpu usage of kubelet becomes high, one bottleneck is the cadvisor stats provider as below: the method iterate all containerInfos for every podStats cause high cpu the method should not iterate allcontainerInfos for every podStats, use map instead Kubernetes version (use ): v1.19.4 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig node
/triage accepted
/assign
I’ve done an analysis on the code and this is what I have found, I believe we have two issues to address. There are multiple range loops being performed over v2.ContainerInfo / PodStat maps within the ListPodStatus method which can be contributing to the high cpu utilisation. The allInfos argument which is supplied to the getCadvisorContainerInfo function is a map which uses a cgroup path as the key, in order to find matching container info to podUID, processing of the keys in the map needs to be performed until a match is found. Examples of keys are as follows: /kubepods/burstable/podf9e3a173-1470-451a-9ca2-/d8426b82582fbcfae16085bc794c49769d3a9648bd5463976aca4071f9342754 /docker/2dfa17fc81e781212bcc26e189ebb724721fe097394b4de3cbeb7e069bf23290 My recommendation would be to apply the following updates: Refactor the getCadvisorContainerInfo function to allow for a filter and key type parameter to be supplied by the caller, this enables reducing the number of range loops being performed over the container info map by removing elements that are not required and the ability to set the key being used in the returned container infos map to enable fast lookup of container info.


Question: "/pod0-i":  getTestContainerInfo(seedPod0Infra, pName0, namespace, leaky.PodInfraContainerName), "/pod0-c0": getTestContainerInfo(seedPod0Container0, pName0, namespace, cName00), } <del> output := removeTerminatedContainerInfo(infos) assert.Len(t, output, 2) </del> <ins> filteredInfos, allInfos := filterTerminatedContainerInfoAndAssembleByPodCgroupKey(infos) assert.Len(t, filteredInfos, 2) assert.Len(t, allInfos, 6) </ins> for _, c := range []string{"/pod0-i", "/pod0-c0"} { <del> if _, found := output[c]; !found { </del> <ins> if _, found := filteredInfos[c]; !found { t.Errorf("%q is expected to be in the outputn", c) } } for _, c := range []string{"pod0-i-terminated-1", "pod0-c0-terminated-1", "pod0-i-terminated-2", "pod0-c0-terminated-2", "pod0-i", "pod0-c0"} { if _, found := allInfos[c]; !found { </ins> t.Errorf("%q is expected to be in the outputn", c) } }

My initial thoughts are to use an enumeration which has two values KEYCGROUPPATH to provide backwards compatibility and KEYPODUID to enable fast lookups via podUID Refactor ListPodStats method to be more efficient in constructing the PodStats slice and taking into account the fast lookup via podUID I think this is the best approach instead of pushing the data structure implementation for the container info upstream to the cadviser clients, this will at least initially provide a better optimisation scoped within the kubelet/stats package and reduce other possible impacts to changing the cadvisor clients Please advise.
Just realized this looks like a duplicate issue and looking as fixed by PR Both this issue and were raised by Zac
/unassign
/close
You can't close an active issue/PR unless you authored it or you are a collaborator. <detailsIn response to : /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Didn't think I could close this, leaving this with
this will be automatically closed once the PR merges


Question: <del> #!/usr/bin/env bash # Copyright 2018 The Kubernetes Authors. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # #     http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # Script that creates a Kubemark cluster for IBM cloud. KUBECTL="${KUBE_ROOT}/cluster/kubectl.sh" KUBEMARK_DIRECTORY="${KUBE_ROOT}/test/kubemark" RESOURCE_DIRECTORY="${KUBEMARK_DIRECTORY}/resources" # Generate secret and configMap for the hollow-node pods to work, prepare # manifests of the hollow-node and heapster replication controllers from # templates, and finally create these resources through kubectl. function create-kube-hollow-node-resources { # Create kubeconfig for Kubelet. KUBELET_KUBECONFIG_CONTENTS="$(cat <<EOF apiVersion: v1 kind: Config users: - name: kubelet user: client-certificate-data: "${KUBELET_CERT_BASE64}" client-key-data: "${KUBELET_KEY_BASE64}" clusters: - name: kubemark cluster: certificate-authority-data: "${CA_CERT_BASE64}" server: https://${MASTER_IP} contexts: - context: cluster: kubemark user: kubelet name: kubemark-context current-context: kubemark-context EOF )" # Create kubeconfig for Kubeproxy. KUBEPROXY_KUBECONFIG_CONTENTS="$(cat <<EOF apiVersion: v1 kind: Config users: - name: kube-proxy user: client-certificate-data: "${KUBELET_CERT_BASE64}" client-key-data: "${KUBELET_KEY_BASE64}" clusters: - name: kubemark cluster: insecure-skip-tls-verify: true server: https://${MASTER_IP} contexts: - context: cluster: kubemark user: kube-proxy name: kubemark-context current-context: kubemark-context EOF )" # Create kubeconfig for Heapster. HEAPSTER_KUBECONFIG_CONTENTS="$(cat <<EOF apiVersion: v1 kind: Config users: - name: heapster user: client-certificate-data: "${KUBELET_CERT_BASE64}" client-key-data: "${KUBELET_KEY_BASE64}" clusters: - name: kubemark cluster: insecure-skip-tls-verify: true server: https://${MASTER_IP} contexts: - context: cluster: kubemark user: heapster name: kubemark-context current-context: kubemark-context EOF )" # Create kubeconfig for Cluster Autoscaler. CLUSTER_AUTOSCALER_KUBECONFIG_CONTENTS="$(cat <<EOF apiVersion: v1 kind: Config users: - name: cluster-autoscaler user: client-certificate-data: "${KUBELET_CERT_BASE64}" client-key-data: "${KUBELET_KEY_BASE64}" clusters: - name: kubemark cluster: insecure-skip-tls-verify: true server: https://${MASTER_IP} contexts: - context: cluster: kubemark user: cluster-autoscaler name: kubemark-context current-context: kubemark-context EOF )" # Create kubeconfig for NodeProblemDetector. NPD_KUBECONFIG_CONTENTS="$(cat <<EOF apiVersion: v1 kind: Config users: - name: node-problem-detector user: client-certificate-data: "${KUBELET_CERT_BASE64}" client-key-data: "${KUBELET_KEY_BASE64}" clusters: - name: kubemark cluster: insecure-skip-tls-verify: true server: https://${MASTER_IP} contexts: - context: cluster: kubemark user: node-problem-detector name: kubemark-context current-context: kubemark-context EOF )" # Create kubeconfig for Kube DNS. KUBE_DNS_KUBECONFIG_CONTENTS="$(cat <<EOF apiVersion: v1 kind: Config users: - name: kube-dns user: client-certificate-data: "${KUBELET_CERT_BASE64}" client-key-data: "${KUBELET_KEY_BASE64}" clusters: - name: kubemark cluster: insecure-skip-tls-verify: true server: https://${MASTER_IP} contexts: - context: cluster: kubemark user: kube-dns name: kubemark-context current-context: kubemark-context EOF )" # Create kubemark namespace. spawn-config if kubectl get ns | grep -Fq "kubemark"; then kubectl delete ns kubemark while kubectl get ns | grep -Fq "kubemark" do sleep 10 done fi "${KUBECTL}" create -f "${RESOURCE_DIRECTORY}/kubemark-ns.json" # Create configmap for configuring hollow- kubelet, proxy and npd. "${KUBECTL}" create configmap "node-configmap" --namespace="kubemark"  --from-file=kernel.monitor="${RESOURCE_DIRECTORY}/kernel-monitor.json" # Create secret for passing kubeconfigs to kubelet, kubeproxy and npd. "${KUBECTL}" create secret generic "kubeconfig" --type=Opaque --namespace="kubemark"  --from-literal=kubelet.kubeconfig="${KUBELET_KUBECONFIG_CONTENTS}"  --from-literal=kubeproxy.kubeconfig="${KUBEPROXY_KUBECONFIG_CONTENTS}"  --from-literal=heapster.kubeconfig="${HEAPSTER_KUBECONFIG_CONTENTS}"  --from-literal=cluster_autoscaler.kubeconfig="${CLUSTER_AUTOSCALER_KUBECONFIG_CONTENTS}"  --from-literal=npd.kubeconfig="${NPD_KUBECONFIG_CONTENTS}"  --from-literal=dns.kubeconfig="${KUBE_DNS_KUBECONFIG_CONTENTS}" # Create addon pods. # Heapster. mkdir -p "${RESOURCE_DIRECTORY}/addons" sed "s/{{MASTER_IP}}/${MASTER_IP}/g" "${RESOURCE_DIRECTORY}/heapster_template.json" > "${RESOURCE_DIRECTORY}/addons/heapster.json" metrics_mem_per_node=4 metrics_mem=$((200 + metrics_mem_per_node*NUM_NODES)) sed -i'' -e "s/{{METRICS_MEM}}/${metrics_mem}/g" "${RESOURCE_DIRECTORY}/addons/heapster.json" metrics_cpu_per_node_numerator=${NUM_NODES} metrics_cpu_per_node_denominator=2 metrics_cpu=$((80 + metrics_cpu_per_node_numerator / metrics_cpu_per_node_denominator)) sed -i'' -e "s/{{METRICS_CPU}}/${metrics_cpu}/g" "${RESOURCE_DIRECTORY}/addons/heapster.json" eventer_mem_per_node=500 eventer_mem=$((200 * 1024 + eventer_mem_per_node*NUM_NODES)) sed -i'' -e "s/{{EVENTER_MEM}}/${eventer_mem}/g" "${RESOURCE_DIRECTORY}/addons/heapster.json" # Cluster Autoscaler. if [[ "${ENABLE_KUBEMARK_CLUSTER_AUTOSCALER:-}" == "true" ]]; then echo "Setting up Cluster Autoscaler" KUBEMARK_AUTOSCALER_MIG_NAME="${KUBEMARK_AUTOSCALER_MIG_NAME:-${NODE_INSTANCE_PREFIX}-group}" KUBEMARK_AUTOSCALER_MIN_NODES="${KUBEMARK_AUTOSCALER_MIN_NODES:-0}" KUBEMARK_AUTOSCALER_MAX_NODES="${KUBEMARK_AUTOSCALER_MAX_NODES:-${DESIRED_NODES}}" NUM_NODES=${KUBEMARK_AUTOSCALER_MAX_NODES} echo "Setting maximum cluster size to ${NUM_NODES}." KUBEMARK_MIG_CONFIG="autoscaling.k8s.io/nodegroup: ${KUBEMARK_AUTOSCALER_MIG_NAME}" sed "s/{{master_ip}}/${MASTER_IP}/g" "${RESOURCE_DIRECTORY}/cluster-autoscaler_template.json" > "${RESOURCE_DIRECTORY}/addons/cluster-autoscaler.json" sed -i'' -e "s/{{kubemark_autoscaler_mig_name}}/${KUBEMARK_AUTOSCALER_MIG_NAME}/g" "${RESOURCE_DIRECTORY}/addons/cluster-autoscaler.json" sed -i'' -e "s/{{kubemark_autoscaler_min_nodes}}/${KUBEMARK_AUTOSCALER_MIN_NODES}/g" "${RESOURCE_DIRECTORY}/addons/cluster-autoscaler.json" sed -i'' -e "s/{{kubemark_autoscaler_max_nodes}}/${KUBEMARK_AUTOSCALER_MAX_NODES}/g" "${RESOURCE_DIRECTORY}/addons/cluster-autoscaler.json" fi # Kube DNS. if [[ "${ENABLE_KUBEMARK_KUBE_DNS:-}" == "true" ]]; then echo "Setting up kube-dns" sed "s/{{dns_domain}}/${KUBE_DNS_DOMAIN}/g" "${RESOURCE_DIRECTORY}/kube_dns_template.yaml" > "${RESOURCE_DIRECTORY}/addons/kube_dns.yaml" fi "${KUBECTL}" create -f "${RESOURCE_DIRECTORY}/addons" --namespace="kubemark" set-registry-secrets # Create the replication controller for hollow-nodes. # We allow to override the NUM_REPLICAS when running Cluster Autoscaler. NUM_REPLICAS=${NUM_REPLICAS:-${KUBEMARK_NUM_NODES}} sed "s/{{numreplicas}}/${NUM_REPLICAS}/g" "${RESOURCE_DIRECTORY}/hollow-node_template.yaml" > "${RESOURCE_DIRECTORY}/hollow-node.yaml" proxy_cpu=20 if [ "${NUM_NODES}" -gt 1000 ]; then proxy_cpu=50 fi proxy_mem_per_node=50 proxy_mem=$((100 * 1024 + proxy_mem_per_node*NUM_NODES)) hollow_kubelet_params=$(eval "for param in ${HOLLOW_KUBELET_TEST_ARGS:-}; do echo -n "$param",; done") hollow_kubelet_params=${hollow_kubelet_params%?} hollow_proxy_params=$(eval "for param in ${HOLLOW_PROXY_TEST_ARGS:-}; do echo -n "$param",; done") hollow_proxy_params=${hollow_proxy_params%?} sed -i'' -e "s/{{HOLLOW_PROXY_CPU}}/${proxy_cpu}/g" "${RESOURCE_DIRECTORY}/hollow-node.yaml" sed -i'' -e "s/{{HOLLOW_PROXY_MEM}}/${proxy_mem}/g" "${RESOURCE_DIRECTORY}/hollow-node.yaml" sed -i'' -e "s'{{kubemark_image_registry}}'${KUBEMARK_IMAGE_REGISTRY}${KUBE_NAMESPACE}'g" "${RESOURCE_DIRECTORY}/hollow-node.yaml" sed -i'' -e "s/{{kubemark_image_tag}}/${KUBEMARK_IMAGE_TAG}/g" "${RESOURCE_DIRECTORY}/hollow-node.yaml" sed -i'' -e "s/{{master_ip}}/${MASTER_IP}/g" "${RESOURCE_DIRECTORY}/hollow-node.yaml" sed -i'' -e "s/{{hollow_kubelet_params}}/${hollow_kubelet_params}/g" "${RESOURCE_DIRECTORY}/hollow-node.yaml" sed -i'' -e "s/{{hollow_proxy_params}}/${hollow_proxy_params}/g" "${RESOURCE_DIRECTORY}/hollow-node.yaml" sed -i'' -e "s'{{kubemark_mig_config}}'${KUBEMARK_MIG_CONFIG:-}'g" "${RESOURCE_DIRECTORY}/hollow-node.yaml" "${KUBECTL}" create -f "${RESOURCE_DIRECTORY}/hollow-node.yaml" --namespace="kubemark" echo "Created secrets, configMaps, replication-controllers required for hollow-nodes." } # Wait until all hollow-nodes are running or there is a timeout. function wait-for-hollow-nodes-to-run-or-timeout { echo -n "Waiting for all hollow-nodes to become Running" start=$(date +%s) # IKS uses a real cluster for hollow master, so need to exclude the real worker nodes nodes=$("${KUBECTL}" --kubeconfig="${KUBECONFIG}" get node | grep hollow-node 2> /dev/null) || true ready=$(($(echo "${nodes}" | grep -vc "NotReady") - 1)) until [[ "${ready}" -ge "${NUM_REPLICAS}" ]]; do echo -n "." sleep 1 now=$(date +%s) # Fail it if it already took more than 30 minutes. if [ $((now - start)) -gt 1800 ]; then echo "" # shellcheck disable=SC2154 # Color defined in sourced script echo -e "${color_red} Timeout waiting for all hollow-nodes to become Running. ${color_norm}" # Try listing nodes again - if it fails it means that API server is not responding if "${KUBECTL}" --kubeconfig="${KUBECONFIG}" get node &> /dev/null; then echo "Found only ${ready} ready hollow-nodes while waiting for ${NUM_NODES}." else echo "Got error while trying to list hollow-nodes. Probably API server is down." fi spawn-config pods=$("${KUBECTL}" get pods -l name=hollow-node --namespace=kubemark) || true running=$(($(echo "${pods}" | grep -c "Running"))) echo "${running} hollow-nodes are reported as 'Running'" not_running=$(($(echo "${pods}" | grep -vc "Running") - 1)) echo "${not_running} hollow-nodes are reported as NOT 'Running'" echo "${pods}" | grep -v "Running" exit 1 fi nodes=$("${KUBECTL}" --kubeconfig="${KUBECONFIG}" get node 2> /dev/null) || true ready=$(($(echo "${nodes}" | grep -vc "NotReady") - 1)) done # shellcheck disable=SC2154 # Color defined in sourced script echo -e "${color_green} Done!${color_norm}" } ############################### Main Function ######################################## # In order for the cluster autoscalar to function, the template file must be changed so that the ":443" # is removed. This is because the port is already given with the MASTER_IP. # Create clusters and populate with hollow nodes complete-login build-kubemark-image choose-clusters generate-values set-hollow-master echo "Creating kube hollow node resources" create-kube-hollow-node-resources master-config # shellcheck disable=SC2154 # Color defined in sourced script echo -e "${color_blue}EXECUTION COMPLETE${color_norm}" # Check status of Kubemark # shellcheck disable=SC2154 # Color defined in sourced script echo -e "${color_yellow}CHECKING STATUS${color_norm}" wait-for-hollow-nodes-to-run-or-timeout # Celebrate echo "" # shellcheck disable=SC2154 # Color defined in sourced script echo -e "${color_blue}SUCCESS${color_norm}" clean-repo exit 0 </del>  No newline at end of file

While trying to run with a valid ibmcloud account, the following error happened I also tried to run and got the same error. Looks like a compatibility issue. ibmcloud CLI shouldn't return error Run with a valid IBM cloud account. No response I am trying to build kubemark from source <details</details<detailsIBM cloud IKS </details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scalability
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:


Question: tc := tests[idx] ginkgo.It(tc.name, func(ctx context.Context) { ginkgo.By("waiting for the node to be ready", func() { <del> if !supportsInPlacePodVerticalScaling(ctx, f) || framework.NodeOSDistroIs("windows") { </del> <ins> if !supportsInPlacePodVerticalScaling(ctx, f) || framework.NodeOSDistroIs("windows") || isRunningOnArm64() { </ins> e2eskipper.Skipf("runtime does not support InPlacePodVerticalScaling -- skipping") } })

/kind failing-test <!-- If this is a flake, please add: /kind flake --/sig node
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
could be related to
/assign
Yes, tests should be skipped with Feature:InPlacePodVerticalScaling on those CIs , it fails because InPlacePodVerticalScaling is not enabled. Will update tomorrow morning, checking more closely how to fix in a smooth way
Thanks , It was failing in 11 jobs for two reasons: In 7 jobs because InPlacePodVerticalScaling was not supported and was missed to identify, attempt to fix by because all impacted CI jobs will skip tests with labels with prefix NodeAlphaFeatures In 4 jobs because Arm64 was not checked in one test, attempt to fix by


Question: if nodeIP.IsUnspecified() { nodeIP = utilnode.GetNodeIP(client, hostname) if nodeIP == nil { <del> return nil, fmt.Errorf("unable to get node IP for hostname %s", hostname) </del> <ins> klog.V(0).Infof("can't determine this node's IP, assuming 127.0.0.1; if this is incorrect, please set the --bind-address flag") nodeIP = net.ParseIP("127.0.0.1") </ins> } } if proxyMode == proxyModeIPTables {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Using Kubernetes 1.16.1, I started kube-proxy in iptables mode as a static pod on a master node. The master node was intentionally (and permanently) not available in the API as a Node object. kube-proxy started up and terminated soon after because it failed to (which is supposed to function as the bound address) and eventually returned an error: What you expected to happen: I expected the same behavior as in Kubernetes 1.15 where failure to look up the Node in the API would result in . How to reproduce it (as minimally and precisely as possible): Configure kube-proxy with iptables mode on Kubernetes 1.16 without the hosting Node being available in the Kubernetes API (under the OS hostname). Anything else we need to know?: PR (Fix for kube-proxy to wait for some duration for the node to be defined) introduced an additional . I believe this is the part that is causing the change in behavior: in Kubernetes 1.15, the check would be omitted and cause to set the default bind address to 127.0.0.1 (when iptables mode is set; I haven't checked other modes). The PR's primary purpose seems to be a different one (retrying node IP retrieval from the API), so my understanding is that the check was not really needed; unless I'm misunderstanding things which I don't want to rule out at all. The quick workaround was to explicitly configure kube-proxy with . However, I'm not entirely sure if that's a good solution long-term compared to letting Kubernetes do The Right Thing, which is what I'd like to clarify through this issue. Environment: Kubernetes version (use ): v1.16.1 Cloud provider or hardware configuration: Kubernetes running on DigitalOcean VMs (droplets) OS (e.g: ): Debian GNU/Linux 9 (stretch) Kernel (e.g.


Question: if nodeIP.IsUnspecified() { nodeIP = utilnode.GetNodeIP(client, hostname) if nodeIP == nil { <del> return nil, fmt.Errorf("unable to get node IP for hostname %s", hostname) </del> <ins> klog.V(0).Infof("can't determine this node's IP, assuming 127.0.0.1; if this is incorrect, please set the --bind-address flag") nodeIP = net.ParseIP("127.0.0.1") </ins> } } if proxyMode == proxyModeIPTables {

): Linux master-k8s-1-16-test1 4.19.0-0.bpo.6-amd64 SMP Debian 4.19.67-2+deb10u1~bpo9+1 (2019-09-30) x86_64 GNU/Linux /sig network /cc who were involved in the referenced PR.
I made a similar comment in , wondering if this is a breaking change in v1.16 that we should revert? Previously if is nil, kube-proxy defaults to but now kube-proxy always exits with error instead of defaulting to local host. I think a lot of external cloud providers relied on this behavior because they needed to talk to API server prior to node addresses being set by the provider. Thoughts /assign
Opened to preserve behavior before v1.16 where node IP defaults to 127.0.01 is none is found. PTAL


Question: } return conditionStatuses } <del> return !reflect.DeepEqual(strip(oldNode.Status.Conditions), strip(newNode.Status.Conditions)) </del> <ins> return !equality.Semantic.DeepEqual(strip(oldNode.Status.Conditions), strip(newNode.Status.Conditions)) </ins> } func nodeSpecUnschedulableChanged(newNode *v1.Node, oldNode *v1.Node) bool { return newNode.Spec.Unschedulable != oldNode.Spec.Unschedulable && !newNode.Spec.Unschedulable } <ins> func nodeAnnotationsChanged(newNode *v1.Node, oldNode *v1.Node) bool { return !equality.Semantic.DeepEqual(oldNode.GetAnnotations(), newNode.GetAnnotations()) } </ins> func preCheckForNode(nodeInfo *framework.NodeInfo) queue.PreEnqueueCheck { // Note: the following checks doesn't take preemption into considerations, in very rare // cases (e.g., node resizing), "pod" may still fail a check but preemption helps. We deliberately

Due to limited character space, I've stored additional information in annotations on nodes. My custom scheduling plugins read these annotations to schedule Pods. However, when a Pod fails to be scheduled on a node, I want it to be removed from the Unschedulable queue. This can be achieved by modifying the annotations of the node, allowing the Pod to be rescheduled. Unfortunately, the eventHandler of the scheduler only supports five types of node events, and it does not include the nodeAnnotationsChanged event. Consequently, I am currently unable to trigger rescheduling by changing the annotations of nodes. /sig scheduling Unschedulable pods will be rescheduled after annotations of nodes changed. None No response <details</details<details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc
/assign
It's more like a new feature, not bug because it doesn't lead to any bad behavior on the upstream. /remove-kind bug /kind feature
My concern is node annotation is arbitrary, change a node annotation doesn't imply a pod is schedulable. But for other events, like node labels, node taints, they're the constraints we consider in scheduling. They're different, I'm afraid this will lead to invalid scheduling attempts. Who is responsible for change the annotations in you scenario?
This enhancement has no impact on the in-tree plugins / upstream scheduler, right? Only (custom) plugins, that want to reschedule Pods via the node annotation change, will register event to EventsToRegister.
I have a controller capable of modifying node annotations to indicate node's current availability. And my custom plugins need to be aware of these changes.
I guess we can still be impacted when scheduling pod failed because of no fit node rather than failed by the plugin.


Question: } return conditionStatuses } <del> return !reflect.DeepEqual(strip(oldNode.Status.Conditions), strip(newNode.Status.Conditions)) </del> <ins> return !equality.Semantic.DeepEqual(strip(oldNode.Status.Conditions), strip(newNode.Status.Conditions)) </ins> } func nodeSpecUnschedulableChanged(newNode *v1.Node, oldNode *v1.Node) bool { return newNode.Spec.Unschedulable != oldNode.Spec.Unschedulable && !newNode.Spec.Unschedulable } <ins> func nodeAnnotationsChanged(newNode *v1.Node, oldNode *v1.Node) bool { return !equality.Semantic.DeepEqual(oldNode.GetAnnotations(), newNode.GetAnnotations()) } </ins> func preCheckForNode(nodeInfo *framework.NodeInfo) queue.PreEnqueueCheck { // Note: the following checks doesn't take preemption into considerations, in very rare // cases (e.g., node resizing), "pod" may still fail a check but preemption helps. We deliberately

see Can we predict whether the pod is schedulable before running into scheduling cycle via the node annotations, then we can leverage the preEnqueue extension point since you already have a external controller.
Thanks for the advice. But I'm afraid it would not be the best solution for us, because doing that outside the scheduling cycle would cause the custom plugins to malfunction in my scenario.
The concerns raised by make sense to me. In my understanding, most in-tree plugins will classify the event as an event.
No. When no fit Pod, UnschedulablePlugins will have plugin(s) rejected the Pod. (e.g., all nodes don't have enough capacity, UnschedulablePlugins has noderesourcefit. all nodes don't satisfy node affinity, UnschedulablePlugins has nodeaffinity) is a rare case like plugins other than PreFilter/Filter returns non-success. I don't think we should care about it.
But, this is a good point, I think we should refactor all plugins that register event with to have a specific Node update event. For example, . It should have with instead of . event with , that should be the last-resort when we don't know which kind of Node update will make Pods schedulable. (I guess no such case at least in the in-tree plugins.)
1) One possibility is Reserve Plugin, I the capability here 2) Another possibility is PreFilter, but not because failed by the plugin, but no fit node exists Have no idea whether we have other places, I guess no. But yes, this only impacts unschedulable pods has no failed plugins.
~I think this is for intended, like ~ ~1. 1) we have a plugin failed by nodeAffinity~ ~2. 2) we add a new node~ ~3. 3) we receive the node add event, but because node not ready, we failed by nodeTaint plugin as ~ 4) node become ready, we get the node update event, but dismatch with nodeAffinity registered event (if we registered with UpdateNodeLabel event)~ ~5. 5) pod will not be moved to the active or backoff queue~ updated below
weigh in for advices as whether this is a good idea to take nodeAnnotations as a new event.


Question: } return conditionStatuses } <del> return !reflect.DeepEqual(strip(oldNode.Status.Conditions), strip(newNode.Status.Conditions)) </del> <ins> return !equality.Semantic.DeepEqual(strip(oldNode.Status.Conditions), strip(newNode.Status.Conditions)) </ins> } func nodeSpecUnschedulableChanged(newNode *v1.Node, oldNode *v1.Node) bool { return newNode.Spec.Unschedulable != oldNode.Spec.Unschedulable && !newNode.Spec.Unschedulable } <ins> func nodeAnnotationsChanged(newNode *v1.Node, oldNode *v1.Node) bool { return !equality.Semantic.DeepEqual(oldNode.GetAnnotations(), newNode.GetAnnotations()) } </ins> func preCheckForNode(nodeInfo *framework.NodeInfo) queue.PreEnqueueCheck { // Note: the following checks doesn't take preemption into considerations, in very rare // cases (e.g., node resizing), "pod" may still fail a check but preemption helps. We deliberately

As I know, some plugins will read the embedded prometheus metrics in node annotations for scoring, but not for filtering. At least we should mitigate the impacts.
iv. node become ready, we get the node update event, but dismatch with nodeAffinity registered event (if we registered with UpdateNodeLabel event) (3) is not exact. When we receive the node add event, the scheduler retry to schedule the Pod because the nodeaffinity has Node add event. And it's rejected by the nodetaint plugin. (not node affinity) So, in (4), we get the node update event, it matches with node taint registered event, and the Pod is moved to active/backoffQ (node taint registers node update event. Yes I think we can change node taint's node update event to more specific one(s) though)
As I replied in your PR, we acknowledged that in , and yes let's have your patch. Well, that's a good point. So, when all nodes are filtered out by PreFilterResult, then no failedplugin can be returned. That's a problem. I think we should return the plugin name that returns PreFilterResult with not all Node names in this case (= the plugins that contributes to reducing the candidate Nodes via PreFilterResult). What do you think?
Sorry, the previous one is not right, a detailed one: a. we have a pod failed by nodeAffinity. pod will be placed into the unschedulablePods b. node autoscaler triggered, a new node , emitted a node add event c. called -d. node not ready right now and holds a notReady taint, then failed, pod still stays in the unschedulablePods e. node gets ready, emitted a node update event, but dismatch with the registered event(if we register with in nodeAffinity plugin) f. pod will stay in unschedulabelPods I guess this is the right flow. Anyway, glad to see it be more effect in the future.
If we have no performance problems, I think it's ok to take nodeAnnotations into consideration.
Wouldn't this be solved with proposal on ClusterEventWithHint? If so, I prefer we leave the existing code as-is.
whatever we do here, we wouldn't backport, so I think is the right move forward.
We put many stuff here.


Question: } return conditionStatuses } <del> return !reflect.DeepEqual(strip(oldNode.Status.Conditions), strip(newNode.Status.Conditions)) </del> <ins> return !equality.Semantic.DeepEqual(strip(oldNode.Status.Conditions), strip(newNode.Status.Conditions)) </ins> } func nodeSpecUnschedulableChanged(newNode *v1.Node, oldNode *v1.Node) bool { return newNode.Spec.Unschedulable != oldNode.Spec.Unschedulable && !newNode.Spec.Unschedulable } <ins> func nodeAnnotationsChanged(newNode *v1.Node, oldNode *v1.Node) bool { return !equality.Semantic.DeepEqual(oldNode.GetAnnotations(), newNode.GetAnnotations()) } </ins> func preCheckForNode(nodeInfo *framework.NodeInfo) queue.PreEnqueueCheck { // Note: the following checks doesn't take preemption into considerations, in very rare // cases (e.g., node resizing), "pod" may still fail a check but preemption helps. We deliberately

Let me summarize each: Born from from is right. But, that's the problem with preCheck. (Some preCheck depends on some in-tree plugins' logic and it affects badly to other plugins enqueueing ) The preCheck is removed after is completed. After preCheck is removed, will be the correct behavior. So, we can do the optimization that changes node to a specific event. will be fixed in: It's kind of rare case, but I feel we need to have some improvement like Edit: create another issue for this: I'd say (2) and (3) are kind of rare cases. So, the blocker here is (1). (And, (1) is blocked by the removal of preCheck, which is blocked by ) As my conclusion, it's safer to add after (1) is done so that we can make sure the performance won't be affected badly. what do you think?
Instead, can we just keep an event for "node updated", then individual plugins can decide whether a annotations change is worth requeuing or not?
Yes, (1) can be replaced with that idea. But, I'd say changing to a specific is slightly better because the scheduling queue could filter out some Node update events without executing the hint functions.
If rescheduling can be triggered with precise events, that would be great. I haven't revisit since last time, I'll take a look again.
Since we are missing an event for , can we add it? But we also have to modify existing events like so: Because change of such field is also an Update. And in future releases we can get rid of some fine-grained events.
Since we will add hintFuncs for all in-tree plugins, so I think it's ok to add nodeAnnotationsChanged event now, we can filter out unrelated events in hitFuncs. Any other concerns? cc Prefer to keep them as people can register fine-gained events instead a general event.
If by "now" you mean v1.30, then I'm ok with that.


Question: //    (localhost) by name. The possible profile names are detailed at //    http://wiki.apparmor.net/index.php/AppArmor_Core_Policy_Reference ApparmorProfile string `protobuf:"bytes,9,opt,name=apparmor_profile,json=apparmorProfile,proto3" json:"apparmor_profile,omitempty"` <ins> // Seccomp profile for the container, candidate values are: // * runtime/default: the default profile for the container runtime // * unconfined: unconfined profile, ie, no seccomp sandboxing // * localhost/<full-path-to-profile>: the profile installed on the node. //   <full-path-to-profile> is the full path of the profile. SeccompProfilePath string `protobuf:"bytes,10,opt,name=seccomp_profile_path,json=seccompProfilePath,proto3" json:"seccomp_profile_path,omitempty"` </ins> } func (m *LinuxContainerSecurityContext) Reset()      { *m = LinuxContainerSecurityContext{} }

CRI supports AppArmor, Seccomp and Sysctls via annotations , we should move them to typed and structured fields instead. (refer ) [x] Apparmor () [x] Sysctls () [x] Seccomp () cc
/cc
/sig node
Seccomp is addressed at .


Question: //    (localhost) by name. The possible profile names are detailed at //    http://wiki.apparmor.net/index.php/AppArmor_Core_Policy_Reference ApparmorProfile string `protobuf:"bytes,9,opt,name=apparmor_profile,json=apparmorProfile,proto3" json:"apparmor_profile,omitempty"` <ins> // Seccomp profile for the container, candidate values are: // * runtime/default: the default profile for the container runtime // * unconfined: unconfined profile, ie, no seccomp sandboxing // * localhost/<full-path-to-profile>: the profile installed on the node. //   <full-path-to-profile> is the full path of the profile. SeccompProfilePath string `protobuf:"bytes,10,opt,name=seccomp_profile_path,json=seccompProfilePath,proto3" json:"seccomp_profile_path,omitempty"` </ins> } func (m *LinuxContainerSecurityContext) Reset()      { *m = LinuxContainerSecurityContext{} }

Seccomp profile is passed to runtime by in CRI: is relative to node's local seccomp profile root, it is defined in kubelet. But runtime doesn't know it. We should pass full profile path in CRI. cc/ Will file a PR to fix this later.
Add support for Seccomp sounds like a new feature. The milestone should be reset to , correct?
not really. The current implementation supports seccomp already. If it's missing critical information to function properly, this would be a bug.
are we blocking releases for CRI in 1.5? i was not sure, so marking this as non-release-blocker and leaving 1.5 milestone. please correct if i am mistaken.
I don't get why this is a 1.5 targetted thing myself. Yes we want CRI to be in a good state in 1.5, but afaik we have zero intention of anyone actually using (at the very least because of the lack of exec auth regressing kubelet security). why should this be part of 1.5?
First, this is a small enough change that allows non-docker integrations to properly support seccomp. Second, if "we have zero intention of anyone actually using" is a valid reason, we should never fix any bugs in CRI. We do want people to try using it (perhaps not in production yet). However, given that seccomp is only in alpha, I am ok with dropping this and just continue the docker support for now.
Perhaps I misunderstood the meaning of code freeze, but I thought it was only bug-fixes regardless of size of change (or exceptions that have gone through some process) What I meant is we don't intend this to be used generally as part of the 1.5 release. Because of that, I don't think we should consider bugs in CRI bugs for the purpose of the 1.5 release freeze. My statement is just that we shouldn't violate code-freeze for this. We should totally fix this right after the 1.5 freeze window ends, and saying that I'm arguing we should never fix anything is disingenuous and taking it to an absurd extreme.
the only thing we don't seem to agree on is whether seccomp support was considered to be supported for CRI in 1.5.


Question: //    (localhost) by name. The possible profile names are detailed at //    http://wiki.apparmor.net/index.php/AppArmor_Core_Policy_Reference ApparmorProfile string `protobuf:"bytes,9,opt,name=apparmor_profile,json=apparmorProfile,proto3" json:"apparmor_profile,omitempty"` <ins> // Seccomp profile for the container, candidate values are: // * runtime/default: the default profile for the container runtime // * unconfined: unconfined profile, ie, no seccomp sandboxing // * localhost/<full-path-to-profile>: the profile installed on the node. //   <full-path-to-profile> is the full path of the profile. SeccompProfilePath string `protobuf:"bytes,10,opt,name=seccomp_profile_path,json=seccompProfilePath,proto3" json:"seccomp_profile_path,omitempty"` </ins> } func (m *LinuxContainerSecurityContext) Reset()      { *m = LinuxContainerSecurityContext{} }

My interpretation is that we assumed seccomp would be supported until we found out that it wouldn't for other runtimes due to the lack of profile path. That's the only reason why I considered this a bug. Of course we could say that this is not really an important feature in CRI, so we can drop it, but we can be more clear on that.
If there's a bug in a feature no one uses, does fixing it make a sound? I think what we're not agreeing on is the role of CRI in 1.5. I understood it as something that is not generally usable. We can't recommend a customer use it, even as an experimental feature, due to security aspects (if nothing else). Correct me if I'm wrong. It's a developer integration point, and developers integrating with it are doing so with the kubernetes master branch, not with some release. Those reasons together result in my opinion that CRI is still not really a customer/release feature, and thus bugs or features related to it should not be exceptions to the freeze window, but rather should targeted at master as soon as the freeze lifts. If this were already post-freeze, would we be cherry-picking this over to 1.5?
This is not true. cri-o is depending on this fix for apparmor support.
is it depending on the 1.5 release branch directly, or on master?
I think it's better to make this in 1.5, so cri-o could work with first release with experimental CRI. It helps to encourage people to use CRI and find features misses or bugs.
Modifying the user-specified annotation is a little bit unsettling. I think adding a new annotation explicitly for the path would be better. Or, we should just add an explicit field for this. I spoke to and separately. I still think this is a bug fix, but we may want to be more careful about the fix. Given the time constraint, let's just list this as a known issue and work on it for 1.6.
OK, let's move this to next release and think carefully about seccomp feature besides this: kubelet may just pass the profile as string via CRI; may assume the profile locate in a file in a known format and pass path to the runtime; or maybe some other solutions.


Question: //    (localhost) by name. The possible profile names are detailed at //    http://wiki.apparmor.net/index.php/AppArmor_Core_Policy_Reference ApparmorProfile string `protobuf:"bytes,9,opt,name=apparmor_profile,json=apparmorProfile,proto3" json:"apparmor_profile,omitempty"` <ins> // Seccomp profile for the container, candidate values are: // * runtime/default: the default profile for the container runtime // * unconfined: unconfined profile, ie, no seccomp sandboxing // * localhost/<full-path-to-profile>: the profile installed on the node. //   <full-path-to-profile> is the full path of the profile. SeccompProfilePath string `protobuf:"bytes,10,opt,name=seccomp_profile_path,json=seccompProfilePath,proto3" json:"seccomp_profile_path,omitempty"` </ins> } func (m *LinuxContainerSecurityContext) Reset()      { *m = LinuxContainerSecurityContext{} }

We need to clearly define it. define profile format clearly
Please document known issues for 1.5 in
(copied from ) And let's move here to discuss how to pass profiles to runtime. Related with .
This is issue is related with two level APIs: CRI, since we may also want to support profiles from ConfigMaps, I think we should pass full profile contents instead of path to runtime. issues is also related with seccomp api in kube-apiserver, it is based on annotations today. We should define seccomp profile format clearly (see ) and decide whether to add struct types to pod spec or just continue the annotations way. I think it's more clear to add specific struct types to pod spec.
Agreed - the path stuff is really a crutch (and implementation-specific) - I think is the answer here.
I don't disagree, but it's not strictly a requirement for supporting ConfigMaps (kubelet could create a file with the contents). An open question is whether to pass it as a raw string, or wait for and pass the structured data. The annotations way is deprecated (). I think the bigger question is where/how to put the data. E.g. structured data in the API, unstructured (string type) data in the API, reference to another object (e.g. ConfigMap). If the Kubelet needs to parse the spec, I think that's a good argument for a structured data type (either a standalone object or embedded in the pod API).
Yes, already opened to replace annotations with typed structs. I personally prefer a structured data type in both kubernetes API and CRI. WDYT?
A counter argument is that as the spec grows or new features are , we need to keep updating the Kubernetes API to maintain parity, or update the code in each CRI implementation to translate the Kubernetes spec. OTOH, if we use unstructured data we can use 3rd party libraries to parse out the data we need (assuming the spec evolves in a backwards compatible way).
What spec?
Whichever seccomp profile spec is being implemented by runtimes. E.g. if we decide to base the Kubernetes seccomp profile API off the OCI seccomp profile spec, and OCI adds some new features, at some point we'd probably want to add those features to Kubernetes.


Question: //    (localhost) by name. The possible profile names are detailed at //    http://wiki.apparmor.net/index.php/AppArmor_Core_Policy_Reference ApparmorProfile string `protobuf:"bytes,9,opt,name=apparmor_profile,json=apparmorProfile,proto3" json:"apparmor_profile,omitempty"` <ins> // Seccomp profile for the container, candidate values are: // * runtime/default: the default profile for the container runtime // * unconfined: unconfined profile, ie, no seccomp sandboxing // * localhost/<full-path-to-profile>: the profile installed on the node. //   <full-path-to-profile> is the full path of the profile. SeccompProfilePath string `protobuf:"bytes,10,opt,name=seccomp_profile_path,json=seccompProfilePath,proto3" json:"seccomp_profile_path,omitempty"` </ins> } func (m *LinuxContainerSecurityContext) Reset()      { *m = LinuxContainerSecurityContext{} }

It also means that if different runtimes support different feature sets, we'll be stuck either tracking only the common features, or specifying features that may be ignored by the runtime. This may be a non-issue if the seccomp feature set rarely changes.
Well, yes, but that applies to everything in the CRI, and is one of the tradeoffs we ma[d|k]e by introducing this abstraction in the first place.
My $0.02: Moving the existing logic/information for seccomp into a first-class field for CRI (as in ) is productive because of the known problems with the annotation approach. My initial proposal for seccomp support included a resource similar to what has proposed in kubernetes/community and we were unable to come to an agreement on the issue refers to here: I do not necessarily expect that we will be able to come to an agreement about the seccomp resource in time for 1.7, so I think we should proceed with and do backward compatible CRI changes later if we need to inject the content of the seccomp profile instead of the path. I think we would deeply regret embedding API surface to describe seccomp profiles in the Pod API :)


Question: expected       []string // any of the items is valid }{ { <del> name:           "No node needs preemption", registerPlugin: st.RegisterPluginAsExtensions(noderesources.FitName, noderesources.NewFit, "Filter", "PreFilter"), nodeNames:      []string{"node1"}, pod:            st.MakePod().Name("p").UID("p").Priority(highPriority).Req(largeRes).Obj(), pods: []*v1.Pod{ st.MakePod().Name("p1").UID("p1").Node("node1").Priority(midPriority).Req(smallRes).StartTime(epochTime).Obj(), }, expected: []string{"node1"}, }, { </del> name:           "a pod that fits on both nodes when lower priority pods are preempted", registerPlugin: st.RegisterPluginAsExtensions(noderesources.FitName, noderesources.NewFit, "Filter", "PreFilter"), nodeNames:      []string{"node1", "node2"},

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --Kubernetes 1.19 (confirmed with 1.19.7 and 1.19.10) kube-scheduler crashes and restarts with the following errors in . panic: runtime error: index out of range [0] with length 0 The problem code is line 389. When is nil or is empty, the error happens. If we skip when it's nil or empty, the error is gone. panic: runtime error: invalid memory address or nil poster dereference The problem is does not exist and returns nil sometimes. Simply skipping it won't solve the problem. The scheduler will reach either line 456 or 341. The scheduler works without failures. When there are preemptions in a cluster. Kubernetes version (use ): v1.19.10 or v1.19.7 Cloud provider or hardware configuration: bare metal OS (e.g: ): centos 7 Kernel (e.g. ): Linux 5.4.77-7.el7pie SMP Sat Nov 21 01:16:27 UTC 2020 x8664 x8664 x8664 GNU/Linux Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scheduling
/cc
/assign
Are you running the vanilla scheduler? or a customized one that leverages the utilities in
It's a custom scheduler with some internal out-of-tree plugins (just like ), but it doesn't have any custom postFilter plugins. Which utility functions are you referring to? Thanks.
are those custom plugins filter plugins? do they maintain state?


Question: expected       []string // any of the items is valid }{ { <del> name:           "No node needs preemption", registerPlugin: st.RegisterPluginAsExtensions(noderesources.FitName, noderesources.NewFit, "Filter", "PreFilter"), nodeNames:      []string{"node1"}, pod:            st.MakePod().Name("p").UID("p").Priority(highPriority).Req(largeRes).Obj(), pods: []*v1.Pod{ st.MakePod().Name("p1").UID("p1").Node("node1").Priority(midPriority).Req(smallRes).StartTime(epochTime).Obj(), }, expected: []string{"node1"}, }, { </del> name:           "a pod that fits on both nodes when lower priority pods are preempted", registerPlugin: st.RegisterPluginAsExtensions(noderesources.FitName, noderesources.NewFit, "Filter", "PreFilter"), nodeNames:      []string{"node1", "node2"},

If yes, then one hypothesis is that those custom filters filter the node in the filter phase, but not when executed in the preemption phase, and so this will result in a candidate node with no victims?
Yes, it includes filter plugins and uses cycle state.
you need to make sure that those filter plugins play nicely when executed again in the preemption phase in the same cycle: i.e., produce the same result.
Thanks What do you mean by performing filter in the preemption phase? would you mind elaborating it a little?
What additional logic is needed to handle the preemption case?
in the preemption phase we run the filters again to check if removing lower priority pods make the node schedulable: we later add them one by one to reduce the set of victim pods to the absolute minimum: My hypothesis is that somehow your filter returns success when adding all the pods back, and so you end up with zero victims and a candidate node. This shouldn't happen because if the pod fits the node without removing any victims, then we shouldn't be running preemption in the first place. So the theory is that the custom filter is returning false when run in the filter phase, but returns true when executed in the preemption phase perhaps because it makes some assumptions about cyclestate that makes it behave this way (e.g. that a the filter will be executed once per node in a scheduling cycle).
Disabled the filter plugin, but still see the same issues.
are you able to change the scheduler code and run the test again? I can send a patch tomorrow to add some debugging messages to help us root cause the issue.
yes, thanks a lot! really appreciate it!
You are right. We still use a deprecated Predicate extender, which causes the problem. After disabling it, it's working fine. It used to work with 1.18 though. We are retiring it. Thank you so much!!!
If the test cases can help debug scheduler extender Webhook (Predicate) too, I'd like to try it. Thanks again!
How can we improve the preemption logic to prevent bugs and issues in custom Filters or Predicates crashing the scheduler?
for starter, we should check the list len, the scheduler should not panic in all cases.


Question: expected       []string // any of the items is valid }{ { <del> name:           "No node needs preemption", registerPlugin: st.RegisterPluginAsExtensions(noderesources.FitName, noderesources.NewFit, "Filter", "PreFilter"), nodeNames:      []string{"node1"}, pod:            st.MakePod().Name("p").UID("p").Priority(highPriority).Req(largeRes).Obj(), pods: []*v1.Pod{ st.MakePod().Name("p1").UID("p1").Node("node1").Priority(midPriority).Req(smallRes).StartTime(epochTime).Obj(), }, expected: []string{"node1"}, }, { </del> name:           "a pod that fits on both nodes when lower priority pods are preempted", registerPlugin: st.RegisterPluginAsExtensions(noderesources.FitName, noderesources.NewFit, "Filter", "PreFilter"), nodeNames:      []string{"node1", "node2"},

Other than that, I think we need to clearly document what the preemption logic does (that it executes the prefilter extensions points and the filter plugins multiple times) and the expectations from filter plugins (that they may get executed more than once in the same cycle)
Is it a predicate extender or preemption extender? If it's a predicate extender, I don't quite think we found the root cause - b/c predicate extenders are not invoked during preemption (). It makes more sense if it's a preemption extender - if the preemption extender mutates the to make it invalid. If it's the case, could you verify it by adding debug info before and after L157 to check the validness of ?
That's always true :)
filed a PR to mitigate it.
It's a predicate extender. I'll debug it more.
Added some debug info. after and . Here's the result. returned 4 candidates, but one of candidates () has zero pod. The candidate happened to be the best candidate and hence caused the "" error. It should have nothing to do with since our extender doesn't support Preemption. Disabling it may change pod placements/candidates selection and bypassed the error. It's like a coincidence. To remove the panic, a simple fix is to check if is empty. If it is, do not add the node to candidates. Alternatively, a change to may work too. Here is the log with debug info.
This looks more promising. Back to digging into the root cause, it seems returns an empty but with . So that implies firstly we have a non-nil , and then in , the is separated into two list: violatingVictims and nonViolatingVictims. Finally the two victims list are tried one by one to run . It looks like returns {fits=true, nil error} for every victim, which is quite abnormal. I think it's problematic that every pod is retrievable and in which we can debug into.
Yes, should not return fit=true with empty pods in the first place, but it's hard to locate the problematic code in this function.
The following code in looks questionable to me. If is empty, should the function return nil, nil? Also, if is true for all extenders, the loop won't do anything and candidates will remain the same.


Question: expected       []string // any of the items is valid }{ { <del> name:           "No node needs preemption", registerPlugin: st.RegisterPluginAsExtensions(noderesources.FitName, noderesources.NewFit, "Filter", "PreFilter"), nodeNames:      []string{"node1"}, pod:            st.MakePod().Name("p").UID("p").Priority(highPriority).Req(largeRes).Obj(), pods: []*v1.Pod{ st.MakePod().Name("p1").UID("p1").Node("node1").Priority(midPriority).Req(smallRes).StartTime(epochTime).Obj(), }, expected: []string{"node1"}, }, { </del> name:           "a pod that fits on both nodes when lower priority pods are preempted", registerPlugin: st.RegisterPluginAsExtensions(noderesources.FitName, noderesources.NewFit, "Filter", "PreFilter"), nodeNames:      []string{"node1", "node2"},

Why do we bother creating a new candidates later? Can it just return candidates?
It can prevent from returning candidates with empty pods, but what about ? It's possible to add nodes without victim pods to candidates if returned by contain such nodes. How about adding the following check to CallExtender?
Updated the PR based on the finding and discussions.
Yes, we can use a flag to mark if all extenders don't support preemption. If yes, simply return the candidates.
Technically this can exist due to faulty extender implementation. If we really want to guard it, instead of logging the error and continue, I'm more inclined to return the error immediately as this is a fatal error - the victimsMap cannot be used either for latter extender or further preemptor nominating.
Is my understanding correct? In , if victimsMap is empty, should it return instead of ?
The only case victimsMap can be empty is empty candidates, and so return either doesn't quite matter, right?
You are right.
As long as one extender returns an invalid victimsMap (with empty pods), s returns an error immediately? What about ? Should it return an error immediately if it returnswith empty pods?
I think so. Because the result it returned is faulty, and we don't want to continue based on it The same. We may don't want to continue the scheduling cycle based on faulty result.


Question: # Compare Godep Licenses if ! _out="$(diff -Naupr ${KUBE_ROOT}/Godeps/LICENSES ${_tmpdir}/Godeps/LICENSES)"; then <del> echo "Your godep licenses file is out of date. Run hack/update-godep-licenses.sh and commit the results." echo "${_out}" </del> <ins> echo "Your godep licenses file is out of date. Run hack/update-godep-licenses.sh and commit the results." >&2 echo "${_out}" >&2 </ins> exit 1 fi

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: When running on my commits locally I got the following error: Affected PRs:
/sig testing
/cc
There's likely something up with your local environment. These jobs are running correctly in CI, although you need to look at the build log for output


Question: } func checkDiskAttached(volPath string, vmdevices object.VirtualDeviceList, dc *object.Datacenter, client *govmomi.Client) (bool, error) { <del> virtualDiskControllerKey, err := getVirtualDiskControllerKey(volPath, vmdevices, dc, client) </del> <ins> _, err := getVirtualDiskControllerKey(volPath, vmdevices, dc, client) </ins> if err != nil { if err == ErrNoDevicesFound { return false, nil

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): v1.4.x v1.5.x v1.6.x master Environment: vSphere based install with vSphere Cloud Provider enabled. Cloud provider or hardware configuration: vSphere OS (e.g. from /etc/os-release): NA Kernel (e.g. ):NA Install tools:kubernetes-anywhere Others: What happened: vSphere Cloud Provider configures additional SCSI adapters to scale the number of volumes that can be attached to a VM (1 adapter = 16 disks). The code within the cloud provider does not handle this correctly. Additional disks attach successfully to the VM but when k8s verifies the attach status it fails leading to k8s retrying the attach that never succeeds. In addition to this when the SCSI adapter is created for the first time the code does not use it to attach the disk leading to a failed attach disk request to the cloud provider. What you expected to happen: Additional disks should be to the VM upto the max number supported. How to reproduce it (as minimally and precisely as possible): Create a single node cluster and create a statefulset with 16 replicas or have a pod with more than 16 disks. Anything else we need to know:
A fix for this is coming up..
cc
Huh, I wonder if something changed between 1.3 and newer?


Question: } func checkDiskAttached(volPath string, vmdevices object.VirtualDeviceList, dc *object.Datacenter, client *govmomi.Client) (bool, error) { <del> virtualDiskControllerKey, err := getVirtualDiskControllerKey(volPath, vmdevices, dc, client) </del> <ins> _, err := getVirtualDiskControllerKey(volPath, vmdevices, dc, client) </ins> if err != nil { if err == ErrNoDevicesFound { return false, nil

We had tested up to 40-something volumes on 1.3. There was a fun discussion internally because it would always fail at SCSI ID 7 and I had to explain SCSI to some younger engineers :)


Question: for i := range pod.Spec.InitContainers { switch resourceName { case v1.ResourceMemory: <del> containerValue.Add(*pod.Spec.Containers[i].Resources.Requests.Memory()) </del> <ins> if initValue.Cmp(*pod.Spec.InitContainers[i].Resources.Requests.Memory()) < 0 { initValue = *pod.Spec.InitContainers[i].Resources.Requests.Memory() } </ins> case resourceDisk: <del> containerValue.Add(*pod.Spec.Containers[i].Resources.Requests.StorageEphemeral()) </del> <ins> if initValue.Cmp(*pod.Spec.InitContainers[i].Resources.Requests.StorageEphemeral()) < 0 { initValue = *pod.Spec.InitContainers[i].Resources.Requests.StorageEphemeral() } </ins> } } if containerValue.Cmp(initValue) > 0 {

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: Kubelet constantly crashes trying to evict pods under memory pressure What you expected to happen: kubelet to be able to evict pods How to reproduce it (as minimally and precisely as possible): Fresh 1.9.1 Kubernetes, installed via kubeadm Here are relevant eviction parameters for kubelet Anything else we need to know?: Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.1", GitCommit:"", GitTreeState:"archive", BuildDate:"2018-01-05T20:28:28Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-01-04T11:40:06Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: 8GiB of RAM OS (e.g. from /etc/os-release): Gentoo Kernel (e.g. ): Linux 4.9.34-gentoo SMP Sat Jan 13 23:06:16 MSK 2018 x86_64 Intel(R) Core(TM) i5-2500K CPU @ 3.30GHz GenuineIntel GNU/Linux Install tools: Others:
contains the following code I think there is a bug in that it is iterating through InitContainers but getting resources of regular containers. I have a pod that has 3 initContainers. Looks like this could be a copy&paste error.
/sig node
indeed there's a mismatch between initContainers and spec containers, I'll take a look.
looks like we need to do the following and the problem was introduced in cc
- Indeed, I'll open a PR for this
reopening until it is cherrypicked to 1.9
It was cherry-picked in 1.9.3.
/close


Question: glog.Infof(volumeToMount.GenerateMsgDetailed("MapVolume.WaitForAttach succeeded", fmt.Sprintf("DevicePath %q", devicePath))) <del> // Update actual state of world to reflect volume is globally mounted markDeviceMappedErr := actualStateOfWorld.MarkDeviceAsMounted( volumeToMount.VolumeName, devicePath, globalMapPath) if markDeviceMappedErr != nil { // On failure, return error. Caller will log and retry. return volumeToMount.GenerateError("MapVolume.MarkDeviceAsMounted failed", markDeviceMappedErr) } </del> } // A plugin doesn't have attacher also needs to map device to global map path with SetUpDevice() pluginDevicePath, mapErr := blockVolumeMapper.SetUpDevice()

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: A Pod is run with local volume mounted in block mode. When that pod is deleted, the tear down process fails because device unmap fails. The device path appears to be empty, causing the unmap to fail. What you expected to happen: The teardown should have succeeded How to reproduce it (as minimally and precisely as possible): Create a pod with a PV for local volume in block mode. Delete the pod. Look at kubelet log files on the minion node to observe the error messages. Anything else we need to know?: Log messages: Feb 18 09:21:43 kubernetes-minion-group-crj1 kubelet[1510]: E0218 09:21:43. 1510 ] Operation for """ ("-148c-11e8-b315-")" failed. No retries permitted until 2018-02-18 09:21:44. +0000 UTC m=+. (durationBeforeRetry 500ms). Error: "UnmapVolume.UnmapDevice on global map path failed for volume "blkdata" (UniqueName: "") pod "-148c-11e8-b315-" (UID: "-148c-11e8-b315-") : Failed to unmap device from map path. mapPath is empty" Feb 18 09:21:43 kubernetes-minion-group-crj1 kubelet[1510]: I0218 09:21:43. 1510 ] operationExecutor.UnmountVolume started for volume "blkdata" (UniqueName: "") pod "-148c-11e8-b315-" (UID: "-148c-11e8-b315-") Environment: Kubernetes version (use ): 1.10+ Cloud provider or hardware configuration: GCE OS (e.g. from /etc/os-release): "Container-Optimized OS from Google" Kernel (e.g. ): Linux 4.4.86+ Install tools: Others:
/sig storage
This error is returned in this . It same that we set the global mount path to only if the volume plugin is attachable(see ).


Question: glog.Infof(volumeToMount.GenerateMsgDetailed("MapVolume.WaitForAttach succeeded", fmt.Sprintf("DevicePath %q", devicePath))) <del> // Update actual state of world to reflect volume is globally mounted markDeviceMappedErr := actualStateOfWorld.MarkDeviceAsMounted( volumeToMount.VolumeName, devicePath, globalMapPath) if markDeviceMappedErr != nil { // On failure, return error. Caller will log and retry. return volumeToMount.GenerateError("MapVolume.MarkDeviceAsMounted failed", markDeviceMappedErr) } </del> } // A plugin doesn't have attacher also needs to map device to global map path with SetUpDevice() pluginDevicePath, mapErr := blockVolumeMapper.SetUpDevice()

However, local volume plugin is not attachable, so will be empty and the operation will fail. I think we should extract the operation from the block, and execute this operation just before we try to map the device to global map path, then this global map path can be cached in for both attachable and non attachable volume plugins. Any comments? If not, I can help to fix this.
Thank you for the investigation. We've discussed this problem by offline yesterday. I proposed same fix which you mentioned above, then tested the fix works correctly with local volume plugin. I'll push a PR soon.
The device path appears to be empty, causing the unmap to fail. Please note, which was stored in is empty. device path isn't related to this problem.
Sorry about that, this is a typo, what I mean is global mount path.
No, no. My above comment is just note for the section. Your investigation is correct.


Question: "fmt" "io" "io/ioutil" <ins> "testing" </ins> "github.com/GoogleCloudPlatform/kubernetes/pkg/api" "github.com/GoogleCloudPlatform/kubernetes/pkg/api/latest"

I stumbled across this while experimenting with kubeconfig load order/merging. There are various helpers constructed from a Factory that use , however this always uses the same default ClientConfig because doesn't use the passed in version param: Am I missing something or is this a bug? /cc
Looks like a bug, clientforversion should accept "" (which can provide a default) and SetKubernetesDefaults has the final fallback protection (by using latest.Version). If version != defaultConfig.Version, we should copy defaultConfig, set version, then apply kubedefaults. The reason we take version here is if a resource is specified via file which only exists in a newer version, ie user says: --api-version=v1beta1 -f RESTMapper can handle that newresource is only valid for v1beta3 and apply the right override, then ask for client. --api-version is expected to bubble up via default config.


Question: set -o nounset set -o pipefail <del> if [[ -z "$1" ]]; then echo "Usage: $0 <version>" </del> <ins> if [ "$#" -ne 1 ]; then echo "Usage: ${0} <version>" </ins> exit 1 fi <del> if ! gsutil ls gs://kubernetes-release/release/${1}/kubernetes.tar.gz; then echo "Release files don't exist, aborting." exit 2 fi STABLE_FILE_LOCATION="kubernetes-release/release/stable.txt" </del> <ins> KUBE_RELEASE_VERSION="${1-}" </ins> <del> version_file=$(mktemp -t stable.XXXXXX) </del> <ins> KUBE_GCS_MAKE_PUBLIC='y' KUBE_GCS_RELEASE_BUCKET='kubernetes-release' KUBE_GCS_RELEASE_PREFIX="release/${KUBE_RELEASE_VERSION}" KUBE_GCS_PUBLISH_VERSION="${KUBE_RELEASE_VERSION}" </ins> <del> echo $1 >> ${version_file} echo "Uploading stable version $1 to google storage" gsutil cp ${version_file} "gs://${STABLE_FILE_LOCATION}" echo "Making it world readable" gsutil acl ch -R -g all:R "gs://${STABLE_FILE_LOCATION}" </del> <ins> KUBE_ROOT="$(dirname "${BASH_SOURCE}")/.." source "${KUBE_ROOT}/build/common.sh" </ins> <del> rm ${version_file} value=$(curl -s https://storage.googleapis.com/${STABLE_FILE_LOCATION}) echo "Validating version file" if [[ "${value}" != "${1}" ]]; then echo "Error validating upload, :${value}: vs expected :${1}:" </del> <ins> if "${KUBE_ROOT}/cluster/kubectl.sh" 'version' | grep 'Client' | grep 'dirty'; then echo "!!! Tag at invalid point, or something else is bad. Build is dirty. Don't push this build." >&2 </ins> exit 1 fi <ins> kube::release::gcs::publish_official 'stable' </ins>

(cc Despite doing very, very similar things, doesn't rely on . It should, or the logic should otherwise be consolidated. I'm going to maybe work on this to make easier.


Question: # See the License for the specific language governing permissions and # limitations under the License. <del> .PHONY: all image push clean </del> <ins> SRCS = nnp ARCH ?= amd64 TARGET ?= $(CURDIR) GOLANG_VERSION ?= latest SRC_DIR = $(notdir $(shell pwd)) export </ins> <del> TAG = 1.2 PREFIX = gcr.io/google_containers </del> <ins> bin: ../image-util.sh bin $(SRCS) </ins> <del> all: push nnp: nnp.c gcc -static -o $@ $@.c image: nnp docker build --pull -t $(PREFIX)/nonewprivs:$(TAG) . push: image gcloud docker -- push $(PREFIX)/nonewprivs:$(TAG) clean: rm -f nnp </del> <ins> .PHONY: bin </ins>

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind feature This issue is for creating image as multiarch
/assign
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig testing


Question: // volumeConfigName is the configmap passed to bootstrapper and provisioner volumeConfigName = "local-volume-config" // bootstrapper and provisioner images used for e2e tests <del> bootstrapperImageName = "quay.io/external_storage/local-volume-provisioner-bootstrap:v1.0.1" provisionerImageName  = "quay.io/external_storage/local-volume-provisioner:v1.0.1" </del> <ins> bootstrapperImageName = "quay.io/external_storage/local-volume-provisioner-bootstrap:v2.0.0" provisionerImageName  = "quay.io/external_storage/local-volume-provisioner:v2.0.0" </ins> // provisioner daemonSetName name, must match the one defined in bootstrapper daemonSetName = "local-volume-provisioner" // provisioner node/pv cluster role binding, must match the one defined in bootstrapper

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: The local-volume bootstrapper/provisioner configuration format changed in kubernetes-incubator/external-storage, and its related e2e tests will fail when the provisioner & bootstrapper images are updated to v2.0.0. What you expected to happen: I'd expect the e2e tests to continue to work properly with the new version of the provisioner and boostrapper. How to reproduce it (as minimally and precisely as possible): Update local-volume provisioner and boostrapper images to v2.0.0, and then the provisioner e2e test will fail.
/sig storage
/assign