Question: cpuQuery                  = "Processor(_Total)% Processor Time" memoryPrivWorkingSetQuery = "Process(_Total)Working Set - Private" memoryCommittedBytesQuery = "MemoryCommitted Bytes" <del> // Perf counters are updated every second. This is the same as the default cadvisor collection period // see https://github.com/google/cadvisor/blob/master/docs/runtime_options.md#housekeeping perfCounterUpdatePeriod = 1 * time.Second </del> <ins> // Perf counters are updated 10 seconds. This is the same as the default cadvisor housekeeping interval // set at https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/cadvisor/cadvisor_linux.go perfCounterUpdatePeriod = 10 * time.Second </ins> // defaultCachePeriod is the default cache period for each cpuUsage. // This matches with the cadvisor setting and the time interval we use for containers. // see https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/cadvisor/cadvisor_linux.go#L63

Perf counters are currently updated every second which doesn't match the default cadvisor collection period as 10 seconds. Increase the update interval of perf counters to 10 seconds to match the house keeping interval of cadvisor. The fresh install of our internal AKS clusters. The perf counter query (pdh!PdhCollectQueryData) is responsible for 20% of CPU time in kubelet process on internal clusters, which can be reduced by lowering the perf counter update interval. <details</details<detailsAzure Kubernetes Service (AKS) </details<detailsOperating System: Windows Server 2019 Datacenter OS Build Number: .3770.amd64fre.rs5release.-1434 </details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node /sig windows
/area cadvisor Since it is a performance overhead and not a bug per se, I will change the kind of this issue: /remove-kind bug /kind feature I bet there is an issue for this already. Can you comment/dedup?
I believe the ask here is to increase the which is only used on Windows to fetch stats from windows specific per counters. Windows does not use cAdvisor and instead uses which reads from the perf counters directly. /cc
(removing cadvisor since it is ask for windows stats) /remove-area cadvisor
agreed that is the ask. I don't have any major issue with it but would be worried about change in behavior for existing users. What are our options for making this configurable?
That would be great if making it configurable.
It might not be worth the overhead to pass this as configuration through kubelet parameters. I am unsure what affect it would have on folks who rely on it for current reporting or if it would be ok to bump the querying time. if you increase it to 10s do you see a large drop in cpu usage?


Question: cpuQuery                  = "Processor(_Total)% Processor Time" memoryPrivWorkingSetQuery = "Process(_Total)Working Set - Private" memoryCommittedBytesQuery = "MemoryCommitted Bytes" <del> // Perf counters are updated every second. This is the same as the default cadvisor collection period // see https://github.com/google/cadvisor/blob/master/docs/runtime_options.md#housekeeping perfCounterUpdatePeriod = 1 * time.Second </del> <ins> // Perf counters are updated 10 seconds. This is the same as the default cadvisor housekeeping interval // set at https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/cadvisor/cadvisor_linux.go perfCounterUpdatePeriod = 10 * time.Second </ins> // defaultCachePeriod is the default cache period for each cpuUsage. // This matches with the cadvisor setting and the time interval we use for containers. // see https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/cadvisor/cadvisor_linux.go#L63

might have thoughts as well
Based on internal cloud profiling data from thousands of Windows clusters, pdh!PdhCollectQueryData takes 33% CPU usage as of today, 01/20/2023. If the updating interval increases from 1s to 10s, pdh!PdhCollectQueryData usage can be reduced 90% and CPU usage of kubelet can potentially drop 30%.
that is a compelling improvement. I am still concerned about changing the default on folks who might rely on it, I am not sure if it is a valid concern though
To understand the impact of changing the interval, what is the purpose for querying those Windows counters?
- Can you share a little bit more information about your setup? What node size are you using in AKS? Are you overprovisioning your nodes? Do you have any addons running in your AKS cluster? Does the drop eventually go down? Have you seen this issue on Windows Server 2022 too? We don't see this spike in CPU usage in our clusters provisioned with CAPZ (cluster api provider for azure) so I suspect something else is going on too. https://perf- I'm not opposed to increasing the timeout but currently I think this issue may be environmental.
1 The node size is D32_Flex which has 32 vCPUs and 128 GiB memory. 2 The perf counter query usage is the same across all clusters regardless of the cluster is running hot or cold. 3 azurepolicy is the only addon enabled. 4 The trending of perf counter query usage is quite stable. 5 Not applicable at this moment, all clusters are running with Windows Server 2019. Do you have CPU traces from CAPZ to check the time spent on PdhCollectQueryData? For our internal profiling data, perf counter usage is pretty stable in kubelet, and it's not related to the CPU spike.
I am not sure I understand this statement. The cpu usage from the the winstats pkg would be attributed to the kubelet since it is the own process. What is the owning process for usage you are seeing?
usage is from kubelet process itself. One thing I misunderstood is that, the usage in kubelet is not stable. Actually, it keeps growing.


Question: cpuQuery                  = "Processor(_Total)% Processor Time" memoryPrivWorkingSetQuery = "Process(_Total)Working Set - Private" memoryCommittedBytesQuery = "MemoryCommitted Bytes" <del> // Perf counters are updated every second. This is the same as the default cadvisor collection period // see https://github.com/google/cadvisor/blob/master/docs/runtime_options.md#housekeeping perfCounterUpdatePeriod = 1 * time.Second </del> <ins> // Perf counters are updated 10 seconds. This is the same as the default cadvisor housekeeping interval // set at https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/cadvisor/cadvisor_linux.go perfCounterUpdatePeriod = 10 * time.Second </ins> // defaultCachePeriod is the default cache period for each cpuUsage. // This matches with the cadvisor setting and the time interval we use for containers. // see https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/cadvisor/cadvisor_linux.go#L63

The usage is almost zero originally, but it grows up to 30+% cpu usage in kubelet. Looking into it...
The tool wasn't getting the correct data at the first time. It turns out usage in kubelet is growing from 20% to 30+% across all clusters, and the usage is never close to zero. I will investigate the growing cost of . It'd be great if we can increase the query interval to reduce the cost.


Question: UID:               types.UID("a671734a-e8e5-11e4-8fde-42010af09327"), ResourceVersion:   "22", CreationTimestamp: parseTimeOrDie("2015-04-22T11:49:36Z"), <del> Annotations: map[string]string{ "kubernetes.io/config.mirror": "mirror", "kubernetes.io/config.source": "file", }, </del> }, Spec: api.PodSpec{ Volumes: []api.Volume{

Nodes can't recognize the static pods that they create possibly because the defaulting is different. always returns false. Here is a json diff of two pod specs before and after they are Semenatic.DeepEqual'd where before is created pod and after is what kubelet things the created pod should be. ! Thus: p0 for triage
I think the solution is to not apply defaults to mirror pods in the API Server.
Yes,we have to add specific code to API Server to detect the mirror pod, then not apply defaults.
The right fix should be , which we didn't get to before v1.0. I will start on this and at least get it in for v1.1 to prevent future complications. Since we cannot update the node/kubelet, options are: suggests to disable applying defaults to mirror pods as a temporary fix. IIRC, was against treating mirror pod differently at the apiserver. Since this is only a workaround, perhaps we can relax the constraint? a script to update the pod manifest files.
Disabling defaults has a number of problems: are version-specific. Conversion may fail if defaults are not applied. may fail if defaults are not applied. using defaulted fields assumes default values have been applied. defaulting pass is not the only place where values can be set, as mentioned in . For instance, if the user were to change the kube-system LimitRange, that would cause the same problem. This is another case where would have been useful. Which gives me another idea. For mirror pods, we could capture the entire metadata+spec provided by the user as an annotation, as we do in the client for kubectl apply, and then we could return that in GET. PUT may not work. We'd have to resolve the size concerns raised by . How serious of a problem is this delete-recreate loop?
Ok, we are going with approach proposed in . I am closing this issue as dup of
"entire metadata+spec" in plain text? It'd be hard for kubelet to diff using this. It'd be better to let kubelet store hash of its own choosing so that it can easily compare when there is any change.
Even with , we can't fix v1.0 kubelet.
I agree with
Yes, I were confused.


Question: UID:               types.UID("a671734a-e8e5-11e4-8fde-42010af09327"), ResourceVersion:   "22", CreationTimestamp: parseTimeOrDie("2015-04-22T11:49:36Z"), <del> Annotations: map[string]string{ "kubernetes.io/config.mirror": "mirror", "kubernetes.io/config.source": "file", }, </del> }, Spec: api.PodSpec{ Volumes: []api.Volume{

The comparison is in Kubelet, and we cannot patch 1.0 kubelet in this case.
There are no serious issues with this bug: static pods are running properly, but corresponding mirror pods coming and gone. cc/ Can we treat this as part of GKE cluster upgrade process? Can we upgrade GKE master nodes, then ask the users to upgrade their fluentd yaml file in their cluster? For OSS kubernetes users, most likely they won't run into this issue. But we should document this as known issue for 1.1.
Or can we update the fluentd yaml files when updating the master?
cc/
This is a cc issue, really. But the fast answer is no, because "then ask the users to upgrade their fluentd yaml file in their cluster?" Historically our users are extremely slow to do maintenance.
Our current plan for this issue is just document as a known issue for 1.1 release.
In a 100 node cluster this is going to cause +100 writes +100 deletes per second. Is that extra load a concern?
This would be +100 writes +100 deletes every 10 seconds in a 100-node cluster. We are still discussing options to let user update their manifest files.
Ok, from the logs in the issue description, it looked more frequent but most of the deletes are failing with key not found
's easy to trigger a sync for all pods in kubelet. Any change in any pod would trigger a sync. In this case, the DELETE/ADD of the mirror pod could also trigger another sync. So, yes, it could be worse than 10s... We are looking at patching this in the apiserver, based on suggestion.
Yeah, my suggestion is fixing defaulting/conversion/validation as needed to make exceptions for mirror pods.
I'm not 100% sure it's needed, but if necessary we can turn the defaulting fix on in apiserver, and leave it off in scheduler, so that it gets the defaults it expects.
Yeah, it is way more frequent than 10s. I pointed this out to earlier. We had a meeting with and looks like: Providing a scripts to convert fluentd yaml file and asking the user to run is not a solution. Our users are very slow on such maintenance work. Piggyback such conversion work on master updating might not a solution for GKE clusters.


Question: UID:               types.UID("a671734a-e8e5-11e4-8fde-42010af09327"), ResourceVersion:   "22", CreationTimestamp: parseTimeOrDie("2015-04-22T11:49:36Z"), <del> Annotations: map[string]string{ "kubernetes.io/config.mirror": "mirror", "kubernetes.io/config.source": "file", }, </del> }, Spec: api.PodSpec{ Volumes: []api.Volume{

We do have access to GKE nodes, and we don't do that. To obtain the permission to run script and sync a file requires more work from PM side. This is a API backward compatibility issue anyway. Given today's version management policy, we should handle such compatibility issue. I checked our default code: request diff was introduced by TerminationGracePeriodSeconds was introduced along with graceful termination support. There is no conversion for both fields There is no validation checking both fields today I think it is safe to disable such defaulting for mirror pods shouldn't cause any issue for 1.1 release. Sending PR shortly.
In order to not break the v1.1+ kubelet (which applies newer defaults), we should only skip defaulting for pods from pre-v1.1 kubelets. An easy way to do this is to reuse , which changes the mirror key annotation value from "mirror" to the hash of the pod manifest file. In short, if the key exists, it's a mirror pod. If the value is "mirror", it is sent by a pre-v1.1 kubelet. Otherwise, it's sent by a v1.1+ kubelet. Note that we will also need to maintain the special defaulting case for the future releases until we drop support for v1.0 nodes.
Just this morning, we discussed providing a script/tool to update static pod configs for people with large clusters. Why have we changed that plan? Any hack in apiserver is going to be confusing and fragile. For instance, a LimitRange installed for kube-system would defeat the hack. If we were to add a hack in apiserver, we'd need to remove it in 1.2.
we talked to and the conclusion was that node update may not be feasible as . I agree that any apiserver fix is hacky and fragile, but it seems the most viable option right now. We would need to carry this fix in apiserver as long as GKE supports v1.N master + v1.0 node, unless we can make updating the configs part of the cluster upgrade process.
Ah. These fields are problems because they existed in 1.0, but weren't used nor populated. Entirely new fields are silently dropped during decoding.


Question: UID:               types.UID("a671734a-e8e5-11e4-8fde-42010af09327"), ResourceVersion:   "22", CreationTimestamp: parseTimeOrDie("2015-04-22T11:49:36Z"), <del> Annotations: map[string]string{ "kubernetes.io/config.mirror": "mirror", "kubernetes.io/config.source": "file", }, </del> }, Spec: api.PodSpec{ Volumes: []api.Volume{

We should cherrypick the solution into 1.0.7, since it appears there will be one.
That is, we should cherrypick into the 1.0 branch.
Ref.
If the hack is going to remain indefinitely, I'm not comfortable circumventing defaulting, conversion, and validation.
Just not performing the defaulting for these specific fields on just mirror pods seems safer.
In particular, I could easily imagine a new version of the pod API in 1.2 or 1.3, which will require defaulting and conversion to be applied, in general.
We'll be discussing this soon after 1.1 lands, I expect. My proposal would be for v1.N masters to support v1.N-1 and v1.N-2 (and maybe v1.N-3) nodes. That gives users 6-9 months to lag their nodes behind their masters. But it warrants much further discussion. This isn't a viable solution either, because we can't depend on users updating their nodes to the latest 1.0 rev. We expect arbitrary 1.0 nodes to work, including 1.0.6 all the way back to 1.0.1, per our current .
As Ike noted, cherrypicking into 1.0 kubelets is not feasible for solving this, because the whole reason for the problem is that people aren't updating their nodes. The conversion fix in is much preferable to the defaulting fix: it's less brittle, and components that need the defaulting (e.g. scheduler) will see the defaulting (because they run the conversion stack themselves), but on the wire the mirror pods from old kubelets will remain undefaulted.
I didn't say cherrypicking into 1.0 would solve the problem for everything. The behavior in 1.0 is pretty broken, even when not considering upgrade. We have a fix. We should cherrypick it into 1.0.
I don't see how different components would see different values for these fields, assuming they are all updated to the new code. As discussed offline, the scheduler and node controller need to handle cases of empty requests and pod termination for 1.0 kubelets, anyway, so I think the conversion approach is fine. Please put the explanation in code comments, though, not just in issue/PR comments.


Question: UID:               types.UID("a671734a-e8e5-11e4-8fde-42010af09327"), ResourceVersion:   "22", CreationTimestamp: parseTimeOrDie("2015-04-22T11:49:36Z"), <del> Annotations: map[string]string{ "kubernetes.io/config.mirror": "mirror", "kubernetes.io/config.source": "file", }, </del> }, Spec: api.PodSpec{ Volumes: []api.Volume{

Since the conversion stack does the defaulting at the end and runs in every component, and scheduler, node-controller, etc all consume api.Pod objects (instead of v1.Pod objects), they will have a view of the object with the fields defaulted. The conversion based fix essentially retains the defaulting in *api.Pod objects, but undoes it in any serialized representation of the object. Over the wire the defaulting will not be baked in to the object. Older clients that have not been recompiled (including these kubelets) would see a version of the object without the fields being defaulted.
Defaulting is run first:
Oh, you're right, I somehow had that backwards in my head. Sorry, everyone. I think that tilts me back in favor of . This means that defaulting is part of the versioned structs and therefore changing the defaulting is an API breaking change, which we claimed we wouldn't do. Can we change our minds about the defaulting? How about moving this particular defaulting into conversion functions, which is the intended way of doing something like this?
In , we only reset the defaults when converting api.Pod to v1.Pod. Shouldn't all components still see the defaulting values? Agreed that this is an API breaking change. Perhaps we should not change the defaults at all? mentioned in that scheduler already deals with empty requests. It's true in , but seems untrue in . Maybe this should be fixed. As for , we can also ask kubelet to apply a default value if we truly on to have the 30 second dfault.
resources.requests and terminationGracePeriodSeconds are effectively newly fields, so setting defaults is not a breaking change for most clients. If Kubelet were comparing the pre-decoded pods returned by apisever, it would barf on every new field we've . It's an accident of the implementation that it broke on exactly these fields.
Actually, neither priorities nor predicates takes values from limits any longer. That code is rely on the defaulting pass to copy limits to requests in the case that the requests field is absent. We would need to change that, yes, if we were to not set requests for the 1.0 mirror pods.
This Kubelet behavior was known to be broken back in May: What we're trying to do now is figure out a workaround for this broken behavior.


Question: UID:               types.UID("a671734a-e8e5-11e4-8fde-42010af09327"), ResourceVersion:   "22", CreationTimestamp: parseTimeOrDie("2015-04-22T11:49:36Z"), <del> Annotations: map[string]string{ "kubernetes.io/config.mirror": "mirror", "kubernetes.io/config.source": "file", }, </del> }, Spec: api.PodSpec{ Volumes: []api.Volume{

My recommended solution is: not set resources.requests and terminationGracePeriodSeconds to non-nil values for just 1.0 mirror pods. All other fields and all other pods, including 1.1 mirror pods, should have default values set normally. Doing this in the defaulting pass makes the most sense, assuming that it's straightforward to do given the structure of that pass. the scheduler fit and priority functions to use limits for pods where requests are entirely absent. that the new code in the node controller to handle termination of pods on 1.0 kubelets deals with unset terminationGracePeriodSeconds appropriately. a kind/workaround-removal issue to remove these hacks once we don't need to worry about 1.0 kubelets.
Just for comparison, an alternative: this default setting from the defaulting function. it to the conversion function instead. mirror pods from 1.0 kubelets (MPF1K), store the original value in an annotation. MPF1K, when converting to v1, restore the original value & remove the annotation. No changes needed anywhere else in the system, maintenance cost is very low, we can carry this as long as needed.
I don't see how that keeps the default values off the wire.
Actually, I think I see. Yes, I think that could work.
The defaults are on the v1-internal transition, and removed on the internal-v1 transition.
(3) is converting v1 to IR, and (4) is converting IR to v1, either for storing to etcd or to communicate to another component, such as Kubelet.
I'm saying stop using the default setting code for setting these "breaking" defaults.
Right.
I don't think we want to do this for pods in general, though, just for MPF1K
So I think we can actually just add a few more changes to PR and it should work. (Just move the defaulting.)
Yes, I agree. Just for MPF1K. We can easily broaden it if needed.
SGTM
We're going to want to do something similar for new fields to alpha API versions that we don't want to expose in the stable, serialized version of the API.
So if this works, we should add it to
Yes. Changing defaults is a breaking change and therefore needs to happen in conversion.


Question: UID:               types.UID("a671734a-e8e5-11e4-8fde-42010af09327"), ResourceVersion:   "22", CreationTimestamp: parseTimeOrDie("2015-04-22T11:49:36Z"), <del> Annotations: map[string]string{ "kubernetes.io/config.mirror": "mirror", "kubernetes.io/config.source": "file", }, </del> }, Spec: api.PodSpec{ Volumes: []api.Volume{

Adding new fields with new defaults is fine, just not changing the behavior of existing fields. course I do think kubelet is being a very inflexible client here, but we can't fix the stuff that's already out there.
+1 to both points. We should've patched kubelet sooner to avoid this, but it's out there and kubelet 1.0 remains a stubborn client. Sorry for that. thanks for coming up with the solution! Since both of you agreed on , I will change my PR acoordingly.
So that there is no misunderstanding that leads to future such problems: Whether fields are new or pre-existing, it is an invalid assumption that unset fields may not be changed, whether by defaulting or by some other process, such as admission control.
This issue will be considered fixed for 1.1 once is submitted. (I didn't read this thread carefully enough yet to know whether additional work is necessary before considering this whole issue closed.)
Update: has just been merged. Closing the issue.
Probably we should not close this issue until we do a cherrypick?
The change was cherrypicked, so closing this issue


Question: if obj.SchedulerName == "" { obj.SchedulerName = v1.DefaultSchedulerName } <del> if obj.EnableServiceLinks == nil { enableServiceLinks := v1.DefaultEnableServiceLinks obj.EnableServiceLinks = &enableServiceLinks } </del> } func SetDefaults_Probe(obj *v1.Probe) { if obj.TimeoutSeconds == 0 {

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug The change in a new field () to the Pod spec that is set to true by default. This is causing problems for DaemonSets in some situations. DaemonSets are using ControllerRevisions to keep track of previous configurations and each revision will contain among other things the Pod spec for the DaemonSet. If a DaemonSet is created with the enableServiceLinks property is set to true, the ControllerRevision will also contain this field. If the cluster is rolled back to a previous version of kubernetes, the DeamonSet struct does not have this field and so the ControllerRevision and the DaemonSet will no longer match. This leads to a new ControllerRevision being created and a rolling update of all pods in the DaemonSet. Would it be a better solution to rename the field to and enable service links whenever the field is not set? This would fix the problem with the DaemonSets unless the field is explicitly set before the rollback. This is causing the master upgrade tests to fail as reported in /sig apps /cc /cc /cc
I guess the change in podSpec and the defaulting will break all the workload resources as they are based on comparing the template's podSpec. ~Is the new value defaulted to true also for the DaemonSet's podSpec template? If not, controller might go hot loop like with mutating admission controllers.~ I don't like us making such changes to v1 podSpec. The other way around seems more sensible. /cc
Why is this a bad thing? We will continue to add fields to podSpec - the issue isn't the defaulting (this would happen regardless of which way it defaulted), it's that the controller can't really be version aware. The new value would default to true in the daemonset as well.
I'm not sure that a rollback requiring daemonsets to recreate is a bug. I think that's potentially "the best possible thing that we can do in that case".
/cc SIG Release blocking dashboard with upgrade test failures - https://k8s-
So the problem isn't the fact that a field has been . It the way in which the field was .


Question: if obj.SchedulerName == "" { obj.SchedulerName = v1.DefaultSchedulerName } <del> if obj.EnableServiceLinks == nil { enableServiceLinks := v1.DefaultEnableServiceLinks obj.EnableServiceLinks = &enableServiceLinks } </del> } func SetDefaults_Probe(obj *v1.Probe) { if obj.TimeoutSeconds == 0 {

enableServiceLinks is bool pointer that defaults to true. To be backward compatible it should be something like where the new default behavior is to assume that service links are enabled, and the field should be set on an opt out basis to disable the service links. compatiblity As it is if you create a new DaemonSet and then roll back cluster to a prior version, or if you have a failover in a HA setup, you can trigger a simultaneous rolling restart of all DaemonSets you created even though you didn't explicitly configure (and may not even be aware of) the new feature. If we respect backward compatibility and make the feature explicitly opt-out (that is the semantics we are going for here in either ca So the problem isn't the fact that a field has been . It the way in which the field was . enableServiceLinks is bool pointer that defaults to true. To be backward compatible it should be something like where the new default behavior is to assume that service links are enabled, and the field should be set on an opt out basis to disable the service links. compatiblity As it is if you create a new DaemonSet and then roll back cluster to a prior version, or if you have a failover in a HA setup, you can trigger a simultaneous rolling restart of all DaemonSets you created even though you didn't explicitly configure (and may not even be aware of) the new feature. If we respect backward compatibility and make the feature explicitly opt-out (that is the semantics we are going for here in either case), you won't trigger the rolling restart on rollback or controller manager failoverse), you won't trigger the rolling restart on rollback or controller manager failover unless you explicitly opt out of the behavior. I don't see why the former behavior is more desirable than the latter.
For background/context, the choice between enable/disable was initially discussed here:
Are you saying that the issue is that default value is not the same as the Go "zero value" ? The concept of zero value is interesting but it forces APIs to bend into unnatural shapes to respect it. Since forever Kubernetes has explicitly said that we have version-specific default values which ARE NOT governed by zero values, but instead are governed by the API version. It is safe to assume that if a field is documented as having a default in a version, you can rely on that field being populated with that default, with a few exceptions.


Question: if obj.SchedulerName == "" { obj.SchedulerName = v1.DefaultSchedulerName } <del> if obj.EnableServiceLinks == nil { enableServiceLinks := v1.DefaultEnableServiceLinks obj.EnableServiceLinks = &enableServiceLinks } </del> } func SetDefaults_Probe(obj *v1.Probe) { if obj.TimeoutSeconds == 0 {

We generally ask for boolean fields to be "in the affirmative direction", as in this case. Now, we could revisit this. It would be a significant change in direction, but it's not impossible. But I don't think that solves your problem in general -- if this field were optional string, with a default value of "Hello there", you would be in the same boat right?
So the underlying issue is that we are defaulting into the PodSpec which gets adopted into all of the PodTemplateSpecs of every object. That sort of defaults the users declared intent. Why not just default the value into the Pod itself . Is the intention really to default this in all Templates or to make it optional in the Template and default it in the Pod itself?
/milestone v1.13
I can't speak to the ideology of whether to default the field in all PodSpecs or just the Pod, but your suggestion makes sense to me. I'm not sure of the past reasoning for adding other defaulted fields, most recently SchedulerName in Jan. 2017 (), to all PodSpecs instead of just to the Pod object. Did we not have these same tests in place then? This would effectively be saying we should never add new defaulted fields to PodSpec. If this is the plan, I can create a PR for the fix and I'll add a big warning comment to SetDefaults_PodSpec.
As an example of where we don't default the template we have a CronJob that embeds JobTemplate, we never default that template, only an actual Job gets defaulting. I agree with that not defaulting any templates does make sense.
Agree to only default this in pods. Agree to adding the comment. Most important defaulting happens in pods to keep higher level pod specs simpler. We may need to raise the general pattern in a discussion with api review in general, since this is a change from previous strategy, but the previous strategy was not explicitly chosen (to my recollection at least)
Hi all do we have an agreed path forward for fixing this issue in 1.13? if so can we have an ETA for the fix to land? the release team would like for the tests to get passing in the blocking suites in an effort to unmask any other failures that might show up once these are fixed.


Question: if obj.SchedulerName == "" { obj.SchedulerName = v1.DefaultSchedulerName } <del> if obj.EnableServiceLinks == nil { enableServiceLinks := v1.DefaultEnableServiceLinks obj.EnableServiceLinks = &enableServiceLinks } </del> } func SetDefaults_Probe(obj *v1.Probe) { if obj.TimeoutSeconds == 0 {

/assign
We have a PR with a fix that will hopefully merge today or tomorrow. I will keep an eye on the dashboard to make sure it fixes the failing daemonset tests.
Thanks Can we link the PR here plz. /cc
Reopening, because of the auto-close. Tests are not passing yet. /reopen
Reopening this issue. <detailsIn response to : Reopening, because of the auto-close. Tests are not passing yet. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
PR that should (hopefully) close this issue is here:
DaemonSet tests passed on run 1842, seems to have been fixed: https://k8s- if it doesn't pop up again - /close
Closing this issue. <detailsIn response to : DaemonSet tests passed on run 1842, seems to have been fixed: https://k8s- Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: import ( "encoding/json" "fmt" <del> "hash/adler32" </del> "time" "github.com/golang/glog"

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT Kubernetes version (use ): Environment: Cloud provider or hardware configuration: AWS OS (e.g. from /etc/os-release): Debian GNU/Linux 8 (jessie) Kernel (e.g. ): 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux What happened: Deployed a scheduled job from this manifest: After some time the jobs are no longer executed: What you expected to happen: Jobs are executed as defined in the manifest. How to reproduce it (as minimally and precisely as possible): Deploy a scheduled job and wait for some time.
Some logs from the controller manager:
Ping. More than 2 weeks and no action at all? Anything else i can do?
I'm also encountering this using both and as concurrency policies, fwiw.
I'm hitting this as well. Kube
Do the jobs start to reschedule if you delete old one? Try running . cc
From kube-controller-manager logs: The job is scheduled for 1:00. I think the "Multiple unmet start times" message is a clue of where things start going downhill. Also, notice how at 1:00 the controller wants to create job (for a whole day, it had failed on ).
Deleting (all) old jobs / pods fixes the problem temporarily. After some time the scheduled jobs face the same issue again.
There's a fix () that I've submitted against master to fix the Replace strategy, I'll mark it for inclusion in 1.4.
The other problem that I see here is this: It is possible that our is creating duplicates. I'll try to investigate that a bit more.
sorry for the delay in response time, I must have missed that while going through issues. Next time you're experiencing problems/have a question wrt Jobs/ScheduledJobs, just ping me directly in the issue.
No problem and I will. Thanks!
I'm still hitting this on
Fix in flight , I'll make sure to cherry-pick it back to 1.4
does this ship in to latest kubernetes release?
yes, this should be present in 1.5, 1.4 cherry-pick hasn't been merged yet.
so this issue is still present in v1.4.6?


Question: import ( "encoding/json" "fmt" <del> "hash/adler32" </del> "time" "github.com/golang/glog"

yes - it is still present there. My fix was merged to 1.4 yesterday. You should wait for the next 1.4.x release.
thanks!
which according to should happen Dec 8th.
any news on the release?
See !msg/kubernetes-dev/o0At2-4KKQ8/_kPVM1XyDAAJ
so v1.4.7 will be released on that same day?
Look like should be available later today: !msg/kubernetes-dev/00lJpT9pxDc/YZfDzCxMDQAJ


Question: Recursive: true, } r.RawMustRegister(metrics.NewPrometheusCollector(prometheusHostAdapter{s.host}, containerPrometheusLabelsFunc(s.host), includedMetrics, clock.RealClock{}, cadvisorOpts)) <ins> r.RawMustRegister(metrics.NewPrometheusMachineCollector(prometheusHostAdapter{s.host}, includedMetrics)) </ins> s.restfulCont.Handle(cadvisorMetricsPath, compbasemetrics.HandlerFor(r, compbasemetrics.HandlerOpts{ErrorHandling: compbasemetrics.ContinueOnError}), )

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: prometheus cadvisor machine metrics from kubelet are missing What you expected to happen: How to reproduce it (as minimally and precisely as possible): install k8s 1.19, compare behavior to k8s <1.19 Anything else we need to know?: Looks like missing call to in , machine metrics collection was split out in cadvisor 0.37 Environment: Kubernetes version (use ):
/sig node (hope that's the correct sig :)
Suggest from as an alternative (declared as "stable").
Related PR Their output format changed too:
Multiple queries is invalide now. Is there any workaroud for this? Please help. Thanks.
if you run see for an alternative metric.
+1 on the issue. This issue has caused the node metric collection for Azure Monitor for Containers to be broken which also impacts alerting.
Exposed metrics are a kind of an API and should not be removed without warning.
any news, when this will be fixed? Thanks in advance!
It is not possible to have a metric available without exposing it. To provide stability information on metrics, similar to APIs, you might want to read up on the . There currently are no metrics in Kubernetes itself that are considered stable, but SIG Instrumentation hopes to be selecting some for the 1.21 release. This particular issue seems like an unintended bug as opposed to an intentional deprecation. Suspect this issue got lost in SIG Node's backlog. I'll put it on the agenda for the next SIG Instrumentation meeting. /sig instrumentation
Backports in and


Question: It("processes in different containers of the same pod should be able to see each other", func() { // TODO(yguo0905): Change this test to run unless the runtime is // Docker and its version is <1.13. <del> By("Check whether shared PID namespace is enabled.") isEnabled, err := isSharedPIDNamespaceEnabled() </del> <ins> By("Check whether shared PID namespace is supported.") isEnabled, err := isSharedPIDNamespaceSupported() </ins> framework.ExpectNoError(err) if !isEnabled { <del> framework.Skipf("Skipped because shared PID namespace is not enabled.") </del> <ins> framework.Skipf("Skipped because shared PID namespace is not supported by this docker version.") </ins> } By("Create a pod with two containers.")

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --BUG REPORT: What happened: I'm running a simple Kubernetes cluster with master and one node. All basic PODs are deploying successfully (ie. weave-net) but when I try to deploy GlusterFS as a POD - container creation fails with error: This is a result of failing to perform from docker file. The weird part is that if I run this image directly on node through docker it runs smoothly. What you expected to happen: I'd expect to be able to deploy GlusterFS through Kubernetess properly. How to reproduce it (as minimally and precisely as possible): Setup minimal cluster and try to deploy the follwing POD Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration**: virtualbox OS (e.g. from /etc/os-release):Kernel (e.g. ):Docker versionInstall tools: vagrant Others:
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig node
Reiterating the mentions to trigger a notification: <detailsIn response to : /sig node Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
One update - I can deploy this POD with no problems on version of k8s.
I am hitting the same bug on a freshly installed k8s cluster running on installed via kubeadm plus weave-net with a currently unsupported docker version. For me, downgrading to a supported docker version helped to resolve this issue (v1.12):
- Thanks for the info! - Well we can confirm that kubernetes is not working properly with Docker 1.29 and 1.3
Reiterating the mentions to trigger a notification: <detailsIn response to : - Thanks for the info!


Question: It("processes in different containers of the same pod should be able to see each other", func() { // TODO(yguo0905): Change this test to run unless the runtime is // Docker and its version is <1.13. <del> By("Check whether shared PID namespace is enabled.") isEnabled, err := isSharedPIDNamespaceEnabled() </del> <ins> By("Check whether shared PID namespace is supported.") isEnabled, err := isSharedPIDNamespaceSupported() </ins> framework.ExpectNoError(err) if !isEnabled { <del> framework.Skipf("Skipped because shared PID namespace is not enabled.") </del> <ins> framework.Skipf("Skipped because shared PID namespace is not supported by this docker version.") </ins> } By("Create a pod with two containers.")

- Well we can confirm that kubernetes is not working properly with Docker 1.29 and 1.3 Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This is not really a bug since kubernetes doesn't support the docker version you use yet, but feel free to send a patch if you find the problem.
I was thinking about your comment and I think it is still a bug since downgrading Kubernetes to 1.6 fixes the problem while having the same setup as described above. So something have changed in 1.7. But yeah. Will try to investigate on my own.
I'm actually getting this same error with 1.7.1 and docker 1.13.1. Anyone have other thoughts on what the issue might be here? I found this so question () but it looks like everything is configured correctly.
In kubernetes 1.7, under pid # 1, there is a "/ pause" process, and since init takes pid 1, it is no longer available. In earlier versions of kubernetes, pid # 1 was free . PID USER TIME COMMAND 1 root 0:00 /pause 7 root 3:41 /tiller Kubernetes now shares a single PID namespace among all containers in a pod when running with docker = 1.13.1. This means processes can now signal processes in other containers in a pod, but it also means that the pattern will cause the pod to be restarted rather than a single container.
I can work around the problem using the option in the kubelet, however this not the desired solution of course.
just FYI, looks like some images have to be run as PID 1.
In 1.7 release, we enabled shared pid namespace if the docker version is docker 1.13 and beyond for debug pod feature. Do anyone know why those images, such as GlusterFS have to run as PID 1? we need to figure out how to resolve this conflict with shared pid namespace. If we cannot, we have to disable that setting for 1.8 since we scheduled to support docker 1.13 in 1.8 timeframe. I am currently assigning you this bug to come up a solution.


Question: It("processes in different containers of the same pod should be able to see each other", func() { // TODO(yguo0905): Change this test to run unless the runtime is // Docker and its version is <1.13. <del> By("Check whether shared PID namespace is enabled.") isEnabled, err := isSharedPIDNamespaceEnabled() </del> <ins> By("Check whether shared PID namespace is supported.") isEnabled, err := isSharedPIDNamespaceSupported() </ins> framework.ExpectNoError(err) if !isEnabled { <del> framework.Skipf("Skipped because shared PID namespace is not enabled.") </del> <ins> framework.Skipf("Skipped because shared PID namespace is not supported by this docker version.") </ins> } By("Create a pod with two containers.")

cc/
gluster/gluster-centos is a container image that runs systemd so that it can run ntpd, crond, gssproxy, glusterd & sshd inside a single container. This isn't how Kubernetes is intended to be used and I don't know how much we should go out of our way to support it (this is a question for & seems to be a good write up of the challenges of running systemd inside of docker. It also prevents Kubernetes from doing process management like: reporting state of processes signaling individual processes for graceful restart ingesting and display daemon output (to stdout/err) Ideally we could change this many-processes-in-a-container to be many-containers-in-a-pod, something like this: This doesn't need a privileged container and doesn't run sshd or ntpd, neither of which are needed in Kubernetes. This also doesn't run crond, but that may actually be needed by gluster. I've never used gluster. (I used flags so I didn't have to investigate how to make these processes log to stdout) If you must run systemd in a container, adding this to the container config will get systemd to run: This tells systemd to run in system mode even though it's not PID 1 and disables its chroot detection (which is unrelated to PID 1 but checked when invoked as "systemd", I guess).
afaict, running in a container is well recommended ( and ) at certain scenarios and a common practice. iic, there are enough containers who make use of as its . If shared namespace which introduced in 1.7 breaks this, I believe its a backward compatibility break :(. I would invite here to share his thoughts as well. :)
Those articles are about how to work around shortcomings in vanilla docker that Kubernetes solves natively: resource sharing via pods, containing via SELinux, reaping orphaned zombies, etc. My earlier question was along the lines of how much effort should go into supporting use cases that are not using Kubernetes as intended. (I'm actually asking, not trying to make a point.) Does Kubernetes commit to continuing support for every container image that's ever run on Kubernetes? For how long? Indefinitely? or can we change it over the course of multiple releases? Anyway, this change is compatible with running an init system in a container.


Question: It("processes in different containers of the same pod should be able to see each other", func() { // TODO(yguo0905): Change this test to run unless the runtime is // Docker and its version is <1.13. <del> By("Check whether shared PID namespace is enabled.") isEnabled, err := isSharedPIDNamespaceEnabled() </del> <ins> By("Check whether shared PID namespace is supported.") isEnabled, err := isSharedPIDNamespaceSupported() </ins> framework.ExpectNoError(err) if !isEnabled { <del> framework.Skipf("Skipped because shared PID namespace is not enabled.") </del> <ins> framework.Skipf("Skipped because shared PID namespace is not supported by this docker version.") </ins> } By("Create a pod with two containers.")

The problem is with images that assume they will only ever have , which has so far been the case with docker due to a technical limitation, but it was never the case with other runtimes such as rkt, hyperv, etc. Brainstorming other options: shared pid namespace and require users of systemd to set an environment variable in their Kubernetes config (or port the multi-process container to a multi-container pod). shared pid namespace and enable it in a future release where we have the same problem. a solution where pid namespace is configurable per-pod, but not supported indefinitely. It would probably be implemented using annotations, and we will have to modify the Container Runtime Interface to support it. Over the course of multiple releases the default behavior would change but could be reverted based on the presence of an annotation. In a future release, users will have to change an annotation in their config to get the old behavior (rather than the environment variable). In a future future release, the annotation will stop working and users will have to set the environment variable. to supporting per-pod shared vs isolated pid namespaces indefinitely and adding it to the Kubernetes API. It doesn't make sense in some runtimes, so we probably can't require all runtimes to implement it, so it'll likely only see use with Docker. Last time we discussed it in SIG Node, and didn't want to support isolated pid namespaces indefinitely.
Yes running systemd in a container is a critical need for us. There are lots of situations where it is and will be used. Biggest use case if for people simply moving current workloads that run in a VM into a container. I have a service that runs fun under systemd in a VM. If I can simply take the init script and move it whole sale into a container, why wouldn't I want to do this. It's simple and does require a great deal of knowledge by the end user. service containers. Not every project can easily be moved into microservices. But they probably can run inside of containers. If we drop support for systemd running inside of containers, then we loose a big group of users. Getting access to journald information inside of the container. Applications that write to syslog and directly to the journal. Currently in container world without systemd, this information is dropped on the floor.


Question: It("processes in different containers of the same pod should be able to see each other", func() { // TODO(yguo0905): Change this test to run unless the runtime is // Docker and its version is <1.13. <del> By("Check whether shared PID namespace is enabled.") isEnabled, err := isSharedPIDNamespaceEnabled() </del> <ins> By("Check whether shared PID namespace is supported.") isEnabled, err := isSharedPIDNamespaceSupported() </ins> framework.ExpectNoError(err) if !isEnabled { <del> framework.Skipf("Skipped because shared PID namespace is not enabled.") </del> <ins> framework.Skipf("Skipped because shared PID namespace is not supported by this docker version.") </ins> } By("Create a pod with two containers.")

if I run systemd and journald, I can actually retrieve and use this information.
If the intension is to force all containers inside of a POD to run with the same PID Namespace? I think this is a mistake since it eliminates another really cool use case. I would love to be able to run two containers inside of a pod, and INIT container with escallated privileges and a locked down container, with no privileges. I want to make sure the locked down container can not see the processes of the privileged container. An example of this, might be a container that loads a kernel module that is needed by the locked down container. Another use case would be a buildah container. The INIT container mounts up a COW file system onto a directory on a location that the locked down container can write, The locked down container writes out the data for his image, when the container completes, the INIT container commits the data to the repo. I believe allowing different containers inside of a POD to run with different security and isolated PIDs should not be removed.
none of the things you've mentioned are removed by this change, save using docker to hide other processes in the pod, and I'm not sure why that's a requirement. Is anyone currently using the "privileged sidecar" pattern you describe or is it hypothetical? It just occurred to me that an interesting compromise here might be to only enable shared pid for multi-container pods. This would enable pod semantics for PID when it would make a difference while drop-in docker monoliths keep the PID 1 behavior they're expecting in docker. I'd have to double check, but I'm pretty sure that behavior could be implemented entirely in the dockershim with no API or CRI changes.
Hypothetical, but I am pushing it for a solution for problems I hear from users fairly frequently, where then need big privileges at start and stop of the pod, but not for the time is is available to the network.
ok, thanks. ftr, I think we should continue to support this pattern as well, and I don't think a shared PID namespace prohibits it. I will bring up this issue at Tuesday's sig node meetup if you'd like to join us. Does the config I mentioned in fix the issue with systemd?


Question: It("processes in different containers of the same pod should be able to see each other", func() { // TODO(yguo0905): Change this test to run unless the runtime is // Docker and its version is <1.13. <del> By("Check whether shared PID namespace is enabled.") isEnabled, err := isSharedPIDNamespaceEnabled() </del> <ins> By("Check whether shared PID namespace is supported.") isEnabled, err := isSharedPIDNamespaceSupported() </ins> framework.ExpectNoError(err) if !isEnabled { <del> framework.Skipf("Skipped because shared PID namespace is not enabled.") </del> <ins> framework.Skipf("Skipped because shared PID namespace is not supported by this docker version.") </ins> } By("Create a pod with two containers.")

Copying from a comment made elsewhere:
Update from SIG Node Meetup: Everyone wants to support these use cases, but we're not yet sure the requirements we want to support medium to long term. is going to sync up with and then we'll figure out what we want to support in a future release. For 1.8 the current plan is to leave docker shared pid namespace enabled by default. Cluster administrators can disable it per-node with the --docker-disable-shared-pid flag. I'll see about whether we should add something back to the release notes.
This seems somewhat backwards. Wouldn't it make more sense to go back to supporting systemd use cases by default NOW (by making the shared PID namespace behavior optional or something) until a final decision is reached and can be advertised to current users?
absolutely ! thank you very much
I think that is the best solution is to make the PID Namespace sharing optional.
- optional at what scope - cluster, node, pod. let's sync up before next sig-node and we can revisit if need be before 1.8 goes out. The main question right now is if pid namespace sharing is opt-in or opt-out. We need to weigh pro/cons of both.
I would expect it at the POD, since pods would have different requirements, not sure why you would want to have this at the cluster or node level.
yes it works :)
awesome, thanks! I'd like to better understand the requirements of people objecting to this change ( I think there's a bit of confusion about what has changed, so let's start with some facts: This container image runs on Kubernetes without changes to the image, but it requires (non-obvious) pod configuration. is nothing inherently incompatible between a shared pid namespace and a process manager like systemd. systemd and other init systems work with pid != 1, but may require special invocation. in this issue is specific to Kubernetes. That is, this container image wouldn't work with vanilla docker PID namespace sharing, either. change does not affect securityContext I can think of a few things that are objectionable about the change, but let's state them as requirements. For example, are these requirements for anyone? Kubernetes must run all containers published on docker hub with no additional pod configuration Kubernetes must run all systemd containers, regardless of how they are built, with zero additional pod configuration.


Question: It("processes in different containers of the same pod should be able to see each other", func() { // TODO(yguo0905): Change this test to run unless the runtime is // Docker and its version is <1.13. <del> By("Check whether shared PID namespace is enabled.") isEnabled, err := isSharedPIDNamespaceEnabled() </del> <ins> By("Check whether shared PID namespace is supported.") isEnabled, err := isSharedPIDNamespaceSupported() </ins> framework.ExpectNoError(err) if !isEnabled { <del> framework.Skipf("Skipped because shared PID namespace is not enabled.") </del> <ins> framework.Skipf("Skipped because shared PID namespace is not supported by this docker version.") </ins> } By("Create a pod with two containers.")

Kubernetes must run this specific container image with zero additional configuration Kubernetes must officially support isolated PID namespaces between containers in a single pod across all runtimes (docker, rkt, hyperv, etc). Configuration changes are ok, but the flag to systemd cannot be used because the documentation says its not useful. I want to use systemd as intended. Rather than having Kubernetes work around this container image, should we patch the Dockerfile to make it compatible with even the native docker shared-pid, unrelated to Kubernetes?
Configuration changes are ok, but the --system flag to systemd cannot be used because the documentation says its not useful. I want to use systemd as intended. We can ask the systemd guys why they recommend never using the --system flag, but I would guess one of the reasons is that processes showing up that were not created in the ancestry of systemd, could cause confusion. I would also guess that running more then one container with systemd --system would not be supported. Kubernetes must officially support isolated PID namespaces between containers in a single pod. This is required so that you can run different containers with different security constraints in the same pod at the same time. Your last comment makes the seem like it is the default, when actually it is the exception. almost no containers run with shared-pid normally. I guess in a side car container, it makes sense to support this scenario, but forcing this as the ONLY way to run containers is limiting.
This is the part I don't understand. Why? What does PID visibility have to do with security constraints? What am I missing? nor do they run with a shared network namespace, but we impose that on docker containers because that's the pod model.
An update from the last SIG Node: we'll disable this by default in 1.8 and revisit for 1.9 I'll use to track that.
if I have a privileged container process in the same pid namespace as a pid in a non priv container, then the container process in the non priv can attack the priv process. Having them in separate pid namespaces gives you better security. Bottom line, you have a potential change where containers that expect to be run as PID 1, whether legitimately or not, could break when you make this change. Kubernetes in my opinion should make it easy to opt out of this behavior.


Question: It("processes in different containers of the same pod should be able to see each other", func() { // TODO(yguo0905): Change this test to run unless the runtime is // Docker and its version is <1.13. <del> By("Check whether shared PID namespace is enabled.") isEnabled, err := isSharedPIDNamespaceEnabled() </del> <ins> By("Check whether shared PID namespace is supported.") isEnabled, err := isSharedPIDNamespaceSupported() </ins> framework.ExpectNoError(err) if !isEnabled { <del> framework.Skipf("Skipped because shared PID namespace is not enabled.") </del> <ins> framework.Skipf("Skipped because shared PID namespace is not supported by this docker version.") </ins> } By("Create a pod with two containers.")

/priority critical-urgent
[MILESTONENOTIFIER] Milestone Labels Complete Issue label settings: sig/node: Issue will be escalated to these SIGs if needed. priority/critical-urgent: Never automatically move out of a release milestone; continually escalate to contributor and SIG through all available channels. kind/bug: Fixes a bug discovered during the current release. <detailsAdditional instructions available <a href=""here</a</details
Update: is pending review and should be to the v1.8 Milestone to resolve this issue.


Question: q.ToDec().d.Dec.Sub(q.d.Dec, y.AsDec()) } <ins> // Mul multiplies the provided y to the current value. // It will return false if the result is inexact. Otherwise, it will return true. func (q *Quantity) Mul(y int64) bool { q.s = "" if q.d.Dec == nil && q.i.Mul(y) { return true } return q.ToDec().d.Dec.Mul(q.d.Dec, inf.NewDec(y, inf.Scale(0))).UnscaledBig().IsInt64() } </ins> // Cmp returns 0 if the quantity is equal to y, -1 if the quantity is less than y, or 1 if the // quantity is greater than y. func (q *Quantity) Cmp(y Quantity) int {

I'd like to add a multiplication method for the resource.Quantity. Currently, the resource package provides helper methods to calculate Quantity, such as and : Batch workloads often replicate the same PodTemplate, and controllers calculate the resources needed for all containers. In that case, It is inconvenient to call the function as many times as podTemplate, so I would like to create a helper function for multiplication.
/cc /sig api-machinary /wg batch
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /cc /sig api-machinary /wg batch </details
/sig api-machinery /wg batch
Do you have an example of usage for these utilities? Just curious?
/triage accepted
Sorry for the late response. Taking k-sigs/kueue as an example, I assumed the below: It helps the controllers for batch workloads such as k-sigs/kueue, k-sigs/jobset, kubeflow/mpi-operator, and kubeflow/training-operator. I'm thinking of implementing this function once the code freeze term is finished.
/assign
Hi, may i take this up?
I took this one, and working on this now.


Question: package transport <ins> // this is just to make the "unused" linter rule happy var _ = isCacheKeyComparable[tlsCacheKey] </ins> // assert at compile time that tlsCacheKey is comparable in a way that will never panic at runtime. <del> var _ = isComparable[tlsCacheKey] </del> <ins> // // Golang 1.20 introduced an exception to type constraints that allows comparable, but not // necessarily strictly comparable type arguments to satisfy the `comparable` type constraint, // thus allowing interfaces to fulfil the `comparable` constraint. // However, by definition, "A comparison of two interface values with identical // dynamic types causes a run-time panic if that type is not comparable". // // We want to make sure that comparing two `tlsCacheKey` elements won't cause a // runtime panic. In order to do that, we'll force the `tlsCacheKey` to be strictly // comparable, thus making it impossible for it to contain interfaces. // To assert strict comparability, we'll use another definition: "Type // parameters are comparable if they are strictly comparable". // Below, we first construct a type parameter from the `tlsCacheKey` type so that // we can then push this type parameter to a comparable check, thus checking these // are strictly comparable. // // Original suggestion from https://github.com/golang/go/issues/56548#issuecomment-1317673963 func isCacheKeyComparable[K tlsCacheKey]() { _ = isComparable[K] } </ins> func isComparable[T comparable]() {}

Originally posted by in
/assign /triage accepted /sig api-machinery auth
/milestone v1.28
hi , v1.28 Bug Triage Lead here. Is this targeted to 1.28 release? If that is the case, can we probably set labels for easier tracking? Thanks!
hi again! Gentle reminder that the code freeze is starting (about 3 weeks from now), we want to ensure that each issue is dealt in timely manner and tracked. As the issue is tagged for 1.28, is it still planned for this release?
hi! I don't seem to find any PR being worked on to close this issue? Since it has not been updated for a long time, I'd like to check what's the status with you once again. FYI, as I informed about it already 2 weeks ago, we are in week 9 currently and the code freeze is starting next week (week 10) at . Please remove the milestone if this not currently tracked for v1.28 release or if it is still, please make sure that the fix is in before the code freeze date above, thanks!
/milestone next-candidate
/milestone v1.29
Hello ! Bug triage for the 1.29 release cycle is here! This issue hasn't been updated for a long time, so I wanted to check what the status is. The code freeze will start (), which is about 2 weeks from now. And while there is still plenty of time, we want to make sure that every PR has a chance to be merged on time. As the issue is tagged for 1.29, is it still planned for that release?
Hello ! Bug triage for the 1.29 release cycle is here! This issue hasn't been updated for a long time, so I wanted to check what the status is. The code freeze will start (), which is this week. We want to make sure that every PR has a chance to be merged on time. As the issue is tagged for 1.29, is it still planned for that release?
/milestone v1.30 kicking the can, we are way past code freeze
does this still target v1.30?
Hello! Release Signal shadow here. I'd like to check what's the status.


Question: package transport <ins> // this is just to make the "unused" linter rule happy var _ = isCacheKeyComparable[tlsCacheKey] </ins> // assert at compile time that tlsCacheKey is comparable in a way that will never panic at runtime. <del> var _ = isComparable[tlsCacheKey] </del> <ins> // // Golang 1.20 introduced an exception to type constraints that allows comparable, but not // necessarily strictly comparable type arguments to satisfy the `comparable` type constraint, // thus allowing interfaces to fulfil the `comparable` constraint. // However, by definition, "A comparison of two interface values with identical // dynamic types causes a run-time panic if that type is not comparable". // // We want to make sure that comparing two `tlsCacheKey` elements won't cause a // runtime panic. In order to do that, we'll force the `tlsCacheKey` to be strictly // comparable, thus making it impossible for it to contain interfaces. // To assert strict comparability, we'll use another definition: "Type // parameters are comparable if they are strictly comparable". // Below, we first construct a type parameter from the `tlsCacheKey` type so that // we can then push this type parameter to a comparable check, thus checking these // are strictly comparable. // // Original suggestion from https://github.com/golang/go/issues/56548#issuecomment-1317673963 func isCacheKeyComparable[K tlsCacheKey]() { _ = isComparable[K] } </ins> func isComparable[T comparable]() {}

The code freeze is starting in a week, and we want to ensure that each PR has a chance to be merged on time.
We are past the code freeze stage. Moving this to the next release. /milestone v1.31
Hello This issue has not been updated for a long time, so I'd like to check what's the status. The , and while there is still plenty of time, we want to ensure that each PR has a chance to be merged on time. As the issue is tagged for , is it still planned for this release?
Hello, This issue/PR hasn't been updated in a while, so I just wanted to check the status. If there's anything we can do, please let us know. Reminder: Code freeze starts (about 1.5 weeks from now). Please make sure the PR has both and labels before the code freeze.
/milestone v1.32
/assign


Question: } // tryUnmount calls plain "umount" and waits for unmountTimeout for it to finish. <del> func tryUnmount(mounter *Mounter, path string, unmountTimeout time.Duration) error { klog.V(4).Infof("Unmounting %s", path) </del> <ins> func tryUnmount(target string, withSafeNotMountedBehavior bool, unmountTimeout time.Duration) error { klog.V(4).Infof("Unmounting %s", target) </ins> ctx, cancel := context.WithTimeout(context.Background(), unmountTimeout) defer cancel() <del> cmd := exec.CommandContext(ctx, "umount", path) out, cmderr := cmd.CombinedOutput() </del> <ins> command := exec.CommandContext(ctx, "umount", target) output, err := command.CombinedOutput() </ins> // CombinedOutput() does not return DeadlineExceeded, make sure it's // propagated on timeout.

A status of Pod mounted NFS PV remain as terminating when a node shutdown and restart. A Pod is completely terminated. Deploy a Pod mounted NFS PV Shutdown the node where the Pod is running Wait for 5 min and the Pod status become Restart the node The Pod status is still and the Pod is never terminated completely <detailsPrepare a NFS Server.(I used ) Deploy a Pod mounted NFS PV like this. Shutdown the node where the Pod is running.(In this case ) And the status of node02 becomes . After 5 min later(by default), the Pod status becomes . (The time a Pod becomes status depends on .) Then, restart the node and the status of node02 becomes . But the Pod status is still and the Pod is never terminated completely. It seems that a faild to delete Pod because of trying to unmount a not mounted directory. Force delete Pod. Delete Pod's directory on the node. </detailsNo response <details</details<detailsnone(on premise) </details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage
/sig node
It dose not happen using , so it seems to depend on NFS volume plugin.
/assign
I try to build and run kubelet with comment out this logic. Then it seems to get well. related: This change checks the result of instead of the mount point check. However, check logic is not implemented in and (these functions are used in NFS volume plugin), despite it is done in .
/remove-sig node /sig storage
Reopening, fix had to be reverted due to unit tests requiring root permissions ()
I am terribly sorry... Please wait a moment. I'll try to fix it.
no worries, just get to review the rework, making sure the units tests don't require root permissions
Thank you! Fixed in .
Are there any plans to backport this to older Kubernetes versions? This seems like a critical fix.


Question: } } } <ins> func TestPodToEndpointAddress(t *testing.T) { podStore := cache.NewStore(cache.DeletionHandlingMetaNamespaceKeyFunc) ns := "test" addPods(podStore, ns, 1, 1, 0) pods := podStore.List() if len(pods) != 1 { t.Errorf("podStore size: expected: %d, got: %d", 1, len(pods)) return } pod := pods[0].(*v1.Pod) epa := podToEndpointAddress(pod) if epa.IP != pod.Status.PodIP { t.Errorf("IP: expected: %s, got: %s", pod.Status.PodIP, epa.IP) } if *(epa.NodeName) != pod.Spec.NodeName { t.Errorf("NodeName: expected: %s, got: %s", pod.Spec.NodeName, *(epa.NodeName)) } if epa.TargetRef.Kind != "Pod" { t.Errorf("TargetRef.Kind: expected: %s, got: %s", "Pod", epa.TargetRef.Kind) } if epa.TargetRef.Namespace != pod.ObjectMeta.Namespace { t.Errorf("TargetRef.Kind: expected: %s, got: %s", pod.ObjectMeta.Namespace, epa.TargetRef.Namespace) } if epa.TargetRef.Name != pod.ObjectMeta.Name { t.Errorf("TargetRef.Kind: expected: %s, got: %s", pod.ObjectMeta.Name, epa.TargetRef.Name) } if epa.TargetRef.UID != pod.ObjectMeta.UID { t.Errorf("TargetRef.Kind: expected: %s, got: %s", pod.ObjectMeta.UID, epa.TargetRef.UID) } if epa.TargetRef.ResourceVersion != pod.ObjectMeta.ResourceVersion { t.Errorf("TargetRef.Kind: expected: %s, got: %s", pod.ObjectMeta.ResourceVersion, epa.TargetRef.ResourceVersion) } } func TestPodAddressChanged(t *testing.T) { podStore := cache.NewStore(cache.DeletionHandlingMetaNamespaceKeyFunc) ns := "test" addPods(podStore, ns, 1, 1, 0) pods := podStore.List() if len(pods) != 1 { t.Errorf("podStore size: expected: %d, got: %d", 1, len(pods)) return } oldPod := pods[0].(*v1.Pod) newPod := oldPod.DeepCopy() if podAddressChanged(oldPod, newPod) { t.Errorf("Expected address to be unchanged for copied pod") } newPod.Spec.NodeName = "changed" if !podAddressChanged(oldPod, newPod) { t.Errorf("Expected address to be changed for pod with NodeName changed") } newPod.Spec.NodeName = oldPod.Spec.NodeName newPod.ObjectMeta.ResourceVersion = "changed" if podAddressChanged(oldPod, newPod) { t.Errorf("Expected address to be unchanged for pod with only ResourceVersion changed") } newPod.ObjectMeta.ResourceVersion = oldPod.ObjectMeta.ResourceVersion newPod.Status.PodIP = "1.2.3.1" if !podAddressChanged(oldPod, newPod) { t.Errorf("Expected address to be changed with pod IP address change") } newPod.Status.PodIP = oldPod.Status.PodIP newPod.ObjectMeta.Name = "wrong-name" if !podAddressChanged(oldPod, newPod) { t.Errorf("Expected address to be changed with pod name change") } newPod.ObjectMeta.Name = oldPod.ObjectMeta.Name } func TestDetermineNeededServiceUpdates(t *testing.T) { testCases := []struct { name  string a     sets.String b     sets.String union sets.String xor   sets.String }{ { name:  "no services changed", a:     sets.NewString("a", "b", "c"), b:     sets.NewString("a", "b", "c"), xor:   sets.NewString(), union: sets.NewString("a", "b", "c"), }, { name:  "all old services removed, new services added", a:     sets.NewString("a", "b", "c"), b:     sets.NewString("d", "e", "f"), xor:   sets.NewString("a", "b", "c", "d", "e", "f"), union: sets.NewString("a", "b", "c", "d", "e", "f"), }, { name:  "all old services removed, no new services added", a:     sets.NewString("a", "b", "c"), b:     sets.NewString(), xor:   sets.NewString("a", "b", "c"), union: sets.NewString("a", "b", "c"), }, { name:  "no old services, but new services added", a:     sets.NewString(), b:     sets.NewString("a", "b", "c"), xor:   sets.NewString("a", "b", "c"), union: sets.NewString("a", "b", "c"), }, { name:  "one service removed, one service added, two unchanged", a:     sets.NewString("a", "b", "c"), b:     sets.NewString("b", "c", "d"), xor:   sets.NewString("a", "d"), union: sets.NewString("a", "b", "c", "d"), }, { name:  "no services", a:     sets.NewString(), b:     sets.NewString(), xor:   sets.NewString(), union: sets.NewString(), }, } for _, testCase := range testCases { retval := determineNeededServiceUpdates(testCase.a, testCase.b, false) if !retval.Equal(testCase.xor) { t.Errorf("%s (with podChanged=false): expected: %v  got: %v", testCase.name, testCase.xor.List(), retval.List()) } retval = determineNeededServiceUpdates(testCase.a, testCase.b, true) if !retval.Equal(testCase.union) { t.Errorf("%s (with podChanged=true): expected: %v  got: %v", testCase.name, testCase.union.List(), retval.List()) } } } </ins>

/kind bug What happened: On a large cluster with enough pods, endpoints take as long as 10 minutes to show up after their associated pods and service are created. We believe that this is because every pod change results in an update of the addresses within the endpoint since they refer to a pod via a whose changes every time a pod change is observed. All these non-update updates eventually overwhelm the endpoint controller's queue rate limit and the queue starts to grow. While an increase in the queue's rate limit can alleviate the problem, an admin can keep a much lower limit if the non-update updates are dropped rather than processed. What you expected to happen: When a pod related to an endpoint is updated, I would expect the endpoint to update only if there is a change in the pod that relates to the endpoint, such as a change in the pod's IP address. How to reproduce it (as minimally and precisely as possible): a pod and associated service: the endpoint for changes: the watch running, and in another session, edit a pod to add a label: Then add another label on the line after the label, such as that the watched endpoint changed despite there not being any meaningful change to the fields the endpoint cares about. Anything else we need to know?: The PR addresses this issue. Environment: Kubernetes version (use ):
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig scalability
what ever happened with the effort to move leases off of Endpoints?


Question: } // LocalEndpoint empty implementation <del> func LocalEndpoint(path, file string) string { return "" </del> <ins> func LocalEndpoint(path, file string) (string, error) { return "", fmt.Errorf("LocalEndpoints are unsupported in this build") </ins> } // GetBootTime empty implementation

What happened: I replaced the kubelet on one of my 1.14 nodes with a build from master, and now I'm getting this error starting up. Test passes since 5/31/2019 commit have been failing on testgrid. If you look at the logs, the Windows nodes are not coming up. What you expected to happen: No errors starting up kubelet How to reproduce it (as minimally and precisely as possible): Build master, then try to run the kubelet on Windows Anything else we need to know?: /sig windows Environment: Kubernetes version (use ): Cloud provider or hardware configuration: Azure OS (e.g: ): Windows Server 2019 Kernel (e.g. ): 10.0..529 Install tools: aks-engine
cc /priority critical-urgent
/milestone v1.15 Release team - it looks like a recent PR broke windows nodes. We will need a fix here for 1.15
Ok, so the problem here is a little more complicated and we're actually seeing a bug that is actually in the code for quite some time, just that we never used that particular code path. But first: TL;DR: Quick fix is to start the kubelet on Windows with: --feature-gates="KubeletPodResources=false" Now, as to why this is happening: PR [1] was merged 3 days ago and it graduates KubeletPodResources feature as Beta. That means kubelet will enable it by default.With the introduction of the above feature, this branch of code is entered when starting kubelet [2]. We can see a call to the function: defined here [3].While innocent enough, gets the necessary npipe by calling: . is defined here [4] for Windows . The problem is actually that url.URL can't handle Windows style paths [5] and will replace "" with "%5C" as it's to be expected. The paths are transformed along the way from the default "/var/lib/kubelet" to "varlibkubelet" by defined here [6]. More specifically in the use of to create a path. It's inputs are unix style but returns the path Windows style as it's to be expected.The solution is to replace the usage of url.URL in with some alternative package or just implement it.


Question: } // LocalEndpoint empty implementation <del> func LocalEndpoint(path, file string) string { return "" </del> <ins> func LocalEndpoint(path, file string) (string, error) { return "", fmt.Errorf("LocalEndpoints are unsupported in this build") </ins> } // GetBootTime empty implementation

[1] [2] [3] [4] [5] [6]
Thanks for the analysis! I was a bit concerned around the fact that this started showing up around my commit: but the root cause here is indeed, as you said, the graduation of feature to Beta and the way the invalid escape sequences are being introduced.
GCE tests failing too /cc
I'm trying a quick to see if anything else is broken. There are other functions in that same file doing that substitution already so it may make sense to use that as a workaround until a better solution is available.
Still working on it - looks like the path to create the directories is also busted. With still getting


Question: return tmpDir, plug } <ins> func getNodeExpandablePlugin(t *testing.T, isBlockDevice bool) (string, volume.NodeExpandableVolumePlugin) { tmpDir, err := utiltesting.MkTmpdir("localVolumeTest") if err != nil { t.Fatalf("can't make a temp dir: %v", err) } plugMgr := volume.VolumePluginMgr{} var pathToFSType map[string]hostutil.FileType if isBlockDevice { pathToFSType = map[string]hostutil.FileType{ tmpDir: hostutil.FileTypeBlockDev, } } else { pathToFSType = map[string]hostutil.FileType{ tmpDir: hostutil.FileTypeDirectory, } } plugMgr.InitPlugins(ProbeVolumePlugins(), nil /* prober */, volumetest.NewFakeKubeletVolumeHostWithMounterFSType(t, tmpDir, nil, nil, pathToFSType)) plug, err := plugMgr.FindNodeExpandablePluginByName(localVolumePluginName) if err != nil { os.RemoveAll(tmpDir) t.Fatalf("Can't find the plugin by name") } if plug.GetPluginName() != localVolumePluginName { t.Errorf("Wrong name: %s", plug.GetPluginName()) } return tmpDir, plug } </ins> func getPersistentPlugin(t *testing.T) (string, volume.PersistentVolumePlugin) { tmpDir, err := utiltesting.MkTmpdir("localVolumeTest") if err != nil {

There are use cases where users want to expand the local volume (may be managed by LVM or something similar), but even though they can edit underlying PV (and expand the underlying device) - the reported size in pvc.Status keeps reporting older value because in-tree local volumes are non-expandable. We could make this easier for users by - implementing interface for local volumes. This would allow following steps for local volume expansion: /admin edits the pvc - requesting new size. expands the underlying device. edits the PV and sets new size. will expand the filesystem and update This would at least get the user from impossible territory to possible. cc /sig storage
/triage accepted
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
Hi & any news on this issue ? The related PR seems stalled.
merged. So this is fixed now.
Oh right ! I didn't looked close enough and thought it was only for directory based volumes. Thank you.


Question: func NewMainKubelet( hn string, dc dockertools.DockerInterface, <del> cc CadvisorInterface, </del> ec tools.EtcdClient, rd string, ni string,

So today, cadvisor became wedged and just stopped paying attention to its socket. Not kubelet's fault. But when I would launch all I would get would be: It took me a bit of trolling through netstat and lsof to figure out that the problem was cadvisor being completely locked up. kill -9 cadvisor and everything started working well. I wonder if we should emit more messages during startup about what operation we are about to attempt, not just the results of operations, as I was blaming docker for a while....
kubelet shouldn't block on this. I'll take a look.
Any ideas what was locking cAdvisor? and was it just not responding to requests?
I don't have any idea why it was locked. If I find another one locked I'll hang onto it to debug. Journal had: obviously the 'kill' came from me. It would not respond to systemctl cadvisor stop. netstat looking at port 4194 showed me At which point I realized cadvisor wasn't handling its sockets right since so many long dead kubelet sockets hadn't been cleaned up, so I kill -9'd it.
I cut some interesting info from the cadvisor journal log (trying to cut hostnames) You can see the last message about failed to update stats for container ... happened Oct 08 01:06:01. It was then silent until I killed it Oct 08 19:20:13, even though I'm pretty sure dnf was still being launched every hour on the hour. So it was wedged for about 18 hours.
What kernel is this on? We've seen there is a kernel bug fixed in 3.12+ that sometimes hangs us forever on exit. We're not quite sure how it gets triggered...
16.3


Question: // Search under globalMapPath dir if all symbolic links from pods have been removed already. // If symbolick links are there, pods may still refer the volume. globalMapPath := deviceToDetach.DeviceMountPath <del> if err != nil { // On failure, return error. Caller will log and retry. return deviceToDetach.GenerateError("UnmapDevice.GetGlobalMapPath failed", err) } </del> refs, err := og.blkUtil.GetDeviceSymlinkRefs(deviceToDetach.DevicePath, globalMapPath) if err != nil { return deviceToDetach.GenerateError("UnmapDevice.GetDeviceSymlinkRefs check failed", err)

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: A Pod is run with local volume mounted in block mode. When that pod is deleted, the tear down process fails because device unmap fails. The device path appears to be empty, causing the unmap to fail. What you expected to happen: The teardown should have succeeded How to reproduce it (as minimally and precisely as possible): Create a pod with a PV for local volume in block mode. Delete the pod. Look at kubelet log files on the minion node to observe the error messages. Anything else we need to know?: Log messages: Feb 18 09:21:43 kubernetes-minion-group-crj1 kubelet[1510]: E0218 09:21:43. 1510 ] Operation for """ ("-148c-11e8-b315-")" failed. No retries permitted until 2018-02-18 09:21:44. +0000 UTC m=+. (durationBeforeRetry 500ms). Error: "UnmapVolume.UnmapDevice on global map path failed for volume "blkdata" (UniqueName: "") pod "-148c-11e8-b315-" (UID: "-148c-11e8-b315-") : Failed to unmap device from map path. mapPath is empty" Feb 18 09:21:43 kubernetes-minion-group-crj1 kubelet[1510]: I0218 09:21:43. 1510 ] operationExecutor.UnmountVolume started for volume "blkdata" (UniqueName: "") pod "-148c-11e8-b315-" (UID: "-148c-11e8-b315-") Environment: Kubernetes version (use ): 1.10+ Cloud provider or hardware configuration: GCE OS (e.g. from /etc/os-release): "Container-Optimized OS from Google" Kernel (e.g. ): Linux 4.4.86+ Install tools: Others:
/sig storage
This error is returned in this . It same that we set the global mount path to only if the volume plugin is attachable(see ).


Question: // Search under globalMapPath dir if all symbolic links from pods have been removed already. // If symbolick links are there, pods may still refer the volume. globalMapPath := deviceToDetach.DeviceMountPath <del> if err != nil { // On failure, return error. Caller will log and retry. return deviceToDetach.GenerateError("UnmapDevice.GetGlobalMapPath failed", err) } </del> refs, err := og.blkUtil.GetDeviceSymlinkRefs(deviceToDetach.DevicePath, globalMapPath) if err != nil { return deviceToDetach.GenerateError("UnmapDevice.GetDeviceSymlinkRefs check failed", err)

However, local volume plugin is not attachable, so will be empty and the operation will fail. I think we should extract the operation from the block, and execute this operation just before we try to map the device to global map path, then this global map path can be cached in for both attachable and non attachable volume plugins. Any comments? If not, I can help to fix this.
Thank you for the investigation. We've discussed this problem by offline yesterday. I proposed same fix which you mentioned above, then tested the fix works correctly with local volume plugin. I'll push a PR soon.
The device path appears to be empty, causing the unmap to fail. Please note, which was stored in is empty. device path isn't related to this problem.
Sorry about that, this is a typo, what I mean is global mount path.
No, no. My above comment is just note for the section. Your investigation is correct.


Question: effect: NoSchedule - key: "CriticalAddonsOnly" operator: "Exists" <ins> nodeSelector: beta.kubernetes.io/os: linux </ins> containers: - name: coredns image: k8s.gcr.io/coredns:1.2.4

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: In a windows cluster, containers can get scheduled on Windows nodes where they will never work. Warning Failed 54m (x3 over 54m) kubelet, 35934k8s9000 Failed to pull image "coredns/coredns:1.2.2": rpc error: code = Unknown desc = no matching manifest for windows/amd64 in the manifest list entries What you expected to happen: CoreDNS starts and runs on Linux master nodes. How to reproduce it (as minimally and precisely as possible): Build a Windows Cluster. Note: I first encountered this issue when deploying with ACS-Engine: but it occurs to me that it should be mirrored here, so that the relevant node selectors can be included in the canonical add-on directory. /sig Windows
I suppose this is fine if there are always linux nodes in the cluster. I don't know how Kubernetes or docker manifests work for windows, but CoreDNS does build and run on windows: That is not included in the manifest, is it possible to get that working? I guess you would probably also have to change the way the mounted config file is referred to in windows? Or is it this change good enough?
For now, IMO yes. what do you think? I'm assuming, perhaps incorrectly, that not all core k8s components support windows yet. do you know the current state of Windows support for core k8s components? Is CoreDNS the only core component whose image doesnt support windows?
Have not looked into it in detail yet, but one complication would be configuration. AFAIK Windows systems don't have an ... and the default CoreDNS configuration makes specific reference to the file (to get upstream servers).
Yeah, good point. I was thinking about the -conf /etc/coredns/Corefile which I think we pass to coredns IIRC. But is a bigger issue.
While it is possible to run Windows coreDNS pods, this is still undocumented and not officially supported yet. Officially, only kubelet & kube-proxy are in beta on Windows and scheduled to GA with 1.13 (assuming it meets the criteria and community votes of course). I think for the time being it should be fine to limit scheduling to Linux only to avoid pods being stuck but we should design in a way to keep the option open of undoing this in the future. what do you think?


Question: // errPodCompleted is returned by PodRunning or PodContainerRunning to indicate that // the pod has already reached completed state. <del> var errPodCompleted = fmt.Errorf("pod ran to completion") </del> <ins> var errPodCompleted = FinalError(errors.New("pod ran to completion successfully")) // errPodFailed is returned by PodRunning or PodContainerRunning to indicate that // the pod has already reached a permanent failue state. var errPodFailed = FinalError(errors.New("pod failed permanently")) </ins> // LabelLogOnPodFailure can be used to mark which Pods will have their logs logged in the case of // a test failure. By default, if there are no Pods with this label, only the first 5 Pods will

treats any error returned by the condition as terminal, but silently ignores any errors that aren't returned along with . We should change WaitForPodCondition to behave like . /sig testing /area e2e-test-framework
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
Was there any reason why you instead of doing this right away? I fully agree with this issue, we should abort early. I just found the TODO because I have a failing test that ran longer than expected after a final pod failure and that made me check out why it didn't abort quicker :grin:


Question: v1 "k8s.io/api/core/v1" "k8s.io/apimachinery/pkg/api/resource" <ins> "k8s.io/apiserver/pkg/util/feature" "k8s.io/component-base/featuregate" featuregatetesting "k8s.io/component-base/featuregate/testing" </ins> v1helper "k8s.io/kubernetes/pkg/apis/core/v1/helper" <ins> "k8s.io/kubernetes/pkg/features" </ins> "k8s.io/kubernetes/pkg/scheduler/apis/config" "k8s.io/kubernetes/pkg/scheduler/framework" )

What happened: In a v1.18 cluster with the LocalStorageCapacityIsolation feature gate disabled setting an ephemeral-storage request or limit causes the pod to be unschedulable: If the ephemeral-storage request is removed the pod is scheduled. This behavior exists in v1.19 and a recent v1.20 build. What you expected to happen: In previous releases such pods are scheduled. I expect that any ephemeral-storage limit is not enforced. How to reproduce it (as minimally and precisely as possible): Disable the LocalStorageCapacityIsolation feature guide by adding to the kube-scheduler. Create a pod like this: Anything else we need to know?: The problem was originally seen after moving from IBM Cloud Red Hat OpenShift v4.4 to v4.5. Pods that worked in 4.4 (Kubernetes v1.17) remained in a Pending state in OpenShift v4.5 (Kubernetes 1.18). We discovered that the IBM Cloud OpenShift service has the LocalStorageCapacityIsolation feature-gate disabled while Red Hat has enabled the feature. We then replicated the behavior with a plain Kubernetes cluster. Environment: Kubernetes version (use ): v1.18.10, v1.19.3, v1.20.0 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig sig-scheduling
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig sig-scheduling Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scheduling
I can confirm it's a bug. Will work on it.


Question: v1 "k8s.io/api/core/v1" "k8s.io/apimachinery/pkg/api/resource" <ins> "k8s.io/apiserver/pkg/util/feature" "k8s.io/component-base/featuregate" featuregatetesting "k8s.io/component-base/featuregate/testing" </ins> v1helper "k8s.io/kubernetes/pkg/apis/core/v1/helper" <ins> "k8s.io/kubernetes/pkg/features" </ins> "k8s.io/kubernetes/pkg/scheduler/apis/config" "k8s.io/kubernetes/pkg/scheduler/framework" )

/assign
Clarifying the recreate steps: I disabled the LocalStorageCapacityIsolation feature guide by adding to the kube-scheduler since this behavior seems to be within the scheduler.
Thanks for confirming!
This bug has been fixed in master branch, and will be backported to [1.18] and [1.19] soon. [1.18]: [1.19]:


Question: '${KUBELET_CONFIG}' create-kube-proxy-opts  '${MASTER_IP}'  <del> '${MASTER_IP}' </del> <ins> '${MASTER_IP}'  '${KUBE_PROXY_EXTRA_OPTS}' </ins> create-flanneld-opts '127.0.0.1' '${MASTER_IP}' FLANNEL_OTHER_NET_CONFIG='${FLANNEL_OTHER_NET_CONFIG}' sudo -E -p '[sudo] password to start master: ' -- /bin/bash -ce '

There should be a way to select the iptables proxy-mode.
Are you interested in modifying ubuntu deploy scripts to use a user provided configure file?
This is fixed by and it will be merged in soon.
This fix breaks cluster/kube- while kubernetes provider is rather than ubuntu, as the following variables are only defined in ubuntu folder PROXYSETTING=${PROXYSETTING:-""} DEBUG=${DEBUG:-"false"} When I run KUBERNETES_PROVIDER=vagrant and run cluster/kube- It will fail with error "==master: [ERROR ] {'pid': 5871, 'retcode': 1, 'stderr': ' line 21: DEBUG: unbound variable', 'stdout': ''}" I will open a separated issue to discuss the solution.
I do not understand. I have not tried KUBERNETESPROVIDER=vagrant and run cluster/kube-, can you please clarify what is going wrong? This PR () affected only files in cluster/ubuntu/; are you saying that some of them are used when KUBERNETESPROVIDER=vagrant? This PR did not introduce PROXY_SETTING or DEBUG.
I tried running kube- with the Vagrant provider, but ran into a different (and much prior) problem; see --- any help would be appreciated.
OK, I see you opened issue for this and PR fixed it. Thanks. I will respond to the larger question under .


Question: if [ "${DRY_RUN}" = false ]; then ls "${CLIENT_REPO}" | { grep -v '_tmp' || true; } | xargs rm -rf mv "${CLIENT_REPO_TEMP}"/* "${CLIENT_REPO}" <ins> git checkout HEAD -- $(find "${CLIENT_REPO}" -name BUILD) </ins> fi

We copy in BUILD files from the saved directories. The BUILD files should be preserved. This was hit in


Question: // Plugin is an implementation of admission.Interface. type Plugin struct { *admission.Handler <del> client kubernetes.Interface lister schedulingv1listers.PriorityClassLister </del> <ins> client                          kubernetes.Interface lister                          schedulingv1listers.PriorityClassLister resourceQuotaFeatureGateEnabled bool nonPreemptingPriority           bool </ins> } var _ admission.MutationInterface = &Plugin{} var _ admission.ValidationInterface = &Plugin{} <ins> var _ genericadmissioninitializers.WantsFeatures = &Plugin{} </ins> var _ = genericadmissioninitializers.WantsExternalKubeInformerFactory(&Plugin{}) var _ = genericadmissioninitializers.WantsExternalKubeClientSet(&Plugin{})

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Following the documentation at the , I tried enabling quota for a namespace other than but I was still not able to create a pod with priorityClass in my namespace with following error: Error from server (Forbidden): error when creating "": pods "memory-demo" is forbidden: pods with system-cluster-critical priorityClass is not permitted in my-namespace namespace What you expected to happen: Expected pod to be created with namespace where I allowed quota. I can see that both ResourceQuota and Priority Plugins are enabled from logs. I0408 17:15:08. ] Loaded 8 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,MutatingAdmissionWebhook. I0408 17:15:08. ] Loaded 6 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,ValidatingAdmissionWebhook,ResourceQuota. How to reproduce it (as minimally and precisely as possible): Follow the steps at Anything else we need to know?: I think, the issue is with priorityPlugin where we are rejecting instead of letting resourceQuota plugin to handle the admission when is enabled. Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Others:
/sig scheduling /cc
Same happens to me. Correction, it does not allow to use the default permissions outside of kube-system
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
See also:
/remove-lifecycle stale
/assign Per offline discussion with
Same issue here.