Problem: Preferred pod/node affinity and preferred PodTopologySpread may not be respected because of conflicts with other scoring functions, like node utilization or image locality. Scheduler Scoring functions have weights, by default they are all set to 1 (apart from one): Having equal weights is problematic because of the way we score: each Scoring plugin returns a score [0, MaxScore], which get multiplied by the configured weight of the plugin, and a node score is the sum over all scores. Therefore, even if the user explicitly indicates that they prefer to place pod x on node y, this preference may not be respected and the scheduler may decide to place the pod on node z because it has lower usage and the container that the pod is requesting. Proposed Solution: Scoring functions that allow users to explicitly express preference should have higher weights than generic scoring functions. This includes: inter-pod affinity, node affinity, taint/toleration and pod topology spread. This issue is proposing to change the default behavior of the scheduler, I am not sure how k8s deprecation policy applies here, although I would argue that this change should help the scheduler make decisions closer to users expectations. We will start with pod topology pod spread since as of creating this issue, the feature is still in beta. /sig scheduling
Quote from the : Rule : Deprecated behaviors must function for no less than 1 year after their announced deprecation. I think the rule doesn't apply to this proposal since it is not impacting the correctness of applications or impact administration of the cluster.
As nodepreferavoidpods already exercised to set its weight to 1000 by default, it sounds fair to me to increase the default weight for inter-pod affinity, node affinity, taint/toleration and pod topology spread. BTW: to some extent, we can use e2e tests verifying the to-proposed default weight. Right now we have some e2e tests covering priorities (scoring), but they depend on to ensure cpu/mem are at the same ratio of usage: and hence IMO a bit fragile.
Strongly agree with this change. But, IMO, we may need to reconsider the default impact weights of each scoring plugin. Maybe they should have more weight ladders, rather than simply 1 or 1000. For example, from practical experience, in most cases should not be set to the same weight as , , etc. (Yes, we need to consider this for users, because most users may not notice this change).
IMO a reasonable criterion to guide which Score plugins should have non-default weight is: whether the Score plugin is associated with an API (e.g., PodTopologySpread Score plugin is associated with .
If we give a higher weight only to the API-facing scores, then workloads that don't use these features are not affected, which is a good thing. I think a good weight to give to these scores is one that avoids it to be diluted among the rest of the scores. So, maybe a weight that is half the sum of all non-API-facing scores. Now, if a user uses the API for more than one score, I think we can safely assume that they want equal weights for them.
/cc Daniel, can you comment on whether or not the proposed change conforms with k8s deprecation policy?
I hate to say it, but I think you have to make the new set of weights an option and use the deprecation process to move people over. It's quite difficult to predict what a change in these weights might do to the overall scheduling behavior, no? If it was already possible to specify the settings (via flag or config file), then maybe there's a case to be made that the deprecation process is heavier than needed here. To make that case, you'd need to test and be VERY sure that the new settings are actually an improvement. The change sounds like an improvement, but I'd like to see that quantified / tested somehow. Perhaps even more importantly, I'd like to know the conditions under which it makes things worse, and along which dimensions. can give a second opinion if you don't like this answer :)
This may have an impact on availability (since by default the scheduler will prefer less under utilized nodes) and pod startup (less preference to node that already has the image) IF the pod have explicit preference for being on a specific node or with a specific pod. Those weights are actually configurable via the component config file, but most cloud providers don't allow customers to provide a custom configuration, they just use the default. Regarding the change of behavior, this effectively create two level scoring: first level is by user-specified preferences (affinity), second level is by workload-agnostic ones (e.g., image existence). If users are not specifying preferences, then this change have no impact.
Those are good points, but how many users specify preferences? Should we encourage those users to use non-optional constraints, instead? I can imagine arguing that making the preferences more likely to be obeyed makes it rarer and therefore more surprising and worse when they can't be honored.
I'm not sure if I'm understanding this correctly. Examples of non-optional scores are "topology spreading", "resources utilization" and "image locality". Users (aka Pods) can not opt-in or opt-out from them. Cluster operators (on prem or cloud-providers) can opt-out or change weights, but for a whole cluster. Remember that we are talking about node scoring. The nodes to be scored should have already passed the filters. Let's take an API-facing score (node affinity) and non-API-facing score (image locality) for illustrating the effect of weights: Imagine a pod is using "preferred" node affinity and there are no nodes available that satisfy that affinity. That means that in this case all nodes would get a low (or zero) score for node affinity. In this case the score given by image locality will provide the difference in score between all nodes. Now, let's say some nodes satisfy the node affinity. Those nodes will have the same high score for node affinity. But they will defer in image locality, which will be (in this case) the determining factor to select a node. However, in real (default) clusters we have more than 2 scores. And the problem is that pod preferences end up being diluted with all the other default scores summed up. What the proposal would achieve is that we first honor pod preferences and then we differentiate among the resulting nodes using the default scores.
OK, so I guess the case where this hurts users is: They specified preferences The preferences are actually bad (plot twist) they're diluted so much that they're not having much effect but now the scheduler starts honoring them more often But it's still probabilistic, so we can expect some percentage of the time, the preferences continue to not get honored? Numerically, suppose 50% have preferences, 2% of those got them wrong, 80% of those are currently diluted (i.e., not already broken), and 10% of those are in environments where they'll suddenly get honored 100% after this change (becoming broken). That's .5.02.8*.1 = .08% of users, or 1 out of 1250 users this change could break. That's still maybe a little high for my taste, but maybe you have an alternative estimate?
I think we can tweak how many of the currently diluted cases become honored. In I proposed that we use a weight of half the sum of the non-API-facing weights. We can be less aggressive and do a quarter or just 2 (we currently put weight of 1 for all).
On another note, I would say weights are actually part of the component config API. In both v1alpha1 and the new v1alpha2 (), you can tweak the weights of individual scores. Users might be setting up these weights assuming that the defaults are 1. This is not a problem with the old Policy API, because you have to specify all the scores with their weights. So, if we decide to change the weights, better do so now for v1alpha2.
Daniel, I don't expect that 50% of k8s workloads have preferred pod/node affinities, but I don't have data to suggest an alternative estimate. I guess we could make the change behind a feature flag that gets graduated over multiple releases.
It is permissible for the default to change over versions, in fact that's the way to do it for API types. Since the type is alpha, I guess it's OK to just do this. So, if the config file is v1alpha1, do the old defaults; if it's v1alpha2, use the new ones. How does that sound?
sgtm. But the question of what's a good weight for api-facing scores still remains. By default we only have 4 non-api-facing scores: NodeResourcesBalancedAllocation ImageLocality NodeResourcesLeastAllocated DefaultPodTopologySpread So my two proposals are actually the same number: 2 :sweat:
What about the case where a config file is not provided at all (which is very common across all cloud providers)? I guess we will have to keep the old defaults, and so we didn't actually solve the issue, right? it will also be confusing to users who pass a config file that doesn't change any of the weights.
If there's no config file at all, I think it's fair game, users can easily get the old behavior if needed. Others might disagree with me, but that's what I think. A completely separate thought: are you sure changing the weights is good? What about changing the way they're evaluated? E.g. in ML it's common to use e.g. a softmax function at the last layer to force a NN to make a choice / assign a consistent set of probabilities to each option. I think the way that might look here is instead of using F(N) as F's priority for node N, use exp(F(N)) / SUM (N0-Nn) { exp(F(N)) }. This would have the effect of magnifying minor preferences. Basically, I'm claiming that priority functions should be distributing the same amount of "probability mass" over the N possible nodes. It's not fair for one function to say ".3" for everything while another says ".6" for everything. You maybe need to normalize this before applying the weighting function anyway. But if you do the softmax, you might not even need the weight, as priority functions that give each node the same score will basically be no-ops, letting ones with differing opinions win.
Interesting proposal, but don't you think weights will still be needed because unlike NN, the scheduler didn't go through a training phase to discover which features (in this case priority functions) provide the most important signal?
Let's put an scenario with three simplified scores: selected: a score that prefers a particular set of nodes (maybe a particular zone). : existence of an image. : low utilization Now let's assume they give the following scores: Notice that they all have the same distribution (so they all give good signal). Even when using softmax the sum of scores (with the same weight for every score) will be: Nodes 1, 4 and 7 have the same score and thus the same probability to be chosen. I would argue that the user intention was that node 4 would be selected. It's still called a preference because if would give 0 to all nodes, then the pod can still go to the nodes, and the automatic scores take all the decision. Now, in the default scheduler, we have 4 automatic scores, so the user preference is quite unlikely to get respected. It's not necessarily a matter of good signal from the score.
Yup good points-- I agree a weight is still needed. If all the numbers are 0 or 1 (I had thought that it was a floating point and not a binary signal) then I don't think softmax is going to be different than a linear normalization (?). I guess it's debatable whether one wants a priority function that selects fewer nodes to have its opinion on those nodes be weighted more highly, which is what normalization would do. Anyway sorry if this is a rabbit hole. :) I think this idea would make more sense if the priority functions could express a strength for their preference and not just yes/no.
It's not binary necessarily... I was over simplifying. Node affinity and image locality certainly are, but node utilization is not.
It might be worth some experiments, then... if you have some way of telling whether or not it's actually better? Aldo Culquicondor <:
We have an HA use case that involves spreading pods across nodes and zone with preferred scheduling since we'd rather schedule the pods than not. Before 1.18, we've been using the something like the following: Now with the 1.18 beta, we tried the following: The later yields better scheduling results than the former on a 9 nodes multi-zone cluster with 3 nodes in each node. So while was better it surprisingly didn't yield a perfect spread. So I decided to combine both and ... boom ... we had a perfect pod spread for our deployment with 9 replicas. I'm not sure how these results fit into the discussions so far but I would have expected either option by itself to yield a better spread. Combining the two lets me reinforce my desires to the scheduler. IMO, I would like to see the scheduler give my explicit request more weight.
thanks a lot for sharing. Giving more weight to explicit requests is what we are proposing. However, I'm a little surprised that you need these constraints. The scheduler already spreads by default when using services, replica set or stateful sets. Are these pods not members of any of those? I assume you are not modifying the scheduler configuration? But, of course, there are other scores in play. It makes sense that spreading constraints are more effective than antiaffinity, because there is a lot of antiaffinity with the node that is ultimately selected as well. Adding both specs works too. However, this increases scheduling latency, as we have to calculate more things. The main motivation for this issue was actually to make scoring more effective for the opposite case, which is not done by default: "packing". But your use case is just as valid.
These pods are part of a deployment. I tried without any scheduling preferences and the results are similar to podAntiAffinity preferred scheduling. Here is the baseline deployment yaml used for the tests: I have not modified the scheduler configuration.
Yes, deployments are affected because they create a replica set that owns the pods. But there are always other scores at play that might also be important, which is why a "preferred" spreading is never going to be perfect. This is kind of the point that was trying to make: guarding users from over-fitting and degrading the clusters in some way.
What if we reduce the scope to just PodTopologySpreading, which is beta since 1.18? We could give it a weight of 2 now, before going to GA.
Aldo, one thing to take into account is the new default spread logic which is based on default pod topology spread constraints. For default spread constraints, I think the weight shouldn't increase, so things will get a little complicated with how we treat this case.
Given that default spread constraints is explicitly specified by users in , maybe we can assume they are already aware of adjusting the "Weight" in ? While for regular users, they won't touch , so a default non-1 weight is a no-op for them.
the default constraints (which we plan to introduce to replace the current defaultpodspreading score plugin) are not explicitly specified in PluginConfig.
Maybe I missed something, are we talking about the same default constraints defined here?
yes we are, but isn't the plan to have a default value for DefaultConstraints when not set: two constraints, one for zone and one for hostname, which mimic the current defaultpodspread behaviour and allows us to deprecate it?
I think the only way around that is to introduce a weight for default constraints in the plugin config. And then set it to 50% for the k8s default.
hmm, this will be a confusing configuration parameter to users. Another approach to consider is to bake it into the TopologySpread plugin instantiation without needing to pass it through PluginConfig (similar to what we did for volumebinder in the past: ).
urgh, this will be tricky to do since defaulting will be done via component config, and so we don't have a way to communicate whether the values were defaults or explicitly set by the user. If defaulting is done in then it might be possible, but we shouldn't do that.
Defaulting is going to happen in in the near future
We can have it in the internal type, but not the external one then. This will hide it from the users. But how do we communicate that to conversion logic?!
It doesn't seem there is a way because defaulting happens before conversion...
To make the new parameter semantically clear, we can treat it as a weight override rather than a percentage of the plugin weight, and so it's default would be 1 rather than 50%, internally the plugin computes the ratio as the override weight divided by the plugin weight, which it uses to scale scores computed for pods that don't have topology spread specified in their spec.
Got it now. I thought the default constraints would default to nil if not specified by users. I think we should introduce a field called "multiplexer" and default to 1, so it's friendly to defaulting and conversion.
+1.
What if we start giving meaning to MaxSkew for scoring? If you set a low MaxSkew, you probably want it to have a higher effect than if you set a bigger value. We can discuss the exact formula, but I think adding these semantics make sense. On Sat., Apr. 18, 2020, 5:50 p.m. Wei Huang, <:
That's possible. Current Score function doesn't take into consideration - it's just following a simple "the less number of matching Pods a topology has, the higher score the Nodes on that topology are given" principle.
Note that if we add another parameter to the args, it would only be relevant for scoring, adding to the confusion.
: This score calculation is somewhat complicated, but can be resumed as: Calculate number of matching pods per topology. There is some new feature that takes in consideration the skew Normalize by . The effect is that pods with the lowest count get a score of . Then the rest of the scores go from a certain number to . A score is given to a node with when is . This formula has the effect of reducing differentiation among nodes as the increases. : The formula is very simple, the score is equal to the percentage of free space (averaged for cpu and memory). Each score on their own give the differentiation we want, but they kind of have different scales. As a result tends to be stronger. I'm using a 9 nodes (3 in each zone) cluster as an example. I've set different initial loads, and in all cases I simulate a Deployment of 30 pods (each assumed to take 1% of node resources). Here are the scenarios, where numbers are percentage of use: For equal score weights (current configuration), this is the distribution of new pods: for both nodes and zones Note how a node with only 40% utilization gets only 1 or 0 pods. For double score weights for both nodes and zones Only at 80% utilization we observe bigger skews, which is reasonable. But the spreading is still pretty good at 40% For planned default spreading See , but this might change, depending on feedback here. I think this is enough justification to increase the weight of to 2 in its 2nd beta release.
Sounds reasonable. It's mathematically equivalent to changing the formula to e.g. , right? Have you considered doing that instead of changing the weight? It seems like that'd help everyone except those who specifically already increased , instead of just people who are using the defaults?
It'd make sense to me if the available functions were in the same scale / units so that they'd compose better, but it's not obvious how to do that--what would the units be? One thought is to make the units "std deviations from the mean"--if we knew the output frequency of each function, we could maybe do that. So basically this would measure how unusual a preference is, I guess. (I don't think this is an actionable thought. It really bugs me that the outputs of these functions are combined when they are not really in the same units, though!)
Yes 100 is the maximum score we have defined any plugins to have. I'm not convinced this would work. The variance in my examples for initial resource usage is quite high. So it would still get a lot of importance. But, you have triggered me to start thinking about the linearity of the scores. The formula is good because it reduce the variance as increases. However, that reduction is too aggressive. On the other hand, is completely linear. I'll experiment changing the linearity of either function, but preferring to do so for the beta plugin.
Here go 2 alternatives, with the same situations as We either solution, we wouldn't need to modify the weights Although we are not getting as good spreading as simply doubling the weight. But this is only in the last scenario, where we have usages of 80%. So I would call this rather a win (we should be paying greater attention to resource usage at this point). I think option B is simpler to understand and gives slightly better results (only noticeable in the last case). So this would be my algorithmic preference. However, I'm fine with option A.
Yeah, that's a great idea. You can generalize that and give each function a both a linear weight and a "gamma" (name from image processing): or even go for (but in this case I'm in favor of just making a targeted fix to that one function)
Awesome, Any concerns against option B? Given that it's an older feature.
how would option B impact node ranking in the absence of spreading? would image size scores start to have bigger impact?
I think SIG Scheduling should probably have a discussion. I could go either way, my naive inclination is to just go for the change, but I'm sure it'll break someone. We probably need a way to get back the old behavior, or a suggested weight change that would approximate it (not clear how well that could do though). I also have question, which is how well does this new value compose with other functions? I feel like the proper answer looks like an evaluation criteria + {long list of scenarios} + {list of workloads to schedule} + randomized testing / benchmarking. Basically the multi-dimensional version of the analysis here. That sounds like a lot of work but it would permit actual optimization of these functions.
configurable gamma is an obvious way... :/
I don't think we want another config parameter, we already have the weight concept and it is configurable, so I think we should go with that, just double the spreading score weight by default, and users who don't favor this behavior can configure it back to 1.
Yeah, that sounds expedient. ahg-g <:
In short, yes. Although that score has a mechanism to avoid the "node heating problem" we could have undesirable consequences unless we carefully analyze the impact. But that same reasoning doesn't apply to option A (improving PodTopologySpreading), for the following reasons: The score in 1.18 had pretty weak signal, that was only useful with very small number of pods. We have drastically improved this . So it can only be compared to other scores now. The score can now be tweaked in the PodSpec through maxSkew I would argue that option ~B~ A is better that simply doubling the weight because its signal is more conservative when it matters (nodes have high usage).
I am not clear on the following: you say "we could have undesirable consequences unless we carefully analyze the impact." but then you argue for the option at the end, can you clarify? We had default spreading that was giving similar strong signal, we should compare with that. My concern with modifying the is that we may break someone. Increasing the weight is something that users can revert, while option B can't be reverted.
Sorry, I meant option A, improve PodTopologySpread
One potential advantage to doubling the weight is that topology spread currently offers a signal comparable to default spreading, which we can think of as a baseline for spreading signal since it has been used for a while now. Doubling the weight will allow admins to revert back to the old behavior if they wish to do so. Also, from my perspective, doubling the weight is giving better placement at high utilization ( vs )


Once we upgrade to go1.14, we can use the new CipherSuites and InsecureCipherSuites functions and CipherSuiteName function to build the supported cipher suite flags. /sig api-machinery
/sig auth
/cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
as the project was upgraded to go-1.14+ is that still something that needs to be done?
yes, these manually maintained maps should be swapped to the stdlib impls:
/assign
/triage accepted Thanks,


What happened: https://k8s- These tests failed consistently: Anything else we need to know?: Volume snapshot tests need to be reworked to use / helper functions that support block volume mode. In addition, csi-driver-host-path fails when restoring block volume snapshots. /assign /sig storage
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : What happened: https://k8s- Anything else we need to know?: Volume snapshot tests need to be reworked to use / helper functions that support block volume mode. In addition, csi-driver-host-path fails when restoring block volume snapshots. /sig stroage Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
hostpath driver fix:
E2e test fix: (needs hostpath driver release first).


<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: pull-kubernetes-e2e-gce Which test(s) are flaking: Testgrid link: Reason for failure: Anything else we need to know: Triage board shows flakes here spiked dramatically on 3/4 /sig node /priority critical-urgent /milestone v1.18
are you able to assist if this is related to recent topology work?
sure, I'll take a look
is this the test you just in ?
As mentioned above, this is a new test that was just , and the test sometimes fails. I don't see that any of the recent topology manager changes would cause these failures. I'll be happy to help out and dig more if or others think otherwise.
This is me, the bugs are untriaged containerd / Kubelet failures with low occurrence rates that this test now exposes. Investigating.
still flaking in non-dockershim environments: pod pod-submit-status-0-14 on node kind-worker2 container unexpected exit code 128: start=1970-01-01 00:00:00 +0000 UTC end=2020-03-06 16:16:54 +0000 UTC reason=StartError message=failed to create containerd task: OCI runtime create failed: starting container process caused " container init caused " mounting "/var/lib/kubelet/pods/-819e-4840-8369-" to rootfs "" at "" caused "stat /var/lib/kubelet/pods/-819e-4840-8369- no such file or directory""": unknown `
Different reason this time. Will continue to enumerate failure modes.


What happened: A SystemOOM event was not reported even when the memory limit hit was the root cgroup memory limit. What you expected to happen: A SystemOOM event should have been reported. How to reproduce it (as minimally and precisely as possible): Do not set or , and set to disable memory eviction on the kubelet. Run a pod without memory limits which slowly consumes memory, such as with from Use arguments -mem-alloc-size", "100Mi", "-mem-alloc-sleep", "10s". This should trigger an OOM on the root cgroup, but no event will be generated. Anything else we need to know?: cAdvisor populates ContainerName and VictimContainerName from matching the regexp: instead in the kubelet. cc
/sig node /priority important-soon


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: k8s 1.17.2 config kubelet with those parameters cgroupDriver: cgroupfs cgroupRoot: /docker cgroupsPerQOS: false kubelet log always prints those E0303 17:25:03. ] Failed to get system container stats for "/docker": failed to get cgroup stats for "/docker": failed to get container info for "/docker": unknown container "/docker" E0303 17:25:03. ] eviction manager: failed to construct signal: "allocatableMemory.available" error: system container "pods" not found in metrics What you expected to happen: kubelet does not print those error information How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig node


Which jobs are flaking: pull-kubernetes-integration Which test(s) are flaking: TestNominatedNodeCleanUp Testgrid link: Reason for failure: /sig scheduling
/assign
any progress on this one?
I was working on it and tried to find the root cause. I will give my solution within this week
Another one
Still happening cc
/assign
/close dupe
Closing this issue. <detailsIn response to : /close dupe Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


/kind cleanup /sig storage /sig node Description: Feature gate VolumeSubpathEnvExpansion was GA in 1.17 and is marked for removal in 1.19 Related Issues: GA Release:
/assign
/cc


When using the package, one can easily add a flag to a CLI tool. However, the output of invocation with is hard-coded to . As such, the package is not very usable in non-Kubernetes projects. /wg component-standard /assign


What happened: there are many kubelet warnings in the systemd logs I initialized the cluster with kubeadm init (no modifications) What you expected to happen: Not to have that many kubelet warnings in the system logs Anything else we need to know?: Environment: Kubernetes version (use kubectl version):Cloud provider or hardware configuration:OS (e.g: cat /etc/os-release):Kernel (e.g. uname -a):Install tools:
I have the same issue
should add a option or an annotationo to resolve this /sig node /area kubelet
Same issue kops, kubernetes 1.18.3, aws cloudprovider
Same issue on v1.17.4 installed with rancher
Same issue on v1.18.0 ~
Same issue on 1.17.6
Happens on v1.18.3 as well. This issue only occurs when container has some fsGroup set in its securityContext AND volume being mounted is either secret or a configmap. I guess they are being re-mounted every 70s or so. It's okay when the volume being mounted is a standard PV - log only appears once and doesn't get repeated.
I'm seeing the same issue. I'm willing to fix it.
I've digged into this a little bit further. I believe this is just a annoying warning message. AFAICS, it does not show any issue... I was reading the kubelet, its volume manager and the volume plugins source files. It seems that's the expected behavior. The desiredStateOfWorldPopulator sync loop always call the . The finds the volume plugin and check if the plugin requires mount. Secret plugin, wich is the plugin in discussion, always returns . Thus, the of the volume manager remounts the volume every time. I think the problem is in the which prints the warning message. It's more and I believe it does not take these plugins into account. The secret plugin is as it is now since . Is that make sense to you?
/assign


What happened: Pods get stuck in terminating state, as kubelet failed to unmount volume: nodev on /var/lib/kubelet/pods/-120c-45c4-aeac- type hugetlbfs (rw,relatime,pagesize=2Mi) What you expected to happen: Pod to get deleted and volumes to be cleaned up. How to reproduce it (as minimally and precisely as possible): kvm based virtual machines on ubuntu 16.04. create a pod with hugepage volumes () Delete the pod. Anything else we need to know?: Changes are related to mentioned commit id: In this file : , line no 72 : pageSize, err := resource.ParseQuantity(strings.TrimPrefix(opt, prefix) + "i") The mount already has a pagesize 2Mi in my case as shown here : (rw,relatime,pagesize=2Mi) But the code here suffix/appends "i" assuming pagesize can be only of type 2M or 1024M resulting in failure to clean up this volume. Documentation from code: / NOTE: Adding suffix 'i' as result should be comparable with a medium size. // pagesize mount option is specified without a suffix, // e.g. pagesize=2M or pagesize=1024M for x86 CPUs pageSize, err := resource.ParseQuantity(strings.TrimPrefix(opt, prefix) + "i") if err != nil { return nil, fmt.Errorf("error getting page size from '%s' mount option: %v", opt, err) } when the pod is deployed, kubelet creates the mount and we can see from logs that it already has pagesize=2Mi for my environment. Environment: Kubernetes version: kubectl version Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2020-03-25T14:58:59Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2020-04-16T11:48:36Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: Kvm hypervisor based virual machine setupOS (e.g: ): NAME="Ubuntu" VERSION="16.04.6 LTS (Xenial Xerus)" ID=ubuntu IDLIKE=debian PRETTYNAME="Ubuntu 16.04.6 LTS" VERSIONID="16.04" HOMEURL="" SUPPORTURL="" BUGREPORTURL="" VERSIONCODENAME=xenial UBUNTUCODENAME=xenial Kernel (e.g. ): Linux master 4.4.0-169-generic -Ubuntu SMP Tue Nov 12 10:38:00 UTC 2019 x8664 x8664 x8664 GNU/LinuxInstall tools: Network plugin and version (if this is a network-related bug): Others:
/assign bart0sh
/sig storage
What about this change ?
The change looks good to me. Would you be willing to submit a PR?
& : Thanks for the quick turnaround.
We are seeing this issue in 1.18.6 version, can you please push the fix to this branch as well.
Could we know when the problem was introduced? As I suppose we don't have the same issue with K8S 1.17.x release?
Hi same issue found in 1.18.2
It is not yet ported into v1.18.6 as well.
Can you submit your PR for release-1.18 branch?


Currently we don't have an easy way of determining what number of requests are apiserver self-requests. This has historically bitten us, since we used to have a 50(?) QPS limit on apiserver self-requests (which thus has been removed). Even so, it would be a good idea to know this generally, since this may be helpful in pinpointing performance regressions for apiserver. /sig api-machinery instrumentation
Can this issue be considered a newbie one? I'm hunting for issues which might be a good first PR.
In the triage meeting was mentioned that would be good to have similar issues to track other well known actors like controller-manager and kubelet. But let's keep this one for api-server calling itself to have it done. /cc
/cc


[x] Original approval issue: 09 July 2020 [x] Two weeks soak end date: 24 July 2020 This issue is created to allow edit by /wip /hold According to this APIsnoop query, there are still an endpoint which is untested. a set of 3 Pods with a static label within a Namespace. that all 3 Pods with the label in the Namespace are created. the set of Namespaced Pods with a label via DeleteCollection. that all Pods with the label have been deleted. Discover useragents: List endpoints hit by the test: Display endpoint coverage change: Two other endpoints tested here have tests in progress to conformance already. If a test with these calls gets merged, test coverage will go up by 1 points This test is also created with the goal of conformance promotion.
/sig testing /sig architecture /area conformance
During Conformance Meeting: Is there a way to write this test so it is resource agnostic? If we are just changing the resource... can be parameterize the resource-type
/assign
GitHub didn't allow me to assign the following users: heyste. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Test to be created using "Polling" and not "Watch tooling"
/reopen Merged Issue open, waiting for Promotion PR
Reopened this issue. <detailsIn response to : /reopen Merged Issue open, waiting for Promotion PR Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
The test appears to be flaking, need to reach out to #sig-node for some feedback.
Can you point us to the link of the flaky tests? Thanks!
The Promotion The testgrid Thank you!


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Same symtoms as The detach is not successful until controller-manager is restarted controller-manager logs: What you expected to happen: The detach operation should know that a node is not active or its been deleted and should assume the disk is detached. How to reproduce it (as minimally and precisely as possible): This seems quite hard to reproduce, but we do run into it quite a few times where the pod is stuck in , because the controller-manager is unable to detach disks from deleted nodes. Anything else we need to know?: Environment: Kubernetes version (use ): v1.15.11 Cloud provider or hardware configuration: azure (self-managed) OS (e.g: ): Ubuntu 18.04.3 LTS Kernel (e.g. ): 5.0.0-1027-azure Install tools: kubeadm Network plugin and version (if this is a network-related bug): Others: /sig cloud-provider /area provider/azure paging
/sig azure
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig azure Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cloud-provider
/area provider/azure
hi have you done sth manually on the node? if node is deleted finally, the disk would be in unattached state finally, then disk attach to another node would succeed finally.
as discussed offline, the issue is probably due to vmss instance compute name change(incorrect format: , correct format: ), user should not change vmss instance compute name, otherwise it would break disk attach/detach, since compute name is used in vmss node name passing, code is here:
Hi We don't change the instance compute name - however what i do see in the azure metadata service is in the form - What you're referring to seems to be the computer name, and we do use the defaults as provided by azure. See The attach/detach operations works fairly well most of the time, and I think we're running into some edge case here. One thing i might have missed out is the fact that there seems to have been an issue with the VM at the time. So the VM was present, but azure platform would return 400 bad request for a detach operation which probably led to this .. But when the VM was eventually deleted by the autoscaler, the re-attach was never attempted by the reconciler it would seem. and the last message about that disk was
should be a valid compute name according to , do you know why we define the format like this?
The function is used to get the name from IMDS, and its format is .
From the above errors, seems the request has been sent to VMSS but failed because it's not active. Do you know the VM state during that time?
I am not sure of the state at the time, but i believe the VM was in some kind of failed state. The VM has since been deleted, but we had errors on the scaleset at the time. The events on the VMSS look like this. The VM was eventually deleted by the autoscaler, but the disk was never re-attached to a different node, and the pod remained in for about 2 days until controller-manager was restarted.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Apparently we hit this issue today as a result of replacing the nodepool. We 2 new nodepools and deleted the old one, and now we have 718 of 1672 pods stuck with this error in one of our clusters. The volumes are failing to be attached to new pods, the VMSS for old nodepool is removed. In AKS portal disks are still reported to be attached to VM instances in the VMSS that was deleted. hundreds of pods are stuck in Init state won't likely go forward. /remove-lifecycle rotten
/remove-lifecycle rotten
I have seen similar issue when handling one customer's case, it's probably caused by broken vmss cluster, pls file a ticket to VMSS team to fix the issue.
I happened to find this issue in one customer case, this issue is caused by obsolete cache in kube-controller-manager, should skip attach/detach retry when it's 400, below are the details, following two instances were deleted 3min ago before kube-controller-manager trying to attach/detach: workaround restart kube-controller-manager should work
/assign


After has been implemented and enabled the 5k node kubemark run has extra ~10k lines of logs like this ones: Example kubemark 5k run: We should get rid of that, it doesn't bring any value only pollutes the log
/sig scalability /cc


Graduate seccomp to GA, as defined on the respective . This is an umbrella issue to track all outstanding work. [x] API Server changes (MERGED) [x] Kubelet changes (MERGED) [x] Mark flag as deprecated (MERGED) [x] Update PSP to support seccomp fields (MERGED) [x] Document Kubernetes seccomp support (MERGED)[x] Add seccomp least privilege for docker sandbox (MERGED) [x] Add seccomp least privilege for kuberuntime (MERGED) ) /area kubelet /area kubelet-api /sig node /area security /assign
/cc
just a heads up that we have concluded all the core items for seccomp to go GA. There are still 2 pending PRs that ideally would also go in 1.19, but we are waiting for them to be reviewed: and
needs to be rebased apparently and isn't just waiting in the merge pool as verify is not passing.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
I think the warning mechanism was blocked by a not existing API. This is now tracked in
/remove-lifecycle stale
Updated the description to given that the warning mechanism is now set to that milestone.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale
/priority backlog
/triage accepted
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/lifecycle frozen What is the remaining work here?
I guess we have to follow-up on and implement: I can take care of that if we agree that we want to ship that in 1.27. /cc


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: I created several Workloads which require up to ten ports for both UDP and TCP protocol. Now my logs get spammed with . What you expected to happen: should consider -combination instead of only . How to reproduce it (as minimally and precisely as possible): Deploy any service with a node-port with both protocols, e.g.: Anything else we need to know?: This was already acknowledged by here: However, as this thread addresses a different bug and seems kind of stuck, i'd like to suggest that this might be addressed in a new context. Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Deployed via Rancher v2.4.3
/sig api-machinery
/remove-sig api-machinery /sig network
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
How did you get into this situation? The allocator should not have granted the same node port twice?
Oh right, that issue. The only real way out is to de-allocate the NodePorts and then re-allocate them, I think
As this happens for all of my ports in the said Production Application (the one where my log gets spammed) as well as complete fresh installation (where i verified the bug), i think there's no "way out" except fixing the bug. As said, for me it's necessary to assign both protocols to the same port. This might be a niche-use-case, but checking the it's actually very likely for ports to allow tcp AND udp. I'm not sure about the history of this, but it might be that issue was the origin, which was adressed in PR ("Allow services which use same port, different protocol to use the same nodePort for both"). But this PR doesn't contain the adjustment of the said , so still considers the behaviour as invalid, while it's actually allowed and supported.
/assign
Ahh, I thought you meant that you had accidentally been allocated the same nodePorts and I was very confused when you said "all of them". I have reproduced this issue, and I can see it by inspection. The obvious fix would be to collect ports split by protocol, but I think that's not right since we don't store it split. An even simpler fix would be for to build a . Then for each protocol, for each port, build that contains all the port numbers, but remove any duplicates ONLY IF they are duplicated across protocols. Duplicates in the same protocol should be retained (to report an error later). I don't have time to tackle this tonight (and write a test), but it seems like a good bug.
perhaps I can fix this problem /assign
things may be a little more complicated. Right now the is not aware of the protocol of the port so I guess simply rewriting function to return is not enough, the port allocator will still return on same node port even with different protocol. perhaps we need to refactor the to make it protocol-aware.
I don't think we need to do make the allocator be protocol aware. IIRC we allow you to use the same nodeport on different protocols ONLY within a single service. This means (I think) that we can do hyper-local deduplication as we collect the nodeports for each service. If we find the same nodeport number with different protocols, just remove the duplicates because it is allowed. I.e. a service that looks like: return - deduped. if ANY OTHER service tried to use it would be an error at the "rebuilt" allocator Likewise if a single Service claims to use the same nodeport number more than once on the same protocol, it should be an error: return - NOT deduped.
obviously, we want a great unit test here :P
ah got it! thanks for your detailed explanation!
Hi! I have filed a PR to resolve this issue, could you take a look?


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: add a CRD resource with kind only a single character while cause kube-apiserver crashslice bounds out of range What you expected to happen: add validation to limit CRD kind definition or fix the bug of slice out of bounds How to reproduce it (as minimally and precisely as possible): apiVersion: kind: CustomResourceDefinition metadata: name: spec: group: version: v1 names: kind: R plural: rs singular: r scope: Cluster Anything else we need to know?: apiserverlogs: 10:05:32. 6 ] Shutting down OpenAPI controller E6606 10:05:32. 6 ] Observed a panic: runtime.boundsError{x:-1, y:1, signed:true, code:0x3(runtime error: slice bounds out of range [-1:]) goroutine 1670 [running]: (0x41458a0, Gxc00ed66d40) /workspace/anago-v1.17.G- +0xa3 (0x0, 0x0, 0x0) /workspace/anago-v1.17.G- +0x82 panic(0x41458a0, Oxc0Oed66d40) +0x1b2 (0x7493232, 0x1, 0x447d778, 0x1, 0x130, 0xc00ed13680) /workspace/anago-v1.17.0- +0x2b8 ("builder).descriptionFor(€xc007b351f0, 0x0, 0x0, 0x4481982, 0x6, Gxc00ed06d00, 0x18) /workspace/anago-v1.17.0- +0x3c6 ("builder).buildRoute(GxcGG7b351fG, 0xc00ed66d00, 0x18, 0x0, 0x0, 0x447f0F3, 0x4, 0x447f71b, 0x4, 0x4481982, ...) /workspace/anago-v1.17.0- .go :281 +0x197 (0xc66b7bcfd0, 0xc005d24a3c, 0x2, 0x101, 0x1, 0x2, Gxc60b76aec8) /workspace/anago-v1.17.0-rc .2.10+70132b0F130acc/src/k8s .io/kubernetes/output/dockerized/go/src/k8s .io/kubernetes/vendor/k8s .io/apiextensions-apiserver/pkg/controller/openapi/builder/builder .go :146 +0x49d (0xc66b7bcfd0, 0x0, Oxb, Gx447f1db, 0x4, 0x1) /workspace/anago-v1.17.0- +0x1a8 ("Controller).Run(0xc0012c8050, Gxc0G1d61200, 0xc007a6d1e0, 0xc069cced80) /workspace/anago-v1.17.0- +0x2da created by /workspace/anago-v1.17.0- .go :236 +0x244 panic: runtime error: slice bounds out of range [-1:] [recovered] panic: runtime error: slice bounds out of range [-1:] Environment: Kubernetes version (use ): 1.18.0 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ):Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig apps
/assign
There is a typo in the subject.


<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: Which test(s) are failing: Since when has it been failing: Testgrid link: Reason for failure: Anything else we need to know: /sig testing /cc /priority critical-urgent /milestone v1.19
/assign
I see the PR merged, however, the job still failing in the recent run 22:31 PDT Failures for bootstrap-e2e-minion-group (if any): W0611 06:08:56.190] 2020/06/11 06:08:56 Step './cluster/log-dump/log- /workspace/artifacts' finished in 2m14.442417098s W0611 06:08:44.597] ERROR: () [/usr/bin/scp] exited with return code [1]. W0611 06:08:46.070] scp: No such file or directory W0611 06:08:46.070] scp: /var/log/node-problem- No such file or directory W0611 06:08:46.072] scp: No such file or directory W0611 06:08:46.072] scp: No such file or directory W0611 06:08:46.073] scp: No such file or directory W0611 06:08:46.074] scp: /var/log/node-problem- No such file or directory W0611 06:08:46.075] scp: No such file or directory W0611 06:08:46.075] scp: No such file or directory W0611 06:08:46.079] ERROR: () [/usr/bin/scp] exited with return code [1]. W0611 06:08:46.084] ERROR: () [/usr/bin/scp] exited with return code [1]. W0611 06:08:51.917] INSTANCEGROUPS=bootstrap-e2e-minion-group W0611 06:08:51.918] NODENAMES=bootstrap-e2e-minion-group-dq3m bootstrap-e2e-minion-group-gjk0 bootstrap-e2e-minion-group-qpxw I0611 06:08:54.017] Failures for bootstrap-e2e-minion-group (if any): W0611 06:08:56.190] 2020/06/11 06:08:56 Step './cluster/log-dump/log- /workspace/artifacts' finished in 2m14.442417098s W0611 06:08:56.191] 2020/06/11 06:08:56 Running: ./hack/e2e-internal/e2e-
I see the issue closed by the PR however the job still failing since 06-10 16:31 PDT
looks like we just turned green :)
cool! :)
Yes it's indeed ... thanks for the support!


<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: https://k8s- Which test(s) are failing: sig-network] SCTP [Feature:SCTP] [LinuxOnly] should create a Pod with SCTP HostPort Since when has it been failing: Test [sig-network] SCTP [Feature:SCTP] [LinuxOnly] should create a Pod with SCTP HostPort has failed 3 times since it last passed at commit/. It started failing between commit/ (Tue Jun 09 13:25 PDT) and commit/ (Wed Jun 10 01:26 PDT): Testgrid link: https://k8s- Reason for failure: Anything else we need to know: I'm not sure if we already did it but there was a proposal to change from dockershim to containerd for the GCE jobs
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign /cc
GitHub didn't allow me to assign the following users: janosi. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign /cc Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
yes, that would break us... the test checks "are we using kubenet", but it didn't occur to me that we would need to deal with kubenet with CRI instead of kubenet with dockershim. It should be easy to make it check for both the dockershim rule and the CNI portmap rule though...
yes, according to the "Changes" link on the dashboard, containerd replaced docker in GCE right before the failures.
Let's see if I can make it :)
So actually the test is pretty generic (it just looks for and ) and it seems like it ought to pass correctly with CNI portmap too. I'm not sure what's failing. You may need to add some debugging...
OK, so as I can see, containerd's CRI implementation just ignores all HostPort definitions that do not have TCP or UDP Protocol value. I.e. containerd will not send HostPort mapping definitions to the CNI if the protocol is not TCP or UDP.
seems this is the reason why sctp was skipped
Yes, exactly. Introduced right after SCTP support was merged in k8s. Just found this in the package: Though not sure it "knows" the right runtime at runtime, because there is no user of it in :)
just curious, why do we skip in containerd and allow it with dockershim?
Hmm, do you mean why do not we skip in case of containerd? The reason is that I did not remember that containerd PR about the SCTP support. Let's see if I can skip it.
Hm, that is not true. My search engine was wrong. After a restart I found usage of that in e2e_node
Has a bug been filed upstream in containerd? Just skipping the tests seems like the wrong approach here.
sorry, I meant why we are not allowing it in containerd, does this feature should work in all container runtimes? CRIO, docker and containerd? if affirmative, we can submit a PR to containerd/cri to allow SCTP, I can do it if you don't have time
Excuse me, I misunderstood your question. Yep, why not. I mean, it could work in all container runtimes. As I understand the containerd implementation is based on a misunderstanding: that the host port mapping loads the SCTP kernel module and that would be bad for those apps that want to use user space SCTP. But as long as there is no SCTP socket created on a node the SCTP kernel module is not loaded.
I can initiate a discussion with containerd about this.
seems this dates back to the original SCTP KEP, seems like it should be a pretty simple fix (?) thanks for filing the bug and for the details there!
CRI-O seems to have the same issue:
Just confirming you want this filter to be gone now. Or do you need the filter to be manageable in some way.. filtered by a config switch?
For TCP and UDP HostPorts, the runtime is expected to do two things: a listening socket on that port so the kernel will think it's in use and not allocate it to anyone else iptables rules (or whatever) to route traffic from the host port to the container port For SCTP HostPorts, the runtime should do the second part (iptables rules), but should not do the first part (listening socket), for two reasons: Some users want to use "userspace SCTP", in which case they will be referring to SCTP at the Kubernetes level but need to keep the kernel module from ever being loaded. (This is discussed more .) If is not loaded when Kubernetes starts, we want to avoid having Kubernetes itself cause it to be auto-loaded, for security reasons. (More discussion of this .) So, runtimes should be filtering out SCTP for the opening-listening-sockets part, but not for the making-iptables-rules part. (Some implementations, notably CNI , don't actually do the opening-listening-sockets part, so they wouldn't be filtering at all.)
Exactly. The current state in the runtimes: Docker (dockershim, actually): OK. The logic was implemented together with the basic SCTP feature, so listening on the host port is filtered for SCTP, but the iptables rules are created Containerd (CRI, actually): the patch which was applied 2 years ago filters out every SCTP host port handling, because it filters SCTP ports from the list of port mappings that are passed to CNI (portmap) later. Actually, containerd (CRI) does not perform the "listen" part at all, so the patch is not needed at all CRI-O: the patch applied 2 years ago filters out everything. CRI-O imports the host port manager implementation from kubelet/dockershim. For this reason it has the logic that can filter out SCTP ports from the "listen" part, but still applies the iptables rules. I.e. the patch is not needed in CRI-O either.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale The test is flaky now :thinking: https://k8s-
I guess it is a timing issue? Do you think some extra polling rounds would help? (I did not disappear, but currently I am very loaded with my regular job...)
I've submitted a PR to fix it and another one to test the sctp connectivity, please review them, I've pinged you there :smile:


The purpose of the file for each Kubernetes repository is to provide a list of people who can assist the Kubernetes in the event that a security issue related to the repository is discovered or disclosed. As described in the file, those on the list should agree to our . Please update the  file for the apimachinery repository. After finding people who are willing to work in this capacity, you should add them to the list, then remove PSC members (except any PSC member will be working as a security contact for this repository). The list is GitHub usernames, optionally followed by an email address. If no email address is listed, the PSC will use the email address found on git commits made by the listed user. The file may already have people listed who are secuirty contacts. In that case, simply remove any PSC members who aren't also security contacts for the repo. See for more information /area security /committee product-security /kind cleanup /lifecycle frozen /priority important-soon /sig api-machinery


The purpose of the file for each Kubernetes repository is to provide a list of people who can assist the Kubernetes in the event that a security issue related to the repository is discovered or disclosed. As described in the file, those on the list should agree to our . Please update the  file for the sample-apiserver repository. After finding people who are willing to work in this capacity, you should add them to the list, then remove PSC members (except any PSC member will be working as a security contact for this repository). The list is GitHub usernames, optionally followed by an email address. If no email address is listed, the PSC will use the email address found on git commits made by the listed user. The file may already have people listed who are secuirty contacts. In that case, simply remove any PSC members who aren't also security contacts for the repo. See for more information /area security /committee product-security /kind cleanup /lifecycle frozen /priority important-soon /sig api-machinery


<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: ci-kubernetes-e2e-aks-engine-azure-master-windows-2004 Which test(s) are failing: [] Container Runtime blackbox test when running a container with a new image should be able to pull from private registry with secret [NodeConformance] Since when has it been failing: since enabling the Windows 2004 SAC release tests Testgrid link: Reason for failure: The manifest for is missing the new windows server version 2004. This has occurred before: I looks like we need someone with appropriate permissions to update the manifests once the images are ready. Anything else we need to know: There was some work done in to automate some of this. Is there something missing? There are a few other tests that are failing in this job but should be covered by updating /cc /sig windows
The test passed 2x consecutively. We can close this now. /close
Closing this issue. <detailsIn response to : The test passed 2x consecutively. We can close this now. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: pull-kubernetes-e2e-gce Which test(s) are flaking: Discovery should validate PreferredVersion for each APIGroup Testgrid link: Reason for failure: The test does not handle API groups which are or removed dynamically during the run of the test. Just because an API group is present in at the beginning of the test does not mean or will be present by the end of the test. This is particularly true for the dynamic/custom-resource API groups registered/unregistered during the e2e test run /sig api-machinery /assign
mitigation opened at
Saw this today in It seems to happen relatively often Likely due to the CRD created in: Should the ValidatingAdmissionPolicy test CRD have its group name template changed to match the existing test exclusion for groups ending in /cc


Which jobs are flaking: ci-kubernetes-node-kubelet-serial-hugepages Which test(s) are flaking: all Testgrid link: https://k8s- Reason for failure:
/sig node
/assign /cc
GitHub didn't allow me to assign the following users: cynepco3hahue. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign /cc Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
testgrid alert:
/cc
I started to look into logs and the issue that one of the kubelet restarts fails on: I will try to reproduce the issue locally.
any news? Have you managed to reproduce the issue locally?
I already have a fix that is waiting for merge -


What happened: This issue was originally reported via the Kubernetes HackerOne bug bounty by HackerOne user but we have decided that due to the high privileges required, it can be addressed publicly. A user able to create CRDs could create a malicious CRD such that listing CRs will cause enormous amounts of CPU usage on the API server. What you expected to happen: The function should not expand strings with more than two consecutive periods. Either it should collapse additional repeated periods or it should return an error. How to reproduce it (as minimally and precisely as possible): /kind bug /area apiserver /area security /committee product-security /priority important-longterm
How about we take it a step further by rejecting the jsonpath which contains when parsing it? as more than 2 periods seem to make no sense IMO
good idea. I have posted to do what you suggest.
/sig api-machinery
/cc


Looks like this test is never able to succeed because service does not exist: /kind failing-test /sig instrumentation
possibly related to
/assign
/assign
/assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
where did we get with this?
Looks like those tests didn't work for at least half a year and nobody cared... Should we delete this e2e test as well? If so I can help with that.
Yes, let's remove them. Would appreciate the help! /remove-lifecycle rotten
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
Still failing: /remove-lifecycle stale


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: For each NLB created, K8s creates an ingress rule on the node group's security group for each port exposed for each CIDR attached to the VPC to allow health checks to succeed. AWS limits the number of rules per security group to 60 by default. This quota is adjustable but multiplied by the quota for security groups per network interface cannot exceed 1000. I have a VPC with 16 CIDR blocks attached to it and a quota for rules per security group set to 80. Each NLB created adds 18 new rules (16 for Health Check, 1 for Client, and 1 for MTU) to the node group's security groups. This high number of unnecessary rules created makes it impossible to create more than 4 NLBs per node group (if each one has a single port exposed). What you expected to happen: For the health check to succeed, the NLB must be able to ping the node port for the service on the cluster's node group. So the node group's security group just needs an inbound rule for each ENI of the NLB, making the number of rules necessary on the cluster's node group directly proportional to the number of AZs the NLB is configured. How to reproduce it (as minimally and precisely as possible): Create a VPC on AWS with 8 CIDRs attached to it; Create a K8s Cluster; Create a Network Load Balancer with 6 ports exposed; If the default quota of rules per security group has not been changed, the NLB creation will fail with RulesPerSecurityGroupLimitExceeded. Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: OS (e.g: ):Kernel (e.g. ):Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig cloud-provider
/area provider/aws


Running : cpu usage goes to 2 cores @ 100% What you expected to happen: Way lower cpu usage. Environment: Kubernetes version (use ):Cloud provider or hardware configuration: GKE OS (e.g: ):Kernel (e.g. ): Linux felix-Latitude-7490 5.3.0-62-generic ~18.04.1-Ubuntu SMP Wed Jun 24 16:17:03 UTC 2020 x8664 x8664 x86_64 GNU/Linux
Hey I just did not understood, the cpu goes to 200% in the node or in the place where kubectl is running? Thanks!
/priority awaiting-more-evidence /sig cli
On my laptop running kubectl
Same issue on mac: kubectl version Client Version: version.Info{Major:"1", Minor:"17+", GitVersion:"v1.17.9-eks-", GitCommit:"", GitTreeState:"clean", BuildDate:"2020-07-17T19:00:19Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"16+", GitVersion:"v1.16.13-eks-", GitCommit:"", GitTreeState:"clean", BuildDate:"2020-07-17T18:48:53Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"} PID COMMAND %CPU TIME #TH #WQ #PORTS MEM PURG CMPRS PGRP PPID STATE BOOSTS %CPUME %CPUOTHRS UID FAULTS COW MSGSENT MSGRECV SYSBSD SYSMACH CSW PAGEINS IDLEW POWER INSTRS CYCLES USER #MREGS RPRVT VPRVT VSIZE KPRVT KSHRD kubectl 100.5 03:30.41 32/1 0 43 16M 0B 0B 1322 running *0[1] 0. 0. 274 36 16 + 117 + 3730 266+ 100.5 costel.lupoaie N/A N/A N/A N/A N/A N/A I noticed the problem after upgrade from 1.15. Maybe it't not kubectl itself but the way it communicates with external auth provider like IAM in my case.
oh I see. Thanks I've no access to cloud providers right now to test, but as you've provisioned in EKS using IAM and is using GKE it MIGHT be a thing. Will wait to see if someone else have access to cloud providers and can test this :)
Maybe I am wrong and it's just some busy waiting. In my case I noticed it when draining nodes with many pods that cannot immediately be stopped because it would break the deployments maxUnavailable limits, so kubectl has to wait until it can stop all the pods on the draining node.
In my case also this was triggered when pods couldn't be evicted immediately because of a violation of pdb.
does not lead to high cpu usage in my case, so I suspect it has nothing to do with IAM/GKE auth, but with evictions failing because of pdb (but I am not experienced enough to know kubectl works in the background).
I can confirm 100% (not 200%) CPU core usage for each process. Currently I'm draining four nodes in parallel (4x ), all four are waiting due to PDB's, and all are at 100% CPU most of the time. I'll try with a newer kubectl next time.
OK, so it seems to be related to PDB and Drain. I'll try to take a look into it, try to reproduce with the last supported clients. Thank you for reporting! /assign
Thanks for looking into it! And my apologies, I forgot to comment on this before: Our cluster is deployed on bare metal on premise, so it's probably not EKS og GKE related.
Is that possible you try draining with "disable-eviction" just to make sure PDB or eviction control is causing this? Thanks!
I found a newer kubectl lying around, so it's now v1.18.8 (1.17 didn't include it seems) I couldn't test on the same cluster, but I just drained a host from our staging env with a similar amount of pods to what we run in prod (~85 vs ~100), and I think you're right. CPU usage certainly didn't go above 20% at any point on any core, and that's basically background noise on this host.
So my theory here is that kubectl keeps waiting for the node to drain and this is causing a high CPU usage. I'm also creating a cluster here to reproduce the behavior, and see what part of the code is this being caused
Confirmed here :) Even in last kubectl v1.19.2 the problem happens. It seems also it's putting some pressure in API Server because of the amount of nodes vs PDB. disable-eviction is a "workaround" not desired if using PDB /remove-priority awaiting-more-evidence /priority important-soon
I'm hitting my head in the wall here trying to figure out why the goroutine causes the peak of CPU usage, so anyone with better skills then me in Go, feel free to join :) is the function taking the whole CPU apparently Will keep digging what part of this function is taking the whole CPU. When removing the and making this 'serial' it seems to keep the CPU in low usage but I don't know if this is the desired scenario, as draining a node could take much longer since other Pods may not be using PDB. EDIT: but it also seems that the drain does not goes further because I forgot to change the return/break stuff inside the loop. So I'm thinking tomorrow maybe create a new 'queue' of the 'failed because of PDB' Pods, and deal with them in a separate way.


Using scheduler like or , it always get unwanted output like: It simply because when adding the flags in scheduler, it will try to look up the And this will invoke: Since flag already has to prevent running other code when having : We should not print the klog info when using or right?
/sig scheduling
that's annoying. Can you help to fix it?
Ok I'll take care of this. /assign


Which test(s) are flaking: TestWatchHTTPTimeout Reason for failure: This test is flaking, but not enough to fail our presubmit CI (which currently retries up to 3 times and passes on a single green run). Observed in when trying to disable this flake toleration. Reproducer steps: test binary: with : This is reproducible on release-1.18 as well, though is happening an order of magnitude more frequently on master: /sig api-machinery
/assign I would like to give it a shot.
thanks for looking


There was an effort at one point to promote it to Beta This may have moved to Service Catalog repo So question, do we remove it from kubernetes/kubernetes ?
/sig apps /sig service-catalog Thoughts please!
I see removal of perma-alpha APIs as a natural next step after the "" work to put end dates on beta APIs
The PodPreset functionality is not used in Service Catalog, we have an issue to remove references to it in the svc-cat repo, see: In the separate issue, we will figure out how to implement the e2e functionality for injecting secrets into an application: IMO it's ok to remove it as right now there is no decision about how to use it in svc-cat and currently we do not have enough capacity to work on that. In Kyma project but we also . /cc
Many of us have been waiting for this to become mature since along time. Removing PodPresets would be quite sad, as it would allow many people to stop having to write their own mutating webhooks for things PodPresets could do really well. Also I had in mind to use it for automatically mounting common volumes into pods and much more.
Me too Doing my usual check of has-this-thing-got-to-beta-so-I-can-finally-use-it only to find PodPresets had entirely disappeared This feature was going to greatly simplify our manifests. We do have the mutating webhooks feature now, which we didn't when PodPresets were created. However PodPresets are a lot cleaner would have received a lot of use. Even chart designs could be greatly simplified with optional PodPreset manifests. So I was looking forward to getting them available for use. Things in alpha can never gain traction or significant interest because managed providers won't let you enable them. This is not really a case of 'perma-beta'. But, if they were not going anywhere then I can understand removing them.
Speaking of gaining traction and momentum, the 'Avoiding permanent beta' policy only has a nine-month countdown for beta features. This new policy clashes with the recent popularity of managed releases, which tend to lag upstream releases by 6-9 months (2-3 releases). E.g. the current . That is before users can start planning upgrades in the 3-6 months after that. So a brand-spanking new beta feature won't reach cloud provider users until 9-15 months after it reaches beta upstream. Then users will start trying it out, and if it works well and get some interest and momentum behind it, only to find it has been already removed from a subsequent releases because of the mandatory 9-month policy In practice AWS probably won't enable a beta feature that is already removed upstream, so we can basically expect beta features will never reach, or get a chance for real-world feedback from cloud provider users. It is going to be tough for when the real world can't test a release with a beta feature until after it has already hit mandatory removal
For long running alpha features I usually recommend announcing depreaction and removing after one release, instead of right away. For extending the beta policy please see:
Please don't remove this feature! We only just discovered it this week and it was positioned to solve a real issue with common volume specs, in the same way mentioned by It would be really sad to now see it removed when it was going to fix some integration issues for us!
Could you elaborate on why PodPresets were never promoted to beta and why you want to remove them now? It seems like a useful feature.
This feature would have been very useful in several operating scenarios, such as setting proxy settings before a deployment.
Can anyone comment on what alternatives are? Without this feature we'll have a lot of duplication. No one likes duplication. We can address the Environment variables via ConfigMap and the envFrom keyword, however I cannot see anything for volumes, imagePullSecret etc.
It seems the alternative is just to use mutating webhooks. And there is even a port of PodPreset to a mutating web hook by redhat:
-1000 Removing PodPreset is terrible. Shame on you!
It doesn't look much maintained and active, sadly
Related:


Currently we unconditionally move pods out of unschedulable status when an event happens. However, we can optimize this by setting some conditions for each event to avoid unnecessarily doing that: 1) PV, PVC, StorageClass or CSINode add/update: move only pods with a PVC. 2) Service add/update: only if ServiceAffinity plugin is enabled in one of the profiles 3) Node add/update: test the pods against the , this include: NodeAffinity, NodeName NodePorts, NodeResources and TaintsTolerations filters. 4) Pod delete: keep it as is (spreading or affinity constraints must be tested, which is expensive to do) This is necessary to avoid wasting scheduling cycles. Related proposal: /sig scheduling /assign
GitHub didn't allow me to assign the following users: adtac. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : Currently we unconditionally move pods out of unschedulable status when an event happens. However, we can optimize this by setting some conditions for each event to avoid unnecessarily doing that: 2) Service add/update: only if ServiceAffinity plugin is enabled in one of the profiles 3) Node add/update: test the pods against the , this include: NodeAffinity, NodeName NodePorts, NodeResources and TaintsTolerations filters. 4) Pod delete: keep it as is (spreading or affinity constraints must be tested, which is expensive to do) /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/cc I'm thinking of if we can amplify the scope: which is to say, not only enable/disable moving (all) pods, but also provides mechanics to let plugins notify the scheduler framework which particular pods should be moved. This would benefits in 's coscheduling implementation to move the preceding (N-1) pods back to activeQ when the Nth pod comes - which satisfies the "minAvaialbe" requirement. Moreover, this would help in getting rid of flushing pods back to activeQ .
The suggestion is not to enable/disable moving all pods in all events, this is only true for the Service events. For the other two, node and volumes, we will test each pod and move the ones that pass the condition. I agree that our proposal of integrating this with filters/permit plugins is more generic, but it adds a lot of complexity and potential for bugs, so we need to be careful with that I think.
One approach could be to start simple, validate and establish a reference point in terms of perf improvements, and then migrate to something more generic. The more generic approach will not be more performant than what is proposed in this issue.
Yes, filters/permit runs in the critical path, and move Pods isn't a lock-free operation. What I want to say is during the design/implementation of this proposal, keep it in mind that plugins may need to actively "suggest" moving particular Pods back to activeQ. Agree.
Right, we already designed the interface for such a potential eventuality:
just fixing double "moving" in title: /retitle Avoid moving pods out of unschedulable status unconditionally
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
/assign /unassign
/cc
what is the current plan? Is it to do the entire proposal in "Efficient Requeuing" doc or just the minimal checks proposed in the issue?
I plan to fix the problems mentioned in this issue, but not entirely following the logic mentioned here. For example, it's inappropriate to move a Pod with PVC once a CSINode happens - the Pod may fail due to a reason irrelevant to its PVC. What I'm proposing is more aligned with the Google doc, but there are 2 major distinctions: I'm not associating the interface to a particular extension point - the EventsToRegister is more a general hook instead of semantically associated with a particular scheduling phase. I'm not going to pass the changed event object to let users decide whether to move or not - that'd increase the latency, also impose an extra burden on plugin developers. Fixing this issue is the minimum goal I can promise in 1.21, but in the future, we may provide other interfaces like enable users to proactively move Pods inside scheduling queue.
That is true, but this is the case right now, not something new this issue is proposing to do. This issue is proposing to address the obvious moves that we can avoid with minimal complexity, which will also likely give us the most bang for the buck. Do you have a more detailed proposal for what you are planning to do?
Yes, the code is almost complete - PR1 () and PR2 (), I will send out another PR later to use ServiceAffinity as an example to show case how it works in details. Basically, the plugin can implement an optional interface called and return a number of ClusterEvents that a Pod failed by that plugin can be made re-schedulable, and also upon a Pod's scheduling failure, we store the "failed plugin" into , so that upon moving Pods in the internal queues, we're able to tell whether the Pod's failed plugins are associated/registered with the current event, and hence perform efficient enqueuing.
You can wait late today or tomorrow when the 3rd PoC PR will be ready, and it'd show you a whole picture.
FYI: the 3rd PR to showcase how it works: PoC: ServiceAffinity enqueue .
Do we have a benchmark we can use to prove that the changes are having a positive impact?
Also, are we planning to keep the existing checks that some events have?
Will do that. Yes, they will be kept.
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/triage accepted
I think we should still do 3: or even just NodeAffinity+TaintsTolerations The problem is that just one node is being and potentially we are retrying all pending pods. And this means that we are testing all Nodes for each of those Pods.
I think we can do sth. like this: upon a Node Add/Update, for each unschedulablePodInfo, only verify if the Pod can pass NodeAffinity/NodePorr/etc. filtering logic when the Pod failed by NodeAffinity/NodePort/etc. plugin. In this way, we can save cycles for Pods not failed with NodeAffinity/NodePort.
Latest status of this issue: Interface and implementation in framework side [x] Surface info of failed plugins during PerFilter, Filter and Permit [] [x] Store a cluster event to plugin map in SchedulerQueue [] Plugin implementation and perf evaluation.
/help Some tips/notes for contributors who are interested: please dedicate some time to the item(s) you claimed as code freeze is around the corner, we'd like to finish the above items as early as possible works as a sample you can refer, basically you need to think about which events can make the unschedulabed Pods failed by the specified plugins schedulable, and specify the reasons. Also give a benchmark test result proving it. to design a desired benchmark test, you may need to try the "churn operator", which means you need to rebase locally (unless they get merged) if you meet timeout issue frequently during running a 5000Nodes test, you can work it around by rebasing it's a good practice to run the benchmark test multiple times and eliminate the outliers. Here is a for your information.
I would like to take ‘ tainttoleration’ to learn something about scheduling plugins.
Thanks! I updated the tips to include a to automate running the benchmark tests multiple times.
I claim "nodeports" and shall learn about it firstly.
Can I try to deal with the "nodeaffinity" plugin
I would like take "podtopologyspread" plugin.
Can I take ?
I can take plugin.
Can I take ?
Can i take ?
I opened and will add testcase and test result tommorrow.
I would like to take on
I think I'm left with
updated with proper owners. Thanks! Please assign the reviewer to me when your PR is ready. /remove-help
Wei, there were followups from , are we tracking them somewhere?
the only followup is I will work on that. Let me know if any other followup I missed.
sg, can you open an issue to track that unless you have an open PR.
Sure, create two issues tracking the follow-ups: [ ] sched: cleanup clusterEventReg from [] [x] sched: requeue optimization upon Node Add/Update []
I submitted a PR ( ) for NodePorts, not yet tested. I shall test it this week.
After discussion with , due to 1.21 code freeze (today), we decided to postpone merging most of the opening PRs here. Apologize for that - we'd better not rush to avoid regressions. So priority your time on the on-going PRs, we will still review them, just take your time :)
Use to convert the json files to table.
That's a really useful tool thanks so much for writing that!
oh bummer. I would have liked at least serviceaffinity to go in.
Actually, to get rid of the reaction to service changes... all plugins would have to opt-out, so nvm.
Yes, to achieve that, every plugin has to implement EventsToRegister()... BTW: one way is work it around is to add some temporary logic on framework side - skip movingPods upon Service events if ServiceAffinity is not enabled. And remove this temporary logic when all plugins opt-in EventsToRegister() in the next release.
That wouldn't qualify for codefreeze though.
Nvm then :)
We should simply not register for service updates at all if ServiceAffinity is not configured, which is the default.
Probably not a great idea to totally disable the registration, as an out-of-tree plugin may choose to respond to Service events.
We need to find a way for external plugins to register events because we will remove this registration once the plugin is deprecated.
Yes, also should take CRD into consideration. Probably dynamic informer.
I still think we should disabled Service event handlers when the plugin is not registered. If an external plugin is using it, then they could enable the service affinity filter plugin with an empty affinity labels or patch event handlers until we figure out a way for external plugins to register extra events. btw, do you know of any external plugins that actually rely on service events?
That's fair, and I can work on this, adding some comments caveating this is a temporary workaround. I'm not aware so far.
tbh, I would say plugins depending on services is an antipattern, given the existence of pod affinity. Probably some CRDs might make sense for specialized plugins.
we should add metrics to track the efficiency of the solution. The metric would track the number of pods that were not put back into the queue broken down by plugin and even type for example.
SG. I compiled a list to include on-going and to-do items here .
Are you still working on this?
did we have PRs for number 2 and number 3?
I am asking about the two points from the four mentioned in the description of this issue:
Oops, misunderstanding, ignore me.
number 2: mitigated the problem. In short, includes two parts: one part is to remove Service event from default list so that only Pods failed by a plugin (e.g., ServiceAffinity) which explicitly are interested in Service events can be moved; the other part is to implement EventsToRegister() in ServcieAfffinity plugin. But that doesn't prevent the internal processing cycles like going through the unschedulabe pods and verify whether each is failed by that plugin, so it's a mitigation. PR will fix this problem thoroughly. number 3:
ah, yeah, I knew we worked on that in the previous release, lol. Thanks.


What happened: We're seeing legacy-cloud-providers/azure/clients (in our case, the synchronised copy used by latest cluster-autoscaler (1.19)) segfaulting under heavy pressure and ARM throttling: In that specific case the cluster-autoscaler triggered the issue, but it can likely affect other ARM clients using that code, including in-tree kubelet or controller-manager. What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): v1.18.8 Cloud provider or hardware configuration: Azure OS (e.g: ): Ubuntu 20.04.1 LTS Kernel (e.g. ): 5.4.0-1022-azure -Ubuntu SMP Install tools: Network plugin and version (if this is a network-related bug): Others: /kind bug /sig cloud-provider /area provider/azure
/milestone v1.20


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: is created on a cgroup2 host What you expected to happen: Such directory should not be created How to reproduce it (as minimally and precisely as possible): Boot Ubuntu 20.04 with Install moby/moby with containerd/containerd and opencontainers/runc Install kubernetes-sigs/kind . This version uses as the base image. Run with Run Run to drop into the shell of the node The following directories are unexpectedly created. (No process seems to belong to these groups actually) Environment: (See above)
cc
/sig node