Question:
Problem: Preferred pod/node affinity and preferred PodTopologySpread may not be respected because of conflicts with other scoring functions, like node utilization or image locality. Scheduler Scoring functions have weights, by default they are all set to 1 (apart from one): Having equal weights is problematic because of the way we score: each Scoring plugin returns a score [0, MaxScore], which get multiplied by the configured weight of the plugin, and a node score is the sum over all scores. Therefore, even if the user explicitly indicates that they prefer to place pod x on node y, this preference may not be respected and the scheduler may decide to place the pod on node z because it has lower usage and the container that the pod is requesting. Proposed Solution: Scoring functions that allow users to explicitly express preference should have higher weights than generic scoring functions. This includes: inter-pod affinity, node affinity, taint/toleration and pod topology spread. This issue is proposing to change the default behavior of the scheduler, I am not sure how k8s deprecation policy applies here, although I would argue that this change should help the scheduler make decisions closer to users expectations. We will start with pod topology pod spread since as of creating this issue, the feature is still in beta. /sig scheduling
Quote from the : Rule : Deprecated behaviors must function for no less than 1 year after their announced deprecation. I think the rule doesn't apply to this proposal since it is not impacting the correctness of applications or impact administration of the cluster.
As nodepreferavoidpods already exercised to set its weight to 1000 by default, it sounds fair to me to increase the default weight for inter-pod affinity, node affinity, taint/toleration and pod topology spread. BTW: to some extent, we can use e2e tests verifying the to-proposed default weight. Right now we have some e2e tests covering priorities (scoring), but they depend on to ensure cpu/mem are at the same ratio of usage: and hence IMO a bit fragile.
Strongly agree with this change. But, IMO, we may need to reconsider the default impact weights of each scoring plugin. Maybe they should have more weight ladders, rather than simply 1 or 1000. For example, from practical experience, in most cases should not be set to the same weight as , , etc. (Yes, we need to consider this for users, because most users may not notice this change).
IMO a reasonable criterion to guide which Score plugins should have non-default weight is: whether the Score plugin is associated with an API (e.g., PodTopologySpread Score plugin is associated with .
If we give a higher weight only to the API-facing scores, then workloads that don't use these features are not affected, which is a good thing. I think a good weight to give to these scores is one that avoids it to be diluted among the rest of the scores. So, maybe a weight that is half the sum of all non-API-facing scores. Now, if a user uses the API for more than one score, I think we can safely assume that they want equal weights for them.
/cc Daniel, can you comment on whether or not the proposed change conforms with k8s deprecation policy?
I hate to say it, but I think you have to make the new set of weights an option and use the deprecation process to move people over. It's quite difficult to predict what a change in these weights might do to the overall scheduling behavior, no? If it was already possible to specify the settings (via flag or config file), then maybe there's a case to be made that the deprecation process is heavier than needed here. To make that case, you'd need to test and be VERY sure that the new settings are actually an improvement. The change sounds like an improvement, but I'd like to see that quantified / tested somehow. Perhaps even more importantly, I'd like to know the conditions under which it makes things worse, and along which dimensions. can give a second opinion if you don't like this answer :)
This may have an impact on availability (since by default the scheduler will prefer less under utilized nodes) and pod startup (less preference to node that already has the image) IF the pod have explicit preference for being on a specific node or with a specific pod. Those weights are actually configurable via the component config file, but most cloud providers don't allow customers to provide a custom configuration, they just use the default. Regarding the change of behavior, this effectively create two level scoring: first level is by user-specified preferences (affinity), second level is by workload-agnostic ones (e.g., image existence). If users are not specifying preferences, then this change have no impact.
Those are good points, but how many users specify preferences? Should we encourage those users to use non-optional constraints, instead? I can imagine arguing that making the preferences more likely to be obeyed makes it rarer and therefore more surprising and worse when they can't be honored.
I'm not sure if I'm understanding this correctly. Examples of non-optional scores are "topology spreading", "resources utilization" and "image locality". Users (aka Pods) can not opt-in or opt-out from them. Cluster operators (on prem or cloud-providers) can opt-out or change weights, but for a whole cluster. Remember that we are talking about node scoring. The nodes to be scored should have already passed the filters. Let's take an API-facing score (node affinity) and non-API-facing score (image locality) for illustrating the effect of weights: Imagine a pod is using "preferred" node affinity and there are no nodes available that satisfy that affinity. That means that in this case all nodes would get a low (or zero) score for node affinity. In this case the score given by image locality will provide the difference in score between all nodes. Now, let's say some nodes satisfy the node affinity. Those nodes will have the same high score for node affinity. But they will defer in image locality, which will be (in this case) the determining factor to select a node. However, in real (default) clusters we have more than 2 scores. And the problem is that pod preferences end up being diluted with all the other default scores summed up. What the proposal would achieve is that we first honor pod preferences and then we differentiate among the resulting nodes using the default scores.
OK, so I guess the case where this hurts users is: They specified preferences The preferences are actually bad (plot twist) they're diluted so much that they're not having much effect but now the scheduler starts honoring them more often But it's still probabilistic, so we can expect some percentage of the time, the preferences continue to not get honored? Numerically, suppose 50% have preferences, 2% of those got them wrong, 80% of those are currently diluted (i.e., not already broken), and 10% of those are in environments where they'll suddenly get honored 100% after this change (becoming broken). That's .5.02.8*.1 = .08% of users, or 1 out of 1250 users this change could break. That's still maybe a little high for my taste, but maybe you have an alternative estimate?
I think we can tweak how many of the currently diluted cases become honored. In I proposed that we use a weight of half the sum of the non-API-facing weights. We can be less aggressive and do a quarter or just 2 (we currently put weight of 1 for all).
On another note, I would say weights are actually part of the component config API. In both v1alpha1 and the new v1alpha2 (), you can tweak the weights of individual scores. Users might be setting up these weights assuming that the defaults are 1. This is not a problem with the old Policy API, because you have to specify all the scores with their weights. So, if we decide to change the weights, better do so now for v1alpha2.
Daniel, I don't expect that 50% of k8s workloads have preferred pod/node affinities, but I don't have data to suggest an alternative estimate. I guess we could make the change behind a feature flag that gets graduated over multiple releases.
It is permissible for the default to change over versions, in fact that's the way to do it for API types. Since the type is alpha, I guess it's OK to just do this. So, if the config file is v1alpha1, do the old defaults; if it's v1alpha2, use the new ones. How does that sound?
sgtm. But the question of what's a good weight for api-facing scores still remains. By default we only have 4 non-api-facing scores: NodeResourcesBalancedAllocation ImageLocality NodeResourcesLeastAllocated DefaultPodTopologySpread So my two proposals are actually the same number: 2 :sweat:
What about the case where a config file is not provided at all (which is very common across all cloud providers)? I guess we will have to keep the old defaults, and so we didn't actually solve the issue, right? it will also be confusing to users who pass a config file that doesn't change any of the weights.
If there's no config file at all, I think it's fair game, users can easily get the old behavior if needed. Others might disagree with me, but that's what I think. A completely separate thought: are you sure changing the weights is good? What about changing the way they're evaluated? E.g. in ML it's common to use e.g. a softmax function at the last layer to force a NN to make a choice / assign a consistent set of probabilities to each option. I think the way that might look here is instead of using F(N) as F's priority for node N, use exp(F(N)) / SUM (N0-Nn) { exp(F(N)) }. This would have the effect of magnifying minor preferences. Basically, I'm claiming that priority functions should be distributing the same amount of "probability mass" over the N possible nodes. It's not fair for one function to say ".3" for everything while another says ".6" for everything. You maybe need to normalize this before applying the weighting function anyway. But if you do the softmax, you might not even need the weight, as priority functions that give each node the same score will basically be no-ops, letting ones with differing opinions win.
Interesting proposal, but don't you think weights will still be needed because unlike NN, the scheduler didn't go through a training phase to discover which features (in this case priority functions) provide the most important signal?
Let's put an scenario with three simplified scores: selected: a score that prefers a particular set of nodes (maybe a particular zone). : existence of an image. : low utilization Now let's assume they give the following scores: Notice that they all have the same distribution (so they all give good signal). Even when using softmax the sum of scores (with the same weight for every score) will be: Nodes 1, 4 and 7 have the same score and thus the same probability to be chosen. I would argue that the user intention was that node 4 would be selected. It's still called a preference because if would give 0 to all nodes, then the pod can still go to the nodes, and the automatic scores take all the decision. Now, in the default scheduler, we have 4 automatic scores, so the user preference is quite unlikely to get respected. It's not necessarily a matter of good signal from the score.
Yup good points-- I agree a weight is still needed. If all the numbers are 0 or 1 (I had thought that it was a floating point and not a binary signal) then I don't think softmax is going to be different than a linear normalization (?). I guess it's debatable whether one wants a priority function that selects fewer nodes to have its opinion on those nodes be weighted more highly, which is what normalization would do. Anyway sorry if this is a rabbit hole. :) I think this idea would make more sense if the priority functions could express a strength for their preference and not just yes/no.
It's not binary necessarily... I was over simplifying. Node affinity and image locality certainly are, but node utilization is not.
It might be worth some experiments, then... if you have some way of telling whether or not it's actually better? Aldo Culquicondor <:
We have an HA use case that involves spreading pods across nodes and zone with preferred scheduling since we'd rather schedule the pods than not. Before 1.18, we've been using the something like the following: Now with the 1.18 beta, we tried the following: The later yields better scheduling results than the former on a 9 nodes multi-zone cluster with 3 nodes in each node. So while was better it surprisingly didn't yield a perfect spread. So I decided to combine both and ... boom ... we had a perfect pod spread for our deployment with 9 replicas. I'm not sure how these results fit into the discussions so far but I would have expected either option by itself to yield a better spread. Combining the two lets me reinforce my desires to the scheduler. IMO, I would like to see the scheduler give my explicit request more weight.
thanks a lot for sharing. Giving more weight to explicit requests is what we are proposing. However, I'm a little surprised that you need these constraints. The scheduler already spreads by default when using services, replica set or stateful sets. Are these pods not members of any of those? I assume you are not modifying the scheduler configuration? But, of course, there are other scores in play. It makes sense that spreading constraints are more effective than antiaffinity, because there is a lot of antiaffinity with the node that is ultimately selected as well. Adding both specs works too. However, this increases scheduling latency, as we have to calculate more things. The main motivation for this issue was actually to make scoring more effective for the opposite case, which is not done by default: "packing". But your use case is just as valid.
These pods are part of a deployment. I tried without any scheduling preferences and the results are similar to podAntiAffinity preferred scheduling. Here is the baseline deployment yaml used for the tests: I have not modified the scheduler configuration.
Yes, deployments are affected because they create a replica set that owns the pods. But there are always other scores at play that might also be important, which is why a "preferred" spreading is never going to be perfect. This is kind of the point that was trying to make: guarding users from over-fitting and degrading the clusters in some way.
What if we reduce the scope to just PodTopologySpreading, which is beta since 1.18? We could give it a weight of 2 now, before going to GA.
Aldo, one thing to take into account is the new default spread logic which is based on default pod topology spread constraints. For default spread constraints, I think the weight shouldn't increase, so things will get a little complicated with how we treat this case.
Given that default spread constraints is explicitly specified by users in , maybe we can assume they are already aware of adjusting the "Weight" in ? While for regular users, they won't touch , so a default non-1 weight is a no-op for them.
the default constraints (which we plan to introduce to replace the current defaultpodspreading score plugin) are not explicitly specified in PluginConfig.
Maybe I missed something, are we talking about the same default constraints defined here?
yes we are, but isn't the plan to have a default value for DefaultConstraints when not set: two constraints, one for zone and one for hostname, which mimic the current defaultpodspread behaviour and allows us to deprecate it?
I think the only way around that is to introduce a weight for default constraints in the plugin config. And then set it to 50% for the k8s default.
hmm, this will be a confusing configuration parameter to users. Another approach to consider is to bake it into the TopologySpread plugin instantiation without needing to pass it through PluginConfig (similar to what we did for volumebinder in the past: ).
urgh, this will be tricky to do since defaulting will be done via component config, and so we don't have a way to communicate whether the values were defaults or explicitly set by the user. If defaulting is done in then it might be possible, but we shouldn't do that.
Defaulting is going to happen in in the near future
We can have it in the internal type, but not the external one then. This will hide it from the users. But how do we communicate that to conversion logic?!
It doesn't seem there is a way because defaulting happens before conversion...
To make the new parameter semantically clear, we can treat it as a weight override rather than a percentage of the plugin weight, and so it's default would be 1 rather than 50%, internally the plugin computes the ratio as the override weight divided by the plugin weight, which it uses to scale scores computed for pods that don't have topology spread specified in their spec.
Got it now. I thought the default constraints would default to nil if not specified by users. I think we should introduce a field called "multiplexer" and default to 1, so it's friendly to defaulting and conversion.
+1.
What if we start giving meaning to MaxSkew for scoring? If you set a low MaxSkew, you probably want it to have a higher effect than if you set a bigger value. We can discuss the exact formula, but I think adding these semantics make sense. On Sat., Apr. 18, 2020, 5:50 p.m. Wei Huang, <:
That's possible. Current Score function doesn't take into consideration - it's just following a simple "the less number of matching Pods a topology has, the higher score the Nodes on that topology are given" principle.
Note that if we add another parameter to the args, it would only be relevant for scoring, adding to the confusion.
: This score calculation is somewhat complicated, but can be resumed as: Calculate number of matching pods per topology. There is some new feature that takes in consideration the skew Normalize by . The effect is that pods with the lowest count get a score of . Then the rest of the scores go from a certain number to . A score is given to a node with when is . This formula has the effect of reducing differentiation among nodes as the increases. : The formula is very simple, the score is equal to the percentage of free space (averaged for cpu and memory). Each score on their own give the differentiation we want, but they kind of have different scales. As a result tends to be stronger. I'm using a 9 nodes (3 in each zone) cluster as an example. I've set different initial loads, and in all cases I simulate a Deployment of 30 pods (each assumed to take 1% of node resources). Here are the scenarios, where numbers are percentage of use: For equal score weights (current configuration), this is the distribution of new pods: for both nodes and zones Note how a node with only 40% utilization gets only 1 or 0 pods. For double score weights for both nodes and zones Only at 80% utilization we observe bigger skews, which is reasonable. But the spreading is still pretty good at 40% For planned default spreading See , but this might change, depending on feedback here. I think this is enough justification to increase the weight of to 2 in its 2nd beta release.
Sounds reasonable. It's mathematically equivalent to changing the formula to e.g. , right? Have you considered doing that instead of changing the weight? It seems like that'd help everyone except those who specifically already increased , instead of just people who are using the defaults?
It'd make sense to me if the available functions were in the same scale / units so that they'd compose better, but it's not obvious how to do that--what would the units be? One thought is to make the units "std deviations from the mean"--if we knew the output frequency of each function, we could maybe do that. So basically this would measure how unusual a preference is, I guess. (I don't think this is an actionable thought. It really bugs me that the outputs of these functions are combined when they are not really in the same units, though!)
Yes 100 is the maximum score we have defined any plugins to have. I'm not convinced this would work. The variance in my examples for initial resource usage is quite high. So it would still get a lot of importance. But, you have triggered me to start thinking about the linearity of the scores. The formula is good because it reduce the variance as increases. However, that reduction is too aggressive. On the other hand, is completely linear. I'll experiment changing the linearity of either function, but preferring to do so for the beta plugin.
Here go 2 alternatives, with the same situations as We either solution, we wouldn't need to modify the weights Although we are not getting as good spreading as simply doubling the weight. But this is only in the last scenario, where we have usages of 80%. So I would call this rather a win (we should be paying greater attention to resource usage at this point). I think option B is simpler to understand and gives slightly better results (only noticeable in the last case). So this would be my algorithmic preference. However, I'm fine with option A.
Yeah, that's a great idea. You can generalize that and give each function a both a linear weight and a "gamma" (name from image processing): or even go for (but in this case I'm in favor of just making a targeted fix to that one function)
Awesome, Any concerns against option B? Given that it's an older feature.
how would option B impact node ranking in the absence of spreading? would image size scores start to have bigger impact?
I think SIG Scheduling should probably have a discussion. I could go either way, my naive inclination is to just go for the change, but I'm sure it'll break someone. We probably need a way to get back the old behavior, or a suggested weight change that would approximate it (not clear how well that could do though). I also have question, which is how well does this new value compose with other functions? I feel like the proper answer looks like an evaluation criteria + {long list of scenarios} + {list of workloads to schedule} + randomized testing / benchmarking. Basically the multi-dimensional version of the analysis here. That sounds like a lot of work but it would permit actual optimization of these functions.
configurable gamma is an obvious way... :/
I don't think we want another config parameter, we already have the weight concept and it is configurable, so I think we should go with that, just double the spreading score weight by default, and users who don't favor this behavior can configure it back to 1.
Yeah, that sounds expedient. ahg-g <:
In short, yes. Although that score has a mechanism to avoid the "node heating problem" we could have undesirable consequences unless we carefully analyze the impact. But that same reasoning doesn't apply to option A (improving PodTopologySpreading), for the following reasons: The score in 1.18 had pretty weak signal, that was only useful with very small number of pods. We have drastically improved this . So it can only be compared to other scores now. The score can now be tweaked in the PodSpec through maxSkew I would argue that option ~B~ A is better that simply doubling the weight because its signal is more conservative when it matters (nodes have high usage).
I am not clear on the following: you say "we could have undesirable consequences unless we carefully analyze the impact." but then you argue for the option at the end, can you clarify? We had default spreading that was giving similar strong signal, we should compare with that. My concern with modifying the is that we may break someone. Increasing the weight is something that users can revert, while option B can't be reverted.
Sorry, I meant option A, improve PodTopologySpread
One potential advantage to doubling the weight is that topology spread currently offers a signal comparable to default spreading, which we can think of as a baseline for spreading signal since it has been used for a while now. Doubling the weight will allow admins to revert back to the old behavior if they wish to do so. Also, from my perspective, doubling the weight is giving better placement at high utilization ( vs )

{Name: "NodePreferAvoidPods", Weight: 10000}, {Name: "DefaultPodTopologySpread", Weight: 1}, {Name: "TaintToleration", Weight: 1}, <del> {Name: "PodTopologySpread", Weight: 1}, </del> <ins> {Name: "PodTopologySpread", Weight: 2}, </ins> }, "ReservePlugin":   {{Name: "VolumeBinding"}}, "UnreservePlugin": {{Name: "VolumeBinding"}},


Question:
Once we upgrade to go1.14, we can use the new CipherSuites and InsecureCipherSuites functions and CipherSuiteName function to build the supported cipher suite flags. /sig api-machinery
/sig auth
/cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
as the project was upgraded to go-1.14+ is that still something that needs to be done?
yes, these manually maintained maps should be swapped to the stdlib impls:
/assign
/triage accepted Thanks,

"k8s.io/apimachinery/pkg/util/sets" ) <del> // ciphers maps strings into tls package cipher constants in // https://golang.org/pkg/crypto/tls/#pkg-constants // to be replaced by tls.CipherSuites() when the project migrates to go1.14. var ciphers = map[string]uint16{ "TLS_RSA_WITH_3DES_EDE_CBC_SHA":           tls.TLS_RSA_WITH_3DES_EDE_CBC_SHA, "TLS_RSA_WITH_AES_128_CBC_SHA":            tls.TLS_RSA_WITH_AES_128_CBC_SHA, "TLS_RSA_WITH_AES_256_CBC_SHA":            tls.TLS_RSA_WITH_AES_256_CBC_SHA, "TLS_RSA_WITH_AES_128_GCM_SHA256":         tls.TLS_RSA_WITH_AES_128_GCM_SHA256, "TLS_RSA_WITH_AES_256_GCM_SHA384":         tls.TLS_RSA_WITH_AES_256_GCM_SHA384, "TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA":    tls.TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, "TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA":    tls.TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, "TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA":     tls.TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA, "TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA":      tls.TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, "TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA":      tls.TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256":   tls.TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256": tls.TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384":   tls.TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, "TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384": tls.TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, "TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305":    tls.TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305, "TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305":  tls.TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305, "TLS_AES_128_GCM_SHA256":                  tls.TLS_AES_128_GCM_SHA256, "TLS_CHACHA20_POLY1305_SHA256":            tls.TLS_CHACHA20_POLY1305_SHA256, "TLS_AES_256_GCM_SHA384":                  tls.TLS_AES_256_GCM_SHA384, } </del> <ins> var ( // ciphers maps strings into tls package cipher constants in // https://golang.org/pkg/crypto/tls/#pkg-constants ciphers         = map[string]uint16{} insecureCiphers = map[string]uint16{} ) </ins> <del> // to be replaced by tls.InsecureCipherSuites() when the project migrates to go1.14. var insecureCiphers = map[string]uint16{ "TLS_RSA_WITH_RC4_128_SHA":                tls.TLS_RSA_WITH_RC4_128_SHA, "TLS_RSA_WITH_AES_128_CBC_SHA256":         tls.TLS_RSA_WITH_AES_128_CBC_SHA256, "TLS_ECDHE_ECDSA_WITH_RC4_128_SHA":        tls.TLS_ECDHE_ECDSA_WITH_RC4_128_SHA, "TLS_ECDHE_RSA_WITH_RC4_128_SHA":          tls.TLS_ECDHE_RSA_WITH_RC4_128_SHA, "TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256": tls.TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, "TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256":   tls.TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, </del> <ins> func init() { for _, suite := range tls.CipherSuites() { ciphers[suite.Name] = suite.ID } // keep legacy names for backward compatibility ciphers["TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305"] = tls.TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256 ciphers["TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305"] = tls.TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256 for _, suite := range tls.InsecureCipherSuites() { insecureCiphers[suite.Name] = suite.ID } </ins> } // InsecureTLSCiphers returns the cipher suites implemented by crypto/tls which have


Question:
Once we upgrade to go1.14, we can use the new CipherSuites and InsecureCipherSuites functions and CipherSuiteName function to build the supported cipher suite flags. /sig api-machinery
/sig auth
/cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
as the project was upgraded to go-1.14+ is that still something that needs to be done?
yes, these manually maintained maps should be swapped to the stdlib impls:
/assign
/triage accepted Thanks,

<del> //go:build go1.14 // +build go1.14 /* Copyright 2020 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package flag import ( "crypto/tls" ) func init() { // support official IANA names as well on go1.14 ciphers["TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256"] = tls.TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256 ciphers["TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256"] = tls.TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256 } </del>


Question:
Once we upgrade to go1.14, we can use the new CipherSuites and InsecureCipherSuites functions and CipherSuiteName function to build the supported cipher suite flags. /sig api-machinery
/sig auth
/cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
as the project was upgraded to go-1.14+ is that still something that needs to be done?
yes, these manually maintained maps should be swapped to the stdlib impls:
/assign
/triage accepted Thanks,

import ( "crypto/tls" <del> "fmt" "go/importer" </del> "reflect" <del> "strings" </del> "testing" )


Question:
Once we upgrade to go1.14, we can use the new CipherSuites and InsecureCipherSuites functions and CipherSuiteName function to build the supported cipher suite flags. /sig api-machinery
/sig auth
/cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
as the project was upgraded to go-1.14+ is that still something that needs to be done?
yes, these manually maintained maps should be swapped to the stdlib impls:
/assign
/triage accepted Thanks,

expected:       nil, expected_error: true, }, <ins> { // All existing cipher suites flag: []string{ "TLS_RSA_WITH_3DES_EDE_CBC_SHA", "TLS_RSA_WITH_AES_128_CBC_SHA", "TLS_RSA_WITH_AES_256_CBC_SHA", "TLS_RSA_WITH_AES_128_GCM_SHA256", "TLS_RSA_WITH_AES_256_GCM_SHA384", "TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA", "TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA", "TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA", "TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA", "TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA", "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256", "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256", "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384", "TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384", "TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305", "TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305", "TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256", "TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256", "TLS_AES_128_GCM_SHA256", "TLS_CHACHA20_POLY1305_SHA256", "TLS_AES_256_GCM_SHA384", "TLS_RSA_WITH_RC4_128_SHA", "TLS_RSA_WITH_AES_128_CBC_SHA256", "TLS_ECDHE_ECDSA_WITH_RC4_128_SHA", "TLS_ECDHE_RSA_WITH_RC4_128_SHA", "TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256", "TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256", }, expected: []uint16{ tls.TLS_RSA_WITH_3DES_EDE_CBC_SHA, tls.TLS_RSA_WITH_AES_128_CBC_SHA, tls.TLS_RSA_WITH_AES_256_CBC_SHA, tls.TLS_RSA_WITH_AES_128_GCM_SHA256, tls.TLS_RSA_WITH_AES_256_GCM_SHA384, tls.TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, tls.TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, tls.TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA, tls.TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, tls.TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, tls.TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, tls.TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, tls.TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, tls.TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, tls.TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305, tls.TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305, tls.TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256, tls.TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256, tls.TLS_AES_128_GCM_SHA256, tls.TLS_CHACHA20_POLY1305_SHA256, tls.TLS_AES_256_GCM_SHA384, tls.TLS_RSA_WITH_RC4_128_SHA, tls.TLS_RSA_WITH_AES_128_CBC_SHA256, tls.TLS_ECDHE_ECDSA_WITH_RC4_128_SHA, tls.TLS_ECDHE_RSA_WITH_RC4_128_SHA, tls.TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, tls.TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, }, }, </ins> } for i, test := range tests { uIntFlags, err := TLSCipherSuites(test.flag) <del> if reflect.DeepEqual(uIntFlags, test.expected) == false { </del> <ins> if !reflect.DeepEqual(uIntFlags, test.expected) { </ins> t.Errorf("%d: expected %+v, got %+v", i, test.expected, uIntFlags) } if test.expected_error && err == nil {


Question:
Once we upgrade to go1.14, we can use the new CipherSuites and InsecureCipherSuites functions and CipherSuiteName function to build the supported cipher suite flags. /sig api-machinery
/sig auth
/cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
as the project was upgraded to go-1.14+ is that still something that needs to be done?
yes, these manually maintained maps should be swapped to the stdlib impls:
/assign
/triage accepted Thanks,

} } } <del> func TestConstantMaps(t *testing.T) { pkg, err := importer.Default().Import("crypto/tls") if err != nil { fmt.Printf("error: %sn", err.Error()) return } discoveredVersions := map[string]bool{} discoveredCiphers := map[string]bool{} for _, declName := range pkg.Scope().Names() { if strings.HasPrefix(declName, "VersionTLS") { discoveredVersions[declName] = true } if strings.HasPrefix(declName, "TLS_") && !strings.HasPrefix(declName, "TLS_FALLBACK_") { discoveredCiphers[declName] = true } } acceptedCiphers := allCiphers() for k := range discoveredCiphers { if _, ok := acceptedCiphers[k]; !ok { t.Errorf("discovered cipher tls.%s not in ciphers map", k) } } for k := range acceptedCiphers { if _, ok := discoveredCiphers[k]; !ok { t.Errorf("ciphers map has %s not in tls package", k) } } for k := range discoveredVersions { if _, ok := versions[k]; !ok { t.Errorf("discovered version tls.%s not in version map", k) } } for k := range versions { if _, ok := discoveredVersions[k]; !ok { t.Errorf("versions map has %s not in tls package", k) } } } </del>


Question:
What happened: https://k8s- These tests failed consistently: Anything else we need to know?: Volume snapshot tests need to be reworked to use / helper functions that support block volume mode. In addition, csi-driver-host-path fails when restoring block volume snapshots. /assign /sig storage
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : What happened: https://k8s- Anything else we need to know?: Volume snapshot tests need to be reworked to use / helper functions that support block volume mode. In addition, csi-driver-host-path fails when restoring block volume snapshots. /sig stroage Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
hostpath driver fix:
E2e test fix: (needs hostpath driver release first).

dc := l.config.Framework.DynamicClient vsc := sDriver.GetSnapshotClass(l.config) <del> dataSource, cleanupFunc := prepareSnapshotDataSourceForProvisioning(l.config.ClientNodeSelection, l.cs, dc, l.pvc, l.sc, vsc) </del> <ins> testConfig := convertTestConfig(l.config) expectedContent := fmt.Sprintf("Hello from namespace %s", f.Namespace.Name) dataSource, cleanupFunc := prepareSnapshotDataSourceForProvisioning(f, testConfig, l.cs, dc, l.pvc, l.sc, vsc, pattern.VolMode, expectedContent) </ins> defer cleanupFunc() l.pvc.Spec.DataSource = dataSource l.testCase.PvCheck = func(claim *v1.PersistentVolumeClaim) { ginkgo.By("checking whether the created volume has the pre-populated data") <del> command := fmt.Sprintf("grep '%s' /mnt/test/initialData", claim.Namespace) RunInPodWithVolume(l.cs, claim.Namespace, claim.Name, "pvc-snapshot-tester", command, l.config.ClientNodeSelection) </del> <ins> tests := []volume.Test{ { Volume:          *createVolumeSource(claim.Name, false /* readOnly */), Mode:            pattern.VolMode, File:            "index.html", ExpectedContent: expectedContent, }, } volume.TestVolumeClient(f, testConfig, nil, "", tests) </ins> } l.testCase.TestDynamicProvisioning() })


Question:
What happened: https://k8s- These tests failed consistently: Anything else we need to know?: Volume snapshot tests need to be reworked to use / helper functions that support block volume mode. In addition, csi-driver-host-path fails when restoring block volume snapshots. /assign /sig storage
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : What happened: https://k8s- Anything else we need to know?: Volume snapshot tests need to be reworked to use / helper functions that support block volume mode. In addition, csi-driver-host-path fails when restoring block volume snapshots. /sig stroage Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
hostpath driver fix:
E2e test fix: (needs hostpath driver release first).

} func prepareSnapshotDataSourceForProvisioning( <del> node e2epod.NodeSelection, </del> <ins> f *framework.Framework, config volume.TestConfig, </ins> client clientset.Interface, dynamicClient dynamic.Interface, initClaim *v1.PersistentVolumeClaim, class *storagev1.StorageClass, snapshotClass *unstructured.Unstructured, <ins> mode v1.PersistentVolumeMode, injectContent string, </ins> ) (*v1.TypedLocalObjectReference, func()) { var err error if class != nil {


Question:
What happened: https://k8s- These tests failed consistently: Anything else we need to know?: Volume snapshot tests need to be reworked to use / helper functions that support block volume mode. In addition, csi-driver-host-path fails when restoring block volume snapshots. /assign /sig storage
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : What happened: https://k8s- Anything else we need to know?: Volume snapshot tests need to be reworked to use / helper functions that support block volume mode. In addition, csi-driver-host-path fails when restoring block volume snapshots. /sig stroage Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
hostpath driver fix:
E2e test fix: (needs hostpath driver release first).

framework.ExpectNoError(err) // write namespace to the /mnt/test (= the volume). <del> ginkgo.By("[Initialize dataSource]write data to volume") command := fmt.Sprintf("echo '%s' > /mnt/test/initialData", updatedClaim.GetNamespace()) RunInPodWithVolume(client, updatedClaim.Namespace, updatedClaim.Name, "pvc-snapshot-writer", command, node) err = e2epv.WaitForPersistentVolumeClaimPhase(v1.ClaimBound, client, updatedClaim.Namespace, updatedClaim.Name, framework.Poll, framework.ClaimProvisionTimeout) framework.ExpectNoError(err) ginkgo.By("[Initialize dataSource]checking the initClaim") // Get new copy of the initClaim _, err = client.CoreV1().PersistentVolumeClaims(updatedClaim.Namespace).Get(context.TODO(), updatedClaim.Name, metav1.GetOptions{}) framework.ExpectNoError(err) </del> <ins> tests := []volume.Test{ { Volume:          *createVolumeSource(updatedClaim.Name, false /* readOnly */), Mode:            mode, File:            "index.html", ExpectedContent: injectContent, }, } volume.InjectContent(f, config, nil, "", tests) </ins> ginkgo.By("[Initialize dataSource]creating a SnapshotClass") snapshotClass, err = dynamicClient.Resource(snapshotClassGVR).Create(snapshotClass, metav1.CreateOptions{}) <ins> framework.ExpectNoError(err) </ins> ginkgo.By("[Initialize dataSource]creating a snapshot") snapshot := getSnapshot(updatedClaim.Name, updatedClaim.Namespace, snapshotClass.GetName())


Question:
What happened: https://k8s- These tests failed consistently: Anything else we need to know?: Volume snapshot tests need to be reworked to use / helper functions that support block volume mode. In addition, csi-driver-host-path fails when restoring block volume snapshots. /assign /sig storage
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : What happened: https://k8s- Anything else we need to know?: Volume snapshot tests need to be reworked to use / helper functions that support block volume mode. In addition, csi-driver-host-path fails when restoring block volume snapshots. /sig stroage Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
hostpath driver fix:
E2e test fix: (needs hostpath driver release first).

name: csi-data-dir - name: hostpath <del> image: quay.io/k8scsi/hostpathplugin:v1.4.0-rc1 </del> <ins> image: quay.io/k8scsi/hostpathplugin:v1.4.0-rc2 </ins> args: - "--drivername=hostpath.csi.k8s.io" - "--v=5"


Question:
<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: pull-kubernetes-e2e-gce Which test(s) are flaking: Testgrid link: Reason for failure: Anything else we need to know: Triage board shows flakes here spiked dramatically on 3/4 /sig node /priority critical-urgent /milestone v1.18
are you able to assist if this is related to recent topology work?
sure, I'll take a look
is this the test you just in ?
As mentioned above, this is a new test that was just , and the test sometimes fails. I don't see that any of the recent topology manager changes would cause these failures. I'll be happy to help out and dig more if or others think otherwise.
This is me, the bugs are untriaged containerd / Kubelet failures with low occurrence rates that this test now exposes. Investigating.
still flaking in non-dockershim environments: pod pod-submit-status-0-14 on node kind-worker2 container unexpected exit code 128: start=1970-01-01 00:00:00 +0000 UTC end=2020-03-06 16:16:54 +0000 UTC reason=StartError message=failed to create containerd task: OCI runtime create failed: starting container process caused " container init caused " mounting "/var/lib/kubelet/pods/-819e-4840-8369-" to rootfs "" at "" caused "stat /var/lib/kubelet/pods/-819e-4840-8369- no such file or directory""": unknown `
Different reason this time. Will continue to enumerate failure modes.

ginkgo.It("should never report success for a pending container", func() { ginkgo.By("creating pods that should always exit 1 and terminating the pod after a random delay") <del> var reBug88766 = regexp.MustCompile(`ContainerCannotRun.*rootfs_linux.go.*kubernetes.io~secret.*no such file or directory`) </del> <ins> var reBug88766 = regexp.MustCompile(`rootfs_linux.*kubernetes.io~secret.*no such file or directory`) </ins> var ( lock sync.Mutex


Question:
<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: pull-kubernetes-e2e-gce Which test(s) are flaking: Testgrid link: Reason for failure: Anything else we need to know: Triage board shows flakes here spiked dramatically on 3/4 /sig node /priority critical-urgent /milestone v1.18
are you able to assist if this is related to recent topology work?
sure, I'll take a look
is this the test you just in ?
As mentioned above, this is a new test that was just , and the test sometimes fails. I don't see that any of the recent topology manager changes would cause these failures. I'll be happy to help out and dig more if or others think otherwise.
This is me, the bugs are untriaged containerd / Kubelet failures with low occurrence rates that this test now exposes. Investigating.
still flaking in non-dockershim environments: pod pod-submit-status-0-14 on node kind-worker2 container unexpected exit code 128: start=1970-01-01 00:00:00 +0000 UTC end=2020-03-06 16:16:54 +0000 UTC reason=StartError message=failed to create containerd task: OCI runtime create failed: starting container process caused " container init caused " mounting "/var/lib/kubelet/pods/-819e-4840-8369-" to rootfs "" at "" caused "stat /var/lib/kubelet/pods/-819e-4840-8369- no such file or directory""": unknown `
Different reason this time. Will continue to enumerate failure modes.

switch { case t.ExitCode == 1: // expected <del> case t.ExitCode == 128 && reBug88766.MatchString(t.Message): </del> <ins> case t.ExitCode == 128 && t.Reason == "ContainerCannotRun" && reBug88766.MatchString(t.Message): </ins> // pod volume teardown races with container start in CRI, which reports a failure framework.Logf("pod %s on node %s failed with the symptoms of https://github.com/kubernetes/kubernetes/issues/88766") default:


Question:
<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: pull-kubernetes-e2e-gce Which test(s) are flaking: Testgrid link: Reason for failure: Anything else we need to know: Triage board shows flakes here spiked dramatically on 3/4 /sig node /priority critical-urgent /milestone v1.18
are you able to assist if this is related to recent topology work?
sure, I'll take a look
is this the test you just in ?
As mentioned above, this is a new test that was just , and the test sometimes fails. I don't see that any of the recent topology manager changes would cause these failures. I'll be happy to help out and dig more if or others think otherwise.
This is me, the bugs are untriaged containerd / Kubelet failures with low occurrence rates that this test now exposes. Investigating.
still flaking in non-dockershim environments: pod pod-submit-status-0-14 on node kind-worker2 container unexpected exit code 128: start=1970-01-01 00:00:00 +0000 UTC end=2020-03-06 16:16:54 +0000 UTC reason=StartError message=failed to create containerd task: OCI runtime create failed: starting container process caused " container init caused " mounting "/var/lib/kubelet/pods/-819e-4840-8369-" to rootfs "" at "" caused "stat /var/lib/kubelet/pods/-819e-4840-8369- no such file or directory""": unknown `
Different reason this time. Will continue to enumerate failure modes.

switch { case t.ExitCode == 1: // expected <del> case t.ExitCode == 128 && t.Reason == "ContainerCannotRun" && reBug88766.MatchString(t.Message): </del> <ins> case t.ExitCode == 128 && (t.Reason == "StartError" || t.Reason == "ContainerCannotRun") && reBug88766.MatchString(t.Message): </ins> // pod volume teardown races with container start in CRI, which reports a failure framework.Logf("pod %s on node %s failed with the symptoms of https://github.com/kubernetes/kubernetes/issues/88766") default:


Question:
What happened: A SystemOOM event was not reported even when the memory limit hit was the root cgroup memory limit. What you expected to happen: A SystemOOM event should have been reported. How to reproduce it (as minimally and precisely as possible): Do not set or , and set to disable memory eviction on the kubelet. Run a pod without memory limits which slowly consumes memory, such as with from Use arguments -mem-alloc-size", "100Mi", "-mem-alloc-sleep", "10s". This should trigger an OOM on the root cgroup, but no event will be generated. Anything else we need to know?: cAdvisor populates ContainerName and VictimContainerName from matching the regexp: instead in the kubelet. cc
/sig node /priority important-soon

defer runtime.HandleCrash() for event := range outStream { <del> if event.ContainerName == recordEventContainerName { </del> <ins> if event.VictimContainerName == recordEventContainerName { </ins> klog.V(1).Infof("Got sys oom event: %v", event) eventMsg := "System OOM encountered" if event.ProcessName != "" && event.Pid != 0 {


Question:
What happened: A SystemOOM event was not reported even when the memory limit hit was the root cgroup memory limit. What you expected to happen: A SystemOOM event should have been reported. How to reproduce it (as minimally and precisely as possible): Do not set or , and set to disable memory eviction on the kubelet. Run a pod without memory limits which slowly consumes memory, such as with from Use arguments -mem-alloc-size", "100Mi", "-mem-alloc-sleep", "10s". This should trigger an OOM on the root cgroup, but no event will be generated. Anything else we need to know?: cAdvisor populates ContainerName and VictimContainerName from matching the regexp: instead in the kubelet. cc
/sig node /priority important-soon

Pid:                 1000, ProcessName:         "fakeProcess", TimeOfDeath:         time.Now(), <del> ContainerName:       recordEventContainerName, VictimContainerName: "some-container", </del> <ins> ContainerName:       recordEventContainerName + "some-container", VictimContainerName: recordEventContainerName, </ins> }, } numExpectedOomEvents := len(oomInstancesToStream)


Question:
What happened: A SystemOOM event was not reported even when the memory limit hit was the root cgroup memory limit. What you expected to happen: A SystemOOM event should have been reported. How to reproduce it (as minimally and precisely as possible): Do not set or , and set to disable memory eviction on the kubelet. Run a pod without memory limits which slowly consumes memory, such as with from Use arguments -mem-alloc-size", "100Mi", "-mem-alloc-sleep", "10s". This should trigger an OOM on the root cgroup, but no event will be generated. Anything else we need to know?: cAdvisor populates ContainerName and VictimContainerName from matching the regexp: instead in the kubelet. cc
/sig node /priority important-soon

Pid:                 1000, ProcessName:         "fakeProcess", TimeOfDeath:         time.Now(), <del> ContainerName:       recordEventContainerName, VictimContainerName: "some-container", </del> <ins> ContainerName:       recordEventContainerName + "some-container", VictimContainerName: recordEventContainerName, </ins> }, { Pid:                 1000, ProcessName:         "fakeProcess", TimeOfDeath:         time.Now(), <del> ContainerName:       "/dont-record-oom-event", VictimContainerName: "some-container", </del> <ins> ContainerName:       recordEventContainerName + "kubepods/some-container", VictimContainerName: recordEventContainerName + "kubepods", </ins> }, } numExpectedOomEvents := len(oomInstancesToStream) - numOomEventsWithIncorrectContainerName


Question:
What happened: A SystemOOM event was not reported even when the memory limit hit was the root cgroup memory limit. What you expected to happen: A SystemOOM event should have been reported. How to reproduce it (as minimally and precisely as possible): Do not set or , and set to disable memory eviction on the kubelet. Run a pod without memory limits which slowly consumes memory, such as with from Use arguments -mem-alloc-size", "100Mi", "-mem-alloc-sleep", "10s". This should trigger an OOM on the root cgroup, but no event will be generated. Anything else we need to know?: cAdvisor populates ContainerName and VictimContainerName from matching the regexp: instead in the kubelet. cc
/sig node /priority important-soon

Pid:                 eventPid, ProcessName:         processName, TimeOfDeath:         time.Now(), <del> ContainerName:       recordEventContainerName, VictimContainerName: "some-container", </del> <ins> ContainerName:       recordEventContainerName + "some-container", VictimContainerName: recordEventContainerName, </ins> }, } numExpectedOomEvents := len(oomInstancesToStream)


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: k8s 1.17.2 config kubelet with those parameters cgroupDriver: cgroupfs cgroupRoot: /docker cgroupsPerQOS: false kubelet log always prints those E0303 17:25:03. ] Failed to get system container stats for "/docker": failed to get cgroup stats for "/docker": failed to get container info for "/docker": unknown container "/docker" E0303 17:25:03. ] eviction manager: failed to construct signal: "allocatableMemory.available" error: system container "pods" not found in metrics What you expected to happen: kubelet does not print those error information How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig node

} var cgroupRoots []string <del> cgroupRoots = append(cgroupRoots, cm.NodeAllocatableRoot(s.CgroupRoot, s.CgroupDriver)) </del> <ins> nodeAllocatableRoot := cm.NodeAllocatableRoot(s.CgroupRoot, s.CgroupsPerQOS, s.CgroupDriver) cgroupRoots = append(cgroupRoots, nodeAllocatableRoot) </ins> kubeletCgroup, err := cm.GetKubeletContainer(s.KubeletCgroups) if err != nil { klog.Warningf("failed to get the kubelet's cgroup: %v.  Kubelet system container metrics may be missing.", err)


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: k8s 1.17.2 config kubelet with those parameters cgroupDriver: cgroupfs cgroupRoot: /docker cgroupsPerQOS: false kubelet log always prints those E0303 17:25:03. ] Failed to get system container stats for "/docker": failed to get cgroup stats for "/docker": failed to get container info for "/docker": unknown container "/docker" E0303 17:25:03. ] eviction manager: failed to construct signal: "allocatableMemory.available" error: system container "pods" not found in metrics What you expected to happen: kubelet does not print those error information How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig node

} // NodeAllocatableRoot returns the literal cgroup path for the node allocatable cgroup <del> func NodeAllocatableRoot(cgroupRoot, cgroupDriver string) string { root := ParseCgroupfsToCgroupName(cgroupRoot) nodeAllocatableRoot := NewCgroupName(root, defaultNodeAllocatableCgroupName) </del> <ins> func NodeAllocatableRoot(cgroupRoot string, cgroupsPerQOS bool, cgroupDriver string) string { nodeAllocatableRoot := ParseCgroupfsToCgroupName(cgroupRoot) if cgroupsPerQOS { nodeAllocatableRoot = NewCgroupName(nodeAllocatableRoot, defaultNodeAllocatableCgroupName) } </ins> if libcontainerCgroupManagerType(cgroupDriver) == libcontainerSystemd { return nodeAllocatableRoot.ToSystemd() }


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: k8s 1.17.2 config kubelet with those parameters cgroupDriver: cgroupfs cgroupRoot: /docker cgroupsPerQOS: false kubelet log always prints those E0303 17:25:03. ] Failed to get system container stats for "/docker": failed to get cgroup stats for "/docker": failed to get container info for "/docker": unknown container "/docker" E0303 17:25:03. ] eviction manager: failed to construct signal: "allocatableMemory.available" error: system container "pods" not found in metrics What you expected to happen: kubelet does not print those error information How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig node

} // NodeAllocatableRoot returns the literal cgroup path for the node allocatable cgroup <del> func NodeAllocatableRoot(cgroupRoot, cgroupDriver string) string { </del> <ins> func NodeAllocatableRoot(cgroupRoot string, cgroupsPerQOS bool, cgroupDriver string) string { </ins> return "" }


Question:
Which jobs are flaking: pull-kubernetes-integration Which test(s) are flaking: TestNominatedNodeCleanUp Testgrid link: Reason for failure: /sig scheduling
/assign
any progress on this one?
I was working on it and tried to find the root cause. I will give my solution within this week
Another one
Still happening cc
/assign
/close dupe
Closing this issue. <detailsIn response to : /close dupe Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question:
Which jobs are flaking: pull-kubernetes-integration Which test(s) are flaking: TestNominatedNodeCleanUp Testgrid link: Reason for failure: /sig scheduling
/assign
any progress on this one?
I was working on it and tried to find the root cause. I will give my solution within this week
Another one
Still happening cc
/assign
/close dupe
Closing this issue. <detailsIn response to : /close dupe Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

import ( "context" <ins> "encoding/json" </ins> "fmt" "io/ioutil" "math/rand"


Question:
Which jobs are flaking: pull-kubernetes-integration Which test(s) are flaking: TestNominatedNodeCleanUp Testgrid link: Reason for failure: /sig scheduling
/assign
any progress on this one?
I was working on it and tried to find the root cause. I will give my solution within this week
Another one
Still happening cc
/assign
/close dupe
Closing this issue. <detailsIn response to : /close dupe Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

v1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/runtime" <ins> "k8s.io/apimachinery/pkg/types" "k8s.io/apimachinery/pkg/util/strategicpatch" </ins> "k8s.io/apimachinery/pkg/util/wait" "k8s.io/client-go/informers" coreinformers "k8s.io/client-go/informers/core/v1"


Question:
Which jobs are flaking: pull-kubernetes-integration Which test(s) are flaking: TestNominatedNodeCleanUp Testgrid link: Reason for failure: /sig scheduling
/assign
any progress on this one?
I was working on it and tried to find the root cause. I will give my solution within this week
Another one
Still happening cc
/assign
/close dupe
Closing this issue. <detailsIn response to : /close dupe Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

func (p *podConditionUpdaterImpl) update(pod *v1.Pod, condition *v1.PodCondition) error { klog.V(3).Infof("Updating pod condition for %s/%s to (%s==%s, Reason=%s)", pod.Namespace, pod.Name, condition.Type, condition.Status, condition.Reason) <del> if podutil.UpdatePodCondition(&pod.Status, condition) { _, err := p.Client.CoreV1().Pods(pod.Namespace).UpdateStatus(context.TODO(), pod, metav1.UpdateOptions{}) </del> <ins> oldData, err := json.Marshal(pod) if err != nil { </ins> return err } <del> return nil </del> <ins> if !podutil.UpdatePodCondition(&pod.Status, condition) { return nil } newData, err := json.Marshal(pod) if err != nil { return err } patchBytes, err := strategicpatch.CreateTwoWayMergePatch(oldData, newData, &v1.Pod{}) if err != nil { return fmt.Errorf("failed to create merge patch for pod %q/%q: %v", pod.Namespace, pod.Name, err) } _, err = p.Client.CoreV1().Pods(pod.Namespace).Patch(context.TODO(), pod.Name, types.StrategicMergePatchType, patchBytes, metav1.PatchOptions{}, "status") return err </ins> } type podPreemptorImpl struct {


Question:
Which jobs are flaking: pull-kubernetes-integration Which test(s) are flaking: TestNominatedNodeCleanUp Testgrid link: Reason for failure: /sig scheduling
/assign
any progress on this one?
I was working on it and tried to find the root cause. I will give my solution within this week
Another one
Still happening cc
/assign
/close dupe
Closing this issue. <detailsIn response to : /close dupe Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

} func (p *podPreemptorImpl) setNominatedNodeName(pod *v1.Pod, nominatedNodeName string) error { <ins> klog.V(3).Infof("Setting nominated node name for %s/%s to "%s"", pod.Namespace, pod.Name, nominatedNodeName) if pod.Status.NominatedNodeName == nominatedNodeName { return nil } </ins> podCopy := pod.DeepCopy() <ins> oldData, err := json.Marshal(podCopy) if err != nil { return err } </ins> podCopy.Status.NominatedNodeName = nominatedNodeName <del> _, err := p.Client.CoreV1().Pods(pod.Namespace).UpdateStatus(context.TODO(), podCopy, metav1.UpdateOptions{}) </del> <ins> newData, err := json.Marshal(podCopy) if err != nil { return err } patchBytes, err := strategicpatch.CreateTwoWayMergePatch(oldData, newData, &v1.Pod{}) if err != nil { return fmt.Errorf("failed to create merge patch for pod %q/%q: %v", pod.Namespace, pod.Name, err) } _, err = p.Client.CoreV1().Pods(pod.Namespace).Patch(context.TODO(), pod.Name, types.StrategicMergePatchType, patchBytes, metav1.PatchOptions{}, "status") </ins> return err }


Question:
Which jobs are flaking: pull-kubernetes-integration Which test(s) are flaking: TestNominatedNodeCleanUp Testgrid link: Reason for failure: /sig scheduling
/assign
any progress on this one?
I was working on it and tried to find the root cause. I will give my solution within this week
Another one
Still happening cc
/assign
/close dupe
Closing this issue. <detailsIn response to : /close dupe Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

"os" "path" "reflect" <ins> "regexp" </ins> "sort" "strings" "sync"


Question:
Which jobs are flaking: pull-kubernetes-integration Which test(s) are flaking: TestNominatedNodeCleanUp Testgrid link: Reason for failure: /sig scheduling
/assign
any progress on this one?
I was working on it and tried to find the root cause. I will give my solution within this week
Another one
Still happening cc
/assign
/close dupe
Closing this issue. <detailsIn response to : /close dupe Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

} } } <ins> func TestSetNominatedNodeName(t *testing.T) { tests := []struct { name                     string currentNominatedNodeName string newNominatedNodeName     string expectedPatchRequests    int expectedPatchData        string }{ { name:                     "Should make patch request to set node name", currentNominatedNodeName: "", newNominatedNodeName:     "node1", expectedPatchRequests:    1, expectedPatchData:        `{"status":{"nominatedNodeName":"node1"}}`, }, { name:                     "Should make patch request to clear node name", currentNominatedNodeName: "node1", newNominatedNodeName:     "", expectedPatchRequests:    1, expectedPatchData:        `{"status":{"nominatedNodeName":null}}`, }, { name:                     "Should not make patch request if nominated node is already set to the specified value", currentNominatedNodeName: "node1", newNominatedNodeName:     "node1", expectedPatchRequests:    0, }, { name:                     "Should not make patch request if nominated node is already cleared", currentNominatedNodeName: "", newNominatedNodeName:     "", expectedPatchRequests:    0, }, } for _, test := range tests { t.Run(test.name, func(t *testing.T) { actualPatchRequests := 0 var actualPatchData string cs := &clientsetfake.Clientset{} cs.AddReactor("patch", "pods", func(action clienttesting.Action) (bool, runtime.Object, error) { actualPatchRequests++ patch := action.(clienttesting.PatchAction) actualPatchData = string(patch.GetPatch()) // For this test, we don't care about the result of the patched pod, just that we got the expected // patch request, so just returning &v1.Pod{} here is OK because scheduler doesn't use the response. return true, &v1.Pod{}, nil }) pod := &v1.Pod{ ObjectMeta: metav1.ObjectMeta{Name: "foo"}, Status:     v1.PodStatus{NominatedNodeName: test.currentNominatedNodeName}, } preemptor := &podPreemptorImpl{Client: cs} if err := preemptor.setNominatedNodeName(pod, test.newNominatedNodeName); err != nil { t.Fatalf("Error calling setNominatedNodeName: %v", err) } if actualPatchRequests != test.expectedPatchRequests { t.Fatalf("Actual patch requests (%d) dos not equal expected patch requests (%d)", actualPatchRequests, test.expectedPatchRequests) } if test.expectedPatchRequests > 0 && actualPatchData != test.expectedPatchData { t.Fatalf("Patch data mismatch: Actual was %v, but expected %v", actualPatchData, test.expectedPatchData) } }) } } func TestUpdatePodCondition(t *testing.T) { tests := []struct { name                     string currentPodConditions     []v1.PodCondition newPodCondition          *v1.PodCondition expectedPatchRequests    int expectedPatchDataPattern string }{ { name:                 "Should make patch request to add pod condition when there are none currently", currentPodConditions: []v1.PodCondition{}, newPodCondition: &v1.PodCondition{ Type:               "newType", Status:             "newStatus", LastProbeTime:      metav1.NewTime(time.Date(2020, 5, 13, 1, 1, 1, 1, time.UTC)), LastTransitionTime: metav1.NewTime(time.Date(2020, 5, 12, 1, 1, 1, 1, time.UTC)), Reason:             "newReason", Message:            "newMessage", }, expectedPatchRequests:    1, expectedPatchDataPattern: `{"status":{"conditions":[{"lastProbeTime":"2020-05-13T01:01:01Z","lastTransitionTime":".*","message":"newMessage","reason":"newReason","status":"newStatus","type":"newType"}]}}`, }, { name: "Should make patch request to add a new pod condition when there is already one with another type", currentPodConditions: []v1.PodCondition{ { Type:               "someOtherType", Status:             "someOtherTypeStatus", LastProbeTime:      metav1.NewTime(time.Date(2020, 5, 11, 0, 0, 0, 0, time.UTC)), LastTransitionTime: metav1.NewTime(time.Date(2020, 5, 10, 0, 0, 0, 0, time.UTC)), Reason:             "someOtherTypeReason", Message:            "someOtherTypeMessage", }, }, newPodCondition: &v1.PodCondition{ Type:               "newType", Status:             "newStatus", LastProbeTime:      metav1.NewTime(time.Date(2020, 5, 13, 1, 1, 1, 1, time.UTC)), LastTransitionTime: metav1.NewTime(time.Date(2020, 5, 12, 1, 1, 1, 1, time.UTC)), Reason:             "newReason", Message:            "newMessage", }, expectedPatchRequests:    1, expectedPatchDataPattern: `{"status":{"$setElementOrder/conditions":[{"type":"someOtherType"},{"type":"newType"}],"conditions":[{"lastProbeTime":"2020-05-13T01:01:01Z","lastTransitionTime":".*","message":"newMessage","reason":"newReason","status":"newStatus","type":"newType"}]}}`, }, { name: "Should make patch request to update an existing pod condition", currentPodConditions: []v1.PodCondition{ { Type:               "currentType", Status:             "currentStatus", LastProbeTime:      metav1.NewTime(time.Date(2020, 5, 13, 0, 0, 0, 0, time.UTC)), LastTransitionTime: metav1.NewTime(time.Date(2020, 5, 12, 0, 0, 0, 0, time.UTC)), Reason:             "currentReason", Message:            "currentMessage", }, }, newPodCondition: &v1.PodCondition{ Type:               "currentType", Status:             "newStatus", LastProbeTime:      metav1.NewTime(time.Date(2020, 5, 13, 1, 1, 1, 1, time.UTC)), LastTransitionTime: metav1.NewTime(time.Date(2020, 5, 12, 1, 1, 1, 1, time.UTC)), Reason:             "newReason", Message:            "newMessage", }, expectedPatchRequests:    1, expectedPatchDataPattern: `{"status":{"$setElementOrder/conditions":[{"type":"currentType"}],"conditions":[{"lastProbeTime":"2020-05-13T01:01:01Z","lastTransitionTime":".*","message":"newMessage","reason":"newReason","status":"newStatus","type":"currentType"}]}}`, }, { name: "Should make patch request to update an existing pod condition, but the transition time should remain unchanged because the status is the same", currentPodConditions: []v1.PodCondition{ { Type:               "currentType", Status:             "currentStatus", LastProbeTime:      metav1.NewTime(time.Date(2020, 5, 13, 0, 0, 0, 0, time.UTC)), LastTransitionTime: metav1.NewTime(time.Date(2020, 5, 12, 0, 0, 0, 0, time.UTC)), Reason:             "currentReason", Message:            "currentMessage", }, }, newPodCondition: &v1.PodCondition{ Type:               "currentType", Status:             "currentStatus", LastProbeTime:      metav1.NewTime(time.Date(2020, 5, 13, 1, 1, 1, 1, time.UTC)), LastTransitionTime: metav1.NewTime(time.Date(2020, 5, 12, 0, 0, 0, 0, time.UTC)), Reason:             "newReason", Message:            "newMessage", }, expectedPatchRequests:    1, expectedPatchDataPattern: `{"status":{"$setElementOrder/conditions":[{"type":"currentType"}],"conditions":[{"lastProbeTime":"2020-05-13T01:01:01Z","message":"newMessage","reason":"newReason","type":"currentType"}]}}`, }, { name: "Should not make patch request if pod condition already exists and is identical", currentPodConditions: []v1.PodCondition{ { Type:               "currentType", Status:             "currentStatus", LastProbeTime:      metav1.NewTime(time.Date(2020, 5, 13, 0, 0, 0, 0, time.UTC)), LastTransitionTime: metav1.NewTime(time.Date(2020, 5, 12, 0, 0, 0, 0, time.UTC)), Reason:             "currentReason", Message:            "currentMessage", }, }, newPodCondition: &v1.PodCondition{ Type:               "currentType", Status:             "currentStatus", LastProbeTime:      metav1.NewTime(time.Date(2020, 5, 13, 0, 0, 0, 0, time.UTC)), LastTransitionTime: metav1.NewTime(time.Date(2020, 5, 12, 0, 0, 0, 0, time.UTC)), Reason:             "currentReason", Message:            "currentMessage", }, expectedPatchRequests: 0, }, } for _, test := range tests { t.Run(test.name, func(t *testing.T) { actualPatchRequests := 0 var actualPatchData string cs := &clientsetfake.Clientset{} cs.AddReactor("patch", "pods", func(action clienttesting.Action) (bool, runtime.Object, error) { actualPatchRequests++ patch := action.(clienttesting.PatchAction) actualPatchData = string(patch.GetPatch()) // For this test, we don't care about the result of the patched pod, just that we got the expected // patch request, so just returning &v1.Pod{} here is OK because scheduler doesn't use the response. return true, &v1.Pod{}, nil }) pod := &v1.Pod{ ObjectMeta: metav1.ObjectMeta{Name: "foo"}, Status:     v1.PodStatus{Conditions: test.currentPodConditions}, } updater := &podConditionUpdaterImpl{Client: cs} if err := updater.update(pod, test.newPodCondition); err != nil { t.Fatalf("Error calling update: %v", err) } if actualPatchRequests != test.expectedPatchRequests { t.Fatalf("Actual patch requests (%d) dos not equal expected patch requests (%d)", actualPatchRequests, test.expectedPatchRequests) } regex, err := regexp.Compile(test.expectedPatchDataPattern) if err != nil { t.Fatalf("Error compiling regexp for %v: %v", test.expectedPatchDataPattern, err) } if test.expectedPatchRequests > 0 && !regex.MatchString(actualPatchData) { t.Fatalf("Patch data mismatch: Actual was %v, but expected to match regexp %v", actualPatchData, test.expectedPatchDataPattern) } }) } } </ins>


Question:
Which jobs are flaking: pull-kubernetes-integration Which test(s) are flaking: TestNominatedNodeCleanUp Testgrid link: Reason for failure: /sig scheduling
/assign
any progress on this one?
I was working on it and tried to find the root cause. I will give my solution within this week
Another one
Still happening cc
/assign
/close dupe
Closing this issue. <detailsIn response to : /close dupe Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

framework "k8s.io/kubernetes/pkg/scheduler/framework/v1alpha1" "k8s.io/kubernetes/plugin/pkg/admission/priority" testutils "k8s.io/kubernetes/test/integration/util" <del> utils "k8s.io/kubernetes/test/utils" </del> <ins> "k8s.io/kubernetes/test/utils" </ins> ) var lowPriority, mediumPriority, highPriority = int32(100), int32(200), int32(300)


Question:
Which jobs are flaking: pull-kubernetes-integration Which test(s) are flaking: TestNominatedNodeCleanUp Testgrid link: Reason for failure: /sig scheduling
/assign
any progress on this one?
I was working on it and tried to find the root cause. I will give my solution within this week
Another one
Still happening cc
/assign
/close dupe
Closing this issue. <detailsIn response to : /close dupe Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

Labels:    map[string]string{"pod": name}, Resources: defaultPodRes, }) <del> // Setting grace period to zero. Otherwise, we may never see the actual deletion // of the pods in integration tests. </del> pod.Spec.TerminationGracePeriodSeconds = &grace return pod }


Question:
Which jobs are flaking: pull-kubernetes-integration Which test(s) are flaking: TestNominatedNodeCleanUp Testgrid link: Reason for failure: /sig scheduling
/assign
any progress on this one?
I was working on it and tried to find the root cause. I will give my solution within this week
Another one
Still happening cc
/assign
/close dupe
Closing this issue. <detailsIn response to : /close dupe Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

} // Step 5. Check that nominated node name of the high priority pod is set. if err := waitForNominatedNodeName(cs, highPriPod); err != nil { <del> t.Errorf("NominatedNodeName annotation was not set for pod %v/%v: %v", medPriPod.Namespace, medPriPod.Name, err) </del> <ins> t.Errorf("NominatedNodeName annotation was not set for pod %v/%v: %v", highPriPod.Namespace, highPriPod.Name, err) </ins> } // And the nominated node name of the medium priority pod is cleared. if err := wait.Poll(100*time.Millisecond, wait.ForeverTestTimeout, func() (bool, error) {


Question:
/kind cleanup /sig storage /sig node Description: Feature gate VolumeSubpathEnvExpansion was GA in 1.17 and is marked for removal in 1.19 Related Issues: GA Release:
/assign
/cc

podSpec.EphemeralContainers = nil } <del> if (!utilfeature.DefaultFeatureGate.Enabled(features.VolumeSubpath) || !utilfeature.DefaultFeatureGate.Enabled(features.VolumeSubpathEnvExpansion)) && !subpathExprInUse(oldPodSpec) { // drop subpath env expansion from the pod if either of the subpath features is disabled and the old spec did not specify subpath env expansion </del> <ins> if !utilfeature.DefaultFeatureGate.Enabled(features.VolumeSubpath) && !subpathExprInUse(oldPodSpec) { // drop subpath env expansion from the pod if subpath feature is disabled and the old spec did not specify subpath env expansion </ins> VisitContainers(podSpec, AllContainers, func(c *api.Container, containerType ContainerType) bool { for i := range c.VolumeMounts { c.VolumeMounts[i].SubPathExpr = ""


Question:
/kind cleanup /sig storage /sig node Description: Feature gate VolumeSubpathEnvExpansion was GA in 1.17 and is marked for removal in 1.19 Related Issues: GA Release:
/assign
/cc

} t.Run(fmt.Sprintf("feature enabled=%v, old pod %v, new pod %v", enabled, oldPodInfo.description, newPodInfo.description), func(t *testing.T) { <del> defer featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, features.VolumeSubpathEnvExpansion, enabled)() </del> var oldPodSpec *api.PodSpec if oldPod != nil {


Question:
/kind cleanup /sig storage /sig node Description: Feature gate VolumeSubpathEnvExpansion was GA in 1.17 and is marked for removal in 1.19 Related Issues: GA Release:
/assign
/cc

} func TestValidateSubpathMutuallyExclusive(t *testing.T) { <del> // Enable feature VolumeSubpathEnvExpansion and VolumeSubpath defer featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, features.VolumeSubpathEnvExpansion, true)() </del> <ins> // Enable feature VolumeSubpath </ins> defer featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, features.VolumeSubpath, true)() volumes := []core.Volume{


Question:
/kind cleanup /sig storage /sig node Description: Feature gate VolumeSubpathEnvExpansion was GA in 1.17 and is marked for removal in 1.19 Related Issues: GA Release:
/assign
/cc

} func TestValidateDisabledSubpathExpr(t *testing.T) { <del> // Enable feature VolumeSubpathEnvExpansion defer featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, features.VolumeSubpathEnvExpansion, true)() </del> volumes := []core.Volume{ {Name: "abc", VolumeSource: core.VolumeSource{PersistentVolumeClaim: &core.PersistentVolumeClaimVolumeSource{ClaimName: "testclaim1"}}},


Question:
/kind cleanup /sig storage /sig node Description: Feature gate VolumeSubpathEnvExpansion was GA in 1.17 and is marked for removal in 1.19 Related Issues: GA Release:
/assign
/cc

// while making decisions. BalanceAttachedNodeVolumes featuregate.Feature = "BalanceAttachedNodeVolumes" <del> // owner: @kevtaylor // alpha: v1.14 // beta: v1.15 // ga: v1.17 // // Allow subpath environment variable substitution // Only applicable if the VolumeSubpath feature is also enabled VolumeSubpathEnvExpansion featuregate.Feature = "VolumeSubpathEnvExpansion" </del> // owner: @vladimirvivien // alpha: v1.11 // beta:  v1.14


Question:
/kind cleanup /sig storage /sig node Description: Feature gate VolumeSubpathEnvExpansion was GA in 1.17 and is marked for removal in 1.19 Related Issues: GA Release:
/assign
/cc

VolumeSubpath:                  {Default: true, PreRelease: featuregate.GA}, ConfigurableFSGroupPolicy:      {Default: false, PreRelease: featuregate.Alpha}, BalanceAttachedNodeVolumes:     {Default: false, PreRelease: featuregate.Alpha}, <del> VolumeSubpathEnvExpansion:      {Default: true, PreRelease: featuregate.GA, LockToDefault: true}, // remove in 1.19, </del> CSIBlockVolume:                 {Default: true, PreRelease: featuregate.GA, LockToDefault: true}, // remove in 1.20 CSIInlineVolume:                {Default: true, PreRelease: featuregate.Beta}, RuntimeClass:                   {Default: true, PreRelease: featuregate.Beta},


Question:
/kind cleanup /sig storage /sig node Description: Feature gate VolumeSubpathEnvExpansion was GA in 1.17 and is marked for removal in 1.19 Related Issues: GA Release:
/assign
/cc

return nil, cleanupAction, fmt.Errorf("volume subpaths are disabled") } <del> if !utilfeature.DefaultFeatureGate.Enabled(features.VolumeSubpathEnvExpansion) { return nil, cleanupAction, fmt.Errorf("volume subpath expansion is disabled") } </del> subPath, err = kubecontainer.ExpandContainerVolumeMounts(mount, expandEnvs) if err != nil {


Question: When using the package, one can easily add a flag to a CLI tool. However, the output of invocation with is hard-coded to . As such, the package is not very usable in non-Kubernetes projects. /wg component-standard /assign

var ( versionFlag = Version(versionFlagName, VersionFalse, "Print version information and quit") <ins> programName = "Kubernetes" </ins> ) // AddFlags registers this package's flags on arbitrary FlagSets, such that they point to the


Question: When using the package, one can easily add a flag to a CLI tool. However, the output of invocation with is hard-coded to . As such, the package is not very usable in non-Kubernetes projects. /wg component-standard /assign

fmt.Printf("%#vn", version.Get()) os.Exit(0) } else if *versionFlag == VersionTrue { <del> fmt.Printf("Kubernetes %sn", version.Get()) </del> <ins> fmt.Printf("%s %sn", programName, version.Get()) </ins> os.Exit(0) } }


Question:
What happened: there are many kubelet warnings in the systemd logs I initialized the cluster with kubeadm init (no modifications) What you expected to happen: Not to have that many kubelet warnings in the system logs Anything else we need to know?: Environment: Kubernetes version (use kubectl version):Cloud provider or hardware configuration:OS (e.g: cat /etc/os-release):Kernel (e.g. uname -a):Install tools:
I have the same issue
should add a option or an annotationo to resolve this /sig node /area kubelet
Same issue kops, kubernetes 1.18.3, aws cloudprovider
Same issue on v1.17.4 installed with rancher
Same issue on v1.18.0 ~
Same issue on 1.17.6
Happens on v1.18.3 as well. This issue only occurs when container has some fsGroup set in its securityContext AND volume being mounted is either secret or a configmap. I guess they are being re-mounted every 70s or so. It's okay when the volume being mounted is a standard PV - log only appears once and doesn't get repeated.
I'm seeing the same issue. I'm willing to fix it.
I've digged into this a little bit further. I believe this is just a annoying warning message. AFAICS, it does not show any issue... I was reading the kubelet, its volume manager and the volume plugins source files. It seems that's the expected behavior. The desiredStateOfWorldPopulator sync loop always call the . The finds the volume plugin and check if the plugin requires mount. Secret plugin, wich is the plugin in discussion, always returns . Thus, the of the volume manager remounts the volume every time. I think the problem is in the which prints the warning message. It's more and I believe it does not take these plugins into account. The secret plugin is as it is now since . Is that make sense to you?
/assign

"syscall" "os" <ins> "time" </ins> v1 "k8s.io/api/core/v1" utilfeature "k8s.io/apiserver/pkg/util/feature"


Question:
What happened: there are many kubelet warnings in the systemd logs I initialized the cluster with kubeadm init (no modifications) What you expected to happen: Not to have that many kubelet warnings in the system logs Anything else we need to know?: Environment: Kubernetes version (use kubectl version):Cloud provider or hardware configuration:OS (e.g: cat /etc/os-release):Kernel (e.g. uname -a):Install tools:
I have the same issue
should add a option or an annotationo to resolve this /sig node /area kubelet
Same issue kops, kubernetes 1.18.3, aws cloudprovider
Same issue on v1.17.4 installed with rancher
Same issue on v1.18.0 ~
Same issue on 1.17.6
Happens on v1.18.3 as well. This issue only occurs when container has some fsGroup set in its securityContext AND volume being mounted is either secret or a configmap. I guess they are being re-mounted every 70s or so. It's okay when the volume being mounted is a standard PV - log only appears once and doesn't get repeated.
I'm seeing the same issue. I'm willing to fix it.
I've digged into this a little bit further. I believe this is just a annoying warning message. AFAICS, it does not show any issue... I was reading the kubelet, its volume manager and the volume plugins source files. It seems that's the expected behavior. The desiredStateOfWorldPopulator sync loop always call the . The finds the volume plugin and check if the plugin requires mount. Secret plugin, wich is the plugin in discussion, always returns . Thus, the of the volume manager remounts the volume every time. I think the problem is in the which prints the warning message. It's more and I believe it does not take these plugins into account. The secret plugin is as it is now since . Is that make sense to you?
/assign

fsGroupPolicyEnabled := utilfeature.DefaultFeatureGate.Enabled(features.ConfigurableFSGroupPolicy) <del> klog.Warningf("Setting volume ownership for %s and fsGroup set. If the volume has a lot of files then setting volume ownership could be slow, see https://github.com/kubernetes/kubernetes/issues/69699", mounter.GetPath()) </del> <ins> timer := time.AfterFunc(30*time.Second, func() { klog.Warningf("Setting volume ownership for %s and fsGroup set. If the volume has a lot of files then setting volume ownership could be slow, see https://github.com/kubernetes/kubernetes/issues/69699", mounter.GetPath()) }) defer timer.Stop() </ins> // This code exists for legacy purposes, so as old behaviour is entirely preserved when feature gate is disabled // TODO: remove this when ConfigurableFSGroupPolicy turns GA.


Question:
What happened: Pods get stuck in terminating state, as kubelet failed to unmount volume: nodev on /var/lib/kubelet/pods/-120c-45c4-aeac- type hugetlbfs (rw,relatime,pagesize=2Mi) What you expected to happen: Pod to get deleted and volumes to be cleaned up. How to reproduce it (as minimally and precisely as possible): kvm based virtual machines on ubuntu 16.04. create a pod with hugepage volumes () Delete the pod. Anything else we need to know?: Changes are related to mentioned commit id: In this file : , line no 72 : pageSize, err := resource.ParseQuantity(strings.TrimPrefix(opt, prefix) + "i") The mount already has a pagesize 2Mi in my case as shown here : (rw,relatime,pagesize=2Mi) But the code here suffix/appends "i" assuming pagesize can be only of type 2M or 1024M resulting in failure to clean up this volume. Documentation from code: / NOTE: Adding suffix 'i' as result should be comparable with a medium size. // pagesize mount option is specified without a suffix, // e.g. pagesize=2M or pagesize=1024M for x86 CPUs pageSize, err := resource.ParseQuantity(strings.TrimPrefix(opt, prefix) + "i") if err != nil { return nil, fmt.Errorf("error getting page size from '%s' mount option: %v", opt, err) } when the pod is deployed, kubelet creates the mount and we can see from logs that it already has pagesize=2Mi for my environment. Environment: Kubernetes version: kubectl version Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2020-03-25T14:58:59Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2020-04-16T11:48:36Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: Kvm hypervisor based virual machine setupOS (e.g: ): NAME="Ubuntu" VERSION="16.04.6 LTS (Xenial Xerus)" ID=ubuntu IDLIKE=debian PRETTYNAME="Ubuntu 16.04.6 LTS" VERSIONID="16.04" HOMEURL="" SUPPORTURL="" BUGREPORTURL="" VERSIONCODENAME=xenial UBUNTUCODENAME=xenial Kernel (e.g. ): Linux master 4.4.0-169-generic -Ubuntu SMP Tue Nov 12 10:38:00 UTC 2019 x8664 x8664 x8664 GNU/LinuxInstall tools: Network plugin and version (if this is a network-related bug): Others:
/assign bart0sh
/sig storage
What about this change ?
The change looks good to me. Would you be willing to submit a PR?
& : Thanks for the quick turnaround.
We are seeing this issue in 1.18.6 version, can you please push the fix to this branch as well.
Could we know when the problem was introduced? As I suppose we don't have the same issue with K8S 1.17.x release?
Hi same issue found in 1.18.2
It is not yet ported into v1.18.6 as well.
Can you submit your PR for release-1.18 branch?

// NOTE: Adding suffix 'i' as result should be comparable with a medium size. // pagesize mount option is specified without a suffix, // e.g. pagesize=2M or pagesize=1024M for x86 CPUs <del> pageSize, err := resource.ParseQuantity(strings.TrimPrefix(opt, prefix) + "i") </del> <ins> trimmedOpt := strings.TrimPrefix(opt, prefix) if !strings.HasSuffix(trimmedOpt, "i") { trimmedOpt = trimmedOpt + "i" } pageSize, err := resource.ParseQuantity(trimmedOpt) </ins> if err != nil { return nil, fmt.Errorf("error getting page size from '%s' mount option: %v", opt, err) }


Question:
What happened: Pods get stuck in terminating state, as kubelet failed to unmount volume: nodev on /var/lib/kubelet/pods/-120c-45c4-aeac- type hugetlbfs (rw,relatime,pagesize=2Mi) What you expected to happen: Pod to get deleted and volumes to be cleaned up. How to reproduce it (as minimally and precisely as possible): kvm based virtual machines on ubuntu 16.04. create a pod with hugepage volumes () Delete the pod. Anything else we need to know?: Changes are related to mentioned commit id: In this file : , line no 72 : pageSize, err := resource.ParseQuantity(strings.TrimPrefix(opt, prefix) + "i") The mount already has a pagesize 2Mi in my case as shown here : (rw,relatime,pagesize=2Mi) But the code here suffix/appends "i" assuming pagesize can be only of type 2M or 1024M resulting in failure to clean up this volume. Documentation from code: / NOTE: Adding suffix 'i' as result should be comparable with a medium size. // pagesize mount option is specified without a suffix, // e.g. pagesize=2M or pagesize=1024M for x86 CPUs pageSize, err := resource.ParseQuantity(strings.TrimPrefix(opt, prefix) + "i") if err != nil { return nil, fmt.Errorf("error getting page size from '%s' mount option: %v", opt, err) } when the pod is deployed, kubelet creates the mount and we can see from logs that it already has pagesize=2Mi for my environment. Environment: Kubernetes version: kubectl version Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2020-03-25T14:58:59Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2020-04-16T11:48:36Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: Kvm hypervisor based virual machine setupOS (e.g: ): NAME="Ubuntu" VERSION="16.04.6 LTS (Xenial Xerus)" ID=ubuntu IDLIKE=debian PRETTYNAME="Ubuntu 16.04.6 LTS" VERSIONID="16.04" HOMEURL="" SUPPORTURL="" BUGREPORTURL="" VERSIONCODENAME=xenial UBUNTUCODENAME=xenial Kernel (e.g. ): Linux master 4.4.0-169-generic -Ubuntu SMP Tue Nov 12 10:38:00 UTC 2019 x8664 x8664 x8664 GNU/LinuxInstall tools: Network plugin and version (if this is a network-related bug): Others:
/assign bart0sh
/sig storage
What about this change ?
The change looks good to me. Would you be willing to submit a PR?
& : Thanks for the quick turnaround.
We are seeing this issue in 1.18.6 version, can you please push the fix to this branch as well.
Could we know when the problem was introduced? As I suppose we don't have the same issue with K8S 1.17.x release?
Hi same issue found in 1.18.2
It is not yet ported into v1.18.6 as well.
Can you submit your PR for release-1.18 branch?

Opts:   []string{"rw", "relatime", "pagesize=2M"}, }, { <ins> Device: "/dev/hugepages", Type:   "hugetlbfs", Path:   "/mnt/hugepages-2Mi", Opts:   []string{"rw", "relatime", "pagesize=2Mi"}, }, { </ins> Device: "sysfs", Type:   "sysfs", Path:   "/sys",


Question:
Currently we don't have an easy way of determining what number of requests are apiserver self-requests. This has historically bitten us, since we used to have a 50(?) QPS limit on apiserver self-requests (which thus has been removed). Even so, it would be a good idea to know this generally, since this may be helpful in pinpointing performance regressions for apiserver. /sig api-machinery instrumentation
Can this issue be considered a newbie one? I'm hunting for issues which might be a good first PR.
In the triage meeting was mentioned that would be good to have similar issues to track other well known actors like controller-manager and kubelet. But let's keep this one for api-server calling itself to have it done. /cc
/cc


Question:
Currently we don't have an easy way of determining what number of requests are apiserver self-requests. This has historically bitten us, since we used to have a 50(?) QPS limit on apiserver self-requests (which thus has been removed). Even so, it would be a good idea to know this generally, since this may be helpful in pinpointing performance regressions for apiserver. /sig api-machinery instrumentation
Can this issue be considered a newbie one? I'm hunting for issues which might be a good first PR.
In the triage meeting was mentioned that would be good to have similar issues to track other well known actors like controller-manager and kubelet. But let's keep this one for api-server calling itself to have it done. /cc
/cc

"k8s.io/apimachinery/pkg/types" utilsets "k8s.io/apimachinery/pkg/util/sets" "k8s.io/apiserver/pkg/audit" <ins> "k8s.io/apiserver/pkg/authentication/user" </ins> "k8s.io/apiserver/pkg/endpoints/request" "k8s.io/apiserver/pkg/features" utilfeature "k8s.io/apiserver/pkg/util/feature"