Question: //    (localhost) by name. The possible profile names are detailed at //    http://wiki.apparmor.net/index.php/AppArmor_Core_Policy_Reference string apparmor_profile = 9; <ins> // Seccomp profile for the container, candidate values are: // * runtime/default: the default profile for the container runtime // * unconfined: unconfined profile, ie, no seccomp sandboxing // * localhost/<full-path-to-profile>: the profile installed on the node. //   <full-path-to-profile> is the full path of the profile. string seccomp_profile_path = 10; </ins> } // LinuxContainerConfig contains platform-specific configuration for

Seccomp profile is passed to runtime by in CRI: is relative to node's local seccomp profile root, it is defined in kubelet. But runtime doesn't know it. We should pass full profile path in CRI. cc/ Will file a PR to fix this later.
Add support for Seccomp sounds like a new feature. The milestone should be reset to , correct?
not really. The current implementation supports seccomp already. If it's missing critical information to function properly, this would be a bug.
are we blocking releases for CRI in 1.5? i was not sure, so marking this as non-release-blocker and leaving 1.5 milestone. please correct if i am mistaken.
I don't get why this is a 1.5 targetted thing myself. Yes we want CRI to be in a good state in 1.5, but afaik we have zero intention of anyone actually using (at the very least because of the lack of exec auth regressing kubelet security). why should this be part of 1.5?
First, this is a small enough change that allows non-docker integrations to properly support seccomp. Second, if "we have zero intention of anyone actually using" is a valid reason, we should never fix any bugs in CRI. We do want people to try using it (perhaps not in production yet). However, given that seccomp is only in alpha, I am ok with dropping this and just continue the docker support for now.
Perhaps I misunderstood the meaning of code freeze, but I thought it was only bug-fixes regardless of size of change (or exceptions that have gone through some process) What I meant is we don't intend this to be used generally as part of the 1.5 release. Because of that, I don't think we should consider bugs in CRI bugs for the purpose of the 1.5 release freeze. My statement is just that we shouldn't violate code-freeze for this. We should totally fix this right after the 1.5 freeze window ends, and saying that I'm arguing we should never fix anything is disingenuous and taking it to an absurd extreme.
the only thing we don't seem to agree on is whether seccomp support was considered to be supported for CRI in 1.5.


Question: //    (localhost) by name. The possible profile names are detailed at //    http://wiki.apparmor.net/index.php/AppArmor_Core_Policy_Reference string apparmor_profile = 9; <ins> // Seccomp profile for the container, candidate values are: // * runtime/default: the default profile for the container runtime // * unconfined: unconfined profile, ie, no seccomp sandboxing // * localhost/<full-path-to-profile>: the profile installed on the node. //   <full-path-to-profile> is the full path of the profile. string seccomp_profile_path = 10; </ins> } // LinuxContainerConfig contains platform-specific configuration for

My interpretation is that we assumed seccomp would be supported until we found out that it wouldn't for other runtimes due to the lack of profile path. That's the only reason why I considered this a bug. Of course we could say that this is not really an important feature in CRI, so we can drop it, but we can be more clear on that.
If there's a bug in a feature no one uses, does fixing it make a sound? I think what we're not agreeing on is the role of CRI in 1.5. I understood it as something that is not generally usable. We can't recommend a customer use it, even as an experimental feature, due to security aspects (if nothing else). Correct me if I'm wrong. It's a developer integration point, and developers integrating with it are doing so with the kubernetes master branch, not with some release. Those reasons together result in my opinion that CRI is still not really a customer/release feature, and thus bugs or features related to it should not be exceptions to the freeze window, but rather should targeted at master as soon as the freeze lifts. If this were already post-freeze, would we be cherry-picking this over to 1.5?
This is not true. cri-o is depending on this fix for apparmor support.
is it depending on the 1.5 release branch directly, or on master?
I think it's better to make this in 1.5, so cri-o could work with first release with experimental CRI. It helps to encourage people to use CRI and find features misses or bugs.
Modifying the user-specified annotation is a little bit unsettling. I think adding a new annotation explicitly for the path would be better. Or, we should just add an explicit field for this. I spoke to and separately. I still think this is a bug fix, but we may want to be more careful about the fix. Given the time constraint, let's just list this as a known issue and work on it for 1.6.
OK, let's move this to next release and think carefully about seccomp feature besides this: kubelet may just pass the profile as string via CRI; may assume the profile locate in a file in a known format and pass path to the runtime; or maybe some other solutions.


Question: //    (localhost) by name. The possible profile names are detailed at //    http://wiki.apparmor.net/index.php/AppArmor_Core_Policy_Reference string apparmor_profile = 9; <ins> // Seccomp profile for the container, candidate values are: // * runtime/default: the default profile for the container runtime // * unconfined: unconfined profile, ie, no seccomp sandboxing // * localhost/<full-path-to-profile>: the profile installed on the node. //   <full-path-to-profile> is the full path of the profile. string seccomp_profile_path = 10; </ins> } // LinuxContainerConfig contains platform-specific configuration for

We need to clearly define it. define profile format clearly
Please document known issues for 1.5 in
(copied from ) And let's move here to discuss how to pass profiles to runtime. Related with .
This is issue is related with two level APIs: CRI, since we may also want to support profiles from ConfigMaps, I think we should pass full profile contents instead of path to runtime. issues is also related with seccomp api in kube-apiserver, it is based on annotations today. We should define seccomp profile format clearly (see ) and decide whether to add struct types to pod spec or just continue the annotations way. I think it's more clear to add specific struct types to pod spec.
Agreed - the path stuff is really a crutch (and implementation-specific) - I think is the answer here.
I don't disagree, but it's not strictly a requirement for supporting ConfigMaps (kubelet could create a file with the contents). An open question is whether to pass it as a raw string, or wait for and pass the structured data. The annotations way is deprecated (). I think the bigger question is where/how to put the data. E.g. structured data in the API, unstructured (string type) data in the API, reference to another object (e.g. ConfigMap). If the Kubelet needs to parse the spec, I think that's a good argument for a structured data type (either a standalone object or embedded in the pod API).
Yes, already opened to replace annotations with typed structs. I personally prefer a structured data type in both kubernetes API and CRI. WDYT?
A counter argument is that as the spec grows or new features are , we need to keep updating the Kubernetes API to maintain parity, or update the code in each CRI implementation to translate the Kubernetes spec. OTOH, if we use unstructured data we can use 3rd party libraries to parse out the data we need (assuming the spec evolves in a backwards compatible way).
What spec?
Whichever seccomp profile spec is being implemented by runtimes. E.g. if we decide to base the Kubernetes seccomp profile API off the OCI seccomp profile spec, and OCI adds some new features, at some point we'd probably want to add those features to Kubernetes.


Question: //    (localhost) by name. The possible profile names are detailed at //    http://wiki.apparmor.net/index.php/AppArmor_Core_Policy_Reference string apparmor_profile = 9; <ins> // Seccomp profile for the container, candidate values are: // * runtime/default: the default profile for the container runtime // * unconfined: unconfined profile, ie, no seccomp sandboxing // * localhost/<full-path-to-profile>: the profile installed on the node. //   <full-path-to-profile> is the full path of the profile. string seccomp_profile_path = 10; </ins> } // LinuxContainerConfig contains platform-specific configuration for

It also means that if different runtimes support different feature sets, we'll be stuck either tracking only the common features, or specifying features that may be ignored by the runtime. This may be a non-issue if the seccomp feature set rarely changes.
Well, yes, but that applies to everything in the CRI, and is one of the tradeoffs we ma[d|k]e by introducing this abstraction in the first place.
My $0.02: Moving the existing logic/information for seccomp into a first-class field for CRI (as in ) is productive because of the known problems with the annotation approach. My initial proposal for seccomp support included a resource similar to what has proposed in kubernetes/community and we were unable to come to an agreement on the issue refers to here: I do not necessarily expect that we will be able to come to an agreement about the seccomp resource in time for 1.7, so I think we should proceed with and do backward compatible CRI changes later if we need to inject the content of the seccomp profile instead of the path. I think we would deeply regret embedding API surface to describe seccomp profiles in the Pod API :)


Question: } } } <ins> func TestContainsIPv4Loopback(t *testing.T) { tests := []struct { name        string cidrStrings []string want        bool }{ { name: "empty", want: true, }, { name:        "all zeros ipv4", cidrStrings: []string{"224.0.0.0/24", "192.168.0.0/16", "fd00:1:d::/64", "0.0.0.0/0"}, want:        true, }, { name:        "all zeros ipv4 and invalid cidr", cidrStrings: []string{"invalid.cidr", "192.168.0.0/16", "fd00:1:d::/64", "0.0.0.0/0"}, want:        true, }, { name:        "all zeros ipv6", // interpret all zeros equal for IPv4 and IPv6 as Golang stdlib cidrStrings: []string{"224.0.0.0/24", "192.168.0.0/16", "fd00:1:d::/64", "::/0"}, want:        true, }, { name:        "ipv4 loopback", cidrStrings: []string{"224.0.0.0/24", "192.168.0.0/16", "fd00:1:d::/64", "127.0.0.0/8"}, want:        true, }, { name:        "ipv6 loopback", cidrStrings: []string{"224.0.0.0/24", "192.168.0.0/16", "fd00:1:d::/64", "::1/128"}, want:        false, }, { name:        "ipv4 loopback smaller range", cidrStrings: []string{"224.0.0.0/24", "192.168.0.0/16", "fd00:1:d::/64", "127.0.2.0/28"}, want:        true, }, { name:        "ipv4 loopback within larger range", cidrStrings: []string{"224.0.0.0/24", "192.168.0.0/16", "fd00:1:d::/64", "64.0.0.0/2"}, want:        true, }, { name:        "non loop loopback", cidrStrings: []string{"128.0.2.0/28", "224.0.0.0/24", "192.168.0.0/16", "fd00:1:d::/64"}, want:        false, }, { name:        "invalid cidr", cidrStrings: []string{"invalid.ip/invalid.mask"}, want:        false, }, } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { if got := ContainsIPv4Loopback(tt.cidrStrings); got != tt.want { t.Errorf("ContainLoopback() = %v, want %v", got, tt.want) } }) } } </ins>

Just similar to , for service with type nodeport , iptable proxy should add IsLoopback check for nodeAddress. Line 1104-1106 check nodeAddress is localhost address or not for ipvs mode proxy but for iptable mode proxy , when deal with nodePort type service, there is no similar check for nodeAddress since the use of localhost addresses for nodeports is the only know reason to require (), Just for the purpose of security ,this requirement shall be removed for proxy-mode=itable. Start a cluster with proxy-mode=iptable and some services Do; iptable -Ln | grep 127.0.0.1 just as says : ipv6+iptables , ::1 is not work for Nodeport all supporting proxy-mode=iptable any any /sig network /kind cleanup /area iptable
/sig network
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /area iptable Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/kind cleanup
maybe I'm missing something, but this was discussed in this issue and iptables needs to use localhost for nodeports, if we remove that we can break users. ipvs is different and it never needed loopback for nodeports
set sysctlRouteLocalnet when running in IPtable mode may still face a risk from CVE-2020-8558, so why not disable it in the future version?
because there are users that depend on that behavior, I can't remember the details , the referenced issue has them
I support finding a way to get to the end-state of not supporting nodeports on localhost, but I don't know the path.
/assign
it seems that localhost;nodeport is also used in blogs and forums the only tool we have for these situations is to add a feature gate and rollout the change in several releases, users can always set the sysctl value by themselves to recover previous behavior. should I create a small KEP to deprecate the behavior of setting the TTL? since is inconsistent with the other kube-proxy modes, there is a security risk and the workaround is quite simple, it seems a reasonable decision
I don't think we have a clear path to changing a defsult behavior, even one that is ill-advised.


Question: } } } <ins> func TestContainsIPv4Loopback(t *testing.T) { tests := []struct { name        string cidrStrings []string want        bool }{ { name: "empty", want: true, }, { name:        "all zeros ipv4", cidrStrings: []string{"224.0.0.0/24", "192.168.0.0/16", "fd00:1:d::/64", "0.0.0.0/0"}, want:        true, }, { name:        "all zeros ipv4 and invalid cidr", cidrStrings: []string{"invalid.cidr", "192.168.0.0/16", "fd00:1:d::/64", "0.0.0.0/0"}, want:        true, }, { name:        "all zeros ipv6", // interpret all zeros equal for IPv4 and IPv6 as Golang stdlib cidrStrings: []string{"224.0.0.0/24", "192.168.0.0/16", "fd00:1:d::/64", "::/0"}, want:        true, }, { name:        "ipv4 loopback", cidrStrings: []string{"224.0.0.0/24", "192.168.0.0/16", "fd00:1:d::/64", "127.0.0.0/8"}, want:        true, }, { name:        "ipv6 loopback", cidrStrings: []string{"224.0.0.0/24", "192.168.0.0/16", "fd00:1:d::/64", "::1/128"}, want:        false, }, { name:        "ipv4 loopback smaller range", cidrStrings: []string{"224.0.0.0/24", "192.168.0.0/16", "fd00:1:d::/64", "127.0.2.0/28"}, want:        true, }, { name:        "ipv4 loopback within larger range", cidrStrings: []string{"224.0.0.0/24", "192.168.0.0/16", "fd00:1:d::/64", "64.0.0.0/2"}, want:        true, }, { name:        "non loop loopback", cidrStrings: []string{"128.0.2.0/28", "224.0.0.0/24", "192.168.0.0/16", "fd00:1:d::/64"}, want:        false, }, { name:        "invalid cidr", cidrStrings: []string{"invalid.ip/invalid.mask"}, want:        false, }, } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { if got := ContainsIPv4Loopback(tt.cidrStrings); got != tt.want { t.Errorf("ContainLoopback() = %v, want %v", got, tt.want) } }) } } </ins>

As much as I'd like it to be the default, I think the only way to get there is to add a flag with compatible default behavior and encourage people to turn it on. e.g. add a kube-proxy config like defaulting to true, and encourage people/tools to set it to false.
I've found a better way ;)
/triage accepted


Question: utiltaints "k8s.io/kubernetes/pkg/util/taints" ) <ins> const ( MODIFIED  = "modified" TAINTED   = "tainted" UNTAINTED = "untainted" ) </ins> // TaintOptions have the data required to perform the taint operation type TaintOptions struct { resources      []string

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"6+", GitVersion:"v1.6.0-alpha.3.1185+-dirty Server Version: version.Info{Major:"1", Minor:"6+", GitVersion:"v1.6.0-alpha.3.1185+-dirty" What happened: Kubectl is always reporting as node <XYZis tainted irrespective of operation when used with local cluster. taint nodes XYZ dedicated- node "XYZ" tainted What you expected to happen: It should have shown exact operation like tainted, untainted modified and overwritten but because of it is always returning tainted. How to reproduce it (as minimally and precisely as possible): taint nodes XYZ dedicated-
cc/
Regarding Yes I agree we should provide more clearer output/feedback. But providing something that makes output more confusing than existing is something would be even more concerning. So as I stated , it would be good to discuss/decide how the output/feedback should like for various operations for one and multi nodes scenarios.
I think having 1 return values is an exponential improvement on what we have now, even if they aren't perfect for all cases. Rather then a disproportionately long debate over a relatively simple improvement.... Could you guys suggest something simple that you're confident it will be easy to agree on, and we can implement it as a second pass ? we are happy to implement anything with n 1 return values :).
I think we'd be happy to consider any reasonable PR. I'm not sure why was closed; I didn't look at it though so I don't know if it solves the problem.
(hit close accidentally)
, I am not sure why was closed. But I raised a PR around 15 days ago() to address this.


Question: // we confirm unreachability. framework.Logf("Failed to execute %q: %v, stdout: %q, stderr: %q", cmd, err, stdout, stderr) } else { <del> framework.Logf("Tries: %d, in try: %d, stdout: %v, stderr: %v, command run in: %#v", tries, i, stdout, stderr, config.TestContainerPod) </del> <ins> podInfo := fmt.Sprintf("name: %v, namespace: %v, hostIp: %v, podIp: %v, conditions: %v", config.TestContainerPod.Name, config.TestContainerPod.Namespace, config.TestContainerPod.Status.HostIP, config.TestContainerPod.Status.PodIP, config.TestContainerPod.Status.Conditions) framework.Logf("Tries: %d, in try: %d, stdout: %v, stderr: %v, command run in Pod { %#v }", tries, i, stdout, stderr, podInfo) </ins> var output NetexecDialResponse if err := json.Unmarshal([]byte(stdout), &output); err != nil { framework.Logf("WARNING: Failed to unmarshal curl response. Cmd %v run in %v, output: %s, err: %v",

What happened: We print out massive go structs during the granular network checks What you expected to happen: not print 7K log messages :) How to reproduce it (as minimally and precisely as possible): run sonobuoy with and you'll get this in the logs :) ! Anything else we need to know?: easy fix, will print out the pod json which is much more concise.
/triage accepted
/assign
/sig network
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.


Question: TTY:       o.TTY, Quiet:     o.Quiet, }, <del> // TODO(verb): kubectl prints an incorrect "Session ended" message for debug containers. </del> CommandName: cmd.Parent().CommandPath() + " attach", Attach: &attach.DefaultRemoteAttach{},

When attaching to an ephemeral container, prints a message suggesting that the container can be reattached after it exists. This is not true because an ephemeral container is never restarted. This cosmetic issue should be resolved for
/assign /sig node /triage accepted /priority important-longterm


Question: import ( "crypto/x509" "errors" <ins> "fmt" </ins> "net/http" "reflect" "strings" utilnet "k8s.io/apimachinery/pkg/util/net" <ins> "k8s.io/apiserver/pkg/audit" </ins> "k8s.io/component-base/metrics" "k8s.io/klog/v2" )

go1.18 dropped support for certificates signed with the SHA-1 hash function - While this does improve security, and publicly trusted Certificate Authorities no longer issue such certificates, it is possible that user-provided certificates issued by private Certificate Authorities and used in webhooks or aggregated API servers do still use this. Similar to removal of support for certificates without subjectAltNames in go1.17, we need to: [x] add metrics when webhook/conversion/aggregated certificates with this hash are encountered - [x] document use of and the metric from 1 as a mitigation for clusters which still need to identify and replace use of such certificates - to release note and known issues [x] add audit annotation when SHA-1 cert is encountered - [x] update docs at and - /sig api-machinery instrumentation docs /priority important-soon /milestone v1.24 /kind bug documentation Upgraded 1.23 clusters to continue working as-is Configure an admission webhook, conversion webhook, or aggregated server with a serving certificate signed with the SHA-1 hash function. Upgrade to HEAD of 1.24. The certificate will no longer work by default No response 24 HEAD (go1.18-based) n/a n/a No response No response No response
it looks like NewMissingSANRoundTripperWrapperConstructor already has all the plumbing, and instead of missing SANs we could check the SignatureAlgorithm and output a distinct metric
had a not-yet-tested WIP branch at with a sketch of what it might look like
/assign who offered to pick this up
GitHub didn't allow me to assign the following users: stlaz. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign who offered to pick this up Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Indeed, please, you should trust Jordan :slightlysmilingface:
/assign
/triage accepted
These restrictions are on the GitHub side, this message is posted when it receives a specific error from the GitHub API while attempting to assign It is an annoying GitHub limitation, and one that seemingly does not exist in the web client vs the API.


Question: import ( "crypto/x509" "errors" <ins> "fmt" </ins> "net/http" "reflect" "strings" utilnet "k8s.io/apimachinery/pkg/util/net" <ins> "k8s.io/apiserver/pkg/audit" </ins> "k8s.io/component-base/metrics" "k8s.io/klog/v2" )

adds the metrics. the release note on that issue describes the go change, links to go docs, describes use of the option, the new metrics, and the kube-apiserver logs what would you expect kubeadm to do with this? GODEBUG off by default and let users override GODEBUG on their own (is this possible)? GODEBUG by default on all clusters? GODEBUG by default only on upgrade? The risk of 1 is that a user upgrades without reading release notes and breaks a cluster that has SHA-1 certs in webhooks, etc The risk of 2 is that we propagate accepting insecure certificates longer, even in new clusters that we know don't have any pre-existing bad certs The risk of 3 is that it makes an upgraded 1.23 → 1.24 cluster and a new 1.24 cluster inconsistent
the majority of kubeadm users use kubeadm managed CA and certificates. there are less users that use external CA to sign their own certificates. sha1 attacks surfaced more than a before kubeadm's creation and Git hash collusion made the problem more known around 2017 - that was at a time when kubeadm did not support external CA. given these facts, an estimate on my side is that there are currently no kubeadm users that use sha1 certificates, but i would not exclude the possibility entirely. if such users exists, they are on they own and my vote goes for option 1, until they migrate their PKI to something secure. if the same users log tickets for us, our response would be "please use GODEBUG and migrate ASAP". all users should be able to see the notes in IMO, the PR needs an "action required" to appear on top of the release notes.
that's helpful, thanks agreed. weird, I set , but the label didn't get . the label manually.
and there's a way for them to set that envvar in kube-apiserver manifests when using kubeadm?
There's been a lot of intermittent github webhook delivery issues in the past few weeks or so, slash commands are going to be somewhat unreliable :/
they could use kubeadm patches to override the YAML manifest defaults - i.e. the contents of v1.Container -Env:


Question: k8s.io/apimachinery v0.0.0 k8s.io/client-go v0.0.0 k8s.io/kube-openapi v0.0.0-20211115234752-e816edb12b65 <ins> k8s.io/utils v0.0.0-20220210201930-3a6ce19ff2f9 </ins> sigs.k8s.io/kustomize/api v0.10.1 sigs.k8s.io/kustomize/kyaml v0.13.0 sigs.k8s.io/yaml v1.2.0

The discovery cache for kubectl is defaulted to 10 minutes: That means that every 10 minutes of using against a cluster, it runs API requests against all group versions, which can take quite some time on clusters with many operators installed (like , , , etc.). I previously wrote that shows that every 10 minutes a simple makes 170 API requests against a test cluster, vs 4 API requests when the cache exists. That results in significant productivity implications against such clusters. Here is a time comparison between a request that runs the discovery cache vs one that doesn't: The discovery cache doesn't really have to run every 10 minutes, as CRDs don't change that often. A lot of unnecessary load is created on clients and servers. This issue is raised to change the discovery cache default to 24 hours and also to make the discovery cache ttl configurable. Running the discovery cache every 10 minutes has a significant productivity impact on using on clusters with many CRDs as it is taking time to run these unnecessary requests. The discovery cache doesn't really have to run every 10 minutes, as CRDs don't change that often. A lot of unnecessary load is created on clients and servers.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cli
I would like to work on this issue.
/assign


Question: github.com/mitchellh/go-wordwrap v1.0.0/go.mod h1:ZXFpozHsX6DPmq2I0TCekCxypsnAUbP2oI0UX1GXzOo= github.com/mitchellh/mapstructure v1.1.2 h1:fmNYVwqnSfB9mZU6OS2O6GsXM+wcskZDuKQzvN1EDeE= github.com/mitchellh/mapstructure v1.1.2/go.mod h1:FVVH3fgwuzCH5S8UJGiWEs2h04kUh9fWfEaFds41c1Y= <del> github.com/moby/ipvs v1.0.0 h1:89i8bPaL2DC0cjyRDv0QQOYUOU4fujziJmhF4ca/mtY= github.com/moby/ipvs v1.0.0/go.mod h1:2pngiyseZbIKXNv7hsKj3O9UEz30c53MT9005gt2hxQ= </del> <ins> github.com/moby/ipvs v1.0.1 h1:aoZ7fhLTXgDbzVrAnvV+XbKOU8kOET7B3+xULDF/1o0= github.com/moby/ipvs v1.0.1/go.mod h1:2pngiyseZbIKXNv7hsKj3O9UEz30c53MT9005gt2hxQ= </ins> github.com/moby/term v0.0.0-20200312100748-672ec06f55cd h1:aY7OQNf2XqY/JQ6qREWamhI/81os/agb2BAGpcx5yWI= github.com/moby/term v0.0.0-20200312100748-672ec06f55cd/go.mod h1:DdlQx2hp0Ss5/fLikoLlEeIYiATotOjgB//nb973jeo= github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd h1:TRLaZ9cD/w8PVh93nsPXa1VrQ6jlwL5oN8l14QlcNfg=

Hi, Since my upgrade to 1.18 version, I have errors in kube-proxy: I have ipv4/ipv6 dualstack enable. No problem with cluster and IPVS works despite errors. Do you have also this issue ? OS: RHEL7 Kubernetes version (use ):
/sig network
Related to cc
/cc
Taking one of the outputs it gives me the following addresses? can you share more details about your environment, Is it possible that the CoreDNS pod has the address ? what are your cluster-cidr and service-cidr? Indeed that new net.Utils change looks suspicious , a quick grep in the repo gives me some suspects, maybe the service/ipallocator? It's important to mention there were similar issues with IPVS before, interpreting IPv4 addresses as IPv6, has some experience with this kind of issues, maybe he can see something:) /cc
A bit tricky to trace since libnetwork/ipvs is moving around at the moment. Problem seems to be in; The "family" must be wrong. It is called from;
Please do a and attach the output.
Hi, Some informations: cluster-cidr and service-cidr: fd00:10:245:0:0:0:0:0/64,10.245.0.0/16 fd00:10:96:0:0:0:0:0/112,10.96.0.0/12 ipvsadm output:
Ref:
The commit in the ref above should be in v1.18.0 which is the version used in this issue. So the commit does not fix the problem.
Maybe something wrong around here Before , it looks like this: Now it looks like this: However is not initializedit is zero :(.
I do not get these faults. Has the netlink API changed? What kernel version are you using?
d.AddressFamily is supposed to be set from the "NetlinkRouteAttr" slice; It should definitely not be hard-coded to ipv4.
Made a trying to reproduce this in my environment but didn't got this error.


Question: github.com/mitchellh/go-wordwrap v1.0.0/go.mod h1:ZXFpozHsX6DPmq2I0TCekCxypsnAUbP2oI0UX1GXzOo= github.com/mitchellh/mapstructure v1.1.2 h1:fmNYVwqnSfB9mZU6OS2O6GsXM+wcskZDuKQzvN1EDeE= github.com/mitchellh/mapstructure v1.1.2/go.mod h1:FVVH3fgwuzCH5S8UJGiWEs2h04kUh9fWfEaFds41c1Y= <del> github.com/moby/ipvs v1.0.0 h1:89i8bPaL2DC0cjyRDv0QQOYUOU4fujziJmhF4ca/mtY= github.com/moby/ipvs v1.0.0/go.mod h1:2pngiyseZbIKXNv7hsKj3O9UEz30c53MT9005gt2hxQ= </del> <ins> github.com/moby/ipvs v1.0.1 h1:aoZ7fhLTXgDbzVrAnvV+XbKOU8kOET7B3+xULDF/1o0= github.com/moby/ipvs v1.0.1/go.mod h1:2pngiyseZbIKXNv7hsKj3O9UEz30c53MT9005gt2hxQ= </ins> github.com/moby/term v0.0.0-20200312100748-672ec06f55cd h1:aY7OQNf2XqY/JQ6qREWamhI/81os/agb2BAGpcx5yWI= github.com/moby/term v0.0.0-20200312100748-672ec06f55cd/go.mod h1:DdlQx2hp0Ss5/fLikoLlEeIYiATotOjgB//nb973jeo= github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd h1:TRLaZ9cD/w8PVh93nsPXa1VrQ6jlwL5oN8l14QlcNfg=

I may try this on some older kernel like from CentOS 7 Added exactly the same IPVS configuration here: And when running this 'gist/program' pointing to the IPVS from the DNS (fd00:10:96::a port 53 UDP) got the expected result: Maybe trying to run this also in some other scenarios could clarify if this is something related to Netlink API changed between Kernels, something with the used library, etc.
After some study, I find that this issue most likely is caused by the low linux kernel version. I have reproduced this problem on . Accroding to the code, netlink will try to get attribute, but I find the kernel just does not support it :( This is the defined in (kernel 3.10) No attribute is defined! But in new kernel version, the is defined like : Obviously, the kernel has some attributes (IPVSDESTATTRADDRFAMILY, IPVSDESTATTRSTATS64...). That is why kube-proxy works well on systems with a higher version of the kernel. So we can address this issue by upgrading our linux kernel. Not sure which kernel version is the minimum requirement, maybe we should document something about this.
Here is the defined in :
Good finding. So, we can conclude that this really is kernel version dependent. It is hard to know what to do about it. We must check the family. I would propose that we make it work for ipv4 with some fallback and put a requirement on kernel version for ipv6-only/dual-stack. But the decision is others to make. /cc
/cc
The symbol in the kernel tree The for service family seems very old. Since the service must have the same family as the real-servers perhaps it can be used for compatibility with old kernels?
The commit that adds the symbol; Seem to go into linux-3.18
I did not know the minimum required for the kernel version, I was using the stable and official version of RHEL7. My kernel version is: Currently the max version in the repository is: According to the documentation (), I should switch to RHEL8 if I want a recent kernel :-
There is no minimum kernel version requirement afaik. So I think this must be fixed for ipv4-only clusters IMO.


Question: github.com/mitchellh/go-wordwrap v1.0.0/go.mod h1:ZXFpozHsX6DPmq2I0TCekCxypsnAUbP2oI0UX1GXzOo= github.com/mitchellh/mapstructure v1.1.2 h1:fmNYVwqnSfB9mZU6OS2O6GsXM+wcskZDuKQzvN1EDeE= github.com/mitchellh/mapstructure v1.1.2/go.mod h1:FVVH3fgwuzCH5S8UJGiWEs2h04kUh9fWfEaFds41c1Y= <del> github.com/moby/ipvs v1.0.0 h1:89i8bPaL2DC0cjyRDv0QQOYUOU4fujziJmhF4ca/mtY= github.com/moby/ipvs v1.0.0/go.mod h1:2pngiyseZbIKXNv7hsKj3O9UEz30c53MT9005gt2hxQ= </del> <ins> github.com/moby/ipvs v1.0.1 h1:aoZ7fhLTXgDbzVrAnvV+XbKOU8kOET7B3+xULDF/1o0= github.com/moby/ipvs v1.0.1/go.mod h1:2pngiyseZbIKXNv7hsKj3O9UEz30c53MT9005gt2hxQ= </ins> github.com/moby/term v0.0.0-20200312100748-672ec06f55cd h1:aY7OQNf2XqY/JQ6qREWamhI/81os/agb2BAGpcx5yWI= github.com/moby/term v0.0.0-20200312100748-672ec06f55cd/go.mod h1:DdlQx2hp0Ss5/fLikoLlEeIYiATotOjgB//nb973jeo= github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd h1:TRLaZ9cD/w8PVh93nsPXa1VrQ6jlwL5oN8l14QlcNfg=

However ipv6-only and dual-stack are in "alpha" so a requirement on kernel version might be ok, but I can't say myself. A complication is that the problem is not in k8s itself but in a 3rd party lib.
ipv6-only is beta since 1.18 :smile:
/assign
I opened a PR to fix this in moby/ipvs If someone w/ a cluster on the older kernel version can test it that would be great, otherwise I will setup a cluster to validate this some time this week.
Instead of bootstraping a whole cluster right now (pretty busy here) I can use the Gist I've mentioned to check in a CentOS 7.1 if it faces the problem before and after the patch. At least to have some more test right now
with a small test, it worked fine. Environment: CentOS 7.1 - Kernel 3.10.0-229.el7.x86_64 + ipvsadm 1.2.1 Pre-loaded some IPv6 rules into ipvs: I've used code, searching for the following structure: I then call ` to get the IPVS object, and then try to fetch the destinations. The following are the results: Before patching the ipvs module: After patching the ipvs module (pinning to your repo/branch) Also I've made those tests with the structure but searching for TCP, and for kube-apiserver (that have only one destination) and it worked fine in all those scenarios. If wanted I can improve the gist to pass the parameters as command arguments instead of hardcoded (sorry, did this in a rush to test)
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/remove-triage unresolved
Thanks just waiting on now
same issue,sloved by update kernel to 5.X
I didn't want to wait, I updated to kernel-lt in the version 4.4 . The problem is solved on my side :)
Thanks for confirming. We'll still try to get in since I'm sure other folks with older kernels will run into this issue.
/retitle parseIP error with IPVS proxy
upgrade your system kernel to 4.xx edition
FYI: just merged, need to run some validation & cut a new release (v1.0.1).


Question: github.com/mitchellh/go-wordwrap v1.0.0/go.mod h1:ZXFpozHsX6DPmq2I0TCekCxypsnAUbP2oI0UX1GXzOo= github.com/mitchellh/mapstructure v1.1.2 h1:fmNYVwqnSfB9mZU6OS2O6GsXM+wcskZDuKQzvN1EDeE= github.com/mitchellh/mapstructure v1.1.2/go.mod h1:FVVH3fgwuzCH5S8UJGiWEs2h04kUh9fWfEaFds41c1Y= <del> github.com/moby/ipvs v1.0.0 h1:89i8bPaL2DC0cjyRDv0QQOYUOU4fujziJmhF4ca/mtY= github.com/moby/ipvs v1.0.0/go.mod h1:2pngiyseZbIKXNv7hsKj3O9UEz30c53MT9005gt2hxQ= </del> <ins> github.com/moby/ipvs v1.0.1 h1:aoZ7fhLTXgDbzVrAnvV+XbKOU8kOET7B3+xULDF/1o0= github.com/moby/ipvs v1.0.1/go.mod h1:2pngiyseZbIKXNv7hsKj3O9UEz30c53MT9005gt2hxQ= </ins> github.com/moby/term v0.0.0-20200312100748-672ec06f55cd h1:aY7OQNf2XqY/JQ6qREWamhI/81os/agb2BAGpcx5yWI= github.com/moby/term v0.0.0-20200312100748-672ec06f55cd/go.mod h1:DdlQx2hp0Ss5/fLikoLlEeIYiATotOjgB//nb973jeo= github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd h1:TRLaZ9cD/w8PVh93nsPXa1VrQ6jlwL5oN8l14QlcNfg=

FYI
/retitle DualStack / IPv6-only: parseIP error with IPVS proxy on CentOS 7
v1.18 cherry-pick
Hi, it seems that you're using v1.18.0 and this has been corrected in v1.18.3, can you please update and check? Tks


Question: <del> // +build linux </del> /* Copyright 2017 The Kubernetes Authors.

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: IPVS proxy UT breaks in macOS platform. What you expected to happen: Should run IPVS proxy UTs everywhere. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: That's because there is a flag in , see Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: /sig network
/assign cc


Question: "client_config_test.go", "loader_test.go", "merged_client_builder_test.go", <ins> "overrides_test.go", </ins> "validation_test.go", ], library = ":go_default_library", deps = [ "//vendor/github.com/ghodss/yaml:go_default_library", "//vendor/github.com/imdario/mergo:go_default_library", <ins> "//vendor/github.com/spf13/pflag:go_default_library", </ins> "//vendor/k8s.io/apimachinery/pkg/runtime:go_default_library", "//vendor/k8s.io/apimachinery/pkg/util/errors:go_default_library", "//vendor/k8s.io/client-go/rest:go_default_library",

Because always return names with a prefix like , it cannot be directly used to iterate over all namespaces to execute other commands. I've compiled from source: Reviewing some scripts, I came across a usage of like this: That line fetches namespace names to later run in a loop: That seemed to be a perfect use case for the flag, but it can't be used without extra hacks to get rid of the prefix (FWIW, is invalid). Changing the current output of might not be possible or desirable, for this could impact existing users that rely on the output format. If I'm wrong about the above limitation, than it would make sense to output just the name when and there is a single resource type, while using the ideas introduced in to show the resource type only when there is more than one type in the output, or when is explicitly set. Before: After:
could you help me ping the appropriate team to triage this? Thanks!
Please re-post your question to . We are trying to consolidate the channels to which questions for help/support are posted so that we can improve our efficiency in responding to your requests, and to make it easier for you to find answers to frequently asked questions and how to address common use cases. We regularly see messages posted in multiple forums, with the full response thread only in one place or, worse, spread across multiple forums. Also, the large volume of support issues on github is making it difficult for us to use issues to identify real bugs. The Kubernetes team scans stackoverflow on a regular basis, and will try to ensure your questions don't go unanswered. Before posting a new question, please search stackoverflow for answers to similar questions, and also familiarize yourself with: Again, thanks for using Kubernetes. The Kubernetes Team
perhaps I didn't do a good job at explaining the problem, but this is not a question of how to do something, but a report on a problem with the CLI design.
Sorry about that :-/. I'll try to route to the proper team.
update: console examples to the description.
the -o format is designed for scripting, the use case is just different from yours. You should be able to paste the output of into . Does do what you want? If this is a common use case we could create another -o flag value.


Question: "client_config_test.go", "loader_test.go", "merged_client_builder_test.go", <ins> "overrides_test.go", </ins> "validation_test.go", ], library = ":go_default_library", deps = [ "//vendor/github.com/ghodss/yaml:go_default_library", "//vendor/github.com/imdario/mergo:go_default_library", <ins> "//vendor/github.com/spf13/pflag:go_default_library", </ins> "//vendor/k8s.io/apimachinery/pkg/runtime:go_default_library", "//vendor/k8s.io/apimachinery/pkg/util/errors:go_default_library", "//vendor/k8s.io/client-go/rest:go_default_library",

the use cases I've observed doing code reviews: over all namespaces to get certain resources from that namespace: over all namespaces, then iterate of over resource types that produce logs and fetch logs one by one: Above we can use for listing pods because later we'll pass it to that accept the syntax. We cannot do the same for namespaces, because doesn't like the format.
Alternatively we could be kubectl recognize as valid
I'll take a look on it if it's not taken by anyone yet. CC
Have you started this?
Only investigated due to being out of office since Thursday last week. I can work on it in upcoming week, but I'll understand if it's piority issue and you'll reassign it.
Ready :-)
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list .
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: t.Error(err) } <del> expectPath(t, swagger, "/apis/apiservicegroup/v1") </del> <ins> expectPath(t, swagger, "/apis/apiservicegroup/v1/path1") </ins> expectPath(t, swagger, "/apis/apiregistration.k8s.io/v1") t.Log("Update APIService OpenAPI")

When installing the metrics-server to implement the API and the prometheus-adapter to implement the API, it is possible to make the OpenAPI handler fail to initialize with this error log in apiserver: This will result in kubectl discovery calls to /openapi/v2 to fail with a 503: The OpenAPI handler should not fail to initialize if both and are serving the same path for different API groups/versions. I can provide exact installation steps if needed, however, a high-level repro is: install metrics-server to serve install prometheus-adapter to serve ensure resourceRules is populated in the prometheus-adapter config so that resource metrics are also served by restarting apiserver will result in the OpenAPI failure and 503 responses to /openapi/v2 path No response <details</details<detailsAWS </details<details</details<details</details<details</details<details</details
Please let me know if I should reopen the issue in
/sig api-machinery
/sig instrumentation [metrics apis]
Would you mind taking a look when have time? Thank you^^ /triage accepted
I have a question. should be serving under . Why does an apiservice serving another gvk register paths under ? cc we've made the assumption that an APIService (such as metrics-server or prometheus-adapter) serves only gvks under its own registered group version. I'll do some more investigation but just wondering if prometheus doesn't follow that assumption?
the prometheus-adapter can also be configured to serve and replace the metrics-server by enabling these metrics, However, if the prometheus-adapter fronts and metrics-server fronts , it does seem like the correct behavior would be for prometheus-adapter to serve until an operator updates the APIService to reference prometheus-adapter rather than metrics-server. We're seeing this issue occur for upgrades between 1.27 and 1.28 so it does look this path duplication did not cause the OpenAPI handler to fail on 1.27.
I just upgraded from 1.27 to 1.28 and running into this problem. Some kubectl apply operations fail, and ArgoCD cannot load its custom resources. Any suggested workaround? Would removing the from the prometheus configuration resolve it? Or perhaps disable metrics-server?
Simply looking at the code diff between 1.27 and 1.28 I can't pinpoint the cause.


Question: t.Error(err) } <del> expectPath(t, swagger, "/apis/apiservicegroup/v1") </del> <ins> expectPath(t, swagger, "/apis/apiservicegroup/v1/path1") </ins> expectPath(t, swagger, "/apis/apiregistration.k8s.io/v1") t.Log("Update APIService OpenAPI")

Can I get your exact repro steps? suggests that the conflicting paths should be removed before the merge. We always call this function with
One other question is if this is reproducible by just applying a set of yaml/installations or does it require a cluster upgrade to trigger? Thanks!
hit same problem and we cannot install any other new helm plugins. maybe we need to move forward
Here are verbose repro instructions. An apiserver restart on 1.28 will induce the issue.
Hello. I'm still having some trouble reproducing although I'm not 100% sure if the configuration is correct. I'm trying to repro via kind. afaik installing via helm has prometheus adapter serving . seems to suggest that prometheus adapter is serving under . Applying the update to the configmap per the instructions didn't change the apiservice on my end. Not sure if this is my misconfiguration or intended. I've also tested via integration test that having two openapis serving the same path doesn't cause errors and one will get filtered out. A simple workaround is probably to delete the metrics-server if it is already superseded by the prometheus-adapter.
The is not set up in an resource for us, we only created an APIService with . When I go fetch the openAPI schema served up by the endpoint, it has schema information for in it if that is enabled using . If I remove then the schema fetched from the server doesn't have stuff in it, and the error is resolved. We have been keeping around because I think something uses it directly, or used to. Maybe ? That could have changed But in any case, this problem is quite severe when it happens so it would be nice to fix it as there may be others who do the same update and have to do some deep diving into the logs and all kinds of crazy diagnostics and guesswork to figure it out.
How does the get reached without an APIService? The resources would be unavailable for the apiserver to query and it'll be through something else? I see, adding resources for me doesn't populate the OpenAPI. I wonder if this could be cloud provider specific. I'm totally supportive of fixing, it's just quite difficult to reproduce on my part .
Hmm good question.


Question: t.Error(err) } <del> expectPath(t, swagger, "/apis/apiservicegroup/v1") </del> <ins> expectPath(t, swagger, "/apis/apiservicegroup/v1/path1") </ins> expectPath(t, swagger, "/apis/apiregistration.k8s.io/v1") t.Log("Update APIService OpenAPI")

In our staging cluster I tried bringing back the in the custom metrics server configuration and restarting all the masters, and the problem did not come back, everything seems OK now. So it maybe isn't that easy to reproduce
After updating the configmap, restarting the prometheus-adapter pod to pickup the changes and restarting the apiserver are both required to induce the issue. I have been able to deterministically reproduce this following those steps, please follow up here if you are still having issues.
That's good to hear. I'll need some more help with reproducing it. I've been able to verify that restarting the prometheus adapter pod picks up the configmap changes but restarting the apiserver hasn't caused any crashes for me. I've tested this both on the latest master branch and 1.28.
While I'm yet to fully understand these changes, the 2 PRs below (both of which went into 1.28) seem likely candidates for a regression here: - introduced lazy loading in openAPI v2 handler to load/compute the aggregator spec when requested by client - switched the spec merging function from to any clue if one of the above changes could be causing mischief?
Also from example request at the top, the impact seems to be to the v2 API: Nathan - when you get a chance, could you also confirm if v1 API doesn't run into this?
OpenAPI only supports v2 and v3.
hi, any clue about this issue?
It's possible but I haven't been able to reproduce it. Can I ask if this is happening on cloud providers outside AWS?
Here are the apiservices and relevant pod objects with a working repro from these steps If it's useful, I can provide yamls for these objects in addition to the settings applied in the prometheus-adapter configmap. Can you confirm that the pods backing your prometheus-adapter and metrics-server are running?
I meet this problem
We the problem in GCP as well.
ACK in Alibaba Cloud also meets this problem
We were able to fix the v2 availability by and re-installing the kube-prometheus helm chart. Unfortunately, after restarting the k8s api server, v2 availability is broken again.
yes, I'm running on kind. I'll do some more digging, assuming it's related to those PRs linked by but still haven't pinpointed anything.


Question: t.Error(err) } <del> expectPath(t, swagger, "/apis/apiservicegroup/v1") </del> <ins> expectPath(t, swagger, "/apis/apiservicegroup/v1/path1") </ins> expectPath(t, swagger, "/apis/apiregistration.k8s.io/v1") t.Log("Update APIService OpenAPI")

I tried to reproduce on my local setup with based on steps + , but I could still get a successful response when I executed . Then I reverted one of the step mentioned in to and waited until the new pod came alive and then did a apiserver restart and I was able to reproduce the same thing on as well i.e; I get the 503 response when I execute following command after the fact. After this point, even if i change it back to and restart the apiserver , the error is persistent. Let me know if this works for you to reproduce ? PS: Also, i want to call out that there is no issue with , its .
Thanks, alright I found a way to reproduce it consistently without restarting the apiserver. Still investigating the root cause. Thanks for the detailed reproduction and
Quick update: This issue was a combination of bugs both in k/k and prometheus adapter. I opened to mitigate on k/k side which should fix the issue. The issue was difficult to reproduce because it requires a very specific ordering of OpenAPI merging to trigger, which is why I suspect many people were able to reproduce after some restarts to kube-apiserver. Prometheus and metrics server serve differing OpenAPIs for the endpoint. Prometheus also doesn't seem to be registering discovery's OpenAPI properly (missing OpenAPI for but having the OpenAPI for whereas it is all present in metrics server. This inconsistency in how prometheus and metrics server handle the OpenAPI surfaced the merging issue.
a little wired question: why do we need to restart kube-apiserver to reproduce this problem?
Tried to reproduce the issue after cherry picking the fix and didn't run into below error. Fix is working as expected.
It can look like restarting prometheus-adapter "fixes" the issue? I am also curious about suggestion of removing the section from the config map works as a temporary workaround until released upstream (we use and cannot delete the apiregistration)? As I understand the fix for this issue is currently in


Question: expectedList: []HostPriority{{"machine1", 0}, {"machine2", 0}}, test:         "no resources requested", pods: []api.Pod{ <del> {CurrentState: machine1State, Labels: labels2}, {CurrentState: machine1State, Labels: labels1}, {CurrentState: machine2State, Labels: labels1}, {CurrentState: machine2State, Labels: labels1}, </del> <ins> {DesiredState: machine1State, Labels: labels2}, {DesiredState: machine1State, Labels: labels1}, {DesiredState: machine2State, Labels: labels1}, {DesiredState: machine2State, Labels: labels1}, </ins> }, }, {

When creating replication controller (e.g. frontend php-redis from guestbook example), scheduler gets stuck and never recovers. This is a race, I have to run several times to see the error (A script to delete controller and pods, then create new controller, leave enough time in between for k8s to react). I've created PR for an attempted fix.
I saw this too while following the guest book and creating the php/frontend step. In my case, I had 2 minions while the replication controller configuration had configured 3 replicas, so 2 pods got scheduled -each on another minion, the third pod had empty Host (saw that when doing 'list pods'), and the 3 of them were stuck in "waiting" status. I deleted the controller and the pods, fixed the configuration in json file to be 2 replicas and recreated the controller and everything worked. Does your fix take care of this use case? (too many replicas, too few minions/one of the pods is not scheduled)
This is different from your use case. If you have more replicas than minions, then the scheduler will get stuck for sure (with the error msg probably being 'failed to find a fit for pod'). AFAICT, it's not support yet.
If you remove the HostPorts, it should work for you. thanks for report.
, thanks for your replies. I understand why it isn't working, but I'd expect one of the 2 options to happen instead of the current situation: the creation of replication controller if there are less minions than replicas, AND there is a hostPort property. In other words - validation of the content during creation. least schedule all replicas that can get a minion (in my case 2 out of 3) and leave only one at "waiting". Is there an issue open about the use case I describe?
should be true today. is less possible than it seems like-- minions can come and go, and become over or under loaded in the time between creation and when the pod shows up at the scheduler. So even if we check up front, we still have to check again later, and sometimes the first check will pass but not the second.


Question: import ( "context" "fmt" <del> "k8s.io/kubernetes/pkg/apis/core/v1/helper/qos" "k8s.io/kubernetes/pkg/kubelet/apis/config" e2eskipper "k8s.io/kubernetes/test/e2e/framework/skipper" "k8s.io/kubernetes/test/e2e/nodefeature" imageutils "k8s.io/kubernetes/test/utils/image" </del> "math/big" "path/filepath" "strconv" "strings" "time" <ins> "k8s.io/kubernetes/pkg/apis/core/v1/helper/qos" "k8s.io/kubernetes/pkg/kubelet/apis/config" e2eskipper "k8s.io/kubernetes/test/e2e/framework/skipper" "k8s.io/kubernetes/test/e2e/nodefeature" imageutils "k8s.io/kubernetes/test/utils/image" </ins> "github.com/onsi/ginkgo/v2" "github.com/onsi/gomega" v1 "k8s.io/api/core/v1"

master-informing: ci-crio-cgroupv1-node-e2e-conformanceE2eNode Suite.[It] [sig-node] Swap [LinuxOnly] [NodeFeature:NodeSwap] [NodeConformance] with configuration QOS Best-effort E2eNode Suite.[It] [sig-node] Swap [LinuxOnly] [NodeFeature:NodeSwap] [NodeConformance] with configuration QOS Burstable E2eNode Suite.[It] [sig-node] Swap [LinuxOnly] [NodeFeature:NodeSwap] [NodeConformance] with configuration QOS Burstable with memory request equals to limit E2eNode Suite.[It] [sig-node] Swap [LinuxOnly] [NodeFeature:NodeSwap] [NodeConformance] with configuration QOS Guaranteed After was merged No response /sig node
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/cc


Question: } return c.Name, nil } <ins> // reattachMessage returns a message to print after attach has completed, or // the empty string if no message should be printed. func (o *AttachOptions) reattachMessage(containerName string, rawTTY bool) string { if o.Quiet || !o.Stdin || !rawTTY || o.Pod.Spec.RestartPolicy != corev1.RestartPolicyAlways { return "" } if _, path := podcmd.FindContainerByName(o.Pod, containerName); strings.HasPrefix(path, "spec.ephemeralContainers") { return fmt.Sprintf("Session ended, the ephemeral container will not be restarted but may be reattached using '%s %s -c %s -i -t' if it is still running", o.CommandName, o.Pod.Name, containerName) } return fmt.Sprintf("Session ended, resume using '%s %s -c %s -i -t' command when the pod is running", o.CommandName, o.Pod.Name, containerName) } </ins>

When attaching to an ephemeral container, prints a message suggesting that the container can be reattached after it exists. This is not true because an ephemeral container is never restarted. This cosmetic issue should be resolved for
/assign /sig node /triage accepted /priority important-longterm


Question: This is enforced throughout this document.   Current pod descriptors use an `emptyDir` for storing data in each data node container. This is meant to be for the sake of simplicity and [should be adapted according to your storage needs](../../../docs/design/persistent-storage.md).    Current pod descriptors use an `emptyDir` for storing data in each data node container. This is meant to be for the sake of simplicity and [should be adapted according to your storage needs](../../../docs/design/persistent-storage.md).  ## Docker image

This includes but effects multiple docs.
This issue was also raised on the repository, see:
What's needed to fix this? Are the broken headers being generated automatically somewhere?
- The warning section is pre-pended to the md docs from the following script The fixes this issue, updating this script and all the docs... However, there is an open question as to whether "old" docs should be removed from this repo, which has prevented this PR from being merged.


Question: } func getNodeDevices() (*podresourcesapi.ListPodResourcesResponse, error) { <del> endpoint := util.LocalEndpoint(defaultPodResourcesPath, podresources.Socket) </del> <ins> endpoint, err := util.LocalEndpoint(defaultPodResourcesPath, podresources.Socket) if err != nil { return nil, fmt.Errorf("Error getting local endpoint: %v", err) } </ins> client, conn, err := podresources.GetClient(endpoint, defaultPodResourcesTimeout, defaultPodResourcesMaxSize) if err != nil { return nil, fmt.Errorf("Error getting grpc client: %v", err)

What happened: I replaced the kubelet on one of my 1.14 nodes with a build from master, and now I'm getting this error starting up. Test passes since 5/31/2019 commit have been failing on testgrid. If you look at the logs, the Windows nodes are not coming up. What you expected to happen: No errors starting up kubelet How to reproduce it (as minimally and precisely as possible): Build master, then try to run the kubelet on Windows Anything else we need to know?: /sig windows Environment: Kubernetes version (use ): Cloud provider or hardware configuration: Azure OS (e.g: ): Windows Server 2019 Kernel (e.g. ): 10.0..529 Install tools: aks-engine
cc /priority critical-urgent
/milestone v1.15 Release team - it looks like a recent PR broke windows nodes. We will need a fix here for 1.15
Ok, so the problem here is a little more complicated and we're actually seeing a bug that is actually in the code for quite some time, just that we never used that particular code path. But first: TL;DR: Quick fix is to start the kubelet on Windows with: --feature-gates="KubeletPodResources=false" Now, as to why this is happening: PR [1] was merged 3 days ago and it graduates KubeletPodResources feature as Beta. That means kubelet will enable it by default.With the introduction of the above feature, this branch of code is entered when starting kubelet [2]. We can see a call to the function: defined here [3].While innocent enough, gets the necessary npipe by calling: . is defined here [4] for Windows . The problem is actually that url.URL can't handle Windows style paths [5] and will replace "" with "%5C" as it's to be expected. The paths are transformed along the way from the default "/var/lib/kubelet" to "varlibkubelet" by defined here [6]. More specifically in the use of to create a path. It's inputs are unix style but returns the path Windows style as it's to be expected.The solution is to replace the usage of url.URL in with some alternative package or just implement it.


Question: } func getNodeDevices() (*podresourcesapi.ListPodResourcesResponse, error) { <del> endpoint := util.LocalEndpoint(defaultPodResourcesPath, podresources.Socket) </del> <ins> endpoint, err := util.LocalEndpoint(defaultPodResourcesPath, podresources.Socket) if err != nil { return nil, fmt.Errorf("Error getting local endpoint: %v", err) } </ins> client, conn, err := podresources.GetClient(endpoint, defaultPodResourcesTimeout, defaultPodResourcesMaxSize) if err != nil { return nil, fmt.Errorf("Error getting grpc client: %v", err)

[1] [2] [3] [4] [5] [6]
Thanks for the analysis! I was a bit concerned around the fact that this started showing up around my commit: but the root cause here is indeed, as you said, the graduation of feature to Beta and the way the invalid escape sequences are being introduced.
GCE tests failing too /cc
I'm trying a quick to see if anything else is broken. There are other functions in that same file doing that substitution already so it may make sense to use that as a workaround until a better solution is available.
Still working on it - looks like the path to create the directories is also busted. With still getting


Question: cancel() // mutate request Headers // Authorization filter does it for example <del> for j := 0; j < 10000; j++ { req.Header.Set("Test", "post") </del> <ins> for { select { case <-testDone: return default: req.Header.Set("Test", "post") } </ins> } }), func(r *http.Request, requestInfo *request.RequestInfo) bool {

Similar to and easily reproducible. The handler occasionally completes and sends a response before timing out. /kind failing-test /kind flake /sig api-machinery
/assign
/assign Thank you very much for fixing these tests.
/triage accepted


Question: sleepingPod := getSleepingPod(f.Namespace.Name) sleepingPod = runPodAndWaitUntilScheduled(f, sleepingPod) <del> gomega.Expect(isPodCgroupV2(f, sleepingPod)).To(gomega.BeTrueBecause("node uses cgroup v1")) </del> <ins> if !isPodCgroupV2(f, sleepingPod) { e2eskipper.Skipf("swap tests require cgroup v2") } </ins> nodeName = sleepingPod.Spec.NodeName gomega.Expect(nodeName).ToNot(gomega.BeEmpty(), "node name is empty")


Question: // Important to wait here to ensure we prime the cache with the initial list // of documents in order to exercise 304 Not Modified <del> require.True(t, waitForEmptyQueue(testCtx.Done(), aggregatedManager)) </del> <ins> require.True(t, waitForQueueComplete(testCtx.Done(), aggregatedManager)) </ins> <del> // Now add all groups. We excluded one group before so that AllServicesSynced // could include it in this round. Now, if AllServicesSynced ever returns </del> <ins> // Now add all groups. We excluded one group before so that waitForQueueIsComplete // could include it in this round. Now, if waitForQueueIsComplete ever returns </ins> // true, it must have synced all the pre-existing groups before, which would // return 304 Not Modified for _, s := range apiServices {

pull-kubernetes-unit sometimes fails with data race. TestDirty I think the test was 3 months ago with the flaking state. There are two issues. First, updating and reading a bool-type variable is not thread-safe. and there is also another problem checking the state before completely syncing it. No response /sig api-machinery
/assign
/triage accepted


Question: if e.stdout != nil { waitCount++ <del> headers.Set(api.StreamType, api.StreamTypeStdout) remoteStdout, err := conn.CreateStream(headers) if err != nil { return err } defer remoteStdout.Reset() </del> go cp(api.StreamTypeStdout, e.stdout, remoteStdout) } if e.stderr != nil && !e.tty { waitCount++ <del> headers.Set(api.StreamType, api.StreamTypeStderr) remoteStderr, err := conn.CreateStream(headers) if err != nil { return err } defer remoteStderr.Reset() </del> go cp(api.StreamTypeStderr, e.stderr, remoteStderr) }

We create the stdin stream and immediately start copying local stdin to the container. We do the same for stdout and stderr (create stream, start copying immediately). In the kubelet, we wait to receive all the streams before we start the exec and copy between the streams and the container. The spdy connection frame handling code has 5 frame workers, each with a queue of up to 50 frames. In the event that we try to push more than 50 frames to a single queue before all streams are ready, the 51st attempt to push a frame blocks until there is space available in the queue. But there won't be any space until the server starts reading from the streams, which won't happen until all the streams have been received, so we're stuck at this point. The fix is to create all the streams first, and only start copying after all the streams have been created and the server is ready to copy between the streams and the container. Originally reported here:
FYI
This seems pretty high priority.
It's an easy fix. has already tested the patch for OpenShift and it works. Are we willing to pull it in before is resolved, as that will take longer to put in place?
Here's the patch, for reference:
5 * 50 = 250 frames before you get stuck? Or did I misunderstand?
Oh, I guess all the frames go to the same worker for some reason?
anyway, I'm fine with taking this and cherry picking it. Order doesn't matter; someone has to rebase either way, right?
frames are assigned to each queue based on stream id, so if you sent 50 frames to the stdin stream, that queue fills up and the loop that processes incoming frames is blocked until something starts reading from the stream. Working on a PR now.
Gotcha, thanks. , Andy Goldstein wrote:
an update: I spent some time trying to create an e2e that would fail more often than not without this fix, but I haven't had any luck so far (I have it stream a copy of the kubectl binary from the local system to the pod). The issue that originally uncovered the problem was from running exec on a Windows system, where it reproduces consistently, but I haven't seen it fail much on Linux. I will keep trying...


Question: // services that need to be synced queue           *workqueue.Type knownClusterSet sets.String <ins> // endpoint worker map contains all the clusters registered with an indication that worker exist // key clusterName endpointWorkerMap map[string]bool // channel for worker to signal that it is going out of existence endpointWorkerDoneChan chan string // service worker map contains all the clusters registered with an indication that worker exist // key clusterName serviceWorkerMap map[string]bool // channel for worker to signal that it is going out of existence serviceWorkerDoneChan chan string </ins> } // New returns a new service controller to keep DNS provider service resources

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): No Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Environment: Cloud provider or hardware configuration: GCP OS (e.g. from /etc/os-release):Kernel (e.g. ):Install tools: Others: What happened: Memory usage of federation-control-manager keeps on increasing to very large size overtime. What you expected to happen: Memory usage is stable How to reproduce it (as minimally and precisely as possible): Register one or more cluster to federation control plane and watch the top output for the federation-control-manager Also can check the pprof, here is the one such output after federation-control-manager is run for an hour. We can see there are around goroutines created in an hour which are leaked and would consume memory. Anything else do we need to know: I do have a fix for the issue and would be happy to submit it, if this issue is accepted.
plz check the issue
feel free to just submit a pull request, we don't really have a concept of an issue being "accepted" :)
This looks like a legitimate bug. could you please clarify what version of the federated service controller you are running? The image path and version of the container, or a git hash of the source should be sufficient.
i am taking the latest code from master branch.


Question: timeout    time.Duration mu         sync.Mutex // protects active active     bool <del> // TODO: make this an net.IP address publicIP []string </del> <ins> publicIP   []string // TODO: make this a []net.IP </ins> } func (si *serviceInfo) isActive() bool {

I'm not sure if it's the intended behaviour, but when I specify publicIPs for a service, external portals are only opened if createExternalLoadBalancer is set, too (this is checked in the OnUpdate function in ). After reading the docs, I would have expected that IPTables rules for external IPs are independent of the createExternalLoadBalancer flag. Also, we're running kubernetes locally, so setting createExternalLoadBalancer doesn't work since there's no cloud provider specified.
I was about to say it should work, but then I went and re-read the code and found at least one bug. Let me dig a bit more
You're right - the IPs actually propagate to the proxy, but it ignores them if createExternal... is not set. I think it is kosher to just do it. I'm prepping a PR
PR is up - if you feel bold you could test it in your environment...
Thanks, it's working fine with your patch.


Question: Now we will turn up two replicas of that image.  They all serve on port 8080, mapped to internal port 80 $ cd kubernetes <del> $ cluster/cloudcfg.sh -p 8080:80 run $DOCKER_USER/data 2 dataController </del> <ins> $ cluster/kubecfg.sh -p 8080:80 run $DOCKER_USER/data 2 dataController </ins> ### Step Three: Turn up the UX for the demo In a different terminal: $ cd kubernetes <del> $ cluster/cloudcfg.sh -proxy -www examples/update-demo/local/ </del> <ins> $ cluster/kubecfg.sh -proxy -www examples/update-demo/local/ </ins> Now visit the the [demo website](http://localhost:8001/static/index.html).  You should see two light blue squares with pod IDs and ip addresses.

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: Which test(s) are failing: Overall Since when has it been failing: Nov 20 between 8:41 PST and 12:43 PST (it's been "failing" before that as well, but now none of the tests are running at all) Testgrid link: Reason for failure: Looks like  failed (), so is not building the env properly Slack discussion: Anything else we need to know*: /sig cluster-lifecycle /area provider/gcp
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
fixed here:
/assign


Question: Source string `json:"source,omitempty" description:"component reporting this event; short machine understandable string"` // Host name on which the event is generated. <del> Host string `json:"host,omitempty"` </del> <ins> Host string `json:"host,omitempty" description:"host name on which this event was generated"` </ins> // The time at which the client recorded the event. (Time of server receipt is in TypeMeta.) Timestamp util.Time `json:"timestamp,omitempty" description:"time at which the client recorded the event"`

Or for some other reason it complains about v1beta1 and 2 on my system even though actually grepping those files shows no problems.
it's griping about Host


Question: RegisterSchedulable:                 true, ExperimentalKernelMemcgNotification: false, RemoteRuntimeEndpoint:               remoteRuntimeEndpoint, <del> RotateCertificates:                  false, </del> // TODO(#54161:v1.11.0): Remove --enable-custom-metrics flag, it is deprecated. EnableCustomMetrics: false, NodeLabels:          make(map[string]string),

/kind cleanup What happened: and both require opt-in configuration knobs to the kubelet. They are kinda global settings, you probably either opt into dynamic cert rotation or you don't, so they qualify for being in the struct. The knob for enabling dynamic serving certs was (I fully support that :+1:). What you expected to happen: I'd like the same to happen to  to be consistent. (Also, we're planning to use the config file for the kubelet by default in kubeadm soon, and it'd be nice to be able to set this value then in a structured manner) /assign


Question: RegisterSchedulable:                 true, ExperimentalKernelMemcgNotification: false, RemoteRuntimeEndpoint:               remoteRuntimeEndpoint, <del> RotateCertificates:                  false, </del> // TODO(#54161:v1.11.0): Remove --enable-custom-metrics flag, it is deprecated. EnableCustomMetrics: false, NodeLabels:          make(map[string]string),

Flag name: Help text: <Warning: Beta featureAuto rotate the kubelet client certificates by requesting new certificates from the kube-apiserver when the certificate expiration approaches. This is part of migrating the Kubelet command-line to a Kubernetes-style API. The flag should either be migrated to the Kubelet's API group, or simply removed from the Kubelet. If this could be considered an instance-specific flag, or a descriptor of local topology managed by the Kubelet, see: If this flag is only registered in os-specific builds, see: /assign /sig node /kind feature /priority important-soon /milestone v1.11 /status approved-for-milestone
[MILESTONENOTIFIER] Milestone Issue: Up-to-date for process <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. : New functionality. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details


Question: <ins> /* Copyright 2024 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ // This is an empty file to avoid "build constraints exclude all Go files" error while running unit tests on non-linux platforms. package ipvs </ins>

Tried running make test against local latest fork of kubernetes master Go version Go env Successfully able to run the test Try running make test on darwin os. No response <details</detailsLatest master <details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
Bro i think there might be problem with your setup , its not an issue .
test cases are working fine
Thanks for verifying, I will double check from my side, I hope you verified it on darwin os.
/sig release
Also seeing this when trying to run on mac


Question: package network import ( <ins> "encoding/hex" </ins> "encoding/json" "fmt" "math" <ins> "net" </ins> "strconv" "strings" "time" v1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" <ins> "k8s.io/apimachinery/pkg/util/wait" </ins> "k8s.io/kubernetes/test/e2e/framework" e2enode "k8s.io/kubernetes/test/e2e/framework/node" e2epod "k8s.io/kubernetes/test/e2e/framework/pod" <del> e2essh "k8s.io/kubernetes/test/e2e/framework/ssh" </del> "k8s.io/kubernetes/test/images/agnhost/net/nat" imageutils "k8s.io/kubernetes/test/utils/image"

Which jobs are failing: kops-aws and many other jobs Which test(s) are failing: This isn't the actual test, but the text we're consistently seeing is Since when has it been failing: Testgrid links: (ref: ) Reason for failure: TBD Anything else we need to know: Very rarely tests pass when one of the tests passes: Seems related to , that was merged about the same time:
/sig cluster-lifecycle /sig aws
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig cluster-lifecycle /sig aws Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/priority critical-urgent this is merge blocking
/assign /cc /cc /cc
The PR we think we want to revert
is also failing, I don't think this is aws-specific
/cc
/assign
it failed with kind too :/ with the same message
Opened to revert
/unassign /sig network /sig testing /remove-sig cluster-lifecycle
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
This may be the actual cause PR to address But we're going with revert first to be sure we've identified the offending PR
I think I've identified the root cause
Revert brought us back to green: Awaiting root cause fix
/remove-triage unresolved
/reopen merged but we're now seeing multiple tests fail since it merged Looking at it went red between these commits:
Reopened this issue. <detailsIn response to : /reopen merged but we're now seeing multiple tests fail since it merged Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
is an attempt to fix
reverting in
is the failure cluster I suspect will go down once the revert goes in
/close Failure cluster has gone down, calling this done <img width="998" alt="Screen Shot 2020-01-10 at 9 56 17 AM" src="https://user-"
Closing this issue.


Question: package network import ( <ins> "encoding/hex" </ins> "encoding/json" "fmt" "math" <ins> "net" </ins> "strconv" "strings" "time" v1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" <ins> "k8s.io/apimachinery/pkg/util/wait" </ins> "k8s.io/kubernetes/test/e2e/framework" e2enode "k8s.io/kubernetes/test/e2e/framework/node" e2epod "k8s.io/kubernetes/test/e2e/framework/pod" <del> e2essh "k8s.io/kubernetes/test/e2e/framework/ssh" </del> "k8s.io/kubernetes/test/images/agnhost/net/nat" imageutils "k8s.io/kubernetes/test/utils/image"

<detailsIn response to : /close Failure cluster has gone down, calling this done <img width="998" alt="Screen Shot 2020-01-10 at 9 56 17 AM" src="https://user-"</details


Question: cpu: 500m # memory units are bytes memory: 64Mi <ins> requests: # cpu units are cores cpu: 500m # memory units are bytes memory: 64Mi </ins> ``` <del> The container will die due to OOM (out of memory) if it exceeds its specified limit, so specifying a value a little higher than expected generally improves reliability. </del> <ins> The container will die due to OOM (out of memory) if it exceeds its specified limit, so specifying a value a little higher than expected generally improves reliability. By specifying request, pod is guaranteed to be able to use that much of resource when needed. See [Resource QoS](../proposals/resource-qos.md) for the difference between resource limits and requests. </ins> If you’re not sure how much resources to request, you can first launch the application without specifying resources, and use [resource usage monitoring](monitoring.md) to determine appropriate values.

did which addresses some of them, but turns up a lot more.
Alright, I will do that.


Question: // Action is a function to be performed by the system. type Action func() error <del> // observeNodeUpdateAfterAction returns true if a node update matching the predicate was emitted // from the system after performing the supplied action. func observeNodeUpdateAfterAction(c clientset.Interface, nodeName string, nodePredicate func(*v1.Node) bool, action Action) (bool, error) { observedMatchingNode := false nodeSelector := fields.OneTermEqualSelector("metadata.name", nodeName) informerStartedChan := make(chan struct{}) var informerStartedGuard sync.Once _, controller := cache.NewInformer( &cache.ListWatch{ ListFunc: func(options metav1.ListOptions) (runtime.Object, error) { options.FieldSelector = nodeSelector.String() ls, err := c.CoreV1().Nodes().List(context.TODO(), options) return ls, err }, WatchFunc: func(options metav1.ListOptions) (watch.Interface, error) { // Signal parent goroutine that watching has begun. defer informerStartedGuard.Do(func() { close(informerStartedChan) }) options.FieldSelector = nodeSelector.String() w, err := c.CoreV1().Nodes().Watch(context.TODO(), options) return w, err }, }, &v1.Node{}, 0, cache.ResourceEventHandlerFuncs{ UpdateFunc: func(oldObj, newObj interface{}) { n, ok := newObj.(*v1.Node) framework.ExpectEqual(ok, true) if nodePredicate(n) { observedMatchingNode = true } }, }, ) // Start the informer and block this goroutine waiting for the started signal. informerStopChan := make(chan struct{}) defer func() { close(informerStopChan) }() go controller.Run(informerStopChan) <-informerStartedChan // Invoke the action function. err := action() if err != nil { return false, err } // Poll whether the informer has found a matching node update with a timeout. // Wait up 2 minutes polling every second. timeout := 2 * time.Minute interval := 1 * time.Second err = wait.Poll(interval, timeout, func() (bool, error) { return observedMatchingNode, nil }) return err == nil, err } </del> // observeEventAfterAction returns true if an event matching the predicate was emitted // from the system after performing the supplied action. func observeEventAfterAction(c clientset.Interface, ns string, eventPredicate func(*v1.Event) bool, action Action) (bool, error) {

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --ci-kubernetes-e2e-gci-gce-serial (gce-cos-master-serial) Kubernetes e2e suite.[sig-scheduling] SchedulerPriorities [Serial] Pod should avoid nodes that have avoidPod annotation June 19th 2021 () Testgrid: https://k8s- Prow history: Looks like pods are scheduled where they shouldn't be? : The tests passes in maybe 20% of the builds, but it looks random, so I don't treat it as a flake. At the moment there are also sig-storage tests consistently failing in this job: , They don't look related though. /sig scheduling /kind failing-test
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/milestone v1.22
try to have a look.
try to have a look.
/cc
This is expected. The feature (which was alpha) is now disabled by default. Users can still enable it by switching to a v1beta1 scheduler component config. Let's remove the test.


Question: testJSONPath( []jsonpathTest{ { <del> "when range is used in a certain way in script, additional line is printed", `{range .items[?(.status.phase=="Running")]}{.metadata.name}{" is Runningn"}`, </del> <ins> "range over pods without selecting the last one", `{range .items[?(.status.phase=="Running")]}{.metadata.name}{" is Runningn"}{end}`, </ins> data, "pod1 is Runningnpod2 is Runningnpod3 is Runningn", false, // expect no error

no longer continues parsing an item when a key is encountered that has no value. This appears to be a behavior change introduced in Kubernetes version 1.22. will continue . Run the following commands against a cluster running CoreDNS pods in the namespace: Nothing. Kubernetes version (use ): v1.22.0 Cloud provider or hardware configuration: N/A OS (e.g: ): N/A Kernel (e.g. ): N/A Install tools: N/A Network plugin and version (if this is a network-related bug): N/A Others: N/A
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Debug data for Kubernetes version 1.21 cluster:
Debug data for Kubernetes version 1.22 cluster:
/sig cli
I reproduce the issue in my 1.23 environment. I'll try to fix it. /assign
PR works on it.
/unassign
to ensure I understand your comment correct, should fix this issue?
Yes, I've tested it. can resolve it. The bug is the line:
Yes, I've tested it. can resolve it. The bug is in this line:


Question: } } <ins> func (p *podWorkers) IsPodKnownTerminated(uid types.UID) bool { p.podLock.Lock() defer p.podLock.Unlock() if status, ok := p.podSyncStatuses[uid]; ok { return status.IsTerminated() } // if the pod is not known, we return false (pod worker is not aware of it) return false } </ins> func (p *podWorkers) CouldHaveRunningContainers(uid types.UID) bool { p.podLock.Lock() defer p.podLock.Unlock()

There appears to be a change of behavior / possible regression in k8s 1.22 with respect to pod lifecycle behavior. We've started to see some cases when multiple pods are being created (often as part of a k8s job) the following will happen: pods get created by e.g. job controller (e.g. say 5). Each pod in job requests majority of node memory so only one such pod can fit on node at a time. schedules [first] pod to node, rest (4 pods) stay in pending accepts first pod, runs it to completion pod get scheduled to the node rejects the pod during pod admission, e.g. with , despite the fact that the first pod has already completed successfully This results in many of pods backing the job getting into Failed phase and having to be recreated by job controller. The job can often get into BackofLimit exceeded to hitting backoffLimit. The root issue seems to be a mismatch of when kubelet actually considers the pod to complete and what it reports to API server (which causes the followup pods to be scheduled to it). I'm still investigating further, but this may be related to the pod lifecycle changes in 1.22 -- I've created a e2e test as minimal repro of this behavior -- (Test fails) 22 build log -- 22 kubelet log -- (Test passes) 21 build log -- 21 kubelet log -- <!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --- Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository.


Question: } } <ins> func (p *podWorkers) IsPodKnownTerminated(uid types.UID) bool { p.podLock.Lock() defer p.podLock.Unlock() if status, ok := p.podSyncStatuses[uid]; ok { return status.IsTerminated() } // if the pod is not known, we return false (pod worker is not aware of it) return false } </ins> func (p *podWorkers) CouldHaveRunningContainers(uid types.UID) bool { p.podLock.Lock() defer p.podLock.Unlock()

</details
/sig node
/cc
Looking at the kubelet log in 1.22 () here's what I see -- First pod starts up... starts dies after ~10 seconds since the job is completed (the container simply runs a ) Pod completes, status is updated to API server Next pod comes in but is rejected with despite the earlier pod already being completed/terminated
did the kubelet pod lifecycle rework result in a reordering of kubelet resource bookeeping w.r.t. pod status? it seems reasonable for the scheduler to send another pod to a kubelet once it reports the pod complete and the use of the resources is done from an API perspective
I am also seeing this same behavior, but with "OutOfPods" rather than "OutOfMemory", when running a job with parallelism=100, completions=400 on GKE (which has a 110 pod per node limit).
this is easily reproducible on a local cluster as well start with this: then create this job: result on 1.22 is the job hangs with 2 completions and a bunch of rejections by the kubelet: on 1.21, the same job completes all pods successfully in a couple minutes with no OutOfpods errors
verified introduced the OutOfPods issue
Shame we didn’t have an e2e which caught this given how easy the reproducer is. This looks to be the opposite of the previous cpu admission bug (ignoring things that are terminating but not terminated). We are weak on admission testing in general. This potentially implies admission is looking at the wrong set of pods (admitted + terminating). Note there is already a bug in 1.21 and before where force deleted pods are incorrectly ignored from the set of allocated resources while they are terminating, but this doesn’t immediately seem to be that issue (since i doubt those jobs are being force deleted)
yeah, there was no force deletion involved in (or any deletion at all, actually... the completed pods just sat there, completed)
The set of admitted pods considered for pod count is everything in the pod manager filtered by filterOutTerminatingPods, which calls CouldHaveRunningContainers which returns true once a pod is seen by the workers (shortly after admission, and until we’ve confirmed via syncTerminatingPod that no containers are running). Whatever the problem is it’s likely in the interaction of that.


Question: } } <ins> func (p *podWorkers) IsPodKnownTerminated(uid types.UID) bool { p.podLock.Lock() defer p.podLock.Unlock() if status, ok := p.podSyncStatuses[uid]; ok { return status.IsTerminated() } // if the pod is not known, we return false (pod worker is not aware of it) return false } </ins> func (p *podWorkers) CouldHaveRunningContainers(uid types.UID) bool { p.podLock.Lock() defer p.podLock.Unlock()

A pod the pod worker hasn’t seen yet is also filtered out (that means bursts of pods can get in, which seems the opposite of this problem, and is the bug i mentioned before).
Edit: one wrong thing spotted while looking - I'm not sure it fixes the issue you're seeing, but it's definitely wrong for admission (admission is filtering the known pod set down to remove pods that are terminated, but shouldn't filter pods that aren't admitted yet). This needs a node e2e reproducer that can run in serial (resource consumption is really hard to test from parallel, is slow in serial, and node e2e is back in a good state now) - / is that something either of you could help me with? I believe the reason the existing tests don't catch this is the job controller is creating pods fast enough that we see races in admission (multiple pods are admitted in one call to HandlePodAdditions) that are not caught in the current serial node e2e admission tests which rarely create more than one at a time.
Job pods still go through the scheduler to get assigned to a node, so even though the job controller can create pods in parallel, I'd still expect them to arrive at the node in an orderly fashion. It's still possible multiple arrive in a single HandlePodAdditions call, of course. It seems like kubelet admission is using laggy information that doesn't reflect pod completion the kubelet itself wrote to the pod API.
The bug turned out to be much simpler. Pods that are completed never go to the pod worker, so once a pod has reached terminal state it was being back to the list of active pods. The set of e2es that would have prevented this regression: node e2e that creates multiple pods at the same time and verifies that admission works correctly when HandlePodAdditions takes two pods at once node e2e that verifies that completed pods don't consume resources for admission purposes in the kubelet job controller e2e that attempts to verify some part of this reliably (ideally in a way that can be run in parallel e2e)
another related issue:
Linking here for completeness - a similar symptom might get exposed after this fix as described in The kubelet considers resources for terminating pods to be in use (they are!), but the scheduler ignores terminating pods and schedules new pods.


Question: } } <ins> func (p *podWorkers) IsPodKnownTerminated(uid types.UID) bool { p.podLock.Lock() defer p.podLock.Unlock() if status, ok := p.podSyncStatuses[uid]; ok { return status.IsTerminated() } // if the pod is not known, we return false (pod worker is not aware of it) return false } </ins> func (p *podWorkers) CouldHaveRunningContainers(uid types.UID) bool { p.podLock.Lock() defer p.podLock.Unlock()

Because the kubelet now considers terminating pods, it rejects those rapidly rescheduled pods. The follow up is more involved but requires the kubelet to allow pods to be potentially admitted that can't be run currently due to those terminating pods. See the linked issue.


Question: ) func TestCleanup(t *testing.T) { <ins> // The control plane is noisy and randomly logs through klog, for example: // E0912 07:08:46.100164   75466 controller.go:254] unable to sync kubernetes service: Endpoints "kubernetes" is invalid: subsets[0].addresses[0].ip: Invalid value: "127.0.0.1": may not be in the loopback range (127.0.0.0/8, ::1/128) // // By creating a ktesting logger and registering that as global // default logger we get the control plane output into the // "go test" output in case of a failure (useful for debugging!) // while keeping it out of the captured Ginkgo output that // the test is comparing below. // // There are some small drawbacks: // - The source code location for control plane log messages //   is shown as klog.go because klog does not properly //   skip its own helper functions. That's okay, normally //   ktesting should not be installed as logging backend like this. // - klog.Infof messages are printed with an extra newline. logger, _ := ktesting.NewTestContext(t) klog.SetLogger(logger) </ins> apiServer := testapiserver.StartAPITestServer(t) // This simulates how test/e2e uses the framework and how users

pull-kubernetes-unit Probably since it was merged in The test checks output from the E2E framework and sometimes some extra log output shows up that isn't expected: No response /sig testing
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign


Question: <ins> check process kube-proxy with pidfile /var/run/kube-proxy.pid group kube-proxy start program = "/etc/init.d/kube-proxy start" stop program = "/etc/init.d/kube-proxy stop" if does not exist then restart if failed port 10249 protocol HTTP request "/healthz" with timeout 10 seconds then restart </ins>

We already "healthcheck" kubelet and docker periodically, but miss one for kube-proxy.
It would make sense to do (only bind to localhost) at the same time we add the healthchecking configuration.


Question: } } for _, c := range pod.Spec.Containers { <del> if c.SecurityContext == nil { </del> <ins> if c.SecurityContext == nil || c.SecurityContext.Privileged == nil { </ins> continue } if *c.SecurityContext.Privileged {

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): No What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): panic apiserver invalid memory address Is this a BUG REPORT or FEATURE REQUEST? (choose one): Bug Report <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Environment: Cloud provider or hardware configuration: aws OS (e.g. from /etc/os-release): Ubuntu 16.04 Kernel (e.g. ): Linux ip-10-50-193-54 4.4.0-59-generic -Ubuntu SMP Fri Jan 6 17:47:47 UTC 2017 x8664 x8664 x86_64 GNU/Linux Install tools: Others: What happened: Tried to exec: . Got back: apiserver logs: It only seems to happen on this one pod - other pods are fine. What you expected to happen: should succeed or at least give a sensible error message. How to reproduce it (as minimally and precisely as possible): Anything else do we need to know:
There are no errors on the node with the pod. I also tried kubectl v1.4.8 and same problem.
It looks like your pod that's having the issue has a container with a security context, but privileged appears to be nil. Can you confirm that is the case?
Yep that's it. It seems when securityContext is specified without setting , it isn't set correctly to its default of false. e.g. the pod yaml has: Explicitly adding allows exec to succeed.
Ah, this was caused by the kubelet using versioned APIs - for his awareness. Basically any code that depends on nil pointers being set is vulnerable.