Question: } apiGroupList, resourcesByGV, failedGVs = SplitGroupsAndResources(aggregatedDiscovery) default: <del> return nil, nil, nil, fmt.Errorf("Unknown discovery response content-type: %s", responseContentType) </del> <ins> // Default is unaggregated discovery v1. err = json.Unmarshal(body, apiGroupList) if err != nil { return nil, nil, nil, err } </ins> } return apiGroupList, resourcesByGV, failedGVs, nil } <ins> // isV2Beta1ContentType checks of the content-type string is both // "application/json" and contains the v2beta1 content-type params. // NOTE: This function is resilient to the ordering of the // content-type parameters, as well as parameters added by // intermediaries such as proxies or gateways. Examples: // //	"application/json; g=apidiscovery.k8s.io;v=v2beta1;as=APIGroupDiscoveryList" = true //	"application/json; as=APIGroupDiscoveryList;v=v2beta1;g=apidiscovery.k8s.io" = true //	"application/json; as=APIGroupDiscoveryList;v=v2beta1;g=apidiscovery.k8s.io;charset=utf-8" = true //	"application/json" = false //	"application/json; charset=UTF-8" = false func isV2Beta1ContentType(contentType string) bool { base, params, err := mime.ParseMediaType(contentType) if err != nil { return false } return runtime.ContentTypeJSON == base && params["g"] == "apidiscovery.k8s.io" && params["v"] == "v2beta1" && params["as"] == "APIGroupDiscoveryList" } </ins> // ServerGroups returns the supported groups, with information like supported versions and the // preferred version. func (d *DiscoveryClient) ServerGroups() (*metav1.APIGroupList, error) {

We are using (and ) to interact with Kubernetes clusters behind a couple of proxy layers. One of these proxies completes the headers to include character encoding details. This was working fine until recently, when client-go/kubectl have started to fail when interacting with the discovery API. It seems that a recent change has made kubectl more restrictive of the content-type that it expects. I suspect that it is related to this commit: For example, this is an error from a kubectl command: We can of course try to fix it on our end, removing that charset information. However, it is a valid content-type and the way kubectl is handling the content-type there seems like it is relying on implementation details of kube-apiserver (that it does not add metadata to the content-type). I propose to fix this issue in , by properly parsing and handling the content-type.
cc /kind regression /kind bug /sig api-machinery /priority important-soon looks like the switch statement in downloadLegacy() needs to be more tolerant of the variety of content-types it can get from a server (including text/plain or no content-type if old servers could have returned that...)
/transfer kubernetes/kubernetes
Something went wrong or the destination repo kubernetes/kubernetes/kubernetes does not exist. <detailsIn response to : /transfer kubernetes/kubernetes Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/transfer kubernetes
Yep, the string comparison for the content-type is flimsy, let's address that. Thanks for the report!
/assign
/triage accepted
/cc
The fix is being cherry-picked back to: : :


Question: return machine, exists, err } <ins> // VirtualMachineClientGetWithRetry invokes az.VirtualMachinesClient.Get with exponential backoff retry func (az *Cloud) VirtualMachineClientGetWithRetry(resourceGroup, vmName string, types compute.InstanceViewTypes) (compute.VirtualMachine, error) { var machine compute.VirtualMachine err := wait.ExponentialBackoff(az.resourceRequestBackoff, func() (bool, error) { var retryErr error machine, retryErr = az.VirtualMachinesClient.Get(resourceGroup, vmName, types) if retryErr != nil { glog.Errorf("backoff: failure, will retry,err=%v", retryErr) return false, nil } glog.V(2).Infof("backoff: success") return true, nil }) return machine, err } // GetIPForMachineWithRetry invokes az.getIPForMachine with exponential backoff retry func (az *Cloud) GetIPForMachineWithRetry(name types.NodeName) (string, error) { var ip string err := wait.ExponentialBackoff(az.resourceRequestBackoff, func() (bool, error) { var retryErr error ip, retryErr = az.getIPForMachine(name) if retryErr != nil { glog.Errorf("backoff: failure, will retry,err=%v", retryErr) return false, nil } glog.V(2).Infof("backoff: success") return true, nil }) return ip, err } </ins> // CreateOrUpdateSGWithRetry invokes az.SecurityGroupsClient.CreateOrUpdate with exponential backoff retry func (az *Cloud) CreateOrUpdateSGWithRetry(sg network.SecurityGroup) error { return wait.ExponentialBackoff(az.resourceRequestBackoff, func() (bool, error) {

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind feature What happened: In certain large cluster scenarios we would like to anticipate kubelet behavior hitting request limits, and backoff in response. E.g., What you expected to happen: This is a request for add'l backoff coverage, so nothing surprising. How to reproduce it (as minimally and precisely as possible): Build a cluster with as many nodes as your subscription will support on azure until kubelet behavior contributes to backend API request limits. Environment: Kubernetes version (use ): and Cloud provider or hardware configuration**: azure OS (e.g. from /etc/os-release): Ubuntu 16.04.2 LTS (Xenial Xerus) Kernel (e.g. ): Linux k8s-pool7--7 4.4.0-83-generic -Ubuntu SMP Mon Jun 26 17:54:43 UTC 2017 x8664 x8664 x86_64 GNU/Linux Install tools: acs-engine Others:
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig azure


Question: <ins> # 3.1 * The pause container gains a signal handler to clean up orphaned zombie processes. ([#36853](https://prs.k8s.io/36853), [@verb](https://github.com/verb)) * `pause -v` will return build information for the pause binary. ([#56762](https://prs.k8s.io/56762), [@verb](https://github.com/verb)) # 3.0 * The pause container was rewritten entirely in C. ([#23009](https://prs.k8s.io/23009), [@uluyol](https://github.com/uluyol)) </ins>

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: the pause-amd64:3.0 image on has an empty schema2 manifest, which prevents is from being pulled by proxy/caches which only support V2 registries. What you expected to happen: Being able to pull ` using a V2 manifest compatible client. Anything else we need to know?: The proxy/cahce is requesting this
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
After talking with , he mentionned the following
AFAIK, does not support well. After inspecting the image, you can find is still using manifest v1.
/cc FYI
Im not sure which exact version x schéma X is needed. What I know is that we can download through the proxy/cache all other Kubernetes kube-system images, only the pause image is giving us trouble. Therefore, I would say that having the image available using the same format as the other kube-system images would be the request here.
Yeah, we haven't updated the pause image in a while, since it doesn't really ever change. I'm not sure whether rebuilding + pushing over the existing tag is safe, or if we should bump to or something. thoughts?
If we do rebuilding/updating the image, that would help and a bunch of similar issues.
it's better/save to go with new tag
I didn't realize there were changes to the pause binary that were never actually deployed. It seems like we should probably just do that (and bump the tag).
that sounds great. Any idea when the existing 3.0 tag could be pushed again?
We shouldn't push over the 3.0 tag; we should push a 3.1 or 4.0 tag instead. I'm not sure who owns the pause container who should drive this. Maybe sig-node and/or
It seems like it should be sig-node. I'll look into it.
The new container has been built, can you test with
Its working! thanks! I was able to pull it through our artifactory proxy , Lee Verberne <:
excellent, thanks for checking. /assign


Question: <ins> # 3.1 * The pause container gains a signal handler to clean up orphaned zombie processes. ([#36853](https://prs.k8s.io/36853), [@verb](https://github.com/verb)) * `pause -v` will return build information for the pause binary. ([#56762](https://prs.k8s.io/56762), [@verb](https://github.com/verb)) # 3.0 * The pause container was rewritten entirely in C. ([#23009](https://prs.k8s.io/23009), [@uluyol](https://github.com/uluyol)) </ins>

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: pause does not reap zombie processes using the default pause container provided by What you expected to happen: should include the up to date pause code to reap zombie processes when using docker = 1.13.1 found here How to reproduce it (as minimally and precisely as possible): a.out will leave a zombie process that is not cleaned up Anything else we need to know?: Building the pause container from master solves this issue Related: Environment: Kubernetes version (use ): 1.7.0+ Cloud provider or hardware configuration**: N/A OS (e.g. from /etc/os-release): CentOS 7 Kernel (e.g. ): 4.10.13+ Install tools: Others: docker = 1.13.1
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
i get a simliar problem when i run crond service in pod container, every day it makes some [anacron] zombie progresses UID PID PPID C STIME TTY TIME CMD root 1 0 0 Nov14 ? 00:00:00 /pause root 7 0 0 Nov14 ? 00:00:00 /usr/local/bin/dumb-init /usr/lo root 13 7 0 Nov14 ? 00:00:00 nginx: master process /usr/local nobody 21 13 0 Nov14 ? 00:00:01 nginx: worker process nobody 22 13 0 Nov14 ? 00:00:07 nginx: worker process nobody 23 13 0 Nov14 ? 00:00:07 nginx: worker process nobody 24 13 0 Nov14 ? 00:00:05 nginx: worker process root 77 1 0 Nov14 ? 00:00:00 [anacron] root 1 0 00:01 ? 00:00:00 [anacron] <defunctroot 1 0 01:01 ? 00:00:00 [anacron] <defunctroot 1 0 02:01 ?


Question: <ins> # 3.1 * The pause container gains a signal handler to clean up orphaned zombie processes. ([#36853](https://prs.k8s.io/36853), [@verb](https://github.com/verb)) * `pause -v` will return build information for the pause binary. ([#56762](https://prs.k8s.io/56762), [@verb](https://github.com/verb)) # 3.0 * The pause container was rewritten entirely in C. ([#23009](https://prs.k8s.io/23009), [@uluyol](https://github.com/uluyol)) </ins>

00:00:00 [anacron] <defunctroot 1 0 03:01 ? 00:00:00 [anacron] <defunctroot 0 0 11:49 ? 00:00:00 /bin/bash root 0 1 19:15 ? 00:00:00 /bin/bash root 0 19:15 ? 00:00:00 ps -ef !
Do we know why this was never resolved / was never deployed?
I just fought through this issue with a rabbitmq helm chart. I had to custom build a pause container from master to fix it. The rabbitmq helm chart uses for its liveness checks. It's a thin wrapper around a call to the rabbitmq-server daemon, which is not PID 1 (/pause is PID 1). The rabbitmq-server process forks some children to do the work and then they exit. Since the pause-amd64:3.0 is not reaping, those processes stick around as zombies. The liveness check runs every 5 seconds. After about 4 hours, there are about 27K zombies and the node begins failing to launch new containers because it's starved for resources. The pause container built from current master fixes that behavior, no zombies at all in the container.
The plan is to build a new version of pause-amd64 and then update the image referenced by Kubernetes. This means the default image will change in the next Kubernetes release. We can make the new image available once the freeze lifts and we can merge .
Looks like this made it into 1.10


Question: Spec: api.ServiceSpec{ Ports: []api.ServicePort{ { <del> Name:     "default", Protocol: api.Protocol("UDP"), Port:     14, </del> <ins> Name:       "default", Protocol:   api.Protocol("UDP"), Port:       14, TargetPort: util.NewIntOrStringFromInt(14), </ins> }, }, Selector: map[string]string{"func": "stream"}, Type:     api.ServiceTypeLoadBalancer, }, }, <del> expected: "you will also need to explicitly open a firewall", status:   200, </del> <ins> status: 200, </ins> }, }

This was broken by , and therefore broken in as well. Any fix should be offered as a cherry pick so "make release" is clean there.
I went ahead and worked something up for this this morning, to get Shippable builds back on the rails.


Question: func TestSecretForDockerRegistryGenerate(t *testing.T) { username, password, email, server := "test-user", "test-password", "test-user@example.org", "https://index.docker.io/v1/" <del> secretData, err := handleDockercfgContent(username, password, email, server) </del> <ins> secretData, err := handleDockerCfgJsonContent(username, password, email, server) </ins> if err != nil { t.Errorf("unexpected error: %v", err) } <del> secretDataNoEmail, err := handleDockercfgContent(username, password, "", server) </del> <ins> secretDataNoEmail, err := handleDockerCfgJsonContent(username, password, "", server) </ins> if err != nil { t.Errorf("unexpected error: %v", err) }

/kind bug We're having trouble retrieving images from a private registry on DockerCloud. For nearly a year the following has worked flawlessly. Last Friday I pulled down the 1.9.0 CLI (via brew) and since then have not been able to access any image in a private registry on DockerCloud. I've tried this on 10+ brand new clusters created with kops v1.8.0. The Kubernetes cluster version have been 1.8.0, 1.8.4, 1.8.5 and 1.9.0 (which is not supported by kops yet) and all exhibit the same problem. We create the secret as such: And verify it using: Which yields: And create a test service with with these contents: The pod fails immediatly with: I've reviewed the 1.9.0 release notes and nothing jumps out as the root cause. For reference I've tried putting quotes around the value when creating the secret to no avail. I've also tried prefixing the value with , so the full value would read: also to no avail. Environment: Kubernetes version: Cloud provider and installation info: AWS cluster built with Kops 1.8.0, using (to isolate this issue) nearly the default settings for everything.
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I have same problem, and I "solved" it creating a docker-registry secret in an old kubernetes version and pasting it on new kubernetes
I'm recently (I know it worked perfectly about last month) facing issues with as well. For me it ended in a similar manner saying: I was able to confirm, that this is a kubernetes CLI related issue and not end user. Based on and this I was able to create properly formatted auth information looking like this: . When encoding this one-liner using (MacOS workstation) and apply like this works fine and images can be pulled from private repository again.


Question: func TestSecretForDockerRegistryGenerate(t *testing.T) { username, password, email, server := "test-user", "test-password", "test-user@example.org", "https://index.docker.io/v1/" <del> secretData, err := handleDockercfgContent(username, password, email, server) </del> <ins> secretData, err := handleDockerCfgJsonContent(username, password, email, server) </ins> if err != nil { t.Errorf("unexpected error: %v", err) } <del> secretDataNoEmail, err := handleDockercfgContent(username, password, "", server) </del> <ins> secretDataNoEmail, err := handleDockerCfgJsonContent(username, password, "", server) </ins> if err != nil { t.Errorf("unexpected error: %v", err) }

However this breaks all automation and can't be considered as solution Confirmed working with kubectl 1.8.5 and server 1.8.4 (guess 1.7.9 as well)
looks like there's an extra "auths" element when adding the secret using 1.9 kubectl
Added in /assign
Many thanks for the quick fix on this. Any idea when this might get rolled out? Also, it might be a good idea if this was to the known issue list for v1.9.0. It would have saved us a bunch of time trying to track down the root cause. Just trying to make sure no one else needs to do the same.
linking to the "v1.9.0 known issues" list
ack, asking Mehdy since he is the patch manager for v1.9 branch. Mehdy, any idea when we can ship 1.9.1 with this fix?
workarounds for folks until 1.9.1 is released: if you have a .dockerconfigjson for your private registry already, you can manually specify the type and data key (e.g. ) if you don't have a .dockerconfigjson file already, you can fix up the secret produced by manually: the type from to the data key from to the modified secret with
1 will be released tomorrow
I can confirm solution works, thanks so much!
I am still having issues in 1.9.1. This worked in the 1.8.x branches: results in the following error:
you have to remove the gitlab-registry secret before you can apply an update... type is immutable, and the fix in 1.9.1 required changing the type of the secret


Question: } require.NoError(t, registerAPIService(ctx, client, groupVersion, service)) <del> defer func() { require.NoError(t, unregisterAPIService(ctx, client, groupVersion)) }() </del> <ins> groupVersions = append(groupVersions, groupVersion) </ins> } // Keep repeatedly fetching document from aggregator. // Check to see if it contains our service within a reasonable amount of time require.NoError(t, WaitForGroups(ctx, client, basicTestGroupWithFixup)) <ins> require.NoError(t, WaitForRootPaths(t, ctx, client, sets.New("/apis/"+basicTestGroup.Name), nil)) // Unregister and ensure the group gets dropped from root paths for _, groupVersion := range groupVersions { require.NoError(t, unregisterAPIService(ctx, client, groupVersion)) } require.NoError(t, WaitForRootPaths(t, ctx, client, nil, sets.New("/apis/"+basicTestGroup.Name))) </ins> } func runTestCases(t *testing.T, cases []testCase) {

reported that conformance testing with the latest version of sonobuoy against a version 1.29.0-alpha.3 cluster fails on the support the 1.17 Sample API Server using the current Aggregator test: , related to I found the reason why the test is working fine in Kubernetes CI jobs but fails with sonobuoy is because the service account used by sonobuoy "sonobuoy-serviceaccount" doesn't have read permission of the non-resource url "/", which is used by the test to confirm the removal of group path: The permission of "sonobuoy-serviceaccount" is as below: There may be two ways to fix it: the RBAC rule of sonobuoy to include "get" of "/". However, could other test runners encounter this issue? Is there a doc listing the required permissions for running e2e tests? the e2e to skip checking the removal of the group path if the service account doesn't have permission to access the root path. Which way do we usually prefer? Running e2e with sonobuoy should succeed. a v1.29.0-alpha.3 cluster. No response <details</details<details</details<details</details<details</details<details</details<details</details
implemented this solution.
/sig api-machinery
thank you for opening this issue and working on a solution. I'm probably not the right person to decide on which solution is preferred. From my perspective, either solution is fine.
Note: Sonobuoy is not owned by any Kubernetes SIG. At most it is between SIG Architecture and the CNCF to choose whether to recommend it for running tests. SIG Testing owns the e2e image that sonobuoy can be used to schedule but not sonobuoy. IMHO this is a bug in running the tests via a pod, which is not how we develop the tests (they're run outside the cluster, by a cluster admin account typically). I would say 1) is the correct answer. xref: /sig architecture /sig testing /triage accepted cc
thanks for your comment. I guess 1) means non of the existing sonobuoy versions can pass K8s conformance test against a cluster 1.29.0-alpha3, as the RBAC rule is hardcoded.


Question: } require.NoError(t, registerAPIService(ctx, client, groupVersion, service)) <del> defer func() { require.NoError(t, unregisterAPIService(ctx, client, groupVersion)) }() </del> <ins> groupVersions = append(groupVersions, groupVersion) </ins> } // Keep repeatedly fetching document from aggregator. // Check to see if it contains our service within a reasonable amount of time require.NoError(t, WaitForGroups(ctx, client, basicTestGroupWithFixup)) <ins> require.NoError(t, WaitForRootPaths(t, ctx, client, sets.New("/apis/"+basicTestGroup.Name), nil)) // Unregister and ensure the group gets dropped from root paths for _, groupVersion := range groupVersions { require.NoError(t, unregisterAPIService(ctx, client, groupVersion)) } require.NoError(t, WaitForRootPaths(t, ctx, client, nil, sets.New("/apis/"+basicTestGroup.Name))) </ins> } func runTestCases(t *testing.T, cases []testCase) {

I will file an issue to sonobuoy first.
question here is what are the set of client permissions required to run the conformance suite? perhaps a level of access or a more granular recommendation should be documented. what is your take on the root path requirement as approver for: EDIT: slack thread:
I agree, we already require e.g. privileged pods for a long time in conformance tests and we should make it clear that we expect a highly privileged client.
+1 for what is saying, that we should expect that the user running the tests has required permissions. These tests are not built for being run unprivileged.
i wanted to PR the admin note in our conformance docs under sig-arch, but then i saw this: so broke this guarantee. sounds like SIG Arch needs to comment.
This statement surprises me. Root discovery is one of the least-privileged permissions you can have on a cluster. For years this was even open for unauthorized users on the internet (if your api endpoint has been exposed). In other words, just add the permission. While thinking about it, it is surprising that the user in question does not have this access already. Don't we allow that for every authorized user?
In contrast, I hope that lots of privileged operations are tested by conformance today. If we don't test things like admission webhooks for example, we have much bigger problems. And admission webhooks are basically equivalent to cluster-admin.
that is outdated, we use hostNetwork pods one some of the tests IIRC
looks like we don't. before this ticket, i was under the impression that the conformance suite already requires admin and that all testers (like Sonobuoy) use admin level credentials. it is non-privileged (e.g., does not require root on nodes, access to raw network interfaces, or cluster admin permissions) looks like we may have to change that sentence completely.
Would have expected it here: The PR in question explicitly fixes the result returned by . If that is part of conformance (IMO it should), every conformance test needs access it obviously. Whether we require admin, or we add it manually, either would work.


Question: } require.NoError(t, registerAPIService(ctx, client, groupVersion, service)) <del> defer func() { require.NoError(t, unregisterAPIService(ctx, client, groupVersion)) }() </del> <ins> groupVersions = append(groupVersions, groupVersion) </ins> } // Keep repeatedly fetching document from aggregator. // Check to see if it contains our service within a reasonable amount of time require.NoError(t, WaitForGroups(ctx, client, basicTestGroupWithFixup)) <ins> require.NoError(t, WaitForRootPaths(t, ctx, client, sets.New("/apis/"+basicTestGroup.Name), nil)) // Unregister and ensure the group gets dropped from root paths for _, groupVersion := range groupVersions { require.NoError(t, unregisterAPIService(ctx, client, groupVersion)) } require.NoError(t, WaitForRootPaths(t, ctx, client, nil, sets.New("/apis/"+basicTestGroup.Name))) </ins> } func runTestCases(t *testing.T, cases []testCase) {

adding to milestone and making release blocking ... we should not release 1.29 in a state that the most commonly used conformance tool will fail
IMO we should clarify the state and go fix the tool, but we should keep the test and we should NOT add skip. Conformance docs are not a release artifact, and sonobuoy release lifecycle is not a conformance artifact. If we can't get sonobuoy working in a reasonable time, we already know the tests can be run by scheduling the image (which is ours) directly with kubectl instead and we can document that in the upstream conformance docs. We need to fix the for this image: ,%20master%20(dev) And it should probably be to release-blocking in the future.
it looks like the e2e change made in was not consciously made part of conformance, it was just trying to functionally test the bugfix being made. For 1.29, I think we should revert the change to and make an integration test that validates the fix instead. We can revisit whether content of is part of conformance or not for 1.30
I opened to revert the e2e change and move the functional test to an integration test
Filed to try to get the image job passing again.
We probably need another issue to track 1.30 follow-up?
If no one has done it, I could create another issue and follow up with the items that have been discussed here: outdated privilege requirement in commonly used conformance tools like sonobuoy aware of the required priviledge to avoid similar situations in the future. permission of GET "/" to authorized users via the role for root discovery.


Question: func TestCheckVolumeFSResize(t *testing.T) { mode := v1.PersistentVolumeFilesystem <del> pv := &v1.PersistentVolume{ ObjectMeta: metav1.ObjectMeta{ Name: "dswp-test-volume-name", </del> <ins> setCapacity := func(pv *v1.PersistentVolume, pvc *v1.PersistentVolumeClaim, capacity int) { pv.Spec.Capacity = volumeCapacity(capacity) pvc.Spec.Resources.Requests = volumeCapacity(capacity) } testcases := []struct { resize       func(*testing.T, *v1.PersistentVolume, *v1.PersistentVolumeClaim, *desiredStateOfWorldPopulator) verify       func(*testing.T, []v1.UniqueVolumeName, v1.UniqueVolumeName) enableResize bool readOnlyVol  bool }{ { // No resize request for volume, volumes in ASW shouldn't be marked as fsResizeRequired resize: func(*testing.T, *v1.PersistentVolume, *v1.PersistentVolumeClaim, *desiredStateOfWorldPopulator) { }, verify: func(t *testing.T, vols []v1.UniqueVolumeName, _ v1.UniqueVolumeName) { if len(vols) > 0 { t.Errorf("No resize request for any volumes, but found resize required volumes in ASW: %v", vols) } }, enableResize: true, </ins> }, <del> Spec: v1.PersistentVolumeSpec{ PersistentVolumeSource: v1.PersistentVolumeSource{RBD: &v1.RBDPersistentVolumeSource{}}, Capacity:               volumeCapacity(1), ClaimRef:               &v1.ObjectReference{Namespace: "ns", Name: "file-bound"}, VolumeMode:             &mode, </del> <ins> { // Disable the feature gate, so volume shouldn't be marked as fsResizeRequired resize: func(_ *testing.T, pv *v1.PersistentVolume, pvc *v1.PersistentVolumeClaim, _ *desiredStateOfWorldPopulator) { setCapacity(pv, pvc, 2) }, verify: func(t *testing.T, vols []v1.UniqueVolumeName, _ v1.UniqueVolumeName) { if len(vols) > 0 { t.Errorf("Feature gate disabled, but found resize required volumes in ASW: %v", vols) } }, enableResize: false, </ins> }, <del> } pvc := &v1.PersistentVolumeClaim{ Spec: v1.PersistentVolumeClaimSpec{ VolumeName: "dswp-test-volume-name", Resources: v1.ResourceRequirements{ Requests: volumeCapacity(1), </del> <ins> { // Make volume used as ReadOnly, so volume shouldn't be marked as fsResizeRequired resize: func(_ *testing.T, pv *v1.PersistentVolume, pvc *v1.PersistentVolumeClaim, _ *desiredStateOfWorldPopulator) { setCapacity(pv, pvc, 2) </ins> }, <ins> verify: func(t *testing.T, vols []v1.UniqueVolumeName, _ v1.UniqueVolumeName) { if len(vols) > 0 { t.Errorf("volume mounted as ReadOnly, but found resize required volumes in ASW: %v", vols) } }, readOnlyVol:  true, enableResize: true, </ins> }, <del> Status: v1.PersistentVolumeClaimStatus{ Phase:    v1.ClaimBound, Capacity: volumeCapacity(1), </del> <ins> { // Clear ASW, so volume shouldn't be marked as fsResizeRequired because they are not mounted resize: func(_ *testing.T, pv *v1.PersistentVolume, pvc *v1.PersistentVolumeClaim, dswp *desiredStateOfWorldPopulator) { setCapacity(pv, pvc, 2) clearASW(dswp.actualStateOfWorld, dswp.desiredStateOfWorld, t) }, verify: func(t *testing.T, vols []v1.UniqueVolumeName, _ v1.UniqueVolumeName) { if len(vols) > 0 { t.Errorf("volume hasn't been mounted, but found resize required volumes in ASW: %v", vols) } }, enableResize: true, </ins> }, <del> } dswp, fakePodManager, fakeDSW := createDswpWithVolume(t, pv, pvc) fakeASW := dswp.actualStateOfWorld // create pod containers := []v1.Container{ </del> { <del> VolumeMounts: []v1.VolumeMount{ { Name:      "dswp-test-volume-name", MountPath: "/mnt", }, </del> <ins> // volume in ASW should be marked as fsResizeRequired resize: func(_ *testing.T, pv *v1.PersistentVolume, pvc *v1.PersistentVolumeClaim, _ *desiredStateOfWorldPopulator) { setCapacity(pv, pvc, 2) }, verify: func(t *testing.T, vols []v1.UniqueVolumeName, volName v1.UniqueVolumeName) { if len(vols) == 0 { t.Fatalf("Request resize for volume, but volume in ASW hasn't been marked as fsResizeRequired") } if len(vols) != 1 { t.Errorf("Some unexpected volumes are marked as fsResizeRequired: %v", vols) } if vols[0] != volName { t.Fatalf("Mark wrong volume as fsResizeRequired: %s", vols[0]) } </ins> }, <ins> enableResize: true, </ins> }, } <del> pod := createPodWithVolume("dswp-test-pod", "dswp-test-volume-name", "file-bound", containers) uniquePodName := types.UniquePodName(pod.UID) uniqueVolumeName := v1.UniqueVolumeName("fake-plugin/" + pod.Spec.Volumes[0].Name) fakePodManager.AddPod(pod) // Fill the dsw to contains volumes and pods. dswp.findAndAddNewPods() reconcileASW(fakeASW, fakeDSW, t) </del> <ins> for _, tc := range testcases { pv := &v1.PersistentVolume{ ObjectMeta: metav1.ObjectMeta{ Name: "dswp-test-volume-name", }, Spec: v1.PersistentVolumeSpec{ PersistentVolumeSource: v1.PersistentVolumeSource{RBD: &v1.RBDPersistentVolumeSource{}}, Capacity:               volumeCapacity(1), ClaimRef:               &v1.ObjectReference{Namespace: "ns", Name: "file-bound"}, VolumeMode:             &mode, }, } pvc := &v1.PersistentVolumeClaim{ Spec: v1.PersistentVolumeClaimSpec{ VolumeName: pv.Name, Resources: v1.ResourceRequirements{ Requests: pv.Spec.Capacity, }, }, Status: v1.PersistentVolumeClaimStatus{ Phase:    v1.ClaimBound, Capacity: pv.Spec.Capacity, }, } </ins> <del> // No resize request for volume, volumes in ASW shouldn't be marked as fsResizeRequired. defer featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, features.ExpandInUsePersistentVolumes, true)() resizeRequiredVolumes := reprocess(dswp, uniquePodName, fakeDSW, fakeASW) if len(resizeRequiredVolumes) > 0 { t.Fatalf("No resize request for any volumes, but found resize required volumes in ASW: %v", resizeRequiredVolumes) } </del> <ins> dswp, fakePodManager, fakeDSW := createDswpWithVolume(t, pv, pvc) fakeASW := dswp.actualStateOfWorld // create pod containers := []v1.Container{ { VolumeMounts: []v1.VolumeMount{ { Name:      pv.Name, MountPath: "/mnt", ReadOnly:  tc.readOnlyVol, }, }, }, } pod := createPodWithVolume("dswp-test-pod", "dswp-test-volume-name", "file-bound", containers) uniquePodName := types.UniquePodName(pod.UID) uniqueVolumeName := v1.UniqueVolumeName("fake-plugin/" + pod.Spec.Volumes[0].Name) </ins> <del> // Add a resize request to volume. pv.Spec.Capacity = volumeCapacity(2) pvc.Spec.Resources.Requests = volumeCapacity(2) </del> <ins> fakePodManager.AddPod(pod) // Fill the dsw to contains volumes and pods. dswp.findAndAddNewPods() reconcileASW(fakeASW, fakeDSW, t) </ins> <del> // Disable the feature gate, so volume shouldn't be marked as fsResizeRequired. defer featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, features.ExpandInUsePersistentVolumes, false)() resizeRequiredVolumes = reprocess(dswp, uniquePodName, fakeDSW, fakeASW) if len(resizeRequiredVolumes) > 0 { t.Fatalf("Feature gate disabled, but found resize required volumes in ASW: %v", resizeRequiredVolumes) } </del> <ins> func() { defer featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, features.ExpandInUsePersistentVolumes, tc.enableResize)() </ins> <del> // Make volume used as ReadOnly, so volume shouldn't be marked as fsResizeRequired. defer featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, features.ExpandInUsePersistentVolumes, true)() pod.Spec.Containers[0].VolumeMounts[0].ReadOnly = true resizeRequiredVolumes = reprocess(dswp, uniquePodName, fakeDSW, fakeASW) if len(resizeRequiredVolumes) > 0 { t.Fatalf("volume mounted as ReadOnly, but found resize required volumes in ASW: %v", resizeRequiredVolumes) } </del> <ins> tc.resize(t, pv, pvc, dswp) </ins> <del> // Clear ASW, so volume shouldn't be marked as fsResizeRequired because they are not mounted. pod.Spec.Containers[0].VolumeMounts[0].ReadOnly = false clearASW(fakeASW, fakeDSW, t) resizeRequiredVolumes = reprocess(dswp, uniquePodName, fakeDSW, fakeASW) if len(resizeRequiredVolumes) > 0 { t.Fatalf("volume hasn't been mounted, but found resize required volumes in ASW: %v", resizeRequiredVolumes) } </del> <ins> resizeRequiredVolumes := reprocess(dswp, uniquePodName, fakeDSW, fakeASW) </ins> <del> // volume in ASW should be marked as fsResizeRequired. reconcileASW(fakeASW, fakeDSW, t) resizeRequiredVolumes = reprocess(dswp, uniquePodName, fakeDSW, fakeASW) if len(resizeRequiredVolumes) == 0 { t.Fatalf("Request resize for volume, but volume in ASW hasn't been marked as fsResizeRequired") } if len(resizeRequiredVolumes) != 1 { t.Fatalf("Some unexpected volumes are marked as fsResizeRequired: %v", resizeRequiredVolumes) } if resizeRequiredVolumes[0] != uniqueVolumeName { t.Fatalf("Mark wrong volume as fsResizeRequired: %s", resizeRequiredVolumes[0]) </del> <ins> tc.verify(t, resizeRequiredVolumes, uniqueVolumeName) }() </ins> } }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: These test cases are very hard to follow because they keep toggling the feature gates on and off, and subsequent test cases depend on the behavior of the previous. <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
/assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle-stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-lifecycle rotten
/help
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hey, I'd like to take this if nobody is working on it
Hi thanks for volunteering! Let us know if you have any questions


Question: MINION_TAG="${INSTANCE_PREFIX}-minion" MINION_NAMES=($(eval echo ${INSTANCE_PREFIX}-minion-{1..${NUM_MINIONS}})) MINION_IP_RANGES=($(eval echo "10.245.{1..${NUM_MINIONS}}.0/24")) <del> MINION_SCOPES="" </del> <ins> MINION_SCOPES="storage-ro,compute-rw" </ins> # Increase the sleep interval value if concerned about API rate limits. 3, in seconds, is the default. POLL_SLEEP_INTERVAL=3 PORTAL_NET="10.0.0.0/16"

07-02 and 13-02 No response No response /sig
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scalability
I might be related to the npd.
The same test failures happening again on 25.02:
/sig node
/sig node /cc The restart of caused the flake. I also opened to track it in NPD repo.
I missed this log. It is caused by oom. Adding some memory may help. I submitted to fix it. Does mean killing metrics-server?
/remove-sig node Sorry, I misread the log. The NPD found the metric server is OOM. It is about the metrics server, not the NPD. Sorry for disturb. I may add some memory for metric-server and check why metric server memory changed. I heard that metrics server recently released a new version. I will check if it is related.
/sig node
OOM metrics shows the metric server is OOMed. This is odd, that only metrics-server-nanny is with a memory limit. It uses whose command is . metrics-server container has no memory limit, but it got killed by OOM. Does this indicate that the node is OOM? (edited)
The last run failed in node-problem-detector ContainerStateTerminated. I0227 05:03:31. ] PodStartupLatency: perc50: 14m29.018886827s, perc90: 28m44.625518807s, perc99: 32m2.19303496s; threshold 1h0m0s I0227 05:03:32. ]


Question: MINION_TAG="${INSTANCE_PREFIX}-minion" MINION_NAMES=($(eval echo ${INSTANCE_PREFIX}-minion-{1..${NUM_MINIONS}})) MINION_IP_RANGES=($(eval echo "10.245.{1..${NUM_MINIONS}}.0/24")) <del> MINION_SCOPES="" </del> <ins> MINION_SCOPES="storage-ro,compute-rw" </ins> # Increase the sleep interval value if concerned about API rate limits. 3, in seconds, is the default. POLL_SLEEP_INTERVAL=3 PORTAL_NET="10.0.0.0/16"

Loaded restart count threshold overrides: map[coredns:2 fluentd-gcp:999 konnectivity-agent:2 kube-controller-manager:1 kube-proxy:2 kube-scheduler:1 l7-lb-controller:1 metadata-proxy:2 prometheus-to-sd-exporter:2] I0227 05:03:32. ] ClusterOOMsTracker: gathering cluster OOMs tracking measurement E0227 05:03:32. ] TestMetrics: [action gather failed for SystemPodMetrics measurement: restart counts violation: RestartCount(node-problem-detector-tgl8h, node-problem-detector)=1, want <= 0; RestartCount(node-problem-detector-vpsmz, node-problem-detector)=1, want <= 0] I0227 05:03:32. ] Step "[step: 29] gathering measurements" ended W0227 05:03:32. ] Got errors during step execution: [measurement call TestMetrics - TestMetrics error: [action gather failed for SystemPodMetrics measurement: restart counts violation: RestartCount(node-problem-detector-tgl8h, node-problem-detector)=1, want <= 0; RestartCount(node-problem-detector-vpsmz, node-problem-detector)=1, want <= 0]] I0227 05:03:32. ] Waiting for the chaos monkey subroutine to end... I0227 05:03:32. ] Chaos monkey ended. I0227 05:03:32. ] I0227 05:03:32. ] Stopping DnsLookupLatency probe... I0227 05:03:47. ] Stopping InClusterNetworkLatency probe... I0227 05:04:22. ] Resources cleanup time: 50.539700215s E0227 05:04:22. ]
E0227 05:04:22. ] Test Finished E0227 05:04:22. ] Test: E0227 05:04:22. ] Status: Fail E0227 05:04:22. ]


Question: MINION_TAG="${INSTANCE_PREFIX}-minion" MINION_NAMES=($(eval echo ${INSTANCE_PREFIX}-minion-{1..${NUM_MINIONS}})) MINION_IP_RANGES=($(eval echo "10.245.{1..${NUM_MINIONS}}.0/24")) <del> MINION_SCOPES="" </del> <ins> MINION_SCOPES="storage-ro,compute-rw" </ins> # Increase the sleep interval value if concerned about API rate limits. 3, in seconds, is the default. POLL_SLEEP_INTERVAL=3 PORTAL_NET="10.0.0.0/16"

Errors: [measurement call TestMetrics - TestMetrics error: [action gather failed for SystemPodMetrics measurement: restart counts violation: RestartCount(node-problem-detector-tgl8h, node-problem-detector)=1, want <= 0; RestartCount(node-problem-detector-vpsmz, node-problem-detector)=1, want <= 0]] E0227 05:04:22. ]
I0227 05:04:22. ]
I0227 05:04:22. ] Running testing/huge- I0227 05:04:22. ]


Question: ) // ValidateLoadBalancerStatus validates required fields on a LoadBalancerStatus <del> func ValidateLoadBalancerStatus(status *core.LoadBalancerStatus, fldPath *field.Path) field.ErrorList { </del> <ins> func ValidateLoadBalancerStatus(status *core.LoadBalancerStatus, fldPath *field.Path, spec *core.ServiceSpec) field.ErrorList { </ins> allErrs := field.ErrorList{} <del> for i, ingress := range status.Ingress { idxPath := fldPath.Child("ingress").Index(i) if len(ingress.IP) > 0 { if isIP := (netutils.ParseIPSloppy(ingress.IP) != nil); !isIP { allErrs = append(allErrs, field.Invalid(idxPath.Child("ip"), ingress.IP, "must be a valid IP address")) } } if utilfeature.DefaultFeatureGate.Enabled(features.LoadBalancerIPMode) && ingress.IPMode == nil { </del> <ins> ingrPath := fldPath.Child("ingress") if !utilfeature.DefaultFeatureGate.Enabled(features.AllowServiceLBStatusOnNonLB) && spec.Type != core.ServiceTypeLoadBalancer && len(status.Ingress) != 0 { allErrs = append(allErrs, field.Forbidden(ingrPath, "may only be used when `spec.type` is 'LoadBalancer'")) } else { for i, ingress := range status.Ingress { idxPath := ingrPath.Index(i) </ins> if len(ingress.IP) > 0 { <del> allErrs = append(allErrs, field.Required(idxPath.Child("ipMode"), "must be specified when `ip` is set")) </del> <ins> if isIP := (netutils.ParseIPSloppy(ingress.IP) != nil); !isIP { allErrs = append(allErrs, field.Invalid(idxPath.Child("ip"), ingress.IP, "must be a valid IP address")) } </ins> } <del> } else if ingress.IPMode != nil && len(ingress.IP) == 0 { allErrs = append(allErrs, field.Forbidden(idxPath.Child("ipMode"), "may not be specified when `ip` is not set")) } else if ingress.IPMode != nil && !supportedLoadBalancerIPMode.Has(string(*ingress.IPMode)) { allErrs = append(allErrs, field.NotSupported(idxPath.Child("ipMode"), ingress.IPMode, supportedLoadBalancerIPMode.List())) } if len(ingress.Hostname) > 0 { for _, msg := range validation.IsDNS1123Subdomain(ingress.Hostname) { allErrs = append(allErrs, field.Invalid(idxPath.Child("hostname"), ingress.Hostname, msg)) </del> <ins> if utilfeature.DefaultFeatureGate.Enabled(features.LoadBalancerIPMode) && ingress.IPMode == nil { if len(ingress.IP) > 0 { allErrs = append(allErrs, field.Required(idxPath.Child("ipMode"), "must be specified when `ip` is set")) } } else if ingress.IPMode != nil && len(ingress.IP) == 0 { allErrs = append(allErrs, field.Forbidden(idxPath.Child("ipMode"), "may not be specified when `ip` is not set")) } else if ingress.IPMode != nil && !supportedLoadBalancerIPMode.Has(string(*ingress.IPMode)) { allErrs = append(allErrs, field.NotSupported(idxPath.Child("ipMode"), ingress.IPMode, supportedLoadBalancerIPMode.List())) </ins> } <del> if isIP := (netutils.ParseIPSloppy(ingress.Hostname) != nil); isIP { allErrs = append(allErrs, field.Invalid(idxPath.Child("hostname"), ingress.Hostname, "must be a DNS name, not an IP address")) </del> <ins> if len(ingress.Hostname) > 0 { for _, msg := range validation.IsDNS1123Subdomain(ingress.Hostname) { allErrs = append(allErrs, field.Invalid(idxPath.Child("hostname"), ingress.Hostname, msg)) } if isIP := (netutils.ParseIPSloppy(ingress.Hostname) != nil); isIP { allErrs = append(allErrs, field.Invalid(idxPath.Child("hostname"), ingress.Hostname, "must be a DNS name, not an IP address")) } </ins> } } }

xref I think this may be one of the few cases where we can actually change the behavior in a "breaking" way. User creates service with , is empty Controller or user updates the status subresource, sets (not wiped or validated, should be) User updates in any way, is wiped It's clear that the is not meant to be set when is not "LoadBalancer". The question is: 1) Fail validation? 2) Silently discard the value? 3) Warn and then discard the value? 4) Just warn and leave it. Because we allowed it in the past, but it is so volatile, I lean towards 3. and - the people most familair with this type. Edit: PrepareForUpdate() runs before warnings. So: (1) is easy (2) is easy (4) is slightly less easy than (2) (3) is slightly less easy than (4)
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
OMG change it to clusterIP as expected Status is wiped update status as ClusterIP ... and if I edit the service back to loadbalancer it keeps the status??
/kind bug
/retitle Service can be set when !=
We talked about this one at the SIG Network meeting today. suggested we use the feature gate for deprecation option to phase the current (problematic) functionality out.
/triage accept
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /triage accept Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: } } <ins> func printStatusAndLogsForNotReadyPods(c *client.Client, oldPods, newPods []*api.Pod) { printFn := func(id, log string, err error, previous bool) { prefix := "Retrieving log for container" if previous { prefix = "Retrieving log for the last terminated container" } if err != nil { Logf("%s %s, err: %v:n%sn", prefix, id, log) } else { Logf("%s %s:n%sn", prefix, id, log) } } for _, oldPod := range oldPods { for _, p := range newPods { if p.Namespace != oldPod.Namespace || p.Name != oldPod.Name { continue } if ok, _ := podRunningReady(p); !ok { Logf("Status for not ready pod %s/%s: %+v", p.Namespace, p.Name, p.Status) // Print the log of the containers if pod is not running and ready. for _, container := range p.Status.ContainerStatuses { cIdentifer := fmt.Sprintf("%s/%s/%s", p.Namespace, p.Name, container.Name) log, err := getPodLogs(c, p.Namespace, p.Name, container.Name) printFn(cIdentifer, log, err, false) // Get log from the previous container. if container.RestartCount > 0 { printFn(cIdentifer, log, err, true) } } } break } } } </ins> // rebootNode takes node name on provider through the following steps using c: //  - ensures the node is ready //  - ensures all pods on the node are running and ready

We are running heapster, dns server, kube-proxy etc. in kube-system namespace. Some e2e tests try to make sure all those kube-system pods are at ready state before firing the real tests, others just simply ingore them. But both cases might cause e2e flakiness when some of those kube-system containers are in crashloop, for example, for heapster crashloop, for influxdb crashloop, etc. When such cases happen, the developer doesn't have much information on why since the nodes might be teared down. We should watch those kube-system pods, and onContainerFailure, we should collect the related logs. One simple way is using kubectl logs <Pod<Container--previous cc/
is this related enough to and that you can bundle it in?
The logs from the pods should be retrieved via the api server. This is different from the kubelet logs. I'll upload a PR for this.
Great job It will be very helpful for debugging.


Question: // parameters, you'll run out of stack space before anything useful happens. Convert(src, dest interface{}, flags FieldMatchingFlags) error <ins> // DefaultConvert performs the default conversion, without calling a conversion func // on the current stack frame. This makes it safe to call from a conversion func. DefaultConvert(src, dest interface{}, flags FieldMatchingFlags) error </ins> // SrcTags and DestTags contain the struct tags that src and dest had, respectively. // If the enclosing object was not a struct, then these will contain no tags, of course. SrcTag() reflect.StructTag

Files storing latest versions for minors are not updated since infix was to the version string. Example Similar issues can be found in 1.25 and 1.24 builds (last 1.23 build infix so it has not happened there yet): I expect the files to be overwritten with newer version of the build. Manually run build job for one of the minors. No response Not related. Not related. No response No response No response No response
I believe this may be also responsible for breaking kubemark tests: since they fail due to version mismatch
/sig release
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /priority high Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Thanks for raising the issue! /cc who is working on before
I assume this will be fixed automatically with the next patches, because then we would compare with . Anyway, I think we should fix that intermediately by manually updating the version markers to match their correct version. I did that now for the bucket
We had to update the versions markers again, because those were pointing to two commits earlier and didn't include the Go updates. The latest version markers are: v1.24.12-35+ v1.25.8-37+ v1.26.3-41+
latest-1.27 has been affected by this as well. I updated it manually to v1.27.0-9+. See this thread for more details:
/triage accepted
This should be fixed now, closing it. Feel free to reopen if we missed anything.
This issue is still present, see this Slack thread for more details: /reopen
Reopened this issue. <detailsIn response to : This issue is still present, see this Slack thread for more details: /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
We have a similar situation with latest-1.28 which is currently pointing to: . This is one commit before the release commit: ! It's mostly like that the release step didn't update version markers, we should look into this.


Question: // parameters, you'll run out of stack space before anything useful happens. Convert(src, dest interface{}, flags FieldMatchingFlags) error <ins> // DefaultConvert performs the default conversion, without calling a conversion func // on the current stack frame. This makes it safe to call from a conversion func. DefaultConvert(src, dest interface{}, flags FieldMatchingFlags) error </ins> // SrcTags and DestTags contain the struct tags that src and dest had, respectively. // If the enclosing object was not a struct, then these will contain no tags, of course. SrcTag() reflect.StructTag

As of now, I'm going to manually update this version marker to unblock 1.28.1 patch release. I'll also take a look into this. /assign
The PR that fixed this issue got reverted so this is still relevant: /reopen
Reopened this issue. <detailsIn response to : The PR that fixed this issue got reverted so this is still relevant: /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
We're running into this issue again with stable-1.29 marker. It's currently pointing to which is It should instead point to I'm going to manually change it following the same procedure as the last time. Version marker updated to
This issue is labeled with but has not been updated in over 90 days, and should be re-triaged. Important-soon issues must be staffed and worked on either currently, or very soon, ideally in time for the next release. You can: Confirm that this issue is still relevant with (org members only) Deprioritize it with or Close this issue with For more details on the triage process, see /remove-triage accepted
/triage accepted
Same issue for 1.30, where I had to update the version marker manually again.


Question: return utilfile.FileExists(pathname) } <ins> // EvalHostSymlinks returns the path name after evaluating symlinks func (mounter *Mounter) EvalHostSymlinks(pathname string) (string, error) { return filepath.EvalSymlinks(pathname) } </ins> // check whether hostPath is within volume path // this func will lock all intermediate subpath directories, need to close handle outside of this func after container started func lockAndCheckSubPath(volumePath, hostPath string) ([]uintptr, error) {

/kind bug Using , if a block device is symlinked to the discovery directory e.g. , then MapVolume will fail with since the containerized kubelet cannot see the path on host. is mounted by containerized kubelet, so if before we try to map the device we set devicePath to the evaluation of symlink and end up with a devicePath of , containerized kubelet will see the device and successfully map the device. Are there any issues with this solution? This is on Kubernetes 1.10
/sig storage
ah yes containerized kubelet has many issues when dealing with symlinks. Check out Mounter.GetMode(), it calls a nsenter EvalSymlinks which should be able to do what you need. But actually, one thing to consider, is do you need to do the losetup on the host, or does inside the container work?
Yes it appears to work from inside the container, on the host I run which disappears after I delete the pod using the device. My understanding of running losetup in MapVolume is not that great but I suppose that as long as that /dev/loop0 is created we are fine. Thanks for the quick response, I'll create a PR using that function and tag you
Does it create it on the host or in the containerized kubelet? What happens when containerized kubelet is restarted? The purpose of losetup is to do "refcounting" on the block device to prevent possible naming collisions when devices are attached/detached under unhealthy conditions. Ie, if a device showed up as /dev/sdb and you gave that to a pod, and then for some reason it become unhealthy and disappeared (while the pod is still running), another device (maybe even another volume type) could get attached to the node and reuse /dev/sdb. Now the pod has access to a device it's not supposed to. With filesystem volumes, the refcounting is done with mount points, but for block device, loopback is the equivalent.
It is created on the host (and also visible in the container), and stays around through restart. I understand now, thank you.
Actually after I restart it becomes /vdb instead of /dev/vdb in the losetup output, idk what that is about.


Question: return utilfile.FileExists(pathname) } <ins> // EvalHostSymlinks returns the path name after evaluating symlinks func (mounter *Mounter) EvalHostSymlinks(pathname string) (string, error) { return filepath.EvalSymlinks(pathname) } </ins> // check whether hostPath is within volume path // this func will lock all intermediate subpath directories, need to close handle outside of this func after container started func lockAndCheckSubPath(volumePath, hostPath string) ([]uintptr, error) {

There is no such thing as /vdb and /dev/vdb is still intact of course. Seems inconsequential since it is disappearing after all pods using the device are deleted, i.e. it's still serving as a refcounter


Question: endpointSlicesEnabled := utilfeature.DefaultFeatureGate.Enabled(features.EndpointSliceProxying) <ins> var incorrectAddresses []string nodePortAddresses, incorrectAddresses = utilproxy.FilterIncorrectCIDRVersion(nodePortAddresses, isIPv6) if len(incorrectAddresses) > 0 { klog.Warning("NodePortAddresses of wrong family; ", incorrectAddresses) } </ins> proxier := &Proxier{ portsMap:              make(map[utilproxy.LocalPort]utilproxy.Closeable), serviceMap:            make(proxy.ServiceMap),

What happened: In a dual-stack cluster when specifying both ipv6 and ipv4 addresses in the entries are inserted in ipvs for the ipv6 nodeports but with an invalid ipv4 address; The example shows the ipv4 loopback address but any ipv4 address in will cause a faulty entry. What you expected to happen: No ipv4 entry shall be inserted for the ipv6 nodeport. How to reproduce it (as minimally and precisely as possible): Configure with both ipv4 and ipv6 addresses Bring up a dual-stack cluster with proxy-mode Create a service with ipFamily:IPv6 Do Anything else we need to know?: Proxy-mode=iptables seems to work. With no ipv4 entries are inserted for ipv6 nodeports. In it is recommended to use but doing so makes kube-proxy enter rules for address (in addition to "127.0.0.1"). While these entries cause no harm they can't be used and should not be there. These entries are created for both proxy-mode "iptables" and "ipvs". This should not be handled in this issue but in another (new) issue if needed. Example; Environment: Kubernetes version (use ):Cloud provider or hardware configuration: Kvm's on Ubuntu OS (e.g: ): xcluster Kernel (e.g. ):Install tools: "from scratch" Network plugin and version (if this is a network-related bug): calico/cni:v3.12.0 Others: /sig network
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/area ipvs /area ipv6
/remove-triage unresolved /assign I take this myself if no one objects.
The ipv4 address is obviously a ipv4-converted ipv6 from .
According to the code here (), we may need to filter the before passing it to .
Looks that that's it. Please go ahead and make a PR, or I can do it if you prefer. I will help test the PR of course. Here are some review comments right away; Use v1.IPFamily as param to the filter function rather than The filter function seem usable and generic and may be placed in some "util" package, but leave it as-is and let the maintainers decide that.
Hi feel free to send a PR please, as you have took this already :) I'd like to help review it!
The "addresses" are really CIDRs which make the fix even simpler.


Question: fi local -r src_dir="${KUBE_HOME}/kube-manifests/kubernetes/gci-trusty" <del> if [[ -n "${KUBE_USER:-}" || ! -e /etc/srv/kubernetes/abac-authz-policy.jsonl ]]; then local -r abac_policy_json="${src_dir}/abac-authz-policy.jsonl" remove-salt-config-comments "${abac_policy_json}" if [[ -n "${KUBE_USER:-}" ]]; then sed -i -e "s/{{kube_user}}/${KUBE_USER}/g" "${abac_policy_json}" else sed -i -e "/{{kube_user}}/d" "${abac_policy_json}" fi cp "${abac_policy_json}" /etc/srv/kubernetes/ fi </del> src_file="${src_dir}/kube-apiserver.manifest" remove-salt-config-comments "${src_file}" # Evaluate variables.

Typically, the dns and dashboard is in crash loop. Ref cc
if you check the file, does it contain lines like ?
Previous comment is ref: from I'll copy that to container-linux as well.


Question: SecurityOpt: fmtSecurityOpts, } <del> // There is no /etc/resolv.conf in Windows, DNS and DNSSearch options would have to be passed to Docker runtime instead if runtime.GOOS == "windows" { hc.DNS = opts.DNS hc.DNSSearch = opts.DNSSearch } </del> <ins> updateHostConfig(hc) </ins> // Set sysctls if requested if container.Name == PodInfraContainerName {

What happened: Latest kubelet + Windows Docker version 1.14.0-dev refused to start any containers, error output: "invalid option: Windows does not support MemorySwap" How to reproduce it (as minimally and precisely as possible): It's always reproducible with Windows Docker 1.14.0-dev + latest kubelet code base. Anything else do we need to know: It didn't happen for Windows Docker 1.12.2-cs2-ws-beta. The reason behind the issue was because new code was for Windows Docker to check Resources parameters, it doesn't accept a non-zero value for MemorySwap: But kubelet was default to set an unlimited (-1) value. The issue was gone in my setup after applied a private fix to disable MemorySwap by set it to zero for Windows.


Question: } _, err := createPodPreset(f.ClientSet, f.Namespace.Name, pip) <ins> if errors.IsNotFound(err) { framework.Skipf("podpresets requires k8s.io/api/settings/v1alpha1 to be enabled") } </ins> Expect(err).NotTo(HaveOccurred()) By("creating the pod")

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: is consistently failing because of PodPreset feature tests, which require settings/v1alpha1, which is no longer enabled by default What you expected to happen: . How to reproduce it (as minimally and precisely as possible): . Anything else we need to know?: This was caused by and kubernetes/test-infra Environment: . Kubernetes version (use ): . Cloud provider or hardware configuration**: . OS (e.g. from /etc/os-release): . Kernel (e.g. ): . Install tools: . Others: .
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig service-catalog
does the alpha CI job enable alpha API versions?
file has "AllAlpha=true" Maybe would be better?


Question: t.Fatalf("AddPod failed. Expected: <no error> Actual: <%v>", podAddErr) } <del> time.Sleep(1000 * time.Millisecond) </del> // Volume is added to asw, volume should be reported as attached to the node. waitForVolumeAddedToNode(t, generatedVolumeName, nodeName1, asw) verifyVolumeAttachedToNode(t, generatedVolumeName, nodeName1, cache.AttachStateAttached, asw) <del> verifyVolumeReportedAsAttachedToNode(t, logger, generatedVolumeName, nodeName1, true, asw) </del> <ins> verifyVolumeReportedAsAttachedToNode(t, logger, generatedVolumeName, nodeName1, true, asw, volumeAttachedCheckTimeout) </ins> // Delete the pod, but detach will fail dsw.DeletePod(types.UniquePodName(podName1), generatedVolumeName, nodeName1)

TestRunOneVolumeDetachFailNodeWithReadWriteOnce 4 times since 03-19 23:00 PDT, intermittently ! https://k8s- Still investigating whether this is a tolerable flake or not with more history than testgrid view, or otherwise would appreciate input from anyone with expertise in this test, especially as regards to whether this should be a blocker for tomorrows beta cut. maybe: /sig storage /sig testing
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
is already open for this, looks like it is attempting to fix it
thank you!!! closing this in favor of that one, which I also to our CI signal project board for visibility. For completeness: since it's a longer standing issue, well identified as a flake and not a failure and also actively in progress, NOT considering it a blocker for the go/nogo signal for the beta tomorrow. /close
Closing this issue. <detailsIn response to : thank you!!! closing this in favor of that one, which I also to our CI signal project board for visibility. For completeness: since it's a longer standing issue, well identified as a flake and not a failure and also actively in progress, NOT considering it a blocker for the go/nogo signal for the beta tomorrow. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: t.Fatalf("AddPod failed. Expected: <no error> Actual: <%v>", podAddErr) } <del> time.Sleep(1000 * time.Millisecond) </del> // Volume is added to asw, volume should be reported as attached to the node. waitForVolumeAddedToNode(t, generatedVolumeName, nodeName1, asw) verifyVolumeAttachedToNode(t, generatedVolumeName, nodeName1, cache.AttachStateAttached, asw) <del> verifyVolumeReportedAsAttachedToNode(t, logger, generatedVolumeName, nodeName1, true, asw) </del> <ins> verifyVolumeReportedAsAttachedToNode(t, logger, generatedVolumeName, nodeName1, true, asw, volumeAttachedCheckTimeout) </ins> // Delete the pod, but detach will fail dsw.DeletePod(types.UniquePodName(podName1), generatedVolumeName, nodeName1)

pull-kubernetes-unit TestRunOneVolumeDetachFailNodeWithReadWriteOnce Jan 7th, 2022 Check volume <fake-plugin/volume-nameis reported as attached to node <fail-detach-node, got false, expected true No response_ /sig storage
/cc
this is the top unit test flake according to
cc
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
I think this test is no longer flaky by checking the test grid, close it now.
Unfortunately this does still seem to be flaky: Here was the output from : /reopen
Reopened this issue. <detailsIn response to : Unfortunately this does still seem to be flaky: Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-lifecycle stale
still the case
How about adding some logging to help troubleshoot this? I looked at it a little, but it is hard to tell which part of the test is failing. PR
/
This is still happening
/triage accepted
From the logs, in the OneVolumeDetachFailNodeWithReadWriteOnce test, succeeded, but the next verification failed. In verifyVolumeAttachedToNode, the attachState is from calling and within asw, it calls . In verifyVolumeReportedAsAttachedToNode, the volumes are obtained from calling , it seems that the node is not in asw.nodesToUpdateStatusFor at this step. Cannot reproduce locally. I tried to run the test 100 times locally with debug.Stack()... 3 * 100 times all passed.
Tried to run stress test on my linux machine as well but still cannot reproduce. Command: Result: Also tried running n process in parallel. Command: Result:
It seems like it happens more frequently.


Question: t.Fatalf("AddPod failed. Expected: <no error> Actual: <%v>", podAddErr) } <del> time.Sleep(1000 * time.Millisecond) </del> // Volume is added to asw, volume should be reported as attached to the node. waitForVolumeAddedToNode(t, generatedVolumeName, nodeName1, asw) verifyVolumeAttachedToNode(t, generatedVolumeName, nodeName1, cache.AttachStateAttached, asw) <del> verifyVolumeReportedAsAttachedToNode(t, logger, generatedVolumeName, nodeName1, true, asw) </del> <ins> verifyVolumeReportedAsAttachedToNode(t, logger, generatedVolumeName, nodeName1, true, asw, volumeAttachedCheckTimeout) </ins> // Delete the pod, but detach will fail dsw.DeletePod(types.UniquePodName(podName1), generatedVolumeName, nodeName1)

But I may have no time to dig into the cause. Let's see.
cc as the test author.
The problem is that this test has a lot of dependency on the sleeps that are hardcoded with a very narrow scope, but our CI seems to be able to stress it much more. As soon as you slightly change one of the sleep duration it starts to flake locally The test can not make an assumption that timing is perfect, even more in the sub second range can someone in sig-storage review this test and make it less dependant on those tight sleeps?
According to most failures, it failed at Line 660. Let me run some stress locally. with 250ms sleep with 100ms sleep
I agree this test case is too brittle due to its specific timing dependencies. To fix this, I suggest reworking the test case to: 1) Add retry with timeout to wait for final conditions 2) Remove the intermediate timing-based state transition checks. Instead, we can keep track of state transitions in the Fakes and validate those stats. wdyt?
Friendly ping from 1.27 CI Signal that this test is still a noisy flake on the master-blocking dashboard as CI Signal has flagged it in and


Question: exit 2 fi <del> /opt/bin/etcdctl mk /coreos.com/network/config "{"Network":"${FLANNEL_NET}", "Backend": {"Type": "vxlan"}}" </del> <ins> /opt/bin/etcdctl mk /coreos.com/network/config "{"Network":"${FLANNEL_NET}", "Backend": {"Type": "vxlan"}${FLANNEL_OTHER_NET_CONFIG}}" </ins> attempt=$((attempt+1)) sleep 3 fi

and provide for configuring Flannel's parameter but none of the others listed in


Question: rm -rf "${LOCAL_OUTPUT_BINPATH}" mkdir -p "${LOCAL_OUTPUT_BINPATH}" <del> # Remove the container if it is left over from some previous aborted run "${DOCKER[@]}" rm -f -v "${KUBE_BUILD_CONTAINER_NAME}" >/dev/null 2>&1 || true </del> <ins> kube::build::destroy_container "${KUBE_BUILD_CONTAINER_NAME}" </ins> "${docker_cmd[@]}" bash -c "cp -r ${REMOTE_OUTPUT_BINPATH} /tmp/bin;touch /tmp/finished;rm /tmp/bin/test_for_remote;/bin/sleep 600" > /dev/null 2>&1 # Wait until binaries have finished coppying

Github automatically adds a artifact to every release, which sounds very similar to the artifact that we generate as the release artifact. Assuming we don't have control over the naming or presence of the autogenerated source code artifact, I think we should consider renaming our published artifact to better disambiguate between the two.
We distribute most of our artifacts outside of GitHub. We should probably stop uploading this one entirely.
cc
We've got a thread going on w/ Tim. /assign
can you please link so folks can follow the bread crumbs? ( cc )
-- It's currently sensitive. Will share once there's an outcome.
ACK. thanks for the heads up
I think it's safe to discuss openly at this point. We had a report that the sha512 hashes on our release artifacts were incorrect. We have confirmed that the mismatch was due to a bug in the release pipeline, and not malicious. This issue here was just a point of confusion that came out of that discussion.
thanks !
/assign
Going one step back since I think the exists for historical reasons: The client and server binaries were part of this bundle, which is not the case any more: So it seems that is mostly a collection of licenses and scripts. Do we really need it or may we cleanup some code paths and remove it completely?
+1 to put the word out and remove it completely if no one depends on it.
I'm not confident that CI is not using these versioned scripts, in fact i'm fairly sure that it is. However it would not be pulling it from github.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale I think this remains as relevant as ever.
/milestone v1.23
/triage accepted
Hi and k8s 1.23 bug triage shadow here.


Question: rm -rf "${LOCAL_OUTPUT_BINPATH}" mkdir -p "${LOCAL_OUTPUT_BINPATH}" <del> # Remove the container if it is left over from some previous aborted run "${DOCKER[@]}" rm -f -v "${KUBE_BUILD_CONTAINER_NAME}" >/dev/null 2>&1 || true </del> <ins> kube::build::destroy_container "${KUBE_BUILD_CONTAINER_NAME}" </ins> "${docker_cmd[@]}" bash -c "cp -r ${REMOTE_OUTPUT_BINPATH} /tmp/bin;touch /tmp/finished;rm /tmp/bin/test_for_remote;/bin/sleep 600" > /dev/null 2>&1 # Wait until binaries have finished coppying

There hasn't been any activity or linked PRs for this issue. Just wanted to check in to see if this is still on track for 1.23.
/milestone v1.24
Hi I am part of the bug triage team for v1.24. We noticed that this issue hasn't been updated for a while. Just wanted to check if there is any work planned that would go into this.
this one keeps moving down the milestones. i don't see any activity since november. So kick the can down the road /milestone v1.25
Hi My name Hossein Salahi from Bug-Triage team. I am just checking to ensure if everything is on track for K8s 1.25. Thanks!
That's a question for or
any remarks on that?
the release 1.25 bug triage Lead. I'm reaching out to see the status of this issue and if we are still targeting the current release. Also, kindly if you can update the priority so we can track it in the future.
this one keeps moving down the milestones yet again!. i don't see any activity since i moved this down the line. So kick the can down the road: /milestone clear (instead of v1.26 as this has been moving down the line for a few releases now)
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
What's the current status of this issue? Is there anything that we can do to move this forward?
We got new conversations going on during Kubecon on whether we should publish our release binaries on GitHub. Given that and I think we should rethink now what we publish on the release pages. I think we should start publishing all binaries for one. It is a common practice everywhere and I think users largely expect them to be there.


Question: rm -rf "${LOCAL_OUTPUT_BINPATH}" mkdir -p "${LOCAL_OUTPUT_BINPATH}" <del> # Remove the container if it is left over from some previous aborted run "${DOCKER[@]}" rm -f -v "${KUBE_BUILD_CONTAINER_NAME}" >/dev/null 2>&1 || true </del> <ins> kube::build::destroy_container "${KUBE_BUILD_CONTAINER_NAME}" </ins> "${docker_cmd[@]}" bash -c "cp -r ${REMOTE_OUTPUT_BINPATH} /tmp/bin;touch /tmp/finished;rm /tmp/bin/test_for_remote;/bin/sleep 600" > /dev/null 2>&1 # Wait until binaries have finished coppying

let's scope the issue to the actual "Kubernetes Source Code" tarball. We can still remove it from what I can see from the discussion.
I agree with Removing the "Kubernetes Source Code" tarball should be as easy as removing this from krel: However, there are a few questions: is it safe to do so? I believe yes, because this "Kubernetes Source Code" tarball is pointing to the GCS bucket, and we'll still continue publishing that tarball to the bucket The link is also available in the changelog, so it's easy to find do we need any deprecation period? Considering the above two points, I don't think so. If you folks agree with that, I can go ahead and create a PR to update krel.
I agree on that, let's drop an additional note on k-dev that others are aware of the change.
/assign
Update: An announcement to proceed with the removal of the "Kubernetes Source Code" artifact has been sent to k-dev: PR to remove this artifact from being published: Removal will go into effect starting the January patch releases.


Question: w.initialValue = results.Unknown } <del> podName := getPodLabelName(w.pod) </del> basicMetricLabels := metrics.Labels{ "probe_type": w.probeType.String(), "container":  w.container.Name, <del> "pod":        podName, </del> <ins> "pod":        w.pod.Name, </ins> "namespace":  w.pod.Namespace, "pod_uid":    string(w.pod.UID), }

The metric used to report acutal pod in the label , but now I see the podname prefix only. The full pod name in the label, so I can trace the metric to the failing pod. Check raw api and verify the actual pod name with uids. <details</detailsAzure AKS <details</detailsNo response No response No response_
/sig node
/triage accepted /priority important-soon /cc As discussed this is a change in functionaloty that wasn't highlighted in release notes. Need to decide on what do do with it. I understand it's a change of functionality. what is the side effect of this for you? Does the label without last part not informational enough?
I've been using the metric in my custom grafana dashboard that shows CPU, RAM, IO usage of pods with graph of increases in probes. If the apps are stable, the probe graph looks steadily. I can see if some irregularities in system metrics are influencing probes of respective pod. Also, if readiness probe is failing, the pod is removed from service, but is not restarted so I must have knowledge of pods with failing readiness probes. I attach the screenshot of my graph. ! So I'd welcome the possibility to pair the probes with exact pod and thank you for looking into it.
are we thinking that this resulted in an undesirable change? Spologies for being a bit late to reply, I'm returning from a brief paternity leave staycation.
This issue is labeled with but has not been updated in over 90 days, and should be re-triaged. Important-soon issues must be staffed and worked on either currently, or very soon, ideally in time for the next release. You can: Confirm that this issue is still relevant with (org members only) Deprioritize it with or Close this issue with For more details on the triage process, see /remove-triage accepted
Well this is awkward... I still believe the actual behavior does not respect the intended idea...
This issue makes it hard to link probe metrics with right pods. The value of pod label in and are also inconsistent with how it is used in other kubelet metrics, see: vs Definitely a bug, regression since 1.25.
I can't get anyone to look at the fix/revert: you may have better luck
/milestone v1.28
discussion in
You've pushed it through, congrats!


Question: <ins> /* Copyright 2016 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package util import ( "fmt" "reflect" "sync" "time" federation_api "k8s.io/kubernetes/federation/apis/federation/v1beta1" federation_release_1_4 "k8s.io/kubernetes/federation/client/clientset_generated/federation_release_1_4" api "k8s.io/kubernetes/pkg/api" api_v1 "k8s.io/kubernetes/pkg/api/v1" "k8s.io/kubernetes/pkg/client/cache" "k8s.io/kubernetes/pkg/client/restclient" "k8s.io/kubernetes/pkg/controller/framework" pkg_runtime "k8s.io/kubernetes/pkg/runtime" "k8s.io/kubernetes/pkg/watch" "github.com/golang/glog" ) const ( clusterSyncPeriod = 10 * time.Minute userAgentName     = "federation-service-controller" ) // FederatedReadOnlyStore is an overlay over multiple stores created in federated clusters. type FederatedReadOnlyStore interface { // Returns all items in the store. List() ([]interface{}, error) // GetByKey returns the item stored under the given key in the specified cluster (if exist). GetByKey(clusterName string, key string) (interface{}, bool, error) // Returns the items stored under the given key in all clusters. GetFromAllClusters(key string) ([]interface{}, error) // Checks whether stores for all clusters form the lists (and only these) are there and // are synced. This is only a basic check whether the data inside of the store is usable. // It is not a full synchronization/locking mechanism it only tries to ensure that out-of-sync // issues occur less often.	All users of the interface should assume // that there may be significant delays in content updates of all kinds and write their // code that it doesn't break if something is slightly out-of-sync. ClustersSynced(clusters []*federation_api.Cluster) bool } // An interface to access federation members and clients. type FederationView interface { // GetClientsetForCluster returns a clientset for the cluster, if present. GetClientsetForCluster(clusterName string) (federation_release_1_4.Interface, error) // GetReadyClusers returns all clusters for which the sub-informers are run. GetReadyClusters() ([]*federation_api.Cluster, error) // GetReadyCluster returns the cluster with the given name, if found. GetReadyCluster(name string) (*federation_api.Cluster, bool, error) // ClustersSynced returns true if the view is synced (for the first time). ClustersSynced() bool } // A structure that combines an informer running agains federated api server and listening for cluster updates // with multiple Kubernetes API informers (called target informers) running against federation members. Whenever a new // cluster is added to the federation an informer is created for it using TargetInformerFactory. Infomrers are stoped // when a cluster is either put offline of deleted. It is assumed that some controller keeps an eye on the cluster list // and thus the clusters in ETCD are up to date. type FederatedInformer interface { FederationView // Returns a store created over all stores from target informers. GetTargetStore() FederatedReadOnlyStore // Starts all the processes. Start() // Stops all the processes inside the informer. Stop() } // A function that should be used to create an informer on the target object. Store should use // framework.DeletionHandlingMetaNamespaceKeyFunc as a keying function. type TargetInformerFactory func(*federation_api.Cluster, federation_release_1_4.Interface) (cache.Store, framework.ControllerInterface) // Builds a FederatedInformer for the given federation client and factory. func NewFederatedInformer(federationClient federation_release_1_4.Interface, targetInformerFactory TargetInformerFactory) FederatedInformer { federatedInformer := &federatedInformerImpl{ targetInformerFactory: targetInformerFactory, clientFactory: func(cluster *federation_api.Cluster) (federation_release_1_4.Interface, error) { clusterConfig, err := BuildClusterConfig(cluster) if err != nil && clusterConfig != nil { clientset := federation_release_1_4.NewForConfigOrDie(restclient.AddUserAgent(clusterConfig, userAgentName)) return clientset, nil } return nil, err }, targetInformers: make(map[string]informer), } federatedInformer.clusterInformer.store, federatedInformer.clusterInformer.controller = framework.NewInformer( &cache.ListWatch{ ListFunc: func(options api.ListOptions) (pkg_runtime.Object, error) { return federationClient.Federation().Clusters().List(options) }, WatchFunc: func(options api.ListOptions) (watch.Interface, error) { return federationClient.Federation().Clusters().Watch(options) }, }, &federation_api.Cluster{}, clusterSyncPeriod, framework.ResourceEventHandlerFuncs{ DeleteFunc: func(old interface{}) { oldCluster, ok := old.(*federation_api.Cluster) if ok { federatedInformer.deleteCluster(oldCluster) } }, AddFunc: func(cur interface{}) { curCluster, ok := cur.(*federation_api.Cluster) if ok && isClusterReady(curCluster) { federatedInformer.addCluster(curCluster) } }, UpdateFunc: func(old, cur interface{}) { oldCluster, ok := old.(*federation_api.Cluster) if !ok { return } curCluster, ok := cur.(*federation_api.Cluster) if !ok { return } if isClusterReady(oldCluster) != isClusterReady(curCluster) || !reflect.DeepEqual(oldCluster.Spec, curCluster.Spec) { federatedInformer.deleteCluster(oldCluster) if isClusterReady(curCluster) { federatedInformer.addCluster(curCluster) } } }, }, ) return federatedInformer } func isClusterReady(cluster *federation_api.Cluster) bool { for _, condition := range cluster.Status.Conditions { if condition.Type == federation_api.ClusterReady { if condition.Status == api_v1.ConditionTrue { return true } } } return false } type informer struct { controller framework.ControllerInterface store      cache.Store stopChan   chan struct{} } type federatedInformerImpl struct { sync.Mutex // Informer on federated clusters. clusterInformer informer // Target informers factory targetInformerFactory TargetInformerFactory // Structures returned by targetInformerFactory targetInformers map[string]informer // A function to build clients. clientFactory func(*federation_api.Cluster) (federation_release_1_4.Interface, error) } type federatedStoreImpl struct { federatedInformer *federatedInformerImpl } func (f *federatedInformerImpl) Stop() { f.Lock() defer f.Unlock() close(f.clusterInformer.stopChan) for _, informer := range f.targetInformers { close(informer.stopChan) } } func (f *federatedInformerImpl) Start() { f.Lock() defer f.Unlock() f.clusterInformer.stopChan = make(chan struct{}) go f.clusterInformer.controller.Run(f.clusterInformer.stopChan) } // GetClientsetForCluster returns a clientset for the cluster, if present. func (f *federatedInformerImpl) GetClientsetForCluster(clusterName string) (federation_release_1_4.Interface, error) { f.Lock() defer f.Unlock() return f.getClientsetForClusterUnlocked(clusterName) } func (f *federatedInformerImpl) getClientsetForClusterUnlocked(clusterName string) (federation_release_1_4.Interface, error) { // No locking needed. Will happen in f.GetCluster. if cluster, found, err := f.getReadyClusterUnlocked(clusterName); found && err == nil { return f.clientFactory(cluster) } else { if err != nil { return nil, err } } return nil, fmt.Errorf("cluster %s not found", clusterName) } // GetReadyClusers returns all clusters for which the sub-informers are run. func (f *federatedInformerImpl) GetReadyClusters() ([]*federation_api.Cluster, error) { f.Lock() defer f.Unlock() items := f.clusterInformer.store.List() result := make([]*federation_api.Cluster, 0, len(items)) for _, item := range items { if cluster, ok := item.(*federation_api.Cluster); ok { if isClusterReady(cluster) { result = append(result, cluster) } } else { return nil, fmt.Errorf("wrong data in FederatedInformerImpl cluster store: %v", item) } } return result, nil } // GetCluster returns the cluster with the given name, if found. func (f *federatedInformerImpl) GetReadyCluster(name string) (*federation_api.Cluster, bool, error) { f.Lock() defer f.Unlock() return f.getReadyClusterUnlocked(name) } func (f *federatedInformerImpl) getReadyClusterUnlocked(name string) (*federation_api.Cluster, bool, error) { if obj, exist, err := f.clusterInformer.store.GetByKey(name); exist && err == nil { if cluster, ok := obj.(*federation_api.Cluster); ok { if isClusterReady(cluster) { return cluster, true, nil } return nil, false, nil } return nil, false, fmt.Errorf("wrong data in FederatedInformerImpl cluster store: %v", obj) } else { return nil, false, err } } // Synced returns true if the view is synced (for the first time) func (f *federatedInformerImpl) ClustersSynced() bool { f.Lock() defer f.Unlock() return f.clusterInformer.controller.HasSynced() } // Adds the given cluster to federated informer. func (f *federatedInformerImpl) addCluster(cluster *federation_api.Cluster) { f.Lock() defer f.Unlock() name := cluster.Name if client, err := f.getClientsetForClusterUnlocked(name); err == nil { store, controller := f.targetInformerFactory(cluster, client) targetInformer := informer{ controller: controller, store:      store, stopChan:   make(chan struct{}), } f.targetInformers[name] = targetInformer go targetInformer.controller.Run(targetInformer.stopChan) } else { // TODO: create also an event for cluster. glog.Errorf("Failed to create a client for cluster: %v", err) } } // Removes the cluster from federated informer. func (f *federatedInformerImpl) deleteCluster(cluster *federation_api.Cluster) { f.Lock() defer f.Unlock() name := cluster.Name if targetInformer, found := f.targetInformers[name]; found { close(targetInformer.stopChan) } delete(f.targetInformers, name) } // Returns a store created over all stores from target informers. func (f *federatedInformerImpl) GetTargetStore() FederatedReadOnlyStore { return &federatedStoreImpl{ federatedInformer: f, } } // Returns all items in the store. func (fs *federatedStoreImpl) List() ([]interface{}, error) { fs.federatedInformer.Lock() defer fs.federatedInformer.Unlock() result := make([]interface{}, 0) for _, targetInformer := range fs.federatedInformer.targetInformers { values := targetInformer.store.List() result = append(result, values...) } return result, nil } // GetByKey returns the item stored under the given key in the specified cluster (if exist). func (fs *federatedStoreImpl) GetByKey(clusterName string, key string) (interface{}, bool, error) { fs.federatedInformer.Lock() defer fs.federatedInformer.Unlock() if targetInformer, found := fs.federatedInformer.targetInformers[clusterName]; found { return targetInformer.store.GetByKey(key) } return nil, false, nil } // Returns the items stored under the given key in all clusters. func (fs *federatedStoreImpl) GetFromAllClusters(key string) ([]interface{}, error) { fs.federatedInformer.Lock() defer fs.federatedInformer.Unlock() result := make([]interface{}, 0) for _, targetInformer := range fs.federatedInformer.targetInformers { value, exist, err := targetInformer.store.GetByKey(key) if err != nil { return nil, err } if exist { result = append(result, value) } } return result, nil } // GetKey for returns the key under which the item would be put in the store. func (fs *federatedStoreImpl) GetKeyFor(item interface{}) string { // TODO: support other keying functions. key, _ := framework.DeletionHandlingMetaNamespaceKeyFunc(item) return key } // Checks whether stores for all clusters form the lists (and only these) are there and // are synced. func (fs *federatedStoreImpl) ClustersSynced(clusters []*federation_api.Cluster) bool { fs.federatedInformer.Lock() defer fs.federatedInformer.Unlock() if len(fs.federatedInformer.targetInformers) != len(clusters) { return false } for _, cluster := range clusters { if targetInformer, found := fs.federatedInformer.targetInformers[cluster.Name]; found { if !targetInformer.controller.HasSynced() { return false } } else { return false } } return true } </ins>

As discussed, similar patterns occur in many of the controller for federated objects. Lift this into a common library that can be used by all of the controllers. Open separate issues for each of the above, and track there. This can be the tracking issue over those.
Hi, , we have been starting develop work about cluster-controller and RS controller, so would you like to share me the plan about this work? Thanks.
Tracked in PR's referenced above. Closing this tracking issue.


Question: sched.Recorder.Eventf(victim, preemptor, v1.EventTypeNormal, "Preempted", "Preempting", "Preempted by %v/%v on node %v", preemptor.Namespace, preemptor.Name, nodeName) } <del> metrics.PreemptionVictims.Set(float64(len(victims))) </del> <ins> metrics.PreemptionVictims.Observe(float64(len(victims))) </ins> } // Clearing nominated pods should happen outside of "if node != nil". Node could // be nil when a pod with nominated node name is eligible to preempt again,

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Change scheduler metric from Gauge to Histogram Why is this needed: is recorded for each preemption attempt. A histogram is better. /sig scheduling /help /good-first-issue
/help /good-first-issue
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I can work on this issue.
/assign
I could work on this as well if doesn't have time to work on this.
I am working on it, I will create the initial PR in less than a week. Sorry for the delay.


Question: var ( // Master nodes are not added to standard load balancer by default. defaultExcludeMasterFromStandardLB = true <ins> // Outbound SNAT is enabled by default. defaultDisableOutboundSNAT = false </ins> ) // Config holds the configuration parsed from the --cloud-config flag

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: is the recommended way to configure the outbound networks together with Azure standard loadbalancer (SLB). However, for load balancer rules are disabled by default. That means all frontend IPs would be used as the SNAT IPs for virtual machines, according to . This would introduce troubles for various security scenarios, as there's no way to get a stable list of egress IPs. Azure cloud provider should support this scenario, and disable the outbound SNAT when outbound rules are configured by provisioning tools. What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Others: /kind bug /sig azure /priority critical-urgent


Question: a.cachedCreds = newCreds // Only close all connections when TLS cert rotates. Token rotation doesn't // need the extra noise. <del> if a.onRotate != nil && oldCreds != nil && !reflect.DeepEqual(oldCreds.cert, a.cachedCreds.cert) { a.onRotate() </del> <ins> if len(a.onRotateList) > 0 && oldCreds != nil && !reflect.DeepEqual(oldCreds.cert, a.cachedCreds.cert) { for _, onRotate := range a.onRotateList { onRotate() } </ins> } return nil }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Data race in client-go with below sample code: Data race: What you expected to happen: No data race How to reproduce it (as minimally and precisely as possible): Using above given sample code Kubernetes version (use ):Cloud provider or hardware configuration:OS (e.g: ): OSX
/sig api-machinery
I'd like to take it. /assign
/remove-sig api-machinery /sig auth cli
Bug triage for release 1.17 here. Since this issue has been tagged for the milestone, we would like to check it's status. Code freeze is on Nov. 14. Will this issue be resolved before then?
Another reminder here that code freeze is in less than two weeks (Nov. 14). We will need to move this issue out of the milestone if isn't going to be resolved before then.


Question: releaseHugepages() ginkgo.By("restarting kubelet to pick up pre-allocated hugepages") <del> restartKubelet() </del> <ins> // stop the kubelet and wait until the server will restart it automatically stopKubelet() </ins> waitForHugepages() })

ci-kubernetes-node-kubelet-serial-hugepages when start the pod with the resources requests that contain multiple hugepages resources should set correct hugetlb mount and limit under the container cgroup Unexpected error: "pod ran to completion", Multiple sizes huge pages is going to be graduated to GA, so having this test job fixed is quite important.
/sig node
/assign
/triage accepted
/cc
I did a small analysis and I can see that the pod failed to start, I can not find any failure messages under the kubelet log, so it really hard to say why it failed.
were you able to reproduce it locally? btw, there is a "right way" to fix this - to fix dynamic update of a hugepage resources. This would eliminate need in a kubelet restart. There is even a PR for that: , we just need to make it working again.
after conversation with on slack I'd say we should stick to the current procedure - restarting kubelet when hugepages allocation changes. So, we should investigate why kubelet doesn't restart properly in this test case and fix it.
Nope, I failed to reproduce it locally, also under the CI, it does not appear often. The best thing will be to describe the pod in case if it failed, so we will see the reason.
under the code we have so from the log , I can see that the kubelet received the pod before the kubelet stopped, so the problem that pod dropped to Failed state because of the kubelet stop.
So some additional details, the one who responsible to restart the kubelet again, is the server service: restart the kubelet service are waiting for hugepages to be available are starting the pod server service restart the kubelet again because it was not healthy for too long - failed to start because of the kubelet restart
great findings! Should we just stop the kubelet and let the server service to start it again? We can also check if the kubelet is restarted successfully before starting the pod.
I tried it under the PR , also I unsure how can we check if the kubelet restarted successfully. All tests that I wrote before I checked if the node has NotReady state and waited until it will be Ready, but in case of the restart probably the node will always have Ready state because the time period is very short. I will continue to investigate and will check locally how can I fix it.