Question: "spec": { "containers": [{ "name": "test-container", <del> "image": "gcr.io/google_containers/pause-amd64:3.0" </del> <ins> "image": "gcr.io/google_containers/pause-amd64:3.1" </ins> }] } }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: the pause-amd64:3.0 image on has an empty schema2 manifest, which prevents is from being pulled by proxy/caches which only support V2 registries. What you expected to happen: Being able to pull ` using a V2 manifest compatible client. Anything else we need to know?: The proxy/cahce is requesting this
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
After talking with , he mentionned the following
AFAIK, does not support well. After inspecting the image, you can find is still using manifest v1.
/cc FYI
Im not sure which exact version x schéma X is needed. What I know is that we can download through the proxy/cache all other Kubernetes kube-system images, only the pause image is giving us trouble. Therefore, I would say that having the image available using the same format as the other kube-system images would be the request here.
Yeah, we haven't updated the pause image in a while, since it doesn't really ever change. I'm not sure whether rebuilding + pushing over the existing tag is safe, or if we should bump to or something. thoughts?
If we do rebuilding/updating the image, that would help and a bunch of similar issues.
it's better/save to go with new tag
I didn't realize there were changes to the pause binary that were never actually deployed. It seems like we should probably just do that (and bump the tag).
that sounds great. Any idea when the existing 3.0 tag could be pushed again?
We shouldn't push over the 3.0 tag; we should push a 3.1 or 4.0 tag instead. I'm not sure who owns the pause container who should drive this. Maybe sig-node and/or
It seems like it should be sig-node. I'll look into it.
The new container has been built, can you test with
Its working! thanks! I was able to pull it through our artifactory proxy , Lee Verberne <:
excellent, thanks for checking. /assign


Question: "spec": { "containers": [{ "name": "test-container", <del> "image": "gcr.io/google_containers/pause-amd64:3.0" </del> <ins> "image": "gcr.io/google_containers/pause-amd64:3.1" </ins> }] } }

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: pause does not reap zombie processes using the default pause container provided by What you expected to happen: should include the up to date pause code to reap zombie processes when using docker = 1.13.1 found here How to reproduce it (as minimally and precisely as possible): a.out will leave a zombie process that is not cleaned up Anything else we need to know?: Building the pause container from master solves this issue Related: Environment: Kubernetes version (use ): 1.7.0+ Cloud provider or hardware configuration**: N/A OS (e.g. from /etc/os-release): CentOS 7 Kernel (e.g. ): 4.10.13+ Install tools: Others: docker = 1.13.1
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
i get a simliar problem when i run crond service in pod container, every day it makes some [anacron] zombie progresses UID PID PPID C STIME TTY TIME CMD root 1 0 0 Nov14 ? 00:00:00 /pause root 7 0 0 Nov14 ? 00:00:00 /usr/local/bin/dumb-init /usr/lo root 13 7 0 Nov14 ? 00:00:00 nginx: master process /usr/local nobody 21 13 0 Nov14 ? 00:00:01 nginx: worker process nobody 22 13 0 Nov14 ? 00:00:07 nginx: worker process nobody 23 13 0 Nov14 ? 00:00:07 nginx: worker process nobody 24 13 0 Nov14 ? 00:00:05 nginx: worker process root 77 1 0 Nov14 ? 00:00:00 [anacron] root 1 0 00:01 ? 00:00:00 [anacron] <defunctroot 1 0 01:01 ? 00:00:00 [anacron] <defunctroot 1 0 02:01 ?


Question: "spec": { "containers": [{ "name": "test-container", <del> "image": "gcr.io/google_containers/pause-amd64:3.0" </del> <ins> "image": "gcr.io/google_containers/pause-amd64:3.1" </ins> }] } }

00:00:00 [anacron] <defunctroot 1 0 03:01 ? 00:00:00 [anacron] <defunctroot 0 0 11:49 ? 00:00:00 /bin/bash root 0 1 19:15 ? 00:00:00 /bin/bash root 0 19:15 ? 00:00:00 ps -ef !
Do we know why this was never resolved / was never deployed?
I just fought through this issue with a rabbitmq helm chart. I had to custom build a pause container from master to fix it. The rabbitmq helm chart uses for its liveness checks. It's a thin wrapper around a call to the rabbitmq-server daemon, which is not PID 1 (/pause is PID 1). The rabbitmq-server process forks some children to do the work and then they exit. Since the pause-amd64:3.0 is not reaping, those processes stick around as zombies. The liveness check runs every 5 seconds. After about 4 hours, there are about 27K zombies and the node begins failing to launch new containers because it's starved for resources. The pause container built from current master fixes that behavior, no zombies at all in the container.
The plan is to build a new version of pause-amd64 and then update the image referenced by Kubernetes. This means the default image will change in the next Kubernetes release. We can make the new image available once the freeze lifts and we can merge .
Looks like this made it into 1.10


Question: } # k8s <del> KUBE_VERSION=${KUBE_VERSION:-"1.1.4"} </del> <ins> KUBE_VERSION=${KUBE_VERSION:-"1.1.7"} </ins> echo "Prepare kubernetes ${KUBE_VERSION} release ..." grep -q "^${KUBE_VERSION}$" binaries/.kubernetes 2>/dev/null || { curl -L https://github.com/kubernetes/kubernetes/releases/download/v${KUBE_VERSION}/kubernetes.tar.gz -o kubernetes.tar.gz

download- refers to release 1.1.4 but the latest is 1.1.7
Resolved by PR


Question: const ( csiNodeLimitUpdateTimeout  = 5 * time.Minute <del> csiPodUnschedulableTimeout = 2 * time.Minute </del> <ins> csiPodUnschedulableTimeout = 5 * time.Minute </ins> ) var _ = utils.SIGDescribe("CSI mock volume", func() {

Looks like CSI volume limit test is flaking. /sig storage /assign
This flake isn't happening in normal clusters, appears to be alpha only issue.


Question: "//staging/src/k8s.io/component-base/metrics:go_default_library", "//vendor/github.com/emicklei/go-restful:go_default_library", "//vendor/github.com/google/cadvisor/info/v1:go_default_library", <ins> "//vendor/github.com/pkg/errors:go_default_library", </ins> "//vendor/k8s.io/klog:go_default_library", ], )

What happened: In handler#handleSummary, return value from request.Request.ParseForm() is ignored. What you expected to happen: The error return should be checked.
/sig node
Hi this seems to be easy to fix. Could I solve this?
Please go ahead.


Question: return 0 } <del> func (ds *dockerService) getSecurityOpts(containerName string, sandboxConfig *runtimeapi.PodSandboxConfig, separator rune) ([]string, error) { hasSeccompSetting := false annotations := sandboxConfig.GetAnnotations() if _, ok := annotations[v1.SeccompContainerAnnotationKeyPrefix+containerName]; !ok { _, hasSeccompSetting = annotations[v1.SeccompPodAnnotationKey] } else { hasSeccompSetting = true </del> <ins> func (ds *dockerService) getSecurityOpts(seccompProfile string, separator rune) ([]string, error) { if seccompProfile != "" { glog.Warningf("seccomp annotations are not supported on windows") </ins> } <del> if hasSeccompSetting { glog.Warningf("seccomp annotations found, but it is not supported on windows") } </del> return nil, nil }

seems to belong to sig node PTAL and fix ASAP
/assign as it seems like having something to do with


Question: setHugepages() ginkgo.By("restarting kubelet to pick up pre-allocated hugepages") <del> restartKubelet() </del> <ins> // stop the kubelet and wait until the server will restart it automatically stopKubelet() // wait until the kubelet health check will fail gomega.Eventually(func() bool { return kubeletHealthCheck(kubeletHealthCheckURL) }, time.Minute, time.Second).Should(gomega.BeFalse()) // wait until the kubelet health check will pass gomega.Eventually(func() bool { return kubeletHealthCheck(kubeletHealthCheckURL) }, 2*time.Minute, 10*time.Second).Should(gomega.BeTrue()) </ins> waitForHugepages()

ci-kubernetes-node-kubelet-serial-hugepages when start the pod with the resources requests that contain multiple hugepages resources should set correct hugetlb mount and limit under the container cgroup Unexpected error: "pod ran to completion", Multiple sizes huge pages is going to be graduated to GA, so having this test job fixed is quite important.
/sig node
/assign
/triage accepted
/cc
I did a small analysis and I can see that the pod failed to start, I can not find any failure messages under the kubelet log, so it really hard to say why it failed.
were you able to reproduce it locally? btw, there is a "right way" to fix this - to fix dynamic update of a hugepage resources. This would eliminate need in a kubelet restart. There is even a PR for that: , we just need to make it working again.
after conversation with on slack I'd say we should stick to the current procedure - restarting kubelet when hugepages allocation changes. So, we should investigate why kubelet doesn't restart properly in this test case and fix it.
Nope, I failed to reproduce it locally, also under the CI, it does not appear often. The best thing will be to describe the pod in case if it failed, so we will see the reason.
under the code we have so from the log , I can see that the kubelet received the pod before the kubelet stopped, so the problem that pod dropped to Failed state because of the kubelet stop.
So some additional details, the one who responsible to restart the kubelet again, is the server service: restart the kubelet service are waiting for hugepages to be available are starting the pod server service restart the kubelet again because it was not healthy for too long - failed to start because of the kubelet restart
great findings! Should we just stop the kubelet and let the server service to start it again? We can also check if the kubelet is restarted successfully before starting the pod.
I tried it under the PR , also I unsure how can we check if the kubelet restarted successfully. All tests that I wrote before I checked if the node has NotReady state and waited until it will be Ready, but in case of the restart probably the node will always have Ready state because the time period is very short. I will continue to investigate and will check locally how can I fix it.


Question: setHugepages() ginkgo.By("restarting kubelet to pick up pre-allocated hugepages") <del> restartKubelet() </del> <ins> // stop the kubelet and wait until the server will restart it automatically stopKubelet() // wait until the kubelet health check will fail gomega.Eventually(func() bool { return kubeletHealthCheck(kubeletHealthCheckURL) }, time.Minute, time.Second).Should(gomega.BeFalse()) // wait until the kubelet health check will pass gomega.Eventually(func() bool { return kubeletHealthCheck(kubeletHealthCheckURL) }, 2*time.Minute, 10*time.Second).Should(gomega.BeTrue()) </ins> waitForHugepages()

So the only reliable solution that I found, it just trying to start the pod number of times. I tested it 5 times and the test passed. Can you please review
Can we access kubelet service from hugepage test cases? If we can we can use its APIs like or to check if kubelet is ready after the restart. Overall I have a feeling of duplication test/e2enode/services functionality in This is a main reason for this and similar bugs I believe. It would be nice to get rid of this duplication if possible. That would nicely isolate kubelet service controls in one place and eliminate conflicts like this one. wdyt?
not really, the used mostly to start all needed components to emulate the cluster(not only the kubelet) and the test does not have access to it. I think you can separate the readiness check under the importable method, but it worth the separate PR.
I updated the PR to check the healthiness of the kubelet and looks like it works just fine.
thank you. it looks good to me so far. I'll add lgtm label when is merged and tests are all green.
the job is still flaky. can you reopen this issue?
I opened a different one - , the issue can affect any job, not only the hugepages one.
I can open a fix to W/A an issue(start the pod until it will succeed), it should fix all problems, WDYT?
i think it's ok as a workaround, until is fixed
would it be better to wait until api server reports correct resource value before running the pod?
We are waiting for it, the problem that the scheduler and the kubelet see a different picture regarding resources requested by pods.
ok. In this case restarting the pod until it succeeds seems reasonable to me to fix the flakiness until is fixed.
any progress on the workaround fix?


Question: match: BASEIMAGE?=k8s.gcr.io/build-image/debian-base-s390x:[a-zA-Z]+-v((([0-9]+).([0-9]+).([0-9]+)(?:-([0-9a-zA-Z-]+(?:.[0-9a-zA-Z-]+)*))?)(?:+([0-9a-zA-Z-]+(?:.[0-9a-zA-Z-]+)*))?) - name: "k8s.gcr.io/debian-iptables: dependents" <del> version: buster-v1.4.0 </del> <ins> version: buster-v1.5.0 </ins> refPaths: - path: build/common.sh match: debian_iptables_version=

The based container images for are shipped with the wrong architecture within the manifest if not amd64: Refers to We updated the images a few minutes ago to buster-v1.3.0, which are now ready to be used as base for the multi-architecture images. If the images contain the right architectures, then it may magically happen that the kube-proxy images are fixed, too. This is because we already use the build syntax for it: /cc
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig release


Question: VolumePlugins:      volumePlugins(), TLSOptions:         nil, OOMAdjuster:        oom.NewFakeOOMAdjuster(), <del> Mounter:            mount.New("" /* default mount path */), </del> <ins> Mounter:            &mount.FakeMounter{}, </ins> Subpather:          &subpath.FakeSubpath{}, HostUtil:           hostutil.NewFakeHostUtil(nil), }

There are CVEs happening in different images - we already migrated Kubmeark from busybox in , but we should go one step further and migrate the base image to distroless. There are some things that would need to be changed mentioned here: /sig scalability
Can I work on this? /assign
/remove-help
- sure that would be great.
Hi based on the discussion in , the kubemark image has dependency on and maybe more other utilities. So it seems impossible to migrate the base image to distroless? Or do we need something like debian-hyperkube-base?
The mount one we may get rid of - basically we should be able to switch to FakeMounter:


Question: )}, wantErr: "duplicate profile with scheduler name "foo"", }, <ins> { name: "With extenders", opts: []Option{ WithProfiles( schedulerapi.KubeSchedulerProfile{ SchedulerName: "default-scheduler", Plugins: &schedulerapi.Plugins{ QueueSort: schedulerapi.PluginSet{Enabled: []schedulerapi.Plugin{{Name: "PrioritySort"}}}, Bind:      schedulerapi.PluginSet{Enabled: []schedulerapi.Plugin{{Name: "DefaultBinder"}}}, }, }, ), WithExtenders( schedulerapi.Extender{ URLPrefix: "http://extender.kube-system/", }, ), }, wantProfiles:  []string{"default-scheduler"}, wantExtenders: []string{"http://extender.kube-system/"}, }, </ins> } for _, tc := range cases {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --here is supposed to be set through , but is not called from anywhere except tests, so always returns . should be set through here. Am I missing something? N/A None Kubernetes version (use ): master () Cloud provider or hardware configuration: N/A OS (e.g: ): N/A Kernel (e.g. ): N/A Install tools: N/A Network plugin and version (if this is a network-related bug): N/A Others: None
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scheduling
Thanks for pointing that out. The plumbing of extenders to the framework impact calling extenders on preemption. That seems to be a path not covered by tests end to end. Can you submit a PR to fix this? as you mentioned we need to pass the extenders on profile creation. We need to patch the fix back to v1.21 and v1.20 (the release where this was refactored). /cc
I have submitted a PR for a fix.


Question: // Flush is part of Interface.  Currently we delete IPVS services one by one func (runner *runner) Flush() error { <del> vss, err := runner.GetVirtualServers() if err != nil { return err } for _, vs := range vss { err := runner.DeleteVirtualServer(vs) // TODO: aggregate errors? if err != nil { return err } } return nil </del> <ins> return runner.ipvsHandle.Flush() </ins> } // AddRealServer is part of Interface.

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind feature What happened: Currently, we flush ipvs services by deleting them one by one. That's because there was no flush API in libnetwork. Now, libnetwork supports flush API, I think it's time to rsync. What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration**: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig network
/area kube-proxy
/assign


Question: "k8s.io/kubernetes/pkg/api/v1" dockertypes "github.com/docker/engine-api/types" <ins> dockercontainer "github.com/docker/engine-api/types/container" </ins> ) <ins> // These two functions are OS specific (for now at least) func updateHostConfig(config *dockercontainer.HostConfig) { // There is no /etc/resolv.conf in Windows, DNS and DNSSearch options would have to be passed to Docker runtime instead hc.DNS = opts.DNS hc.DNSSearch = opts.DNSSearch // MemorySwap == -1 is not currently supported in Docker 1.14 on Windows // https://github.com/docker/docker/blob/master/daemon/daemon_windows.go#L175 hc.Resources.MemorySwap = 0 } func DefaultMemorySwap() int64 { return 0 } </ins> func getContainerIP(container *dockertypes.ContainerJSON) string { if container.NetworkSettings != nil { for _, network := range container.NetworkSettings.Networks {

What happened: Latest kubelet + Windows Docker version 1.14.0-dev refused to start any containers, error output: "invalid option: Windows does not support MemorySwap" How to reproduce it (as minimally and precisely as possible): It's always reproducible with Windows Docker 1.14.0-dev + latest kubelet code base. Anything else do we need to know: It didn't happen for Windows Docker 1.12.2-cs2-ws-beta. The reason behind the issue was because new code was for Windows Docker to check Resources parameters, it doesn't accept a non-zero value for MemorySwap: But kubelet was default to set an unlimited (-1) value. The issue was gone in my setup after applied a private fix to disable MemorySwap by set it to zero for Windows.


Question: const ( DefaultEtcdPathPrefix = "/registry" <ins> globalTimeout         = time.Minute </ins> ) // StorageDestinations is a mapping from API group & resource to

this job failed with It seems it is imssing some dependency The job history confirms it is totally broken Originally posted by in
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig testing


Question: w.WriteHeader(http.StatusNotFound) } <del> func startComponents(manifestURL, apiVersion string) (string, string) { </del> <ins> func startComponents(firstManifestURL, secondManifestURL, apiVersion string) (string, string) { </ins> // Setup servers := []string{} glog.Infof("Creating etcd client pointing to %v", servers)

Currently it assumes it can read ContainerManifest - it should try both
Why not hold on this until we have Kubelet-read-Pod in?
Also the HTTP source has the same problem.
It should read Pod. Changed the title.
Ideally, I'd like all Pods created directly via Kubelet to be to etcd and surfaced via the API. If updates are made via the API, I'd like those to propagate back to the node. We can discuss whether the config file should be updated or ignored. At some point, I'd like Kubelet to persistently cache pods received from the apiserver, perhaps by just writing them all out as config files.
is the issue to sync local pods back to apiserver.
this is somewhat related to other issues we talked about.
Detail: We want locally specified pods to be able to use YAML config files, but we also want to rip YAML out of the API server . The config-file behavior should be as similar to the kubectl experience. Kubelet could use parts of the kubectl library to translate YAML to JSON. Vision: Eventually, I'd like Kubelet to use the apiserver library and support exactly the same Pod API as the master, at which time I'd like kubectl be able to create pods directly on Kubelets, as well as reading from config files. I also expect to have a per-node controller . I'd like to be able to use the per-node controller to update pods started via config files (or direct kubectl-kubelet pods), overriding those original files. For availability and self-hosting, we'll eventually need local checkpointing , ideally just using serialized JSON for each object. If Kubelet used apiserver, this could be done using a custom registry implementation.
I think that is a prerequisite for it.
If noone picks it up, I'm going to start working on it tomorrow.
thanks, wojtek-t
fixed it for http channel


Question: Priorities []PriorityPolicy // Holds the information to communicate with the extender(s) ExtenderConfigs []ExtenderConfig <ins> // RequiredDuringScheduling affinity is not symmetric, but there is an implicit PreferredDuringScheduling affinity rule // corresponding to every RequiredDuringScheduling affinity rule. // HardPodAffinitySymmetricWeight represents the weight of implicit PreferredDuringScheduling affinity rule, in the range 1-100. HardPodAffinitySymmetricWeight int </ins> } type PredicatePolicy struct {

( noticed this) HardPodAffinitySymmetricWeight should be in the , not in KubeSchedulerConfiguration If/when fixing this, we need to be careful to do it in a backward-compatilble way.
Do you mean ? It seems scheduler policy config's args can not passed to FitPredicates; it need to update related interface. For the : mark to be deprecated and set scheduler policy config to HardPodAffinitySymmetricWeight if policy config is empty.
Are you working on this? if no, I can take it.
, nop; please go ahead.
/ / / , When reviewing the PR, there's one question on : the default value of is 1, which means the weight (1) will always add to priorities if term matched (), does it our expectation? And the user has to set it to 0 to disable it. IMO, the default value of should be 0.
The value 1 (instead of 0) was chosen so that if you say A must run with B, there is an automatic preference that B should run with A. This seems like reasonable behavior.
Thanks very much; reasonable case to me. But it seems we can not disable it: if set to 0, the default funcs will set it to default (1); similar result to un-set. Anyway, no user complain this now :).


Question: func addConversionFuncs(scheme *runtime.Scheme) { // Add non-generated conversion functions err := scheme.AddConversionFuncs( <del> v1.Convert_api_ServiceSpec_To_v1_ServiceSpec, </del> <ins> v1.Convert_v1_DeleteOptions_To_api_DeleteOptions, v1.Convert_api_DeleteOptions_To_v1_DeleteOptions, v1.Convert_v1_ExportOptions_To_api_ExportOptions, v1.Convert_api_ExportOptions_To_v1_ExportOptions, v1.Convert_v1_List_To_api_List, v1.Convert_api_List_To_v1_List, v1.Convert_v1_ListOptions_To_api_ListOptions, v1.Convert_api_ListOptions_To_v1_ListOptions, v1.Convert_v1_ObjectFieldSelector_To_api_ObjectFieldSelector, v1.Convert_api_ObjectFieldSelector_To_v1_ObjectFieldSelector, v1.Convert_v1_ObjectMeta_To_api_ObjectMeta, v1.Convert_api_ObjectMeta_To_v1_ObjectMeta, v1.Convert_v1_ObjectReference_To_api_ObjectReference, v1.Convert_api_ObjectReference_To_v1_ObjectReference, v1.Convert_v1_OwnerReference_To_api_OwnerReference, v1.Convert_api_OwnerReference_To_v1_OwnerReference, v1.Convert_v1_Service_To_api_Service, v1.Convert_api_Service_To_v1_Service, v1.Convert_v1_ServiceList_To_api_ServiceList, v1.Convert_api_ServiceList_To_v1_ServiceList, v1.Convert_v1_ServicePort_To_api_ServicePort, v1.Convert_api_ServicePort_To_v1_ServicePort, v1.Convert_v1_ServiceProxyOptions_To_api_ServiceProxyOptions, v1.Convert_api_ServiceProxyOptions_To_v1_ServiceProxyOptions, </ins> v1.Convert_v1_ServiceSpec_To_api_ServiceSpec, <ins> v1.Convert_api_ServiceSpec_To_v1_ServiceSpec, v1.Convert_v1_ServiceStatus_To_api_ServiceStatus, v1.Convert_api_ServiceStatus_To_v1_ServiceStatus, </ins> ) if err != nil { // If one of the conversion functions is malformed, detect it immediately.

I started the federation-apiserver and went to and got the following output: Expected: I should have got an empty v1.ServiceList Any idea on what we are doing wrong? Have you guys seen this error?
I think your deep copiers or conversion functions are not getting registered properly. If you look at my change in , some places are inappropriately registering with api.Scheme instead of the passed in scheme. Maybe you have that problem or a similar one, since I think you guys copied some code. , Nikhil Jindal wrote:
Deep copy generation was slightly broken as well - you may want to double check that they are registered. On May 23, 2016, at 5:50 PM, Daniel Smith wrote: I think your deep copiers or conversion functions are not getting registered properly. If you look at my change in , some places are inappropriately registering with api.Scheme instead of the passed in scheme. Maybe you have that problem or a similar one, since I think you guys copied some code. , Nikhil Jindal wrote: — You are receiving this because you are on a team that was mentioned. Reply to this email directly or view it on GitHub
I got the same error with latest branch, as I was swamping on service controller, so I use a previous version and worked around it.
deep copies were indeed missing, but adding them hasnt fixed the issue. Will debug more. is my PR to add the deep copies cc Please feel free to file issues for any such problem you face. We need to fix all these issues. Its good if we know about them early. Thanks!
fixes this


Question: "os" "k8s.io/kubernetes/pkg/api" <ins> "k8s.io/kubernetes/pkg/api/meta" </ins> "k8s.io/kubernetes/pkg/api/unversioned" "k8s.io/kubernetes/pkg/apimachinery/registered" "k8s.io/kubernetes/pkg/kubectl"

BUG REPORT When using with multiple files (like when referencing a directory in ), the json or yaml output is not a valid list format, and when fed to just the first file gets created. Kubernetes version (use ): latest master What happened: only the first file in directory gets created. What you expected to happen: all files in directory must be created. How to reproduce it (as minimally and precisely as possible):


Question: } // DiskIsAttached returns if disk is attached to the VM using controllers supported by the plugin. <del> func (vs *VSphere) DiskIsAttached(volPath string, nodeName k8stypes.NodeName) (bool, error) { diskIsAttachedInternal := func(volPath string, nodeName k8stypes.NodeName) (bool, error) { </del> <ins> func (vs *VSphere) DiskIsAttached(volPath string, nodeName k8stypes.NodeName) (bool, string, error) { diskIsAttachedInternal := func(volPath string, nodeName k8stypes.NodeName) (bool, string, error) { </ins> var vSphereInstance string if nodeName == "" { vSphereInstance = vs.hostName

When user creates a static PV using folder name in path: on vSAN backed datastore then detach disk does not work because check fails - The code expects folder name to be UUID but it does not and hence it determines disk is not attached to the node and hence no detach is ever performed. /sig storage cc


Question: terminating := apiextensions.IsCRDConditionTrue(crd, apiextensions.Terminating) <del> crdInfo, err := r.getOrCreateServingInfoFor(crd) </del> <ins> crdInfo, err := r.getOrCreateServingInfoFor(crd.UID, crd.Name) if apierrors.IsNotFound(err) { r.delegate.ServeHTTP(w, req) return } </ins> if err != nil { http.Error(w, err.Error(), http.StatusInternalServerError) return } <ins> if !hasServedCRDVersion(crdInfo.spec, requestInfo.APIVersion) { r.delegate.ServeHTTP(w, req) return } </ins> verb := strings.ToUpper(requestInfo.Verb) resource := requestInfo.Resource

Which jobs are failing: pull-kubernetes-integration Which test(s) are failing: TestWebhookConverterWithDefaulting /assign /priority important-soon /area custom-resources /sig api-machinery /kind flake /milestone v1.16
/cc
several of the webhook converter tests have similar flakes. it looks like the wait for storage version to become effective times out occasionally Rare flakes seen in all of these tests: TestWebhookConverterWithWatchCache/nontrivial-converter TestWebhookConverterWithWatchCache/metadata-mutating-converter TestWebhookConverterWithDefaulting/metadata-mutating-converter TestWebhookConverterWithWatchCache/noop-converter TestWebhookConverterWithDefaulting/empty-response TestWebhookConverterWithoutWatchCache/nontrivial-converter/check-2 TestWebhookConverterWithoutWatchCache/noop-converter/check-0/v1beta2 TestWebhookConverterWithoutWatchCache/nontrivial-converter/check-0/v1beta2
cc
Yikes, I’ll sort it out.
iiuc sending an empty patch gets short-circuited and doesn't bump CR generation, even if the storage version gets changed, which means we do not re-write to etcd and go through the encoding path this would be racing and doing noop's if the first pass of didn't get the expected storage version I will send a fix to make the patch do actual mutation
It doesn't bump generation, but the bytes to persist in etcd would still be different if the storage version had changed, so they should persist. You can verify that by changing the storage version on a CRD yourself, do an empty patch, and see if the resourceVersion changes.
you're right. Generation is for non-metadata change only, and the RV changes. I queried etcd locally and the apiVersion did change for an empty patch (sidetracking: another observation is after changing the storage version (say v1 -v2; noop convertor), empty-patching v2 endpoint would bump generation, while empty-patching v1 endpoint wouldn't)
another theory: our cached storage strategy could be stale if the CRD informer merges multiple events together (e.g. the informer does a re-list).


Question: terminating := apiextensions.IsCRDConditionTrue(crd, apiextensions.Terminating) <del> crdInfo, err := r.getOrCreateServingInfoFor(crd) </del> <ins> crdInfo, err := r.getOrCreateServingInfoFor(crd.UID, crd.Name) if apierrors.IsNotFound(err) { r.delegate.ServeHTTP(w, req) return } </ins> if err != nil { http.Error(w, err.Error(), http.StatusInternalServerError) return } <ins> if !hasServedCRDVersion(crdInfo.spec, requestInfo.APIVersion) { r.delegate.ServeHTTP(w, req) return } </ins> verb := strings.ToUpper(requestInfo.Verb) resource := requestInfo.Resource

Two scenarios: -create-update events merged into an update event: strategy with UID:a was created on demand for CRD Foo Foo was deleted and recreated strategy with UID:b was created on demand for the new Foo new Foo got updated (e.g. switching storage version) the strategy with UID:b is supposed to be deleted and , to reflect the last update. But if the CRD informer merges the delete-create-update events into , the strategy with UID:a will be deleted, but the strategy with UID:b won't. -update events merged into a create event: strategy with UID:a was created on demand for CRD Foo Foo was updated (e.g. switching storage version) the strategy with UID:a is supposed to be deleted and , to reflect the last update. But if the CRD informer merges the create-update events into , the strategy with UID:a won't be deleted (as we don't react on create events). another race found in , which could happen in a single update.
I think I :


Question: // Check that request <= limit. limitQuantity, exists := requirements.Limits[resourceName] if exists { <del> // For GPUs, not only requests can't exceed limits, they also can't be lower, i.e. must be equal. </del> <ins> // For non overcommitable resources, not only requests can't exceed limits, they also can't be lower, i.e. must be equal. </ins> if quantity.Cmp(limitQuantity) != 0 && !helper.IsOvercommitAllowed(resourceName) { allErrs = append(allErrs, field.Invalid(reqPath, quantity.String(), fmt.Sprintf("must be equal to %s limit", resourceName))) } else if quantity.Cmp(limitQuantity) > 0 { allErrs = append(allErrs, field.Invalid(reqPath, quantity.String(), fmt.Sprintf("must be less than or equal to %s limit", resourceName))) } <del> } else if resourceName == core.ResourceNvidiaGPU { allErrs = append(allErrs, field.Invalid(reqPath, quantity.String(), fmt.Sprintf("must be equal to %s request", core.ResourceNvidiaGPU))) </del> <ins> } else if !helper.IsOvercommitAllowed(resourceName) { allErrs = append(allErrs, field.Required(limPath, "Limit must be set for non overcommitable resources")) </ins> } }

/kind bug /sig node /area hw-accelerators What happened: In devices are allocated from a function called . This function inspects only container resource limits, ignoring container resource requests. This is a usability problem because it is valid to omit limits for extended resources. If limit is omitted for a device resource, then the scheduler properly places the pod with respect to available node resources. However, the device manager never dispatches to the device plugin for that resource. The result is that the user application cannot access the requested device from inside the container. What you expected to happen: Following the documented behavior of extended resources, I expected that specifying a container resource request for a device would cause the device subsystem to properly configure access for my workload. How to reproduce it (as minimally and precisely as possible): Launch an alpha GKE cluster with GPU support enabled. Attempt to run any program that accesses the GPU from a container where the resource is specified in requests but omitted from limits. Environment: Kubernetes version (use ): v1.8+ Cloud provider or hardware configuration: device plugins enabled, devices available.
Related issue:
cc
/assign
FYI. The said issue has been fixed in . There are several valid cases when requesting extended resources: The user can specify 'limits' only. The API server will automatically set 'requests' to be equal to 'limits'. The user specify 'limits' and 'requests', and they are equal. The user specify 'requests' only. We can either reject the request saying that 'limits' must be set, or implicitly treat 'limits' as equal to 'requests'.
Hi went with to begin since it was ready to go and smaller (easier to revert if necessary.) Hopefully it's easy to rebase your work on top of that. Sorry for the inconvenience.
I feel the documented behavior is incorrect. The original intention was to always require for extended resources until we have better APIs to express bursting scenarios. I vote for fixing the documentation and fixing the validation logic and having kubelet use limits only.
/reopen


Question: return nil } <ins> func TestUpgradeServicePreferToDualStack(t *testing.T) { // Create an IPv4 only dual stack control-plane serviceCIDR := "192.168.0.0/24" defer featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, features.IPv6DualStack, true)() cfg := framework.NewIntegrationTestControlPlaneConfig() _, cidr, err := net.ParseCIDR(serviceCIDR) if err != nil { t.Fatalf("bad cidr: %v", err) } cfg.ExtraConfig.ServiceIPRange = *cidr _, s, closeFn := framework.RunAnAPIServer(cfg) client := clientset.NewForConfigOrDie(&restclient.Config{Host: s.URL}) // Wait until the default "kubernetes" service is created. if err = wait.Poll(250*time.Millisecond, time.Minute, func() (bool, error) { _, err := client.CoreV1().Services(metav1.NamespaceDefault).Get(context.TODO(), "kubernetes", metav1.GetOptions{}) if err != nil && !apierrors.IsNotFound(err) { return false, err } return !apierrors.IsNotFound(err), nil }); err != nil { t.Fatalf("creating kubernetes service timed out") } preferDualStack := v1.IPFamilyPolicyPreferDualStack svc := &v1.Service{ ObjectMeta: metav1.ObjectMeta{ Name: "svc-prefer-dual", }, Spec: v1.ServiceSpec{ Type:           v1.ServiceTypeClusterIP, ClusterIPs:     nil, IPFamilies:     nil, IPFamilyPolicy: &preferDualStack, Ports: []v1.ServicePort{ { Name:       "svc-port-1", Port:       443, TargetPort: intstr.IntOrString{IntVal: 443}, Protocol:   "TCP", }, }, }, } // create the service _, err = client.CoreV1().Services(metav1.NamespaceDefault).Create(context.TODO(), svc, metav1.CreateOptions{}) if err != nil { t.Fatalf("Unexpected error: %v", err) } // validate the service was created correctly if it was not expected to fail svc, err = client.CoreV1().Services(metav1.NamespaceDefault).Get(context.TODO(), svc.Name, metav1.GetOptions{}) if err != nil { t.Fatalf("Unexpected error to get the service %s %v", svc.Name, err) } if err := validateServiceAndClusterIPFamily(svc, []v1.IPFamily{v1.IPv4Protocol}); err != nil { t.Fatalf("Unexpected error validating the service %s %v", svc.Name, err) } // reconfigure the apiserver to be dual-stack closeFn() secondaryServiceCIDR := "2001:db8:1::/48" _, secCidr, err := net.ParseCIDR(secondaryServiceCIDR) if err != nil { t.Fatalf("bad cidr: %v", err) } cfg.ExtraConfig.SecondaryServiceIPRange = *secCidr _, s, closeFn = framework.RunAnAPIServer(cfg) defer closeFn() client = clientset.NewForConfigOrDie(&restclient.Config{Host: s.URL}) // Wait until the default "kubernetes" service is created. if err = wait.Poll(250*time.Millisecond, time.Minute, func() (bool, error) { _, err := client.CoreV1().Services(metav1.NamespaceDefault).Get(context.TODO(), "kubernetes", metav1.GetOptions{}) if err != nil && !apierrors.IsNotFound(err) { return false, err } return !apierrors.IsNotFound(err), nil }); err != nil { t.Fatalf("creating kubernetes service timed out") } // validate the service was created correctly if it was not expected to fail svc, err = client.CoreV1().Services(metav1.NamespaceDefault).Get(context.TODO(), svc.Name, metav1.GetOptions{}) if err != nil { t.Fatalf("Unexpected error to get the service %s %v", svc.Name, err) } // service should remain single stack if err = validateServiceAndClusterIPFamily(svc, []v1.IPFamily{v1.IPv4Protocol}); err != nil { t.Fatalf("Unexpected error validating the service %s %v", svc.Name, err) } } func TestDowngradeServicePreferToDualStack(t *testing.T) { // Create a dual stack control-plane serviceCIDR := "192.168.0.0/24" secondaryServiceCIDR := "2001:db8:1::/48" defer featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, features.IPv6DualStack, true)() dualStackCfg := framework.NewIntegrationTestControlPlaneConfig() _, cidr, err := net.ParseCIDR(serviceCIDR) if err != nil { t.Fatalf("bad cidr: %v", err) } dualStackCfg.ExtraConfig.ServiceIPRange = *cidr _, secCidr, err := net.ParseCIDR(secondaryServiceCIDR) if err != nil { t.Fatalf("bad cidr: %v", err) } dualStackCfg.ExtraConfig.SecondaryServiceIPRange = *secCidr _, s, closeFn := framework.RunAnAPIServer(dualStackCfg) client := clientset.NewForConfigOrDie(&restclient.Config{Host: s.URL}) // Wait until the default "kubernetes" service is created. if err = wait.Poll(250*time.Millisecond, time.Minute, func() (bool, error) { _, err := client.CoreV1().Services(metav1.NamespaceDefault).Get(context.TODO(), "kubernetes", metav1.GetOptions{}) if err != nil && !apierrors.IsNotFound(err) { return false, err } return !apierrors.IsNotFound(err), nil }); err != nil { t.Fatalf("creating kubernetes service timed out") } preferDualStack := v1.IPFamilyPolicyPreferDualStack svc := &v1.Service{ ObjectMeta: metav1.ObjectMeta{ Name: "svc-prefer-dual01", }, Spec: v1.ServiceSpec{ Type:           v1.ServiceTypeClusterIP, ClusterIPs:     nil, IPFamilies:     nil, IPFamilyPolicy: &preferDualStack, Ports: []v1.ServicePort{ { Name:       "svc-port-1", Port:       443, TargetPort: intstr.IntOrString{IntVal: 443}, Protocol:   "TCP", }, }, }, } // create the service _, err = client.CoreV1().Services(metav1.NamespaceDefault).Create(context.TODO(), svc, metav1.CreateOptions{}) if err != nil { t.Fatalf("Unexpected error: %v", err) } // validate the service was created correctly if it was not expected to fail svc, err = client.CoreV1().Services(metav1.NamespaceDefault).Get(context.TODO(), svc.Name, metav1.GetOptions{}) if err != nil { t.Fatalf("Unexpected error to get the service %s %v", svc.Name, err) } if err := validateServiceAndClusterIPFamily(svc, []v1.IPFamily{v1.IPv4Protocol, v1.IPv6Protocol}); err != nil { t.Fatalf("Unexpected error validating the service %s %v", svc.Name, err) } // reconfigure the apiserver to be sinlge stack closeFn() // reset secondary var emptyCidr net.IPNet dualStackCfg.ExtraConfig.SecondaryServiceIPRange = emptyCidr _, s, closeFn = framework.RunAnAPIServer(dualStackCfg) defer closeFn() client = clientset.NewForConfigOrDie(&restclient.Config{Host: s.URL}) // Wait until the default "kubernetes" service is created. if err = wait.Poll(250*time.Millisecond, time.Minute, func() (bool, error) { _, err := client.CoreV1().Services(metav1.NamespaceDefault).Get(context.TODO(), "kubernetes", metav1.GetOptions{}) if err != nil && !apierrors.IsNotFound(err) { return false, err } return !apierrors.IsNotFound(err), nil }); err != nil { t.Fatalf("creating kubernetes service timed out") } // validate the service is still there. svc, err = client.CoreV1().Services(metav1.NamespaceDefault).Get(context.TODO(), svc.Name, metav1.GetOptions{}) if err != nil { t.Fatalf("Unexpected error to get the service %s %v", svc.Name, err) } // service should be single stack if err = validateServiceAndClusterIPFamily(svc, []v1.IPFamily{v1.IPv4Protocol, v1.IPv6Protocol}); err != nil { t.Fatalf("Unexpected error validating the service %s %v", svc.Name, err) } } </ins>

1) start with a single stack cluster 2) create a PreferDualStack service 3) configure API server with a 2nd service CIDR 4) Confirm the above service is not dual-stack 5) Add a label to the above service 6) Observe that the service becomes dual-stack I think the correct answer is to have a "repair loop" that upgrades (and downgrades!) PreferDualStack Services to match the config. I think this is a GA blocker.
/triage accepted The current thought is: of type with are special new breed of api objects that I don't think we have any similar precedent in the apis, i could be wrong. Their behavior is tied to how the cluster is configured. Thus their behavior must change as the cluster behavior change. a way the user is stating . If it is cluster then i am good with two ClusterIPs. If it is single stack cluster then I am also good with a single ClusterIP. One can assume that if the user was ok was with getting a additional IP when possible then they must be ok with taking it away when it is no longer possible. Based on the above I believe best course of action is create a new repair loop or piggyback on any of the existing (or better: fix it in rest stack during list and get calls). The new logic takes care of matching to the .
reproducer and regression test if somebody takes this, feel free to add it
ack - Scheduled to tackle 3rd week of May. Will definitely include that patch.


Question: }) }) <ins> ginkgo.It("should ensure an IP overlapping both IPBlock.CIDR and IPBlock.Except is allowed [Feature:NetworkPolicy]", func() { protocolUDP := v1.ProtocolUDP // Getting podServer's status to get podServer's IP, to create the CIDR with except clause podServerStatus, err := f.ClientSet.CoreV1().Pods(f.Namespace.Name).Get(context.TODO(), podServer.Name, metav1.GetOptions{}) if err != nil { framework.ExpectNoError(err, "Error occurred while getting pod status.") } podServerAllowCIDR := fmt.Sprintf("%s/24", podServerStatus.Status.PodIP) podServerIP := fmt.Sprintf("%s/32", podServerStatus.Status.PodIP) // Exclude podServer's IP with an Except clause podServerExceptList := []string{podServerIP} // Create NetworkPolicy which blocks access to podServer with except clause. policyAllowCIDRWithExceptServerPod := &networkingv1.NetworkPolicy{ ObjectMeta: metav1.ObjectMeta{ Namespace: f.Namespace.Name, Name:      "deny-client-a-via-except-cidr-egress-rule", }, Spec: networkingv1.NetworkPolicySpec{ // Apply this policy to the client. PodSelector: metav1.LabelSelector{ MatchLabels: map[string]string{ "pod-name": "client-a", }, }, PolicyTypes: []networkingv1.PolicyType{networkingv1.PolicyTypeEgress}, // Allow traffic to only one CIDR block except subnet which includes Server. Egress: []networkingv1.NetworkPolicyEgressRule{ { Ports: []networkingv1.NetworkPolicyPort{ // Allow DNS look-ups { Protocol: &protocolUDP, Port:     &intstr.IntOrString{Type: intstr.Int, IntVal: 53}, }, }, }, { To: []networkingv1.NetworkPolicyPeer{ { IPBlock: &networkingv1.IPBlock{ CIDR:   podServerAllowCIDR, Except: podServerExceptList, }, }, }, }, }, }, } policyAllowCIDRWithExceptServerPodObj, err := f.ClientSet.NetworkingV1().NetworkPolicies(f.Namespace.Name).Create(context.TODO(), policyAllowCIDRWithExceptServerPod, metav1.CreateOptions{}) framework.ExpectNoError(err, "Error occurred while creating policy: policyAllowCIDRWithExceptServerPod.") ginkgo.By("Creating client-a which should not be able to contact the server.", func() { testCannotConnect(f, f.Namespace, "client-a", service, 80) }) // Create NetworkPolicy which allows access to the podServer using podServer's IP in allow CIDR. policyAllowCIDRServerPod := &networkingv1.NetworkPolicy{ ObjectMeta: metav1.ObjectMeta{ Namespace: f.Namespace.Name, Name:      "allow-client-a-via-cidr-egress-rule", }, Spec: networkingv1.NetworkPolicySpec{ // Apply this policy to the client. PodSelector: metav1.LabelSelector{ MatchLabels: map[string]string{ "pod-name": "client-a", }, }, PolicyTypes: []networkingv1.PolicyType{networkingv1.PolicyTypeEgress}, // Allow traffic to only one CIDR block which includes Server. Egress: []networkingv1.NetworkPolicyEgressRule{ { Ports: []networkingv1.NetworkPolicyPort{ // Allow DNS look-ups { Protocol: &protocolUDP, Port:     &intstr.IntOrString{Type: intstr.Int, IntVal: 53}, }, }, }, { To: []networkingv1.NetworkPolicyPeer{ { IPBlock: &networkingv1.IPBlock{ CIDR: podServerIP, }, }, }, }, }, }, } policyAllowCIDRServerPod, err = f.ClientSet.NetworkingV1().NetworkPolicies(f.Namespace.Name).Create(context.TODO(), policyAllowCIDRServerPod, metav1.CreateOptions{}) framework.ExpectNoError(err, "Error occurred while creating policy: policyAllowCIDRServerPod.") defer cleanupNetworkPolicy(f, policyAllowCIDRServerPod) ginkgo.By("Creating client-a which should now be able to contact the server.", func() { testCanConnect(f, f.Namespace, "client-a", service, 80) }) ginkgo.By("Deleting the network policy with except podServer IP which disallows access to podServer.") cleanupNetworkPolicy(f, policyAllowCIDRWithExceptServerPodObj) ginkgo.By("Creating client-a which should still be able to contact the server after deleting the network policy with except clause.", func() { testCanConnect(f, f.Namespace, "client-a", service, 80) }) // Recreate the NetworkPolicy which contains the podServer's IP in the except list. policyAllowCIDRWithExceptServerPod, err = f.ClientSet.NetworkingV1().NetworkPolicies(f.Namespace.Name).Create(context.TODO(), policyAllowCIDRWithExceptServerPod, metav1.CreateOptions{}) framework.ExpectNoError(err, "Error occurred while creating policy: policyAllowCIDRWithExceptServerPod.") defer cleanupNetworkPolicy(f, policyAllowCIDRWithExceptServerPod) ginkgo.By("Creating client-a which should still be able to contact the server after recreating the network policy with except clause.", func() { testCanConnect(f, f.Namespace, "client-a", service, 80) }) }) </ins> ginkgo.It("should enforce policies to check ingress and egress policies can be controlled independently based on PodSelector [Feature:NetworkPolicy]", func() { var serviceA, serviceB *v1.Service var podA, podB *v1.Pod

needs a test to ensure that clauses are implemented correctly. Specifically, that they are not implemented as "deny" rules. eg, given a client pod with IP , if you add a NetworkPolicy with: spec: ingress: from: ipBlock: cidr: 0.0.0.0/0 except: A.B.C.0/24 then it should block ingress from that pod. But if you then add a second policy with: spec: ingress: from: ipBlock: cidr: A.B.C.D/32 then it should reallow ingress from that pod. Then if you delete the first policy, ingress should still be allowed, and if you create the first policy again, ingress should still be allowed. (ie, the result does not depend on which order the two policies were created in). /sig network /priority important-longterm
I can work on this as my first issue related to sig-network.
just noticed this issue while working on a similar effort. since there was no activity for the past 2 months I pushed a PR for this scenario


Question: <ins> /* Copyright 2017 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package util import ( "os" "reflect" "strings" "testing" "time" "github.com/google/go-cmp/cmp" v1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" kubetypes "k8s.io/apimachinery/pkg/types" "k8s.io/client-go/informers" csitrans "k8s.io/csi-translation-lib" fakeframework "k8s.io/kubernetes/pkg/scheduler/framework/fake" "k8s.io/kubernetes/pkg/volume/csimigration" "k8s.io/kubernetes/pkg/volume/fc" utilfeature "k8s.io/apiserver/pkg/util/feature" "k8s.io/client-go/kubernetes/fake" utiltesting "k8s.io/client-go/util/testing" "k8s.io/kubernetes/pkg/volume" volumetest "k8s.io/kubernetes/pkg/volume/testing" ) const ( testHostName      = "test-hostname" socketPath        = "/var/run/kmsplugin" migratedVolume    = "migrated-volume-name" nonMigratedVolume = "non-migrated-volume-name" testNodeName      = "test-node-name" ) var ( dirOrCreate = v1.HostPathType(v1.HostPathDirectoryOrCreate) nodeName    = kubetypes.NodeName(testNodeName) hostPath    = &v1.HostPathVolumeSource{ Path: socketPath, Type: &dirOrCreate, } migratedObjectReference    = v1.ObjectReference{Namespace: "default", Name: "migrated-pvc"} nonMigratedObjectReference = v1.ObjectReference{Namespace: "default", Name: "non-migrated-pvc"} fsVolumeMode               = new(v1.PersistentVolumeMode) ) type vaTest struct { desc                 string createNodeName       kubetypes.NodeName pod                  *v1.Pod wantVolume           *v1.Volume wantPersistentVolume *v1.PersistentVolume wantErrorMessage     string } func Test_CreateVolumeSpec(t *testing.T) { for _, test := range []vaTest{ { desc:           "inline volume type that does not support csi migration", createNodeName: nodeName, pod: &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name:      "kube-apiserver", Namespace: "default", }, Spec: v1.PodSpec{ Volumes: []v1.Volume{ { Name: migratedVolume, VolumeSource: v1.VolumeSource{ HostPath: hostPath, }, }, }, }, }, wantVolume: &v1.Volume{ Name: migratedVolume, VolumeSource: v1.VolumeSource{ HostPath: hostPath, }, }, }, { desc:           "inline volume type that supports csi migration", createNodeName: nodeName, pod: &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name:      "kube-apiserver", Namespace: "default", }, Spec: v1.PodSpec{ Volumes: []v1.Volume{ { Name: migratedVolume, VolumeSource: v1.VolumeSource{ GCEPersistentDisk: &v1.GCEPersistentDiskVolumeSource{ PDName:    "test-disk", FSType:    "ext4", Partition: 0, ReadOnly:  false, }, }, }, }, }, }, wantPersistentVolume: &v1.PersistentVolume{ ObjectMeta: metav1.ObjectMeta{ Name: "pd.csi.storage.gke.io-test-disk", }, Spec: v1.PersistentVolumeSpec{ PersistentVolumeSource: v1.PersistentVolumeSource{ CSI: &v1.CSIPersistentVolumeSource{ Driver:           "pd.csi.storage.gke.io", VolumeHandle:     "projects/UNSPECIFIED/zones/UNSPECIFIED/disks/test-disk", FSType:           "ext4", ReadOnly:         false, VolumeAttributes: map[string]string{"partition": ""}, }, }, AccessModes: []v1.PersistentVolumeAccessMode{"ReadWriteOnce"}, VolumeMode:  fsVolumeMode, }, }, }, { desc:           "pv type that does not support csi migration", createNodeName: nodeName, pod: &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name:      "kube-apiserver", Namespace: "default", }, Spec: v1.PodSpec{ Volumes: []v1.Volume{ { Name: nonMigratedVolume, VolumeSource: v1.VolumeSource{ PersistentVolumeClaim: &v1.PersistentVolumeClaimVolumeSource{ ClaimName: "non-migrated-pvc", ReadOnly:  false, }, }, }, }, }, }, wantPersistentVolume: &v1.PersistentVolume{ ObjectMeta: metav1.ObjectMeta{ Name: nonMigratedVolume, }, Spec: v1.PersistentVolumeSpec{ PersistentVolumeSource: v1.PersistentVolumeSource{ ScaleIO: &v1.ScaleIOPersistentVolumeSource{}, }, ClaimRef: &nonMigratedObjectReference, }, }, }, { desc:           "pv type that supports csi migration", createNodeName: nodeName, pod: &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name:      "kube-apiserver", Namespace: "default", }, Spec: v1.PodSpec{ Volumes: []v1.Volume{ { Name: migratedVolume, VolumeSource: v1.VolumeSource{ PersistentVolumeClaim: &v1.PersistentVolumeClaimVolumeSource{ ClaimName: "migrated-pvc", ReadOnly:  false, }, }, }, }, }, }, wantPersistentVolume: &v1.PersistentVolume{ ObjectMeta: metav1.ObjectMeta{ Name: migratedVolume, }, Spec: v1.PersistentVolumeSpec{ PersistentVolumeSource: v1.PersistentVolumeSource{ CSI: &v1.CSIPersistentVolumeSource{ Driver:           "pd.csi.storage.gke.io", VolumeHandle:     "projects/UNSPECIFIED/zones/UNSPECIFIED/disks/test-disk", FSType:           "ext4", ReadOnly:         false, VolumeAttributes: map[string]string{"partition": ""}, }, }, ClaimRef: &migratedObjectReference, }, }, }, { desc:           "CSINode not found for a volume type that supports csi migration", createNodeName: kubetypes.NodeName("another-node"), pod: &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name:      "kube-apiserver", Namespace: "default", }, Spec: v1.PodSpec{ Volumes: []v1.Volume{ { Name: migratedVolume, VolumeSource: v1.VolumeSource{ PersistentVolumeClaim: &v1.PersistentVolumeClaimVolumeSource{ ClaimName: "migrated-pvc", ReadOnly:  false, }, }, }, }, }, }, wantErrorMessage: "csiNode "another-node" not found", }, } { t.Run(test.desc, func(t *testing.T) { plugMgr, intreeToCSITranslator, csiTranslator, pvLister, pvcLister := setup(testNodeName, t) actualSpec, err := CreateVolumeSpec(test.pod.Spec.Volumes[0], test.pod, test.createNodeName, plugMgr, pvcLister, pvLister, intreeToCSITranslator, csiTranslator) if actualSpec == nil && (test.wantPersistentVolume != nil || test.wantVolume != nil) { t.Errorf("got volume spec is nil") } if (len(test.wantErrorMessage) > 0 && err == nil) || (err != nil && !strings.Contains(err.Error(), test.wantErrorMessage)) { t.Errorf("got err %v, want err with message %v", err, test.wantErrorMessage) } if test.wantPersistentVolume != nil { if actualSpec.PersistentVolume == nil { t.Errorf("gotVolumeWithCSIMigration is nil") } gotVolumeWithCSIMigration := *actualSpec.PersistentVolume if gotVolumeWithCSIMigration.Name != test.wantPersistentVolume.Name { t.Errorf("got volume name is %v, want volume name is %v", gotVolumeWithCSIMigration.Name, test.wantPersistentVolume.Name) } if !reflect.DeepEqual(gotVolumeWithCSIMigration.Spec, test.wantPersistentVolume.Spec) { t.Errorf("got volume.Spec and want.Spec diff is %s", cmp.Diff(gotVolumeWithCSIMigration.Spec, test.wantPersistentVolume.Spec)) } } if test.wantVolume != nil { if actualSpec.Volume == nil { t.Errorf("gotVolume is nil") } gotVolume := *actualSpec.Volume if !reflect.DeepEqual(gotVolume, *test.wantVolume) { t.Errorf("got volume and want diff is %s", cmp.Diff(gotVolume, test.wantVolume)) } } }) } } func setup(nodeName string, t *testing.T) (*volume.VolumePluginMgr, csimigration.PluginManager, csitrans.CSITranslator, fakeframework.PersistentVolumeLister, fakeframework.PersistentVolumeClaimLister) { tmpDir, err := utiltesting.MkTmpdir("csi-test") if err != nil { t.Fatalf("can't make a temp dir: %v", err) } defer os.RemoveAll(tmpDir) *fsVolumeMode = v1.PersistentVolumeFilesystem csiTranslator := csitrans.New() intreeToCSITranslator := csimigration.NewPluginManager(csiTranslator, utilfeature.DefaultFeatureGate) kubeClient := fake.NewSimpleClientset() factory := informers.NewSharedInformerFactory(kubeClient, time.Minute) csiDriverInformer := factory.Storage().V1().CSIDrivers() csiDriverLister := csiDriverInformer.Lister() volumeAttachmentInformer := factory.Storage().V1().VolumeAttachments() volumeAttachmentLister := volumeAttachmentInformer.Lister() plugMgr := &volume.VolumePluginMgr{} fakeAttachDetachVolumeHost := volumetest.NewFakeAttachDetachVolumeHostWithCSINodeName(t, tmpDir, kubeClient, fc.ProbeVolumePlugins(), nodeName, csiDriverLister, volumeAttachmentLister, ) plugMgr.Host = fakeAttachDetachVolumeHost pvLister := fakeframework.PersistentVolumeLister{ { ObjectMeta: metav1.ObjectMeta{Name: migratedVolume}, Spec: v1.PersistentVolumeSpec{ PersistentVolumeSource: v1.PersistentVolumeSource{ GCEPersistentDisk: &v1.GCEPersistentDiskVolumeSource{ PDName:    "test-disk", FSType:    "ext4", Partition: 0, ReadOnly:  false, }, }, ClaimRef: &migratedObjectReference, }, }, { ObjectMeta: metav1.ObjectMeta{Name: nonMigratedVolume}, Spec: v1.PersistentVolumeSpec{ PersistentVolumeSource: v1.PersistentVolumeSource{ ScaleIO: &v1.ScaleIOPersistentVolumeSource{}, }, ClaimRef: &nonMigratedObjectReference, }, }, } pvcLister := fakeframework.PersistentVolumeClaimLister{ { ObjectMeta: metav1.ObjectMeta{Name: "migrated-pvc", Namespace: "default"}, Spec:       v1.PersistentVolumeClaimSpec{VolumeName: migratedVolume}, Status: v1.PersistentVolumeClaimStatus{ Phase: v1.ClaimBound, }, }, { ObjectMeta: metav1.ObjectMeta{Name: "non-migrated-pvc", Namespace: "default"}, Spec:       v1.PersistentVolumeClaimSpec{VolumeName: nonMigratedVolume}, Status: v1.PersistentVolumeClaimStatus{ Phase: v1.ClaimBound, }, }, } return plugMgr, intreeToCSITranslator, csiTranslator, pvLister, pvcLister } </ins>

After the cluster was updated to enable PD CSI Driver, there was a race condition where AD Controller (kube-controller-manager) started to BEFORE the PD CSI Driver was up, thus AD Controller gave us the error Then AD Controller triggered the disk detach because the disk/volume was not populated in the desired state of world. Unfortunately the disk was in use during this detach and got corrupted. Since cluster upgrades can cause the controller manager and CSI drivers to start at different times, the AD controller needs to be more conservative when detaching We have not been able to reproduce this, it came up from a customer. /sig storage /assign <detailsk8s 1.23, CSI migration enabled </details<detailsGKE </details<details</details<details</details<details</details<details</details
/triage accepted
I think there's potentially two changes we should do: 1) for the initial DSW population, we should skip the check that if the csi driver is not installed. 2) in general, we should not skip volumes during dsw initial population if an error is returned. If we had some temporary api server error then that would cause volumes to not get to dsw and similar behavior happen. cc
for a repro, maybe you can try something like: a cluster with csi migration enabled (control plane + nodes) a pod with intree-volume and get it running the csi driver node daemonset. verify the driver is removed from CSINode kube-controller-manager
cc
/cc was trying to repro this, but on GKE with the managed driver which made it complicated.
Yeah maybe gce kube-up would be easier for a repro
If CSINode says that it has migrated GCP PD volume plugin to CSI, then IMO A/D controller should always attach GCP PD volumes to it using CSI, regardless if the CSI driver is present or not. In-tree volume plugin in the kubelet is disabled and can't mount any in-tree volumes anyway. To flip the migration feature gate / upgrade / downgrade kubelet, the cluster admin must drain the node first to make sure everything is detached. A/D controller should not see change of CSINode migration annotations with any volume attached to it.
/priority important-soon
Another option could be to process these volumes anyways but add them to DSOW with migration status as "uncertain", so as they can be re-processed later.


Question: <ins> /* Copyright 2017 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package util import ( "os" "reflect" "strings" "testing" "time" "github.com/google/go-cmp/cmp" v1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" kubetypes "k8s.io/apimachinery/pkg/types" "k8s.io/client-go/informers" csitrans "k8s.io/csi-translation-lib" fakeframework "k8s.io/kubernetes/pkg/scheduler/framework/fake" "k8s.io/kubernetes/pkg/volume/csimigration" "k8s.io/kubernetes/pkg/volume/fc" utilfeature "k8s.io/apiserver/pkg/util/feature" "k8s.io/client-go/kubernetes/fake" utiltesting "k8s.io/client-go/util/testing" "k8s.io/kubernetes/pkg/volume" volumetest "k8s.io/kubernetes/pkg/volume/testing" ) const ( testHostName      = "test-hostname" socketPath        = "/var/run/kmsplugin" migratedVolume    = "migrated-volume-name" nonMigratedVolume = "non-migrated-volume-name" testNodeName      = "test-node-name" ) var ( dirOrCreate = v1.HostPathType(v1.HostPathDirectoryOrCreate) nodeName    = kubetypes.NodeName(testNodeName) hostPath    = &v1.HostPathVolumeSource{ Path: socketPath, Type: &dirOrCreate, } migratedObjectReference    = v1.ObjectReference{Namespace: "default", Name: "migrated-pvc"} nonMigratedObjectReference = v1.ObjectReference{Namespace: "default", Name: "non-migrated-pvc"} fsVolumeMode               = new(v1.PersistentVolumeMode) ) type vaTest struct { desc                 string createNodeName       kubetypes.NodeName pod                  *v1.Pod wantVolume           *v1.Volume wantPersistentVolume *v1.PersistentVolume wantErrorMessage     string } func Test_CreateVolumeSpec(t *testing.T) { for _, test := range []vaTest{ { desc:           "inline volume type that does not support csi migration", createNodeName: nodeName, pod: &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name:      "kube-apiserver", Namespace: "default", }, Spec: v1.PodSpec{ Volumes: []v1.Volume{ { Name: migratedVolume, VolumeSource: v1.VolumeSource{ HostPath: hostPath, }, }, }, }, }, wantVolume: &v1.Volume{ Name: migratedVolume, VolumeSource: v1.VolumeSource{ HostPath: hostPath, }, }, }, { desc:           "inline volume type that supports csi migration", createNodeName: nodeName, pod: &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name:      "kube-apiserver", Namespace: "default", }, Spec: v1.PodSpec{ Volumes: []v1.Volume{ { Name: migratedVolume, VolumeSource: v1.VolumeSource{ GCEPersistentDisk: &v1.GCEPersistentDiskVolumeSource{ PDName:    "test-disk", FSType:    "ext4", Partition: 0, ReadOnly:  false, }, }, }, }, }, }, wantPersistentVolume: &v1.PersistentVolume{ ObjectMeta: metav1.ObjectMeta{ Name: "pd.csi.storage.gke.io-test-disk", }, Spec: v1.PersistentVolumeSpec{ PersistentVolumeSource: v1.PersistentVolumeSource{ CSI: &v1.CSIPersistentVolumeSource{ Driver:           "pd.csi.storage.gke.io", VolumeHandle:     "projects/UNSPECIFIED/zones/UNSPECIFIED/disks/test-disk", FSType:           "ext4", ReadOnly:         false, VolumeAttributes: map[string]string{"partition": ""}, }, }, AccessModes: []v1.PersistentVolumeAccessMode{"ReadWriteOnce"}, VolumeMode:  fsVolumeMode, }, }, }, { desc:           "pv type that does not support csi migration", createNodeName: nodeName, pod: &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name:      "kube-apiserver", Namespace: "default", }, Spec: v1.PodSpec{ Volumes: []v1.Volume{ { Name: nonMigratedVolume, VolumeSource: v1.VolumeSource{ PersistentVolumeClaim: &v1.PersistentVolumeClaimVolumeSource{ ClaimName: "non-migrated-pvc", ReadOnly:  false, }, }, }, }, }, }, wantPersistentVolume: &v1.PersistentVolume{ ObjectMeta: metav1.ObjectMeta{ Name: nonMigratedVolume, }, Spec: v1.PersistentVolumeSpec{ PersistentVolumeSource: v1.PersistentVolumeSource{ ScaleIO: &v1.ScaleIOPersistentVolumeSource{}, }, ClaimRef: &nonMigratedObjectReference, }, }, }, { desc:           "pv type that supports csi migration", createNodeName: nodeName, pod: &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name:      "kube-apiserver", Namespace: "default", }, Spec: v1.PodSpec{ Volumes: []v1.Volume{ { Name: migratedVolume, VolumeSource: v1.VolumeSource{ PersistentVolumeClaim: &v1.PersistentVolumeClaimVolumeSource{ ClaimName: "migrated-pvc", ReadOnly:  false, }, }, }, }, }, }, wantPersistentVolume: &v1.PersistentVolume{ ObjectMeta: metav1.ObjectMeta{ Name: migratedVolume, }, Spec: v1.PersistentVolumeSpec{ PersistentVolumeSource: v1.PersistentVolumeSource{ CSI: &v1.CSIPersistentVolumeSource{ Driver:           "pd.csi.storage.gke.io", VolumeHandle:     "projects/UNSPECIFIED/zones/UNSPECIFIED/disks/test-disk", FSType:           "ext4", ReadOnly:         false, VolumeAttributes: map[string]string{"partition": ""}, }, }, ClaimRef: &migratedObjectReference, }, }, }, { desc:           "CSINode not found for a volume type that supports csi migration", createNodeName: kubetypes.NodeName("another-node"), pod: &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name:      "kube-apiserver", Namespace: "default", }, Spec: v1.PodSpec{ Volumes: []v1.Volume{ { Name: migratedVolume, VolumeSource: v1.VolumeSource{ PersistentVolumeClaim: &v1.PersistentVolumeClaimVolumeSource{ ClaimName: "migrated-pvc", ReadOnly:  false, }, }, }, }, }, }, wantErrorMessage: "csiNode "another-node" not found", }, } { t.Run(test.desc, func(t *testing.T) { plugMgr, intreeToCSITranslator, csiTranslator, pvLister, pvcLister := setup(testNodeName, t) actualSpec, err := CreateVolumeSpec(test.pod.Spec.Volumes[0], test.pod, test.createNodeName, plugMgr, pvcLister, pvLister, intreeToCSITranslator, csiTranslator) if actualSpec == nil && (test.wantPersistentVolume != nil || test.wantVolume != nil) { t.Errorf("got volume spec is nil") } if (len(test.wantErrorMessage) > 0 && err == nil) || (err != nil && !strings.Contains(err.Error(), test.wantErrorMessage)) { t.Errorf("got err %v, want err with message %v", err, test.wantErrorMessage) } if test.wantPersistentVolume != nil { if actualSpec.PersistentVolume == nil { t.Errorf("gotVolumeWithCSIMigration is nil") } gotVolumeWithCSIMigration := *actualSpec.PersistentVolume if gotVolumeWithCSIMigration.Name != test.wantPersistentVolume.Name { t.Errorf("got volume name is %v, want volume name is %v", gotVolumeWithCSIMigration.Name, test.wantPersistentVolume.Name) } if !reflect.DeepEqual(gotVolumeWithCSIMigration.Spec, test.wantPersistentVolume.Spec) { t.Errorf("got volume.Spec and want.Spec diff is %s", cmp.Diff(gotVolumeWithCSIMigration.Spec, test.wantPersistentVolume.Spec)) } } if test.wantVolume != nil { if actualSpec.Volume == nil { t.Errorf("gotVolume is nil") } gotVolume := *actualSpec.Volume if !reflect.DeepEqual(gotVolume, *test.wantVolume) { t.Errorf("got volume and want diff is %s", cmp.Diff(gotVolume, test.wantVolume)) } } }) } } func setup(nodeName string, t *testing.T) (*volume.VolumePluginMgr, csimigration.PluginManager, csitrans.CSITranslator, fakeframework.PersistentVolumeLister, fakeframework.PersistentVolumeClaimLister) { tmpDir, err := utiltesting.MkTmpdir("csi-test") if err != nil { t.Fatalf("can't make a temp dir: %v", err) } defer os.RemoveAll(tmpDir) *fsVolumeMode = v1.PersistentVolumeFilesystem csiTranslator := csitrans.New() intreeToCSITranslator := csimigration.NewPluginManager(csiTranslator, utilfeature.DefaultFeatureGate) kubeClient := fake.NewSimpleClientset() factory := informers.NewSharedInformerFactory(kubeClient, time.Minute) csiDriverInformer := factory.Storage().V1().CSIDrivers() csiDriverLister := csiDriverInformer.Lister() volumeAttachmentInformer := factory.Storage().V1().VolumeAttachments() volumeAttachmentLister := volumeAttachmentInformer.Lister() plugMgr := &volume.VolumePluginMgr{} fakeAttachDetachVolumeHost := volumetest.NewFakeAttachDetachVolumeHostWithCSINodeName(t, tmpDir, kubeClient, fc.ProbeVolumePlugins(), nodeName, csiDriverLister, volumeAttachmentLister, ) plugMgr.Host = fakeAttachDetachVolumeHost pvLister := fakeframework.PersistentVolumeLister{ { ObjectMeta: metav1.ObjectMeta{Name: migratedVolume}, Spec: v1.PersistentVolumeSpec{ PersistentVolumeSource: v1.PersistentVolumeSource{ GCEPersistentDisk: &v1.GCEPersistentDiskVolumeSource{ PDName:    "test-disk", FSType:    "ext4", Partition: 0, ReadOnly:  false, }, }, ClaimRef: &migratedObjectReference, }, }, { ObjectMeta: metav1.ObjectMeta{Name: nonMigratedVolume}, Spec: v1.PersistentVolumeSpec{ PersistentVolumeSource: v1.PersistentVolumeSource{ ScaleIO: &v1.ScaleIOPersistentVolumeSource{}, }, ClaimRef: &nonMigratedObjectReference, }, }, } pvcLister := fakeframework.PersistentVolumeClaimLister{ { ObjectMeta: metav1.ObjectMeta{Name: "migrated-pvc", Namespace: "default"}, Spec:       v1.PersistentVolumeClaimSpec{VolumeName: migratedVolume}, Status: v1.PersistentVolumeClaimStatus{ Phase: v1.ClaimBound, }, }, { ObjectMeta: metav1.ObjectMeta{Name: "non-migrated-pvc", Namespace: "default"}, Spec:       v1.PersistentVolumeClaimSpec{VolumeName: nonMigratedVolume}, Status: v1.PersistentVolumeClaimStatus{ Phase: v1.ClaimBound, }, }, } return plugMgr, intreeToCSITranslator, csiTranslator, pvLister, pvcLister } </ins>

But surely it looks like might be easiest unless it has any downsides.
There wasn't a node update in this case---there was a control plane update which restarted the PD CSI driver, and it looks like the CSIDriver resource was updated or changed in some way which raced with the kube-controller-manager restart.
Is there anyway we can know if the csi driver is up and running(like a health check ping) before populating DSW?
We use the CSINode registration to determine that. The discussion here I think is concluding that we should not use that as a criteria for adding the volume to DSW. We only need to check that CSI migration is enabled on the node.
+1 we need to be very resilient against races, I think we have to assume a driver is in process of being installed at any point in the AD controller reconcilliation.
/assign sunnylovestiramisu
Reproduced with the steps: In /var/log/kube-controller-
Tested with the and doing the reproduce steps above. The error of does not show up anymore. The error message still get logged in the kube-controller- for debugging purpose: Ran Ran On the GCP console, the disk is used by node: Ran I redeploy the pd-csi-driver and then make sure it is up and running. Ran


Question: } func (mounter *Mounter) IsNotMountPoint(dir string) (bool, error) { <del> return IsNotMountPoint(mounter, dir) </del> <ins> return isNotMountPoint(mounter, dir) </ins> } // IsLikelyNotMountPoint determines if a directory is not a mountpoint.

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: There's , and . The former is used by local and nfs volume plugins, while the latter is used by , which is also used by local and nfs volume plugins. What you expected to happen: Can we consolidate these? This is really confusing and could easily be the source of bugs. <!-- DO NOT EDIT BELOW THIS LINE --/kind cleanup
cc
/assign I'd like to clean up code.


Question: test/e2e/network/service_latency.go: "should not be very high" test/e2e/node/events.go: "should be sent by kubelets and the scheduler about pods scheduling and running" test/e2e/node/pods.go: "should be submitted and removed" <ins> test/e2e/node/pods.go: "should be submitted and removed" </ins> test/e2e/node/pre_stop.go: "should call prestop when killing a pod" test/e2e/scheduling/predicates.go: "validates resource limits of pods that are allowed to run" test/e2e/scheduling/predicates.go: "validates that NodeSelector is respected if not matching"

We need to fix this potentially flaky test and add it back to conformance suite. Please see discussion in Also waiting on:
cc /sig testing
/area conformance
/cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
can we convert it to conformance now??
Once 1.15 opens up


Question: if [[ "${KUBE_PROXY_DAEMONSET:-}" == "true" && "${master}" != "true" ]]; then # Add kube-proxy daemonset label to node to avoid situation during cluster # upgrade/downgrade when there are two instances of kube-proxy running on a node. <del> # TODO(liggitt): drop beta.kubernetes.io/kube-proxy-ds-ready in 1.16 node_labels="node.kubernetes.io/kube-proxy-ds-ready=true,beta.kubernetes.io/kube-proxy-ds-ready=true" </del> <ins> node_labels="node.kubernetes.io/kube-proxy-ds-ready=true" </ins> fi if [[ -n "${NODE_LABELS:-}" ]]; then node_labels="${node_labels:+${node_labels},}${NODE_LABELS}"

xref In 1.16: change the metadata-proxy addon to use change the kube-proxy addon to use change the masq-agent addon to use stop setting , , and in kube-up
Hey I'm trying to start to contribute to kubernetes and this seem to be an easy task. Can I help with this one?


Question: execAffinityTestForLBService(f, cs, svc) }) <del> // TODO: Get rid of [DisabledForLargeClusters] tag when issue #56138 is fixed. </del> // [LinuxOnly]: Windows does not support session affinity. <del> ginkgo.It("should be able to switch session affinity for LoadBalancer service with ESIPP on [Slow] [DisabledForLargeClusters] [LinuxOnly]", func() { </del> <ins> ginkgo.It("should be able to switch session affinity for LoadBalancer service with ESIPP on [Slow] [LinuxOnly]", func() { </ins> // L4 load balancer affinity `ClientIP` is not supported on AWS ELB. e2eskipper.SkipIfProviderIs("aws")

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug This is a followup from filed separately now that we're not tracking this as part of 1.9 release. The tests have been disabled with [DisabledForLargeClusters] tag for now, and this fact will be mentioned in the scalability validation report for v1.9. This issue tracks fixing the test so the tag can be removed and the test ran again in the large cluster. /assign /kind e2e-test-failure
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Is this e2e test still broken?
Sorry for the confusion here. Currently this e2e test is disabled on extra large cluster due to some scalability issues with the load balancer related components. At the moment the issues haven't been completely resolved yet.
ok, adding priority so that I know it's not a 1.13 blocker. /priority important-longterm
cc - this is why the relevant tests for are not part of the e2e suite.
This issue is still valid --- some action items on top of my head: Figure out a plan to make service controller properly handle operations in parallel. Check on the status for cloud providers (especially GCP, which the test ran against) on handling large number of nodes as LB backends. /remove-triage unresolved
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or .


Question: execAffinityTestForLBService(f, cs, svc) }) <del> // TODO: Get rid of [DisabledForLargeClusters] tag when issue #56138 is fixed. </del> // [LinuxOnly]: Windows does not support session affinity. <del> ginkgo.It("should be able to switch session affinity for LoadBalancer service with ESIPP on [Slow] [DisabledForLargeClusters] [LinuxOnly]", func() { </del> <ins> ginkgo.It("should be able to switch session affinity for LoadBalancer service with ESIPP on [Slow] [LinuxOnly]", func() { </ins> // L4 load balancer affinity `ClientIP` is not supported on AWS ELB. e2eskipper.SkipIfProviderIs("aws")

/lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten /lifecycle frozen
/reopen Reopening as is being reverted in
Reopened this issue. <detailsIn response to : /reopen Reopening as is being reverted in Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
these tests are flaking in xref
cc since these were enabled by
Yeah - though these failures doesn't seem to be scale related, they seem to be test related to me. This basically means nodes without public ips: It's pretty much the same thing as (just for a different test). should be addressing these kinds of errors. [Not a release blocker for sure]
BTW - has been opened directly for this one.
... so since is closed, Can we close this and re-enable for large clusters ? (i.e. remove the Tag in that disables this test) ? I'm not up to speed on the history but 2 related notes I came across digging into related test today... 's a couple of TODOs in the code referencing this specific issue, but your above direct issue is now closed, so we can close this as well right?... And if so, we can remove the TODO on the test... -enabling the test ... not sure wether the timing is good right now (given that is open right now , which sparked my stumbling upon this issue :))
We tried to enable all of them and that caused us a bunch of issues and those had to be reverted: A couple tests were enabled later in: and those seem fine. The remaining (mostly session-affinity ones) had some issue, which so far we didn't have time to debug and I haven't heard a good explanation why those were failing. It's possible that they were improved with , but I doubt those would really cause the failures we've seen in the past. So until this is debugged (or at least understood) I'm reluctant to enabling them.


Question: // api.Registry.GroupOrDie(v1.GroupName).GroupVersions[0].String() is changed // to "v1"? <del> runtimeapi "k8s.io/cri-api/pkg/apis/runtime/v1alpha2" </del> _ "k8s.io/kubernetes/pkg/apis/core/install" "k8s.io/kubernetes/pkg/features" kubecontainer "k8s.io/kubernetes/pkg/kubelet/container"

What happened: It's most visible in the test <img width="323" alt="Screen Shot 2020-05-20 at 6 09 10 PM" src="https://user-"It shows up in kind, kube- and kops jobs The regression was introduced somewhere within , possible culprits are: (FYI (FYI (FYI There are more tests affected, just not as prominently (visually inspecting ) [sig-apps] StatefulSet [] Basic StatefulSet functionality [StatefulSetBasic] should perform rolling updates and roll backs of template modifications with PVCs [sig-apps] StatefulSet [] Basic StatefulSet functionality [StatefulSetBasic] should perform rolling updates and roll backs of template modifications [Conformance] [sig-storage] In-tree Volumes [Driver: nfs] [Testpattern: Inline-volume (default fs)] volumes should store data [sig-storage] In-tree Volumes [Driver: nfs] [Testpattern: Pre-provisioned PV (default fs)] volumes should store data I wasn't sure whether to tag this as kind/regression or not, I'll leave that up to or someone /sig node /sig api-machimery /sig scheduling
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : What happened: It's most visible in the test (FYI (FYI (FYI StatefulSet [] Basic StatefulSet functionality [StatefulSetBasic] should perform rolling updates and roll backs of template modifications with PVCs [sig-apps] StatefulSet [] Basic StatefulSet functionality [StatefulSetBasic] should perform rolling updates and roll backs of template modifications [Conformance] [sig-storage] In-tree Volumes [Driver: nfs] [Testpattern: Inline-volume (default fs)] volumes should store data [sig-storage] In-tree Volumes [Driver: nfs] [Testpattern: Pre-provisioned PV (default fs)] volumes should store data /sig api-machimery /sig scheduling Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository.


Question: // api.Registry.GroupOrDie(v1.GroupName).GroupVersions[0].String() is changed // to "v1"? <del> runtimeapi "k8s.io/cri-api/pkg/apis/runtime/v1alpha2" </del> _ "k8s.io/kubernetes/pkg/apis/core/install" "k8s.io/kubernetes/pkg/features" kubecontainer "k8s.io/kubernetes/pkg/kubelet/container"

</details
/cc
will increase the time for deletion of pod and it should be expected because currently pod is deleted from apiserver before removing the sandbox container but after this fix pod will be deleted from the apiserver only after deleting the sandbox container.
/remove-sig scheduling doesn't affect scheduling running time at all.
Yeah, is almost definitely the culprit. Last time we changed that code path, it also caused a regression. The degradation in performance is expected, but a 5x slowdown is definitely larger than it should be.
do we know if that's because the status used by the path is stale, or if removing sandboxes is actually that slow?
if there's not a quick fix, should we revert and bring it back in once the root issue is understood?
I think a revert is appropriate in this case.
ok, I'll open it
open at
I agree with that 5x is something which shouldn't happen and i can take a look on why thats happening and that might require changing the way how the sandbox is currently deleted which is only during GC.


Question: github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 github.com/onsi/ginkgo/v2 v2.9.1 github.com/onsi/gomega v1.27.4 <del> github.com/opencontainers/runc v1.1.4 </del> <ins> github.com/opencontainers/runc v1.1.5 </ins> github.com/opencontainers/selinux v1.10.0 github.com/pkg/errors v0.9.1 github.com/pmezard/go-difflib v1.0.0

opencontainers/runc has fixed three CVEs: Latest runc is pulled by k/k Check runc version in file No response n/a n/a <details</details<details</details<details</details<details</details
/assign /sig security architecture release
/triage accepted


Question: return "", err } <ins> // try and get canonical path for disk and if we can't throw error vmDiskPath, err = getcanonicalVolumePath(ctx, vm.Datacenter, vmDiskPath) if err != nil { klog.Errorf("failed to get canonical path for %s on node %s: %v", vmDiskPath, convertToString(nodeName), err) return "", err } </ins> diskUUID, err = vm.AttachDisk(ctx, vmDiskPath, &vclib.VolumeOptions{SCSIControllerType: vclib.PVSCSIControllerType, StoragePolicyName: storagePolicyName}) if err != nil { klog.Errorf("Failed to attach disk: %s for node: %s. err: +%v", vmDiskPath, convertToString(nodeName), err)


Question: } func (sched *Scheduler) updatePodInSchedulingQueue(oldObj, newObj interface{}) { <del> pod := newObj.(*v1.Pod) if sched.skipPodUpdate(pod) { </del> <ins> oldPod, newPod := oldObj.(*v1.Pod), newObj.(*v1.Pod) // Bypass update event that carries identical objects; otherwise, a duplicated // Pod may go through scheduling and cause unexpected behavior (see #96071). if oldPod.ResourceVersion == newPod.ResourceVersion { </ins> return } <del> if err := sched.SchedulingQueue.Update(oldObj.(*v1.Pod), pod); err != nil { </del> <ins> if sched.skipPodUpdate(newPod) { return } if err := sched.SchedulingQueue.Update(oldPod, newPod); err != nil { </ins> utilruntime.HandleError(fmt.Errorf("unable to update %T: %v", newObj, err)) } }

Which jobs are failing: pull-kubernetes-integration Which test(s) are failing: TestPreemptionRaces Testgrid link: /sig scheduling /help
/reopen I just saw this again
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Seems to be green: https://k8s-
It's still flaky on both master and PR testing: master: https://k8s- () PR testing: (). /reopen
Reopened this issue. <detailsIn response to : It's still flaky on both master and PR testing: PR testing: (). </details
From the above two logs, I'm feeling that there is a "phantom" event (very likely of type Update by enforced resync) triggering the pod in the completed N<subth</subtest, to be still alive in the N+1<subth</subtest.
/triage accepted working on a fix.
It would be great if we assign a different name to the Nth round test's preemptor pod. Currently, whatever this is Nth test, the preemptor pod is always called 'preemptor-pod'.


Question: ExitCode: 1, }, }, <del> map[string]error{}, []api.ContainerStatus{}, map[string]api.ContainerState{ </del> <ins> reasons:     map[string]error{}, oldStatuses: []api.ContainerStatus{}, expectedState: map[string]api.ContainerState{ </ins> "running": {Running: &api.ContainerStateRunning{ StartedAt: unversioned.NewTime(testTimestamp), }}, }, <del> map[string]api.ContainerState{ </del> <ins> expectedLastTerminationState: map[string]api.ContainerState{ </ins> "running": {Terminated: &api.ContainerStateTerminated{ ExitCode:    1, ContainerID: emptyContainerID,

is this a Node bug? cc as next oncall to follow up on this.
This is a node bug due to init containers
I may need help debugging - I see:
So I think the problem is in the error case - we're not properly recording that the start failed, and the next time we come in we somehow think the first container succeeded. I'll add some debugging to see if we can't recreate it.
Let's discuss on instead.


Question: ExitCode: 1, }, }, <del> map[string]error{}, []api.ContainerStatus{}, map[string]api.ContainerState{ </del> <ins> reasons:     map[string]error{}, oldStatuses: []api.ContainerStatus{}, expectedState: map[string]api.ContainerState{ </ins> "running": {Running: &api.ContainerStateRunning{ StartedAt: unversioned.NewTime(testTimestamp), }}, }, <del> map[string]api.ContainerState{ </del> <ins> expectedLastTerminationState: map[string]api.ContainerState{ </ins> "running": {Terminated: &api.ContainerStateTerminated{ ExitCode:    1, ContainerID: emptyContainerID,

Kubelet issues a docker inspect call after creating a container, but for very short lived containers there is a window in which they could exit before being inspected in the docker manager. If the container PID is 0, we should check the ExitCode and other relevant fields and handle them appropriately. [1] This is the suspected cause of several flakes: /cc
looking at the logs 2 weird things: race described second container ran, which suggests inconsistent reporting of initialized status of init1 and choosing init2 as the next container ( vs ). I think 1 got it right and 2 got it wrong. From logs: I'd expect the second one to not have run in this case? Node logs:
Init containers have an implicit assumption that container statuses include results of previous SyncPod invocations. If that's not the case we may need to fix that (or at least mitigate it). On May 18, 2016, at 8:28 PM, Prashanth B wrote: looking at the logs 2 weird things: race described second container ran, which suggests inconsistent reporting of initialized status of init1 and choosing init2 as the next container ( vs ). I think 1 got it right and 2 got it wrong. From logs: May 18 15:58:33.380: INFO: pod-init--1d4b-11e6-80a4- started at 2016-05-18T15:58:04-07:00 (2+1 container statuses recorded) May 18 15:58:33.380: INFO: Init container init1 ready: false, restart count 0 May 18 15:58:33.380: INFO: Init container init2 ready: true, restart count 0 I'd expect the second one to not have run in this case? Node logs: — You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub
This should be true. Is there any evidence that suggests the opposite? only shows conflicting view of whether init1 has run.
If the container exits already and it's not an infra container, kubelet should not error out. Although even if it does, I failed to see how this could affect the readiness of the init container without checking the code further.


Question: ExitCode: 1, }, }, <del> map[string]error{}, []api.ContainerStatus{}, map[string]api.ContainerState{ </del> <ins> reasons:     map[string]error{}, oldStatuses: []api.ContainerStatus{}, expectedState: map[string]api.ContainerState{ </ins> "running": {Running: &api.ContainerStateRunning{ StartedAt: unversioned.NewTime(testTimestamp), }}, }, <del> map[string]api.ContainerState{ </del> <ins> expectedLastTerminationState: map[string]api.ContainerState{ </ins> "running": {Terminated: &api.ContainerStateTerminated{ ExitCode:    1, ContainerID: emptyContainerID,

Each syncPod invocation, we inspect the list of init container in order looking for lastTerminationState. There could be a bug in that logic, or if we didn't get the full status, we might make a bad decision. When determining which container to run next, we always look for the last init container in the chain that has success, then run the next container (unless the pod is RestartNever). , Yu-Ju Hong wrote:
Saw this flake again. How do we verify that container status is not lost between syn pods?
... between calls to syncPods
The second sync clearly saw the init1 container, and decided to launch init2. See "InitContainersToKeep" in the message below.
containers to keep can include failed containers. , Yu-Ju Hong wrote:
Discussed on slack. In summary, the problem has to do with how we convert the kubecontainer.PodStatus to the api.PodStatus. If a container is terminated, the function makes the decisions: the pod restart policy is "Always", and there is a "failure reason" recorded for the given container, this means the pod worker tried to (re)start the container but failed. Set the state to "Waiting" and populate the reason. , the container state is simply "terminated". Init containers don't obey the restart policy like regular containers, and the logic will need to be amended to work with them. is working on a PR. In addition to that, kubelet should not error out if a user container exited before it could apply the oom score (which is best effort in this case). fixes that.
Ahh, thanks for digging into the issue. This is the exact the concern I had when I reviewed the initial implementation of init container, which blocks pr for a while. I even asked to help me verify this case, unfortunately we didn't reproduce the issue back then. :-(
I wish I'd seen the flakes in the runs prior to merge.
We didn't notice this until we started running the tests against docker v1.11.1, which is much faster and hit the race condition differently.
I suppose that's a good thing...
It is hard to reproduce this.


Question: ExitCode: 1, }, }, <del> map[string]error{}, []api.ContainerStatus{}, map[string]api.ContainerState{ </del> <ins> reasons:     map[string]error{}, oldStatuses: []api.ContainerStatus{}, expectedState: map[string]api.ContainerState{ </ins> "running": {Running: &api.ContainerStateRunning{ StartedAt: unversioned.NewTime(testTimestamp), }}, }, <del> map[string]api.ContainerState{ </del> <ins> expectedLastTerminationState: map[string]api.ContainerState{ </ins> "running": {Terminated: &api.ContainerStateTerminated{ ExitCode:    1, ContainerID: emptyContainerID,

The extra check of last terminated container in your initial code makes me nervous, but we couldn't reproduce it due to that small race windows until we have a faster docker running on the node. :-)
I got "Pods should invoke init containers on a RestartAlways pod" flake (referenced from ): Logs:
All the fixes went in. Please reopen this issue if you see any more flakes.