Question:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind cleanup What happened: In PR we tightened the --provider parameter checking of the binary such that it only accepted known providers. This turned out to be problematic () because people had started to depend on the previous behavior, which was to accept anything and merely warn about it. For the sake of giving people time to adapt, PR reverted to the older behavior. What you expected to happen: In 1.14 we want to allow only known providers. While at it, the error handling for unknown providers should be enhanced (print proper error message instead of calling Failf, which panics). /assign /sig testing
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
kubeadm was one of the tools which got broken by the tightening of the parameter checking. Does kubeadm still rely on the acceptance of unknown provider parameters or can the current warning be turned into an error now? /remove-lifecycle stale
yes the kubernetes-anywhere deployer is still bound to provider=kubernetes-anywhere (unknown) in test infra jobs. we went back and forth on this topic, but if you wish to replace this support for unknown deployers i can switch those jobs to . LMK.
given one release has passed with the deprecated message on the CLI, it's probably safe to remove this functionality now. :)
"Lubomir I. Ivanov" <: Yes, please do. Once that is complete, I'll turn the warning into an error.
done:

flag.StringVar(&TestContext.KubeVolumeDir, "volume-dir", "/var/lib/kubelet", "Path to the directory containing the kubelet volumes.") flag.StringVar(&TestContext.CertDir, "cert-dir", "", "Path to the directory containing the certs. Default is empty, which doesn't use certs.") flag.StringVar(&TestContext.RepoRoot, "repo-root", "../../", "Root directory of kubernetes repository, for finding test files.") <del> flag.StringVar(&TestContext.Provider, "provider", "", "The name of the Kubernetes provider (gce, gke, local, etc.)") </del> <ins> flag.StringVar(&TestContext.Provider, "provider", "", "The name of the Kubernetes provider (gce, gke, local, skeleton (the fallback if not set), etc.)") </ins> flag.StringVar(&TestContext.Tooling, "tooling", "", "The tooling in use (kops, gke, etc.)") flag.StringVar(&TestContext.KubectlPath, "kubectl-path", "kubectl", "The kubectl binary to use. For development, you might use 'cluster/kubectl.sh' here.") flag.StringVar(&TestContext.OutputDir, "e2e-output-dir", "/tmp", "Output directory for interesting/useful test data, like performance data, benchmarks, and other metrics.")


Question:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind cleanup What happened: In PR we tightened the --provider parameter checking of the binary such that it only accepted known providers. This turned out to be problematic () because people had started to depend on the previous behavior, which was to accept anything and merely warn about it. For the sake of giving people time to adapt, PR reverted to the older behavior. What you expected to happen: In 1.14 we want to allow only known providers. While at it, the error handling for unknown providers should be enhanced (print proper error message instead of calling Failf, which panics). /assign /sig testing
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
kubeadm was one of the tools which got broken by the tightening of the parameter checking. Does kubeadm still rely on the acceptance of unknown provider parameters or can the current warning be turned into an error now? /remove-lifecycle stale
yes the kubernetes-anywhere deployer is still bound to provider=kubernetes-anywhere (unknown) in test infra jobs. we went back and forth on this topic, but if you wish to replace this support for unknown deployers i can switch those jobs to . LMK.
given one release has passed with the deprecated message on the CLI, it's probably safe to remove this functionality now. :)
"Lubomir I. Ivanov" <: Yes, please do. Once that is complete, I'll turn the warning into an error.
done:

} // Make sure that all test runs have a valid TestContext.CloudConfig.Provider. <ins> // TODO: whether and how long this code is needed is getting discussed // in https://github.com/kubernetes/kubernetes/issues/70194. if TestContext.Provider == "" { // Some users of the e2e.test binary pass --provider=. // We need to support that, changing it would break those usages. Logf("The --provider flag is not set. Continuing as if --provider=skeleton had been used.") TestContext.Provider = "skeleton" } </ins> var err error TestContext.CloudConfig.Provider, err = SetupProviderConfig(TestContext.Provider) <del> if err == nil { return } if !os.IsNotExist(errors.Cause(err)) { Failf("Failed to setup provider config: %v", err) } // We allow unknown provider parameters for historic reasons. At least log a // warning to catch typos. // TODO (https://github.com/kubernetes/kubernetes/issues/70200): // - remove the fallback for unknown providers // - proper error message instead of Failf (which panics) klog.Warningf("Unknown provider %q, proceeding as for --provider=skeleton.", TestContext.Provider) TestContext.CloudConfig.Provider, err = SetupProviderConfig("skeleton") </del> if err != nil { <del> Failf("Failed to setup fallback skeleton provider config: %v", err) </del> <ins> if os.IsNotExist(errors.Cause(err)) { // Provide a more helpful error message when the provider is unknown. var providers []string for _, name := range GetProviders() { // The empty string is accepted, but looks odd in the output below unless we quote it. if name == "" { name = `""` } providers = append(providers, name) } sort.Strings(providers) klog.Errorf("Unknown provider %q. The following providers are known: %v", TestContext.Provider, strings.Join(providers, " ")) } else { klog.Errorf("Failed to setup provider config for %q: %v", TestContext.Provider, err) } os.Exit(1) </ins> } }


Question:
<!-- Please only use this template for submitting enhancement requests --What would you like to be : a new addon to addon manager "that automatically installs CSI CRDs if CSIDriverRegistry or CSINodeInfo feature gates are true". But this is either broken or this addon manager isn't active in a cluster brough up with: This should work automatically also for such local clusters. Why is this needed: The CSI E2E tests are now expecting that the cluster has the CRDs and fail when that isn't the case. /kind feature (or perhaps a bug, not sure) /cc
/assign
Is this expected to be installed in other clusters? A lot of clusters are not going to be using cluster/* scripts.

fi } <ins> create_csi_crd() { echo "create_csi_crd $1" YAML_FILE=${KUBE_ROOT}/cluster/addons/storage-crds/$1.yaml if [ -e $YAML_FILE ]; then echo "Create $1 crd" ${KUBECTL} --kubeconfig="${CERT_DIR}/admin.kubeconfig" create -f $YAML_FILE else echo "No $1 available." fi } </ins> function print_success { if [[ "${START_MODE}" != "kubeletonly" ]]; then if [[ "${ENABLE_DAEMON}" = false ]]; then


Question:
<!-- Please only use this template for submitting enhancement requests --What would you like to be : a new addon to addon manager "that automatically installs CSI CRDs if CSIDriverRegistry or CSINodeInfo feature gates are true". But this is either broken or this addon manager isn't active in a cluster brough up with: This should work automatically also for such local clusters. Why is this needed: The CSI E2E tests are now expecting that the cluster has the CRDs and fail when that isn't the case. /kind feature (or perhaps a bug, not sure) /cc
/assign
Is this expected to be installed in other clusters? A lot of clusters are not going to be using cluster/* scripts.

create_storage_class fi <ins> if [[ "${FEATURE_GATES:-}" == "AllAlpha=true" || "${FEATURE_GATES:-}" =~ "CSIDriverRegistry=true" ]]; then create_csi_crd "csidriver" fi if [[ "${FEATURE_GATES:-}" == "AllAlpha=true" || "${FEATURE_GATES:-}" =~ "CSINodeInfo=true" ]]; then create_csi_crd "csinodeinfo" fi </ins> print_success if [[ "${ENABLE_DAEMON}" = false ]]; then


Question:
<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: gce-new-master-upgrade-cluster-new-parallel Which test(s) are failing: Varies, but all subpath failures, including: [sig-storage] CSI Volumes [Driver: csi-hostpath-v0] [Testpattern: Dynamic PV (default fs)] subPath should fail if subpath with backstepping is outside the volume [Slow] [sig-storage] CSI Volumes [Driver: csi-hostpath] [Testpattern: Dynamic PV (default fs)] subPath should fail if subpath directory is outside the volume [Slow] [sig-storage] CSI Volumes [Driver: csi-hostpath-v0] [Testpattern: Dynamic PV (default fs)] subPath should fail if subpath file is outside the volume [Slow] [sig-storage] CSI Volumes [Driver: csi-hostpath] [Testpattern: Dynamic PV (default fs)] subPath should fail if subpath file is outside the volume [Slow] ... and pretty much every other subpath test, but never all of them at once. There's also a few other storage tests failing, such as: [sig-storage] Volume expand [Slow] Verify if editing PVC allows resize [sig-storage] Detaching volumes should not work when mount is in progress Since when has it been failing: 11/22 Testgrid link: https://k8s- Reason for failure: These flakes started around the time that merged, but doesn't match up with the exact merge stamp, so it's probably coincidental. The subpath test failures seem to be mostly timeouts: ... so possibly this is just GCE fail. Anything else we need to know: This test job has always been flaky, with around a 40% failure rate. <!-- DO NOT EDIT BELOW THIS LINE --/kind flake /sig storage /priority important-soon cc
can you please help investigate this. It looks to me like the tests are flaking for and . I wonder if the v0 tests are racing with the non-v0 tests for something in this test?
The other tests including for the other CSI driver and non-CSI in-tree plugins are solid green which also makes me think this is a issue with the hostpath tests or driver.
(just adding additional similar flakes here for tracking) Similar flakes in :
adding to the milestone for triage since the flakes look new
These all 4 subpath tests call and it fails in finding a specific failed event by using . As uses to check if the specific error event happens, and , I guess that tests may flake if other failed event happens before the expected one. Also, already wait for the event, we might be able to delete  in . I will create a PR to fix above.
/priority critical-urgent /remove-priority important-soon /milestone v1.13
while we should update the test with your PR, I'm not sure if it's going to fix the failure. Looking at one failing test log, these are the events I see on the failing pod: So it seems to be failing before even reaching the subpath logic.
Timeline of events: Test pod started getting processed by kubelet: It took a minute after the test pod was created for the csi driver to finish registering and mount to succeed, and then subpath event to trigger: So should address the issue. I was initially confused because the test logs didn't show the subpath event for some reason
I opened up to investigate if 1 minute for csi plugin registration is expected
I've a tracker issue for the complete list of storage flakes: This is so that we can track whether we're addressing most of the flakes with the subpath fixes, or if additional fixes are required.
keeping open until flakes are verified cleaned up
/reopen Reopening this issue until we can verify the fix in CI
As of this morning we are seeing CSI "subpath should fail" flakes in the following job Multiple "subpath should fail" flakes in this slow job alone Multiple "subpath should fail" flakes in the 1.13 slow job, similar to master. The following 2 tests are flaky only in this upgrade job can you help us determine if these are cases are flaky tests that you are fixing or is there an underlying CSI bug that needs addressing?
root cause of this is a delay in an external sidecar containers and the tests not waiting long enough to accommodate the delay. There are no changes required in (outside of test) to handle this. These should not be 1.13 blocking. We'll continue to look in to getting them non-flaky today. In the meantime, we can mark these tests as flaky to reduce the noise, if you'd like. looks like the same issue () is still being hit. See : And logs from (): We need to make the tests handle this condition more gracefully. The suggestion here is one possible solution.
Would be good to capture this as a known issue in release notes.
to followup on capturing this in Release notes
per , can you create a targeted fix increasing the timeout? Looks like marking these tests as flaky will be difficult for 1.13, but if that doesn't fix it we will need to do that.
Short term fix to bump the test timeout:
As per discussion in release burndown meeting, this is not a 1.13 blocker. please move this issue to 1.14 if there are no pending issues you are concerned about
Ok looks like the test still flaked. Looking at , the mount succeeded after a minute and proceeded to subpath processing, which (correctly) failed. However, for some resaon, the test is not seeing the subpath event and still failed after 4 minutes. On kubelet, subpath processing failed after a minute and generated the event: But the test doesn't see it for some reason, and still times out after 4 more minutes:
I saw the same on CSI volume mount succeeded at: Kubelet correctly failed the pod because of subpath But test still failed: Because the test failed to find the pod event. This looks like a test issue. Not a 1.13 blocker.
can you PTAL at this
Is there a way to check if events are overloaded and not able to handle new events? The MountVolume retries while the driver is not registered is very frequent and could have overloaded the event server.
Checking for the existence of an event seems like the wrong way to test this. Would checking pod status alone be insufficient? Ultimately that is what we really care about.
Checking for the specific event guarantees that we are failing with the correct error. But perhaps events are not reliable enough to depend on. Checking if the Pod never came up running would probably reduce the flakes but could potentially mask false negatives. cc for any thoughts.
Tests are still flaking and actually it seems like in general storage flakes in slow are flaking more since 11-21 or so: https://k8s- My theory is that the driver registration delay is causing a flood of events, which causes tests that check for events to fail. I think this is revealing multiple issues in both the tests and product, and we should work on fixing them all, although any one of the fixes should mitigate the test flakes: Driver registrar can sometimes take a minute to come up: Volume manager operation executor doesn't seem to be throttling retries or backing off: Tests should not rely on events: Tests should wait for CSI driver to be registered on all nodes before creating test pods:
and have been merged and will address the biggest issues. has been backported to 1.13 already, backport is still pending.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

if err != nil { return false, fmt.Errorf("got error while getting pod events: %s", err) } <del> if len(events.Items) == 0 { return false, nil // no events have occurred yet </del> <ins> for _, event := range events.Items { if strings.Contains(event.Message, msg) { return true, nil } </ins> } <del> return strings.Contains(events.Items[0].Message, msg), nil </del> <ins> return false, nil </ins> } }


Question:
<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: gce-new-master-upgrade-cluster-new-parallel Which test(s) are failing: Varies, but all subpath failures, including: [sig-storage] CSI Volumes [Driver: csi-hostpath-v0] [Testpattern: Dynamic PV (default fs)] subPath should fail if subpath with backstepping is outside the volume [Slow] [sig-storage] CSI Volumes [Driver: csi-hostpath] [Testpattern: Dynamic PV (default fs)] subPath should fail if subpath directory is outside the volume [Slow] [sig-storage] CSI Volumes [Driver: csi-hostpath-v0] [Testpattern: Dynamic PV (default fs)] subPath should fail if subpath file is outside the volume [Slow] [sig-storage] CSI Volumes [Driver: csi-hostpath] [Testpattern: Dynamic PV (default fs)] subPath should fail if subpath file is outside the volume [Slow] ... and pretty much every other subpath test, but never all of them at once. There's also a few other storage tests failing, such as: [sig-storage] Volume expand [Slow] Verify if editing PVC allows resize [sig-storage] Detaching volumes should not work when mount is in progress Since when has it been failing: 11/22 Testgrid link: https://k8s- Reason for failure: These flakes started around the time that merged, but doesn't match up with the exact merge stamp, so it's probably coincidental. The subpath test failures seem to be mostly timeouts: ... so possibly this is just GCE fail. Anything else we need to know: This test job has always been flaky, with around a 40% failure rate. <!-- DO NOT EDIT BELOW THIS LINE --/kind flake /sig storage /priority important-soon cc
can you please help investigate this. It looks to me like the tests are flaking for and . I wonder if the v0 tests are racing with the non-v0 tests for something in this test?
The other tests including for the other CSI driver and non-CSI in-tree plugins are solid green which also makes me think this is a issue with the hostpath tests or driver.
(just adding additional similar flakes here for tracking) Similar flakes in :
adding to the milestone for triage since the flakes look new
These all 4 subpath tests call and it fails in finding a specific failed event by using . As uses to check if the specific error event happens, and , I guess that tests may flake if other failed event happens before the expected one. Also, already wait for the event, we might be able to delete  in . I will create a PR to fix above.
/priority critical-urgent /remove-priority important-soon /milestone v1.13
while we should update the test with your PR, I'm not sure if it's going to fix the failure. Looking at one failing test log, these are the events I see on the failing pod: So it seems to be failing before even reaching the subpath logic.
Timeline of events: Test pod started getting processed by kubelet: It took a minute after the test pod was created for the csi driver to finish registering and mount to succeed, and then subpath event to trigger: So should address the issue. I was initially confused because the test logs didn't show the subpath event for some reason
I opened up to investigate if 1 minute for csi plugin registration is expected
I've a tracker issue for the complete list of storage flakes: This is so that we can track whether we're addressing most of the flakes with the subpath fixes, or if additional fixes are required.
keeping open until flakes are verified cleaned up
/reopen Reopening this issue until we can verify the fix in CI
As of this morning we are seeing CSI "subpath should fail" flakes in the following job Multiple "subpath should fail" flakes in this slow job alone Multiple "subpath should fail" flakes in the 1.13 slow job, similar to master. The following 2 tests are flaky only in this upgrade job can you help us determine if these are cases are flaky tests that you are fixing or is there an underlying CSI bug that needs addressing?
root cause of this is a delay in an external sidecar containers and the tests not waiting long enough to accommodate the delay. There are no changes required in (outside of test) to handle this. These should not be 1.13 blocking. We'll continue to look in to getting them non-flaky today. In the meantime, we can mark these tests as flaky to reduce the noise, if you'd like. looks like the same issue () is still being hit. See : And logs from (): We need to make the tests handle this condition more gracefully. The suggestion here is one possible solution.
Would be good to capture this as a known issue in release notes.
to followup on capturing this in Release notes
per , can you create a targeted fix increasing the timeout? Looks like marking these tests as flaky will be difficult for 1.13, but if that doesn't fix it we will need to do that.
Short term fix to bump the test timeout:
As per discussion in release burndown meeting, this is not a 1.13 blocker. please move this issue to 1.14 if there are no pending issues you are concerned about
Ok looks like the test still flaked. Looking at , the mount succeeded after a minute and proceeded to subpath processing, which (correctly) failed. However, for some resaon, the test is not seeing the subpath event and still failed after 4 minutes. On kubelet, subpath processing failed after a minute and generated the event: But the test doesn't see it for some reason, and still times out after 4 more minutes:
I saw the same on CSI volume mount succeeded at: Kubelet correctly failed the pod because of subpath But test still failed: Because the test failed to find the pod event. This looks like a test issue. Not a 1.13 blocker.
can you PTAL at this
Is there a way to check if events are overloaded and not able to handle new events? The MountVolume retries while the driver is not registered is very frequent and could have overloaded the event server.
Checking for the existence of an event seems like the wrong way to test this. Would checking pod status alone be insufficient? Ultimately that is what we really care about.
Checking for the specific event guarantees that we are failing with the correct error. But perhaps events are not reliable enough to depend on. Checking if the Pod never came up running would probably reduce the flakes but could potentially mask false negatives. cc for any thoughts.
Tests are still flaking and actually it seems like in general storage flakes in slow are flaking more since 11-21 or so: https://k8s- My theory is that the driver registration delay is causing a flood of events, which causes tests that check for events to fail. I think this is revealing multiple issues in both the tests and product, and we should work on fixing them all, although any one of the fixes should mitigate the test flakes: Driver registrar can sometimes take a minute to come up: Volume manager operation executor doesn't seem to be throttling retries or backing off: Tests should not rely on events: Tests should wait for CSI driver to be registered on all nodes before creating test pods:
and have been merged and will address the biggest issues. has been backported to 1.13 already, backport is still pending.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

defer func() { framework.DeletePodWithWait(f, f.ClientSet, pod) }() <del> err = framework.WaitForPodRunningInNamespace(f.ClientSet, pod) Expect(err).To(HaveOccurred(), "while waiting for pod to be running") </del> By("Checking for subpath error event") selector := fields.Set{


Question:
What happened: POST or PUT to a custom resource with a JSON body missing "kind" results in a 500 error: A missing "apiVersion" also results in a 500 error: Then with both "kind" and "apiVersion" defined we get the normal 4XX return codes. What you expected to happen: All bad inputs for create and update requests should be 4XX return codes. E.g. Creating a with body of results in the 422 error: How to reproduce it: Create a CRD or use a cluster that already has a one prepared (GKE 1.11.0+ clusters have the CRD installed which you could test against). HTTP to that custom resource with a JSON body of . E.g. Environment: Kubernetes version (use ): 1.11.2 Cloud provider or hardware configuration: GKE OS (e.g. from /etc/os-release): Kernel (e.g. ): /sig api-machinery <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
This is fixed on master, but I can reproduce it on 1.11. Posting a custom resource with a JSON body of : on master: on 1.11:
we are transforming all non-status error into 500 status error. in this case, it's the decoder returning a which is non-status. a quick fix by . (i like this number)
instead of fixed, it's rather been shadowed by i suppose. now when that empty unstructured obj going thru the decoder, the gvk is explicitly coerced, so this case is not reproducable on master for now.
/cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
I get the same response when trying to update the custom resource's 'status' sub-resource. On k8s v1.11.4
/assign
submitting the specified object on master returns a 400 badrequest, as desired /close
Closing this issue. <detailsIn response to : submitting the specified object on master returns a 400 badrequest, as desired Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question:
What happened: POST or PUT to a custom resource with a JSON body missing "kind" results in a 500 error: A missing "apiVersion" also results in a 500 error: Then with both "kind" and "apiVersion" defined we get the normal 4XX return codes. What you expected to happen: All bad inputs for create and update requests should be 4XX return codes. E.g. Creating a with body of results in the 422 error: How to reproduce it: Create a CRD or use a cluster that already has a one prepared (GKE 1.11.0+ clusters have the CRD installed which you could test against). HTTP to that custom resource with a JSON body of . E.g. Environment: Kubernetes version (use ): 1.11.2 Cloud provider or hardware configuration: GKE OS (e.g. from /etc/os-release): Kernel (e.g. ): /sig api-machinery <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
This is fixed on master, but I can reproduce it on 1.11. Posting a custom resource with a JSON body of : on master: on 1.11:
we are transforming all non-status error into 500 status error. in this case, it's the decoder returning a which is non-status. a quick fix by . (i like this number)
instead of fixed, it's rather been shadowed by i suppose. now when that empty unstructured obj going thru the decoder, the gvk is explicitly coerced, so this case is not reproducable on master for now.
/cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
I get the same response when trying to update the custom resource's 'status' sub-resource. On k8s v1.11.4
/assign
submitting the specified object on master returns a 400 badrequest, as desired /close
Closing this issue. <detailsIn response to : submitting the specified object on master returns a 400 badrequest, as desired Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

"//staging/src/k8s.io/apiserver/pkg/apis/example/v1:go_default_library", "//staging/src/k8s.io/apiserver/pkg/endpoints/request:go_default_library", "//staging/src/k8s.io/apiserver/pkg/registry/rest:go_default_library", <ins> "//staging/src/k8s.io/client-go/kubernetes/scheme:go_default_library", </ins> "//vendor/github.com/evanphx/json-patch:go_default_library", "//vendor/github.com/google/gofuzz:go_default_library", "//vendor/k8s.io/utils/trace:go_default_library",


Question:
What happened: POST or PUT to a custom resource with a JSON body missing "kind" results in a 500 error: A missing "apiVersion" also results in a 500 error: Then with both "kind" and "apiVersion" defined we get the normal 4XX return codes. What you expected to happen: All bad inputs for create and update requests should be 4XX return codes. E.g. Creating a with body of results in the 422 error: How to reproduce it: Create a CRD or use a cluster that already has a one prepared (GKE 1.11.0+ clusters have the CRD installed which you could test against). HTTP to that custom resource with a JSON body of . E.g. Environment: Kubernetes version (use ): 1.11.2 Cloud provider or hardware configuration: GKE OS (e.g. from /etc/os-release): Kernel (e.g. ): /sig api-machinery <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
This is fixed on master, but I can reproduce it on 1.11. Posting a custom resource with a JSON body of : on master: on 1.11:
we are transforming all non-status error into 500 status error. in this case, it's the decoder returning a which is non-status. a quick fix by . (i like this number)
instead of fixed, it's rather been shadowed by i suppose. now when that empty unstructured obj going thru the decoder, the gvk is explicitly coerced, so this case is not reproducable on master for now.
/cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
I get the same response when trying to update the custom resource's 'status' sub-resource. On k8s v1.11.4
/assign
submitting the specified object on master returns a 400 badrequest, as desired /close
Closing this issue. <detailsIn response to : submitting the specified object on master returns a 400 badrequest, as desired Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

} } <del> // transformDecodeError adds additional information when a decode fails. </del> <ins> // transformDecodeError adds additional information into a bad-request api error when a decode fails. </ins> func transformDecodeError(typer runtime.ObjectTyper, baseErr error, into runtime.Object, gvk *schema.GroupVersionKind, body []byte) error { objGVKs, _, err := typer.ObjectKinds(into) if err != nil { <del> return err </del> <ins> return errors.NewBadRequest(err.Error()) </ins> } objGVK := objGVKs[0] if gvk != nil && len(gvk.Kind) > 0 {


Question:
What happened: POST or PUT to a custom resource with a JSON body missing "kind" results in a 500 error: A missing "apiVersion" also results in a 500 error: Then with both "kind" and "apiVersion" defined we get the normal 4XX return codes. What you expected to happen: All bad inputs for create and update requests should be 4XX return codes. E.g. Creating a with body of results in the 422 error: How to reproduce it: Create a CRD or use a cluster that already has a one prepared (GKE 1.11.0+ clusters have the CRD installed which you could test against). HTTP to that custom resource with a JSON body of . E.g. Environment: Kubernetes version (use ): 1.11.2 Cloud provider or hardware configuration: GKE OS (e.g. from /etc/os-release): Kernel (e.g. ): /sig api-machinery <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
This is fixed on master, but I can reproduce it on 1.11. Posting a custom resource with a JSON body of : on master: on 1.11:
we are transforming all non-status error into 500 status error. in this case, it's the decoder returning a which is non-status. a quick fix by . (i like this number)
instead of fixed, it's rather been shadowed by i suppose. now when that empty unstructured obj going thru the decoder, the gvk is explicitly coerced, so this case is not reproducable on master for now.
/cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
I get the same response when trying to update the custom resource's 'status' sub-resource. On k8s v1.11.4
/assign
submitting the specified object on master returns a 400 badrequest, as desired /close
Closing this issue. <detailsIn response to : submitting the specified object on master returns a 400 badrequest, as desired Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

apierrors "k8s.io/apimachinery/pkg/api/errors" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" <ins> testapigroupv1 "k8s.io/apimachinery/pkg/apis/testapigroup/v1" </ins> "k8s.io/apimachinery/pkg/runtime" "k8s.io/apimachinery/pkg/runtime/schema" "k8s.io/apimachinery/pkg/runtime/serializer"


Question:
What happened: POST or PUT to a custom resource with a JSON body missing "kind" results in a 500 error: A missing "apiVersion" also results in a 500 error: Then with both "kind" and "apiVersion" defined we get the normal 4XX return codes. What you expected to happen: All bad inputs for create and update requests should be 4XX return codes. E.g. Creating a with body of results in the 422 error: How to reproduce it: Create a CRD or use a cluster that already has a one prepared (GKE 1.11.0+ clusters have the CRD installed which you could test against). HTTP to that custom resource with a JSON body of . E.g. Environment: Kubernetes version (use ): 1.11.2 Cloud provider or hardware configuration: GKE OS (e.g. from /etc/os-release): Kernel (e.g. ): /sig api-machinery <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
This is fixed on master, but I can reproduce it on 1.11. Posting a custom resource with a JSON body of : on master: on 1.11:
we are transforming all non-status error into 500 status error. in this case, it's the decoder returning a which is non-status. a quick fix by . (i like this number)
instead of fixed, it's rather been shadowed by i suppose. now when that empty unstructured obj going thru the decoder, the gvk is explicitly coerced, so this case is not reproducable on master for now.
/cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
I get the same response when trying to update the custom resource's 'status' sub-resource. On k8s v1.11.4
/assign
submitting the specified object on master returns a 400 badrequest, as desired /close
Closing this issue. <detailsIn response to : submitting the specified object on master returns a 400 badrequest, as desired Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

examplev1 "k8s.io/apiserver/pkg/apis/example/v1" "k8s.io/apiserver/pkg/endpoints/request" "k8s.io/apiserver/pkg/registry/rest" <ins> clientgoscheme "k8s.io/client-go/kubernetes/scheme" </ins> utiltrace "k8s.io/utils/trace" )


Question:
What happened: POST or PUT to a custom resource with a JSON body missing "kind" results in a 500 error: A missing "apiVersion" also results in a 500 error: Then with both "kind" and "apiVersion" defined we get the normal 4XX return codes. What you expected to happen: All bad inputs for create and update requests should be 4XX return codes. E.g. Creating a with body of results in the 422 error: How to reproduce it: Create a CRD or use a cluster that already has a one prepared (GKE 1.11.0+ clusters have the CRD installed which you could test against). HTTP to that custom resource with a JSON body of . E.g. Environment: Kubernetes version (use ): 1.11.2 Cloud provider or hardware configuration: GKE OS (e.g. from /etc/os-release): Kernel (e.g. ): /sig api-machinery <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
This is fixed on master, but I can reproduce it on 1.11. Posting a custom resource with a JSON body of : on master: on 1.11:
we are transforming all non-status error into 500 status error. in this case, it's the decoder returning a which is non-status. a quick fix by . (i like this number)
instead of fixed, it's rather been shadowed by i suppose. now when that empty unstructured obj going thru the decoder, the gvk is explicitly coerced, so this case is not reproducable on master for now.
/cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
I get the same response when trying to update the custom resource's 'status' sub-resource. On k8s v1.11.4
/assign
submitting the specified object on master returns a 400 badrequest, as desired /close
Closing this issue. <detailsIn response to : submitting the specified object on master returns a 400 badrequest, as desired Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

func (f mutateObjectUpdateFunc) Admit(a admission.Attributes, o admission.ObjectInterfaces) (err error) { return f(a.GetObject(), a.GetOldObject()) } <ins> func TestTransformDecodeErrorEnsuresBadRequestError(t *testing.T) { testCases := []struct { name             string typer            runtime.ObjectTyper decodedGVK       *schema.GroupVersionKind decodeIntoObject runtime.Object baseErr          error expectedErr      error }{ { name:  "decoding normal objects fails and returns a bad-request error", typer: clientgoscheme.Scheme, decodedGVK: &schema.GroupVersionKind{ Group:   testapigroupv1.GroupName, Version: "v1", Kind:    "Carp", }, decodeIntoObject: &testapigroupv1.Carp{}, // which client-go's scheme doesn't recognize baseErr:          fmt.Errorf("plain error"), }, { name:             "decoding objects with unknown GVK fails and returns a bad-request error", typer:            alwaysErrorTyper{}, decodedGVK:       nil, decodeIntoObject: &testapigroupv1.Carp{}, // which client-go's scheme doesn't recognize baseErr:          nil, }, } for _, testCase := range testCases { err := transformDecodeError(testCase.typer, testCase.baseErr, testCase.decodeIntoObject, testCase.decodedGVK, []byte(``)) if apiStatus, ok := err.(apierrors.APIStatus); !ok || apiStatus.Status().Code != http.StatusBadRequest { t.Errorf("expected bad request error but got: %v", err) } } } var _ runtime.ObjectTyper = alwaysErrorTyper{} type alwaysErrorTyper struct{} func (alwaysErrorTyper) ObjectKinds(runtime.Object) ([]schema.GroupVersionKind, bool, error) { return nil, false, fmt.Errorf("always error") } func (alwaysErrorTyper) Recognizes(gvk schema.GroupVersionKind) bool { return false } </ins>


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: If kubelet plugin registration fails during the registration process, the kubelet never attempts to register the socket again (assuming the error was transient), unless the socket for the driver is deleted & recreated to trigger a new registration. What you expected to happen: Maybe retry with exponential backoff when kubelet plugin registration fails instead of just logging and waiting for another filesystem event like it does today. Of course, if a socket returns a terminal error (does not implement the kubelet plugin registration service, for example), we should not retry. Follow up on /milestone v1.14 /sig storage How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
I'd like to work on this issue
/assign
Re opening. was a temporary work around for this. is the permanent fix.
Hello! We are starting Bug Freeze for 1.15. Is this issue still planned on being incorporated into 1.15?
the dependent PR is not yet merged. can we get this in for 1.15?
said that is the "permanent fix." and that has merged. So let's close this out /close
Closing this issue. <detailsIn response to : said that is the "permanent fix." and that has merged. So let's close this out Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

"fmt" "io" "net" <ins> "strings" </ins> "time" csipbv1 "github.com/container-storage-interface/spec/lib/go/csi" "google.golang.org/grpc" api "k8s.io/api/core/v1" utilversion "k8s.io/apimachinery/pkg/util/version" <ins> "k8s.io/apimachinery/pkg/util/wait" </ins> utilfeature "k8s.io/apiserver/pkg/util/feature" "k8s.io/klog" "k8s.io/kubernetes/pkg/features"


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: If kubelet plugin registration fails during the registration process, the kubelet never attempts to register the socket again (assuming the error was transient), unless the socket for the driver is deleted & recreated to trigger a new registration. What you expected to happen: Maybe retry with exponential backoff when kubelet plugin registration fails instead of just logging and waiting for another filesystem event like it does today. Of course, if a socket returns a terminal error (does not implement the kubelet plugin registration service, for example), we should not retry. Follow up on /milestone v1.14 /sig storage How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
I'd like to work on this issue
/assign
Re opening. was a temporary work around for this. is the permanent fix.
Hello! We are starting Bug Freeze for 1.15. Is this issue still planned on being incorporated into 1.15?
the dependent PR is not yet merged. can we get this in for 1.15?
said that is the "permanent fix." and that has merged. So let's close this out /close
Closing this issue. <detailsIn response to : said that is the "permanent fix." and that has merged. So let's close this out Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

err error, ) <ins> const ( initialDuration = 1 * time.Second factor          = 2.0 steps           = 5 ) </ins> // newV1NodeClient creates a new NodeClient with the internally used gRPC // connection set up. It also returns a closer which must to be called to close // the gRPC connection when the NodeClient is not used anymore.


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: If kubelet plugin registration fails during the registration process, the kubelet never attempts to register the socket again (assuming the error was transient), unless the socket for the driver is deleted & recreated to trigger a new registration. What you expected to happen: Maybe retry with exponential backoff when kubelet plugin registration fails instead of just logging and waiting for another filesystem event like it does today. Of course, if a socket returns a terminal error (does not implement the kubelet plugin registration service, for example), we should not retry. Follow up on /milestone v1.14 /sig storage How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
I'd like to work on this issue
/assign
Re opening. was a temporary work around for this. is the permanent fix.
Hello! We are starting Bug Freeze for 1.15. Is this issue still planned on being incorporated into 1.15?
the dependent PR is not yet merged. can we get this in for 1.15?
said that is the "permanent fix." and that has merged. So let's close this out /close
Closing this issue. <detailsIn response to : said that is the "permanent fix." and that has merged. So let's close this out Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

accessibleTopology map[string]string, err error) { klog.V(4).Info(log("calling NodeGetInfo rpc")) <del> if c.nodeV1ClientCreator != nil { return c.nodeGetInfoV1(ctx) } else if c.nodeV0ClientCreator != nil { return c.nodeGetInfoV0(ctx) } err = fmt.Errorf("failed to call NodeGetInfo. Both nodeV1ClientCreator and nodeV0ClientCreator are nil") </del> <ins> // TODO retries should happen at a lower layer (issue #73371) backoff := wait.Backoff{Duration: initialDuration, Factor: factor, Steps: steps} err = wait.ExponentialBackoff(backoff, func() (bool, error) { var getNodeInfoError error if c.nodeV1ClientCreator != nil { nodeID, maxVolumePerNode, accessibleTopology, getNodeInfoError = c.nodeGetInfoV1(ctx) } else if c.nodeV0ClientCreator != nil { nodeID, maxVolumePerNode, accessibleTopology, getNodeInfoError = c.nodeGetInfoV0(ctx) } if nodeID != "" { return true, nil } // kubelet plugin registration service not implemented is a terminal error, no need to retry if strings.Contains(getNodeInfoError.Error(), "no handler registered for plugin type") { return false, getNodeInfoError } // Continue with exponential backoff return false, nil }) </ins> return nodeID, maxVolumePerNode, accessibleTopology, err }


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: If kubelet plugin registration fails during the registration process, the kubelet never attempts to register the socket again (assuming the error was transient), unless the socket for the driver is deleted & recreated to trigger a new registration. What you expected to happen: Maybe retry with exponential backoff when kubelet plugin registration fails instead of just logging and waiting for another filesystem event like it does today. Of course, if a socket returns a terminal error (does not implement the kubelet plugin registration service, for example), we should not retry. Follow up on /milestone v1.14 /sig storage How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
I'd like to work on this issue
/assign
Re opening. was a temporary work around for this. is the permanent fix.
Hello! We are starting Bug Freeze for 1.15. Is this issue still planned on being incorporated into 1.15?
the dependent PR is not yet merged. can we get this in for 1.15?
said that is the "permanent fix." and that has merged. So let's close this out /close
Closing this issue. <detailsIn response to : said that is the "permanent fix." and that has merged. So let's close this out Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

return err } <del> // TODO (verult) retry with exponential backoff, possibly added in csi client library. </del> ctx, cancel := context.WithTimeout(context.Background(), csiTimeout) defer cancel()


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: If kubelet plugin registration fails during the registration process, the kubelet never attempts to register the socket again (assuming the error was transient), unless the socket for the driver is deleted & recreated to trigger a new registration. What you expected to happen: Maybe retry with exponential backoff when kubelet plugin registration fails instead of just logging and waiting for another filesystem event like it does today. Of course, if a socket returns a terminal error (does not implement the kubelet plugin registration service, for example), we should not retry. Follow up on /milestone v1.14 /sig storage How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
I'd like to work on this issue
/assign
Re opening. was a temporary work around for this. is the permanent fix.
Hello! We are starting Bug Freeze for 1.15. Is this issue still planned on being incorporated into 1.15?
the dependent PR is not yet merged. can we get this in for 1.15?
said that is the "permanent fix." and that has merged. So let's close this out /close
Closing this issue. <detailsIn response to : said that is the "permanent fix." and that has merged. So let's close this out Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

if err != nil { klog.Error(log("registrationHandler.RegisterPlugin failed at CSI.NodeGetInfo: %v", err)) if unregErr := unregisterDriver(pluginName); unregErr != nil { <del> klog.Error(log("registrationHandler.RegisterPlugin failed to unregister plugin due to previous: %v", unregErr)) </del> <ins> klog.Error(log("registrationHandler.RegisterPlugin failed to unregister plugin due to previous error: %v", unregErr)) </ins> return unregErr } return err


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: In the test failed due to namespace leakage: we created e2e-tests-load-30-nodepods-1-pfn9m, but never got response from api server (got 0-length response instead) we retried and created e2e-tests-load-30-nodepods-1-m9tq5 e2e-tests-load-30-nodepods-1-m9tq5 to f.namespacesToDelete and leaked -pfn9m next test case failed because e2e-tests-load-30-nodepods-1-pfn9m wasn't deleted. What you expected to happen: While it's unclear why we got 0-length response instead ( investigates that), we always can get error while the namespace was created (e.g. connection reset, timeout, io error). We should handle this case in retrying correctly. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): v1.14.0-alpha.0.793+ Cloud provider or hardware configuration: gce OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
There are no sig labels on this issue. Please add a sig label by either: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. See the . The in method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc

return fmt.Errorf("PersistentVolumeClaims %v not all in phase %s within %v", pvcNames, phase, timeout) } <ins> // findAvailableNamespaceName random namespace name starting with baseName. func findAvailableNamespaceName(baseName string, c clientset.Interface) (string, error) { var name string err := wait.PollImmediate(Poll, 30*time.Second, func() (bool, error) { name = fmt.Sprintf("%v-%v", baseName, randomSuffix()) _, err := c.CoreV1().Namespaces().Get(name, metav1.GetOptions{}) if err == nil { // Already taken return false, nil } if apierrs.IsNotFound(err) { return true, nil } Logf("Unexpected error while getting namespace: %v", err) return false, nil }) return name, err } </ins> // CreateTestingNS should be used by every test, note that we append a common prefix to the provided test name. // Please see NewFramework instead of using this directly. func CreateTestingNS(baseName string, c clientset.Interface, labels map[string]string) (*v1.Namespace, error) {


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: In the test failed due to namespace leakage: we created e2e-tests-load-30-nodepods-1-pfn9m, but never got response from api server (got 0-length response instead) we retried and created e2e-tests-load-30-nodepods-1-m9tq5 e2e-tests-load-30-nodepods-1-m9tq5 to f.namespacesToDelete and leaked -pfn9m next test case failed because e2e-tests-load-30-nodepods-1-pfn9m wasn't deleted. What you expected to happen: While it's unclear why we got 0-length response instead ( investigates that), we always can get error while the namespace was created (e.g. connection reset, timeout, io error). We should handle this case in retrying correctly. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): v1.14.0-alpha.0.793+ Cloud provider or hardware configuration: gce OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
There are no sig labels on this issue. Please add a sig label by either: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. See the . The in method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc

} labels["e2e-run"] = string(RunId) <ins> // We don't use ObjectMeta.GenerateName feature, as in case of API call // failure we don't know whether the namespace was created and what is its // name. name, err := findAvailableNamespaceName(baseName, c) if err != nil { return nil, err } </ins> namespaceObj := &v1.Namespace{ ObjectMeta: metav1.ObjectMeta{ <del> GenerateName: fmt.Sprintf("e2e-tests-%v-", baseName), Namespace:    "", Labels:       labels, </del> <ins> Name:      name, Namespace: "", Labels:    labels, </ins> }, Status: v1.NamespaceStatus{}, }


Question:
What happened: AAD support in kubectl does not work for sovereign cloud The 'environment' parameter was introduced to support sovereign cloud, but it works in first round. The 'environment' value is not persisted, so follow-up requests will fail. What you expected to happen: 'environment' parameter persisted in kubectl config file. How to reproduce it (as minimally and precisely as possible): Use kubectl to connect to Azure sovereign cloud, make the token expire, and do auth. Anything else we need to know?: Environment: Kubernetes version (use ): 1.11 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
There are no sig labels on this issue. Please add a sig label by either: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. See the . The in method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

type azureToken struct { token       adal.Token <ins> environment string </ins> clientID    string tenantID    string apiserverID string


Question:
What happened: AAD support in kubectl does not work for sovereign cloud The 'environment' parameter was introduced to support sovereign cloud, but it works in first round. The 'environment' value is not persisted, so follow-up requests will fail. What you expected to happen: 'environment' parameter persisted in kubectl config file. How to reproduce it (as minimally and precisely as possible): Use kubectl to connect to Azure sovereign cloud, make the token expire, and do auth. Anything else we need to know?: Environment: Kubernetes version (use ): 1.11 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
There are no sig labels on this issue. Please add a sig label by either: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. See the . The in method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

if refreshToken == "" { return nil, fmt.Errorf("no refresh token in cfg: %s", cfgRefreshToken) } <ins> environment := ts.cfg[cfgEnvironment] if environment == "" { return nil, fmt.Errorf("no environment in cfg: %s", cfgEnvironment) } </ins> clientID := ts.cfg[cfgClientID] if clientID == "" { return nil, fmt.Errorf("no client ID in cfg: %s", cfgClientID)


Question:
What happened: AAD support in kubectl does not work for sovereign cloud The 'environment' parameter was introduced to support sovereign cloud, but it works in first round. The 'environment' value is not persisted, so follow-up requests will fail. What you expected to happen: 'environment' parameter persisted in kubectl config file. How to reproduce it (as minimally and precisely as possible): Use kubectl to connect to Azure sovereign cloud, make the token expire, and do auth. Anything else we need to know?: Environment: Kubernetes version (use ): 1.11 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
There are no sig labels on this issue. Please add a sig label by either: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. See the . The in method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

Resource:     fmt.Sprintf("spn:%s", apiserverID), Type:         tokenType, }, <ins> environment: environment, </ins> clientID:    clientID, tenantID:    tenantID, apiserverID: apiserverID,


Question:
What happened: AAD support in kubectl does not work for sovereign cloud The 'environment' parameter was introduced to support sovereign cloud, but it works in first round. The 'environment' value is not persisted, so follow-up requests will fail. What you expected to happen: 'environment' parameter persisted in kubectl config file. How to reproduce it (as minimally and precisely as possible): Use kubectl to connect to Azure sovereign cloud, make the token expire, and do auth. Anything else we need to know?: Environment: Kubernetes version (use ): 1.11 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
There are no sig labels on this issue. Please add a sig label by either: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. See the . The in method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

newCfg := make(map[string]string) newCfg[cfgAccessToken] = token.token.AccessToken newCfg[cfgRefreshToken] = token.token.RefreshToken <ins> newCfg[cfgEnvironment] = token.environment </ins> newCfg[cfgClientID] = token.clientID newCfg[cfgTenantID] = token.tenantID newCfg[cfgApiserverID] = token.apiserverID


Question:
What happened: AAD support in kubectl does not work for sovereign cloud The 'environment' parameter was introduced to support sovereign cloud, but it works in first round. The 'environment' value is not persisted, so follow-up requests will fail. What you expected to happen: 'environment' parameter persisted in kubectl config file. How to reproduce it (as minimally and precisely as possible): Use kubectl to connect to Azure sovereign cloud, make the token expire, and do auth. Anything else we need to know?: Environment: Kubernetes version (use ): 1.11 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
There are no sig labels on this issue. Please add a sig label by either: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. See the . The in method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

} func (ts *azureTokenSource) refreshToken(token *azureToken) (*azureToken, error) { <del> oauthConfig, err := adal.NewOAuthConfig(azure.PublicCloud.ActiveDirectoryEndpoint, token.tenantID) </del> <ins> env, err := azure.EnvironmentFromName(token.environment) if err != nil { return nil, err } oauthConfig, err := adal.NewOAuthConfig(env.ActiveDirectoryEndpoint, token.tenantID) </ins> if err != nil { return nil, fmt.Errorf("building the OAuth configuration for token refresh: %v", err) }


Question:
What happened: AAD support in kubectl does not work for sovereign cloud The 'environment' parameter was introduced to support sovereign cloud, but it works in first round. The 'environment' value is not persisted, so follow-up requests will fail. What you expected to happen: 'environment' parameter persisted in kubectl config file. How to reproduce it (as minimally and precisely as possible): Use kubectl to connect to Azure sovereign cloud, make the token expire, and do auth. Anything else we need to know?: Environment: Kubernetes version (use ): 1.11 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
There are no sig labels on this issue. Please add a sig label by either: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. See the . The in method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

return &azureToken{ token:       spt.Token(), <ins> environment: token.environment, </ins> clientID:    token.clientID, tenantID:    token.tenantID, apiserverID: token.apiserverID,


Question:
What happened: AAD support in kubectl does not work for sovereign cloud The 'environment' parameter was introduced to support sovereign cloud, but it works in first round. The 'environment' value is not persisted, so follow-up requests will fail. What you expected to happen: 'environment' parameter persisted in kubectl config file. How to reproduce it (as minimally and precisely as possible): Use kubectl to connect to Azure sovereign cloud, make the token expire, and do auth. Anything else we need to know?: Environment: Kubernetes version (use ): 1.11 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
There are no sig labels on this issue. Please add a sig label by either: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. See the . The in method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

return &azureToken{ token:       *token, <ins> environment: ts.environment.Name, </ins> clientID:    ts.clientID, tenantID:    ts.tenantID, apiserverID: ts.apiserverID,


Question:
What happened: AAD support in kubectl does not work for sovereign cloud The 'environment' parameter was introduced to support sovereign cloud, but it works in first round. The 'environment' value is not persisted, so follow-up requests will fail. What you expected to happen: 'environment' parameter persisted in kubectl config file. How to reproduce it (as minimally and precisely as possible): Use kubectl to connect to Azure sovereign cloud, make the token expire, and do auth. Anything else we need to know?: Environment: Kubernetes version (use ): 1.11 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
There are no sig labels on this issue. Please add a sig label by either: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. See the . The in method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

wantCfg := token2Cfg(token) persistedCfg := persiter.Cache() <ins> wantCfgLen := len(wantCfg) persistedCfgLen := len(persistedCfg) if wantCfgLen != persistedCfgLen { t.Errorf("wantCfgLen and persistedCfgLen do not match, wantCfgLen=%v, persistedCfgLen=%v", wantCfgLen, persistedCfgLen) } </ins> for k, v := range persistedCfg { if strings.Compare(v, wantCfg[k]) != 0 { t.Errorf("Token() persisted cfg %s: got %v, want %v", k, v, wantCfg[k])


Question:
What happened: AAD support in kubectl does not work for sovereign cloud The 'environment' parameter was introduced to support sovereign cloud, but it works in first round. The 'environment' value is not persisted, so follow-up requests will fail. What you expected to happen: 'environment' parameter persisted in kubectl config file. How to reproduce it (as minimally and precisely as possible): Use kubectl to connect to Azure sovereign cloud, make the token expire, and do auth. Anything else we need to know?: Environment: Kubernetes version (use ): 1.11 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
There are no sig labels on this issue. Please add a sig label by either: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. See the . The in method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

func (ts *fakeTokenSource) Token() (*azureToken, error) { return &azureToken{ token:       newFackeAzureToken(ts.accessToken, ts.expiresOn), <ins> environment: "testenv", </ins> clientID:    "fake", tenantID:    "fake", apiserverID: "fake",


Question:
What happened: AAD support in kubectl does not work for sovereign cloud The 'environment' parameter was introduced to support sovereign cloud, but it works in first round. The 'environment' value is not persisted, so follow-up requests will fail. What you expected to happen: 'environment' parameter persisted in kubectl config file. How to reproduce it (as minimally and precisely as possible): Use kubectl to connect to Azure sovereign cloud, make the token expire, and do auth. Anything else we need to know?: Environment: Kubernetes version (use ): 1.11 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
There are no sig labels on this issue. Please add a sig label by either: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. See the . The in method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

cfg := make(map[string]string) cfg[cfgAccessToken] = token.token.AccessToken cfg[cfgRefreshToken] = token.token.RefreshToken <ins> cfg[cfgEnvironment] = token.environment </ins> cfg[cfgClientID] = token.clientID cfg[cfgTenantID] = token.tenantID cfg[cfgApiserverID] = token.apiserverID


Question:
What happened: removed reporting of the OutOfDisk condition in the kubelet. A 1.12 kubelet upgraded to a 1.13 kubelet leaves the previously reported OutOfDisk condition in place, and it eventually gets switched to Unknown status. What you expected to happen: If this condition is no longer reported, it should be removed by the kubelet. How to reproduce it (as minimally and precisely as possible): a 1.12 kubelet, let it report OutOfDisk condition status. to 1.13 the condition is no longer updated a while, the node controller flips the status to Unknown Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug /sig node
The fix for this should be picked to 1.13
/assign happened to have a fix handy, if no one is working on it.
after upgrade kubelet should ignore this condition . or change it to 'DiskPressure'
Better not. What if both condition and exist?
should be picked to 1.13 also, I think.
13.2 doesn't have this patch; would it be possible to make it for 1.13.3?
Unfortunately this was not mentioned in the release notes (or I missed it), got hit by this too. 13.2 is affected too, would be good for me to see this getting fixed in 1.13.3

nodestatus.PIDPressureCondition(kl.clock.Now, kl.evictionManager.IsUnderPIDPressure, kl.recordNodeStatusEvent), nodestatus.ReadyCondition(kl.clock.Now, kl.runtimeState.runtimeErrors, kl.runtimeState.networkErrors, validateHostFunc, kl.containerManager.Status, kl.recordNodeStatusEvent), nodestatus.VolumesInUse(kl.volumeManager.ReconcilerStatesHasBeenSynced, kl.volumeManager.GetVolumesInUse), <ins> nodestatus.RemoveOutOfDiskCondition(), </ins> // TODO(mtaufen): I decided not to move this setter for now, since all it does is send an event // and record state back to the Kubelet runtime object. In the future, I'd like to isolate // these side-effects by decoupling the decisions to send events and partial status recording


Question:
What happened: removed reporting of the OutOfDisk condition in the kubelet. A 1.12 kubelet upgraded to a 1.13 kubelet leaves the previously reported OutOfDisk condition in place, and it eventually gets switched to Unknown status. What you expected to happen: If this condition is no longer reported, it should be removed by the kubelet. How to reproduce it (as minimally and precisely as possible): a 1.12 kubelet, let it report OutOfDisk condition status. to 1.13 the condition is no longer updated a while, the node controller flips the status to Unknown Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug /sig node
The fix for this should be picked to 1.13
/assign happened to have a fix handy, if no one is working on it.
after upgrade kubelet should ignore this condition . or change it to 'DiskPressure'
Better not. What if both condition and exist?
should be picked to 1.13 also, I think.
13.2 doesn't have this patch; would it be possible to make it for 1.13.3?
Unfortunately this was not mentioned in the release notes (or I missed it), got hit by this too. 13.2 is affected too, would be good for me to see this getting fixed in 1.13.3

return nil } } <ins> // RemoveOutOfDiskCondition removes stale OutOfDisk condition // OutOfDisk condition has been removed from kubelet in 1.12 func RemoveOutOfDiskCondition() Setter { return func(node *v1.Node) error { var conditions []v1.NodeCondition for i := range node.Status.Conditions { if node.Status.Conditions[i].Type != v1.NodeOutOfDisk { conditions = append(conditions, node.Status.Conditions[i]) } } node.Status.Conditions = conditions return nil } } </ins>


Question:
What happened: removed reporting of the OutOfDisk condition in the kubelet. A 1.12 kubelet upgraded to a 1.13 kubelet leaves the previously reported OutOfDisk condition in place, and it eventually gets switched to Unknown status. What you expected to happen: If this condition is no longer reported, it should be removed by the kubelet. How to reproduce it (as minimally and precisely as possible): a 1.12 kubelet, let it report OutOfDisk condition status. to 1.13 the condition is no longer updated a while, the node controller flips the status to Unknown Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug /sig node
The fix for this should be picked to 1.13
/assign happened to have a fix handy, if no one is working on it.
after upgrade kubelet should ignore this condition . or change it to 'DiskPressure'
Better not. What if both condition and exist?
should be picked to 1.13 also, I think.
13.2 doesn't have this patch; would it be possible to make it for 1.13.3?
Unfortunately this was not mentioned in the release notes (or I missed it), got hit by this too. 13.2 is affected too, would be good for me to see this getting fixed in 1.13.3

} } <ins> func TestRemoveOutOfDiskCondition(t *testing.T) { now := time.Now() var cases = []struct { desc       string inputNode  *v1.Node expectNode *v1.Node }{ { desc: "should remove stale OutOfDiskCondition from node status", inputNode: &v1.Node{ Status: v1.NodeStatus{ Conditions: []v1.NodeCondition{ *makeMemoryPressureCondition(false, now, now), { Type:   v1.NodeOutOfDisk, Status: v1.ConditionFalse, }, *makeDiskPressureCondition(false, now, now), }, }, }, expectNode: &v1.Node{ Status: v1.NodeStatus{ Conditions: []v1.NodeCondition{ *makeMemoryPressureCondition(false, now, now), *makeDiskPressureCondition(false, now, now), }, }, }, }, } for _, tc := range cases { t.Run(tc.desc, func(t *testing.T) { // construct setter setter := RemoveOutOfDiskCondition() // call setter on node if err := setter(tc.inputNode); err != nil { t.Fatalf("unexpected error: %v", err) } // check expected node assert.True(t, apiequality.Semantic.DeepEqual(tc.expectNode, tc.inputNode), "Diff: %s", diff.ObjectDiff(tc.expectNode, tc.inputNode)) }) } } </ins> // Test Helpers: // sortableNodeAddress is a type for sorting []v1.NodeAddress


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: When using the function from Python's Kuberneets API, we noticed that Pods are terminated bythe cluster without respecting their preStop hooks or terminationGracePeriodSeconds seconds. What you expected to happen: When using the eviction API, just as when using delete, Pods should be allowed to run their preStop hooks, as well as be given terminationGracePeriodSeconds before being terminated with SIGKILL. How to reproduce it (as minimally and precisely as possible): the following pod: the pod logs in another window to evict the pod. Here is a sample curl equivalent of what (1.13) is doing - happy path: Here is a sample curl equivalent of what Python's is doing (when not passed ) - broken path: The log output generated by the guinea pod on the happy path looks like this: The "Got signal 15" message shows up only after the preStop hook is finished, so ~10 seconds after the evict call is made What happens on the broken path is: We seem to be getting 2x SIGTERM, there are no messages indicating continued processing after the signal, and the signal itself is registered immediately after the eviction call is made, suggesting that preStop didn't run. Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: AWS OS (e.g. from /etc/os-release): CentOS Linux release 7.5.1804 Kernel (e.g. ): 3.10.0-862.14.4.el7.x8664 SMP Wed Sep 26 15:12:11 UTC 2018 x8664 x8664 x8664 GNU/Linux Install tools: in-house Others:
/sig api-machinery
this is because of this code: invoked from here: it is passing in the eviction delete options field as-is, meaning nil gets propagated and defaulted to mean delete immediately. when calling delete APIs normally, delete options defaults to a non-nil empty set of options:
Fixed in
patch looks legit. Thanks for the quick reaction and for the pointers!
I've a elasticsearch stateful set which have a preHook which never completes and the pod is never killed despite terminationGracePeriodSeconds being set. Is it related ? If so can this be considered to be cherry-picked on v1.13.x release branch ?

go_test( name = "go_default_test", <del> srcs = ["storage_test.go"], </del> <ins> srcs = [ "eviction_test.go", "storage_test.go", ], </ins> embed = [":go_default_library"], deps = [ "//pkg/apis/core:go_default_library", <ins> "//pkg/apis/policy:go_default_library", "//pkg/client/clientset_generated/internalclientset/fake:go_default_library", </ins> "//pkg/registry/registrytest:go_default_library", "//pkg/securitycontext:go_default_library", "//staging/src/k8s.io/api/core/v1:go_default_library",


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: When using the function from Python's Kuberneets API, we noticed that Pods are terminated bythe cluster without respecting their preStop hooks or terminationGracePeriodSeconds seconds. What you expected to happen: When using the eviction API, just as when using delete, Pods should be allowed to run their preStop hooks, as well as be given terminationGracePeriodSeconds before being terminated with SIGKILL. How to reproduce it (as minimally and precisely as possible): the following pod: the pod logs in another window to evict the pod. Here is a sample curl equivalent of what (1.13) is doing - happy path: Here is a sample curl equivalent of what Python's is doing (when not passed ) - broken path: The log output generated by the guinea pod on the happy path looks like this: The "Got signal 15" message shows up only after the preStop hook is finished, so ~10 seconds after the evict call is made What happens on the broken path is: We seem to be getting 2x SIGTERM, there are no messages indicating continued processing after the signal, and the signal itself is registered immediately after the eviction call is made, suggesting that preStop didn't run. Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: AWS OS (e.g. from /etc/os-release): CentOS Linux release 7.5.1804 Kernel (e.g. ): 3.10.0-862.14.4.el7.x8664 SMP Wed Sep 26 15:12:11 UTC 2018 x8664 x8664 x8664 GNU/Linux Install tools: in-house Others:
/sig api-machinery
this is because of this code: invoked from here: it is passing in the eviction delete options field as-is, meaning nil gets propagated and defaulted to mean delete immediately. when calling delete APIs normally, delete options defaults to a non-nil empty set of options:
Fixed in
patch looks legit. Thanks for the quick reaction and for the pointers!
I've a elasticsearch stateful set which have a preHook which never completes and the pod is never killed despite terminationGracePeriodSeconds being set. Is it related ? If so can this be considered to be cherry-picked on v1.13.x release branch ?

// At this point there was either no PDB or we succeeded in decrementing // Try the delete <del> _, _, err = r.store.Delete(ctx, eviction.Name, eviction.DeleteOptions) </del> <ins> deletionOptions := eviction.DeleteOptions if deletionOptions == nil { // default to non-nil to trigger graceful deletion deletionOptions = &metav1.DeleteOptions{} } _, _, err = r.store.Delete(ctx, eviction.Name, deletionOptions) </ins> if err != nil { return nil, err }


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: When using the function from Python's Kuberneets API, we noticed that Pods are terminated bythe cluster without respecting their preStop hooks or terminationGracePeriodSeconds seconds. What you expected to happen: When using the eviction API, just as when using delete, Pods should be allowed to run their preStop hooks, as well as be given terminationGracePeriodSeconds before being terminated with SIGKILL. How to reproduce it (as minimally and precisely as possible): the following pod: the pod logs in another window to evict the pod. Here is a sample curl equivalent of what (1.13) is doing - happy path: Here is a sample curl equivalent of what Python's is doing (when not passed ) - broken path: The log output generated by the guinea pod on the happy path looks like this: The "Got signal 15" message shows up only after the preStop hook is finished, so ~10 seconds after the evict call is made What happens on the broken path is: We seem to be getting 2x SIGTERM, there are no messages indicating continued processing after the signal, and the signal itself is registered immediately after the eviction call is made, suggesting that preStop didn't run. Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: AWS OS (e.g. from /etc/os-release): CentOS Linux release 7.5.1804 Kernel (e.g. ): 3.10.0-862.14.4.el7.x8664 SMP Wed Sep 26 15:12:11 UTC 2018 x8664 x8664 x8664 GNU/Linux Install tools: in-house Others:
/sig api-machinery
this is because of this code: invoked from here: it is passing in the eviction delete options field as-is, meaning nil gets propagated and defaulted to mean delete immediately. when calling delete APIs normally, delete options defaults to a non-nil empty set of options:
Fixed in
patch looks legit. Thanks for the quick reaction and for the pointers!
I've a elasticsearch stateful set which have a preHook which never completes and the pod is never killed despite terminationGracePeriodSeconds being set. Is it related ? If so can this be considered to be cherry-picked on v1.13.x release branch ?

<ins> /* Copyright 2019 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package storage import ( "testing" apierrors "k8s.io/apimachinery/pkg/api/errors" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/runtime" genericapirequest "k8s.io/apiserver/pkg/endpoints/request" api "k8s.io/kubernetes/pkg/apis/core" "k8s.io/kubernetes/pkg/apis/policy" "k8s.io/kubernetes/pkg/client/clientset_generated/internalclientset/fake" ) func TestEviction(t *testing.T) { testcases := []struct { name     string pdbs     []runtime.Object pod      *api.Pod eviction *policy.Eviction expectError   bool expectDeleted bool }{ { name:          "no pdbs, unscheduled pod, nil delete options, deletes immediately", pdbs:          nil, pod:           validNewPod(), eviction:      &policy.Eviction{ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}}, expectDeleted: true, }, { name:          "no pdbs, scheduled pod, nil delete options, deletes gracefully", pdbs:          nil, pod:           func() *api.Pod { pod := validNewPod(); pod.Spec.NodeName = "foo"; return pod }(), eviction:      &policy.Eviction{ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}}, expectDeleted: false, // not deleted immediately because of graceful deletion }, { name:          "no pdbs, scheduled pod, empty delete options, deletes gracefully", pdbs:          nil, pod:           func() *api.Pod { pod := validNewPod(); pod.Spec.NodeName = "foo"; return pod }(), eviction:      &policy.Eviction{ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}, DeleteOptions: &metav1.DeleteOptions{}}, expectDeleted: false, // not deleted immediately because of graceful deletion }, { name:          "no pdbs, scheduled pod, graceless delete options, deletes immediately", pdbs:          nil, pod:           func() *api.Pod { pod := validNewPod(); pod.Spec.NodeName = "foo"; return pod }(), eviction:      &policy.Eviction{ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}, DeleteOptions: metav1.NewDeleteOptions(0)}, expectDeleted: true, }, { name: "matching pdbs with no disruptions allowed", pdbs: []runtime.Object{&policy.PodDisruptionBudget{ ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}, Spec:       policy.PodDisruptionBudgetSpec{Selector: &metav1.LabelSelector{MatchLabels: map[string]string{"a": "true"}}}, Status:     policy.PodDisruptionBudgetStatus{PodDisruptionsAllowed: 0}, }}, pod: func() *api.Pod { pod := validNewPod() pod.Labels = map[string]string{"a": "true"} pod.Spec.NodeName = "foo" return pod }(), eviction:    &policy.Eviction{ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}, DeleteOptions: metav1.NewDeleteOptions(0)}, expectError: true, }, { name: "matching pdbs with disruptions allowed", pdbs: []runtime.Object{&policy.PodDisruptionBudget{ ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}, Spec:       policy.PodDisruptionBudgetSpec{Selector: &metav1.LabelSelector{MatchLabels: map[string]string{"a": "true"}}}, Status:     policy.PodDisruptionBudgetStatus{PodDisruptionsAllowed: 1}, }}, pod: func() *api.Pod { pod := validNewPod() pod.Labels = map[string]string{"a": "true"} pod.Spec.NodeName = "foo" return pod }(), eviction:      &policy.Eviction{ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}, DeleteOptions: metav1.NewDeleteOptions(0)}, expectDeleted: true, }, { name: "non-matching pdbs", pdbs: []runtime.Object{&policy.PodDisruptionBudget{ ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}, Spec:       policy.PodDisruptionBudgetSpec{Selector: &metav1.LabelSelector{MatchLabels: map[string]string{"b": "true"}}}, Status:     policy.PodDisruptionBudgetStatus{PodDisruptionsAllowed: 0}, }}, pod: func() *api.Pod { pod := validNewPod() pod.Labels = map[string]string{"a": "true"} pod.Spec.NodeName = "foo" return pod }(), eviction:      &policy.Eviction{ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}, DeleteOptions: metav1.NewDeleteOptions(0)}, expectDeleted: true, }, } for _, tc := range testcases { t.Run(tc.name, func(t *testing.T) { testContext := genericapirequest.WithNamespace(genericapirequest.NewContext(), metav1.NamespaceDefault) storage, _, _, server := newStorage(t) defer server.Terminate(t) defer storage.Store.DestroyFunc() if tc.pod != nil { if _, err := storage.Create(testContext, tc.pod, nil, &metav1.CreateOptions{}); err != nil { t.Error(err) } } client := fake.NewSimpleClientset(tc.pdbs...) evictionRest := newEvictionStorage(storage.Store, client.Policy()) _, err := evictionRest.Create(testContext, tc.eviction, nil, &metav1.CreateOptions{}) if (err != nil) != tc.expectError { t.Errorf("expected error=%v, got %v", tc.expectError, err) return } if tc.expectError { return } if tc.pod != nil { existingPod, err := storage.Get(testContext, tc.pod.Name, &metav1.GetOptions{}) if tc.expectDeleted { if !apierrors.IsNotFound(err) { t.Errorf("expected to be deleted, lookup returned %#v", existingPod) } return } else if apierrors.IsNotFound(err) { t.Errorf("expected graceful deletion, got %v", err) return } if err != nil { t.Errorf("%#v", err) return } if existingPod.(*api.Pod).DeletionTimestamp == nil { t.Errorf("expected gracefully deleted pod with deletionTimestamp set, got %#v", existingPod) } } }) } } </ins>


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: When using the function from Python's Kuberneets API, we noticed that Pods are terminated bythe cluster without respecting their preStop hooks or terminationGracePeriodSeconds seconds. What you expected to happen: When using the eviction API, just as when using delete, Pods should be allowed to run their preStop hooks, as well as be given terminationGracePeriodSeconds before being terminated with SIGKILL. How to reproduce it (as minimally and precisely as possible): the following pod: the pod logs in another window to evict the pod. Here is a sample curl equivalent of what (1.13) is doing - happy path: Here is a sample curl equivalent of what Python's is doing (when not passed ) - broken path: The log output generated by the guinea pod on the happy path looks like this: The "Got signal 15" message shows up only after the preStop hook is finished, so ~10 seconds after the evict call is made What happens on the broken path is: We seem to be getting 2x SIGTERM, there are no messages indicating continued processing after the signal, and the signal itself is registered immediately after the eviction call is made, suggesting that preStop didn't run. Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: AWS OS (e.g. from /etc/os-release): CentOS Linux release 7.5.1804 Kernel (e.g. ): 3.10.0-862.14.4.el7.x8664 SMP Wed Sep 26 15:12:11 UTC 2018 x8664 x8664 x8664 GNU/Linux Install tools: in-house Others:
/sig api-machinery
this is because of this code: invoked from here: it is passing in the eviction delete options field as-is, meaning nil gets propagated and defaulted to mean delete immediately. when calling delete APIs normally, delete options defaults to a non-nil empty set of options:
Fixed in
patch looks legit. Thanks for the quick reaction and for the pointers!
I've a elasticsearch stateful set which have a preHook which never completes and the pod is never killed despite terminationGracePeriodSeconds being set. Is it related ? If so can this be considered to be cherry-picked on v1.13.x release branch ?

} createNode2NormalPodEviction := func(client clientset.Interface) func() error { return func() error { <ins> zero := int64(0) </ins> return client.Policy().Evictions("ns").Evict(&policy.Eviction{ TypeMeta: metav1.TypeMeta{ APIVersion: "policy/v1beta1",