Question:
Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: The deployment controller for a deployment with a single replica failed to create a new ReplicaSet when the old ReplicaSet included an Evicted pod. Typically, the eviction was caused by "node low on resource: nodefs". When we manually deleted the evicted pod, the new RS and accompanying pod were created immediately. We have seen this happen at least a dozen times. Here are the audit logs for a recent case: ! What you expected to happen: The deployment controller ignores the evicted pod and creates the updated replicaset/pod. How to reproduce it (as minimally and precisely as possible): To reproduce, you would need a way to trigger a kubelet-driven eviction on a node that is currently running a single-replica Deployment with . Wait for the pod to be evicted and the new pod to be created, confirm that the evicted pod still exists, and then make a trivial change to the deployment spec. The running pod will be deleted, and no new replicaset/pod will be created. This issue does not seem to occur for manual evictions (via ); pods I evict that way disappear from the API server immediately. Anything else we need to know?: Possibly related to and/or Environment: Kubernetes version (use ): v1.8.5-gke.0 Cloud provider or hardware configuration: GKE OS (e.g. from /etc/os-release): Container-Optimized OS
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Thanks for the detailed bug report! From Here's what happened: Deployment was updated by the user, so it scaled all old ReplicaSets to 0 old ReplicaSet was scaled to 0 (ReplicaSet's == 0), but still owned the evicted pod, which wouldn't be killed automatically because of the above comment (ReplicaSet's 0) Deployment found that there were still old Pods, and kept waiting for them to be gone ~(wait for old ReplicaSet's to be 0)~ (Edit: Deployment also checks those old Pods by listing them to make sure there are none, not just by checking ReplicaSet's status) before starting the new rollout. We can either modify Deployment controller (to ignore evicted old pods) or ReplicaSet controller (to kill evicted pods ~when it gets scaled down~), if this is not the intended behavior. WDYT?
I am actually debugging likely the same production issue with Recreate strategy in two clusters right now. (We run on patched 1.9.1.) I have traced it down to broken RS as well. in 1 cluster RS leaves around pod failed matching the selector in the other one those are evicted pods. all replicas are scaled down to 0 and report 0 in status but pods with valid controllerRefs to them are still present. Same is 0 for me but pod is still there I think the correct behaviour is for replicaSet scaled to 0 to delete all of its pod, not to leave them laying around
xref - this should fix it

oldRSs []*extensions.ReplicaSet podMap map[types.UID]*v1.PodList <del> expected bool </del> <ins> hasOldPodsRunning bool </ins> }{ { <del> name:     "no old RSs", expected: false, </del> <ins> name:              "no old RSs", hasOldPodsRunning: false, </ins> }, { <del> name:     "old RSs with running pods", oldRSs:   []*extensions.ReplicaSet{rsWithUID("some-uid"), rsWithUID("other-uid")}, podMap:   podMapWithUIDs([]string{"some-uid", "other-uid"}), expected: true, </del> <ins> name:              "old RSs with running pods", oldRSs:            []*extensions.ReplicaSet{rsWithUID("some-uid"), rsWithUID("other-uid")}, podMap:            podMapWithUIDs([]string{"some-uid", "other-uid"}), hasOldPodsRunning: true, </ins> }, { <del> name:     "old RSs without pods but with non-zero status replicas", oldRSs:   []*extensions.ReplicaSet{newRSWithStatus("rs-blabla", 0, 1, nil)}, expected: true, </del> <ins> name:              "old RSs without pods but with non-zero status replicas", oldRSs:            []*extensions.ReplicaSet{newRSWithStatus("rs-1", 0, 1, nil)}, hasOldPodsRunning: true, </ins> }, { <del> name:     "old RSs without pods or non-zero status replicas", oldRSs:   []*extensions.ReplicaSet{newRSWithStatus("rs-blabla", 0, 0, nil)}, expected: false, </del> <ins> name:              "old RSs without pods or non-zero status replicas", oldRSs:            []*extensions.ReplicaSet{newRSWithStatus("rs-1", 0, 0, nil)}, hasOldPodsRunning: false, }, { name:   "old RSs with zero status replicas but pods in terminal state are present", oldRSs: []*extensions.ReplicaSet{newRSWithStatus("rs-1", 0, 0, nil)}, podMap: map[types.UID]*v1.PodList{ "uid-1": { Items: []v1.Pod{ { Status: v1.PodStatus{ Phase: v1.PodFailed, }, }, { Status: v1.PodStatus{ Phase: v1.PodSucceeded, }, }, }, }, }, hasOldPodsRunning: false, }, { name:   "old RSs with zero status replicas but pod in unknown phase present", oldRSs: []*extensions.ReplicaSet{newRSWithStatus("rs-1", 0, 0, nil)}, podMap: map[types.UID]*v1.PodList{ "uid-1": { Items: []v1.Pod{ { Status: v1.PodStatus{ Phase: v1.PodUnknown, }, }, }, }, }, hasOldPodsRunning: true, }, { name:   "old RSs with zero status replicas with pending pod present", oldRSs: []*extensions.ReplicaSet{newRSWithStatus("rs-1", 0, 0, nil)}, podMap: map[types.UID]*v1.PodList{ "uid-1": { Items: []v1.Pod{ { Status: v1.PodStatus{ Phase: v1.PodPending, }, }, }, }, }, hasOldPodsRunning: true, }, { name:   "old RSs with zero status replicas with running pod present", oldRSs: []*extensions.ReplicaSet{newRSWithStatus("rs-1", 0, 0, nil)}, podMap: map[types.UID]*v1.PodList{ "uid-1": { Items: []v1.Pod{ { Status: v1.PodStatus{ Phase: v1.PodRunning, }, }, }, }, }, hasOldPodsRunning: true, }, { name:   "old RSs with zero status replicas but pods in terminal state and pending are present", oldRSs: []*extensions.ReplicaSet{newRSWithStatus("rs-1", 0, 0, nil)}, podMap: map[types.UID]*v1.PodList{ "uid-1": { Items: []v1.Pod{ { Status: v1.PodStatus{ Phase: v1.PodFailed, }, }, { Status: v1.PodStatus{ Phase: v1.PodSucceeded, }, }, }, }, "uid-2": { Items: []v1.Pod{}, }, "uid-3": { Items: []v1.Pod{ { Status: v1.PodStatus{ Phase: v1.PodPending, }, }, }, }, }, hasOldPodsRunning: true, </ins> }, } for _, test := range tests { t.Run(test.name, func(t *testing.T) { <del> if expected, got := test.expected, oldPodsRunning(test.newRS, test.oldRSs, test.podMap); expected != got { </del> <ins> if expected, got := test.hasOldPodsRunning, oldPodsRunning(test.newRS, test.oldRSs, test.podMap); expected != got { </ins> t.Errorf("%s: expected %t, got %t", test.name, expected, got) } })


Question:
There are no sig labels on this issue. Please add a sig label. <detailsA sig label can be by either: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. See the . The in method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
probably caused by
Could be a clash between these two?
cc
I still see the issue with local-cluster on the latest master: F0225 17:06:49. ] error: v1alpha1.KubeProxyConfiguration: FeatureGates: ReadMapCB: expect { or n, but found ", error found in byte of ......
looks like that's another issue, can you please file a new bug? with more logs? this one was about none of the hyperkube based processes starting up.
will do as it is also broke container kube-proxy as well. Same type of error.

"//cmd/kube-proxy/app:go_default_library", "//pkg/client/metrics/prometheus:go_default_library", "//pkg/version/prometheus:go_default_library", <ins> "//vendor/github.com/spf13/pflag:go_default_library", </ins> "//vendor/k8s.io/apiserver/pkg/util/flag:go_default_library", "//vendor/k8s.io/apiserver/pkg/util/logs:go_default_library", ],


Question:
There are no sig labels on this issue. Please add a sig label. <detailsA sig label can be by either: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. See the . The in method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
probably caused by
Could be a clash between these two?
cc
I still see the issue with local-cluster on the latest master: F0225 17:06:49. ] error: v1alpha1.KubeProxyConfiguration: FeatureGates: ReadMapCB: expect { or n, but found ", error found in byte of ......
looks like that's another issue, can you please file a new bug? with more logs? this one was about none of the hyperkube based processes starting up.
will do as it is also broke container kube-proxy as well. Same type of error.

glog.Fatalf("unable to create flag defaults: %v", err) } <del> opts.AddFlags(pflag.CommandLine) </del> <ins> opts.AddFlags(cmd.Flags()) </ins> cmd.MarkFlagFilename("config", "yaml", "yml", "json")


Question:
There are no sig labels on this issue. Please add a sig label. <detailsA sig label can be by either: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. See the . The in method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
probably caused by
Could be a clash between these two?
cc
I still see the issue with local-cluster on the latest master: F0225 17:06:49. ] error: v1alpha1.KubeProxyConfiguration: FeatureGates: ReadMapCB: expect { or n, but found ", error found in byte of ......
looks like that's another issue, can you please file a new bug? with more logs? this one was about none of the hyperkube based processes starting up.
will do as it is also broke container kube-proxy as well. Same type of error.

package main import ( <ins> goflag "flag" </ins> "fmt" "math/rand" "os" "time" <ins> "github.com/spf13/pflag" </ins> utilflag "k8s.io/apiserver/pkg/util/flag" "k8s.io/apiserver/pkg/util/logs" "k8s.io/kubernetes/cmd/kube-proxy/app"


Question:
There are no sig labels on this issue. Please add a sig label. <detailsA sig label can be by either: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. See the . The in method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
probably caused by
Could be a clash between these two?
cc
I still see the issue with local-cluster on the latest master: F0225 17:06:49. ] error: v1alpha1.KubeProxyConfiguration: FeatureGates: ReadMapCB: expect { or n, but found ", error found in byte of ......
looks like that's another issue, can you please file a new bug? with more logs? this one was about none of the hyperkube based processes starting up.
will do as it is also broke container kube-proxy as well. Same type of error.

command := app.NewProxyCommand() <del> utilflag.InitFlags() </del> <ins> // TODO: once we switch everything over to Cobra commands, we can go back to calling // utilflag.InitFlags() (by removing its pflag.Parse() call). For now, we have to set the // normalize func and add the go flag set by hand. pflag.CommandLine.SetNormalizeFunc(utilflag.WordSepNormalizeFunc) pflag.CommandLine.AddGoFlagSet(goflag.CommandLine) // utilflag.InitFlags() </ins> logs.InitLogs() defer logs.FlushLogs()


Question:
Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: The field passed to is never set. In where is a object that is initialized in the method in Notice the field is never set. What you expected to happen: The field should be set to How to reproduce it (as minimally and precisely as possible): Create a CSI volume and set to true. Check the value of in the CSI call. It should be true, but is false. CC Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
I think is only used for mounting pod volume path to container, here(for ), we can set to , just like other volume plugins. I can help to fix this.
What about the case when a user wants to mount a volume for one of containers in a POD as read only, but the actual volume has ReadOnly False? I would like to suggest an alternative solution which passes the value of ReadOnly attribute, taken from POD's spec.
Thank you and for jumping on this! I feel like the other volume plugins have the wrong behavior here. The purpose of the call is to mount the volume. Therefore, the should receive the pod volume mount readonly field. That said, as mentioned , I think the fact that " is used for mounting pod volume path to container" might be hiding a bug: that is never set correctly. I haven't had a chance to validate this but my reading of the code suggests that the is never set. But because the container mount uses , even though the plugin did not get the correct readonly value, the mount ends up getting mounted with the correctly readonly value. Thoughts?
[MILESTONENOTIFIER] Milestone Issue: Up-to-date for process Note: This issue is marked as , and must be updated every 1 day during code freeze. Example update: <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
We discussed the proposed fixes in the Kubernetes CSI implementation meeting today (see ). We decided it would be better for CSI to be consistent with the other plugins (and still broken), and have CSI fixed, along with all the other plugins, whenever we fix the issue (likely post-1.10). Therefore, I will accept fix and we will close Then we can remove the 1.10 tag from this issue and track it independent of the 1.10 release.
Spoke with offline. Details here: Based on that conversation, I don't think there is anything else we need to do here.

if err != nil { return nil, err } <ins> readOnly, err := getReadOnlyFromSpec(spec) if err != nil { return nil, err } </ins> // before it is used in any paths such as socket etc addr := fmt.Sprintf(csiAddrTemplate, pvSource.Driver)


Question:
Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: The field passed to is never set. In where is a object that is initialized in the method in Notice the field is never set. What you expected to happen: The field should be set to How to reproduce it (as minimally and precisely as possible): Create a CSI volume and set to true. Check the value of in the CSI call. It should be true, but is false. CC Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
I think is only used for mounting pod volume path to container, here(for ), we can set to , just like other volume plugins. I can help to fix this.
What about the case when a user wants to mount a volume for one of containers in a POD as read only, but the actual volume has ReadOnly False? I would like to suggest an alternative solution which passes the value of ReadOnly attribute, taken from POD's spec.
Thank you and for jumping on this! I feel like the other volume plugins have the wrong behavior here. The purpose of the call is to mount the volume. Therefore, the should receive the pod volume mount readonly field. That said, as mentioned , I think the fact that " is used for mounting pod volume path to container" might be hiding a bug: that is never set correctly. I haven't had a chance to validate this but my reading of the code suggests that the is never set. But because the container mount uses , even though the plugin did not get the correct readonly value, the mount ends up getting mounted with the correctly readonly value. Thoughts?
[MILESTONENOTIFIER] Milestone Issue: Up-to-date for process Note: This issue is marked as , and must be updated every 1 day during code freeze. Example update: <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
We discussed the proposed fixes in the Kubernetes CSI implementation meeting today (see ). We decided it would be better for CSI to be consistent with the other plugins (and still broken), and have CSI fixed, along with all the other plugins, whenever we fix the issue (likely post-1.10). Therefore, I will accept fix and we will close Then we can remove the 1.10 tag from this issue and track it independent of the 1.10 release.
Spoke with offline. Details here: Based on that conversation, I don't think there is anything else we need to do here.

volumeID:     pvSource.VolumeHandle, specVolumeID: spec.Name(), csiClient:    client, <ins> readOnly:     readOnly, </ins> } return mounter, nil }


Question:
Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: The field passed to is never set. In where is a object that is initialized in the method in Notice the field is never set. What you expected to happen: The field should be set to How to reproduce it (as minimally and precisely as possible): Create a CSI volume and set to true. Check the value of in the CSI call. It should be true, but is false. CC Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
I think is only used for mounting pod volume path to container, here(for ), we can set to , just like other volume plugins. I can help to fix this.
What about the case when a user wants to mount a volume for one of containers in a POD as read only, but the actual volume has ReadOnly False? I would like to suggest an alternative solution which passes the value of ReadOnly attribute, taken from POD's spec.
Thank you and for jumping on this! I feel like the other volume plugins have the wrong behavior here. The purpose of the call is to mount the volume. Therefore, the should receive the pod volume mount readonly field. That said, as mentioned , I think the fact that " is used for mounting pod volume path to container" might be hiding a bug: that is never set correctly. I haven't had a chance to validate this but my reading of the code suggests that the is never set. But because the container mount uses , even though the plugin did not get the correct readonly value, the mount ends up getting mounted with the correctly readonly value. Thoughts?
[MILESTONENOTIFIER] Milestone Issue: Up-to-date for process Note: This issue is marked as , and must be updated every 1 day during code freeze. Example update: <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
We discussed the proposed fixes in the Kubernetes CSI implementation meeting today (see ). We decided it would be better for CSI to be consistent with the other plugins (and still broken), and have CSI fixed, along with all the other plugins, whenever we fix the issue (likely post-1.10). Therefore, I will accept fix and we will close Then we can remove the 1.10 tag from this issue and track it independent of the 1.10 release.
Spoke with offline. Details here: Based on that conversation, I don't think there is anything else we need to do here.

return nil, fmt.Errorf("CSIPersistentVolumeSource not defined in spec") } <ins> func getReadOnlyFromSpec(spec *volume.Spec) (bool, error) { if spec.PersistentVolume != nil && spec.PersistentVolume.Spec.CSI != nil { return spec.ReadOnly, nil } return false, fmt.Errorf("CSIPersistentVolumeSource not defined in spec") } </ins> // log prepends log string with `kubernetes.io/csi` func log(msg string, parts ...interface{}) string { return fmt.Sprintf(fmt.Sprintf("%s: %s", csiPluginName, msg), parts...)


Question:
/kind bug /sig cli /area kubectl Hello. This is my first interaction with the Kubernetes community, so please forgive me if i have left out any crucial details, or if, perhaps, this is PEBKAC after all! I am keen to have a working feature, as described in and I am running K8s v1.9.3 and realise that this feature is in alpha, but i thought i would ask anyway. We're not on v1.9.4 yet, but the changelog doesn't mention the feature. What you expected to happen: I have some services deployed to Kubernetes with YAML files, that has been working reliably. We always use to deploy things. When a service is already deployed, and i make a minor change in the config file, for example the service port, i expect to see that in the diff. Instead, gives no output and exits with a . How to reproduce it (as minimally and precisely as possible): Consider the file : I make sure that it is correctly running on the cluster: Great, looking good. Now, i change the value to use Docker image tag and save. Now, i run: The stderr/stdout output is empty, and the exit code is . Anything else we need to know?: Environment: Kubernetes version: Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.3", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-02-09T21:50:44Z", GoVersion:"go1.9.4", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.3+coreos.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-02-10T01:42:55Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}
I have this same issue - if I change, say, to in my , running returns empty. If I try tweaking the version I'm diffing (which I don't understand) - e.g. - then I see a bunch of differences, but not actually the change I was expecting in the number of replicas.
I'll take this. Sorry for the inconvenience.

set +o errexit } <ins> # Runs tests for kubectl alpha diff run_kubectl_diff_tests() { set -o nounset set -o errexit create_and_use_new_namespace kube::log::status "Testing kubectl alpha diff" kubectl apply -f hack/testdata/pod.yaml # Ensure that selfLink has been added, and shown in the diff output_message=$(kubectl alpha diff -f hack/testdata/pod.yaml) kube::test::if_has_string "${output_message}" 'selfLink' output_message=$(kubectl alpha diff LOCAL LIVE -f hack/testdata/pod.yaml) kube::test::if_has_string "${output_message}" 'selfLink' output_message=$(kubectl alpha diff LOCAL MERGED -f hack/testdata/pod.yaml) kube::test::if_has_string "${output_message}" 'selfLink' output_message=$(kubectl alpha diff MERGED MERGED -f hack/testdata/pod.yaml) kube::test::if_empty_string "${output_message}" output_message=$(kubectl alpha diff LIVE LIVE -f hack/testdata/pod.yaml) kube::test::if_empty_string "${output_message}" output_message=$(kubectl alpha diff LAST LAST -f hack/testdata/pod.yaml) kube::test::if_empty_string "${output_message}" output_message=$(kubectl alpha diff LOCAL LOCAL -f hack/testdata/pod.yaml) kube::test::if_empty_string "${output_message}" kubectl delete -f  hack/testdata/pod.yaml set +o nounset set +o errexit } </ins> # Runs tests for --save-config tests. run_save_config_tests() { set -o nounset


Question:
/kind bug /sig cli /area kubectl Hello. This is my first interaction with the Kubernetes community, so please forgive me if i have left out any crucial details, or if, perhaps, this is PEBKAC after all! I am keen to have a working feature, as described in and I am running K8s v1.9.3 and realise that this feature is in alpha, but i thought i would ask anyway. We're not on v1.9.4 yet, but the changelog doesn't mention the feature. What you expected to happen: I have some services deployed to Kubernetes with YAML files, that has been working reliably. We always use to deploy things. When a service is already deployed, and i make a minor change in the config file, for example the service port, i expect to see that in the diff. Instead, gives no output and exits with a . How to reproduce it (as minimally and precisely as possible): Consider the file : I make sure that it is correctly running on the cluster: Great, looking good. Now, i change the value to use Docker image tag and save. Now, i run: The stderr/stdout output is empty, and the exit code is . Anything else we need to know?: Environment: Kubernetes version: Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.3", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-02-09T21:50:44Z", GoVersion:"go1.9.4", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.3+coreos.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-02-10T01:42:55Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}
I have this same issue - if I change, say, to in my , running returns empty. If I try tweaking the version I'm diffing (which I don't understand) - e.g. - then I see a bunch of differences, but not actually the change I was expecting in the number of replicas.
I'll take this. Sorry for the inconvenience.

record_command run_kubectl_apply_deployments_tests fi <ins> ################ # Kubectl diff # ################ record_command run_kubectl_diff_tests </ins> ############### # Kubectl get # ###############


Question:
/kind bug /sig cli /area kubectl Hello. This is my first interaction with the Kubernetes community, so please forgive me if i have left out any crucial details, or if, perhaps, this is PEBKAC after all! I am keen to have a working feature, as described in and I am running K8s v1.9.3 and realise that this feature is in alpha, but i thought i would ask anyway. We're not on v1.9.4 yet, but the changelog doesn't mention the feature. What you expected to happen: I have some services deployed to Kubernetes with YAML files, that has been working reliably. We always use to deploy things. When a service is already deployed, and i make a minor change in the config file, for example the service port, i expect to see that in the diff. Instead, gives no output and exits with a . How to reproduce it (as minimally and precisely as possible): Consider the file : I make sure that it is correctly running on the cluster: Great, looking good. Now, i change the value to use Docker image tag and save. Now, i run: The stderr/stdout output is empty, and the exit code is . Anything else we need to know?: Environment: Kubernetes version: Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.3", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-02-09T21:50:44Z", GoVersion:"go1.9.4", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.3+coreos.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-02-10T01:42:55Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}
I have this same issue - if I change, say, to in my , running returns empty. If I try tweaking the version I'm diffing (which I don't understand) - e.g. - then I see a bunch of differences, but not actually the change I was expecting in the number of replicas.
I'll take this. Sorry for the inconvenience.

"github.com/ghodss/yaml" "github.com/spf13/cobra" <del> "k8s.io/apimachinery/pkg/api/errors" </del> "k8s.io/apimachinery/pkg/api/meta" <ins> metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" </ins> "k8s.io/apimachinery/pkg/runtime" <ins> "k8s.io/client-go/dynamic" </ins> api "k8s.io/kubernetes/pkg/apis/core" "k8s.io/kubernetes/pkg/kubectl/apply/parse" "k8s.io/kubernetes/pkg/kubectl/apply/strategy"


Question:
/kind bug /sig cli /area kubectl Hello. This is my first interaction with the Kubernetes community, so please forgive me if i have left out any crucial details, or if, perhaps, this is PEBKAC after all! I am keen to have a working feature, as described in and I am running K8s v1.9.3 and realise that this feature is in alpha, but i thought i would ask anyway. We're not on v1.9.4 yet, but the changelog doesn't mention the feature. What you expected to happen: I have some services deployed to Kubernetes with YAML files, that has been working reliably. We always use to deploy things. When a service is already deployed, and i make a minor change in the config file, for example the service port, i expect to see that in the diff. Instead, gives no output and exits with a . How to reproduce it (as minimally and precisely as possible): Consider the file : I make sure that it is correctly running on the cluster: Great, looking good. Now, i change the value to use Docker image tag and save. Now, i run: The stderr/stdout output is empty, and the exit code is . Anything else we need to know?: Environment: Kubernetes version: Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.3", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-02-09T21:50:44Z", GoVersion:"go1.9.4", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.3+coreos.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-02-10T01:42:55Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}
I have this same issue - if I change, say, to in my , running returns empty. If I try tweaking the version I'm diffing (which I don't understand) - e.g. - then I see a bunch of differences, but not actually the change I was expecting in the number of replicas.
I'll take this. Sorry for the inconvenience.

// InfoObject is an implementation of the Object interface. It gets all // the information from the Info object. type InfoObject struct { <ins> Remote  runtime.Unstructured </ins> Info    *resource.Info Encoder runtime.Encoder Parser  *parse.Factory


Question:
/kind bug /sig cli /area kubectl Hello. This is my first interaction with the Kubernetes community, so please forgive me if i have left out any crucial details, or if, perhaps, this is PEBKAC after all! I am keen to have a working feature, as described in and I am running K8s v1.9.3 and realise that this feature is in alpha, but i thought i would ask anyway. We're not on v1.9.4 yet, but the changelog doesn't mention the feature. What you expected to happen: I have some services deployed to Kubernetes with YAML files, that has been working reliably. We always use to deploy things. When a service is already deployed, and i make a minor change in the config file, for example the service port, i expect to see that in the diff. Instead, gives no output and exits with a . How to reproduce it (as minimally and precisely as possible): Consider the file : I make sure that it is correctly running on the cluster: Great, looking good. Now, i change the value to use Docker image tag and save. Now, i run: The stderr/stdout output is empty, and the exit code is . Anything else we need to know?: Environment: Kubernetes version: Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.3", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-02-09T21:50:44Z", GoVersion:"go1.9.4", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.3+coreos.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-02-10T01:42:55Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}
I have this same issue - if I change, say, to in my , running returns empty. If I try tweaking the version I'm diffing (which I don't understand) - e.g. - then I see a bunch of differences, but not actually the change I was expecting in the number of replicas.
I'll take this. Sorry for the inconvenience.

} func (obj InfoObject) Live() (map[string]interface{}, error) { <del> if obj.Info.Object == nil { </del> <ins> if obj.Remote == nil { </ins> return nil, nil // Object doesn't exist on cluster. } <del> data, err := runtime.Encode(obj.Encoder, obj.Info.Object) if err != nil { return nil, err } return obj.toMap(data) </del> <ins> return obj.Remote.UnstructuredContent(), nil </ins> } func (obj InfoObject) Merged() (map[string]interface{}, error) {


Question:
/kind bug /sig cli /area kubectl Hello. This is my first interaction with the Kubernetes community, so please forgive me if i have left out any crucial details, or if, perhaps, this is PEBKAC after all! I am keen to have a working feature, as described in and I am running K8s v1.9.3 and realise that this feature is in alpha, but i thought i would ask anyway. We're not on v1.9.4 yet, but the changelog doesn't mention the feature. What you expected to happen: I have some services deployed to Kubernetes with YAML files, that has been working reliably. We always use to deploy things. When a service is already deployed, and i make a minor change in the config file, for example the service port, i expect to see that in the diff. Instead, gives no output and exits with a . How to reproduce it (as minimally and precisely as possible): Consider the file : I make sure that it is correctly running on the cluster: Great, looking good. Now, i change the value to use Docker image tag and save. Now, i run: The stderr/stdout output is empty, and the exit code is . Anything else we need to know?: Environment: Kubernetes version: Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.3", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-02-09T21:50:44Z", GoVersion:"go1.9.4", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.3+coreos.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-02-10T01:42:55Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}
I have this same issue - if I change, say, to in my , running returns empty. If I try tweaking the version I'm diffing (which I don't understand) - e.g. - then I see a bunch of differences, but not actually the change I was expecting in the number of replicas.
I'll take this. Sorry for the inconvenience.

} func (obj InfoObject) Last() (map[string]interface{}, error) { <del> if obj.Info.Object == nil { </del> <ins> if obj.Remote == nil { </ins> return nil, nil // No object is live, return empty } <del> accessor, err := meta.Accessor(obj.Info.Object) </del> <ins> accessor, err := meta.Accessor(obj.Remote) </ins> if err != nil { return nil, err }


Question:
/kind bug /sig cli /area kubectl Hello. This is my first interaction with the Kubernetes community, so please forgive me if i have left out any crucial details, or if, perhaps, this is PEBKAC after all! I am keen to have a working feature, as described in and I am running K8s v1.9.3 and realise that this feature is in alpha, but i thought i would ask anyway. We're not on v1.9.4 yet, but the changelog doesn't mention the feature. What you expected to happen: I have some services deployed to Kubernetes with YAML files, that has been working reliably. We always use to deploy things. When a service is already deployed, and i make a minor change in the config file, for example the service port, i expect to see that in the diff. Instead, gives no output and exits with a . How to reproduce it (as minimally and precisely as possible): Consider the file : I make sure that it is correctly running on the cluster: Great, looking good. Now, i change the value to use Docker image tag and save. Now, i run: The stderr/stdout output is empty, and the exit code is . Anything else we need to know?: Environment: Kubernetes version: Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.3", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-02-09T21:50:44Z", GoVersion:"go1.9.4", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.3+coreos.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-02-10T01:42:55Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}
I have this same issue - if I change, say, to in my , running returns empty. If I try tweaking the version I'm diffing (which I don't understand) - e.g. - then I see a bunch of differences, but not actually the change I was expecting in the number of replicas.
I'll take this. Sorry for the inconvenience.

d.To.Dir.Delete()   // Ignore error } <ins> type Downloader struct { mapper  meta.RESTMapper dclient dynamic.Interface ns      string } func NewDownloader(f cmdutil.Factory) (*Downloader, error) { var err error var d Downloader d.mapper, err = f.RESTMapper() if err != nil { return nil, err } d.dclient, err = f.DynamicClient() if err != nil { return nil, err } d.ns, _, _ = f.DefaultNamespace() return &d, nil } func (d *Downloader) Download(info *resource.Info) (*unstructured.Unstructured, error) { gvk := info.Object.GetObjectKind().GroupVersionKind() mapping, err := d.mapper.RESTMapping(gvk.GroupKind(), gvk.Version) if err != nil { return nil, err } var resource dynamic.ResourceInterface switch mapping.Scope.Name() { case meta.RESTScopeNameNamespace: if info.Namespace == "" { info.Namespace = d.ns } resource = d.dclient.Resource(mapping.Resource).Namespace(info.Namespace) case meta.RESTScopeNameRoot: resource = d.dclient.Resource(mapping.Resource) } return resource.Get(info.Name, metav1.GetOptions{}) } </ins> // RunDiff uses the factory to parse file arguments, find the version to // diff, and find each Info object for each files, and runs against the // differ.


Question:
/kind bug /sig cli /area kubectl Hello. This is my first interaction with the Kubernetes community, so please forgive me if i have left out any crucial details, or if, perhaps, this is PEBKAC after all! I am keen to have a working feature, as described in and I am running K8s v1.9.3 and realise that this feature is in alpha, but i thought i would ask anyway. We're not on v1.9.4 yet, but the changelog doesn't mention the feature. What you expected to happen: I have some services deployed to Kubernetes with YAML files, that has been working reliably. We always use to deploy things. When a service is already deployed, and i make a minor change in the config file, for example the service port, i expect to see that in the diff. Instead, gives no output and exits with a . How to reproduce it (as minimally and precisely as possible): Consider the file : I make sure that it is correctly running on the cluster: Great, looking good. Now, i change the value to use Docker image tag and save. Now, i run: The stderr/stdout output is empty, and the exit code is . Anything else we need to know?: Environment: Kubernetes version: Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.3", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-02-09T21:50:44Z", GoVersion:"go1.9.4", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.3+coreos.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-02-10T01:42:55Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}
I have this same issue - if I change, say, to in my , running returns empty. If I try tweaking the version I'm diffing (which I don't understand) - e.g. - then I see a bunch of differences, but not actually the change I was expecting in the number of replicas.
I'll take this. Sorry for the inconvenience.

Unstructured(). NamespaceParam(cmdNamespace).DefaultNamespace(). FilenameParam(enforceNamespace, &options.FilenameOptions). <ins> Local(). </ins> Flatten(). Do() if err := r.Err(); err != nil { return err } <ins> dl, err := NewDownloader(f) if err != nil { return err } </ins> err = r.Visit(func(info *resource.Info, err error) error { if err != nil { return err } <del> if err := info.Get(); err != nil { if !errors.IsNotFound(err) { return cmdutil.AddSourceToErr(fmt.Sprintf("retrieving current configuration of:n%snfrom server for:", info.String()), info.Source, err) } info.Object = nil } </del> <ins> remote, _ := dl.Download(info) </ins> obj := InfoObject{ <ins> Remote:  remote, </ins> Info:    info, Parser:  parser, Encoder: cmdutil.InternalVersionJSONEncoder(),


Question:
Anything else we need to know?: I notice there are TODOs in apimachinerypkgapimeta,return bool instead of error func CommonAccessor, ListAccessor, Accessor. If is it necessary to do,I push a PR for it. Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
cc
/sig api-machinery
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close

// CommonAccessor returns a Common interface for the provided object or an error if the object does // not provide List. <del> // TODO: return bool instead of error </del> func CommonAccessor(obj interface{}) (metav1.Common, error) { switch t := obj.(type) { case List:


Question:
Anything else we need to know?: I notice there are TODOs in apimachinerypkgapimeta,return bool instead of error func CommonAccessor, ListAccessor, Accessor. If is it necessary to do,I push a PR for it. Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
cc
/sig api-machinery
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close

// not provide List. // IMPORTANT: Objects are NOT a superset of lists. Do not use this check to determine whether an // object *is* a List. <del> // TODO: return bool instead of error </del> func ListAccessor(obj interface{}) (List, error) { switch t := obj.(type) { case List:


Question:
Anything else we need to know?: I notice there are TODOs in apimachinerypkgapimeta,return bool instead of error func CommonAccessor, ListAccessor, Accessor. If is it necessary to do,I push a PR for it. Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
cc
/sig api-machinery
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close

// obj must be a pointer to an API type. An error is returned if the minimum // required fields are missing. Fields that are not required return the default // value and are a no-op if set. <del> // TODO: return bool instead of error </del> func Accessor(obj interface{}) (metav1.Object, error) { switch t := obj.(type) { case metav1.Object:


Question:
IMO it's surprising that does not force delete a resource. only takes effect when . The description of reads this: We should make it more clear what does and even print warnings when is specified without .
Changing the meaning of force to actually break cluster consistency be default could lead to data loss, which is why we haven’t changed it so far. agree we should print more info and clarfiy

cmd.Flags().BoolVar(&options.DeleteAll, "all", options.DeleteAll, "Delete all resources, including uninitialized ones, in the namespace of the specified resource types.") cmd.Flags().BoolVar(&options.IgnoreNotFound, "ignore-not-found", options.IgnoreNotFound, "Treat "resource not found" as a successful delete. Defaults to "true" when --all is specified.") cmd.Flags().BoolVar(&options.Cascade, "cascade", options.Cascade, "If true, cascade the deletion of the resources managed by this resource (e.g. Pods created by a ReplicationController).  Default true.") <del> cmd.Flags().IntVar(&options.GracePeriod, "grace-period", options.GracePeriod, "Period of time in seconds given to the resource to terminate gracefully. Ignored if negative.") </del> <ins> cmd.Flags().IntVar(&options.GracePeriod, "grace-period", options.GracePeriod, "Period of time in seconds given to the resource to terminate gracefully. Ignored if negative. Set to 1 for immediate shutdown. Can only be set to 0 when --force is true (force deletion).") </ins> cmd.Flags().BoolVar(&options.DeleteNow, "now", options.DeleteNow, "If true, resources are signaled for immediate shutdown (same as --grace-period=1).") <del> cmd.Flags().BoolVar(&options.ForceDeletion, "force", options.ForceDeletion, "Immediate deletion of some resources may result in inconsistency or data loss and requires confirmation.") </del> <ins> cmd.Flags().BoolVar(&options.ForceDeletion, "force", options.ForceDeletion, "Only used when grace-period=0. If true, immediately remove resources from API and bypass graceful deletion. Note that immediate deletion of some resources may result in inconsistency or data loss and requires confirmation.") </ins> cmd.Flags().DurationVar(&options.Timeout, "timeout", options.Timeout, "The length of time to wait before giving up on a delete, zero means determine a timeout from the size of the object") cmdutil.AddOutputVarFlagsForMutation(cmd, &options.Output) cmdutil.AddInclude3rdPartyVarFlags(cmd, &options.Include3rdParty)


Question:
IMO it's surprising that does not force delete a resource. only takes effect when . The description of reads this: We should make it more clear what does and even print warnings when is specified without .
Changing the meaning of force to actually break cluster consistency be default could lead to data loss, which is why we haven’t changed it so far. agree we should print more info and clarfiy

o.WaitForDeletion = true o.GracePeriod = 1 } <ins> } else if o.ForceDeletion { fmt.Fprintf(o.ErrOut, "warning: --force is ignored because --grace-period is not 0.n") </ins> } return nil }


Question:
IMO it's surprising that does not force delete a resource. only takes effect when . The description of reads this: We should make it more clear what does and even print warnings when is specified without .
Changing the meaning of force to actually break cluster consistency be default could lead to data loss, which is why we haven’t changed it so far. agree we should print more info and clarfiy

// If there is no reaper for this resources and the user didn't explicitly ask for stop. if kubectl.IsNoSuchReaperError(err) && isDefaultDelete { // No client side reaper found. Let the server do cascading deletion. <del> return cascadingDeleteResource(info, out, shortOutput) </del> <ins> return cascadingDeleteResource(info, out, shortOutput, gracePeriod) </ins> } return cmdutil.AddSourceToErr("reaping", info.Source, err) }


Question:
IMO it's surprising that does not force delete a resource. only takes effect when . The description of reads this: We should make it more clear what does and even print warnings when is specified without .
Changing the meaning of force to actually break cluster consistency be default could lead to data loss, which is why we haven’t changed it so far. agree we should print more info and clarfiy

} } if !quiet { <del> printDeletion(info, out, shortOutput) </del> <ins> printDeletion(info, out, shortOutput, gracePeriod) </ins> } return nil })


Question:
IMO it's surprising that does not force delete a resource. only takes effect when . The description of reads this: We should make it more clear what does and even print warnings when is specified without .
Changing the meaning of force to actually break cluster consistency be default could lead to data loss, which is why we haven’t changed it so far. agree we should print more info and clarfiy

options = metav1.NewDeleteOptions(int64(gracePeriod)) } options.OrphanDependents = &orphan <del> return deleteResource(info, out, shortOutput, options) </del> <ins> return deleteResource(info, out, shortOutput, options, gracePeriod) </ins> }) if err != nil { return err


Question:
IMO it's surprising that does not force delete a resource. only takes effect when . The description of reads this: We should make it more clear what does and even print warnings when is specified without .
Changing the meaning of force to actually break cluster consistency be default could lead to data loss, which is why we haven’t changed it so far. agree we should print more info and clarfiy

return nil } <del> func cascadingDeleteResource(info *resource.Info, out io.Writer, shortOutput bool) error { </del> <ins> func cascadingDeleteResource(info *resource.Info, out io.Writer, shortOutput bool, gracePeriod int) error { </ins> falseVar := false deleteOptions := &metav1.DeleteOptions{OrphanDependents: &falseVar} <del> return deleteResource(info, out, shortOutput, deleteOptions) </del> <ins> return deleteResource(info, out, shortOutput, deleteOptions, gracePeriod) </ins> } <del> func deleteResource(info *resource.Info, out io.Writer, shortOutput bool, deleteOptions *metav1.DeleteOptions) error { </del> <ins> func deleteResource(info *resource.Info, out io.Writer, shortOutput bool, deleteOptions *metav1.DeleteOptions, gracePeriod int) error { </ins> if err := resource.NewHelper(info.Client, info.Mapping).DeleteWithOptions(info.Namespace, info.Name, deleteOptions); err != nil { return cmdutil.AddSourceToErr("deleting", info.Source, err) } <del> printDeletion(info, out, shortOutput) </del> <ins> printDeletion(info, out, shortOutput, gracePeriod) </ins> return nil } // deletion printing is special because they don't have an object to print.  This logic mirrors PrintSuccess <del> func printDeletion(info *resource.Info, out io.Writer, shortOutput bool) { </del> <ins> func printDeletion(info *resource.Info, out io.Writer, shortOutput bool, gracePeriod int) { </ins> operation := "deleted" groupKind := info.Mapping.GroupVersionKind kindString := fmt.Sprintf("%s.%s", strings.ToLower(groupKind.Kind), groupKind.Group)


Question:
IMO it's surprising that does not force delete a resource. only takes effect when . The description of reads this: We should make it more clear what does and even print warnings when is specified without .
Changing the meaning of force to actually break cluster consistency be default could lead to data loss, which is why we haven’t changed it so far. agree we should print more info and clarfiy

kindString = strings.ToLower(groupKind.Kind) } <ins> if gracePeriod == 0 { operation = "force deleted" } </ins> if shortOutput { // -o name: prints resource/name fmt.Fprintf(out, "%s/%sn", kindString, info.Name)


Question:
What happened: Deployed a webhook backend and service as follows: Please note that there is only 8443 port on the service. Register the webhook as follows: Got the following error when creating pods: What you expected to happen: Based on , should be called. Not the 443 port shown in the error message. How to reproduce it (as minimally and precisely as possible): Just deploy the same setup and register webhook. Anything else we need to know?: Environment: Kubernetes version (use ): HEAD Client Version: version.Info{Major:"1", Minor:"10+", GitVersion:"v1.10.0-alpha.1.97+-dirty", GitCommit:"", GitTreeState:"dirty", BuildDate:"2018-01-22T20:00:24Z", GoVersion:"go1.9.1", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"11+", GitVersion:"v1.11.0-alpha.0.470+", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-03-12T19:36:42Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: GCE OS (e.g. from /etc/os-release): COS Kernel (e.g. ): COS
thanks for the detailed bug report!
I actually think that the API specification (or documentation) must change. Not all API servers will have access to look up the Service object and see what ports it defines. If we want to support ports other than 443, the port information must be specified in the webhook object, not discovered by direct Service object lookup
/assign
Looks like it will always use 443 no matter what ports are available, right? Should our documentation reflect this or should it behave differently? (also just sets 443 in the custom dial wrapper thing )
As a first step, I would adjust the documentation. As noted in , if we want to support ports other than 443, that information needs to be to the webhook config object

"format": "byte" }, "service": { <del> "description": "`service` is a reference to the service for this webhook. Either `service` or `url` must be specified.nnIf the webhook is running within the cluster, then you should use `service`.nnIf there is only one port open for the service, that port will be used. If there are multiple ports open, port 443 will be used if it is open, otherwise it is an error.", </del> <ins> "description": "`service` is a reference to the service for this webhook. Either `service` or `url` must be specified.nnIf the webhook is running within the cluster, then you should use `service`.nnPort 443 will be used if it is open, otherwise it is an error.", </ins> "$ref": "#/definitions/io.k8s.api.admissionregistration.v1beta1.ServiceReference" }, "url": {


Question:
What happened: Deployed a webhook backend and service as follows: Please note that there is only 8443 port on the service. Register the webhook as follows: Got the following error when creating pods: What you expected to happen: Based on , should be called. Not the 443 port shown in the error message. How to reproduce it (as minimally and precisely as possible): Just deploy the same setup and register webhook. Anything else we need to know?: Environment: Kubernetes version (use ): HEAD Client Version: version.Info{Major:"1", Minor:"10+", GitVersion:"v1.10.0-alpha.1.97+-dirty", GitCommit:"", GitTreeState:"dirty", BuildDate:"2018-01-22T20:00:24Z", GoVersion:"go1.9.1", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"11+", GitVersion:"v1.11.0-alpha.0.470+", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-03-12T19:36:42Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: GCE OS (e.g. from /etc/os-release): COS Kernel (e.g. ): COS
thanks for the detailed bug report!
I actually think that the API specification (or documentation) must change. Not all API servers will have access to look up the Service object and see what ports it defines. If we want to support ports other than 443, the port information must be specified in the webhook object, not discovered by direct Service object lookup
/assign
Looks like it will always use 443 no matter what ports are available, right? Should our documentation reflect this or should it behave differently? (also just sets 443 in the custom dial wrapper thing )
As a first step, I would adjust the documentation. As noted in , if we want to support ports other than 443, that information needs to be to the webhook config object

}, "service": { "$ref": "v1beta1.ServiceReference", <del> "description": "`service` is a reference to the service for this webhook. Either `service` or `url` must be specified.nnIf the webhook is running within the cluster, then you should use `service`.nnIf there is only one port open for the service, that port will be used. If there are multiple ports open, port 443 will be used if it is open, otherwise it is an error." </del> <ins> "description": "`service` is a reference to the service for this webhook. Either `service` or `url` must be specified.nnIf the webhook is running within the cluster, then you should use `service`.nnPort 443 will be used if it is open, otherwise it is an error." </ins> }, "caBundle": { "type": "string",


Question:
What happened: Deployed a webhook backend and service as follows: Please note that there is only 8443 port on the service. Register the webhook as follows: Got the following error when creating pods: What you expected to happen: Based on , should be called. Not the 443 port shown in the error message. How to reproduce it (as minimally and precisely as possible): Just deploy the same setup and register webhook. Anything else we need to know?: Environment: Kubernetes version (use ): HEAD Client Version: version.Info{Major:"1", Minor:"10+", GitVersion:"v1.10.0-alpha.1.97+-dirty", GitCommit:"", GitTreeState:"dirty", BuildDate:"2018-01-22T20:00:24Z", GoVersion:"go1.9.1", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"11+", GitVersion:"v1.11.0-alpha.0.470+", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-03-12T19:36:42Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: GCE OS (e.g. from /etc/os-release): COS Kernel (e.g. ): COS
thanks for the detailed bug report!
I actually think that the API specification (or documentation) must change. Not all API servers will have access to look up the Service object and see what ports it defines. If we want to support ports other than 443, the port information must be specified in the webhook object, not discovered by direct Service object lookup
/assign
Looks like it will always use 443 no matter what ports are available, right? Should our documentation reflect this or should it behave differently? (also just sets 443 in the custom dial wrapper thing )
As a first step, I would adjust the documentation. As noted in , if we want to support ports other than 443, that information needs to be to the webhook config object

 If the webhook is running within the cluster, then you should use service.

  If there is only one port open for the service, that port will be used. If there are multiple ports open, port 443 will be used if it is open, otherwise it is an error.
	   Port 443 will be used if it is open, otherwise it is an error.

true

v1beta1.ServiceReference


Question:
What happened: Deployed a webhook backend and service as follows: Please note that there is only 8443 port on the service. Register the webhook as follows: Got the following error when creating pods: What you expected to happen: Based on , should be called. Not the 443 port shown in the error message. How to reproduce it (as minimally and precisely as possible): Just deploy the same setup and register webhook. Anything else we need to know?: Environment: Kubernetes version (use ): HEAD Client Version: version.Info{Major:"1", Minor:"10+", GitVersion:"v1.10.0-alpha.1.97+-dirty", GitCommit:"", GitTreeState:"dirty", BuildDate:"2018-01-22T20:00:24Z", GoVersion:"go1.9.1", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"11+", GitVersion:"v1.11.0-alpha.0.470+", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-03-12T19:36:42Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: GCE OS (e.g. from /etc/os-release): COS Kernel (e.g. ): COS
thanks for the detailed bug report!
I actually think that the API specification (or documentation) must change. Not all API servers will have access to look up the Service object and see what ports it defines. If we want to support ports other than 443, the port information must be specified in the webhook object, not discovered by direct Service object lookup
/assign
Looks like it will always use 443 no matter what ports are available, right? Should our documentation reflect this or should it behave differently? (also just sets 443 in the custom dial wrapper thing )
As a first step, I would adjust the documentation. As noted in , if we want to support ports other than 443, that information needs to be to the webhook config object

// // If the webhook is running within the cluster, then you should use `service`. // <del> // If there is only one port open for the service, that port will be // used. If there are multiple ports open, port 443 will be used if it // is open, otherwise it is an error. </del> <ins> // Port 443 will be used if it is open, otherwise it is an error. </ins> // // +optional Service *ServiceReference


Question:
What happened: Deployed a webhook backend and service as follows: Please note that there is only 8443 port on the service. Register the webhook as follows: Got the following error when creating pods: What you expected to happen: Based on , should be called. Not the 443 port shown in the error message. How to reproduce it (as minimally and precisely as possible): Just deploy the same setup and register webhook. Anything else we need to know?: Environment: Kubernetes version (use ): HEAD Client Version: version.Info{Major:"1", Minor:"10+", GitVersion:"v1.10.0-alpha.1.97+-dirty", GitCommit:"", GitTreeState:"dirty", BuildDate:"2018-01-22T20:00:24Z", GoVersion:"go1.9.1", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"11+", GitVersion:"v1.11.0-alpha.0.470+", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-03-12T19:36:42Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: GCE OS (e.g. from /etc/os-release): COS Kernel (e.g. ): COS
thanks for the detailed bug report!
I actually think that the API specification (or documentation) must change. Not all API servers will have access to look up the Service object and see what ports it defines. If we want to support ports other than 443, the port information must be specified in the webhook object, not discovered by direct Service object lookup
/assign
Looks like it will always use 443 no matter what ports are available, right? Should our documentation reflect this or should it behave differently? (also just sets 443 in the custom dial wrapper thing )
As a first step, I would adjust the documentation. As noted in , if we want to support ports other than 443, that information needs to be to the webhook config object

// // If the webhook is running within the cluster, then you should use `service`. // <del> // If there is only one port open for the service, that port will be // used. If there are multiple ports open, port 443 will be used if it // is open, otherwise it is an error. </del> <ins> // Port 443 will be used if it is open, otherwise it is an error. </ins> // // +optional optional ServiceReference service = 1;


Question:
What happened: Deployed a webhook backend and service as follows: Please note that there is only 8443 port on the service. Register the webhook as follows: Got the following error when creating pods: What you expected to happen: Based on , should be called. Not the 443 port shown in the error message. How to reproduce it (as minimally and precisely as possible): Just deploy the same setup and register webhook. Anything else we need to know?: Environment: Kubernetes version (use ): HEAD Client Version: version.Info{Major:"1", Minor:"10+", GitVersion:"v1.10.0-alpha.1.97+-dirty", GitCommit:"", GitTreeState:"dirty", BuildDate:"2018-01-22T20:00:24Z", GoVersion:"go1.9.1", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"11+", GitVersion:"v1.11.0-alpha.0.470+", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-03-12T19:36:42Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: GCE OS (e.g. from /etc/os-release): COS Kernel (e.g. ): COS
thanks for the detailed bug report!
I actually think that the API specification (or documentation) must change. Not all API servers will have access to look up the Service object and see what ports it defines. If we want to support ports other than 443, the port information must be specified in the webhook object, not discovered by direct Service object lookup
/assign
Looks like it will always use 443 no matter what ports are available, right? Should our documentation reflect this or should it behave differently? (also just sets 443 in the custom dial wrapper thing )
As a first step, I would adjust the documentation. As noted in , if we want to support ports other than 443, that information needs to be to the webhook config object

// // If the webhook is running within the cluster, then you should use `service`. // <del> // If there is only one port open for the service, that port will be // used. If there are multiple ports open, port 443 will be used if it // is open, otherwise it is an error. </del> <ins> // Port 443 will be used if it is open, otherwise it is an error. </ins> // // +optional Service *ServiceReference `json:"service" protobuf:"bytes,1,opt,name=service"`


Question:
What happened: Deployed a webhook backend and service as follows: Please note that there is only 8443 port on the service. Register the webhook as follows: Got the following error when creating pods: What you expected to happen: Based on , should be called. Not the 443 port shown in the error message. How to reproduce it (as minimally and precisely as possible): Just deploy the same setup and register webhook. Anything else we need to know?: Environment: Kubernetes version (use ): HEAD Client Version: version.Info{Major:"1", Minor:"10+", GitVersion:"v1.10.0-alpha.1.97+-dirty", GitCommit:"", GitTreeState:"dirty", BuildDate:"2018-01-22T20:00:24Z", GoVersion:"go1.9.1", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"11+", GitVersion:"v1.11.0-alpha.0.470+", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-03-12T19:36:42Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: GCE OS (e.g. from /etc/os-release): COS Kernel (e.g. ): COS
thanks for the detailed bug report!
I actually think that the API specification (or documentation) must change. Not all API servers will have access to look up the Service object and see what ports it defines. If we want to support ports other than 443, the port information must be specified in the webhook object, not discovered by direct Service object lookup
/assign
Looks like it will always use 443 no matter what ports are available, right? Should our documentation reflect this or should it behave differently? (also just sets 443 in the custom dial wrapper thing )
As a first step, I would adjust the documentation. As noted in , if we want to support ports other than 443, that information needs to be to the webhook config object

var map_WebhookClientConfig = map[string]string{ "":         "WebhookClientConfig contains the information to make a TLS connection with the webhook", "url":      "`url` gives the location of the webhook, in standard URL form (`[scheme://]host:port/path`). Exactly one of `url` or `service` must be specified.nnThe `host` should not refer to a service running in the cluster; use the `service` field instead. The host might be resolved via external DNS in some apiservers (e.g., `kube-apiserver` cannot resolve in-cluster DNS as that would be a layering violation). `host` may also be an IP address.nnPlease note that using `localhost` or `127.0.0.1` as a `host` is risky unless you take great care to run this webhook on all hosts which run an apiserver which might need to make calls to this webhook. Such installs are likely to be non-portable, i.e., not easy to turn up in a new cluster.nnThe scheme must be "https"; the URL must begin with "https://".nnA path is optional, and if present may be any string permissible in a URL. You may use the path to pass an arbitrary string to the webhook, for example, a cluster identifier.nnAttempting to use a user or basic auth e.g. "user:password@" is not allowed. Fragments ("#...") and query parameters ("?...") are not allowed, either.", <del> "service":  "`service` is a reference to the service for this webhook. Either `service` or `url` must be specified.nnIf the webhook is running within the cluster, then you should use `service`.nnIf there is only one port open for the service, that port will be used. If there are multiple ports open, port 443 will be used if it is open, otherwise it is an error.", </del> <ins> "service":  "`service` is a reference to the service for this webhook. Either `service` or `url` must be specified.nnIf the webhook is running within the cluster, then you should use `service`.nnPort 443 will be used if it is open, otherwise it is an error.", </ins> "caBundle": "`caBundle` is a PEM encoded CA bundle which will be used to validate the webhook's server certificate. Required.", }


Question:
The new test (introduced by ) seems to be quite flaky. Example failures: https://k8s- https://k8s- https://k8s- Should a "performance" test even be run in the pre-submit PR runs? /cc
It does connect to etcd and does a fair amount of node additions and deletions. Most of the times I've looked at the error, it seems to be at the startup when connecting to etcd. Is there a way to retain this capability so we can continue to check on demand but not necessarily run every time?
The error in has already been fixed, if you notice , we now have a lock and it should not be happening now. For the rest, the error is in the startup itself, namely, , which is starting the framework connecting to etcd. Any idea's on why this should be causing flakiness? This was the same pattern used in schedulerperf -
Can we please disable this test for the moment? It's causing significant flakiness and blocking other PRs from being merged...
I'd suggest disable the tests first to stop the bleeding, and debug the issue at your own pace. It's probably not a good idea to run performance tests in this environment anyway since the tests are run in parallel (IIRC). Maybe you can set up a separate post-submit test job for these special-purpose tests.
can you take a look at PR ? It adds skip for short tests, which should avoid running this for very PR?
https://k8s- shows that ipamperf tests are now skipped during integration testing. All checks have passed and has given /lgtm. Waiting for auto-merge. can you close this issue once verified? thanks!
the bot will do that automatically since your PR description says it fixes this issue.

} func TestPerformance(t *testing.T) { <ins> if testing.Short() { // TODO (#61854) find why flakiness is caused by etcd connectivity before enabling always t.Skip("Skipping because we want to run short tests") } </ins> apiURL, masterShutdown := util.StartApiserver() defer masterShutdown()


Question:
This is continuation of and . What happened: When StorageClass refers to a key that is not accessible to Kubernetes (i.e. AWS API returns AccessDeniedException), AWS plugin still tries to provision a volume under assumption that while Kubernetes can't access the key, the other components may access it and create the volume and attach it to a node later. Related code is - checks only for . Other errors are just logged and provisioning continues. Problem is that when AWS really can't access the key, it will provision some volume (ec2.CreateVolume returns success + valid AWS EBS), but this AWS EBS disappears after few seconds. Therefore Kubernetes does not know that something went wrong and creates a PV for the returned EBS. Kubernetes later fails to run a pod that refers to such PV, because the PV does not exist. What you expected to happen: Kubernetes must not create PV in this case and must report error during provisioning of such volume. Ideas: According to , the only signal that AWS produces is an event in Amazon CloudWatch Events that a key could not be used for encryption. Kubernetes AWS cloud provider is not subscribed to these events and does not get this notification. We would need new permission for controller-manager to get the events and some new code to filter them and report back to user. Kubernetes AWS cloud provider could wait for X seconds and re-check that created volume still exists and did not disappear because of inaccessible key. What would be reliable value of X? Kubernetes volume plugin for AWS could return both a warning "I failed to check encryption key, created volume may not work" that would be propagated to user and the suspicious volume at the same time, but the API between volume plugins and Kubernetes allows only one - either error or volume. It could be extended, still the user experience would be suboptimal. Kubernetes could send an error to user during volume attach (!) that a volume does not exist, please check your KMS key permissions. Again, UX is quite bad here. /kind bug /sig aws
Checking the KMS key feels wrong here, as you could still create an encrypted EBS volume and not have the DescribeKey permission in IAM or via the key policy. If the main source of errors are sent via CloudWatch Events, then best practice would seem to be to subscribe via SNS to receive those events. Possibly fall back to vague "send an error to user during volume attach (!) that a volume does not exist, please check your KMS key permissions," where a SNS topic isn't provided, but otherwise use CloudWatch events to enrich state/messages. Would then also need to change docs to reflect needing to set up CloudWatch Events -SNS topic. Big drawback of going down this route is a fair few downstream cluster bootstrap projects would likely need updating to support it in their CloudFormation / Terraform bits.
According to it takes quite a long time until an event is available in CloudWatch events. what was the average / maximum delay?
I haven't calculated the metrics accurately but the cloudwatch events arrival vary(and standard deviation was high). Using cloudwatch events was also considered for maintaining instance cache and as well concluded that - they are not realtime (AWS itself documents them as "near realtime"). It may be - we need to measure them again and find out if we can use them for the purpose here. Checking the KMS key feels wrong here, as you could still create an encrypted EBS volume and not have the DescribeKey permission in IAM or via the key policy. from jan's analysis above it appears that - if IAM does not have permission to DescribeKey then EBS volume gets created but then it disappears soon. So net result still is - if "current user" can't describekey, she can't provision the volume. Is there any documentation that indicates otherwise? cc
Testing this for real now. User creates volume with no access to underlying CMK, or insufficient permissions does not seem to generate events with sufficient information as to what went wrong, i.e. don't appear to receive events for create volume failure, only deletion. E.g.: This policy is sufficient to allow the creation of volumes - isn't necessary for the volume to be created. It isn't sufficient to attach the volume however, creating the following error, The permission's required to fix this, to delegate access to EC2 itself: Bit of experimentation shows that only is required for attaching - the ones for creating the volume aren't needed at the attachment stage. The docs at is therefore correct for volume creation, but doesn't describe the volume attachment scenario.
So, what is our solution then? From the above it seems Kubernetes can't reliably check if a key is usable for encryption of a PV. And it can't get an usable event either.
Possibly one of the following: change. Live with the pain. "GenerateDataKey" and "Decrypt" on the generated ciphertext blob prior to volume creation as a preflight check. The advantage being that we can throw an error that this isn't going to work beforehand, but with the disadvantage of doubling up API calls. This is probably fine, and I think if you're creating encrypted volumes at a rate that's likely to run into API call limits or get super expensive, you might have other problems? We then also do the CreateGrant check during volume attachment (as an optional extra). the same check, but only after an error. Problem being that we get much less in the way of guaranteeing what the root cause is, but we don't add to the API calls in the happy path. In any case, is this bug biting sufficiently to fix here, or should we start thinking about the CSI implementation?
thanks a lot for detailed investigation. I'll try to get some time to implement 2. even before CSI. Users are very innovative in setting up their cluster permissions wrong and the messaging to user is very confusing in this particular issue.
/assign
2. does not work. Kubernetes does not need to have GenerateDataKey and Decrypt permissions to create encrypted volume. My AWS account is severely limited (I am using shared company account) and I can't call GenerateDataKey on a key: But I can create encrypted EBS using the same key: The volume gets created successfully. It's not the user who runs Kubernetes who needs permissions for GenerateDataKey, it's AWS itself.
And the volume stayed created and didn't get reaped after a few seconds? I was confused about this part, it didn't seem to follow how the APIs should work. Has anyone ever created a volume with that key, or is it a new one? I'll try what I did previously in a bit across a few regions.
Yes, the volume stayed around for several minutes, then I deleted it manually. The key is our default encryption key, so when I create an encrypted volume with no key, I get this one. No zones, no regions involved.
I got an idea, the volume is "creating" and then it gets "available". I could in a loop until newly created volume either gets "Available" or deleted. This will abuse the API quota ( is cheap, right?) + it will slow down provisioning a bit - all volumes I tried were Available in a second or two. Will it work reliably enough?
Might be the difference then, as I don't think the same perms as I described above are needed for the default key as I assume that it gets automatically delegated to the EC2 service. I only tested & documented the explicit, alternate key path above. Your approach should work, but I don't know when EC2 API limits will kick in and how soon users would see that happen. Worth a shot though?
It takes ~4 seconds for a volume to reach "available", it's quite a long time. This will affect encrypted volumes with explicit keys only.
Fix:

createTagFactor       = 2.0 createTagSteps        = 9 <ins> // encryptedCheck* is configuration of poll for created volume to check // it has not been silently removed by AWS. // On a random AWS account (shared among several developers) it took 4s on // average. encryptedCheckInterval = 1 * time.Second encryptedCheckTimeout  = 30 * time.Second </ins> // Number of node names that can be added to a filter. The AWS limit is 200 // but we are using a lower limit on purpose filterNodeLimit = 150


Question:
This is continuation of and . What happened: When StorageClass refers to a key that is not accessible to Kubernetes (i.e. AWS API returns AccessDeniedException), AWS plugin still tries to provision a volume under assumption that while Kubernetes can't access the key, the other components may access it and create the volume and attach it to a node later. Related code is - checks only for . Other errors are just logged and provisioning continues. Problem is that when AWS really can't access the key, it will provision some volume (ec2.CreateVolume returns success + valid AWS EBS), but this AWS EBS disappears after few seconds. Therefore Kubernetes does not know that something went wrong and creates a PV for the returned EBS. Kubernetes later fails to run a pod that refers to such PV, because the PV does not exist. What you expected to happen: Kubernetes must not create PV in this case and must report error during provisioning of such volume. Ideas: According to , the only signal that AWS produces is an event in Amazon CloudWatch Events that a key could not be used for encryption. Kubernetes AWS cloud provider is not subscribed to these events and does not get this notification. We would need new permission for controller-manager to get the events and some new code to filter them and report back to user. Kubernetes AWS cloud provider could wait for X seconds and re-check that created volume still exists and did not disappear because of inaccessible key. What would be reliable value of X? Kubernetes volume plugin for AWS could return both a warning "I failed to check encryption key, created volume may not work" that would be propagated to user and the suspicious volume at the same time, but the API between volume plugins and Kubernetes allows only one - either error or volume. It could be extended, still the user experience would be suboptimal. Kubernetes could send an error to user during volume attach (!) that a volume does not exist, please check your KMS key permissions. Again, UX is quite bad here. /kind bug /sig aws
Checking the KMS key feels wrong here, as you could still create an encrypted EBS volume and not have the DescribeKey permission in IAM or via the key policy. If the main source of errors are sent via CloudWatch Events, then best practice would seem to be to subscribe via SNS to receive those events. Possibly fall back to vague "send an error to user during volume attach (!) that a volume does not exist, please check your KMS key permissions," where a SNS topic isn't provided, but otherwise use CloudWatch events to enrich state/messages. Would then also need to change docs to reflect needing to set up CloudWatch Events -SNS topic. Big drawback of going down this route is a fair few downstream cluster bootstrap projects would likely need updating to support it in their CloudFormation / Terraform bits.
According to it takes quite a long time until an event is available in CloudWatch events. what was the average / maximum delay?
I haven't calculated the metrics accurately but the cloudwatch events arrival vary(and standard deviation was high). Using cloudwatch events was also considered for maintaining instance cache and as well concluded that - they are not realtime (AWS itself documents them as "near realtime"). It may be - we need to measure them again and find out if we can use them for the purpose here. Checking the KMS key feels wrong here, as you could still create an encrypted EBS volume and not have the DescribeKey permission in IAM or via the key policy. from jan's analysis above it appears that - if IAM does not have permission to DescribeKey then EBS volume gets created but then it disappears soon. So net result still is - if "current user" can't describekey, she can't provision the volume. Is there any documentation that indicates otherwise? cc
Testing this for real now. User creates volume with no access to underlying CMK, or insufficient permissions does not seem to generate events with sufficient information as to what went wrong, i.e. don't appear to receive events for create volume failure, only deletion. E.g.: This policy is sufficient to allow the creation of volumes - isn't necessary for the volume to be created. It isn't sufficient to attach the volume however, creating the following error, The permission's required to fix this, to delegate access to EC2 itself: Bit of experimentation shows that only is required for attaching - the ones for creating the volume aren't needed at the attachment stage. The docs at is therefore correct for volume creation, but doesn't describe the volume attachment scenario.
So, what is our solution then? From the above it seems Kubernetes can't reliably check if a key is usable for encryption of a PV. And it can't get an usable event either.
Possibly one of the following: change. Live with the pain. "GenerateDataKey" and "Decrypt" on the generated ciphertext blob prior to volume creation as a preflight check. The advantage being that we can throw an error that this isn't going to work beforehand, but with the disadvantage of doubling up API calls. This is probably fine, and I think if you're creating encrypted volumes at a rate that's likely to run into API call limits or get super expensive, you might have other problems? We then also do the CreateGrant check during volume attachment (as an optional extra). the same check, but only after an error. Problem being that we get much less in the way of guaranteeing what the root cause is, but we don't add to the API calls in the happy path. In any case, is this bug biting sufficiently to fix here, or should we start thinking about the CSI implementation?
thanks a lot for detailed investigation. I'll try to get some time to implement 2. even before CSI. Users are very innovative in setting up their cluster permissions wrong and the messaging to user is very confusing in this particular issue.
/assign
2. does not work. Kubernetes does not need to have GenerateDataKey and Decrypt permissions to create encrypted volume. My AWS account is severely limited (I am using shared company account) and I can't call GenerateDataKey on a key: But I can create encrypted EBS using the same key: The volume gets created successfully. It's not the user who runs Kubernetes who needs permissions for GenerateDataKey, it's AWS itself.
And the volume stayed created and didn't get reaped after a few seconds? I was confused about this part, it didn't seem to follow how the APIs should work. Has anyone ever created a volume with that key, or is it a new one? I'll try what I did previously in a bit across a few regions.
Yes, the volume stayed around for several minutes, then I deleted it manually. The key is our default encryption key, so when I create an encrypted volume with no key, I get this one. No zones, no regions involved.
I got an idea, the volume is "creating" and then it gets "available". I could in a loop until newly created volume either gets "Available" or deleted. This will abuse the API quota ( is cheap, right?) + it will slow down provisioning a bit - all volumes I tried were Available in a second or two. Will it work reliably enough?
Might be the difference then, as I don't think the same perms as I described above are needed for the default key as I assume that it gets automatically delegated to the EC2 service. I only tested & documented the explicit, alternate key path above. Your approach should work, but I don't know when EC2 API limits will kick in and how soon users would see that happen. Worth a shot though?
It takes ~4 seconds for a volume to reach "available", it's quite a long time. This will affect encrypted volumes with explicit keys only.
Fix:

request.VolumeType = aws.String(createType) request.Encrypted = aws.Bool(volumeOptions.Encrypted) if len(volumeOptions.KmsKeyId) > 0 { <del> if missing, err := c.checkEncryptionKey(volumeOptions.KmsKeyId); err != nil { if missing { // KSM key is missing, provisioning would fail return "", err } // Log checkEncryptionKey error and try provisioning anyway. glog.Warningf("Cannot check KSM key %s: %v", volumeOptions.KmsKeyId, err) } </del> request.KmsKeyId = aws.String(volumeOptions.KmsKeyId) request.Encrypted = aws.Bool(true) }


Question:
This is continuation of and . What happened: When StorageClass refers to a key that is not accessible to Kubernetes (i.e. AWS API returns AccessDeniedException), AWS plugin still tries to provision a volume under assumption that while Kubernetes can't access the key, the other components may access it and create the volume and attach it to a node later. Related code is - checks only for . Other errors are just logged and provisioning continues. Problem is that when AWS really can't access the key, it will provision some volume (ec2.CreateVolume returns success + valid AWS EBS), but this AWS EBS disappears after few seconds. Therefore Kubernetes does not know that something went wrong and creates a PV for the returned EBS. Kubernetes later fails to run a pod that refers to such PV, because the PV does not exist. What you expected to happen: Kubernetes must not create PV in this case and must report error during provisioning of such volume. Ideas: According to , the only signal that AWS produces is an event in Amazon CloudWatch Events that a key could not be used for encryption. Kubernetes AWS cloud provider is not subscribed to these events and does not get this notification. We would need new permission for controller-manager to get the events and some new code to filter them and report back to user. Kubernetes AWS cloud provider could wait for X seconds and re-check that created volume still exists and did not disappear because of inaccessible key. What would be reliable value of X? Kubernetes volume plugin for AWS could return both a warning "I failed to check encryption key, created volume may not work" that would be propagated to user and the suspicious volume at the same time, but the API between volume plugins and Kubernetes allows only one - either error or volume. It could be extended, still the user experience would be suboptimal. Kubernetes could send an error to user during volume attach (!) that a volume does not exist, please check your KMS key permissions. Again, UX is quite bad here. /kind bug /sig aws
Checking the KMS key feels wrong here, as you could still create an encrypted EBS volume and not have the DescribeKey permission in IAM or via the key policy. If the main source of errors are sent via CloudWatch Events, then best practice would seem to be to subscribe via SNS to receive those events. Possibly fall back to vague "send an error to user during volume attach (!) that a volume does not exist, please check your KMS key permissions," where a SNS topic isn't provided, but otherwise use CloudWatch events to enrich state/messages. Would then also need to change docs to reflect needing to set up CloudWatch Events -SNS topic. Big drawback of going down this route is a fair few downstream cluster bootstrap projects would likely need updating to support it in their CloudFormation / Terraform bits.
According to it takes quite a long time until an event is available in CloudWatch events. what was the average / maximum delay?
I haven't calculated the metrics accurately but the cloudwatch events arrival vary(and standard deviation was high). Using cloudwatch events was also considered for maintaining instance cache and as well concluded that - they are not realtime (AWS itself documents them as "near realtime"). It may be - we need to measure them again and find out if we can use them for the purpose here. Checking the KMS key feels wrong here, as you could still create an encrypted EBS volume and not have the DescribeKey permission in IAM or via the key policy. from jan's analysis above it appears that - if IAM does not have permission to DescribeKey then EBS volume gets created but then it disappears soon. So net result still is - if "current user" can't describekey, she can't provision the volume. Is there any documentation that indicates otherwise? cc
Testing this for real now. User creates volume with no access to underlying CMK, or insufficient permissions does not seem to generate events with sufficient information as to what went wrong, i.e. don't appear to receive events for create volume failure, only deletion. E.g.: This policy is sufficient to allow the creation of volumes - isn't necessary for the volume to be created. It isn't sufficient to attach the volume however, creating the following error, The permission's required to fix this, to delegate access to EC2 itself: Bit of experimentation shows that only is required for attaching - the ones for creating the volume aren't needed at the attachment stage. The docs at is therefore correct for volume creation, but doesn't describe the volume attachment scenario.
So, what is our solution then? From the above it seems Kubernetes can't reliably check if a key is usable for encryption of a PV. And it can't get an usable event either.
Possibly one of the following: change. Live with the pain. "GenerateDataKey" and "Decrypt" on the generated ciphertext blob prior to volume creation as a preflight check. The advantage being that we can throw an error that this isn't going to work beforehand, but with the disadvantage of doubling up API calls. This is probably fine, and I think if you're creating encrypted volumes at a rate that's likely to run into API call limits or get super expensive, you might have other problems? We then also do the CreateGrant check during volume attachment (as an optional extra). the same check, but only after an error. Problem being that we get much less in the way of guaranteeing what the root cause is, but we don't add to the API calls in the happy path. In any case, is this bug biting sufficiently to fix here, or should we start thinking about the CSI implementation?
thanks a lot for detailed investigation. I'll try to get some time to implement 2. even before CSI. Users are very innovative in setting up their cluster permissions wrong and the messaging to user is very confusing in this particular issue.
/assign
2. does not work. Kubernetes does not need to have GenerateDataKey and Decrypt permissions to create encrypted volume. My AWS account is severely limited (I am using shared company account) and I can't call GenerateDataKey on a key: But I can create encrypted EBS using the same key: The volume gets created successfully. It's not the user who runs Kubernetes who needs permissions for GenerateDataKey, it's AWS itself.
And the volume stayed created and didn't get reaped after a few seconds? I was confused about this part, it didn't seem to follow how the APIs should work. Has anyone ever created a volume with that key, or is it a new one? I'll try what I did previously in a bit across a few regions.
Yes, the volume stayed around for several minutes, then I deleted it manually. The key is our default encryption key, so when I create an encrypted volume with no key, I get this one. No zones, no regions involved.
I got an idea, the volume is "creating" and then it gets "available". I could in a loop until newly created volume either gets "Available" or deleted. This will abuse the API quota ( is cheap, right?) + it will slow down provisioning a bit - all volumes I tried were Available in a second or two. Will it work reliably enough?
Might be the difference then, as I don't think the same perms as I described above are needed for the default key as I assume that it gets automatically delegated to the EC2 service. I only tested & documented the explicit, alternate key path above. Your approach should work, but I don't know when EC2 API limits will kick in and how soon users would see that happen. Worth a shot though?
It takes ~4 seconds for a volume to reach "available", it's quite a long time. This will affect encrypted volumes with explicit keys only.
Fix:

return "", fmt.Errorf("error tagging volume %s: %q", volumeName, err) } <ins> // AWS has a bad habbit of reporting success when creating a volume with // encryption keys that either don't exists or have wrong permissions. // Such volume lives for couple of seconds and then it's silently deleted // by AWS. There is no other check to ensure that given KMS key is correct, // because Kubernetes may have limited permissions to the key. if len(volumeOptions.KmsKeyId) > 0 { err := c.waitUntilVolumeAvailable(volumeName) if err != nil { if isAWSErrorVolumeNotFound(err) { err = fmt.Errorf("failed to create encrypted volume: the volume disappeared after creation, most likely due to inaccessible KMS encryption key") } return "", err } } </ins> return volumeName, nil } <del> // checkEncryptionKey tests that given encryption key exists. func (c *Cloud) checkEncryptionKey(keyId string) (missing bool, err error) { input := &kms.DescribeKeyInput{ KeyId: aws.String(keyId), } _, err = c.kms.DescribeKey(input) if err == nil { return false, nil </del> <ins> func (c *Cloud) waitUntilVolumeAvailable(volumeName KubernetesVolumeID) error { disk, err := newAWSDisk(c, volumeName) if err != nil { // Unreachable code return err </ins> } <del> if awsError, ok := err.(awserr.Error); ok { if awsError.Code() == "NotFoundException" { return true, fmt.Errorf("KMS key %s not found: %q", keyId, err) </del> <ins> err = wait.Poll(encryptedCheckInterval, encryptedCheckTimeout, func() (done bool, err error) { vol, err := disk.describeVolume() if err != nil { return true, err </ins> } <del> } return false, fmt.Errorf("Error checking KSM key %s: %q", keyId, err) </del> <ins> if vol.State != nil { switch *vol.State { case "available": // The volume is Available, it won't be deleted now. return true, nil case "creating": return false, nil default: return true, fmt.Errorf("unexpected State of newly created AWS EBS volume %s: %q", volumeName, *vol.State) } } return false, nil }) return err </ins> } // DeleteDisk implements Volumes.DeleteDisk