Question:
Is this a request for help? No What keywords did you search in Kubernetes issues before filing this one? http proxy, httpproxy, httpsproxy, AWS, kubeadm, kubectl, cloud-provider Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT Kubernetes version (use ): v1.4.4 Environment: Cloud provider or hardware configuration: kubeadm on EC2 instances on AWS OS (e.g. from /etc/os-release): Ubuntu 16.04.1 LTS Kernel (e.g. ): Linux ip-10-0-133-99 4.4.0-45-generic -Ubuntu SMP Wed Oct 19 14:12:37 UTC 2016 x8664 x8664 x8664 GNU/Linux Install tools: kubeadm Others: httpproxy, httpsproxy and noproxy set in /etc/environment, Docker config as well on startup scripts that run What happened: won't complete and gets stuck on it's first deployment. When one looks through the logs it shows that the pods are downloading fine but we are stuck due to the kube-controller-manager continually crashing as it can't contact AWS. Elsewhere in kubernetes I see references to and if the AWS cloud provider could use that it would be able to connect as well. From the POD if I manually set httpproxy and httpsproxy and no_proxy I am able to use the AWS CLI without any issues. What you expected to happen: I would expect to successfully complete and use the environment proxies set up on the system so as not to depend on a default Internet Gateway being connected to the VPC just for AWS access. How to reproduce it (as minimally and precisely as possible): Set up a public VPC with Squid set up as a proxy Set up a private VPC peered to the public VPC Set up a master node and run Set up a regular node and run will never fully initialize. If you remove the --cloud-provider aws` then it will complete and nodes will register. Anything else do we need to know:
what do you have in proxy variables ? (env | grep -i proxy) it tries to connect to (tcp 176.32.118.53:443) directly. You're expecting to connect there via proxy ? then do you have something like in your no_proxy ?
Unfortunately on the pod itself for the the environment variables aren't passed through. As this is created using I don't have control over the pod definition. So basically none of the *_proxy variables are set inside the pod in the docker image, however if I and export the variables and try to wget that URL it works. Basically I'm hoping we can update to respect and pass the environment variables down to the system pods if they exist, eg, add: Without it I can't stand up a cluster with AWS integration on a completely private VPC and for such a small change it would make a rather big difference.
ok, so request is actually to pass proxy variables into containers. interesting. I'll check how it can be done. BTW, for noproxy value in your example: things like 10.0.0.0/8 would not work on 99% of software that respects noproxy variable. it must be always exact IP match.
you're right that what I'm actually asking for is to pass proxy variables through either from a commandline setting on kubeadm or pick it up from /etc/environment or shell environment that launches kubeadm. I didn't realize it till I dug in to try and think about how one might go about fixing this. At first I thought we would just pick up the variables from the system environment but then I realized kube-controller-manager is deployed in a container and so we'd need to update the yaml file that launches it to contain the environment variables. As I'd like to go to production with this soon hopefully this is a small change that can be .
This issue was moved to kubernetes/kubeadm

Command:       getSchedulerCommand(cfg), LivenessProbe: componentProbe(10251, "/healthz"), Resources:     componentResources("100m"), <ins> Env:           getProxyEnvVars(), </ins> }), }


Question:
Is this a request for help? No What keywords did you search in Kubernetes issues before filing this one? http proxy, httpproxy, httpsproxy, AWS, kubeadm, kubectl, cloud-provider Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT Kubernetes version (use ): v1.4.4 Environment: Cloud provider or hardware configuration: kubeadm on EC2 instances on AWS OS (e.g. from /etc/os-release): Ubuntu 16.04.1 LTS Kernel (e.g. ): Linux ip-10-0-133-99 4.4.0-45-generic -Ubuntu SMP Wed Oct 19 14:12:37 UTC 2016 x8664 x8664 x8664 GNU/Linux Install tools: kubeadm Others: httpproxy, httpsproxy and noproxy set in /etc/environment, Docker config as well on startup scripts that run What happened: won't complete and gets stuck on it's first deployment. When one looks through the logs it shows that the pods are downloading fine but we are stuck due to the kube-controller-manager continually crashing as it can't contact AWS. Elsewhere in kubernetes I see references to and if the AWS cloud provider could use that it would be able to connect as well. From the POD if I manually set httpproxy and httpsproxy and no_proxy I am able to use the AWS CLI without any issues. What you expected to happen: I would expect to successfully complete and use the environment proxies set up on the system so as not to depend on a default Internet Gateway being connected to the VPC just for AWS access. How to reproduce it (as minimally and precisely as possible): Set up a public VPC with Squid set up as a proxy Set up a private VPC peered to the public VPC Set up a master node and run Set up a regular node and run will never fully initialize. If you remove the --cloud-provider aws` then it will complete and nodes will register. Anything else do we need to know:
what do you have in proxy variables ? (env | grep -i proxy) it tries to connect to (tcp 176.32.118.53:443) directly. You're expecting to connect there via proxy ? then do you have something like in your no_proxy ?
Unfortunately on the pod itself for the the environment variables aren't passed through. As this is created using I don't have control over the pod definition. So basically none of the *_proxy variables are set inside the pod in the docker image, however if I and export the variables and try to wget that URL it works. Basically I'm hoping we can update to respect and pass the environment variables down to the system pods if they exist, eg, add: Without it I can't stand up a cluster with AWS integration on a completely private VPC and for such a small change it would make a rather big difference.
ok, so request is actually to pass proxy variables into containers. interesting. I'll check how it can be done. BTW, for noproxy value in your example: things like 10.0.0.0/8 would not work on 99% of software that respects noproxy variable. it must be always exact IP match.
you're right that what I'm actually asking for is to pass proxy variables through either from a commandline setting on kubeadm or pick it up from /etc/environment or shell environment that launches kubeadm. I didn't realize it till I dug in to try and think about how one might go about fixing this. At first I thought we would just pick up the variables from the system environment but then I realized kube-controller-manager is deployed in a container and so we'd need to update the yaml file that launches it to contain the environment variables. As I'd like to go to production with this soon hopefully this is a small change that can be .
This issue was moved to kubernetes/kubeadm

return } <ins> func getProxyEnvVars() []api.EnvVar { envs := []api.EnvVar{} for _, env := range os.Environ() { pos := strings.Index(env, "=") if pos == -1 { // malformed environment variable, skip it. continue } name := env[:pos] value := env[pos+1:] if strings.HasSuffix(strings.ToLower(name), "_proxy") && value != "" { envVar := api.EnvVar{Name: name, Value: value} envs = append(envs, envVar) } } return envs } </ins>


Question:
Kubernetes version: HEAD What happened: Using interface to delete Job throws an error . What you expected to happen: Job should be deleted. We've found that issue after dependency update on dashboard: We are using to delete resources. It works fine for all other resources. I've investigated that and it appears to be the core issue. Other groups have registered as a known type. however batch is missing it After adding I was able to successfully delete Job using our code. I can quickly create PR for that. Just need a confirmation that this is indeed a bug. cc
+1 apis/autoscaling is similar
I thought we only needed to register it in the external versions (). I'll double check. cc
Registering api.DeleteOptions in batch looks fine. In the future, I think it's more proper to register DeleteOptions and ListOptions as unversioned, like how we treat .
DeleteOptions and ListOptions are expected to be in their group. They have to be in internal version and external version. Most of this code will be fixed in 1.6 (we'll standardize registering the "common" things). , Chao Xu wrote:
I've created PR to fix this issue. Could anyone check that?

&HorizontalPodAutoscaler{}, &HorizontalPodAutoscalerList{}, &api.ListOptions{}, <ins> &api.DeleteOptions{}, </ins> ) return nil }


Question:
Kubernetes version: HEAD What happened: Using interface to delete Job throws an error . What you expected to happen: Job should be deleted. We've found that issue after dependency update on dashboard: We are using to delete resources. It works fine for all other resources. I've investigated that and it appears to be the core issue. Other groups have registered as a known type. however batch is missing it After adding I was able to successfully delete Job using our code. I can quickly create PR for that. Just need a confirmation that this is indeed a bug. cc
+1 apis/autoscaling is similar
I thought we only needed to register it in the external versions (). I'll double check. cc
Registering api.DeleteOptions in batch looks fine. In the future, I think it's more proper to register DeleteOptions and ListOptions as unversioned, like how we treat .
DeleteOptions and ListOptions are expected to be in their group. They have to be in internal version and external version. Most of this code will be fixed in 1.6 (we'll standardize registering the "common" things). , Chao Xu wrote:
I've created PR to fix this issue. Could anyone check that?

&CronJob{}, &CronJobList{}, &api.ListOptions{}, <ins> &api.DeleteOptions{}, </ins> ) scheme.AddKnownTypeWithName(SchemeGroupVersion.WithKind("ScheduledJob"), &CronJob{}) scheme.AddKnownTypeWithName(SchemeGroupVersion.WithKind("ScheduledJobList"), &CronJobList{})


Question:
Docker daemon complains "invalid hostname format" if the truncated host name ends with "." The truncation occurs .
Docker checks hostname with the regexp (See ): It means that the hostname must end with an alnum, not or . This issue has been here for a long time. This usually happens to static pod, because we compose a looooong name for static pod. We didn't hit this just because we are lucky. :P Will send a PR to fix this. /cc
: Needed for v1.5? please add milestone if so
Thanks for reminding. :) This is a bug fix in kubelet. Mark 1.5. /cc
Are you working on a fix?
Yeah, the fix is here

return } <ins> // truncatePodHostnameIfNeeded truncates the pod hostname if it's longer than 63 chars. func truncatePodHostnameIfNeeded(podName, hostname string) (string, error) { // Cap hostname at 63 chars (specification is 64bytes which is 63 chars and the null terminating char). const hostnameMaxLen = 63 if len(hostname) <= hostnameMaxLen { return hostname, nil } truncated := hostname[:hostnameMaxLen] glog.Errorf("hostname for pod:%q was longer than %d. Truncated hostname to :%q", podName, hostnameMaxLen, truncated) // hostname should not end with '-' or '.' truncated = strings.TrimRight(truncated, "-.") if len(truncated) == 0 { // This should never happen. return "", fmt.Errorf("hostname for pod %q was invalid: %q", podName, hostname) } return truncated, nil } </ins> // GeneratePodHostNameAndDomain creates a hostname and domain name for a pod, // given that pod's spec and annotations or returns an error. func (kl *Kubelet) GeneratePodHostNameAndDomain(pod *api.Pod) (string, string, error) { // TODO(vmarmol): Handle better. <del> // Cap hostname at 63 chars (specification is 64bytes which is 63 chars and the null terminating char). </del> clusterDomain := kl.clusterDomain <del> const hostnameMaxLen = 63 </del> podAnnotations := pod.Annotations if podAnnotations == nil { podAnnotations = make(map[string]string)


Question:
Docker daemon complains "invalid hostname format" if the truncated host name ends with "." The truncation occurs .
Docker checks hostname with the regexp (See ): It means that the hostname must end with an alnum, not or . This issue has been here for a long time. This usually happens to static pod, because we compose a looooong name for static pod. We didn't hit this just because we are lucky. :P Will send a PR to fix this. /cc
: Needed for v1.5? please add milestone if so
Thanks for reminding. :) This is a bug fix in kubelet. Mark 1.5. /cc
Are you working on a fix?
Yeah, the fix is here

hostname = hostnameCandidate } } <del> if len(hostname) > hostnameMaxLen { hostname = hostname[:hostnameMaxLen] glog.Errorf("hostname for pod:%q was longer than %d. Truncated hostname to :%q", pod.Name, hostnameMaxLen, hostname) </del> <ins> hostname, err := truncatePodHostnameIfNeeded(pod.Name, hostname) if err != nil { return "", "", err </ins> } hostDomain := ""


Question:
Docker daemon complains "invalid hostname format" if the truncated host name ends with "." The truncation occurs .
Docker checks hostname with the regexp (See ): It means that the hostname must end with an alnum, not or . This issue has been here for a long time. This usually happens to static pod, because we compose a looooong name for static pod. We didn't hit this just because we are lucky. :P Will send a PR to fix this. /cc
: Needed for v1.5? please add milestone if so
Thanks for reminding. :) This is a bug fix in kubelet. Mark 1.5. /cc
Are you working on a fix?
Yeah, the fix is here

} } } <ins> func TestTruncatePodHostname(t *testing.T) { for c, test := range map[string]struct { input  string output string }{ "valid hostname": { input:  "test.pod.hostname", output: "test.pod.hostname", }, "too long hostname": { input:  "1234567.1234567.1234567.1234567.1234567.1234567.1234567.1234567.1234567.", // 8*9=72 chars output: "1234567.1234567.1234567.1234567.1234567.1234567.1234567.1234567",          //8*8-1=63 chars }, "hostname end with .": { input:  "1234567.1234567.1234567.1234567.1234567.1234567.1234567.123456.1234567.", // 8*9-1=71 chars output: "1234567.1234567.1234567.1234567.1234567.1234567.1234567.123456",          //8*8-2=62 chars }, "hostname end with -": { input:  "1234567.1234567.1234567.1234567.1234567.1234567.1234567.123456-1234567.", // 8*9-1=71 chars output: "1234567.1234567.1234567.1234567.1234567.1234567.1234567.123456",          //8*8-2=62 chars }, } { t.Logf("TestCase: %q", c) output, err := truncatePodHostnameIfNeeded("test-pod", test.input) assert.NoError(t, err) assert.Equal(t, test.output, output) } } </ins>


Question:
The error reporting functions here all repeat a URI without sanitizing it. An XSS attacker could use this to get the server to embed a reference to a malicious script. (As always, please note that kubernetes apiserver is NOT xss hardened and we do NOT recommend using the api directly from a browser.)
The error correctly sets the content type to , so this is only an issue using a browser that does content sniffing. Would it be sufficient to just add the header here?
Yes, that will work for browsers that honor the header. It's probably sufficient-- we could do that globally, we set Content-Type correctly everywhere, I think.

// badGatewayError renders a simple bad gateway error. func badGatewayError(w http.ResponseWriter, req *http.Request) { <ins> w.Header().Set("Content-Type", "text/plain") w.Header().Set("X-Content-Type-Options", "nosniff") </ins> w.WriteHeader(http.StatusBadGateway) fmt.Fprintf(w, "Bad Gateway: %#v", req.RequestURI) }


Question:
The error reporting functions here all repeat a URI without sanitizing it. An XSS attacker could use this to get the server to embed a reference to a malicious script. (As always, please note that kubernetes apiserver is NOT xss hardened and we do NOT recommend using the api directly from a browser.)
The error correctly sets the content type to , so this is only an issue using a browser that does content sniffing. Would it be sufficient to just add the header here?
Yes, that will work for browsers that honor the header. It's probably sufficient-- we could do that globally, we set Content-Type correctly everywhere, I think.

// forbidden renders a simple forbidden error func forbidden(attributes authorizer.Attributes, w http.ResponseWriter, req *http.Request, reason string) { msg := forbiddenMessage(attributes) <ins> w.Header().Set("Content-Type", "text/plain") w.Header().Set("X-Content-Type-Options", "nosniff") </ins> w.WriteHeader(http.StatusForbidden) fmt.Fprintf(w, "%s: %q", msg, reason) }


Question:
The error reporting functions here all repeat a URI without sanitizing it. An XSS attacker could use this to get the server to embed a reference to a malicious script. (As always, please note that kubernetes apiserver is NOT xss hardened and we do NOT recommend using the api directly from a browser.)
The error correctly sets the content type to , so this is only an issue using a browser that does content sniffing. Would it be sufficient to just add the header here?
Yes, that will work for browsers that honor the header. It's probably sufficient-- we could do that globally, we set Content-Type correctly everywhere, I think.

// internalError renders a simple internal error func internalError(w http.ResponseWriter, req *http.Request, err error) { <ins> w.Header().Set("Content-Type", "text/plain") w.Header().Set("X-Content-Type-Options", "nosniff") </ins> w.WriteHeader(http.StatusInternalServerError) fmt.Fprintf(w, "Internal Server Error: %#v", req.RequestURI) runtime.HandleError(err)


Question:
Failed: https://k8s- Run so broken it didn't make JUnit output!
Sen what is the plan for migrating these to bootstrap?
isn't it a manual run test?
I guess I'll move it over with empty cron-string?
Why did run this test? Please don't run large and enormous tests - this requires coordination w SREs etcd and may potentially break GCE.
I don't think we've run these test ourselves, someone ran it and failed(10 days ago) thus this test failure occurred.

var ( // TODO: Deprecate gitMajor and gitMinor, use only gitVersion instead. gitMajor     string = "0"              // major version, always numeric <del> gitMinor     string = "4.2+"           // minor version, numeric possibly followed by "+" gitVersion   string = "v0.4.2-dev"     // version from git, output of $(git describe) </del> <ins> gitMinor     string = "4.3+"           // minor version, numeric possibly followed by "+" gitVersion   string = "v0.4.3-dev"     // version from git, output of $(git describe) </ins> gitCommit    string = ""               // sha1 from git, output of $(git rev-parse HEAD) gitTreeState string = "not a git tree" // state of git tree, either "clean" or "dirty" )


Question:
Failed: https://k8s- Run so broken it didn't make JUnit output!
Bootstrap?
Why did run this test? Please don't run large and enormous tests - this requires coordination w SREs etcd and may potentially break GCE.
/close hummm I just see the index is -1


Question:
Kubernetes version (use ): 6 Environment: GCE What happened: I created a cluster for Heapster testing purposes to debug failing test in It turned out that Summary API does not export system/misc stats in SystemContainers section (though there are kubelet and docker stats). Everything works as expected on nodes. See also https://k8s-
What node OS were you using? We don't create the system (misc) container on systemd, so it's expected that GCI nodes don't report those stats.
The default configuration for 1.4, which is: master: GCI nodes: Container VM
Are the nodes running 1.4.6 too? There was a bug with this in 1.4.5, but it should have been fixed in 1.4.6.
Yes, both are running in 1.4.6. If you want to reproduce from Heapster current head run . It will take some time and as a side effect you should have a cluster with the mentioned problem. This was catch by integration tests and I temporary disabled the check here
if you have spare cycles, can you look into this?
friendly ping
Looking at it now
I have found a couple things so far. On the containerVM nodes, cadvisor is exporting a container with "name": "/system" On the GCI Master, cadvisor is exporting a container with "name": "" The has this: "{% set system_container = "--system-cgroups=/system" -%}" (/system is the default misc container name) So the real question is: Should cadvisor export "/system" instead of "" on GCI? Or should we be configuring GCI to use "--system-"?
I thought we fix this very issue for 1.4 branch by a while back. Are you running into the same issue with 1.5?
I think I told you the wrong thing earlier. We don't support the misc (/system) cgroup on systemd systems (I can't remember why, but it was an explicit decision - or might know). The is something different, I believe a base systemd cgroup (slice). As for the misc container on ContainerVM nodes, we've been testing for the misc container in 1.5+ in the summary api test (https://k8s-) for a while, and I haven't seen this issue. So if this is still reproducible, it's limited to 1.4. See this the bug was created for the post-fix.
Found the explanation here: The should use the flag to know the location that has the processes that are associated with , but it should not modify the cgroups of existing processes on the system during bootstrapping of the node. This is because is the on the host and it has not delegated authority to the to change how it manages .
IIUC this is intended behavior on systemd nodes. Based on the explanation provided, the kubelet doesnt have the authority to control "misc" processes on systemd.
+1. "misc" is just a bandaid for cvm. , David Ashpole <:
Ok, so I'm closing this issue.

"//test/e2e_node/services:go_default_library", "//test/e2e_node/system:go_default_library", "//test/utils:go_default_library", <ins> "//vendor:github.com/coreos/go-systemd/util", </ins> "//vendor:github.com/davecgh/go-spew/spew", "//vendor:github.com/golang/glog", "//vendor:github.com/kardianos/osext",


Question:
Kubernetes version (use ): 6 Environment: GCE What happened: I created a cluster for Heapster testing purposes to debug failing test in It turned out that Summary API does not export system/misc stats in SystemContainers section (though there are kubelet and docker stats). Everything works as expected on nodes. See also https://k8s-
What node OS were you using? We don't create the system (misc) container on systemd, so it's expected that GCI nodes don't report those stats.
The default configuration for 1.4, which is: master: GCI nodes: Container VM
Are the nodes running 1.4.6 too? There was a bug with this in 1.4.5, but it should have been fixed in 1.4.6.
Yes, both are running in 1.4.6. If you want to reproduce from Heapster current head run . It will take some time and as a side effect you should have a cluster with the mentioned problem. This was catch by integration tests and I temporary disabled the check here
if you have spare cycles, can you look into this?
friendly ping
Looking at it now
I have found a couple things so far. On the containerVM nodes, cadvisor is exporting a container with "name": "/system" On the GCI Master, cadvisor is exporting a container with "name": "" The has this: "{% set system_container = "--system-cgroups=/system" -%}" (/system is the default misc container name) So the real question is: Should cadvisor export "/system" instead of "" on GCI? Or should we be configuring GCI to use "--system-"?
I thought we fix this very issue for 1.4 branch by a while back. Are you running into the same issue with 1.5?
I think I told you the wrong thing earlier. We don't support the misc (/system) cgroup on systemd systems (I can't remember why, but it was an explicit decision - or might know). The is something different, I believe a base systemd cgroup (slice). As for the misc container on ContainerVM nodes, we've been testing for the misc container in 1.5+ in the summary api test (https://k8s-) for a while, and I haven't seen this issue. So if this is still reproducible, it's limited to 1.4. See this the bug was created for the post-fix.
Found the explanation here: The should use the flag to know the location that has the processes that are associated with , but it should not modify the cgroups of existing processes on the system during bootstrapping of the node. This is because is the on the host and it has not delegated authority to the to change how it manages .
IIUC this is intended behavior on systemd nodes. Based on the explanation provided, the kubelet doesnt have the authority to control "misc" processes on systemd.
+1. "misc" is just a bandaid for cvm. , David Ashpole <:
Ok, so I'm closing this issue.

"k8s.io/kubernetes/pkg/kubelet/api/v1alpha1/stats" "k8s.io/kubernetes/test/e2e/framework" <ins> systemdutil "github.com/coreos/go-systemd/util" </ins> . "github.com/onsi/ginkgo" . "github.com/onsi/gomega" "github.com/onsi/gomega/gstruct"


Question:
Kubernetes version (use ): 6 Environment: GCE What happened: I created a cluster for Heapster testing purposes to debug failing test in It turned out that Summary API does not export system/misc stats in SystemContainers section (though there are kubelet and docker stats). Everything works as expected on nodes. See also https://k8s-
What node OS were you using? We don't create the system (misc) container on systemd, so it's expected that GCI nodes don't report those stats.
The default configuration for 1.4, which is: master: GCI nodes: Container VM
Are the nodes running 1.4.6 too? There was a bug with this in 1.4.5, but it should have been fixed in 1.4.6.
Yes, both are running in 1.4.6. If you want to reproduce from Heapster current head run . It will take some time and as a side effect you should have a cluster with the mentioned problem. This was catch by integration tests and I temporary disabled the check here
if you have spare cycles, can you look into this?
friendly ping
Looking at it now
I have found a couple things so far. On the containerVM nodes, cadvisor is exporting a container with "name": "/system" On the GCI Master, cadvisor is exporting a container with "name": "" The has this: "{% set system_container = "--system-cgroups=/system" -%}" (/system is the default misc container name) So the real question is: Should cadvisor export "/system" instead of "" on GCI? Or should we be configuring GCI to use "--system-"?
I thought we fix this very issue for 1.4 branch by a while back. Are you running into the same issue with 1.5?
I think I told you the wrong thing earlier. We don't support the misc (/system) cgroup on systemd systems (I can't remember why, but it was an explicit decision - or might know). The is something different, I believe a base systemd cgroup (slice). As for the misc container on ContainerVM nodes, we've been testing for the misc container in 1.5+ in the summary api test (https://k8s-) for a while, and I haven't seen this issue. So if this is still reproducible, it's limited to 1.4. See this the bug was created for the post-fix.
Found the explanation here: The should use the flag to know the location that has the processes that are associated with , but it should not modify the cgroups of existing processes on the system during bootstrapping of the node. This is because is the on the host and it has not delegated authority to the to change how it manages .
IIUC this is intended behavior on systemd nodes. Based on the explanation provided, the kubelet doesnt have the authority to control "misc" processes on systemd.
+1. "misc" is just a bandaid for cvm. , David Ashpole <:
Ok, so I'm closing this issue.

"Logs":               BeNil(), "UserDefinedMetrics": BeEmpty(), }) <ins> systemContainers := gstruct.Elements{ "kubelet": sysContExpectations, "runtime": sysContExpectations, } // The Kubelet only manages the 'misc' system container if the host is not running systemd. if !systemdutil.IsRunningSystemd() { framework.Logf("Host not running systemd; expecting 'misc' system container.") systemContainers["misc"] = sysContExpectations } </ins> // Expectations for pods. podExpectations := gstruct.MatchAllFields(gstruct.Fields{ "PodRef":    gstruct.Ignore(),


Question:
Kubernetes version (use ): 6 Environment: GCE What happened: I created a cluster for Heapster testing purposes to debug failing test in It turned out that Summary API does not export system/misc stats in SystemContainers section (though there are kubelet and docker stats). Everything works as expected on nodes. See also https://k8s-
What node OS were you using? We don't create the system (misc) container on systemd, so it's expected that GCI nodes don't report those stats.
The default configuration for 1.4, which is: master: GCI nodes: Container VM
Are the nodes running 1.4.6 too? There was a bug with this in 1.4.5, but it should have been fixed in 1.4.6.
Yes, both are running in 1.4.6. If you want to reproduce from Heapster current head run . It will take some time and as a side effect you should have a cluster with the mentioned problem. This was catch by integration tests and I temporary disabled the check here
if you have spare cycles, can you look into this?
friendly ping
Looking at it now
I have found a couple things so far. On the containerVM nodes, cadvisor is exporting a container with "name": "/system" On the GCI Master, cadvisor is exporting a container with "name": "" The has this: "{% set system_container = "--system-cgroups=/system" -%}" (/system is the default misc container name) So the real question is: Should cadvisor export "/system" instead of "" on GCI? Or should we be configuring GCI to use "--system-"?
I thought we fix this very issue for 1.4 branch by a while back. Are you running into the same issue with 1.5?
I think I told you the wrong thing earlier. We don't support the misc (/system) cgroup on systemd systems (I can't remember why, but it was an explicit decision - or might know). The is something different, I believe a base systemd cgroup (slice). As for the misc container on ContainerVM nodes, we've been testing for the misc container in 1.5+ in the summary api test (https://k8s-) for a while, and I haven't seen this issue. So if this is still reproducible, it's limited to 1.4. See this the bug was created for the post-fix.
Found the explanation here: The should use the flag to know the location that has the processes that are associated with , but it should not modify the cgroups of existing processes on the system during bootstrapping of the node. This is because is the on the host and it has not delegated authority to the to change how it manages .
IIUC this is intended behavior on systemd nodes. Based on the explanation provided, the kubelet doesnt have the authority to control "misc" processes on systemd.
+1. "misc" is just a bandaid for cvm. , David Ashpole <:
Ok, so I'm closing this issue.

}) matchExpectations := ptrMatchAllFields(gstruct.Fields{ "Node": gstruct.MatchAllFields(gstruct.Fields{ <del> "NodeName":  Equal(framework.TestContext.NodeName), "StartTime": recent(maxStartAge), "SystemContainers": gstruct.MatchElements(summaryObjectID, gstruct.IgnoreExtras, gstruct.Elements{ "kubelet": sysContExpectations, "runtime": sysContExpectations, }), </del> <ins> "NodeName":         Equal(framework.TestContext.NodeName), "StartTime":        recent(maxStartAge), "SystemContainers": gstruct.MatchAllElements(summaryObjectID, systemContainers), </ins> "CPU": ptrMatchAllFields(gstruct.Fields{ "Time":                 recent(maxStatsAge), "UsageNanoCores":       bounded(100E3, 2E9),


Question:
https://k8s- Failed: [] Network Partition [Disruptive] [Slow] [] [ReplicationController] should eagerly create replacement pod during network partition when termination grace is non-zero {Kubernetes e2e suite}
https://k8s- seems to be the first failure here. It seems to have gone in with a bunch of RC/RS changes () cc/

// list all pods to include the pods that don't match the rs`s selector // anymore but has the stale controller ref. // TODO: Do the List and Filter in a single pass, or use an index. <del> pods, err := rsc.podLister.Pods(rs.Namespace).List(labels.Everything()) </del> <ins> allPods, err := rsc.podLister.Pods(rs.Namespace).List(labels.Everything()) </ins> if err != nil { return err } // Ignore inactive pods. var filteredPods []*v1.Pod <del> for _, pod := range pods { </del> <ins> for _, pod := range allPods { </ins> if controller.IsPodActive(pod) { filteredPods = append(filteredPods, pod) }


Question:
https://k8s- Failed: [] Network Partition [Disruptive] [Slow] [] [ReplicationController] should eagerly create replacement pod during network partition when termination grace is non-zero {Kubernetes e2e suite}
https://k8s- seems to be the first failure here. It seems to have gone in with a bunch of RC/RS changes () cc/

defer close(stopCh) manager, informers := testNewReplicaSetControllerFromClient(client, stopCh, BurstReplicas) <del> // A controller with 2 replicas and no pods in the store, 2 creates expected </del> <ins> // A controller with 2 replicas and no active pods in the store. // Inactive pods should be ignored. 2 creates expected. </ins> labelMap := map[string]string{"foo": "bar"} rs := newReplicaSet(2, labelMap) informers.Extensions().V1beta1().ReplicaSets().Informer().GetIndexer().Add(rs) <ins> failedPod := newPod("failed-pod", rs, v1.PodFailed, nil, true) deletedPod := newPod("deleted-pod", rs, v1.PodRunning, nil, true) deletedPod.DeletionTimestamp = &metav1.Time{Time: time.Now()} informers.Core().V1().Pods().Informer().GetIndexer().Add(failedPod) informers.Core().V1().Pods().Informer().GetIndexer().Add(deletedPod) </ins> fakePodControl := controller.FakePodControl{} manager.podControl = &fakePodControl


Question:
https://k8s- Failed: [] Network Partition [Disruptive] [Slow] [] [ReplicationController] should eagerly create replacement pod during network partition when termination grace is non-zero {Kubernetes e2e suite}
https://k8s- seems to be the first failure here. It seems to have gone in with a bunch of RC/RS changes () cc/

// list all pods to include the pods that don't match the rc's selector // anymore but has the stale controller ref. // TODO: Do the List and Filter in a single pass, or use an index. <del> pods, err := rm.podLister.Pods(rc.Namespace).List(labels.Everything()) </del> <ins> allPods, err := rm.podLister.Pods(rc.Namespace).List(labels.Everything()) </ins> if err != nil { return err } // Ignore inactive pods. var filteredPods []*v1.Pod <del> for _, pod := range pods { </del> <ins> for _, pod := range allPods { </ins> if controller.IsPodActive(pod) { filteredPods = append(filteredPods, pod) }


Question:
https://k8s- Failed: [] Network Partition [Disruptive] [Slow] [] [ReplicationController] should eagerly create replacement pod during network partition when termination grace is non-zero {Kubernetes e2e suite}
https://k8s- seems to be the first failure here. It seems to have gone in with a bunch of RC/RS changes () cc/

cm := controller.NewPodControllerRefManager(rm.podControl, rc, labels.Set(rc.Spec.Selector).AsSelectorPreValidated(), controllerKind) // NOTE: filteredPods are pointing to objects from cache - if you need to // modify them, you need to copy it first. <del> filteredPods, err = cm.ClaimPods(pods) </del> <ins> filteredPods, err = cm.ClaimPods(filteredPods) </ins> if err != nil { return err }


Question:
https://k8s- Failed: [] Network Partition [Disruptive] [Slow] [] [ReplicationController] should eagerly create replacement pod during network partition when termination grace is non-zero {Kubernetes e2e suite}
https://k8s- seems to be the first failure here. It seems to have gone in with a bunch of RC/RS changes () cc/

func TestSyncReplicationControllerCreates(t *testing.T) { c := clientset.NewForConfigOrDie(&restclient.Config{Host: "", ContentConfig: restclient.ContentConfig{GroupVersion: &api.Registry.GroupOrDie(v1.GroupName).GroupVersion}}) <del> manager, _, rcInformer := newReplicationManagerFromClient(c, BurstReplicas) </del> <ins> manager, podInformer, rcInformer := newReplicationManagerFromClient(c, BurstReplicas) </ins> <del> // A controller with 2 replicas and no pods in the store, 2 creates expected </del> <ins> // A controller with 2 replicas and no active pods in the store. // Inactive pods should be ignored. 2 creates expected. </ins> rc := newReplicationController(2) rcInformer.Informer().GetIndexer().Add(rc) <ins> failedPod := newPod("failed-pod", rc, v1.PodFailed, nil, true) deletedPod := newPod("deleted-pod", rc, v1.PodRunning, nil, true) deletedPod.DeletionTimestamp = &metav1.Time{Time: time.Now()} podInformer.Informer().GetIndexer().Add(failedPod) podInformer.Informer().GetIndexer().Add(deletedPod) </ins> fakePodControl := controller.FakePodControl{} manager.podControl = &fakePodControl


Question:
Is this a BUG REPORT or FEATURE REQUEST? BUG REPORT Environment: Cloud provider or hardware configuration: GCE OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: What happened: $ kubectl describe pods test-ipj93 Name: test-ipj93 Namespace: default Node: kubernetes-minion-group-yxra/10.128.0.3 Start Time: Mon, 28 Nov 2016 14:39:28 -0800 Labels: <noneStatus: Pending IP: Controllers: <noneContainers: busybox: Container ID: Image: busybox Image ID: Port: Command: sh -c while true; do echo (hostname)-(cat /proc/stat cut -f 4 -d" "); sleep 30; done QoS Tier: cpu: Burstable memory: BestEffort Requests: cpu: 100m State: Waiting Reason: ContainerCreating Ready: False Restart Count: 0 Environment Variables: Conditions: Type Status Initialized True Ready False PodScheduled True Volumes: data: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: test ReadOnly: false default-token-kjoqg: Type: Secret (a volume populated by a Secret) SecretName: default-token-kjoqg Events: FirstSeen LastSeen Count From SubobjectPath Type Reason Message --------- -------- ----- ---- ------------- -------- ------ ------- 57m 57m 1 {default-scheduler } Normal Scheduled Successfully assigned test-ipj93 to kubernetes-minion-group-yxra 57m 57m 1 {kubelet kubernetes-minion-group-yxra} spec.containers{busybox} Normal Pulling pulling image "busybox" 57m 57m 1 {kubelet kubernetes-minion-group-yxra} spec.containers{busybox} Normal Pulled Successfully pulled image "busybox" 57m 57m 1 {kubelet kubernetes-minion-group-yxra} spec.containers{busybox} Normal Created Created container with docker id ; Security:[seccomp=unconfined] 57m 57m 1 {kubelet kubernetes-minion-group-yxra} spec.containers{busybox} Normal Started Started container with docker id 47m 47m 6 {kubelet kubernetes-minion-group-yxra} Warning FailedMount MountVolume.SetUp failed for volume "" (spec.Name: "default-token-kjoqg") pod "-b5bb-11e6-bab0-" (UID: "-b5bb-11e6-bab0-") with: Get https://kubernetes-master/api/v1/namespaces/default/secrets/default-token-kjoqg: dial tcp: lookup kubernetes-master on 169.254.169.254:53: no such host 46m 46m 1 {kubelet kubernetes-minion-group-yxra} Warning FailedMount MountVolume.SetUp failed for volume "" (spec.Name: "default-token-kjoqg") pod "-b5bb-11e6-bab0-" (UID: "-b5bb-11e6-bab0-") with: Get https://kubernetes-master/api/v1/namespaces/default/secrets/default-token-kjoqg: dial tcp 10.128.0.2:443: i/o timeout 45m 45m 1 {kubelet kubernetes-minion-group-yxra} Warning FailedMount MountVolume.SetUp failed for volume "" (spec.Name: "default-token-kjoqg") pod "-b5bb-11e6-bab0-" (UID: "-b5bb-11e6-bab0-") with: Get https://kubernetes-master/api/v1/namespaces/default/secrets/default-token-kjoqg: dial tcp 10.128.0.2:443: getsockopt: connection refused 11m 11m 2 {controller-manager } Warning FailedMount Failed to attach volume "test" on node "kubernetes-minion-group-yxra" with: error getting instance "kubernetes-minion-group-yxra" 9m 9m 1 {kubelet kubernetes-minion-group-yxra} Warning FailedSync Error syncing pod, skipping: network is not ready: [Kubenet does not have netConfig. This is most likely due to lack of PodCIDR] 9m 9m 2 {kubelet kubernetes-minion-group-yxra} Warning FailedSync Error syncing pod, skipping: network is not ready: [Kubenet does not have netConfig. This is most likely due to lack of PodCIDR] 6m 14s 4 {kubelet kubernetes-minion-group-yxra} Warning FailedMount Unable to mount volumes for pod "test-ipj93_default(-b5bb-11e6-bab0-)": timeout expired waiting for volumes to attach/mount for pod "test-ipj93"/"default". list of unattached/unmounted volumes=[data] 6m 14s 4 {kubelet kubernetes-minion-group-yxra} Warning FailedSync Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod "test-ipj93"/"default". list of unattached/unmounted volumes=[data] What you expected to happen: GCE PD to attach and mount successfully post the upgrade How to reproduce it (as minimally and precisely as possible): Running the upgrade steps in the doc for upgrade of manual test from 1.4 to 1.5 Anything else do we need to know:
Ran the upgrade again. This time the .
kube-controller-
Current theory is because kubelet deletes and recreates the node API object during upgrade (because externalID changed), see the new object does not contain the correct status (it is written by the Attach/Detach controller to the old object but not subseqently updated). If this proves to be the issue, the proposed fix is to force the attach/detach controller to update the node status when the node's external ID changes. is working to verify this theory by executing this code path (without a full upgrade). Once he has validated the theory, he will work on the fix, verify it, merge it to master, and get it cherry-picked to 1.5.
: is expected to fix this problem?
: Yes

func (adc *attachDetachController) nodeAdd(obj interface{}) { node, ok := obj.(*v1.Node) <ins> // TODO: investigate if nodeName is empty then if we can return // kubernetes/kubernetes/issues/37777 if node == nil || !ok { return } nodeName := types.NodeName(node.Name) adc.nodeUpdate(nil, obj) // kubernetes/kubernetes/issues/37586 // This is to workaround the case when a node add causes to wipe out // the attached volumes field. This function ensures that we sync with // the actual status. adc.actualStateOfWorld.SetNodeStatusUpdateNeeded(nodeName) } func (adc *attachDetachController) nodeUpdate(oldObj, newObj interface{}) { node, ok := newObj.(*v1.Node) // TODO: investigate if nodeName is empty then if we can return </ins> if node == nil || !ok { return }


Question:
Is this a BUG REPORT or FEATURE REQUEST? BUG REPORT Environment: Cloud provider or hardware configuration: GCE OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: What happened: $ kubectl describe pods test-ipj93 Name: test-ipj93 Namespace: default Node: kubernetes-minion-group-yxra/10.128.0.3 Start Time: Mon, 28 Nov 2016 14:39:28 -0800 Labels: <noneStatus: Pending IP: Controllers: <noneContainers: busybox: Container ID: Image: busybox Image ID: Port: Command: sh -c while true; do echo (hostname)-(cat /proc/stat cut -f 4 -d" "); sleep 30; done QoS Tier: cpu: Burstable memory: BestEffort Requests: cpu: 100m State: Waiting Reason: ContainerCreating Ready: False Restart Count: 0 Environment Variables: Conditions: Type Status Initialized True Ready False PodScheduled True Volumes: data: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: test ReadOnly: false default-token-kjoqg: Type: Secret (a volume populated by a Secret) SecretName: default-token-kjoqg Events: FirstSeen LastSeen Count From SubobjectPath Type Reason Message --------- -------- ----- ---- ------------- -------- ------ ------- 57m 57m 1 {default-scheduler } Normal Scheduled Successfully assigned test-ipj93 to kubernetes-minion-group-yxra 57m 57m 1 {kubelet kubernetes-minion-group-yxra} spec.containers{busybox} Normal Pulling pulling image "busybox" 57m 57m 1 {kubelet kubernetes-minion-group-yxra} spec.containers{busybox} Normal Pulled Successfully pulled image "busybox" 57m 57m 1 {kubelet kubernetes-minion-group-yxra} spec.containers{busybox} Normal Created Created container with docker id ; Security:[seccomp=unconfined] 57m 57m 1 {kubelet kubernetes-minion-group-yxra} spec.containers{busybox} Normal Started Started container with docker id 47m 47m 6 {kubelet kubernetes-minion-group-yxra} Warning FailedMount MountVolume.SetUp failed for volume "" (spec.Name: "default-token-kjoqg") pod "-b5bb-11e6-bab0-" (UID: "-b5bb-11e6-bab0-") with: Get https://kubernetes-master/api/v1/namespaces/default/secrets/default-token-kjoqg: dial tcp: lookup kubernetes-master on 169.254.169.254:53: no such host 46m 46m 1 {kubelet kubernetes-minion-group-yxra} Warning FailedMount MountVolume.SetUp failed for volume "" (spec.Name: "default-token-kjoqg") pod "-b5bb-11e6-bab0-" (UID: "-b5bb-11e6-bab0-") with: Get https://kubernetes-master/api/v1/namespaces/default/secrets/default-token-kjoqg: dial tcp 10.128.0.2:443: i/o timeout 45m 45m 1 {kubelet kubernetes-minion-group-yxra} Warning FailedMount MountVolume.SetUp failed for volume "" (spec.Name: "default-token-kjoqg") pod "-b5bb-11e6-bab0-" (UID: "-b5bb-11e6-bab0-") with: Get https://kubernetes-master/api/v1/namespaces/default/secrets/default-token-kjoqg: dial tcp 10.128.0.2:443: getsockopt: connection refused 11m 11m 2 {controller-manager } Warning FailedMount Failed to attach volume "test" on node "kubernetes-minion-group-yxra" with: error getting instance "kubernetes-minion-group-yxra" 9m 9m 1 {kubelet kubernetes-minion-group-yxra} Warning FailedSync Error syncing pod, skipping: network is not ready: [Kubenet does not have netConfig. This is most likely due to lack of PodCIDR] 9m 9m 2 {kubelet kubernetes-minion-group-yxra} Warning FailedSync Error syncing pod, skipping: network is not ready: [Kubenet does not have netConfig. This is most likely due to lack of PodCIDR] 6m 14s 4 {kubelet kubernetes-minion-group-yxra} Warning FailedMount Unable to mount volumes for pod "test-ipj93_default(-b5bb-11e6-bab0-)": timeout expired waiting for volumes to attach/mount for pod "test-ipj93"/"default". list of unattached/unmounted volumes=[data] 6m 14s 4 {kubelet kubernetes-minion-group-yxra} Warning FailedSync Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod "test-ipj93"/"default". list of unattached/unmounted volumes=[data] What you expected to happen: GCE PD to attach and mount successfully post the upgrade How to reproduce it (as minimally and precisely as possible): Running the upgrade steps in the doc for upgrade of manual test from 1.4 to 1.5 Anything else do we need to know:
Ran the upgrade again. This time the .
kube-controller-
Current theory is because kubelet deletes and recreates the node API object during upgrade (because externalID changed), see the new object does not contain the correct status (it is written by the Attach/Detach controller to the old object but not subseqently updated). If this proves to be the issue, the proposed fix is to force the attach/detach controller to update the node status when the node's external ID changes. is working to verify this theory by executing this code path (without a full upgrade). Once he has validated the theory, he will work on the fix, verify it, merge it to master, and get it cherry-picked to 1.5.
: is expected to fix this problem?
: Yes

// detach controller. Add it to desired state of world. adc.desiredStateOfWorld.AddNode(nodeName) } adc.processVolumesInUse(nodeName, node.Status.VolumesInUse) } <del> func (adc *attachDetachController) nodeUpdate(oldObj, newObj interface{}) { // The flow for update is the same as add. adc.nodeAdd(newObj) } </del> func (adc *attachDetachController) nodeDelete(obj interface{}) { node, ok := obj.(*v1.Node) if node == nil || !ok {


Question:
Is this a BUG REPORT or FEATURE REQUEST? BUG REPORT Environment: Cloud provider or hardware configuration: GCE OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: What happened: $ kubectl describe pods test-ipj93 Name: test-ipj93 Namespace: default Node: kubernetes-minion-group-yxra/10.128.0.3 Start Time: Mon, 28 Nov 2016 14:39:28 -0800 Labels: <noneStatus: Pending IP: Controllers: <noneContainers: busybox: Container ID: Image: busybox Image ID: Port: Command: sh -c while true; do echo (hostname)-(cat /proc/stat cut -f 4 -d" "); sleep 30; done QoS Tier: cpu: Burstable memory: BestEffort Requests: cpu: 100m State: Waiting Reason: ContainerCreating Ready: False Restart Count: 0 Environment Variables: Conditions: Type Status Initialized True Ready False PodScheduled True Volumes: data: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: test ReadOnly: false default-token-kjoqg: Type: Secret (a volume populated by a Secret) SecretName: default-token-kjoqg Events: FirstSeen LastSeen Count From SubobjectPath Type Reason Message --------- -------- ----- ---- ------------- -------- ------ ------- 57m 57m 1 {default-scheduler } Normal Scheduled Successfully assigned test-ipj93 to kubernetes-minion-group-yxra 57m 57m 1 {kubelet kubernetes-minion-group-yxra} spec.containers{busybox} Normal Pulling pulling image "busybox" 57m 57m 1 {kubelet kubernetes-minion-group-yxra} spec.containers{busybox} Normal Pulled Successfully pulled image "busybox" 57m 57m 1 {kubelet kubernetes-minion-group-yxra} spec.containers{busybox} Normal Created Created container with docker id ; Security:[seccomp=unconfined] 57m 57m 1 {kubelet kubernetes-minion-group-yxra} spec.containers{busybox} Normal Started Started container with docker id 47m 47m 6 {kubelet kubernetes-minion-group-yxra} Warning FailedMount MountVolume.SetUp failed for volume "" (spec.Name: "default-token-kjoqg") pod "-b5bb-11e6-bab0-" (UID: "-b5bb-11e6-bab0-") with: Get https://kubernetes-master/api/v1/namespaces/default/secrets/default-token-kjoqg: dial tcp: lookup kubernetes-master on 169.254.169.254:53: no such host 46m 46m 1 {kubelet kubernetes-minion-group-yxra} Warning FailedMount MountVolume.SetUp failed for volume "" (spec.Name: "default-token-kjoqg") pod "-b5bb-11e6-bab0-" (UID: "-b5bb-11e6-bab0-") with: Get https://kubernetes-master/api/v1/namespaces/default/secrets/default-token-kjoqg: dial tcp 10.128.0.2:443: i/o timeout 45m 45m 1 {kubelet kubernetes-minion-group-yxra} Warning FailedMount MountVolume.SetUp failed for volume "" (spec.Name: "default-token-kjoqg") pod "-b5bb-11e6-bab0-" (UID: "-b5bb-11e6-bab0-") with: Get https://kubernetes-master/api/v1/namespaces/default/secrets/default-token-kjoqg: dial tcp 10.128.0.2:443: getsockopt: connection refused 11m 11m 2 {controller-manager } Warning FailedMount Failed to attach volume "test" on node "kubernetes-minion-group-yxra" with: error getting instance "kubernetes-minion-group-yxra" 9m 9m 1 {kubelet kubernetes-minion-group-yxra} Warning FailedSync Error syncing pod, skipping: network is not ready: [Kubenet does not have netConfig. This is most likely due to lack of PodCIDR] 9m 9m 2 {kubelet kubernetes-minion-group-yxra} Warning FailedSync Error syncing pod, skipping: network is not ready: [Kubenet does not have netConfig. This is most likely due to lack of PodCIDR] 6m 14s 4 {kubelet kubernetes-minion-group-yxra} Warning FailedMount Unable to mount volumes for pod "test-ipj93_default(-b5bb-11e6-bab0-)": timeout expired waiting for volumes to attach/mount for pod "test-ipj93"/"default". list of unattached/unmounted volumes=[data] 6m 14s 4 {kubelet kubernetes-minion-group-yxra} Warning FailedSync Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod "test-ipj93"/"default". list of unattached/unmounted volumes=[data] What you expected to happen: GCE PD to attach and mount successfully post the upgrade How to reproduce it (as minimally and precisely as possible): Running the upgrade steps in the doc for upgrade of manual test from 1.4 to 1.5 Anything else do we need to know:
Ran the upgrade again. This time the .
kube-controller-
Current theory is because kubelet deletes and recreates the node API object during upgrade (because externalID changed), see the new object does not contain the correct status (it is written by the Attach/Detach controller to the old object but not subseqently updated). If this proves to be the issue, the proposed fix is to force the attach/detach controller to update the node status when the node's external ID changes. is working to verify this theory by executing this code path (without a full upgrade). Once he has validated the theory, he will work on the fix, verify it, merge it to master, and get it cherry-picked to 1.5.
: is expected to fix this problem?
: Yes

// since volumes should be removed from this list as soon a detach operation // is considered, before the detach operation is triggered). GetVolumesToReportAttached() map[types.NodeName][]v1.AttachedVolume <ins> // GetNodesToUpdateStatusFor returns the map of nodeNames to nodeToUpdateStatusFor GetNodesToUpdateStatusFor() map[types.NodeName]nodeToUpdateStatusFor </ins> } // AttachedVolume represents a volume that is attached to a node.


Question:
Is this a BUG REPORT or FEATURE REQUEST? BUG REPORT Environment: Cloud provider or hardware configuration: GCE OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: What happened: $ kubectl describe pods test-ipj93 Name: test-ipj93 Namespace: default Node: kubernetes-minion-group-yxra/10.128.0.3 Start Time: Mon, 28 Nov 2016 14:39:28 -0800 Labels: <noneStatus: Pending IP: Controllers: <noneContainers: busybox: Container ID: Image: busybox Image ID: Port: Command: sh -c while true; do echo (hostname)-(cat /proc/stat cut -f 4 -d" "); sleep 30; done QoS Tier: cpu: Burstable memory: BestEffort Requests: cpu: 100m State: Waiting Reason: ContainerCreating Ready: False Restart Count: 0 Environment Variables: Conditions: Type Status Initialized True Ready False PodScheduled True Volumes: data: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: test ReadOnly: false default-token-kjoqg: Type: Secret (a volume populated by a Secret) SecretName: default-token-kjoqg Events: FirstSeen LastSeen Count From SubobjectPath Type Reason Message --------- -------- ----- ---- ------------- -------- ------ ------- 57m 57m 1 {default-scheduler } Normal Scheduled Successfully assigned test-ipj93 to kubernetes-minion-group-yxra 57m 57m 1 {kubelet kubernetes-minion-group-yxra} spec.containers{busybox} Normal Pulling pulling image "busybox" 57m 57m 1 {kubelet kubernetes-minion-group-yxra} spec.containers{busybox} Normal Pulled Successfully pulled image "busybox" 57m 57m 1 {kubelet kubernetes-minion-group-yxra} spec.containers{busybox} Normal Created Created container with docker id ; Security:[seccomp=unconfined] 57m 57m 1 {kubelet kubernetes-minion-group-yxra} spec.containers{busybox} Normal Started Started container with docker id 47m 47m 6 {kubelet kubernetes-minion-group-yxra} Warning FailedMount MountVolume.SetUp failed for volume "" (spec.Name: "default-token-kjoqg") pod "-b5bb-11e6-bab0-" (UID: "-b5bb-11e6-bab0-") with: Get https://kubernetes-master/api/v1/namespaces/default/secrets/default-token-kjoqg: dial tcp: lookup kubernetes-master on 169.254.169.254:53: no such host 46m 46m 1 {kubelet kubernetes-minion-group-yxra} Warning FailedMount MountVolume.SetUp failed for volume "" (spec.Name: "default-token-kjoqg") pod "-b5bb-11e6-bab0-" (UID: "-b5bb-11e6-bab0-") with: Get https://kubernetes-master/api/v1/namespaces/default/secrets/default-token-kjoqg: dial tcp 10.128.0.2:443: i/o timeout 45m 45m 1 {kubelet kubernetes-minion-group-yxra} Warning FailedMount MountVolume.SetUp failed for volume "" (spec.Name: "default-token-kjoqg") pod "-b5bb-11e6-bab0-" (UID: "-b5bb-11e6-bab0-") with: Get https://kubernetes-master/api/v1/namespaces/default/secrets/default-token-kjoqg: dial tcp 10.128.0.2:443: getsockopt: connection refused 11m 11m 2 {controller-manager } Warning FailedMount Failed to attach volume "test" on node "kubernetes-minion-group-yxra" with: error getting instance "kubernetes-minion-group-yxra" 9m 9m 1 {kubelet kubernetes-minion-group-yxra} Warning FailedSync Error syncing pod, skipping: network is not ready: [Kubenet does not have netConfig. This is most likely due to lack of PodCIDR] 9m 9m 2 {kubelet kubernetes-minion-group-yxra} Warning FailedSync Error syncing pod, skipping: network is not ready: [Kubenet does not have netConfig. This is most likely due to lack of PodCIDR] 6m 14s 4 {kubelet kubernetes-minion-group-yxra} Warning FailedMount Unable to mount volumes for pod "test-ipj93_default(-b5bb-11e6-bab0-)": timeout expired waiting for volumes to attach/mount for pod "test-ipj93"/"default". list of unattached/unmounted volumes=[data] 6m 14s 4 {kubelet kubernetes-minion-group-yxra} Warning FailedSync Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod "test-ipj93"/"default". list of unattached/unmounted volumes=[data] What you expected to happen: GCE PD to attach and mount successfully post the upgrade How to reproduce it (as minimally and precisely as possible): Running the upgrade steps in the doc for upgrade of manual test from 1.4 to 1.5 Anything else do we need to know:
Ran the upgrade again. This time the .
kube-controller-
Current theory is because kubelet deletes and recreates the node API object during upgrade (because externalID changed), see the new object does not contain the correct status (it is written by the Attach/Detach controller to the old object but not subseqently updated). If this proves to be the issue, the proposed fix is to force the attach/detach controller to update the node status when the node's external ID changes. is working to verify this theory by executing this code path (without a full upgrade). Once he has validated the theory, he will work on the fix, verify it, merge it to master, and get it cherry-picked to 1.5.
: is expected to fix this problem?
: Yes

"Failed to set statusUpdateNeeded to needed %t because nodeName=%q  does not exist", needed, nodeName) <ins> return </ins> } nodeToUpdate.statusUpdateNeeded = needed


Question:
Is this a BUG REPORT or FEATURE REQUEST? BUG REPORT Environment: Cloud provider or hardware configuration: GCE OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: What happened: $ kubectl describe pods test-ipj93 Name: test-ipj93 Namespace: default Node: kubernetes-minion-group-yxra/10.128.0.3 Start Time: Mon, 28 Nov 2016 14:39:28 -0800 Labels: <noneStatus: Pending IP: Controllers: <noneContainers: busybox: Container ID: Image: busybox Image ID: Port: Command: sh -c while true; do echo (hostname)-(cat /proc/stat cut -f 4 -d" "); sleep 30; done QoS Tier: cpu: Burstable memory: BestEffort Requests: cpu: 100m State: Waiting Reason: ContainerCreating Ready: False Restart Count: 0 Environment Variables: Conditions: Type Status Initialized True Ready False PodScheduled True Volumes: data: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: test ReadOnly: false default-token-kjoqg: Type: Secret (a volume populated by a Secret) SecretName: default-token-kjoqg Events: FirstSeen LastSeen Count From SubobjectPath Type Reason Message --------- -------- ----- ---- ------------- -------- ------ ------- 57m 57m 1 {default-scheduler } Normal Scheduled Successfully assigned test-ipj93 to kubernetes-minion-group-yxra 57m 57m 1 {kubelet kubernetes-minion-group-yxra} spec.containers{busybox} Normal Pulling pulling image "busybox" 57m 57m 1 {kubelet kubernetes-minion-group-yxra} spec.containers{busybox} Normal Pulled Successfully pulled image "busybox" 57m 57m 1 {kubelet kubernetes-minion-group-yxra} spec.containers{busybox} Normal Created Created container with docker id ; Security:[seccomp=unconfined] 57m 57m 1 {kubelet kubernetes-minion-group-yxra} spec.containers{busybox} Normal Started Started container with docker id 47m 47m 6 {kubelet kubernetes-minion-group-yxra} Warning FailedMount MountVolume.SetUp failed for volume "" (spec.Name: "default-token-kjoqg") pod "-b5bb-11e6-bab0-" (UID: "-b5bb-11e6-bab0-") with: Get https://kubernetes-master/api/v1/namespaces/default/secrets/default-token-kjoqg: dial tcp: lookup kubernetes-master on 169.254.169.254:53: no such host 46m 46m 1 {kubelet kubernetes-minion-group-yxra} Warning FailedMount MountVolume.SetUp failed for volume "" (spec.Name: "default-token-kjoqg") pod "-b5bb-11e6-bab0-" (UID: "-b5bb-11e6-bab0-") with: Get https://kubernetes-master/api/v1/namespaces/default/secrets/default-token-kjoqg: dial tcp 10.128.0.2:443: i/o timeout 45m 45m 1 {kubelet kubernetes-minion-group-yxra} Warning FailedMount MountVolume.SetUp failed for volume "" (spec.Name: "default-token-kjoqg") pod "-b5bb-11e6-bab0-" (UID: "-b5bb-11e6-bab0-") with: Get https://kubernetes-master/api/v1/namespaces/default/secrets/default-token-kjoqg: dial tcp 10.128.0.2:443: getsockopt: connection refused 11m 11m 2 {controller-manager } Warning FailedMount Failed to attach volume "test" on node "kubernetes-minion-group-yxra" with: error getting instance "kubernetes-minion-group-yxra" 9m 9m 1 {kubelet kubernetes-minion-group-yxra} Warning FailedSync Error syncing pod, skipping: network is not ready: [Kubenet does not have netConfig. This is most likely due to lack of PodCIDR] 9m 9m 2 {kubelet kubernetes-minion-group-yxra} Warning FailedSync Error syncing pod, skipping: network is not ready: [Kubenet does not have netConfig. This is most likely due to lack of PodCIDR] 6m 14s 4 {kubelet kubernetes-minion-group-yxra} Warning FailedMount Unable to mount volumes for pod "test-ipj93_default(-b5bb-11e6-bab0-)": timeout expired waiting for volumes to attach/mount for pod "test-ipj93"/"default". list of unattached/unmounted volumes=[data] 6m 14s 4 {kubelet kubernetes-minion-group-yxra} Warning FailedSync Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod "test-ipj93"/"default". list of unattached/unmounted volumes=[data] What you expected to happen: GCE PD to attach and mount successfully post the upgrade How to reproduce it (as minimally and precisely as possible): Running the upgrade steps in the doc for upgrade of manual test from 1.4 to 1.5 Anything else do we need to know:
Ran the upgrade again. This time the .
kube-controller-
Current theory is because kubelet deletes and recreates the node API object during upgrade (because externalID changed), see the new object does not contain the correct status (it is written by the Attach/Detach controller to the old object but not subseqently updated). If this proves to be the issue, the proposed fix is to force the attach/detach controller to update the node status when the node's external ID changes. is working to verify this theory by executing this code path (without a full upgrade). Once he has validated the theory, he will work on the fix, verify it, merge it to master, and get it cherry-picked to 1.5.
: is expected to fix this problem?
: Yes

return volumesToReportAttached } <ins> func (asw *actualStateOfWorld) GetNodesToUpdateStatusFor() map[types.NodeName]nodeToUpdateStatusFor { return asw.nodesToUpdateStatusFor } </ins> func getAttachedVolume( attachedVolume *attachedVolume, nodeAttachedTo *nodeAttachedTo) AttachedVolume {


Question:
Is this a BUG REPORT or FEATURE REQUEST? BUG REPORT Environment: Cloud provider or hardware configuration: GCE OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: What happened: $ kubectl describe pods test-ipj93 Name: test-ipj93 Namespace: default Node: kubernetes-minion-group-yxra/10.128.0.3 Start Time: Mon, 28 Nov 2016 14:39:28 -0800 Labels: <noneStatus: Pending IP: Controllers: <noneContainers: busybox: Container ID: Image: busybox Image ID: Port: Command: sh -c while true; do echo (hostname)-(cat /proc/stat cut -f 4 -d" "); sleep 30; done QoS Tier: cpu: Burstable memory: BestEffort Requests: cpu: 100m State: Waiting Reason: ContainerCreating Ready: False Restart Count: 0 Environment Variables: Conditions: Type Status Initialized True Ready False PodScheduled True Volumes: data: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: test ReadOnly: false default-token-kjoqg: Type: Secret (a volume populated by a Secret) SecretName: default-token-kjoqg Events: FirstSeen LastSeen Count From SubobjectPath Type Reason Message --------- -------- ----- ---- ------------- -------- ------ ------- 57m 57m 1 {default-scheduler } Normal Scheduled Successfully assigned test-ipj93 to kubernetes-minion-group-yxra 57m 57m 1 {kubelet kubernetes-minion-group-yxra} spec.containers{busybox} Normal Pulling pulling image "busybox" 57m 57m 1 {kubelet kubernetes-minion-group-yxra} spec.containers{busybox} Normal Pulled Successfully pulled image "busybox" 57m 57m 1 {kubelet kubernetes-minion-group-yxra} spec.containers{busybox} Normal Created Created container with docker id ; Security:[seccomp=unconfined] 57m 57m 1 {kubelet kubernetes-minion-group-yxra} spec.containers{busybox} Normal Started Started container with docker id 47m 47m 6 {kubelet kubernetes-minion-group-yxra} Warning FailedMount MountVolume.SetUp failed for volume "" (spec.Name: "default-token-kjoqg") pod "-b5bb-11e6-bab0-" (UID: "-b5bb-11e6-bab0-") with: Get https://kubernetes-master/api/v1/namespaces/default/secrets/default-token-kjoqg: dial tcp: lookup kubernetes-master on 169.254.169.254:53: no such host 46m 46m 1 {kubelet kubernetes-minion-group-yxra} Warning FailedMount MountVolume.SetUp failed for volume "" (spec.Name: "default-token-kjoqg") pod "-b5bb-11e6-bab0-" (UID: "-b5bb-11e6-bab0-") with: Get https://kubernetes-master/api/v1/namespaces/default/secrets/default-token-kjoqg: dial tcp 10.128.0.2:443: i/o timeout 45m 45m 1 {kubelet kubernetes-minion-group-yxra} Warning FailedMount MountVolume.SetUp failed for volume "" (spec.Name: "default-token-kjoqg") pod "-b5bb-11e6-bab0-" (UID: "-b5bb-11e6-bab0-") with: Get https://kubernetes-master/api/v1/namespaces/default/secrets/default-token-kjoqg: dial tcp 10.128.0.2:443: getsockopt: connection refused 11m 11m 2 {controller-manager } Warning FailedMount Failed to attach volume "test" on node "kubernetes-minion-group-yxra" with: error getting instance "kubernetes-minion-group-yxra" 9m 9m 1 {kubelet kubernetes-minion-group-yxra} Warning FailedSync Error syncing pod, skipping: network is not ready: [Kubenet does not have netConfig. This is most likely due to lack of PodCIDR] 9m 9m 2 {kubelet kubernetes-minion-group-yxra} Warning FailedSync Error syncing pod, skipping: network is not ready: [Kubenet does not have netConfig. This is most likely due to lack of PodCIDR] 6m 14s 4 {kubelet kubernetes-minion-group-yxra} Warning FailedMount Unable to mount volumes for pod "test-ipj93_default(-b5bb-11e6-bab0-)": timeout expired waiting for volumes to attach/mount for pod "test-ipj93"/"default". list of unattached/unmounted volumes=[data] 6m 14s 4 {kubelet kubernetes-minion-group-yxra} Warning FailedSync Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod "test-ipj93"/"default". list of unattached/unmounted volumes=[data] What you expected to happen: GCE PD to attach and mount successfully post the upgrade How to reproduce it (as minimally and precisely as possible): Running the upgrade steps in the doc for upgrade of manual test from 1.4 to 1.5 Anything else do we need to know:
Ran the upgrade again. This time the .
kube-controller-
Current theory is because kubelet deletes and recreates the node API object during upgrade (because externalID changed), see the new object does not contain the correct status (it is written by the Attach/Detach controller to the old object but not subseqently updated). If this proves to be the issue, the proposed fix is to force the attach/detach controller to update the node status when the node's external ID changes. is working to verify this theory by executing this code path (without a full upgrade). Once he has validated the theory, he will work on the fix, verify it, merge it to master, and get it cherry-picked to 1.5.
: is expected to fix this problem?
: Yes

verifyAttachedVolume(t, attachedVolumes, generatedVolumeName2, string(volumeName), node2Name, devicePath2, true /* expectedMountedByNode */, false /* expectNonZeroDetachRequestedTime */) } <ins> // Test_SetNodeStatusUpdateNeededError expects the map nodesToUpdateStatusFor // to be empty if the SetNodeStatusUpdateNeeded is called on a node that // does not exist in the actual state of the world func Test_SetNodeStatusUpdateNeededError(t *testing.T) { // Arrange volumePluginMgr, _ := volumetesting.GetTestVolumePluginMgr(t) asw := NewActualStateOfWorld(volumePluginMgr) nodeName := types.NodeName("node-1") // Act asw.SetNodeStatusUpdateNeeded(nodeName) // Assert nodesToUpdateStatusFor := asw.GetNodesToUpdateStatusFor() if len(nodesToUpdateStatusFor) != 0 { t.Fatalf("nodesToUpdateStatusFor should be empty as nodeName does not exist") } } </ins> func verifyAttachedVolume( t *testing.T, attachedVolumes []AttachedVolume,


Question:
Is this a BUG REPORT or FEATURE REQUEST? BUG REPORT Environment: Cloud provider or hardware configuration: GCE OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: What happened: $ kubectl describe pods test-ipj93 Name: test-ipj93 Namespace: default Node: kubernetes-minion-group-yxra/10.128.0.3 Start Time: Mon, 28 Nov 2016 14:39:28 -0800 Labels: <noneStatus: Pending IP: Controllers: <noneContainers: busybox: Container ID: Image: busybox Image ID: Port: Command: sh -c while true; do echo (hostname)-(cat /proc/stat cut -f 4 -d" "); sleep 30; done QoS Tier: cpu: Burstable memory: BestEffort Requests: cpu: 100m State: Waiting Reason: ContainerCreating Ready: False Restart Count: 0 Environment Variables: Conditions: Type Status Initialized True Ready False PodScheduled True Volumes: data: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: test ReadOnly: false default-token-kjoqg: Type: Secret (a volume populated by a Secret) SecretName: default-token-kjoqg Events: FirstSeen LastSeen Count From SubobjectPath Type Reason Message --------- -------- ----- ---- ------------- -------- ------ ------- 57m 57m 1 {default-scheduler } Normal Scheduled Successfully assigned test-ipj93 to kubernetes-minion-group-yxra 57m 57m 1 {kubelet kubernetes-minion-group-yxra} spec.containers{busybox} Normal Pulling pulling image "busybox" 57m 57m 1 {kubelet kubernetes-minion-group-yxra} spec.containers{busybox} Normal Pulled Successfully pulled image "busybox" 57m 57m 1 {kubelet kubernetes-minion-group-yxra} spec.containers{busybox} Normal Created Created container with docker id ; Security:[seccomp=unconfined] 57m 57m 1 {kubelet kubernetes-minion-group-yxra} spec.containers{busybox} Normal Started Started container with docker id 47m 47m 6 {kubelet kubernetes-minion-group-yxra} Warning FailedMount MountVolume.SetUp failed for volume "" (spec.Name: "default-token-kjoqg") pod "-b5bb-11e6-bab0-" (UID: "-b5bb-11e6-bab0-") with: Get https://kubernetes-master/api/v1/namespaces/default/secrets/default-token-kjoqg: dial tcp: lookup kubernetes-master on 169.254.169.254:53: no such host 46m 46m 1 {kubelet kubernetes-minion-group-yxra} Warning FailedMount MountVolume.SetUp failed for volume "" (spec.Name: "default-token-kjoqg") pod "-b5bb-11e6-bab0-" (UID: "-b5bb-11e6-bab0-") with: Get https://kubernetes-master/api/v1/namespaces/default/secrets/default-token-kjoqg: dial tcp 10.128.0.2:443: i/o timeout 45m 45m 1 {kubelet kubernetes-minion-group-yxra} Warning FailedMount MountVolume.SetUp failed for volume "" (spec.Name: "default-token-kjoqg") pod "-b5bb-11e6-bab0-" (UID: "-b5bb-11e6-bab0-") with: Get https://kubernetes-master/api/v1/namespaces/default/secrets/default-token-kjoqg: dial tcp 10.128.0.2:443: getsockopt: connection refused 11m 11m 2 {controller-manager } Warning FailedMount Failed to attach volume "test" on node "kubernetes-minion-group-yxra" with: error getting instance "kubernetes-minion-group-yxra" 9m 9m 1 {kubelet kubernetes-minion-group-yxra} Warning FailedSync Error syncing pod, skipping: network is not ready: [Kubenet does not have netConfig. This is most likely due to lack of PodCIDR] 9m 9m 2 {kubelet kubernetes-minion-group-yxra} Warning FailedSync Error syncing pod, skipping: network is not ready: [Kubenet does not have netConfig. This is most likely due to lack of PodCIDR] 6m 14s 4 {kubelet kubernetes-minion-group-yxra} Warning FailedMount Unable to mount volumes for pod "test-ipj93_default(-b5bb-11e6-bab0-)": timeout expired waiting for volumes to attach/mount for pod "test-ipj93"/"default". list of unattached/unmounted volumes=[data] 6m 14s 4 {kubelet kubernetes-minion-group-yxra} Warning FailedSync Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod "test-ipj93"/"default". list of unattached/unmounted volumes=[data] What you expected to happen: GCE PD to attach and mount successfully post the upgrade How to reproduce it (as minimally and precisely as possible): Running the upgrade steps in the doc for upgrade of manual test from 1.4 to 1.5 Anything else do we need to know:
Ran the upgrade again. This time the .
kube-controller-
Current theory is because kubelet deletes and recreates the node API object during upgrade (because externalID changed), see the new object does not contain the correct status (it is written by the Attach/Detach controller to the old object but not subseqently updated). If this proves to be the issue, the proposed fix is to force the attach/detach controller to update the node status when the node's external ID changes. is working to verify this theory by executing this code path (without a full upgrade). Once he has validated the theory, he will work on the fix, verify it, merge it to master, and get it cherry-picked to 1.5.
: is expected to fix this problem?
: Yes

} func (nsu *nodeStatusUpdater) UpdateNodeStatuses() error { <ins> // TODO: investigate right behavior if nodeName is empty // kubernetes/kubernetes/issues/37777 </ins> nodesToUpdate := nsu.actualStateOfWorld.GetVolumesToReportAttached() for nodeName, attachedVolumes := range nodesToUpdate { nodeObj, exists, err := nsu.nodeInformer.GetStore().GetByKey(string(nodeName))


Question:
Slack user "nalin" reported an issue Looking at the code this makes sense. Will fix.
For anyone following along, the fix for this bug unfortunately didn't get cherrypicked because of a process issue, and is broken until 1.5.2. cc
I just the new lines from As a temperatur fix ONLY. If anyone like me dont want to wait. Worked like at charm for AWS cluster setup. :)
A quick note, just as confirmation: the above temporary fix does resolve the issue for AWS - even though it's a change to a GCE-related file (cluster/gce/configure-). It seems that the GCE files are used for certain aws functions.
In based AWS deployment, the script is shared between AWS and GCE.
Also, this fix was cherry-picked into the branch a while ago, so instead of manually adding the lines, you can also just checkout the branch and run from there.

e2e_storage_test_environment: '$(echo "$E2E_STORAGE_TEST_ENVIRONMENT" | sed -e "s/'/''/g")' kube_uid: '$(echo "${KUBE_UID}" | sed -e "s/'/''/g")' initial_etcd_cluster: '$(echo "${INITIAL_ETCD_CLUSTER:-}" | sed -e "s/'/''/g")' <del> etcd_quorum_read: '$(echo "${ETCD_QUORUM_READ:-}" | sed -e "s/'/''/g")' </del> hostname: $(hostname -s) enable_default_storage_class: '$(echo "$ENABLE_DEFAULT_STORAGE_CLASS" | sed -e "s/'/''/g")'


Question:
Slack user "nalin" reported an issue Looking at the code this makes sense. Will fix.
For anyone following along, the fix for this bug unfortunately didn't get cherrypicked because of a process issue, and is broken until 1.5.2. cc
I just the new lines from As a temperatur fix ONLY. If anyone like me dont want to wait. Worked like at charm for AWS cluster setup. :)
A quick note, just as confirmation: the above temporary fix does resolve the issue for AWS - even though it's a change to a GCE-related file (cluster/gce/configure-). It seems that the GCE files are used for certain aws functions.
In based AWS deployment, the script is shared between AWS and GCE.
Also, this fix was cherry-picked into the branch a while ago, so instead of manually adding the lines, you can also just checkout the branch and run from there.

etcd_over_ssl: 'false' EOF fi <ins> if [ -n "${ETCD_QUORUM_READ:-}" ]; then cat <<EOF >>/srv/salt-overlay/pillar/cluster-params.sls etcd_quorum_read: '$(echo "${ETCD_QUORUM_READ}" | sed -e "s/'/''/g")' EOF fi </ins> # Configuration changes for test clusters if [ -n "${APISERVER_TEST_ARGS:-}" ]; then cat <<EOF >>/srv/salt-overlay/pillar/cluster-params.sls


Question:
Slack user "nalin" reported an issue Looking at the code this makes sense. Will fix.
For anyone following along, the fix for this bug unfortunately didn't get cherrypicked because of a process issue, and is broken until 1.5.2. cc
I just the new lines from As a temperatur fix ONLY. If anyone like me dont want to wait. Worked like at charm for AWS cluster setup. :)
A quick note, just as confirmation: the above temporary fix does resolve the issue for AWS - even though it's a change to a GCE-related file (cluster/gce/configure-). It seems that the GCE files are used for certain aws functions.
In based AWS deployment, the script is shared between AWS and GCE.
Also, this fix was cherry-picked into the branch a while ago, so instead of manually adding the lines, you can also just checkout the branch and run from there.

e2e_storage_test_environment: '$(echo "$E2E_STORAGE_TEST_ENVIRONMENT" | sed -e "s/'/''/g")' kube_uid: '$(echo "${KUBE_UID}" | sed -e "s/'/''/g")' initial_etcd_cluster: '$(echo "${INITIAL_ETCD_CLUSTER:-}" | sed -e "s/'/''/g")' <del> etcd_quorum_read: '$(echo "${ETCD_QUORUM_READ:-}" | sed -e "s/'/''/g")' </del> hostname: $(hostname -s) EOF


Question:
Slack user "nalin" reported an issue Looking at the code this makes sense. Will fix.
For anyone following along, the fix for this bug unfortunately didn't get cherrypicked because of a process issue, and is broken until 1.5.2. cc
I just the new lines from As a temperatur fix ONLY. If anyone like me dont want to wait. Worked like at charm for AWS cluster setup. :)
A quick note, just as confirmation: the above temporary fix does resolve the issue for AWS - even though it's a change to a GCE-related file (cluster/gce/configure-). It seems that the GCE files are used for certain aws functions.
In based AWS deployment, the script is shared between AWS and GCE.
Also, this fix was cherry-picked into the branch a while ago, so instead of manually adding the lines, you can also just checkout the branch and run from there.


Question:
Is this a BUG REPORT or FEATURE REQUEST? (choose one): bug Kubernetes version (use ): master Environment: Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: What happened: When testing , I found kube-proxy can not record event with following errors. What you expected to happen: No errors How to reproduce it (as minimally and precisely as possible): cc
I am still getting this with the latest build
, thanks for your feedback, I'll check it.

tags = ["automanaged"], deps = [ "//pkg/api:go_default_library", <ins> "//pkg/api/v1:go_default_library", </ins> "//pkg/apis/componentconfig:go_default_library", "//pkg/apis/componentconfig/v1alpha1:go_default_library", "//pkg/kubelet/qos:go_default_library",


Question:
Is this a BUG REPORT or FEATURE REQUEST? (choose one): bug Kubernetes version (use ): master Environment: Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: What happened: When testing , I found kube-proxy can not record event with following errors. What you expected to happen: No errors How to reproduce it (as minimally and precisely as possible): cc
I am still getting this with the latest build
, thanks for your feedback, I'll check it.

"time" "k8s.io/kubernetes/pkg/api" <ins> "k8s.io/kubernetes/pkg/api/v1" </ins> "k8s.io/kubernetes/pkg/apis/componentconfig" "k8s.io/kubernetes/pkg/apis/componentconfig/v1alpha1" "k8s.io/kubernetes/pkg/kubelet/qos"


Question:
Is this a BUG REPORT or FEATURE REQUEST? (choose one): bug Kubernetes version (use ): master Environment: Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: What happened: When testing , I found kube-proxy can not record event with following errors. What you expected to happen: No errors How to reproduce it (as minimally and precisely as possible): cc
I am still getting this with the latest build
, thanks for your feedback, I'll check it.

KubeAPIBurst      int32 ConfigSyncPeriod  time.Duration CleanupAndExit    bool <del> NodeRef           *api.ObjectReference </del> <ins> NodeRef           *v1.ObjectReference </ins> Master            string Kubeconfig        string }


Question:
Is this a BUG REPORT or FEATURE REQUEST? (choose one): bug Kubernetes version (use ): master Environment: Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: What happened: When testing , I found kube-proxy can not record event with following errors. What you expected to happen: No errors How to reproduce it (as minimally and precisely as possible): cc
I am still getting this with the latest build
, thanks for your feedback, I'll check it.

endpointsConfig.Channel("api"), ) <del> config.NodeRef = &api.ObjectReference{ </del> <ins> config.NodeRef = &v1.ObjectReference{ </ins> Kind:      "Node", Name:      hostname, UID:       types.UID(hostname),


Question:
Is this a BUG REPORT or FEATURE REQUEST? (choose one): bug Kubernetes version (use ): master Environment: Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: What happened: When testing , I found kube-proxy can not record event with following errors. What you expected to happen: No errors How to reproduce it (as minimally and precisely as possible): cc
I am still getting this with the latest build
, thanks for your feedback, I'll check it.

"//cmd/kube-proxy/app/options:go_default_library", "//cmd/kubelet/app:go_default_library", "//pkg/api:go_default_library", <ins> "//pkg/api/v1:go_default_library", </ins> "//pkg/apis/componentconfig:go_default_library", "//pkg/apis/componentconfig/v1alpha1:go_default_library", "//pkg/client/clientset_generated/clientset:go_default_library",


Question:
Is this a BUG REPORT or FEATURE REQUEST? (choose one): bug Kubernetes version (use ): master Environment: Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: What happened: When testing , I found kube-proxy can not record event with following errors. What you expected to happen: No errors How to reproduce it (as minimally and precisely as possible): cc
I am still getting this with the latest build
, thanks for your feedback, I'll check it.

proxyapp "k8s.io/kubernetes/cmd/kube-proxy/app" "k8s.io/kubernetes/cmd/kube-proxy/app/options" "k8s.io/kubernetes/pkg/api" <ins> "k8s.io/kubernetes/pkg/api/v1" </ins> clientset "k8s.io/kubernetes/pkg/client/clientset_generated/internalclientset" "k8s.io/kubernetes/pkg/client/record" proxyconfig "k8s.io/kubernetes/pkg/proxy/config"


Question:
Is this a BUG REPORT or FEATURE REQUEST? (choose one): bug Kubernetes version (use ): master Environment: Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: What happened: When testing , I found kube-proxy can not record event with following errors. What you expected to happen: No errors How to reproduce it (as minimally and precisely as possible): cc
I am still getting this with the latest build
, thanks for your feedback, I'll check it.

config := options.NewProxyConfig() config.OOMScoreAdj = util.Int32Ptr(0) config.ResourceContainer = "" <del> config.NodeRef = &api.ObjectReference{ </del> <ins> config.NodeRef = &v1.ObjectReference{ </ins> Kind:      "Node", Name:      nodeName, UID:       types.UID(nodeName),


Question: kubectl edit is not working on HEAD. It's working on 1.5.

docs/man/man1/kubectl-delete.1 docs/man/man1/kubectl-describe.1 docs/man/man1/kubectl-drain.1 <del> docs/man/man1/kubectl-edit-configmap.1 </del> docs/man/man1/kubectl-edit.1 docs/man/man1/kubectl-exec.1 docs/man/man1/kubectl-explain.1


Question: kubectl edit is not working on HEAD. It's working on 1.5.

docs/user-guide/kubectl/kubectl_describe.md docs/user-guide/kubectl/kubectl_drain.md docs/user-guide/kubectl/kubectl_edit.md <del> docs/user-guide/kubectl/kubectl_edit_configmap.md </del> docs/user-guide/kubectl/kubectl_exec.md docs/user-guide/kubectl/kubectl_explain.md docs/user-guide/kubectl/kubectl_expose.md


Question: kubectl edit is not working on HEAD. It's working on 1.5.

<del> This file is autogenerated, but we've stopped checking such files into the repository to reduce the need for rebases. Please run hack/generate-docs.sh to populate this file. </del>


Question: kubectl edit is not working on HEAD. It's working on 1.5.

<del> This file is autogenerated, but we've stopped checking such files into the repository to reduce the need for rebases. Please run hack/generate-docs.sh to populate this file. <!-- BEGIN MUNGE: GENERATED_ANALYTICS --> [![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/user-guide/kubectl/kubectl_edit_configmap.md?pixel)]() <!-- END MUNGE: GENERATED_ANALYTICS --> </del>


Question: kubectl edit is not working on HEAD. It's working on 1.5.

concurrent-serviceaccount-token-syncs concurrent-service-syncs config-map <del> config-map-data </del> config-map-namespace config-sync-period configure-cloud-routes


Question: kubectl edit is not working on HEAD. It's working on 1.5.

"describe.go", "drain.go", "edit.go", <del> "edit_configmap.go", </del> "exec.go", "explain.go", "expose.go",


Question: kubectl edit is not working on HEAD. It's working on 1.5.

Long:    editLong, Example: fmt.Sprintf(editExample), Run: func(cmd *cobra.Command, args []string) { <del> args = append([]string{"configmap"}, args...) </del> err := RunEdit(f, out, errOut, cmd, args, options) cmdutil.CheckErr(err) }, ValidArgs:  validArgs, ArgAliases: argAliases, } <del> addEditFlags(cmd, options) cmd.AddCommand(NewCmdEditConfigMap(f, out, errOut)) return cmd } func addEditFlags(cmd *cobra.Command, options *resource.FilenameOptions) { </del> usage := "to use to edit the resource" cmdutil.AddFilenameOptionFlags(cmd, options, usage) cmdutil.AddValidateFlags(cmd)


Question: kubectl edit is not working on HEAD. It's working on 1.5.

cmdutil.AddApplyAnnotationFlags(cmd) cmdutil.AddRecordFlag(cmd) cmdutil.AddInclude3rdPartyFlags(cmd) <ins> return cmd </ins> } func RunEdit(f cmdutil.Factory, out, errOut io.Writer, cmd *cobra.Command, args []string, options *resource.FilenameOptions) error {


Question: kubectl edit is not working on HEAD. It's working on 1.5.

<del> /* Copyright 2016 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package cmd import ( "bytes" "fmt" "io" "os" "github.com/spf13/cobra" "k8s.io/apimachinery/pkg/apis/meta/v1" cmdutil "k8s.io/kubernetes/pkg/kubectl/cmd/util" "k8s.io/kubernetes/pkg/kubectl/cmd/util/editor" "k8s.io/kubernetes/pkg/kubectl/resource" ) // NewCmdEditConfigMap is a macro command to edit config maps func NewCmdEditConfigMap(f cmdutil.Factory, cmdOut, errOut io.Writer) *cobra.Command { options := &resource.FilenameOptions{} cmd := &cobra.Command{ Use:     "configmap", Aliases: []string{"cm"}, Short:   "Edit a config map object.", Long:    "Edit and update a config map object", Run: func(cmd *cobra.Command, args []string) { RunEditConfigMap(cmd, f, args, cmdOut, errOut, options) }, } addEditFlags(cmd, options) cmd.Flags().String("config-map-data", "", "If non-empty, specify the name of a data slot in a config map to edit.") return cmd } // RunEditConfigMap runs the edit command for config maps. It either edits the complete map // or it edits individual files inside the config map. func RunEditConfigMap(cmd *cobra.Command, f cmdutil.Factory, args []string, cmdOut, errOut io.Writer, options *resource.FilenameOptions) error { dataFile := cmdutil.GetFlagString(cmd, "config-map-data") if len(dataFile) == 0 { // We need to add the resource type back on to the front args = append([]string{"configmap"}, args...) return RunEdit(f, cmdOut, errOut, cmd, args, options) } cmdNamespace, _, err := f.DefaultNamespace() if err != nil { return err } cs, err := f.ClientSet() if err != nil { return err } configMap, err := cs.Core().ConfigMaps(cmdNamespace).Get(args[0], v1.GetOptions{}) if err != nil { return err } value, found := configMap.Data[dataFile] if !found { keys := []string{} for key := range configMap.Data { keys = append(keys, key) } return fmt.Errorf("No such data file (%s), filenames are: %vn", dataFile, keys) } edit := editor.NewDefaultEditor(os.Environ()) data, file, err := edit.LaunchTempFile(fmt.Sprintf("%s-edit-", dataFile), "", bytes.NewBuffer([]byte(value))) defer func() { os.Remove(file) }() if err != nil { return err } configMap.Data[dataFile] = string(data) if _, err := cs.Core().ConfigMaps(cmdNamespace).Update(configMap); err != nil { return err } return nil } </del>


Question:
<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): no What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): subpath, emptyDir, mode, permissions Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-01-12T04:57:25Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-01-12T04:52:34Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"} Environment: Cloud provider or hardware configuration: Bare metal, SSD, but also VMs OS (e.g. from /etc/os-release): Debian GNU/Linux 8 (jessie) Kernel (e.g. ): 4.8.0-0.bpo.2-amd64 Install tools: Others: Docker v1.12.6 What happened: When I mount an volume using a subpath, I'm unable to write into it if my process is not running as . When I leave out the subpath, I can. What you expected to happen: I can write to the directory in both cases. How to reproduce it (as minimally and precisely as possible): Output is: Anything else we need to know: This issue is related to . I know this is caused by the Docker daemon creating those directories on a bind mount. Nevertheless, I expect Kubernetes to abstract this away, since this behavior is neither documented nor IMHO intuitive.
Could someone from please take a look? Thanks! cc:
cc/
attempted fix created here
I created a cherrypick PR for 1.6 here: since this bug is against 1.6. Could someone (who agrees this could be cherrypicked) please add the cherrypick-candidate label + milestone to the original PR , thank you! edit: sorry, it's against 1.5. Should it be cherrypicked back to that as well?
I'm still facing this issue with Kubernetes version 1.12.8 on a AKS cluster when I mount a Azure Storage as azureFile share. Did anyone know if the issue is fixed for all types of mounts and not only just the emptyDir?
same issue on GCP @ k8s 1.14
k8s 1.17 via eks as well
So what's the recommendation? Not using subPath?

return nil, err } if mount.SubPath != "" { <ins> fileinfo, err := os.Lstat(hostPath) if err != nil { return nil, err } perm := fileinfo.Mode() </ins> hostPath = filepath.Join(hostPath, mount.SubPath) <ins> // Create the sub path now because if it's auto-created later when referenced, it may have an // incorrect ownership and mode. For example, the sub path directory must have at least g+rwx // when the pod specifies an fsGroup, and if the directory is not created here, Docker will // later auto-create it with the incorrect mode 0750 if err := os.MkdirAll(hostPath, perm); err != nil { glog.Errorf("failed to mkdir:%s", hostPath) return nil, err } // chmod the sub path because umask may have prevented us from making the sub path with the same // permissions as the mounter path if err := os.Chmod(hostPath, perm); err != nil { return nil, err } </ins> } // Docker Volume Mounts fail on Windows if it is not of the form C:/


Question:
getNodeConditionPredicate() in makes our code hard to understand because it hides the node-condition-based filtering in the node lister, which is totally non-obvious. We should get rid of this function and have NodeController add NoSchedule taints for these situations instead. (Alas, we might not actually be able to get rid of getNodeConditionPredicate() completely until we get rid of the Unschedulable field of PodSpec, but at least we can get rid of all the other code in this function.) If there are pods that should still be able to schedule in any of these situations (e.g. DaemonSet pods?) we should add tolerations in admission control for them (e.g. see ). cc/ cc/ Sub-tasks according to the design doc: [x] Add node taints label and feature flag () [x] In the node controller, taint Nodes according to the Node Condition () [x] In DaemonSet, update DaemonSetController to Tolerant new Taints () [x] In admissionController, add MemoryPressure/DiskPressure toleration for no BestEffort pod () [x] In scheduler, disable the current behaviour of filtering out Nodes. Instead, pods will not be scheduled to tainted nodes if a toleration does not exist in its PodSpec () [x] Add e2e test for this feature () [x] Update doc for this feature ()
I think you are referring Unschedulable field of NodeSpec?
Yes sorry, that was a typo.
- please CC me to all PRs
See also
Note that memory and disk pressure are checked in separate predicates, not in the NodeConditionPredicate. These should be represented as NoSchedule taints too.
Setting NoSchedule taints corresponding to the node conditions that today cause pods not to schedule onto a node, should also allow us to use tolerations rather than special-case code in the DaemonSet controller (and thereby make it easier to have DaemonSet pods be scheduled by the default scheduler.)
ref/
The task list was moved to the issue description .
/cc
/reopen
you can't re-open an issue/PR unless you authored it or you are assigned to it. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/reopen
/reopen
Currently we already have toleration on dsc for taint related to node problems (e.g NodeNotReady and NodeUnreachable). But to me ds should respect the new taints (outOfDisk, NodeMemoryPressure etc..), any insights ?
/reopen
Yes, it's one of sub-task of this project. This issue is closed by 'partially fixes' :(, I had updated PRs to 'part of #'.
- I can handle the dsc tolerations if you want. cc to keep you on the loop.
, sure, thanks very much :).
- just saw that you moved node conditions into predicate func, once adding node taiting and the according tolerations to dsc, do we'll still need to have node conditions checking ? (since the we're tainting by node condition). Also, correct me if I'm wrong, but from what I understood: dsc tolerates new taints with effect noExec since we don't wanna evict any already-running pods ?
We won't, but it'll take time for us to get there. We'll roll out Tainting as alpha (obviously), later move it to beta and GA in 1.10 at the earliest. We can't remove Condition checking logic before that. dsc is a daemon set right? New taints are NoSchedule only, so running Pods won't be affected (at least on the control plane level). Kubelet can still decide to evict them based on problem it observes.
- seems logic for the timeline. From sub-tasks Since new Taints are NoSchedule Only, Daemon set shouldn't have toleration for these ? It should instead respect the new taints when applied and not schedule on these nodes, or I'm I missing something ?
The task for DS in my mind: For all DS pods, it should tolerant MemoryPressure & DiskPressure For critical DS pods, it should tolerant OutOfDisk And if the taint by condition feature is enabled, we should check OutOfDisk by taints instead of predicates; the refactor is trying to make this simpler :).
- You're right, I'll send a PR after yours get merged :)
Do we have a to do list so folks can help out with I'd like to help when I get some time. I have a question about timeline. If alpha makes it into v1.8, will I be able to schedule to NotReady nodes (no CNI) if I reference the alpha tolerations? Or do you mean it won't be functional at all until v1.10?
It means you'll need to flip feature gate flag for it to work.
, here is the task list: I'm working on related PRs for them; please ping me when you got time :).
- i anticipate we will also add a CPUPressure condition which will also mean that no BestEffort pod can be scheduled to that node if the kubelet is running with a static cpu assignment policy. /cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
I think it's done (as alpha). - do we want to keep this issue open?
We can close this.

//     running on that node. func (dsc *DaemonSetsController) nodeShouldRunDaemonPod(node *v1.Node, ds *extensions.DaemonSet) (wantToRun, shouldSchedule, shouldContinueRunning bool, err error) { newPod := NewPod(ds, node.Name) <del> critical := utilfeature.DefaultFeatureGate.Enabled(features.ExperimentalCriticalPodAnnotation) && kubelettypes.IsCriticalPod(newPod) </del> // Because these bools require an && of all their required conditions, we start // with all bools set to true and set a bool to false if a condition is not met.