Question:
Kubernetes version (use ): Environment: Cloud provider or hardware configuration: GKE OS (e.g. from /etc/os-release): container-os Kernel (e.g. ): Linux gke-wordpress-cluster-default-pool--m764 4.4.21+ SMP Fri Feb 17 15:34:45 PST 2017 x86_64 Intel(R) Xeon(R) CPU @ 2.60GHz GenuineIntel GNU/Linux Install tools: Others: What happened: ` I was able to bring up wordpress okay the first time, except GKE wasn't creating loadbalancer ips due to a quota issue, which I resolved, note at this point the mysql pod was up and had attached to it's volume. Upon deleting the wordpress deployment and creating it again I started getting the above errors. I deleted the mysql pod as well and brought it up again to see that it had the same issue. The volumes are backed by a gluster cluster on GCE. Looking at the brick logs on one of the gluster nodes I see I've tried restarting kubelet on the node. I can't find the kubelet log file on any of gke nodes. And I don't know how to get the kube-controller and apiserver logs from the master (GKE). I suspect it's a failing of the glusterfs client on the gke nodes? What you expected to happen: Deployment to mount the volumes successfully. How to reproduce it (as minimally and precisely as possible): Not sure how, but I've run into this intermittently Anything else we need to know: The volumes are backed by gluster
/cc
If you delete pod and create pods soon, it is possible that you could see some errors like "timeout expired waiting for volumes to attach/mount" temporally. Could you wait for a few more minutes to see whether the pods could be started correctly?
I actually tried that, deleted the pods, then waited 5 minutes and recreated them with the same error. Maybe it wasn't long enough
Can you post output of ? It may have some other events for failure..
I deleted the cluster this afternoon, but I checked constantly and those were the only errors, Timeout on mount and a failed to sync error
If you use level 4 log, it should also give the detailed messages of failed mount so that we can figure out why it failed.
Can you email me your project name, cluster name, and zone information? I can take a look at the master logs to see if there's anything useful there.
This is starting to happen more frequently..
could you provide more details about in what situation it happened frequently? Kubelet log could give us more information. If you use GKE cluster, run "journctlctl -u kubelet /tmp/log" to get the log. You could also share with us the log if it is ok.
This seems to be the relevant part of the log I re-created that and my pods sprang back up. Weird thing is I had created that, along with the pvcs and pvs but seems it's somehow getting deleted. There's almost certainly certain I'm doing wrong but I'm not sure what. ]
HI Jonathan, can you clarify what you had to recreate?
The glusterfs endpoints , Michelle Au <:
I have a very similar problem running OpenShift 3.4 in AWS EC2 doing the same exact thing. Brand new OSE install [root master]# oc version oc v3.4.1.12 kubernetes v1.4.0+ features: Basic-Auth GSSAPI Kerberos SPNEGO Server https://172.31.40.126:8443 openshift v3.4.1.12 kubernetes v1.4.0+ POD in question is the docker-registry (kind of important for OSE to work) I've followed this to see if it would fix the issue: But no luck or change. Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod "docker-registry-3-afbbt"/"default". list of unattached/unmounted volumes=[registry-storage] then: Unable to mount volumes for pod "docker-registry-3-afbbt_default(-1a45-11e7-abf8-)": timeout expired waiting for volumes to attach/mount for pod "docker-registry-3-afbbt"/"default". list of unattached/unmounted volumes=[registry-storage] I have no idea what log(s) to look at, this was of no help when I deployed a new docker-registry: oc logs -f dc/docker-registry Any help or guidance would be greatly appreciated as this is a blocker issue for me. thanks and best wishes,
are you using glusterfs for registry-storage volume or other type of volume? Could you share us your kubelet log if possible over slack Thanks!
Hi Jing Xu, Would you know where this log would be? Or is the name kublet<somethingso I may find it for you? I just requested to join the slack channel, as soon as I'm signed in I will share the log. Probably tomorrow as I'm on the east coast USA Thank you! Joel
For you are using GKE with GCI image, you can use "journalctl -u kubelet " to get the log. Otherwise, it should be in " , shepp67 <:
I also experienced this. Kubernetes version:Cloud provider or hardware configuration: OpenStack OS (e.g. from /etc/os-release): Ubuntu 16.04.1 LTS Kernel (e.g. ): Install tools: custom That kubernetes worker vm actually had volumes attached. I saw it on OpenStack dashboard. But: and actually: Besides, this is a new worker vm. I have other worker vms already running, using cinder volumes and there were no such errors. I use Helm to deploy this Pod. I have deployed it earlier and removed it. But it was 2 days ago. I found the solution here: . On that worker vm I run: Edit: or upgrade Kubernetes to = 1.6.0-beta.3 I don't know if this is the solution to issue.
I'm seeing this same error on all my manually launched private Ubuntu based clusters (6), running 1.5.3 and 1.6.0. In my case the messages reference only stopped pods launched by CronJob Jobs. The messages indicate issues mounting every volume in the pod including PVs and node-local volumes. These pods, and future ones launched by these CronJobs, are fully successful when they run. It is only after they run to successful completion and exit that these messages show up. Glancing at the kubelet code, this appears to be coming out of some housekeeping/synch loop.
I've identified a bug and am working on a fix for when "Unable to mount volumes for pod..." + "Error syncing pod" happen indefinitely BUT the pod is actually running just fine, as observed by . For others, if you see this message but your pod is Running, you can safely ignore it. It's caused by a small limitation of the reconstruction of state (what volumes are mounted) so there is a chance you will see this only if you restart kubelet while your pods had volumes mounted. correction: you say they even Completed, they weren't just Running fine. So yeah, it's just misleading logging and not really an indication of anything breaking
Because the error message is very generic, I think there might be different problems mentioned in this thread. 's issue could be fixed by 's PR are you still have any issues with mount? could you also let us know whether you still have the issue? Thanks! Jing
Hi my issue was related to which helped me sort out. I was creating the gluster endpoints without a service and turns out the kube-controller deletes all such endpoints that are not attached to a service at regular intervals. So after every so often when a pod went down and tried to come up again and mount its respective pv it failed because it couldn't connect to gluster. The fix is to create a service for the gluster endpoints.
I also just ran into such timeout issue: After running a load test against my postgres backed web application, the database didn't respond anymore for still unknown reasons. According to kubernetes it was up and running The log showed a bunch of these messages. Not sure yet why but possibly it couldn't access the volume anymore: Short after that, the pod entered "error" state: The event log showed: This error disappeared after a few minutes (not sure how long exactly, but it took a while), but then I got the same error as described here: I couldn't figure out anything more on my own. Nothing in the syslog, nothing more than this in the kubelet log. Finally I rebooted the machine which fixed the issue.
cc do you know what error of the network not ready? It might be helpful if you could share the kubelet log with us to debug. Thanks!
AFAIK, the "network not ready error" is benign, kubelet will retry after this error () cc
I hope I didn't miss some secrets in there, quite a talky log. But here it is: The database didn't respond anymore because it ran OOM and was killed, as also shown in the log. Although it appears like it was OOM killed after it became unresponsive in the log.. Either way, a bunch of other things got killed too, so granted - "not ideal" situation :). Yet it shouldn't drop the volume and should re-attach it.
I think you are hitting the issue mentioned above. He has a PR to fix it. I will make sure it can merge soon. Thanks!
Great, thanks! Do you know roughly when this will be available in GCE?
you mean GKE or GCE? For GKE it will take a little bit longer. On GCE, when the PR gets merged, the fix should be available when the next release is out (I think should be soon)
Running on GKE. I'd rather not launch my product with this bug around. Is there a workaround for it or something I can monitor for? Or maybe it could get hot fixed on GKE? If there is no workaround, this effectively renders GKE useless for stateful stuff.
It's hard to say if volumes are actually being dropped/unmounted or if the FailedMount errors are wrong & misleading, so I wouldn't go so far as to say it's broken because of some volume issues. It could be something else entirely. If nothing else, the PR jing's referring to will clean up the event noise and help us debug the root cause. In the meantime, I wouldn't trust the FailedMount events to be accurate, so if your applications are unresponsive, there may be other causes worth considering
Well, I only can say that my volumes were dropped and it didn't recover on it's own (only after reboot). Dropped a seen by kubernetes, dunno if the volumes were still attached but they weren't usable by my scheduled database. To be clear: I'm not "complaining" and this was just test/R&D, but it would this is the scariest kind of issue for me to happen in prod, given that it's a managed service where I can't find a workaround on my own. Rebooting fixed it in my case, but that's a big hammer..
You don't need to reboot the node. Delete the pod and restart it should work for this issue. Is this workaround ok for your case? , discordianfish <:
Good to know, thanks! Will try this next time.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Hi, I'm facing the same issue with k8s 1.9, installed by kops on AWS. volume created successfully on AWS and attached to the right node, but the pod is failed to start.
/remove-lifecycle stale
could you please provide more details about your issue? What are the steps to reproduce it? Thanks!
yeah can you open a new bug for EBS and cc me with details? I was hoping we fixed most of EBS problems in 1.9 - I will be more than happy to walk through and find what is causing volume to not mount. As I understand this is different from typical EBS problems where volume fails to attach. In your case, volume is being attached to right node but it is not being seen by kubelet.
I found the issue, and it was the fact that i used c5 instances with nvme, this is why the volume never been attached as k8s looking for /dev/xvdXX and with nvme that path is different. I did notice that in 1.9 k8s should support nvme natively but it seems that it's still not stable. thanks
oh weird. You should open a bug about NVMe support then. I was thinking it would just work.
cc Is there an issue related to nvme support? please let us know any issues you still have with 1.9. Thanks!
This is nvme on AWS, not related to the work is doing on gce.
Thanks Definitely related to NVMe support. Ran into this trying to mount a gp2 EBS volume on a node which was an m5 EC2 instance.
I think I am seeing like this with , but unsure of "why" at the moment. It would seem works perfectly fine.
did you guys had a chance to open a BUG for NVMe support on EBS? If you do - please paste output of and PersistentVolume spec. We can't fix it unless we know bit more. :(
sorry, we worked around this by reverting back to m4 class instances. It should be simple enough to reproduce however... create a k8s cluster w/ m5 EBS backed nodes and create a simple pod w/ a persistent volume. You'll notice that the pod will never launch and you should be able to debug further. If I do run into this again I'll be sure to get some more data for you.
I too have the error, error syncing pod. kubectl describe pod kafka1-kafka-0 .... .... .... Warning FailedSync 5m (x141 over 35m) kubelet, 10.186.. Error syncing pod Warning BackOff 57s (x164 over 35m) kubelet, 10.186.. Back-off restarting failed container kubectl logs kafka1-kafka-0 -p export KAFKABROKERID=kafka1-kafka-0 export KAFKAADVERTISEDLISTENERS=PLAINTEXT://172.30.*.:9092 /etc/confluent/docker/run sh: 3: /etc/confluent/docker/run: not found

openssh-client  nfs-common  socat  <ins> udev  </ins> util-linux COPY cni-bin/bin /opt/cni/bin


Question:
Kubernetes version (use ): Environment: Cloud provider or hardware configuration: GKE OS (e.g. from /etc/os-release): container-os Kernel (e.g. ): Linux gke-wordpress-cluster-default-pool--m764 4.4.21+ SMP Fri Feb 17 15:34:45 PST 2017 x86_64 Intel(R) Xeon(R) CPU @ 2.60GHz GenuineIntel GNU/Linux Install tools: Others: What happened: ` I was able to bring up wordpress okay the first time, except GKE wasn't creating loadbalancer ips due to a quota issue, which I resolved, note at this point the mysql pod was up and had attached to it's volume. Upon deleting the wordpress deployment and creating it again I started getting the above errors. I deleted the mysql pod as well and brought it up again to see that it had the same issue. The volumes are backed by a gluster cluster on GCE. Looking at the brick logs on one of the gluster nodes I see I've tried restarting kubelet on the node. I can't find the kubelet log file on any of gke nodes. And I don't know how to get the kube-controller and apiserver logs from the master (GKE). I suspect it's a failing of the glusterfs client on the gke nodes? What you expected to happen: Deployment to mount the volumes successfully. How to reproduce it (as minimally and precisely as possible): Not sure how, but I've run into this intermittently Anything else we need to know: The volumes are backed by gluster
/cc
If you delete pod and create pods soon, it is possible that you could see some errors like "timeout expired waiting for volumes to attach/mount" temporally. Could you wait for a few more minutes to see whether the pods could be started correctly?
I actually tried that, deleted the pods, then waited 5 minutes and recreated them with the same error. Maybe it wasn't long enough
Can you post output of ? It may have some other events for failure..
I deleted the cluster this afternoon, but I checked constantly and those were the only errors, Timeout on mount and a failed to sync error
If you use level 4 log, it should also give the detailed messages of failed mount so that we can figure out why it failed.
Can you email me your project name, cluster name, and zone information? I can take a look at the master logs to see if there's anything useful there.
This is starting to happen more frequently..
could you provide more details about in what situation it happened frequently? Kubelet log could give us more information. If you use GKE cluster, run "journctlctl -u kubelet /tmp/log" to get the log. You could also share with us the log if it is ok.
This seems to be the relevant part of the log I re-created that and my pods sprang back up. Weird thing is I had created that, along with the pvcs and pvs but seems it's somehow getting deleted. There's almost certainly certain I'm doing wrong but I'm not sure what. ]
HI Jonathan, can you clarify what you had to recreate?
The glusterfs endpoints , Michelle Au <:
I have a very similar problem running OpenShift 3.4 in AWS EC2 doing the same exact thing. Brand new OSE install [root master]# oc version oc v3.4.1.12 kubernetes v1.4.0+ features: Basic-Auth GSSAPI Kerberos SPNEGO Server https://172.31.40.126:8443 openshift v3.4.1.12 kubernetes v1.4.0+ POD in question is the docker-registry (kind of important for OSE to work) I've followed this to see if it would fix the issue: But no luck or change. Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod "docker-registry-3-afbbt"/"default". list of unattached/unmounted volumes=[registry-storage] then: Unable to mount volumes for pod "docker-registry-3-afbbt_default(-1a45-11e7-abf8-)": timeout expired waiting for volumes to attach/mount for pod "docker-registry-3-afbbt"/"default". list of unattached/unmounted volumes=[registry-storage] I have no idea what log(s) to look at, this was of no help when I deployed a new docker-registry: oc logs -f dc/docker-registry Any help or guidance would be greatly appreciated as this is a blocker issue for me. thanks and best wishes,
are you using glusterfs for registry-storage volume or other type of volume? Could you share us your kubelet log if possible over slack Thanks!
Hi Jing Xu, Would you know where this log would be? Or is the name kublet<somethingso I may find it for you? I just requested to join the slack channel, as soon as I'm signed in I will share the log. Probably tomorrow as I'm on the east coast USA Thank you! Joel
For you are using GKE with GCI image, you can use "journalctl -u kubelet " to get the log. Otherwise, it should be in " , shepp67 <:
I also experienced this. Kubernetes version:Cloud provider or hardware configuration: OpenStack OS (e.g. from /etc/os-release): Ubuntu 16.04.1 LTS Kernel (e.g. ): Install tools: custom That kubernetes worker vm actually had volumes attached. I saw it on OpenStack dashboard. But: and actually: Besides, this is a new worker vm. I have other worker vms already running, using cinder volumes and there were no such errors. I use Helm to deploy this Pod. I have deployed it earlier and removed it. But it was 2 days ago. I found the solution here: . On that worker vm I run: Edit: or upgrade Kubernetes to = 1.6.0-beta.3 I don't know if this is the solution to issue.
I'm seeing this same error on all my manually launched private Ubuntu based clusters (6), running 1.5.3 and 1.6.0. In my case the messages reference only stopped pods launched by CronJob Jobs. The messages indicate issues mounting every volume in the pod including PVs and node-local volumes. These pods, and future ones launched by these CronJobs, are fully successful when they run. It is only after they run to successful completion and exit that these messages show up. Glancing at the kubelet code, this appears to be coming out of some housekeeping/synch loop.
I've identified a bug and am working on a fix for when "Unable to mount volumes for pod..." + "Error syncing pod" happen indefinitely BUT the pod is actually running just fine, as observed by . For others, if you see this message but your pod is Running, you can safely ignore it. It's caused by a small limitation of the reconstruction of state (what volumes are mounted) so there is a chance you will see this only if you restart kubelet while your pods had volumes mounted. correction: you say they even Completed, they weren't just Running fine. So yeah, it's just misleading logging and not really an indication of anything breaking
Because the error message is very generic, I think there might be different problems mentioned in this thread. 's issue could be fixed by 's PR are you still have any issues with mount? could you also let us know whether you still have the issue? Thanks! Jing
Hi my issue was related to which helped me sort out. I was creating the gluster endpoints without a service and turns out the kube-controller deletes all such endpoints that are not attached to a service at regular intervals. So after every so often when a pod went down and tried to come up again and mount its respective pv it failed because it couldn't connect to gluster. The fix is to create a service for the gluster endpoints.
I also just ran into such timeout issue: After running a load test against my postgres backed web application, the database didn't respond anymore for still unknown reasons. According to kubernetes it was up and running The log showed a bunch of these messages. Not sure yet why but possibly it couldn't access the volume anymore: Short after that, the pod entered "error" state: The event log showed: This error disappeared after a few minutes (not sure how long exactly, but it took a while), but then I got the same error as described here: I couldn't figure out anything more on my own. Nothing in the syslog, nothing more than this in the kubelet log. Finally I rebooted the machine which fixed the issue.
cc do you know what error of the network not ready? It might be helpful if you could share the kubelet log with us to debug. Thanks!
AFAIK, the "network not ready error" is benign, kubelet will retry after this error () cc
I hope I didn't miss some secrets in there, quite a talky log. But here it is: The database didn't respond anymore because it ran OOM and was killed, as also shown in the log. Although it appears like it was OOM killed after it became unresponsive in the log.. Either way, a bunch of other things got killed too, so granted - "not ideal" situation :). Yet it shouldn't drop the volume and should re-attach it.
I think you are hitting the issue mentioned above. He has a PR to fix it. I will make sure it can merge soon. Thanks!
Great, thanks! Do you know roughly when this will be available in GCE?
you mean GKE or GCE? For GKE it will take a little bit longer. On GCE, when the PR gets merged, the fix should be available when the next release is out (I think should be soon)
Running on GKE. I'd rather not launch my product with this bug around. Is there a workaround for it or something I can monitor for? Or maybe it could get hot fixed on GKE? If there is no workaround, this effectively renders GKE useless for stateful stuff.
It's hard to say if volumes are actually being dropped/unmounted or if the FailedMount errors are wrong & misleading, so I wouldn't go so far as to say it's broken because of some volume issues. It could be something else entirely. If nothing else, the PR jing's referring to will clean up the event noise and help us debug the root cause. In the meantime, I wouldn't trust the FailedMount events to be accurate, so if your applications are unresponsive, there may be other causes worth considering
Well, I only can say that my volumes were dropped and it didn't recover on it's own (only after reboot). Dropped a seen by kubernetes, dunno if the volumes were still attached but they weren't usable by my scheduled database. To be clear: I'm not "complaining" and this was just test/R&D, but it would this is the scariest kind of issue for me to happen in prod, given that it's a managed service where I can't find a workaround on my own. Rebooting fixed it in my case, but that's a big hammer..
You don't need to reboot the node. Delete the pod and restart it should work for this issue. Is this workaround ok for your case? , discordianfish <:
Good to know, thanks! Will try this next time.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Hi, I'm facing the same issue with k8s 1.9, installed by kops on AWS. volume created successfully on AWS and attached to the right node, but the pod is failed to start.
/remove-lifecycle stale
could you please provide more details about your issue? What are the steps to reproduce it? Thanks!
yeah can you open a new bug for EBS and cc me with details? I was hoping we fixed most of EBS problems in 1.9 - I will be more than happy to walk through and find what is causing volume to not mount. As I understand this is different from typical EBS problems where volume fails to attach. In your case, volume is being attached to right node but it is not being seen by kubelet.
I found the issue, and it was the fact that i used c5 instances with nvme, this is why the volume never been attached as k8s looking for /dev/xvdXX and with nvme that path is different. I did notice that in 1.9 k8s should support nvme natively but it seems that it's still not stable. thanks
oh weird. You should open a bug about NVMe support then. I was thinking it would just work.
cc Is there an issue related to nvme support? please let us know any issues you still have with 1.9. Thanks!
This is nvme on AWS, not related to the work is doing on gce.
Thanks Definitely related to NVMe support. Ran into this trying to mount a gp2 EBS volume on a node which was an m5 EC2 instance.
I think I am seeing like this with , but unsure of "why" at the moment. It would seem works perfectly fine.
did you guys had a chance to open a BUG for NVMe support on EBS? If you do - please paste output of and PersistentVolume spec. We can't fix it unless we know bit more. :(
sorry, we worked around this by reverting back to m4 class instances. It should be simple enough to reproduce however... create a k8s cluster w/ m5 EBS backed nodes and create a simple pod w/ a persistent volume. You'll notice that the pod will never launch and you should be able to debug further. If I do run into this again I'll be sure to get some more data for you.
I too have the error, error syncing pod. kubectl describe pod kafka1-kafka-0 .... .... .... Warning FailedSync 5m (x141 over 35m) kubelet, 10.186.. Error syncing pod Warning BackOff 57s (x164 over 35m) kubelet, 10.186.. Back-off restarting failed container kubectl logs kafka1-kafka-0 -p export KAFKABROKERID=kafka1-kafka-0 export KAFKAADVERTISEDLISTENERS=PLAINTEXT://172.30.*.:9092 /etc/confluent/docker/run sh: 3: /etc/confluent/docker/run: not found

REGISTRY?=staging-k8s.gcr.io IMAGE?=debian-hyperkube-base <del> TAG=0.9 </del> <ins> TAG=0.10 </ins> ARCH?=amd64 CACHEBUST?=1


Question:
Kubernetes version (use ): Environment: Cloud provider or hardware configuration: GKE OS (e.g. from /etc/os-release): container-os Kernel (e.g. ): Linux gke-wordpress-cluster-default-pool--m764 4.4.21+ SMP Fri Feb 17 15:34:45 PST 2017 x86_64 Intel(R) Xeon(R) CPU @ 2.60GHz GenuineIntel GNU/Linux Install tools: Others: What happened: ` I was able to bring up wordpress okay the first time, except GKE wasn't creating loadbalancer ips due to a quota issue, which I resolved, note at this point the mysql pod was up and had attached to it's volume. Upon deleting the wordpress deployment and creating it again I started getting the above errors. I deleted the mysql pod as well and brought it up again to see that it had the same issue. The volumes are backed by a gluster cluster on GCE. Looking at the brick logs on one of the gluster nodes I see I've tried restarting kubelet on the node. I can't find the kubelet log file on any of gke nodes. And I don't know how to get the kube-controller and apiserver logs from the master (GKE). I suspect it's a failing of the glusterfs client on the gke nodes? What you expected to happen: Deployment to mount the volumes successfully. How to reproduce it (as minimally and precisely as possible): Not sure how, but I've run into this intermittently Anything else we need to know: The volumes are backed by gluster
/cc
If you delete pod and create pods soon, it is possible that you could see some errors like "timeout expired waiting for volumes to attach/mount" temporally. Could you wait for a few more minutes to see whether the pods could be started correctly?
I actually tried that, deleted the pods, then waited 5 minutes and recreated them with the same error. Maybe it wasn't long enough
Can you post output of ? It may have some other events for failure..
I deleted the cluster this afternoon, but I checked constantly and those were the only errors, Timeout on mount and a failed to sync error
If you use level 4 log, it should also give the detailed messages of failed mount so that we can figure out why it failed.
Can you email me your project name, cluster name, and zone information? I can take a look at the master logs to see if there's anything useful there.
This is starting to happen more frequently..
could you provide more details about in what situation it happened frequently? Kubelet log could give us more information. If you use GKE cluster, run "journctlctl -u kubelet /tmp/log" to get the log. You could also share with us the log if it is ok.
This seems to be the relevant part of the log I re-created that and my pods sprang back up. Weird thing is I had created that, along with the pvcs and pvs but seems it's somehow getting deleted. There's almost certainly certain I'm doing wrong but I'm not sure what. ]
HI Jonathan, can you clarify what you had to recreate?
The glusterfs endpoints , Michelle Au <:
I have a very similar problem running OpenShift 3.4 in AWS EC2 doing the same exact thing. Brand new OSE install [root master]# oc version oc v3.4.1.12 kubernetes v1.4.0+ features: Basic-Auth GSSAPI Kerberos SPNEGO Server https://172.31.40.126:8443 openshift v3.4.1.12 kubernetes v1.4.0+ POD in question is the docker-registry (kind of important for OSE to work) I've followed this to see if it would fix the issue: But no luck or change. Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod "docker-registry-3-afbbt"/"default". list of unattached/unmounted volumes=[registry-storage] then: Unable to mount volumes for pod "docker-registry-3-afbbt_default(-1a45-11e7-abf8-)": timeout expired waiting for volumes to attach/mount for pod "docker-registry-3-afbbt"/"default". list of unattached/unmounted volumes=[registry-storage] I have no idea what log(s) to look at, this was of no help when I deployed a new docker-registry: oc logs -f dc/docker-registry Any help or guidance would be greatly appreciated as this is a blocker issue for me. thanks and best wishes,
are you using glusterfs for registry-storage volume or other type of volume? Could you share us your kubelet log if possible over slack Thanks!
Hi Jing Xu, Would you know where this log would be? Or is the name kublet<somethingso I may find it for you? I just requested to join the slack channel, as soon as I'm signed in I will share the log. Probably tomorrow as I'm on the east coast USA Thank you! Joel
For you are using GKE with GCI image, you can use "journalctl -u kubelet " to get the log. Otherwise, it should be in " , shepp67 <:
I also experienced this. Kubernetes version:Cloud provider or hardware configuration: OpenStack OS (e.g. from /etc/os-release): Ubuntu 16.04.1 LTS Kernel (e.g. ): Install tools: custom That kubernetes worker vm actually had volumes attached. I saw it on OpenStack dashboard. But: and actually: Besides, this is a new worker vm. I have other worker vms already running, using cinder volumes and there were no such errors. I use Helm to deploy this Pod. I have deployed it earlier and removed it. But it was 2 days ago. I found the solution here: . On that worker vm I run: Edit: or upgrade Kubernetes to = 1.6.0-beta.3 I don't know if this is the solution to issue.
I'm seeing this same error on all my manually launched private Ubuntu based clusters (6), running 1.5.3 and 1.6.0. In my case the messages reference only stopped pods launched by CronJob Jobs. The messages indicate issues mounting every volume in the pod including PVs and node-local volumes. These pods, and future ones launched by these CronJobs, are fully successful when they run. It is only after they run to successful completion and exit that these messages show up. Glancing at the kubelet code, this appears to be coming out of some housekeeping/synch loop.
I've identified a bug and am working on a fix for when "Unable to mount volumes for pod..." + "Error syncing pod" happen indefinitely BUT the pod is actually running just fine, as observed by . For others, if you see this message but your pod is Running, you can safely ignore it. It's caused by a small limitation of the reconstruction of state (what volumes are mounted) so there is a chance you will see this only if you restart kubelet while your pods had volumes mounted. correction: you say they even Completed, they weren't just Running fine. So yeah, it's just misleading logging and not really an indication of anything breaking
Because the error message is very generic, I think there might be different problems mentioned in this thread. 's issue could be fixed by 's PR are you still have any issues with mount? could you also let us know whether you still have the issue? Thanks! Jing
Hi my issue was related to which helped me sort out. I was creating the gluster endpoints without a service and turns out the kube-controller deletes all such endpoints that are not attached to a service at regular intervals. So after every so often when a pod went down and tried to come up again and mount its respective pv it failed because it couldn't connect to gluster. The fix is to create a service for the gluster endpoints.
I also just ran into such timeout issue: After running a load test against my postgres backed web application, the database didn't respond anymore for still unknown reasons. According to kubernetes it was up and running The log showed a bunch of these messages. Not sure yet why but possibly it couldn't access the volume anymore: Short after that, the pod entered "error" state: The event log showed: This error disappeared after a few minutes (not sure how long exactly, but it took a while), but then I got the same error as described here: I couldn't figure out anything more on my own. Nothing in the syslog, nothing more than this in the kubelet log. Finally I rebooted the machine which fixed the issue.
cc do you know what error of the network not ready? It might be helpful if you could share the kubelet log with us to debug. Thanks!
AFAIK, the "network not ready error" is benign, kubelet will retry after this error () cc
I hope I didn't miss some secrets in there, quite a talky log. But here it is: The database didn't respond anymore because it ran OOM and was killed, as also shown in the log. Although it appears like it was OOM killed after it became unresponsive in the log.. Either way, a bunch of other things got killed too, so granted - "not ideal" situation :). Yet it shouldn't drop the volume and should re-attach it.
I think you are hitting the issue mentioned above. He has a PR to fix it. I will make sure it can merge soon. Thanks!
Great, thanks! Do you know roughly when this will be available in GCE?
you mean GKE or GCE? For GKE it will take a little bit longer. On GCE, when the PR gets merged, the fix should be available when the next release is out (I think should be soon)
Running on GKE. I'd rather not launch my product with this bug around. Is there a workaround for it or something I can monitor for? Or maybe it could get hot fixed on GKE? If there is no workaround, this effectively renders GKE useless for stateful stuff.
It's hard to say if volumes are actually being dropped/unmounted or if the FailedMount errors are wrong & misleading, so I wouldn't go so far as to say it's broken because of some volume issues. It could be something else entirely. If nothing else, the PR jing's referring to will clean up the event noise and help us debug the root cause. In the meantime, I wouldn't trust the FailedMount events to be accurate, so if your applications are unresponsive, there may be other causes worth considering
Well, I only can say that my volumes were dropped and it didn't recover on it's own (only after reboot). Dropped a seen by kubernetes, dunno if the volumes were still attached but they weren't usable by my scheduled database. To be clear: I'm not "complaining" and this was just test/R&D, but it would this is the scariest kind of issue for me to happen in prod, given that it's a managed service where I can't find a workaround on my own. Rebooting fixed it in my case, but that's a big hammer..
You don't need to reboot the node. Delete the pod and restart it should work for this issue. Is this workaround ok for your case? , discordianfish <:
Good to know, thanks! Will try this next time.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Hi, I'm facing the same issue with k8s 1.9, installed by kops on AWS. volume created successfully on AWS and attached to the right node, but the pod is failed to start.
/remove-lifecycle stale
could you please provide more details about your issue? What are the steps to reproduce it? Thanks!
yeah can you open a new bug for EBS and cc me with details? I was hoping we fixed most of EBS problems in 1.9 - I will be more than happy to walk through and find what is causing volume to not mount. As I understand this is different from typical EBS problems where volume fails to attach. In your case, volume is being attached to right node but it is not being seen by kubelet.
I found the issue, and it was the fact that i used c5 instances with nvme, this is why the volume never been attached as k8s looking for /dev/xvdXX and with nvme that path is different. I did notice that in 1.9 k8s should support nvme natively but it seems that it's still not stable. thanks
oh weird. You should open a bug about NVMe support then. I was thinking it would just work.
cc Is there an issue related to nvme support? please let us know any issues you still have with 1.9. Thanks!
This is nvme on AWS, not related to the work is doing on gce.
Thanks Definitely related to NVMe support. Ran into this trying to mount a gp2 EBS volume on a node which was an m5 EC2 instance.
I think I am seeing like this with , but unsure of "why" at the moment. It would seem works perfectly fine.
did you guys had a chance to open a BUG for NVMe support on EBS? If you do - please paste output of and PersistentVolume spec. We can't fix it unless we know bit more. :(
sorry, we worked around this by reverting back to m4 class instances. It should be simple enough to reproduce however... create a k8s cluster w/ m5 EBS backed nodes and create a simple pod w/ a persistent volume. You'll notice that the pod will never launch and you should be able to debug further. If I do run into this again I'll be sure to get some more data for you.
I too have the error, error syncing pod. kubectl describe pod kafka1-kafka-0 .... .... .... Warning FailedSync 5m (x141 over 35m) kubelet, 10.186.. Error syncing pod Warning BackOff 57s (x164 over 35m) kubelet, 10.186.. Back-off restarting failed container kubectl logs kafka1-kafka-0 -p export KAFKABROKERID=kafka1-kafka-0 export KAFKAADVERTISEDLISTENERS=PLAINTEXT://172.30.*.:9092 /etc/confluent/docker/run sh: 3: /etc/confluent/docker/run: not found

docker_pull( name = "debian-hyperkube-base-amd64", <del> digest = "sha256:d83594ecd85345144584523e7fa5388467edf5d2dfa30d0a1bcbf184cddf4a7b", </del> <ins> digest = "sha256:cc782ed16599000ca4c85d47ec6264753747ae1e77520894dca84b104a7621e2", </ins> registry = "k8s.gcr.io", repository = "debian-hyperkube-base-amd64", <del> tag = "0.9",  # ignored, but kept here for documentation </del> <ins> tag = "0.10",  # ignored, but kept here for documentation </ins> ) docker_pull(


Question:
Kubernetes version (use ): Environment: Cloud provider or hardware configuration: GKE OS (e.g. from /etc/os-release): container-os Kernel (e.g. ): Linux gke-wordpress-cluster-default-pool--m764 4.4.21+ SMP Fri Feb 17 15:34:45 PST 2017 x86_64 Intel(R) Xeon(R) CPU @ 2.60GHz GenuineIntel GNU/Linux Install tools: Others: What happened: ` I was able to bring up wordpress okay the first time, except GKE wasn't creating loadbalancer ips due to a quota issue, which I resolved, note at this point the mysql pod was up and had attached to it's volume. Upon deleting the wordpress deployment and creating it again I started getting the above errors. I deleted the mysql pod as well and brought it up again to see that it had the same issue. The volumes are backed by a gluster cluster on GCE. Looking at the brick logs on one of the gluster nodes I see I've tried restarting kubelet on the node. I can't find the kubelet log file on any of gke nodes. And I don't know how to get the kube-controller and apiserver logs from the master (GKE). I suspect it's a failing of the glusterfs client on the gke nodes? What you expected to happen: Deployment to mount the volumes successfully. How to reproduce it (as minimally and precisely as possible): Not sure how, but I've run into this intermittently Anything else we need to know: The volumes are backed by gluster
/cc
If you delete pod and create pods soon, it is possible that you could see some errors like "timeout expired waiting for volumes to attach/mount" temporally. Could you wait for a few more minutes to see whether the pods could be started correctly?
I actually tried that, deleted the pods, then waited 5 minutes and recreated them with the same error. Maybe it wasn't long enough
Can you post output of ? It may have some other events for failure..
I deleted the cluster this afternoon, but I checked constantly and those were the only errors, Timeout on mount and a failed to sync error
If you use level 4 log, it should also give the detailed messages of failed mount so that we can figure out why it failed.
Can you email me your project name, cluster name, and zone information? I can take a look at the master logs to see if there's anything useful there.
This is starting to happen more frequently..
could you provide more details about in what situation it happened frequently? Kubelet log could give us more information. If you use GKE cluster, run "journctlctl -u kubelet /tmp/log" to get the log. You could also share with us the log if it is ok.
This seems to be the relevant part of the log I re-created that and my pods sprang back up. Weird thing is I had created that, along with the pvcs and pvs but seems it's somehow getting deleted. There's almost certainly certain I'm doing wrong but I'm not sure what. ]
HI Jonathan, can you clarify what you had to recreate?
The glusterfs endpoints , Michelle Au <:
I have a very similar problem running OpenShift 3.4 in AWS EC2 doing the same exact thing. Brand new OSE install [root master]# oc version oc v3.4.1.12 kubernetes v1.4.0+ features: Basic-Auth GSSAPI Kerberos SPNEGO Server https://172.31.40.126:8443 openshift v3.4.1.12 kubernetes v1.4.0+ POD in question is the docker-registry (kind of important for OSE to work) I've followed this to see if it would fix the issue: But no luck or change. Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod "docker-registry-3-afbbt"/"default". list of unattached/unmounted volumes=[registry-storage] then: Unable to mount volumes for pod "docker-registry-3-afbbt_default(-1a45-11e7-abf8-)": timeout expired waiting for volumes to attach/mount for pod "docker-registry-3-afbbt"/"default". list of unattached/unmounted volumes=[registry-storage] I have no idea what log(s) to look at, this was of no help when I deployed a new docker-registry: oc logs -f dc/docker-registry Any help or guidance would be greatly appreciated as this is a blocker issue for me. thanks and best wishes,
are you using glusterfs for registry-storage volume or other type of volume? Could you share us your kubelet log if possible over slack Thanks!
Hi Jing Xu, Would you know where this log would be? Or is the name kublet<somethingso I may find it for you? I just requested to join the slack channel, as soon as I'm signed in I will share the log. Probably tomorrow as I'm on the east coast USA Thank you! Joel
For you are using GKE with GCI image, you can use "journalctl -u kubelet " to get the log. Otherwise, it should be in " , shepp67 <:
I also experienced this. Kubernetes version:Cloud provider or hardware configuration: OpenStack OS (e.g. from /etc/os-release): Ubuntu 16.04.1 LTS Kernel (e.g. ): Install tools: custom That kubernetes worker vm actually had volumes attached. I saw it on OpenStack dashboard. But: and actually: Besides, this is a new worker vm. I have other worker vms already running, using cinder volumes and there were no such errors. I use Helm to deploy this Pod. I have deployed it earlier and removed it. But it was 2 days ago. I found the solution here: . On that worker vm I run: Edit: or upgrade Kubernetes to = 1.6.0-beta.3 I don't know if this is the solution to issue.
I'm seeing this same error on all my manually launched private Ubuntu based clusters (6), running 1.5.3 and 1.6.0. In my case the messages reference only stopped pods launched by CronJob Jobs. The messages indicate issues mounting every volume in the pod including PVs and node-local volumes. These pods, and future ones launched by these CronJobs, are fully successful when they run. It is only after they run to successful completion and exit that these messages show up. Glancing at the kubelet code, this appears to be coming out of some housekeeping/synch loop.
I've identified a bug and am working on a fix for when "Unable to mount volumes for pod..." + "Error syncing pod" happen indefinitely BUT the pod is actually running just fine, as observed by . For others, if you see this message but your pod is Running, you can safely ignore it. It's caused by a small limitation of the reconstruction of state (what volumes are mounted) so there is a chance you will see this only if you restart kubelet while your pods had volumes mounted. correction: you say they even Completed, they weren't just Running fine. So yeah, it's just misleading logging and not really an indication of anything breaking
Because the error message is very generic, I think there might be different problems mentioned in this thread. 's issue could be fixed by 's PR are you still have any issues with mount? could you also let us know whether you still have the issue? Thanks! Jing
Hi my issue was related to which helped me sort out. I was creating the gluster endpoints without a service and turns out the kube-controller deletes all such endpoints that are not attached to a service at regular intervals. So after every so often when a pod went down and tried to come up again and mount its respective pv it failed because it couldn't connect to gluster. The fix is to create a service for the gluster endpoints.
I also just ran into such timeout issue: After running a load test against my postgres backed web application, the database didn't respond anymore for still unknown reasons. According to kubernetes it was up and running The log showed a bunch of these messages. Not sure yet why but possibly it couldn't access the volume anymore: Short after that, the pod entered "error" state: The event log showed: This error disappeared after a few minutes (not sure how long exactly, but it took a while), but then I got the same error as described here: I couldn't figure out anything more on my own. Nothing in the syslog, nothing more than this in the kubelet log. Finally I rebooted the machine which fixed the issue.
cc do you know what error of the network not ready? It might be helpful if you could share the kubelet log with us to debug. Thanks!
AFAIK, the "network not ready error" is benign, kubelet will retry after this error () cc
I hope I didn't miss some secrets in there, quite a talky log. But here it is: The database didn't respond anymore because it ran OOM and was killed, as also shown in the log. Although it appears like it was OOM killed after it became unresponsive in the log.. Either way, a bunch of other things got killed too, so granted - "not ideal" situation :). Yet it shouldn't drop the volume and should re-attach it.
I think you are hitting the issue mentioned above. He has a PR to fix it. I will make sure it can merge soon. Thanks!
Great, thanks! Do you know roughly when this will be available in GCE?
you mean GKE or GCE? For GKE it will take a little bit longer. On GCE, when the PR gets merged, the fix should be available when the next release is out (I think should be soon)
Running on GKE. I'd rather not launch my product with this bug around. Is there a workaround for it or something I can monitor for? Or maybe it could get hot fixed on GKE? If there is no workaround, this effectively renders GKE useless for stateful stuff.
It's hard to say if volumes are actually being dropped/unmounted or if the FailedMount errors are wrong & misleading, so I wouldn't go so far as to say it's broken because of some volume issues. It could be something else entirely. If nothing else, the PR jing's referring to will clean up the event noise and help us debug the root cause. In the meantime, I wouldn't trust the FailedMount events to be accurate, so if your applications are unresponsive, there may be other causes worth considering
Well, I only can say that my volumes were dropped and it didn't recover on it's own (only after reboot). Dropped a seen by kubernetes, dunno if the volumes were still attached but they weren't usable by my scheduled database. To be clear: I'm not "complaining" and this was just test/R&D, but it would this is the scariest kind of issue for me to happen in prod, given that it's a managed service where I can't find a workaround on my own. Rebooting fixed it in my case, but that's a big hammer..
You don't need to reboot the node. Delete the pod and restart it should work for this issue. Is this workaround ok for your case? , discordianfish <:
Good to know, thanks! Will try this next time.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Hi, I'm facing the same issue with k8s 1.9, installed by kops on AWS. volume created successfully on AWS and attached to the right node, but the pod is failed to start.
/remove-lifecycle stale
could you please provide more details about your issue? What are the steps to reproduce it? Thanks!
yeah can you open a new bug for EBS and cc me with details? I was hoping we fixed most of EBS problems in 1.9 - I will be more than happy to walk through and find what is causing volume to not mount. As I understand this is different from typical EBS problems where volume fails to attach. In your case, volume is being attached to right node but it is not being seen by kubelet.
I found the issue, and it was the fact that i used c5 instances with nvme, this is why the volume never been attached as k8s looking for /dev/xvdXX and with nvme that path is different. I did notice that in 1.9 k8s should support nvme natively but it seems that it's still not stable. thanks
oh weird. You should open a bug about NVMe support then. I was thinking it would just work.
cc Is there an issue related to nvme support? please let us know any issues you still have with 1.9. Thanks!
This is nvme on AWS, not related to the work is doing on gce.
Thanks Definitely related to NVMe support. Ran into this trying to mount a gp2 EBS volume on a node which was an m5 EC2 instance.
I think I am seeing like this with , but unsure of "why" at the moment. It would seem works perfectly fine.
did you guys had a chance to open a BUG for NVMe support on EBS? If you do - please paste output of and PersistentVolume spec. We can't fix it unless we know bit more. :(
sorry, we worked around this by reverting back to m4 class instances. It should be simple enough to reproduce however... create a k8s cluster w/ m5 EBS backed nodes and create a simple pod w/ a persistent volume. You'll notice that the pod will never launch and you should be able to debug further. If I do run into this again I'll be sure to get some more data for you.
I too have the error, error syncing pod. kubectl describe pod kafka1-kafka-0 .... .... .... Warning FailedSync 5m (x141 over 35m) kubelet, 10.186.. Error syncing pod Warning BackOff 57s (x164 over 35m) kubelet, 10.186.. Back-off restarting failed container kubectl logs kafka1-kafka-0 -p export KAFKABROKERID=kafka1-kafka-0 export KAFKAADVERTISEDLISTENERS=PLAINTEXT://172.30.*.:9092 /etc/confluent/docker/run sh: 3: /etc/confluent/docker/run: not found

OUT_DIR?=_output HYPERKUBE_BIN?=$(OUT_DIR)/dockerized/bin/linux/$(ARCH)/hyperkube <del> BASEIMAGE=k8s.gcr.io/debian-hyperkube-base-$(ARCH):0.8 </del> <ins> BASEIMAGE=k8s.gcr.io/debian-hyperkube-base-$(ARCH):0.10 </ins> TEMP_DIR:=$(shell mktemp -d -t hyperkubeXXXXXX) all: build


Question:
Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Environment: Cloud provider or hardware configuration: bare-metal OS (e.g. from /etc/os-release): Ubuntu 14.04 Kernel (e.g. ): Linux dev-k8s-controller-p1 4.4.0-53-generic ~14.04.1-Ubuntu SMP Fri Dec 2 03:43:31 UTC 2016 x8664 x8664 x86_64 GNU/Linux Install tools: chef cookbook Others: Docker 1.12.6, calico 2.1 CNI binaries: kubelet config: What happened: When trying to run pod error message is generated: So it seems that kubelet is trying to find CNI binaries in paths: /opt/loopback/bin /etc/cni/net.d /opt/calico/bin But it should look at since this is the default valur for parameter. What you expected to happen: Pod starts. How to reproduce it (as minimally and precisely as possible): Run pod: with kubelet 1.6 and CNI binaries in those path (default for calico): Anything else we need to know: After copy to and pod starts fine.
Good, very helpful to me
When set and not set , default path of CNI binaries will not take effect, the path specified by will be used instead. We should use and instead of if use CNI plugin.
I will push a PR to deprecate the flag. It is confusing. Closing this issue.
Was that ? It was closed without merging. I agree should be deprecated.
I think is needed before is removed. The flag is still not deprecated, I will send a PR to deprecate it.

fi # Network plugin if [[ -n "${NETWORK_PROVIDER:-}" ]]; then <del> if [[ "${NETWORK_PROVIDER:-}" == "cni" ]]; then flags+=" --cni-bin-dir=/opt/kubernetes/bin" else flags+=" --network-plugin-dir=/opt/kubernetes/bin" fi </del> <ins> flags+=" --cni-bin-dir=/opt/kubernetes/bin" </ins> flags+=" --network-plugin=${NETWORK_PROVIDER}" fi if [[ -n "${NON_MASQUERADE_CIDR:-}" ]]; then


Question:
Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Environment: Cloud provider or hardware configuration: bare-metal OS (e.g. from /etc/os-release): Ubuntu 14.04 Kernel (e.g. ): Linux dev-k8s-controller-p1 4.4.0-53-generic ~14.04.1-Ubuntu SMP Fri Dec 2 03:43:31 UTC 2016 x8664 x8664 x86_64 GNU/Linux Install tools: chef cookbook Others: Docker 1.12.6, calico 2.1 CNI binaries: kubelet config: What happened: When trying to run pod error message is generated: So it seems that kubelet is trying to find CNI binaries in paths: /opt/loopback/bin /etc/cni/net.d /opt/calico/bin But it should look at since this is the default valur for parameter. What you expected to happen: Pod starts. How to reproduce it (as minimally and precisely as possible): Run pod: with kubelet 1.6 and CNI binaries in those path (default for calico): Anything else we need to know: After copy to and pod starts fine.
Good, very helpful to me
When set and not set , default path of CNI binaries will not take effect, the path specified by will be used instead. We should use and instead of if use CNI plugin.
I will push a PR to deprecate the flag. It is confusing. Closing this issue.
Was that ? It was closed without merging. I agree should be deprecated.
I think is needed before is removed. The flag is still not deprecated, I will send a PR to deprecate it.

fi # Network plugin if [[ -n "${NETWORK_PROVIDER:-}" || -n "${NETWORK_POLICY_PROVIDER:-}" ]]; then <del> if [[ "${NETWORK_PROVIDER:-}" == "cni" || "${NETWORK_POLICY_PROVIDER:-}" == "calico" ]]; then flags+=" --cni-bin-dir=/home/kubernetes/bin" else flags+=" --network-plugin-dir=/home/kubernetes/bin" fi </del> <ins> flags+=" --cni-bin-dir=/home/kubernetes/bin" </ins> if [[ "${NETWORK_POLICY_PROVIDER:-}" == "calico" ]]; then # Calico uses CNI always. flags+=" --network-plugin=cni"


Question:
Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Environment: Cloud provider or hardware configuration: bare-metal OS (e.g. from /etc/os-release): Ubuntu 14.04 Kernel (e.g. ): Linux dev-k8s-controller-p1 4.4.0-53-generic ~14.04.1-Ubuntu SMP Fri Dec 2 03:43:31 UTC 2016 x8664 x8664 x86_64 GNU/Linux Install tools: chef cookbook Others: Docker 1.12.6, calico 2.1 CNI binaries: kubelet config: What happened: When trying to run pod error message is generated: So it seems that kubelet is trying to find CNI binaries in paths: /opt/loopback/bin /etc/cni/net.d /opt/calico/bin But it should look at since this is the default valur for parameter. What you expected to happen: Pod starts. How to reproduce it (as minimally and precisely as possible): Run pod: with kubelet 1.6 and CNI binaries in those path (default for calico): Anything else we need to know: After copy to and pod starts fine.
Good, very helpful to me
When set and not set , default path of CNI binaries will not take effect, the path specified by will be used instead. We should use and instead of if use CNI plugin.
I will push a PR to deprecate the flag. It is confusing. Closing this issue.
Was that ? It was closed without merging. I agree should be deprecated.
I think is needed before is removed. The flag is still not deprecated, I will send a PR to deprecate it.

{% if pillar.get('network_provider', '').lower() == 'opencontrail' %} {% set network_plugin = "--network-plugin=opencontrail" %} {% elif pillar.get('network_provider', '').lower() == 'cni' %} <del> {% set network_plugin = "--network-plugin=cni --network-plugin-dir=/etc/cni/net.d/" %} </del> <ins> {% set network_plugin = "--network-plugin=cni --cni-bin-dir=/etc/cni/net.d/" %} </ins> {%elif pillar.get('network_policy_provider', '').lower() == 'calico' and grains['roles'][0] != 'kubernetes-master' -%} <del> {% set network_plugin = "--network-plugin=cni --network-plugin-dir=/etc/cni/net.d/ --cni-bin-dir=/home/kubernetes/bin/" %} </del> <ins> {% set network_plugin = "--network-plugin=cni --cni-conf-dir=/etc/cni/net.d/ --cni-bin-dir=/home/kubernetes/bin/" %} </ins> {% elif pillar.get('network_provider', '').lower() == 'kubenet' %} {% set network_plugin = "--network-plugin=kubenet" -%} {% endif -%}


Question:
Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Environment: Cloud provider or hardware configuration: bare-metal OS (e.g. from /etc/os-release): Ubuntu 14.04 Kernel (e.g. ): Linux dev-k8s-controller-p1 4.4.0-53-generic ~14.04.1-Ubuntu SMP Fri Dec 2 03:43:31 UTC 2016 x8664 x8664 x86_64 GNU/Linux Install tools: chef cookbook Others: Docker 1.12.6, calico 2.1 CNI binaries: kubelet config: What happened: When trying to run pod error message is generated: So it seems that kubelet is trying to find CNI binaries in paths: /opt/loopback/bin /etc/cni/net.d /opt/calico/bin But it should look at since this is the default valur for parameter. What you expected to happen: Pod starts. How to reproduce it (as minimally and precisely as possible): Run pod: with kubelet 1.6 and CNI binaries in those path (default for calico): Anything else we need to know: After copy to and pod starts fine.
Good, very helpful to me
When set and not set , default path of CNI binaries will not take effect, the path specified by will be used instead. We should use and instead of if use CNI plugin.
I will push a PR to deprecate the flag. It is confusing. Closing this issue.
Was that ? It was closed without merging. I agree should be deprecated.
I think is needed before is removed. The flag is still not deprecated, I will send a PR to deprecate it.

// Network plugin settings. Shared by both docker and rkt. fs.StringVar(&s.NetworkPluginName, "network-plugin", s.NetworkPluginName, "<Warning: Alpha feature> The name of the network plugin to be invoked for various events in kubelet/pod lifecycle") <ins> //TODO(#46410): Remove the network-plugin-dir flag. </ins> fs.StringVar(&s.NetworkPluginDir, "network-plugin-dir", s.NetworkPluginDir, "<Warning: Alpha feature> The full path of the directory in which to search for network plugins or CNI config") <ins> fs.MarkDeprecated("network-plugin-dir", "Use --cni-bin-dir instead. This flag will be removed in a future version.") </ins> fs.StringVar(&s.CNIConfDir, "cni-conf-dir", s.CNIConfDir, "<Warning: Alpha feature> The full path of the directory in which to search for CNI config files. Default: /etc/cni/net.d") fs.StringVar(&s.CNIBinDir, "cni-bin-dir", s.CNIBinDir, "<Warning: Alpha feature> The full path of the directory in which to search for CNI plugin binaries. Default: /opt/cni/bin") fs.Int32Var(&s.NetworkPluginMTU, "network-plugin-mtu", s.NetworkPluginMTU, "<Warning: Alpha feature> The MTU to be passed to the network plugin, to override the default. Set to 0 to use the default 1460 MTU.")


Question:
Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Environment: Cloud provider or hardware configuration: bare-metal OS (e.g. from /etc/os-release): Ubuntu 14.04 Kernel (e.g. ): Linux dev-k8s-controller-p1 4.4.0-53-generic ~14.04.1-Ubuntu SMP Fri Dec 2 03:43:31 UTC 2016 x8664 x8664 x86_64 GNU/Linux Install tools: chef cookbook Others: Docker 1.12.6, calico 2.1 CNI binaries: kubelet config: What happened: When trying to run pod error message is generated: So it seems that kubelet is trying to find CNI binaries in paths: /opt/loopback/bin /etc/cni/net.d /opt/calico/bin But it should look at since this is the default valur for parameter. What you expected to happen: Pod starts. How to reproduce it (as minimally and precisely as possible): Run pod: with kubelet 1.6 and CNI binaries in those path (default for calico): Anything else we need to know: After copy to and pod starts fine.
Good, very helpful to me
When set and not set , default path of CNI binaries will not take effect, the path specified by will be used instead. We should use and instead of if use CNI plugin.
I will push a PR to deprecate the flag. It is confusing. Closing this issue.
Was that ? It was closed without merging. I agree should be deprecated.
I think is needed before is removed. The flag is still not deprecated, I will send a PR to deprecate it.

KUBELET_FLAGS=${KUBELET_FLAGS:-""} # Name of the network plugin, eg: "kubenet" NET_PLUGIN=${NET_PLUGIN:-""} <del> # Place the binaries required by NET_PLUGIN in this directory, eg: "/home/kubernetes/bin". NET_PLUGIN_DIR=${NET_PLUGIN_DIR:-""} </del> <ins> # Place the config files and binaries required by NET_PLUGIN in these directory, # eg: "/etc/cni/net.d" for config files, and "/opt/cni/bin" for binaries. CNI_CONF_DIR=${CNI_CONF_DIR:-""} CNI_BIN_DIR=${CNI_BIN_DIR:-""} </ins> SERVICE_CLUSTER_IP_RANGE=${SERVICE_CLUSTER_IP_RANGE:-10.0.0.0/24} FIRST_SERVICE_CLUSTER_IP=${FIRST_SERVICE_CLUSTER_IP:-10.0.0.1} # if enabled, must set CGROUP_ROOT


Question:
Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Environment: Cloud provider or hardware configuration: bare-metal OS (e.g. from /etc/os-release): Ubuntu 14.04 Kernel (e.g. ): Linux dev-k8s-controller-p1 4.4.0-53-generic ~14.04.1-Ubuntu SMP Fri Dec 2 03:43:31 UTC 2016 x8664 x8664 x86_64 GNU/Linux Install tools: chef cookbook Others: Docker 1.12.6, calico 2.1 CNI binaries: kubelet config: What happened: When trying to run pod error message is generated: So it seems that kubelet is trying to find CNI binaries in paths: /opt/loopback/bin /etc/cni/net.d /opt/calico/bin But it should look at since this is the default valur for parameter. What you expected to happen: Pod starts. How to reproduce it (as minimally and precisely as possible): Run pod: with kubelet 1.6 and CNI binaries in those path (default for calico): Anything else we need to know: After copy to and pod starts fine.
Good, very helpful to me
When set and not set , default path of CNI binaries will not take effect, the path specified by will be used instead. We should use and instead of if use CNI plugin.
I will push a PR to deprecate the flag. It is confusing. Closing this issue.
Was that ? It was closed without merging. I agree should be deprecated.
I think is needed before is removed. The flag is still not deprecated, I will send a PR to deprecate it.

auth_args="${auth_args} --client-ca-file=${CLIENT_CA_FILE}" fi <del> net_plugin_dir_args="" if [[ -n "${NET_PLUGIN_DIR}" ]]; then net_plugin_dir_args="--network-plugin-dir=${NET_PLUGIN_DIR}" </del> <ins> cni_conf_dir_args="" if [[ -n "${CNI_CONF_DIR}" ]]; then cni_conf_dir_args="--cni-conf-dir=${CNI_CONF_DIR}" fi cni_bin_dir_args="" if [[ -n "${CNI_BIN_DIR}" ]]; then cni_bin_dir_args="--cni-bin-dir=${CNI_BIN_DIR}" </ins> fi container_runtime_endpoint_args=""


Question:
Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Environment: Cloud provider or hardware configuration: bare-metal OS (e.g. from /etc/os-release): Ubuntu 14.04 Kernel (e.g. ): Linux dev-k8s-controller-p1 4.4.0-53-generic ~14.04.1-Ubuntu SMP Fri Dec 2 03:43:31 UTC 2016 x8664 x8664 x86_64 GNU/Linux Install tools: chef cookbook Others: Docker 1.12.6, calico 2.1 CNI binaries: kubelet config: What happened: When trying to run pod error message is generated: So it seems that kubelet is trying to find CNI binaries in paths: /opt/loopback/bin /etc/cni/net.d /opt/calico/bin But it should look at since this is the default valur for parameter. What you expected to happen: Pod starts. How to reproduce it (as minimally and precisely as possible): Run pod: with kubelet 1.6 and CNI binaries in those path (default for calico): Anything else we need to know: After copy to and pod starts fine.
Good, very helpful to me
When set and not set , default path of CNI binaries will not take effect, the path specified by will be used instead. We should use and instead of if use CNI plugin.
I will push a PR to deprecate the flag. It is confusing. Closing this issue.
Was that ? It was closed without merging. I agree should be deprecated.
I think is needed before is removed. The flag is still not deprecated, I will send a PR to deprecate it.

--pod-manifest-path="${POD_MANIFEST_PATH}"  ${auth_args}  ${dns_args}  <del> ${net_plugin_dir_args}  </del> <ins> ${cni_conf_dir_args}  ${cni_bin_dir_args}  </ins> ${net_plugin_args}  ${container_runtime_endpoint_args}  ${image_service_endpoint_args}


Question:
Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Environment: Cloud provider or hardware configuration: bare-metal OS (e.g. from /etc/os-release): Ubuntu 14.04 Kernel (e.g. ): Linux dev-k8s-controller-p1 4.4.0-53-generic ~14.04.1-Ubuntu SMP Fri Dec 2 03:43:31 UTC 2016 x8664 x8664 x86_64 GNU/Linux Install tools: chef cookbook Others: Docker 1.12.6, calico 2.1 CNI binaries: kubelet config: What happened: When trying to run pod error message is generated: So it seems that kubelet is trying to find CNI binaries in paths: /opt/loopback/bin /etc/cni/net.d /opt/calico/bin But it should look at since this is the default valur for parameter. What you expected to happen: Pod starts. How to reproduce it (as minimally and precisely as possible): Run pod: with kubelet 1.6 and CNI binaries in those path (default for calico): Anything else we need to know: After copy to and pod starts fine.
Good, very helpful to me
When set and not set , default path of CNI binaries will not take effect, the path specified by will be used instead. We should use and instead of if use CNI plugin.
I will push a PR to deprecate the flag. It is confusing. Closing this issue.
Was that ? It was closed without merging. I agree should be deprecated.
I think is needed before is removed. The flag is still not deprecated, I will send a PR to deprecate it.

# Do not use any network plugin by default. User could override the flags with # test_args. <del> test_args='--kubelet-flags="--network-plugin= --network-plugin-dir=" '$test_args </del> <ins> test_args='--kubelet-flags="--network-plugin= --cni-bin-dir=" '$test_args </ins> # Runtime flags test_args='--kubelet-flags="--container-runtime='$runtime'" '$test_args


Question:
Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Environment: Cloud provider or hardware configuration: bare-metal OS (e.g. from /etc/os-release): Ubuntu 14.04 Kernel (e.g. ): Linux dev-k8s-controller-p1 4.4.0-53-generic ~14.04.1-Ubuntu SMP Fri Dec 2 03:43:31 UTC 2016 x8664 x8664 x86_64 GNU/Linux Install tools: chef cookbook Others: Docker 1.12.6, calico 2.1 CNI binaries: kubelet config: What happened: When trying to run pod error message is generated: So it seems that kubelet is trying to find CNI binaries in paths: /opt/loopback/bin /etc/cni/net.d /opt/calico/bin But it should look at since this is the default valur for parameter. What you expected to happen: Pod starts. How to reproduce it (as minimally and precisely as possible): Run pod: with kubelet 1.6 and CNI binaries in those path (default for calico): Anything else we need to know: After copy to and pod starts fine.
Good, very helpful to me
When set and not set , default path of CNI binaries will not take effect, the path specified by will be used instead. We should use and instead of if use CNI plugin.
I will push a PR to deprecate the flag. It is confusing. Closing this issue.
Was that ? It was closed without merging. I agree should be deprecated.
I think is needed before is removed. The flag is still not deprecated, I will send a PR to deprecate it.

# plugin by default. NETWORK_PLUGIN=${NETWORK_PLUGIN:-""} <del> # NETWORK_PLUGIN_PATH is the path to network plugin binary. NETWORK_PLUGIN_PATH=${NETWORK_PLUGIN_PATH:-""} </del> <ins> # CNI_CONF_DIR is the path to network plugin binaries. CNI_CONF_DIR=${CNI_CONF_DIR:-""} # CNI_BIN_DIR is the path to network plugin config files. CNI_BIN_DIR=${CNI_BIN_DIR:-""} </ins> # start_kubelet starts kubelet and redirect kubelet log to $LOG_DIR/kubelet.log. kubelet_log=kubelet.log


Question:
Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Environment: Cloud provider or hardware configuration: bare-metal OS (e.g. from /etc/os-release): Ubuntu 14.04 Kernel (e.g. ): Linux dev-k8s-controller-p1 4.4.0-53-generic ~14.04.1-Ubuntu SMP Fri Dec 2 03:43:31 UTC 2016 x8664 x8664 x86_64 GNU/Linux Install tools: chef cookbook Others: Docker 1.12.6, calico 2.1 CNI binaries: kubelet config: What happened: When trying to run pod error message is generated: So it seems that kubelet is trying to find CNI binaries in paths: /opt/loopback/bin /etc/cni/net.d /opt/calico/bin But it should look at since this is the default valur for parameter. What you expected to happen: Pod starts. How to reproduce it (as minimally and precisely as possible): Run pod: with kubelet 1.6 and CNI binaries in those path (default for calico): Anything else we need to know: After copy to and pod starts fine.
Good, very helpful to me
When set and not set , default path of CNI binaries will not take effect, the path specified by will be used instead. We should use and instead of if use CNI plugin.
I will push a PR to deprecate the flag. It is confusing. Closing this issue.
Was that ? It was closed without merging. I agree should be deprecated.
I think is needed before is removed. The flag is still not deprecated, I will send a PR to deprecate it.

--system-cgroups=/system  --cgroup-root=/  --network-plugin=$NETWORK_PLUGIN  <del> --network-plugin-dir=$NETWORK_PLUGIN_PATH  </del> <ins> --cni-conf-dir=$CNI_CONF_DIR  --cni-bin-dir=$CNI_BIN_DIR  </ins> --v=$log_level  --logtostderr


Question:
In order to support running tests in parallel, we randomized NodePort selection. There appears to be some evidence that this is not sufficient and is causing e2e test flakes: https://k8s-
Hi, As a need contributor, I'm interesting of solving this issue. I already check the current code, and propose to retry the service creation until success or all possible ports (between FederatedSvcNodePortFirst and FederatedSvcNodePortLast) have been tried. If this approach is fine for you, I can submit a PR soon.
How do you plan to check if the NodePorts are available? In our presubmit test environment, tests run across many instances of federation control planes in parallel, but share the same underlying clusters. So this approach might be viable depending on how and where you plan to check for these used NodePorts.
my approach for getting an available port is a "Dummy" approach: create a list of possible available port, get one of them randomly, then try to create the Service. If Service creation failed. try to use another possible available port, until service creation success or all possible available ports have been tested.
Sounds reasonable to me. Go for it. A couple of caveats: need to create a real list, you could just use the range boundaries. careful about service creation failure. You need to check if the service shard creation failed in the underlying clusters and rollback all those shards before retrying.
thanks for the caveats, I will do soon a PR.
Hi, what is the state of this issue? Is working in this issues with the others issues related to e2e tests?
has a PR to fix this issue -

clusters = f.GetRegisteredClusters() ns = f.FederationNamespace.Name // create backend service <del> service = createLBServiceOrFail(f.FederationClientset, ns, FederatedIngressServiceName) </del> <ins> service = createLBServiceOrFail(f.FederationClientset, ns, FederatedIngressServiceName, clusters) </ins> // create the TLS secret secret = createTLSSecretOrFail(f.FederationClientset, ns, FederatedIngressTLSSecretName) // wait for services objects sync


Question:
In order to support running tests in parallel, we randomized NodePort selection. There appears to be some evidence that this is not sufficient and is causing e2e test flakes: https://k8s-
Hi, As a need contributor, I'm interesting of solving this issue. I already check the current code, and propose to retry the service creation until success or all possible ports (between FederatedSvcNodePortFirst and FederatedSvcNodePortLast) have been tried. If this approach is fine for you, I can submit a PR soon.
How do you plan to check if the NodePorts are available? In our presubmit test environment, tests run across many instances of federation control planes in parallel, but share the same underlying clusters. So this approach might be viable depending on how and where you plan to check for these used NodePorts.
my approach for getting an available port is a "Dummy" approach: create a list of possible available port, get one of them randomly, then try to create the Service. If Service creation failed. try to use another possible available port, until service creation success or all possible available ports have been tested.
Sounds reasonable to me. Go for it. A couple of caveats: need to create a real list, you could just use the range boundaries. careful about service creation failure. You need to check if the service shard creation failed in the underlying clusters and rollback all those shards before retrying.
thanks for the caveats, I will do soon a PR.
Hi, what is the state of this issue? Is working in this issues with the others issues related to e2e tests?
has a PR to fix this issue -

backendPods = createBackendPodsOrFail(clusters, nsName, FederatedServicePodName) <del> service = createLBServiceOrFail(f.FederationClientset, nsName, FederatedServiceName) </del> <ins> service = createLBServiceOrFail(f.FederationClientset, nsName, FederatedServiceName, clusters) </ins> obj, err := scheme.Scheme.DeepCopy(service) // Cloning shouldn't fail. On the off-chance it does, we // should shallow copy service to serviceShard before


Question:
In order to support running tests in parallel, we randomized NodePort selection. There appears to be some evidence that this is not sufficient and is causing e2e test flakes: https://k8s-
Hi, As a need contributor, I'm interesting of solving this issue. I already check the current code, and propose to retry the service creation until success or all possible ports (between FederatedSvcNodePortFirst and FederatedSvcNodePortLast) have been tried. If this approach is fine for you, I can submit a PR soon.
How do you plan to check if the NodePorts are available? In our presubmit test environment, tests run across many instances of federation control planes in parallel, but share the same underlying clusters. So this approach might be viable depending on how and where you plan to check for these used NodePorts.
my approach for getting an available port is a "Dummy" approach: create a list of possible available port, get one of them randomly, then try to create the Service. If Service creation failed. try to use another possible available port, until service creation success or all possible available ports have been tested.
Sounds reasonable to me. Go for it. A couple of caveats: need to create a real list, you could just use the range boundaries. careful about service creation failure. You need to check if the service shard creation failed in the underlying clusters and rollback all those shards before retrying.
thanks for the caveats, I will do soon a PR.
Hi, what is the state of this issue? Is working in this issues with the others issues related to e2e tests?
has a PR to fix this issue -

return clientset.CoreV1().Services(namespace).Create(service) } <del> func createLBService(clientset *fedclientset.Clientset, namespace, name string) (*v1.Service, error) { </del> <ins> func createLBService(clientset *fedclientset.Clientset, namespace, name string, clusters fedframework.ClusterSlice) (*v1.Service, error) { </ins> if clientset == nil || len(namespace) == 0 { return nil, fmt.Errorf("Internal error: invalid parameters passed to createService: clientset: %v, namespace: %v", clientset, namespace) }


Question:
In order to support running tests in parallel, we randomized NodePort selection. There appears to be some evidence that this is not sufficient and is causing e2e test flakes: https://k8s-
Hi, As a need contributor, I'm interesting of solving this issue. I already check the current code, and propose to retry the service creation until success or all possible ports (between FederatedSvcNodePortFirst and FederatedSvcNodePortLast) have been tried. If this approach is fine for you, I can submit a PR soon.
How do you plan to check if the NodePorts are available? In our presubmit test environment, tests run across many instances of federation control planes in parallel, but share the same underlying clusters. So this approach might be viable depending on how and where you plan to check for these used NodePorts.
my approach for getting an available port is a "Dummy" approach: create a list of possible available port, get one of them randomly, then try to create the Service. If Service creation failed. try to use another possible available port, until service creation success or all possible available ports have been tested.
Sounds reasonable to me. Go for it. A couple of caveats: need to create a real list, you could just use the range boundaries. careful about service creation failure. You need to check if the service shard creation failed in the underlying clusters and rollback all those shards before retrying.
thanks for the caveats, I will do soon a PR.
Hi, what is the state of this issue? Is working in this issues with the others issues related to e2e tests?
has a PR to fix this issue -

// Tests can be run in parallel, so we need a different nodePort for // each test. <del> // We add 1 to FederatedSvcNodePortLast because IntnRange's range end // is not inclusive. nodePort := int32(rand.IntnRange(FederatedSvcNodePortFirst, FederatedSvcNodePortLast+1)) </del> <ins> // we add in a array all the "available" ports availablePorts := make([]int32, FederatedSvcNodePortLast-FederatedSvcNodePortFirst) for i := range availablePorts { availablePorts[i] = int32(FederatedSvcNodePortFirst + i) } var err error var service *v1.Service retry := 10 // the function should retry the service creation on different port only 10 time. // until the availablePort list is not empty, lets try to create the service for len(availablePorts) > 0 && retry > 0 { // select the Id of an available port i := rand.Intn(len(availablePorts)) By(fmt.Sprintf("try creating federated service %q in namespace %q with nodePort %d", name, namespace, availablePorts[i])) service, err = createServiceWithNodePort(clientset, namespace, name, availablePorts[i]) if err == nil { // check if service have been created properly in all clusters. // if the service is not present in one of the clusters, we should cleanup all services if err = checkServicesCreation(namespace, name, clusters); err == nil { // everything was created properly so returns the federated service. return service, nil } } // in case of error, cleanup everything if service != nil { if err = deleteService(clientset, namespace, name, nil); err != nil { framework.ExpectNoError(err, "Deleting service %q after a partial createService() error", service.Name) return nil, err } cleanupServiceShardsAndProviderResources(namespace, service, clusters) } // creation failed, lets try with another port // first remove from the availablePorts the port with which the creation failed availablePorts = append(availablePorts[:i], availablePorts[i+1:]...) retry-- } return nil, err } func createServiceWithNodePort(clientset *fedclientset.Clientset, namespace, name string, nodePort int32) (*v1.Service, error) { </ins> service := &v1.Service{ ObjectMeta: metav1.ObjectMeta{ Name:      name,


Question:
In order to support running tests in parallel, we randomized NodePort selection. There appears to be some evidence that this is not sufficient and is causing e2e test flakes: https://k8s-
Hi, As a need contributor, I'm interesting of solving this issue. I already check the current code, and propose to retry the service creation until success or all possible ports (between FederatedSvcNodePortFirst and FederatedSvcNodePortLast) have been tried. If this approach is fine for you, I can submit a PR soon.
How do you plan to check if the NodePorts are available? In our presubmit test environment, tests run across many instances of federation control planes in parallel, but share the same underlying clusters. So this approach might be viable depending on how and where you plan to check for these used NodePorts.
my approach for getting an available port is a "Dummy" approach: create a list of possible available port, get one of them randomly, then try to create the Service. If Service creation failed. try to use another possible available port, until service creation success or all possible available ports have been tested.
Sounds reasonable to me. Go for it. A couple of caveats: need to create a real list, you could just use the range boundaries. careful about service creation failure. You need to check if the service shard creation failed in the underlying clusters and rollback all those shards before retrying.
thanks for the caveats, I will do soon a PR.
Hi, what is the state of this issue? Is working in this issues with the others issues related to e2e tests?
has a PR to fix this issue -

return clientset.CoreV1().Services(namespace).Create(service) } <ins> // checkServicesCreation checks if the service have been created successfuly in all the clusters. // if the service is not present in at least one of the clusters, this function returns an error. func checkServicesCreation(namespace, serviceName string, clusters fedframework.ClusterSlice) error { framework.Logf("check if service %q have been created in %d clusters", serviceName, len(clusters)) for _, cluster := range clusters { name := cluster.Name err := wait.PollImmediate(framework.Poll, fedframework.FederatedDefaultTestTimeout, func() (bool, error) { var err error _, err = cluster.Clientset.CoreV1().Services(namespace).Get(serviceName, metav1.GetOptions{}) if err != nil && !errors.IsNotFound(err) { // Get failed with an error, try again. framework.Logf("Failed to find service %q in namespace %q, in cluster %q: %v. Trying again in %s", serviceName, namespace, name, err, framework.Poll) return false, err } else if errors.IsNotFound(err) { framework.Logf("Service %q in namespace %q in cluster %q not found. Trying again in %s", serviceName, namespace, name, framework.Poll) return false, nil } By(fmt.Sprintf("Service %q in namespace %q in cluster %q found", serviceName, namespace, name)) return true, nil }) if err != nil { return err } } return nil } </ins> func createServiceOrFail(clientset *fedclientset.Clientset, namespace, name string) *v1.Service { service, err := createService(clientset, namespace, name) framework.ExpectNoError(err, "Creating service %q in namespace %q", service.Name, namespace)


Question:
In order to support running tests in parallel, we randomized NodePort selection. There appears to be some evidence that this is not sufficient and is causing e2e test flakes: https://k8s-
Hi, As a need contributor, I'm interesting of solving this issue. I already check the current code, and propose to retry the service creation until success or all possible ports (between FederatedSvcNodePortFirst and FederatedSvcNodePortLast) have been tried. If this approach is fine for you, I can submit a PR soon.
How do you plan to check if the NodePorts are available? In our presubmit test environment, tests run across many instances of federation control planes in parallel, but share the same underlying clusters. So this approach might be viable depending on how and where you plan to check for these used NodePorts.
my approach for getting an available port is a "Dummy" approach: create a list of possible available port, get one of them randomly, then try to create the Service. If Service creation failed. try to use another possible available port, until service creation success or all possible available ports have been tested.
Sounds reasonable to me. Go for it. A couple of caveats: need to create a real list, you could just use the range boundaries. careful about service creation failure. You need to check if the service shard creation failed in the underlying clusters and rollback all those shards before retrying.
thanks for the caveats, I will do soon a PR.
Hi, what is the state of this issue? Is working in this issues with the others issues related to e2e tests?
has a PR to fix this issue -

return service } <del> func createLBServiceOrFail(clientset *fedclientset.Clientset, namespace, name string) *v1.Service { service, err := createLBService(clientset, namespace, name) </del> <ins> func createLBServiceOrFail(clientset *fedclientset.Clientset, namespace, name string, clusters fedframework.ClusterSlice) *v1.Service { service, err := createLBService(clientset, namespace, name, clusters) </ins> framework.ExpectNoError(err, "Creating service %q in namespace %q", service.Name, namespace) By(fmt.Sprintf("Successfully created federated service (type: load balancer) %q in namespace %q", name, namespace)) return service


Question:
In order to support running tests in parallel, we randomized NodePort selection. There appears to be some evidence that this is not sufficient and is causing e2e test flakes: https://k8s-
Hi, As a need contributor, I'm interesting of solving this issue. I already check the current code, and propose to retry the service creation until success or all possible ports (between FederatedSvcNodePortFirst and FederatedSvcNodePortLast) have been tried. If this approach is fine for you, I can submit a PR soon.
How do you plan to check if the NodePorts are available? In our presubmit test environment, tests run across many instances of federation control planes in parallel, but share the same underlying clusters. So this approach might be viable depending on how and where you plan to check for these used NodePorts.
my approach for getting an available port is a "Dummy" approach: create a list of possible available port, get one of them randomly, then try to create the Service. If Service creation failed. try to use another possible available port, until service creation success or all possible available ports have been tested.
Sounds reasonable to me. Go for it. A couple of caveats: need to create a real list, you could just use the range boundaries. careful about service creation failure. You need to check if the service shard creation failed in the underlying clusters and rollback all those shards before retrying.
thanks for the caveats, I will do soon a PR.
Hi, what is the state of this issue? Is working in this issues with the others issues related to e2e tests?
has a PR to fix this issue -

Fail(fmt.Sprintf("Internal error: invalid parameters passed to deleteServiceOrFail: clientset: %v, namespace: %v, service: %v", clientset, namespace, serviceName)) } framework.Logf("Deleting service %q in namespace %v", serviceName, namespace) <ins> err := deleteService(clientset, namespace, serviceName, orphanDependents) if err != nil { framework.ExpectNoError(err, "Error deleting service %q from namespace %q", serviceName, namespace) } } func deleteService(clientset *fedclientset.Clientset, namespace string, serviceName string, orphanDependents *bool) error { </ins> err := clientset.CoreV1().Services(namespace).Delete(serviceName, &metav1.DeleteOptions{OrphanDependents: orphanDependents}) <del> framework.ExpectNoError(err, "Error deleting service %q from namespace %q", serviceName, namespace) </del> <ins> if err != nil { return err } </ins> // Wait for the service to be deleted. err = wait.Poll(5*time.Second, fedframework.FederatedDefaultTestTimeout, func() (bool, error) { _, err := clientset.Core().Services(namespace).Get(serviceName, metav1.GetOptions{})


Question:
In order to support running tests in parallel, we randomized NodePort selection. There appears to be some evidence that this is not sufficient and is causing e2e test flakes: https://k8s-
Hi, As a need contributor, I'm interesting of solving this issue. I already check the current code, and propose to retry the service creation until success or all possible ports (between FederatedSvcNodePortFirst and FederatedSvcNodePortLast) have been tried. If this approach is fine for you, I can submit a PR soon.
How do you plan to check if the NodePorts are available? In our presubmit test environment, tests run across many instances of federation control planes in parallel, but share the same underlying clusters. So this approach might be viable depending on how and where you plan to check for these used NodePorts.
my approach for getting an available port is a "Dummy" approach: create a list of possible available port, get one of them randomly, then try to create the Service. If Service creation failed. try to use another possible available port, until service creation success or all possible available ports have been tested.
Sounds reasonable to me. Go for it. A couple of caveats: need to create a real list, you could just use the range boundaries. careful about service creation failure. You need to check if the service shard creation failed in the underlying clusters and rollback all those shards before retrying.
thanks for the caveats, I will do soon a PR.
Hi, what is the state of this issue? Is working in this issues with the others issues related to e2e tests?
has a PR to fix this issue -

} return false, err }) <del> if err != nil { framework.DescribeSvc(namespace) framework.Failf("Error in deleting service %s: %v", serviceName, err) } </del> <ins> return err </ins> } func cleanupServiceShardsAndProviderResources(namespace string, service *v1.Service, clusters fedframework.ClusterSlice) {


Question: We copy in BUILD files from the saved directories. The BUILD files should be preserved. This was hit in

if [ "${DRY_RUN}" = false ]; then ls "${CLIENT_REPO}" | { grep -v '_tmp' || true; } | xargs rm -rf mv "${CLIENT_REPO_TEMP}"/* "${CLIENT_REPO}" <ins> git checkout HEAD -- $(find "${CLIENT_REPO}" -name BUILD) </ins> fi