Question:
This test is failing in both gce serial suite: http://k8s- /sig network /priority failing-test /priority critical-urgent /kind bug /status approved-for-milestone cc /assign PTAL
xref
/milestone v1.10
this also fails in http://k8s-
Failing to connect to Services at all. I can't even ping pod IPs:
So I can ping pods same-node and cross-node right up until it kills the apiserver. Rebooting fixes a given node. The apiserver demonstrably comes back - I can query it for info...
- I could use help - have to shift attention soon.
FWIW, I am testing on a Calico-enabled node
I'll take a look at this as well.
Weird, now even after a reboot, this condition is sticky. I have a cluster in this state, if you want to poke.
ugh, now it is resolved.
I just repro'ed without the e2e - have a cluster, proved ping works, killed apiserver ping fails.
Interesting - docker shows the pods not running, but apiserver says they are alive.
Update: docker sees no running pods. API server thinks kubelet is running them. kubelet is getting "" from apiserver, even after kubelet restart.
Update and recap: Bring up a cluster Run some pods Prove that pods are running and services work pkill -9 apiserver Wait. Watch kubelets "lose" pods that were assigned to them Observe that pods are not running and services do not work Observe that apiserver says the pods are running pkill -9 kubelet Observe that it is still broken kubectl delete one pod Observe that kubelet is fixed for that one pod, but not others Q: Why is kubelet not getting a full listing when it restarts?
What exactly does it mean for it to be fixed for a deleted pod?
Kubelet started running the pod that was deleted - it got the update for that one.
The cluster I was testing was a week or 2 old. I built a cluster from HEAD and I can not repro - said the same. Is this test running against head of master?
I also ran the test manually against my new cluster and it passed
FWIW I was also unable to reproduce this yesterday on a cluster from HEAD.
Did some investigations and had some findings for this specific failure: Traced down a bit on the history, the first occurrence was https://k8s- From the , seems suspicious because it modified . Also found reported a relevant symptom: The fix () for that was merged a day ago and may explained why we can't repro from HEAD anymore.
nice, is this issue reproducible without the fix ()?
Sorry should have had confirmed this, bringing up a cluster now.
Quick update: I was able to repro the same failure () from HEAD with reverted. Though this test is now flaky on some other errors: https://k8s- https://k8s- They all seem to failed while communicating with apiserver.
Thanks ! So fixed the issue and I am curious about what introduced the issue. Do you happen to know when did this test start to fail? Just trying to find out what went wrong in between. We have some other possibly related issues so finding out the root cause of this one would be super helpful.
/assign
The first occurrence (for that exact failure) I noticed was https://k8s-, which happened on 2018-02-15 09:49 PST.
You all rock! I see a couple greens on it now, but still mostly red?
The test is now flaky mostly on creating an exec pod after restarting apiserver. Some sample failures: To elaborate a bit more, a snippet of the test procedure is: ssh to master and (sending a SIGTERM) Keep polling on /healthz on apiserver until 200 is returned. Create an exec pod for later steps (and failed on various erros --- EOF, connection refused, apiserver terminating). According to above test failures, my suspicion is that the test is assuming the apiserver is restarted too soon. It is possible that apiserver captured SIGTERM and did a graceful termination, so that polling on /healthz may succeed even in the termination phase. I'm not surprised trying to create a pod via a terminating apiserver will hit those errors. Could anyone confirme the SIGTERM theory?
Spoke with folks, sounds like after SIGTERM apiserver does wait a bit for remaining requests to finish. I will take a stab to refine the test. cc
Sent .
Test is now passing on gce but failing on gke. Apparently we don't have a way to trigger apiserver restart on gke at this moment. Sending a PR to skip this test on gke.
[MILESTONENOTIFIER] Milestone Issue Needs Attention Action required: During code freeze, issues in the milestone should be in progress. If this issue is not being actively worked on, please remove it from the milestone. If it is being worked on, please add the label so it can be tracked with other in-flight issues. Note: This issue is marked as , and must be updated every 1 day during code freeze. Example update: <details<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
Commented too quick. On gke we do a same-version master upgrade. Will send another fix :/

return fmt.Errorf("waiting for apiserver timed out") } <del> // WaitForApiserverRestarted waits until apiserver's restart count increased. func WaitForApiserverRestarted(c clientset.Interface, initialRestartCount int32) error { </del> <ins> // waitForApiserverRestarted waits until apiserver's restart count increased. func waitForApiserverRestarted(c clientset.Interface, initialRestartCount int32) error { </ins> for start := time.Now(); time.Since(start) < time.Minute; time.Sleep(5 * time.Second) { <del> restartCount, err := GetApiserverRestartCount(c) </del> <ins> restartCount, err := getApiserverRestartCount(c) </ins> if err != nil { Logf("Failed to get apiserver's restart count: %v", err) continue


Question:
This test is failing in both gce serial suite: http://k8s- /sig network /priority failing-test /priority critical-urgent /kind bug /status approved-for-milestone cc /assign PTAL
xref
/milestone v1.10
this also fails in http://k8s-
Failing to connect to Services at all. I can't even ping pod IPs:
So I can ping pods same-node and cross-node right up until it kills the apiserver. Rebooting fixes a given node. The apiserver demonstrably comes back - I can query it for info...
- I could use help - have to shift attention soon.
FWIW, I am testing on a Calico-enabled node
I'll take a look at this as well.
Weird, now even after a reboot, this condition is sticky. I have a cluster in this state, if you want to poke.
ugh, now it is resolved.
I just repro'ed without the e2e - have a cluster, proved ping works, killed apiserver ping fails.
Interesting - docker shows the pods not running, but apiserver says they are alive.
Update: docker sees no running pods. API server thinks kubelet is running them. kubelet is getting "" from apiserver, even after kubelet restart.
Update and recap: Bring up a cluster Run some pods Prove that pods are running and services work pkill -9 apiserver Wait. Watch kubelets "lose" pods that were assigned to them Observe that pods are not running and services do not work Observe that apiserver says the pods are running pkill -9 kubelet Observe that it is still broken kubectl delete one pod Observe that kubelet is fixed for that one pod, but not others Q: Why is kubelet not getting a full listing when it restarts?
What exactly does it mean for it to be fixed for a deleted pod?
Kubelet started running the pod that was deleted - it got the update for that one.
The cluster I was testing was a week or 2 old. I built a cluster from HEAD and I can not repro - said the same. Is this test running against head of master?
I also ran the test manually against my new cluster and it passed
FWIW I was also unable to reproduce this yesterday on a cluster from HEAD.
Did some investigations and had some findings for this specific failure: Traced down a bit on the history, the first occurrence was https://k8s- From the , seems suspicious because it modified . Also found reported a relevant symptom: The fix () for that was merged a day ago and may explained why we can't repro from HEAD anymore.
nice, is this issue reproducible without the fix ()?
Sorry should have had confirmed this, bringing up a cluster now.
Quick update: I was able to repro the same failure () from HEAD with reverted. Though this test is now flaky on some other errors: https://k8s- https://k8s- They all seem to failed while communicating with apiserver.
Thanks ! So fixed the issue and I am curious about what introduced the issue. Do you happen to know when did this test start to fail? Just trying to find out what went wrong in between. We have some other possibly related issues so finding out the root cause of this one would be super helpful.
/assign
The first occurrence (for that exact failure) I noticed was https://k8s-, which happened on 2018-02-15 09:49 PST.
You all rock! I see a couple greens on it now, but still mostly red?
The test is now flaky mostly on creating an exec pod after restarting apiserver. Some sample failures: To elaborate a bit more, a snippet of the test procedure is: ssh to master and (sending a SIGTERM) Keep polling on /healthz on apiserver until 200 is returned. Create an exec pod for later steps (and failed on various erros --- EOF, connection refused, apiserver terminating). According to above test failures, my suspicion is that the test is assuming the apiserver is restarted too soon. It is possible that apiserver captured SIGTERM and did a graceful termination, so that polling on /healthz may succeed even in the termination phase. I'm not surprised trying to create a pod via a terminating apiserver will hit those errors. Could anyone confirme the SIGTERM theory?
Spoke with folks, sounds like after SIGTERM apiserver does wait a bit for remaining requests to finish. I will take a stab to refine the test. cc
Sent .
Test is now passing on gce but failing on gke. Apparently we don't have a way to trigger apiserver restart on gke at this moment. Sending a PR to skip this test on gke.
[MILESTONENOTIFIER] Milestone Issue Needs Attention Action required: During code freeze, issues in the milestone should be in progress. If this issue is not being actively worked on, please remove it from the milestone. If it is being worked on, please add the label so it can be tracked with other in-flight issues. Note: This issue is marked as , and must be updated every 1 day during code freeze. Example update: <details<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
Commented too quick. On gke we do a same-version master upgrade. Will send another fix :/

return fmt.Errorf("timed out waiting for apiserver to be restarted") } <del> func GetApiserverRestartCount(c clientset.Interface) (int32, error) { </del> <ins> func getApiserverRestartCount(c clientset.Interface) (int32, error) { </ins> label := labels.SelectorFromSet(labels.Set(map[string]string{"component": "kube-apiserver"})) listOpts := metav1.ListOptions{LabelSelector: label.String()} pods, err := c.CoreV1().Pods(metav1.NamespaceSystem).List(listOpts)


Question:
This test is failing in both gce serial suite: http://k8s- /sig network /priority failing-test /priority critical-urgent /kind bug /status approved-for-milestone cc /assign PTAL
xref
/milestone v1.10
this also fails in http://k8s-
Failing to connect to Services at all. I can't even ping pod IPs:
So I can ping pods same-node and cross-node right up until it kills the apiserver. Rebooting fixes a given node. The apiserver demonstrably comes back - I can query it for info...
- I could use help - have to shift attention soon.
FWIW, I am testing on a Calico-enabled node
I'll take a look at this as well.
Weird, now even after a reboot, this condition is sticky. I have a cluster in this state, if you want to poke.
ugh, now it is resolved.
I just repro'ed without the e2e - have a cluster, proved ping works, killed apiserver ping fails.
Interesting - docker shows the pods not running, but apiserver says they are alive.
Update: docker sees no running pods. API server thinks kubelet is running them. kubelet is getting "" from apiserver, even after kubelet restart.
Update and recap: Bring up a cluster Run some pods Prove that pods are running and services work pkill -9 apiserver Wait. Watch kubelets "lose" pods that were assigned to them Observe that pods are not running and services do not work Observe that apiserver says the pods are running pkill -9 kubelet Observe that it is still broken kubectl delete one pod Observe that kubelet is fixed for that one pod, but not others Q: Why is kubelet not getting a full listing when it restarts?
What exactly does it mean for it to be fixed for a deleted pod?
Kubelet started running the pod that was deleted - it got the update for that one.
The cluster I was testing was a week or 2 old. I built a cluster from HEAD and I can not repro - said the same. Is this test running against head of master?
I also ran the test manually against my new cluster and it passed
FWIW I was also unable to reproduce this yesterday on a cluster from HEAD.
Did some investigations and had some findings for this specific failure: Traced down a bit on the history, the first occurrence was https://k8s- From the , seems suspicious because it modified . Also found reported a relevant symptom: The fix () for that was merged a day ago and may explained why we can't repro from HEAD anymore.
nice, is this issue reproducible without the fix ()?
Sorry should have had confirmed this, bringing up a cluster now.
Quick update: I was able to repro the same failure () from HEAD with reverted. Though this test is now flaky on some other errors: https://k8s- https://k8s- They all seem to failed while communicating with apiserver.
Thanks ! So fixed the issue and I am curious about what introduced the issue. Do you happen to know when did this test start to fail? Just trying to find out what went wrong in between. We have some other possibly related issues so finding out the root cause of this one would be super helpful.
/assign
The first occurrence (for that exact failure) I noticed was https://k8s-, which happened on 2018-02-15 09:49 PST.
You all rock! I see a couple greens on it now, but still mostly red?
The test is now flaky mostly on creating an exec pod after restarting apiserver. Some sample failures: To elaborate a bit more, a snippet of the test procedure is: ssh to master and (sending a SIGTERM) Keep polling on /healthz on apiserver until 200 is returned. Create an exec pod for later steps (and failed on various erros --- EOF, connection refused, apiserver terminating). According to above test failures, my suspicion is that the test is assuming the apiserver is restarted too soon. It is possible that apiserver captured SIGTERM and did a graceful termination, so that polling on /healthz may succeed even in the termination phase. I'm not surprised trying to create a pod via a terminating apiserver will hit those errors. Could anyone confirme the SIGTERM theory?
Spoke with folks, sounds like after SIGTERM apiserver does wait a bit for remaining requests to finish. I will take a stab to refine the test. cc
Sent .
Test is now passing on gce but failing on gke. Apparently we don't have a way to trigger apiserver restart on gke at this moment. Sending a PR to skip this test on gke.
[MILESTONENOTIFIER] Milestone Issue Needs Attention Action required: During code freeze, issues in the milestone should be in progress. If this issue is not being actively worked on, please remove it from the milestone. If it is being worked on, please add the label so it can be tracked with other in-flight issues. Note: This issue is marked as , and must be updated every 1 day during code freeze. Example update: <details<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
Commented too quick. On gke we do a same-version master upgrade. Will send another fix :/

framework.ExpectNoError(framework.VerifyServeHostnameServiceUp(cs, ns, host, podNames1, svc1IP, servicePort)) // Restart apiserver <del> initialRestartCount, err := framework.GetApiserverRestartCount(cs) Expect(err).NotTo(HaveOccurred(), "failed to get apiserver's restart count") </del> By("Restarting apiserver") <del> if err := framework.RestartApiserver(cs.Discovery()); err != nil { </del> <ins> if err := framework.RestartApiserver(cs); err != nil { </ins> framework.Failf("error restarting apiserver: %v", err) } <del> By("Waiting for apiserver to be restarted") if err := framework.WaitForApiserverRestarted(cs, initialRestartCount); err != nil { framework.Failf("error while waiting for apiserver to be restarted: %v", err) } </del> By("Waiting for apiserver to come up by polling /healthz") if err := framework.WaitForApiserverUp(cs); err != nil { framework.Failf("error while waiting for apiserver up: %v", err)


Question:
/kind bug /sig storage What happened: When mounting iSCSI PV kubelet doesn't give multipathd enough time to create multipath device and fallbacks to mounting single SCSI device What you expected to happen: it uses multipath device to mount How to reproduce it (as minimally and precisely as possible): create ~5 pods on a single node, each with dynamically provisioned PV (we use Trident for that), observe how some mounts are /dev/sd and not /dev/mapper/ Environment: Kubelet 1.9.3 Baremetal + NetApp ONTAP ove iSCSI + Trident dynamic volume provisioner CoreOS (kernel 4.14.24, multipath-tools 0.6.4)
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
I can reproduce this issue fairly often (although not reliably). The problem is in in the AttachDisk() function. The basic problem is that the method that finds the multipath device looks at each device path one time, and if none of them have an associated multipath device, it just falls back to formatting and mounting the first device path. There is no wait loop for the multipath device to appear. It would be easy to add a wait loop, but that would slow down the non-multipath use case. A possible optimization would be to only wait when 2 or more paths are known, but that still opens up the possibility of incorrect behavior when attaching while all but one paths are down, or when only one path is discovered (for whatever reason).
After thinking about this over lunch, I think a possible fix would be: add a 10 second wait loop for the multipath device if and only if the PV are 2 or more target portals specified. This puts control in the hands of the provisioner without modifying the API at all, and not breaking backwards compatibility. The one corner case I can see is what would happen the PV has 2 portals, but the system is only able to log in to one. This would occur when multipath was correctly configured on the storage controller and in the provisioner, but one of the paths was down (for whatever reason) at attach time. I'm inclined to make the attachment FAIL if the multipath device can't be found within 10 seconds, which would be a change to current behavior, but this can easily be worked around by the cluster administrator setting up multipathd to create devices for single path disks. Provisioners would want to document the recommended way to do that (i.e. set find_multipaths = no, and configure the blacklist appropriately).
is vaguely described as "target portal ips for high availability", without describing what the "high availability" means. Should we assume that it means multipath? Then Kubernetes can always wait for multipath device to appear when more that one is specified. Would it break anything? Does anyone run multiple portals for the same volume without multipath? Is it something we should support? any opinion?
yes, the iscsi portals were made possible by for HA/mpio use case (see ). The lack of concrete description makes it less obvious though.
was it working perfectly with older kube versions and the issue started to pop up recently or from ? Why I am asking is, multipath support was in kube () and atleast I havent seen this issue reported till this report or very recently. Gluster team were also using multipathing feature for quite some time where we always got mpath device mounted reliably, but noticed this issue very recently or with latest OCP builds. iic, There are good number of changes happened in between especially on device discovery/attach ( ex: ) disk paths. I am not concretely saying its a regression or a behavioral change or side effect of some other patch, but trying to figure out if thats the case. any thoughts here ? my understanding was/is that with more target portal IPs means ( atleast) in ISCSI context, but we could have mention as well in the description. Suppose the backup portal IPs are filled by provisioner/admin but initiator havent enabled ( dm_mulitpath is not loaded or deamon service is not running) software in the kubelet/node, what will happen ? Is it going to be infinite wait for failure ? my understanding is that, if multipath is enabled, even if it failed ( for some reason) for backup paths, a mpath device will be created for single path in the system. If thats the case above shouldnt be a concern and device mapper layer will take care of the path discovery later when path becomes stable/available. I dont think so.
we started with 1.9.x on kubelets, so can't tell much about mounting behaviour of previous versions. it is going to fail, but it is fine, nodes should be correctly configured. Lets say you specify fsType, but given FS module is not loaded by kernel, or now mkfs.<FSbinary is available to the kubelet, mount is going to fail and thats OK IMHO. Kubelet shouldn't try to fallback to anything less than was asked, mounting via single portal when multiple portals are specified is similar to trying "default FS" if required fsType can't be used. Yes, default configuration for multipathd is option (note, that default in Linux distributions might be different), which creates multipath device for every non-blacklisted block device.
I'm fairly confident that adding a wait loop for the multipath device in the case when there are 2 or more portals in the PV spec won't cause any regressions. The thing that gives me pause is that there are cases when there's a single portal in the PV spec, but the device still supports multipath and the existing sendtargets command is the mechanism for finding those other paths. I'm wondering if it's possible to detect that case (reliably) and use a wait loop for the multipath device in those cases too.
is the default on RHEL 7, see
Did you mean Multiple Connection Per Session (MC/S)? I don't think this is supported in open-iscsi. Open-iSCSI only supports MPIO (multiple portals)
I don't understand the move by RedHat to default to findmultipaths = no. It basically guarantees bad behavior in cloud-oriented use cases when there's a temporary path failure. My reading of the linked document is that the main motivation for changing the default is to avoid the need to setup a correct blacklist. It seems to me that correct behavior is more important that avoiding some work up front when configuring multipath. No, I'm referring to the . If a storage backend has multiple target portals, but the dynamic provisioner only puts one portal in the PV spec, then as long as that one portal is accessible at attach time, open-iscsi will discover the other portals and log into them. The only weakness to this approach is what happens when that one path is inaccessible at attach time due to a network switch/network port/NIC failure, while the other ports are still available.
, AFAIK multipathd (binary) in RHEL 7 defaults to , but configuration utility they ship writes config with , that utility needs to be called explicitly , IMHO that should be direct scsi mount unconditionally, without multipath involved
The thing that gives me pause is that there are cases when there's a single portal in the PV spec I would be inclined to agree, except that someone the sendtargets logic to the iscsi code. This code exists for no reason other than to allow finding multiple path given a single path. In fact the existing NetApp dynamic provisioner (Trident) relies on that behavior currently to get multipath support (this is something I've fixed and will upstream shortly). If we decide that PVs with exactly one portal should not support multipath, then we should remove that sendtargets logic from the iscsi module in kubelet.

return nil } <ins> func waitForMultiPathToExist(devicePaths []string, maxRetries int, deviceUtil volumeutil.DeviceUtil) string { if 0 == len(devicePaths) { return "" } for i := 0; i < maxRetries; i++ { for _, path := range devicePaths { // There shouldnt be any empty device paths. However adding this check // for safer side to avoid the possibility of an empty entry. if path == "" { continue } // check if the dev is using mpio and if so mount it via the dm-XX device if mappedDevicePath := deviceUtil.FindMultipathDeviceForDevice(path); mappedDevicePath != "" { return mappedDevicePath } } if i == maxRetries-1 { break } time.Sleep(time.Second) } return "" } </ins> // AttachDisk returns devicePath of volume if attach succeeded otherwise returns error func (util *ISCSIUtil) AttachDisk(b iscsiDiskMounter) (string, error) { var devicePath string


Question:
/kind bug /sig storage What happened: When mounting iSCSI PV kubelet doesn't give multipathd enough time to create multipath device and fallbacks to mounting single SCSI device What you expected to happen: it uses multipath device to mount How to reproduce it (as minimally and precisely as possible): create ~5 pods on a single node, each with dynamically provisioned PV (we use Trident for that), observe how some mounts are /dev/sd and not /dev/mapper/ Environment: Kubelet 1.9.3 Baremetal + NetApp ONTAP ove iSCSI + Trident dynamic volume provisioner CoreOS (kernel 4.14.24, multipath-tools 0.6.4)
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
I can reproduce this issue fairly often (although not reliably). The problem is in in the AttachDisk() function. The basic problem is that the method that finds the multipath device looks at each device path one time, and if none of them have an associated multipath device, it just falls back to formatting and mounting the first device path. There is no wait loop for the multipath device to appear. It would be easy to add a wait loop, but that would slow down the non-multipath use case. A possible optimization would be to only wait when 2 or more paths are known, but that still opens up the possibility of incorrect behavior when attaching while all but one paths are down, or when only one path is discovered (for whatever reason).
After thinking about this over lunch, I think a possible fix would be: add a 10 second wait loop for the multipath device if and only if the PV are 2 or more target portals specified. This puts control in the hands of the provisioner without modifying the API at all, and not breaking backwards compatibility. The one corner case I can see is what would happen the PV has 2 portals, but the system is only able to log in to one. This would occur when multipath was correctly configured on the storage controller and in the provisioner, but one of the paths was down (for whatever reason) at attach time. I'm inclined to make the attachment FAIL if the multipath device can't be found within 10 seconds, which would be a change to current behavior, but this can easily be worked around by the cluster administrator setting up multipathd to create devices for single path disks. Provisioners would want to document the recommended way to do that (i.e. set find_multipaths = no, and configure the blacklist appropriately).
is vaguely described as "target portal ips for high availability", without describing what the "high availability" means. Should we assume that it means multipath? Then Kubernetes can always wait for multipath device to appear when more that one is specified. Would it break anything? Does anyone run multiple portals for the same volume without multipath? Is it something we should support? any opinion?
yes, the iscsi portals were made possible by for HA/mpio use case (see ). The lack of concrete description makes it less obvious though.
was it working perfectly with older kube versions and the issue started to pop up recently or from ? Why I am asking is, multipath support was in kube () and atleast I havent seen this issue reported till this report or very recently. Gluster team were also using multipathing feature for quite some time where we always got mpath device mounted reliably, but noticed this issue very recently or with latest OCP builds. iic, There are good number of changes happened in between especially on device discovery/attach ( ex: ) disk paths. I am not concretely saying its a regression or a behavioral change or side effect of some other patch, but trying to figure out if thats the case. any thoughts here ? my understanding was/is that with more target portal IPs means ( atleast) in ISCSI context, but we could have mention as well in the description. Suppose the backup portal IPs are filled by provisioner/admin but initiator havent enabled ( dm_mulitpath is not loaded or deamon service is not running) software in the kubelet/node, what will happen ? Is it going to be infinite wait for failure ? my understanding is that, if multipath is enabled, even if it failed ( for some reason) for backup paths, a mpath device will be created for single path in the system. If thats the case above shouldnt be a concern and device mapper layer will take care of the path discovery later when path becomes stable/available. I dont think so.
we started with 1.9.x on kubelets, so can't tell much about mounting behaviour of previous versions. it is going to fail, but it is fine, nodes should be correctly configured. Lets say you specify fsType, but given FS module is not loaded by kernel, or now mkfs.<FSbinary is available to the kubelet, mount is going to fail and thats OK IMHO. Kubelet shouldn't try to fallback to anything less than was asked, mounting via single portal when multiple portals are specified is similar to trying "default FS" if required fsType can't be used. Yes, default configuration for multipathd is option (note, that default in Linux distributions might be different), which creates multipath device for every non-blacklisted block device.
I'm fairly confident that adding a wait loop for the multipath device in the case when there are 2 or more portals in the PV spec won't cause any regressions. The thing that gives me pause is that there are cases when there's a single portal in the PV spec, but the device still supports multipath and the existing sendtargets command is the mechanism for finding those other paths. I'm wondering if it's possible to detect that case (reliably) and use a wait loop for the multipath device in those cases too.
is the default on RHEL 7, see
Did you mean Multiple Connection Per Session (MC/S)? I don't think this is supported in open-iscsi. Open-iSCSI only supports MPIO (multiple portals)
I don't understand the move by RedHat to default to findmultipaths = no. It basically guarantees bad behavior in cloud-oriented use cases when there's a temporary path failure. My reading of the linked document is that the main motivation for changing the default is to avoid the need to setup a correct blacklist. It seems to me that correct behavior is more important that avoiding some work up front when configuring multipath. No, I'm referring to the . If a storage backend has multiple target portals, but the dynamic provisioner only puts one portal in the PV spec, then as long as that one portal is accessible at attach time, open-iscsi will discover the other portals and log into them. The only weakness to this approach is what happens when that one path is inaccessible at attach time due to a network switch/network port/NIC failure, while the other ports are still available.
, AFAIK multipathd (binary) in RHEL 7 defaults to , but configuration utility they ship writes config with , that utility needs to be called explicitly , IMHO that should be direct scsi mount unconditionally, without multipath involved
The thing that gives me pause is that there are cases when there's a single portal in the PV spec I would be inclined to agree, except that someone the sendtargets logic to the iscsi code. This code exists for no reason other than to allow finding multiple path given a single path. In fact the existing NetApp dynamic provisioner (Trident) relies on that behavior currently to get multipath support (this is something I've fixed and will upstream shortly). If we decide that PVs with exactly one portal should not support multipath, then we should remove that sendtargets logic from the iscsi module in kubelet.

glog.Errorf("iscsi: last error occurred during iscsi init:n%v", lastErr) } <del> //Make sure we use a valid devicepath to find mpio device. devicePath = devicePaths[0] for _, path := range devicePaths { // There shouldnt be any empty device paths. However adding this check // for safer side to avoid the possibility of an empty entry. if path == "" { continue } // check if the dev is using mpio and if so mount it via the dm-XX device if mappedDevicePath := b.deviceUtil.FindMultipathDeviceForDevice(path); mappedDevicePath != "" { devicePath = mappedDevicePath break } </del> <ins> // Try to find a multipath device for the volume if 1 < len(bkpPortal) { // If the PV has 2 or more portals, wait up to 10 seconds for the multipath // device to appear devicePath = waitForMultiPathToExist(devicePaths, 10, b.deviceUtil) } else { // For PVs with 1 portal, just try one time to find the multipath device. This // avoids a long pause when the multipath device will never get created, and // matches legacy behavior. devicePath = waitForMultiPathToExist(devicePaths, 1, b.deviceUtil) } // When no multipath device is found, just use the first (and presumably only) device if devicePath == "" { devicePath = devicePaths[0] </ins> } glog.V(5).Infof("iscsi: AttachDisk devicePath: %s", devicePath)


# When a 'git archive' is exported, the '$Format:%D$' below will look # something like 'HEAD -> release-1.8, tag: v1.8.3' where then 'tag: ' # can be extracted from it. <del> if [[ '$Format:%D$' =~ tag: (v[^ ]+) ]]; then </del> <ins> if [[ '$Format:%D$' =~ tag: (v[^ ,]+) ]]; then </ins> KUBE_GIT_VERSION="${BASH_REMATCH[1]}" fi fi


Question:
Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Service loadbalancer source range don't work in ipvs proxy mode. Which result in gci-gce-ipvs CI failed. http://k8s- How to reproduce it (as minimally and precisely as possible): In gce environment. a service with backend. service type to service util service external IP is specified. the service from pod inside and pod out of the . Environment: Kubernetes version (use ): master Cloud provider or hardware configuration: gce OS (e.g. from /etc/os-release): ubuntu Kernel (e.g. ): Install tools: cluster/kube- Others:
/kind bug /sig network /label ipvs
/area ipvs
The test seems to be flaky now. Can you take a look please?
I don't know why, but I think this flaky have the similiar reason with other normal service-test cases, like Those flaky are pretty annoying, problem is, I have run those cases in my own environment many times, never fail... : ( But still, I'll do my best to find the reason.

} if !proxier.lbWhiteListCIDRSet.isEmpty() || !proxier.lbWhiteListIPSet.isEmpty() { // link kube-services chain -> kube-fire-wall chain <del> args := []string{"-m", "set", "--match-set", proxier.lbIngressSet.Name, "dst,dst", "-j", string(KubeFireWallChain)} if _, err := proxier.iptables.EnsureRule(utiliptables.Append, utiliptables.TableNAT, kubeServicesChain, args...); err != nil { glog.Errorf("Failed to ensure that ipset %s chain %s jumps to %s: %v", proxier.lbIngressSet.Name, kubeServicesChain, KubeFireWallChain, err) </del> <ins> args := []string{ "-A", string(kubeServicesChain), "-m", "set", "--match-set", proxier.lbIngressSet.Name, "dst,dst", "-j", string(KubeFireWallChain), </ins> } <ins> writeLine(proxier.natRules, args...) </ins> if !proxier.lbWhiteListCIDRSet.isEmpty() { args = append(args[:0], "-A", string(KubeFireWallChain),


Question:
Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Service loadbalancer source range don't work in ipvs proxy mode. Which result in gci-gce-ipvs CI failed. http://k8s- How to reproduce it (as minimally and precisely as possible): In gce environment. a service with backend. service type to service util service external IP is specified. the service from pod inside and pod out of the . Environment: Kubernetes version (use ): master Cloud provider or hardware configuration: gce OS (e.g. from /etc/os-release): ubuntu Kernel (e.g. ): Install tools: cluster/kube- Others:
/kind bug /sig network /label ipvs
/area ipvs
The test seems to be flaky now. Can you take a look please?
I don't know why, but I think this flaky have the similiar reason with other normal service-test cases, like Those flaky are pretty annoying, problem is, I have run those cases in my own environment many times, never fail... : ( But still, I'll do my best to find the reason.

"fmt" "net" "reflect" <ins> "strings" </ins> "testing" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"


Question:
Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Service loadbalancer source range don't work in ipvs proxy mode. Which result in gci-gce-ipvs CI failed. http://k8s- How to reproduce it (as minimally and precisely as possible): In gce environment. a service with backend. service type to service util service external IP is specified. the service from pod inside and pod out of the . Environment: Kubernetes version (use ): master Cloud provider or hardware configuration: gce OS (e.g. from /etc/os-release): ubuntu Kernel (e.g. ): Install tools: cluster/kube- Others:
/kind bug /sig network /label ipvs
/area ipvs
The test seems to be flaky now. Can you take a look please?
I don't know why, but I think this flaky have the similiar reason with other normal service-test cases, like Those flaky are pretty annoying, problem is, I have run those cases in my own environment many times, never fail... : ( But still, I'll do my best to find the reason.

t.Errorf("Expect node port type service, got none") } } <ins> func TestLoadBalanceSourceRanges(t *testing.T) { ipt := iptablestest.NewFake() ipvs := ipvstest.NewFake() ipset := ipsettest.NewFake(testIPSetVersion) fp := NewFakeProxier(ipt, ipvs, ipset, nil) svcIP := "10.20.30.41" svcPort := 80 svcLBIP := "1.2.3.4" svcLBSource := "10.0.0.0/8" svcPortName := proxy.ServicePortName{ NamespacedName: makeNSN("ns1", "svc1"), Port:           "p80", } epIP := "10.180.0.1" makeServiceMap(fp, makeTestService(svcPortName.Namespace, svcPortName.Name, func(svc *api.Service) { svc.Spec.Type = "LoadBalancer" svc.Spec.ClusterIP = svcIP svc.Spec.Ports = []api.ServicePort{{ Name:     svcPortName.Port, Port:     int32(svcPort), Protocol: api.ProtocolTCP, }} svc.Status.LoadBalancer.Ingress = []api.LoadBalancerIngress{{ IP: svcLBIP, }} svc.Spec.LoadBalancerSourceRanges = []string{ svcLBSource, } }), ) makeEndpointsMap(fp, makeTestEndpoints(svcPortName.Namespace, svcPortName.Name, func(ept *api.Endpoints) { ept.Subsets = []api.EndpointSubset{{ Addresses: []api.EndpointAddress{{ IP:       epIP, NodeName: strPtr(testHostname), }}, Ports: []api.EndpointPort{{ Name: svcPortName.Port, Port: int32(svcPort), }}, }} }), ) fp.syncProxyRules() // Check ipvs service and destinations services, err := ipvs.GetVirtualServers() if err != nil { t.Errorf("Failed to get ipvs services, err: %v", err) } found := false for _, svc := range services { fmt.Printf("address: %s:%d, %s", svc.Address.String(), svc.Port, svc.Protocol) if svc.Address.Equal(net.ParseIP(svcLBIP)) && svc.Port == uint16(svcPort) && svc.Protocol == string(api.ProtocolTCP) { destinations, _ := ipvs.GetRealServers(svc) if len(destinations) != 1 { t.Errorf("Unexpected %d destinations, expect 0 destinations", len(destinations)) } for _, ep := range destinations { if ep.Address.String() == epIP && ep.Port == uint16(svcPort) { found = true } } } } if !found { t.Errorf("Did not got expected loadbalance service") } // Check ipset entry expectIPSet := map[string]*utilipset.Entry{ KubeLoadBalancerSet: { IP:       svcLBIP, Port:     svcPort, Protocol: strings.ToLower(string(api.ProtocolTCP)), SetType:  utilipset.HashIPPort, }, KubeLoadBalancerMasqSet: { IP:       svcLBIP, Port:     svcPort, Protocol: strings.ToLower(string(api.ProtocolTCP)), SetType:  utilipset.HashIPPort, }, KubeLoadBalancerSourceCIDRSet: { IP:       svcLBIP, Port:     svcPort, Protocol: strings.ToLower(string(api.ProtocolTCP)), Net:      svcLBSource, SetType:  utilipset.HashIPPortNet, }, } for set, entry := range expectIPSet { ents, err := ipset.ListEntries(set) if err != nil || len(ents) != 1 { t.Errorf("Check ipset entries failed for ipset: %q", set) continue } if ents[0] != entry.String() { t.Errorf("Check ipset entries failed for ipset: %q", set) } } // Check iptables chain and rules kubeSvcRules := ipt.GetRules(string(kubeServicesChain)) kubeFWRules := ipt.GetRules(string(KubeFireWallChain)) if !hasJump(kubeSvcRules, string(KubeMarkMasqChain), KubeLoadBalancerMasqSet) { t.Errorf("Didn't find jump from chain %v match set %v to MASQUERADE", kubeServicesChain, KubeLoadBalancerMasqSet) } if !hasJump(kubeSvcRules, string(KubeFireWallChain), KubeLoadBalancerSet) { t.Errorf("Didn't find jump from chain %v match set %v to %v", kubeServicesChain, KubeLoadBalancerSet, KubeFireWallChain) } if !hasJump(kubeFWRules, "ACCEPT", KubeLoadBalancerSourceCIDRSet) { t.Errorf("Didn't find jump from chain %v match set %v to ACCEPT", kubeServicesChain, KubeLoadBalancerSourceCIDRSet) } } </ins> func TestOnlyLocalLoadBalancing(t *testing.T) { ipt := iptablestest.NewFake()


Question:
Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Service loadbalancer source range don't work in ipvs proxy mode. Which result in gci-gce-ipvs CI failed. http://k8s- How to reproduce it (as minimally and precisely as possible): In gce environment. a service with backend. service type to service util service external IP is specified. the service from pod inside and pod out of the . Environment: Kubernetes version (use ): master Cloud provider or hardware configuration: gce OS (e.g. from /etc/os-release): ubuntu Kernel (e.g. ): Install tools: cluster/kube- Others:
/kind bug /sig network /label ipvs
/area ipvs
The test seems to be flaky now. Can you take a look please?
I don't know why, but I think this flaky have the similiar reason with other normal service-test cases, like Those flaky are pretty annoying, problem is, I have run those cases in my own environment many times, never fail... : ( But still, I'll do my best to find the reason.

} } } <ins> func hasJump(rules []iptablestest.Rule, destChain, ipSet string) bool { match := false for _, r := range rules { if r[iptablestest.Jump] == destChain { match = true if ipSet != "" { if strings.Contains(r[iptablestest.MatchSet], ipSet) { return true } match = false } } } return match } </ins>


Question:
Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Service loadbalancer source range don't work in ipvs proxy mode. Which result in gci-gce-ipvs CI failed. http://k8s- How to reproduce it (as minimally and precisely as possible): In gce environment. a service with backend. service type to service util service external IP is specified. the service from pod inside and pod out of the . Environment: Kubernetes version (use ): master Cloud provider or hardware configuration: gce OS (e.g. from /etc/os-release): ubuntu Kernel (e.g. ): Install tools: cluster/kube- Others:
/kind bug /sig network /label ipvs
/area ipvs
The test seems to be flaky now. Can you take a look please?
I don't know why, but I think this flaky have the similiar reason with other normal service-test cases, like Those flaky are pretty annoying, problem is, I have run those cases in my own environment many times, never fail... : ( But still, I'll do my best to find the reason.

Reject      = "REJECT" ToDest      = "--to-destination " Recent      = "recent " <ins> MatchSet    = "--match-set " </ins> ) type Rule map[string]string


Question:
Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Service loadbalancer source range don't work in ipvs proxy mode. Which result in gci-gce-ipvs CI failed. http://k8s- How to reproduce it (as minimally and precisely as possible): In gce environment. a service with backend. service type to service util service external IP is specified. the service from pod inside and pod out of the . Environment: Kubernetes version (use ): master Cloud provider or hardware configuration: gce OS (e.g. from /etc/os-release): ubuntu Kernel (e.g. ): Install tools: cluster/kube- Others:
/kind bug /sig network /label ipvs
/area ipvs
The test seems to be flaky now. Can you take a look please?
I don't know why, but I think this flaky have the similiar reason with other normal service-test cases, like Those flaky are pretty annoying, problem is, I have run those cases in my own environment many times, never fail... : ( But still, I'll do my best to find the reason.

for _, l := range strings.Split(string(f.Lines), "n") { if strings.Contains(l, fmt.Sprintf("-A %v", chainName)) { newRule := Rule(map[string]string{}) <del> for _, arg := range []string{Destination, Source, DPort, Protocol, Jump, ToDest, Recent} { </del> <ins> for _, arg := range []string{Destination, Source, DPort, Protocol, Jump, ToDest, Recent, MatchSet} { </ins> tok := getToken(l, arg) if tok != "" { newRule[arg] = tok


Question:
New tests for subpath consistently fail in ci-kubernetes-e2e-gce-multizone: https://k8s- /kind bug /sig storage
Is this a blocker for 1.10?
I would expect so, yes. /milestone v1.10
I'm on it /assign
The problem is likely due to inline GCE PD volumes not having enough zone information. Considering changing volumes to PVs instead.
fix was in progress in to change to a PV
/unassign
[MILESTONENOTIFIER] Milestone Issue: Up-to-date for process Note: This issue is marked as , and must be updated every 1 day during code freeze. Example update: <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details

} type gcepdSource struct { <del> diskName string </del> <ins> pvc *v1.PersistentVolumeClaim </ins> } func initGCEPD() volSource {


Question:
New tests for subpath consistently fail in ci-kubernetes-e2e-gce-multizone: https://k8s- /kind bug /sig storage
Is this a blocker for 1.10?
I would expect so, yes. /milestone v1.10
I'm on it /assign
The problem is likely due to inline GCE PD volumes not having enough zone information. Considering changing volumes to PVs instead.
fix was in progress in to change to a PV
/unassign
[MILESTONENOTIFIER] Milestone Issue: Up-to-date for process Note: This issue is marked as , and must be updated every 1 day during code freeze. Example update: <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details

func (s *gcepdSource) createVolume(f *framework.Framework) volInfo { var err error <del> framework.Logf("Creating GCE PD volume") s.diskName, err = framework.CreatePDWithRetry() framework.ExpectNoError(err, "Error creating PD") </del> <ins> framework.Logf("Creating GCE PD volume via dynamic provisioning") testCase := storageClassTest{ name:      "subpath", claimSize: "2G", } pvc := newClaim(testCase, f.Namespace.Name, "subpath") s.pvc, err = framework.CreatePVC(f.ClientSet, f.Namespace.Name, pvc) framework.ExpectNoError(err, "Error creating PVC") </ins> return volInfo{ source: &v1.VolumeSource{ <del> GCEPersistentDisk: &v1.GCEPersistentDiskVolumeSource{PDName: s.diskName}, </del> <ins> PersistentVolumeClaim: &v1.PersistentVolumeClaimVolumeSource{ ClaimName: s.pvc.Name, }, </ins> }, } } func (s *gcepdSource) cleanupVolume(f *framework.Framework) { <del> if s.diskName != "" { err := framework.DeletePDWithRetry(s.diskName) framework.ExpectNoError(err, "Error deleting PD") </del> <ins> if s.pvc != nil { err := f.ClientSet.CoreV1().PersistentVolumeClaims(f.Namespace.Name).Delete(s.pvc.Name, nil) framework.ExpectNoError(err, "Error deleting PVC") </ins> } }


Question:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: e2e test "gci-gce" is running in 2 versions: with kube-dns as DNS Service : https://k8s- with CoreDNS as DNS Service : https://k8s- In CoreDNS use case the test is failing What you expected to happen: This e2e tests should not depend on what DNS service is running as default in K8s. CoreDNS is currently BETA, will move to GA soon (after v1.10 though) See : How to reproduce it (as minimally and precisely as possible): look at logs of dashboard : https://k8s- go run -v --test --test_args='--[sig-autoscaling]sDNSshorizontalsautoscalings[DisabledForLargeClusters]skube-dns-autoscalersshouldsscaleskube-dnsspodssinsbothsnonfaultysandsfaultysscenarios$' Anything else we need to know?: Environment: Kubernetes version (use ): branch master Cloud provider or hardware configuration: GCE Install tools: deployment intra-test Others: see https://k8s-
/sig autoscaling
/unsig cli
Looks like the test is searching specifically for a deployment labeled "kube-dns". There are several references to "kube-dns" scattered around it, actually.
Right now the tool "dns-horizontal-autoscaller" seems dedicated for kube-dns. This tools is deployed as "add-on" when enabled by this ENV variable : I guess we should have the definition of another dns-horizontal-autoscaller, dedicated to coredns (unless we generate the yaml file). and the choice of deployment is based on the ENV variable . see
Currently, the translation of the deployment of "dns-horizontal-autoscaller" for CoreDNS is in progress by It is expected that the corresponding e2e test will work as is (as the artifacts are selected by labels, which are the same for CroeDNS and kube-dns).

- --namespace=kube-system - --configmap=kube-dns-autoscaler # Should keep target in sync with cluster/addons/dns/kube-dns.yaml.base <del> - --target=Deployment/kube-dns </del> <ins> - --target={{.Target}} </ins> # When cluster is using large nodes(with more cores), "coresPerReplica" should dominate. # If using small nodes, "nodesPerReplica" should dominate. - --default-params={"linear":{"coresPerReplica":256,"nodesPerReplica":16,"preventSinglePointFailure":true}}


Question:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: e2e test "gci-gce" is running in 2 versions: with kube-dns as DNS Service : https://k8s- with CoreDNS as DNS Service : https://k8s- In CoreDNS use case the test is failing What you expected to happen: This e2e tests should not depend on what DNS service is running as default in K8s. CoreDNS is currently BETA, will move to GA soon (after v1.10 though) See : How to reproduce it (as minimally and precisely as possible): look at logs of dashboard : https://k8s- go run -v --test --test_args='--[sig-autoscaling]sDNSshorizontalsautoscalings[DisabledForLargeClusters]skube-dns-autoscalersshouldsscaleskube-dnsspodssinsbothsnonfaultysandsfaultysscenarios$' Anything else we need to know?: Environment: Kubernetes version (use ): branch master Cloud provider or hardware configuration: GCE Install tools: deployment intra-test Others: see https://k8s-
/sig autoscaling
/unsig cli
Looks like the test is searching specifically for a deployment labeled "kube-dns". There are several references to "kube-dns" scattered around it, actually.
Right now the tool "dns-horizontal-autoscaller" seems dedicated for kube-dns. This tools is deployed as "add-on" when enabled by this ENV variable : I guess we should have the definition of another dns-horizontal-autoscaller, dedicated to coredns (unless we generate the yaml file). and the choice of deployment is based on the ENV variable . see
Currently, the translation of the deployment of "dns-horizontal-autoscaller" for CoreDNS is in progress by It is expected that the corresponding e2e test will work as is (as the artifacts are selected by labels, which are the same for CroeDNS and kube-dns).

apiVersion: v1 kind: Service metadata: <del> name: coredns </del> <ins> name: kube-dns </ins> namespace: kube-system labels: <del> k8s-app: coredns </del> <ins> k8s-app: kube-dns </ins> kubernetes.io/cluster-service: "true" addonmanager.kubernetes.io/mode: Reconcile kubernetes.io/name: "CoreDNS" spec: selector: <del> k8s-app: coredns </del> <ins> k8s-app: kube-dns </ins> clusterIP: __PILLAR__DNS__SERVER__ ports: - name: dns


Question:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: e2e test "gci-gce" is running in 2 versions: with kube-dns as DNS Service : https://k8s- with CoreDNS as DNS Service : https://k8s- In CoreDNS use case the test is failing What you expected to happen: This e2e tests should not depend on what DNS service is running as default in K8s. CoreDNS is currently BETA, will move to GA soon (after v1.10 though) See : How to reproduce it (as minimally and precisely as possible): look at logs of dashboard : https://k8s- go run -v --test --test_args='--[sig-autoscaling]sDNSshorizontalsautoscalings[DisabledForLargeClusters]skube-dns-autoscalersshouldsscaleskube-dnsspodssinsbothsnonfaultysandsfaultysscenarios$' Anything else we need to know?: Environment: Kubernetes version (use ): branch master Cloud provider or hardware configuration: GCE Install tools: deployment intra-test Others: see https://k8s-
/sig autoscaling
/unsig cli
Looks like the test is searching specifically for a deployment labeled "kube-dns". There are several references to "kube-dns" scattered around it, actually.
Right now the tool "dns-horizontal-autoscaller" seems dedicated for kube-dns. This tools is deployed as "add-on" when enabled by this ENV variable : I guess we should have the definition of another dns-horizontal-autoscaller, dedicated to coredns (unless we generate the yaml file). and the choice of deployment is based on the ENV variable . see
Currently, the translation of the deployment of "dns-horizontal-autoscaller" for CoreDNS is in progress by It is expected that the corresponding e2e test will work as is (as the artifacts are selected by labels, which are the same for CroeDNS and kube-dns).

apiVersion: v1 kind: Service metadata: <del> name: coredns </del> <ins> name: kube-dns </ins> namespace: kube-system labels: <del> k8s-app: coredns </del> <ins> k8s-app: kube-dns </ins> kubernetes.io/cluster-service: "true" addonmanager.kubernetes.io/mode: Reconcile kubernetes.io/name: "CoreDNS" spec: selector: <del> k8s-app: coredns </del> <ins> k8s-app: kube-dns </ins> clusterIP: {{ pillar['dns_server'] }} ports: - name: dns


Question:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: e2e test "gci-gce" is running in 2 versions: with kube-dns as DNS Service : https://k8s- with CoreDNS as DNS Service : https://k8s- In CoreDNS use case the test is failing What you expected to happen: This e2e tests should not depend on what DNS service is running as default in K8s. CoreDNS is currently BETA, will move to GA soon (after v1.10 though) See : How to reproduce it (as minimally and precisely as possible): look at logs of dashboard : https://k8s- go run -v --test --test_args='--[sig-autoscaling]sDNSshorizontalsautoscalings[DisabledForLargeClusters]skube-dns-autoscalersshouldsscaleskube-dnsspodssinsbothsnonfaultysandsfaultysscenarios$' Anything else we need to know?: Environment: Kubernetes version (use ): branch master Cloud provider or hardware configuration: GCE Install tools: deployment intra-test Others: see https://k8s-
/sig autoscaling
/unsig cli
Looks like the test is searching specifically for a deployment labeled "kube-dns". There are several references to "kube-dns" scattered around it, actually.
Right now the tool "dns-horizontal-autoscaller" seems dedicated for kube-dns. This tools is deployed as "add-on" when enabled by this ENV variable : I guess we should have the definition of another dns-horizontal-autoscaller, dedicated to coredns (unless we generate the yaml file). and the choice of deployment is based on the ENV variable . see
Currently, the translation of the deployment of "dns-horizontal-autoscaller" for CoreDNS is in progress by It is expected that the corresponding e2e test will work as is (as the artifacts are selected by labels, which are the same for CroeDNS and kube-dns).

name: coredns namespace: kube-system labels: <del> k8s-app: coredns </del> <ins> k8s-app: kube-dns </ins> kubernetes.io/cluster-service: "true" addonmanager.kubernetes.io/mode: Reconcile kubernetes.io/name: "CoreDNS" spec: <del> replicas: 2 </del> <ins> # replicas: not specified here: # 1. In order to make Addon Manager do not reconcile this replicas parameter. # 2. Default is 1. # 3. Will be tuned in real time if DNS horizontal auto-scaling is turned on. </ins> strategy: type: RollingUpdate rollingUpdate: maxUnavailable: 1 selector: matchLabels: <del> k8s-app: coredns </del> <ins> k8s-app: kube-dns </ins> template: metadata: labels: <del> k8s-app: coredns </del> <ins> k8s-app: kube-dns </ins> spec: serviceAccountName: coredns tolerations:


Question:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: e2e test "gci-gce" is running in 2 versions: with kube-dns as DNS Service : https://k8s- with CoreDNS as DNS Service : https://k8s- In CoreDNS use case the test is failing What you expected to happen: This e2e tests should not depend on what DNS service is running as default in K8s. CoreDNS is currently BETA, will move to GA soon (after v1.10 though) See : How to reproduce it (as minimally and precisely as possible): look at logs of dashboard : https://k8s- go run -v --test --test_args='--[sig-autoscaling]sDNSshorizontalsautoscalings[DisabledForLargeClusters]skube-dns-autoscalersshouldsscaleskube-dnsspodssinsbothsnonfaultysandsfaultysscenarios$' Anything else we need to know?: Environment: Kubernetes version (use ): branch master Cloud provider or hardware configuration: GCE Install tools: deployment intra-test Others: see https://k8s-
/sig autoscaling
/unsig cli
Looks like the test is searching specifically for a deployment labeled "kube-dns". There are several references to "kube-dns" scattered around it, actually.
Right now the tool "dns-horizontal-autoscaller" seems dedicated for kube-dns. This tools is deployed as "add-on" when enabled by this ENV variable : I guess we should have the definition of another dns-horizontal-autoscaller, dedicated to coredns (unless we generate the yaml file). and the choice of deployment is based on the ENV variable . see
Currently, the translation of the deployment of "dns-horizontal-autoscaller" for CoreDNS is in progress by It is expected that the corresponding e2e test will work as is (as the artifacts are selected by labels, which are the same for CroeDNS and kube-dns).

apiVersion: v1 kind: Service metadata: <del> name: coredns </del> <ins> name: kube-dns </ins> namespace: kube-system labels: <del> k8s-app: coredns </del> <ins> k8s-app: kube-dns </ins> kubernetes.io/cluster-service: "true" addonmanager.kubernetes.io/mode: Reconcile kubernetes.io/name: "CoreDNS" spec: selector: <del> k8s-app: coredns </del> <ins> k8s-app: kube-dns </ins> clusterIP: $DNS_SERVER_IP ports: - name: dns


Question:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: e2e test "gci-gce" is running in 2 versions: with kube-dns as DNS Service : https://k8s- with CoreDNS as DNS Service : https://k8s- In CoreDNS use case the test is failing What you expected to happen: This e2e tests should not depend on what DNS service is running as default in K8s. CoreDNS is currently BETA, will move to GA soon (after v1.10 though) See : How to reproduce it (as minimally and precisely as possible): look at logs of dashboard : https://k8s- go run -v --test --test_args='--[sig-autoscaling]sDNSshorizontalsautoscalings[DisabledForLargeClusters]skube-dns-autoscalersshouldsscaleskube-dnsspodssinsbothsnonfaultysandsfaultysscenarios$' Anything else we need to know?: Environment: Kubernetes version (use ): branch master Cloud provider or hardware configuration: GCE Install tools: deployment intra-test Others: see https://k8s-
/sig autoscaling
/unsig cli
Looks like the test is searching specifically for a deployment labeled "kube-dns". There are several references to "kube-dns" scattered around it, actually.
Right now the tool "dns-horizontal-autoscaller" seems dedicated for kube-dns. This tools is deployed as "add-on" when enabled by this ENV variable : I guess we should have the definition of another dns-horizontal-autoscaller, dedicated to coredns (unless we generate the yaml file). and the choice of deployment is based on the ENV variable . see
Currently, the translation of the deployment of "dns-horizontal-autoscaller" for CoreDNS is in progress by It is expected that the corresponding e2e test will work as is (as the artifacts are selected by labels, which are the same for CroeDNS and kube-dns).

readonly UUID_MNT_PREFIX="/mnt/disks/by-uuid/google-local-ssds" readonly UUID_BLOCK_PREFIX="/dev/disk/by-uuid/google-local-ssds" <ins> readonly COREDNS_AUTOSCALER="Deployment/coredns" readonly KUBEDNS_AUTOSCALER="Deployment/kube-dns" </ins> # Use --retry-connrefused opt only if it's supported by curl. CURL_RETRY_CONNREFUSED=""


Question:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: e2e test "gci-gce" is running in 2 versions: with kube-dns as DNS Service : https://k8s- with CoreDNS as DNS Service : https://k8s- In CoreDNS use case the test is failing What you expected to happen: This e2e tests should not depend on what DNS service is running as default in K8s. CoreDNS is currently BETA, will move to GA soon (after v1.10 though) See : How to reproduce it (as minimally and precisely as possible): look at logs of dashboard : https://k8s- go run -v --test --test_args='--[sig-autoscaling]sDNSshorizontalsautoscalings[DisabledForLargeClusters]skube-dns-autoscalersshouldsscaleskube-dnsspodssinsbothsnonfaultysandsfaultysscenarios$' Anything else we need to know?: Environment: Kubernetes version (use ): branch master Cloud provider or hardware configuration: GCE Install tools: deployment intra-test Others: see https://k8s-
/sig autoscaling
/unsig cli
Looks like the test is searching specifically for a deployment labeled "kube-dns". There are several references to "kube-dns" scattered around it, actually.
Right now the tool "dns-horizontal-autoscaller" seems dedicated for kube-dns. This tools is deployed as "add-on" when enabled by this ENV variable : I guess we should have the definition of another dns-horizontal-autoscaller, dedicated to coredns (unless we generate the yaml file). and the choice of deployment is based on the ENV variable . see
Currently, the translation of the deployment of "dns-horizontal-autoscaller" for CoreDNS is in progress by It is expected that the corresponding e2e test will work as is (as the artifacts are selected by labels, which are the same for CroeDNS and kube-dns).

sed -i -e "s@{{ *pillar['dns_domain'] *}}@${DNS_DOMAIN}@g" "${coredns_file}" sed -i -e "s@{{ *pillar['dns_server'] *}}@${DNS_SERVER_IP}@g" "${coredns_file}" sed -i -e "s@{{ *pillar['service_cluster_ip_range'] *}}@${SERVICE_CLUSTER_IP_RANGE}@g" "${coredns_file}" <ins> if [[ "${ENABLE_DNS_HORIZONTAL_AUTOSCALER:-}" == "true" ]]; then setup-addon-manifests "addons" "dns-horizontal-autoscaler" "gce" local -r dns_autoscaler_file="${dst_dir}/dns-horizontal-autoscaler/dns-horizontal-autoscaler.yaml" sed -i'' -e "s@{{.Target}}@${COREDNS_AUTOSCALER}@g" "${dns_autoscaler_file}" fi </ins> } # Sets up the manifests of Fluentd configmap and yamls for k8s addons.


Question:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: e2e test "gci-gce" is running in 2 versions: with kube-dns as DNS Service : https://k8s- with CoreDNS as DNS Service : https://k8s- In CoreDNS use case the test is failing What you expected to happen: This e2e tests should not depend on what DNS service is running as default in K8s. CoreDNS is currently BETA, will move to GA soon (after v1.10 though) See : How to reproduce it (as minimally and precisely as possible): look at logs of dashboard : https://k8s- go run -v --test --test_args='--[sig-autoscaling]sDNSshorizontalsautoscalings[DisabledForLargeClusters]skube-dns-autoscalersshouldsscaleskube-dnsspodssinsbothsnonfaultysandsfaultysscenarios$' Anything else we need to know?: Environment: Kubernetes version (use ): branch master Cloud provider or hardware configuration: GCE Install tools: deployment intra-test Others: see https://k8s-
/sig autoscaling
/unsig cli
Looks like the test is searching specifically for a deployment labeled "kube-dns". There are several references to "kube-dns" scattered around it, actually.
Right now the tool "dns-horizontal-autoscaller" seems dedicated for kube-dns. This tools is deployed as "add-on" when enabled by this ENV variable : I guess we should have the definition of another dns-horizontal-autoscaller, dedicated to coredns (unless we generate the yaml file). and the choice of deployment is based on the ENV variable . see
Currently, the translation of the deployment of "dns-horizontal-autoscaller" for CoreDNS is in progress by It is expected that the corresponding e2e test will work as is (as the artifacts are selected by labels, which are the same for CroeDNS and kube-dns).

if [[ "${ENABLE_DNS_HORIZONTAL_AUTOSCALER:-}" == "true" ]]; then setup-addon-manifests "addons" "dns-horizontal-autoscaler" "gce" <ins> local -r dns_autoscaler_file="${dst_dir}/dns-horizontal-autoscaler/dns-horizontal-autoscaler.yaml" sed -i'' -e "s@{{.Target}}@${KUBEDNS_AUTOSCALER}@g" "${dns_autoscaler_file}" </ins> fi }


Question:
/sig gcp /sig cluster-life-cycle /kind flake /milestone v1.10 cc https://k8s- starting last night /assign can one of you verify?
/kind bug /priority important-soon
/priority critical-urgent /priority failing-test /status approved-for-milestone
[MILESTONENOTIFIER] Milestone Issue Needs Attention Action required: During code freeze, issues in the milestone should be in progress. If this issue is not being actively worked on, please remove it from the milestone. If it is being worked on, please add the label so it can be tracked with other in-flight issues. Note: This issue is marked as , and must be updated every 1 day during code freeze. Example update: <details<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
/assign according to , gonna move those jobs to us-central1-c instead

containers: - name: influxdb image: gcr.io/google_containers/heapster_influxdb:v0.3 <ins> ports: - containerPort: 8083 hostPort: 8083 - containerPort: 8086 hostPort: 8086 </ins> - name: grafana image: gcr.io/google_containers/heapster_grafana:v0.6 env:


Question:
/sig gcp /sig cluster-life-cycle /kind flake /milestone v1.10 cc https://k8s- starting last night /assign can one of you verify?
/kind bug /priority important-soon
/priority critical-urgent /priority failing-test /status approved-for-milestone
[MILESTONENOTIFIER] Milestone Issue Needs Attention Action required: During code freeze, issues in the milestone should be in progress. If this issue is not being actively worked on, please remove it from the milestone. If it is being worked on, please add the label so it can be tracked with other in-flight issues. Note: This issue is marked as , and must be updated every 1 day during code freeze. Example update: <details<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
/assign according to , gonna move those jobs to us-central1-c instead

containers: - image: gcr.io/google_containers/heapster_influxdb:v0.3 name: influxdb <ins> ports: - containerPort: 8083 hostPort: 8083 - containerPort: 8086 hostPort: 8086 </ins> - name: grafana image: gcr.io/google_containers/heapster_grafana:v0.6 env:


Question:
/kind bug What happened: We had a pod with persistent storage. We drained the node. The node VM was deleted. The PV was not attached to the node that the Pod migrated too. The kube-controller-manager logs contained the following. What you expected to happen: The PV can be attached to the node it was migrated to. How to reproduce it (as minimally and precisely as possible): Create a 2 node cluster on vSphere Deploy a pod with persistent storage attached Drain the node Delete the node VM immediately after draining Anything else we need to know?: The issue seems to be in the call to getVSphereInstance. This will return an error if the Node doesn't exist. Precluding the logic on line 804-813 from being executed. A restart of the kube-controller-manager resolved the issue. The following e2e test is not triggered in vSphere environments - Which might expose the issue. Environment: Kubernetes version (use ): 1.9.5 Cloud provider or hardware configuration: vSphere OS (e.g. from /etc/os-release): ubuntu Kernel (e.g. ): Install tools: CFCR Others: /sig storage /sig vmware
More logs:
/assign

defer cancel() vsi, err := vs.getVSphereInstance(nodeName) if err != nil { <ins> // If node doesn't exist, disk is already detached from node. if err == vclib.ErrNoVMFound { glog.Infof("Node %q does not exist, disk %s is already detached from node.", convertToString(nodeName), volPath) return nil } </ins> return err } // Ensure client is logged in and session is valid


Question:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: Pods stuck in terminating state on Windows node after deleting it What you expected to happen: Pods get terminated and should disappear from kubectl How to reproduce it (as minimally and precisely as possible): Run a pod with windows container Delete it Pods are still terminating Anything else we need to know?: Could you pls help to resolve this issue. K8s pods stuck as Terminating after getting deleted on windows node with windows container(IIS). Docker ps -a not showing this container, so it is completely deleted, but in kubectl it is marked as Terminating. For networking we using OVN. kubectl logs: kubelet logs: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: Hyper-V VMs OS (e.g. from /etc/os-release): Windows Node: Master Node: NAME="Ubuntu" VERSION="16.04.4 LTS (Xenial Xerus)" ID=ubuntu IDLIKE=debian PRETTYNAME="Ubuntu 16.04.4 LTS" VERSIONID="16.04" HOMEURL="" SUPPORTURL="" BUGREPORTURL="" VERSIONCODENAME=xenial UBUNTUCODENAME=xenial Kernel (e.g. ): Linux esd-kbrnts-lnx-1 4.13.0-37-generic ~16.04.1-Ubuntu SMP Wed Mar 7 16:03:28 UTC 2018 x8664 x8664 x8664 GNU/Linux Install tools: OVN 2.7 on windows OVN 2.7.2 on linux Others: Docker Version on Windows node: Server Version: 17.06.2-ee-6
/sig windows
It needs sometime for kubelet to remove the pod. Could you wait a while and check whether the Pod removed?
Hi actually I have waited for 1 day, and it is not deleting. Can delete it only using --grace-period=0 --force. But I don't want to use this manual approach). Could you pls give some advice on this? Thanks in advance!
is a reference to this issue. Checking kubelet logs will always help to troubleshoot such problem.
Actually, I have attached kubelet log in the ticket and think the main reason is Also as there any option for kubelet to automatically force delete Terminating pods after some period? And --containerized is also not solving my issue. Thanks!
Kubelet won't delete the pod from apiserver. Since it is already not exist on the node, you could force delete it by kubectl.
It is clear, so how we can resolve this error: ? I think this is the core item of our issue. Thanks.
I'm not using OVN. Have you met this problem when using OVN?
adding Bob to comment on this. i have seen this issue before, but it has been quite a while.
Hi, Guys will be very grateful if you will provide some solution for it. We have tried to add to deployment spec but it doesn't help. Thanks.
The error message of is actually a warning message. It shouldn't block the removal of pods. Could you recheck the following things: the node, check whether the pod is still existing, e.g. by the pod exists, check if there are other error or warning message in kubelet logs; or else, the pod could be force removed safely from apiserver
Hi, I had posted this info in the ticket description. not showing this container, so it is completely deleted, but in kubectl it is displayed as Terminating.
i think you may get more support and faster responses in our slack channel for sig-windows.
So it could be safely force removed from API. And agreed with slack is a more faster way to such troubleshooting issues.
The issue seems to be related to . The only platform specific implementation for PodContainerManager is the one for . There is a default implementation (), that always returns true in the Exists call. Disabling the CgroupsPerQOS option on the Windows nodes resolves the issue.
Windows doesn't support cgroups. You should always disable it.
it is clear enough that Windows does not support cgroups). Unfortunately, it's not specified in the documentation. So to disable CgroupsPerQOS you should need to add this flags to kubelet: Maybe it will help someone... Thank you all for discussion!

go_library( name = "go_default_library", <del> srcs = ["validation.go"], </del> <ins> srcs = [ "validation.go", ] + select({ "@io_bazel_rules_go//go/platform:android": [ "validation_others.go", ], "@io_bazel_rules_go//go/platform:darwin": [ "validation_others.go", ], "@io_bazel_rules_go//go/platform:dragonfly": [ "validation_others.go", ], "@io_bazel_rules_go//go/platform:freebsd": [ "validation_others.go", ], "@io_bazel_rules_go//go/platform:linux": [ "validation_others.go", ], "@io_bazel_rules_go//go/platform:nacl": [ "validation_others.go", ], "@io_bazel_rules_go//go/platform:netbsd": [ "validation_others.go", ], "@io_bazel_rules_go//go/platform:openbsd": [ "validation_others.go", ], "@io_bazel_rules_go//go/platform:plan9": [ "validation_others.go", ], "@io_bazel_rules_go//go/platform:solaris": [ "validation_others.go", ], "@io_bazel_rules_go//go/platform:windows": [ "validation_windows.go", ], "//conditions:default": [], }), </ins> importpath = "k8s.io/kubernetes/pkg/kubelet/apis/kubeletconfig/validation", deps = [ "//pkg/features:go_default_library",


Question:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: Pods stuck in terminating state on Windows node after deleting it What you expected to happen: Pods get terminated and should disappear from kubectl How to reproduce it (as minimally and precisely as possible): Run a pod with windows container Delete it Pods are still terminating Anything else we need to know?: Could you pls help to resolve this issue. K8s pods stuck as Terminating after getting deleted on windows node with windows container(IIS). Docker ps -a not showing this container, so it is completely deleted, but in kubectl it is marked as Terminating. For networking we using OVN. kubectl logs: kubelet logs: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: Hyper-V VMs OS (e.g. from /etc/os-release): Windows Node: Master Node: NAME="Ubuntu" VERSION="16.04.4 LTS (Xenial Xerus)" ID=ubuntu IDLIKE=debian PRETTYNAME="Ubuntu 16.04.4 LTS" VERSIONID="16.04" HOMEURL="" SUPPORTURL="" BUGREPORTURL="" VERSIONCODENAME=xenial UBUNTUCODENAME=xenial Kernel (e.g. ): Linux esd-kbrnts-lnx-1 4.13.0-37-generic ~16.04.1-Ubuntu SMP Wed Mar 7 16:03:28 UTC 2018 x8664 x8664 x8664 GNU/Linux Install tools: OVN 2.7 on windows OVN 2.7.2 on linux Others: Docker Version on Windows node: Server Version: 17.06.2-ee-6
/sig windows
It needs sometime for kubelet to remove the pod. Could you wait a while and check whether the Pod removed?
Hi actually I have waited for 1 day, and it is not deleting. Can delete it only using --grace-period=0 --force. But I don't want to use this manual approach). Could you pls give some advice on this? Thanks in advance!
is a reference to this issue. Checking kubelet logs will always help to troubleshoot such problem.
Actually, I have attached kubelet log in the ticket and think the main reason is Also as there any option for kubelet to automatically force delete Terminating pods after some period? And --containerized is also not solving my issue. Thanks!
Kubelet won't delete the pod from apiserver. Since it is already not exist on the node, you could force delete it by kubectl.
It is clear, so how we can resolve this error: ? I think this is the core item of our issue. Thanks.
I'm not using OVN. Have you met this problem when using OVN?
adding Bob to comment on this. i have seen this issue before, but it has been quite a while.
Hi, Guys will be very grateful if you will provide some solution for it. We have tried to add to deployment spec but it doesn't help. Thanks.
The error message of is actually a warning message. It shouldn't block the removal of pods. Could you recheck the following things: the node, check whether the pod is still existing, e.g. by the pod exists, check if there are other error or warning message in kubelet logs; or else, the pod could be force removed safely from apiserver
Hi, I had posted this info in the ticket description. not showing this container, so it is completely deleted, but in kubectl it is displayed as Terminating.
i think you may get more support and faster responses in our slack channel for sig-windows.
So it could be safely force removed from API. And agreed with slack is a more faster way to such troubleshooting issues.
The issue seems to be related to . The only platform specific implementation for PodContainerManager is the one for . There is a default implementation (), that always returns true in the Exists call. Disabling the CgroupsPerQOS option on the Windows nodes resolves the issue.
Windows doesn't support cgroups. You should always disable it.
it is clear enough that Windows does not support cgroups). Unfortunately, it's not specified in the documentation. So to disable CgroupsPerQOS you should need to add this flags to kubelet: Maybe it will help someone... Thank you all for discussion!

allErrors = append(allErrors, fmt.Errorf("invalid configuration: option %q specified for HairpinMode (--hairpin-mode). Valid options are %q, %q or %q", kc.HairpinMode, kubeletconfig.HairpinNone, kubeletconfig.HairpinVeth, kubeletconfig.PromiscuousBridge)) } <ins> if err := validateKubeletOSConfiguration(kc); err != nil { allErrors = append(allErrors, err) } </ins> return utilerrors.NewAggregate(allErrors) }


Question:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: Pods stuck in terminating state on Windows node after deleting it What you expected to happen: Pods get terminated and should disappear from kubectl How to reproduce it (as minimally and precisely as possible): Run a pod with windows container Delete it Pods are still terminating Anything else we need to know?: Could you pls help to resolve this issue. K8s pods stuck as Terminating after getting deleted on windows node with windows container(IIS). Docker ps -a not showing this container, so it is completely deleted, but in kubectl it is marked as Terminating. For networking we using OVN. kubectl logs: kubelet logs: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: Hyper-V VMs OS (e.g. from /etc/os-release): Windows Node: Master Node: NAME="Ubuntu" VERSION="16.04.4 LTS (Xenial Xerus)" ID=ubuntu IDLIKE=debian PRETTYNAME="Ubuntu 16.04.4 LTS" VERSIONID="16.04" HOMEURL="" SUPPORTURL="" BUGREPORTURL="" VERSIONCODENAME=xenial UBUNTUCODENAME=xenial Kernel (e.g. ): Linux esd-kbrnts-lnx-1 4.13.0-37-generic ~16.04.1-Ubuntu SMP Wed Mar 7 16:03:28 UTC 2018 x8664 x8664 x8664 GNU/Linux Install tools: OVN 2.7 on windows OVN 2.7.2 on linux Others: Docker Version on Windows node: Server Version: 17.06.2-ee-6
/sig windows
It needs sometime for kubelet to remove the pod. Could you wait a while and check whether the Pod removed?
Hi actually I have waited for 1 day, and it is not deleting. Can delete it only using --grace-period=0 --force. But I don't want to use this manual approach). Could you pls give some advice on this? Thanks in advance!
is a reference to this issue. Checking kubelet logs will always help to troubleshoot such problem.
Actually, I have attached kubelet log in the ticket and think the main reason is Also as there any option for kubelet to automatically force delete Terminating pods after some period? And --containerized is also not solving my issue. Thanks!
Kubelet won't delete the pod from apiserver. Since it is already not exist on the node, you could force delete it by kubectl.
It is clear, so how we can resolve this error: ? I think this is the core item of our issue. Thanks.
I'm not using OVN. Have you met this problem when using OVN?
adding Bob to comment on this. i have seen this issue before, but it has been quite a while.
Hi, Guys will be very grateful if you will provide some solution for it. We have tried to add to deployment spec but it doesn't help. Thanks.
The error message of is actually a warning message. It shouldn't block the removal of pods. Could you recheck the following things: the node, check whether the pod is still existing, e.g. by the pod exists, check if there are other error or warning message in kubelet logs; or else, the pod could be force removed safely from apiserver
Hi, I had posted this info in the ticket description. not showing this container, so it is completely deleted, but in kubectl it is displayed as Terminating.
i think you may get more support and faster responses in our slack channel for sig-windows.
So it could be safely force removed from API. And agreed with slack is a more faster way to such troubleshooting issues.
The issue seems to be related to . The only platform specific implementation for PodContainerManager is the one for . There is a default implementation (), that always returns true in the Exists call. Disabling the CgroupsPerQOS option on the Windows nodes resolves the issue.
Windows doesn't support cgroups. You should always disable it.
it is clear enough that Windows does not support cgroups). Unfortunately, it's not specified in the documentation. So to disable CgroupsPerQOS you should need to add this flags to kubelet: Maybe it will help someone... Thank you all for discussion!

<ins> // +build !windows /* Copyright 2018 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package validation import ( "k8s.io/kubernetes/pkg/kubelet/apis/kubeletconfig" ) // validateKubeletOSConfiguration validates os specific kubelet configuration and returns an error if it is invalid. func validateKubeletOSConfiguration(kc *kubeletconfig.KubeletConfiguration) error { return nil } </ins>


Question:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: Pods stuck in terminating state on Windows node after deleting it What you expected to happen: Pods get terminated and should disappear from kubectl How to reproduce it (as minimally and precisely as possible): Run a pod with windows container Delete it Pods are still terminating Anything else we need to know?: Could you pls help to resolve this issue. K8s pods stuck as Terminating after getting deleted on windows node with windows container(IIS). Docker ps -a not showing this container, so it is completely deleted, but in kubectl it is marked as Terminating. For networking we using OVN. kubectl logs: kubelet logs: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: Hyper-V VMs OS (e.g. from /etc/os-release): Windows Node: Master Node: NAME="Ubuntu" VERSION="16.04.4 LTS (Xenial Xerus)" ID=ubuntu IDLIKE=debian PRETTYNAME="Ubuntu 16.04.4 LTS" VERSIONID="16.04" HOMEURL="" SUPPORTURL="" BUGREPORTURL="" VERSIONCODENAME=xenial UBUNTUCODENAME=xenial Kernel (e.g. ): Linux esd-kbrnts-lnx-1 4.13.0-37-generic ~16.04.1-Ubuntu SMP Wed Mar 7 16:03:28 UTC 2018 x8664 x8664 x8664 GNU/Linux Install tools: OVN 2.7 on windows OVN 2.7.2 on linux Others: Docker Version on Windows node: Server Version: 17.06.2-ee-6
/sig windows
It needs sometime for kubelet to remove the pod. Could you wait a while and check whether the Pod removed?
Hi actually I have waited for 1 day, and it is not deleting. Can delete it only using --grace-period=0 --force. But I don't want to use this manual approach). Could you pls give some advice on this? Thanks in advance!
is a reference to this issue. Checking kubelet logs will always help to troubleshoot such problem.
Actually, I have attached kubelet log in the ticket and think the main reason is Also as there any option for kubelet to automatically force delete Terminating pods after some period? And --containerized is also not solving my issue. Thanks!
Kubelet won't delete the pod from apiserver. Since it is already not exist on the node, you could force delete it by kubectl.
It is clear, so how we can resolve this error: ? I think this is the core item of our issue. Thanks.
I'm not using OVN. Have you met this problem when using OVN?
adding Bob to comment on this. i have seen this issue before, but it has been quite a while.
Hi, Guys will be very grateful if you will provide some solution for it. We have tried to add to deployment spec but it doesn't help. Thanks.
The error message of is actually a warning message. It shouldn't block the removal of pods. Could you recheck the following things: the node, check whether the pod is still existing, e.g. by the pod exists, check if there are other error or warning message in kubelet logs; or else, the pod could be force removed safely from apiserver
Hi, I had posted this info in the ticket description. not showing this container, so it is completely deleted, but in kubectl it is displayed as Terminating.
i think you may get more support and faster responses in our slack channel for sig-windows.
So it could be safely force removed from API. And agreed with slack is a more faster way to such troubleshooting issues.
The issue seems to be related to . The only platform specific implementation for PodContainerManager is the one for . There is a default implementation (), that always returns true in the Exists call. Disabling the CgroupsPerQOS option on the Windows nodes resolves the issue.
Windows doesn't support cgroups. You should always disable it.
it is clear enough that Windows does not support cgroups). Unfortunately, it's not specified in the documentation. So to disable CgroupsPerQOS you should need to add this flags to kubelet: Maybe it will help someone... Thank you all for discussion!

<ins> // +build windows /* Copyright 2018 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package validation import ( "fmt" utilerrors "k8s.io/apimachinery/pkg/util/errors" "k8s.io/kubernetes/pkg/kubelet/apis/kubeletconfig" ) // validateKubeletOSConfiguration validates os specific kubelet configuration and returns an error if it is invalid. func validateKubeletOSConfiguration(kc *kubeletconfig.KubeletConfiguration) error { message := "invalid configuration: %v (%v) %v is not supported on Windows" allErrors := []error{} if kc.CgroupsPerQOS { allErrors = append(allErrors, fmt.Errorf(message, "CgroupsPerQOS", "--cgroups-per-qos", kc.CgroupsPerQOS)) } if len(kc.EnforceNodeAllocatable) > 0 { allErrors = append(allErrors, fmt.Errorf(message, "EnforceNodeAllocatable", "--enforce-node-allocatable", kc.EnforceNodeAllocatable)) } return utilerrors.NewAggregate(allErrors) } </ins>


Question:
Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: In the kubelet, when using the kubelet creates static pods but doesn't update the status accordingly in the PodList (kubelet ). You can find that kind of Pods with the following annotation: The status will stay like that: What you expected to happen: The status is updated. How to reproduce it (as minimally and precisely as possible): Use a static pod. Anything else we need to know?: I tried to fix it in but the test-grid continuously failed at the reboot phase (see ). I had to revert it (, ). I'm creating this issue to track this bug. The problem in the e2e tests are probably from . This is an old output of the failure: https://k8s-
I don't think this is a bug. only exposes what kubelet sees from the sources, and doesn't include statuses known only to kubelet. Since for static pods, their statuses are not written back to the source, they do not show up in . The fact that it has the condition is unintentional and a bug. It has just been fixed recently.
I confirm that this behaviour is still happening. Having pods with status where they are actually . The kubernetes API shows them as when Kubelet endpoint as .
/sig node
I agree with that this is working as intended.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
I understand that the source of truth is currently the API server and that details can be retrieved there. We have however found this to be a scaling challenge. In our use case, we are a monitoring application that runs as a DaemonSet. Monitoring systems such as ours need to continuously refresh the PodList because Kubernetes operators want a real time feedback on application health via metrics, logs and traces. Having each node that is monitoring query the Kubernetes apiserver to get each of the Pod statuses results in high load on the API server. This is exacerbated and becomes difficult to scale as clusters grow to a large number of nodes. We have however found that having each node query its own Kubelet API locally is quite scaleable and allows this work to be distributed across the cluster. Ideally we would be able to use this API for static pods as well since the state of the static Pods is stored inside the kubelet. Other benefits of using the kublet API are that: It provides the link to the container runtime in the section, which is helpful for metadata collection and association. Is easily managed the RBAC system through webhooks If there is another approach or API that you’d suggest we’re happy to rethink how we collect this data.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
I also meet this issue.
Reporting the status as "Pending" might be consistent, but it's incorrect. If the pod is created statically and is health checking OK, it's not "Pending". The Kubelet API is almost very useful but the status flag seems to make it much less so since it's not possible to tell from the API endpoint if a static Pod is actually up successfully or not.
Hi team, I'm reaching out to see if we can revisit this issue. We need this to be able to monitor pods locally without having agents reach out to the apiserver. Currently the pod is in a wrong state (Pending when it's actually Running), and we're missing all status information (pod IP, container statuses, etc.). We would be happy and willing to update to solve the issue if we get the green light that this will be accepted. cc
I think as long as we fix the test failures the change caused, we can re-introduce it. Feel free to assign me on a PR which accomplishes this, and make sure the reboot test passes.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale

// GetPods returns all pods bound to the kubelet and their spec, and the mirror // pods. func (kl *Kubelet) GetPods() []*v1.Pod { <del> return kl.podManager.GetPods() </del> <ins> pods := kl.podManager.GetPods() // a kubelet running without apiserver requires an additional // update of the static pod status. See #57106 for _, p := range pods { if status, ok := kl.statusManager.GetPodStatus(p.UID); ok { p.Status = status } } return pods </ins> } // GetRunningPods returns all pods running on kubelet from looking at the


Question:
Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: says: What you expected to happen: Don't suggest that -- as it doesn't actually do that when used. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: (This was reported in but that issue was closed) Environment: Kubernetes version (use ): (This is indeed head from today) Cloud provider or hardware configuration: macOS OS (e.g. from /etc/os-release): Kernel (e.g. ):Install tools:Others:
/sig cli
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale

// whether or not a container name was given via --container ContainerNameSpecified bool <del> Interactive            bool </del> Selector               string Object           runtime.Object