Question:
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that InterPodAntiAffinity is respected if matching 2 {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that InterPodAntiAffinity is respected if matching 2 {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that InterPodAntiAffinity is respected if matching 2 {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that InterPodAntiAffinity is respected if matching 2 {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that InterPodAntiAffinity is respected if matching 2 {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that InterPodAntiAffinity is respected if matching 2 {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that InterPodAntiAffinity is respected if matching 2 {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that InterPodAntiAffinity is respected if matching 2 {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that InterPodAntiAffinity is respected if matching 2 {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that InterPodAntiAffinity is respected if matching 2 {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that InterPodAntiAffinity is respected if matching 2 {Kubernetes e2e suite}
Looking into this...
trying to track this flake down. It looks like the pod is launched despite AntiAffinity in . In my local setup I saw the same, but noticed that the new admission controller was not activate. I have activate it and now get this:
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that InterPodAntiAffinity is respected if matching 2 {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that InterPodAntiAffinity is respected if matching 2 {Kubernetes e2e suite}
Should be fixed by

// test when the pod anti affinity rule is not satisfied, the pod would stay pending. It("validates that InterPodAntiAffinity is respected if matching 2", func() { <del> // launch a pod to find a node which can launch a pod. We intentionally do // not just take the node list and choose the first of them. Depending on the // cluster and the scheduler it might be that a "normal" pod cannot be // scheduled onto it. By("Trying to launch a pod with a label to get a node which can launch it.") </del> <ins> // launch pods to find nodes which can launch a pod. We intentionally do // not just take the node list and choose the first and the second of them. // Depending on the cluster and the scheduler it might be that a "normal" pod // cannot be scheduled onto it. By("Launching two pods on two distinct nodes to get two node names") CreateHostPortPods(f, "host-port", 2, true) defer framework.DeleteRC(f.Client, f.Namespace.Name, "host-port") podList, err := c.Pods(ns).List(api.ListOptions{}) ExpectNoError(err) Expect(len(podList.Items)).To(Equal(2)) nodeNames := []string{podList.Items[0].Spec.NodeName, podList.Items[1].Spec.NodeName} Expect(nodeNames[0]).ToNot(Equal(nodeNames[1])) By("Applying a random label to both nodes.") k := "e2e.inter-pod-affinity.kubernetes.io/zone" v := "china-e2etest" for _, nodeName := range nodeNames { framework.AddOrUpdateLabelOnNode(c, nodeName, k, v) framework.ExpectNodeHasLabel(c, nodeName, k, v) defer framework.RemoveLabelOffNode(c, nodeName, k) } By("Trying to launch another pod on the first node with the service label.") </ins> podName := "with-label-" + string(uuid.NewUUID()) pod, err := c.Pods(ns).Create(&api.Pod{ TypeMeta: unversioned.TypeMeta{


Question:
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that InterPodAntiAffinity is respected if matching 2 {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that InterPodAntiAffinity is respected if matching 2 {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that InterPodAntiAffinity is respected if matching 2 {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that InterPodAntiAffinity is respected if matching 2 {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that InterPodAntiAffinity is respected if matching 2 {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that InterPodAntiAffinity is respected if matching 2 {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that InterPodAntiAffinity is respected if matching 2 {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that InterPodAntiAffinity is respected if matching 2 {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that InterPodAntiAffinity is respected if matching 2 {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that InterPodAntiAffinity is respected if matching 2 {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that InterPodAntiAffinity is respected if matching 2 {Kubernetes e2e suite}
Looking into this...
trying to track this flake down. It looks like the pod is launched despite AntiAffinity in . In my local setup I saw the same, but noticed that the new admission controller was not activate. I have activate it and now get this:
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that InterPodAntiAffinity is respected if matching 2 {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that InterPodAntiAffinity is respected if matching 2 {Kubernetes e2e suite}
Should be fixed by

Image: framework.GetPauseImageName(f.Client), }, }, <ins> NodeSelector: map[string]string{k: v}, // only launch on our two nodes </ins> }, }) framework.ExpectNoError(err)


Question:
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that InterPodAntiAffinity is respected if matching 2 {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that InterPodAntiAffinity is respected if matching 2 {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that InterPodAntiAffinity is respected if matching 2 {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that InterPodAntiAffinity is respected if matching 2 {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that InterPodAntiAffinity is respected if matching 2 {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that InterPodAntiAffinity is respected if matching 2 {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that InterPodAntiAffinity is respected if matching 2 {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that InterPodAntiAffinity is respected if matching 2 {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that InterPodAntiAffinity is respected if matching 2 {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that InterPodAntiAffinity is respected if matching 2 {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that InterPodAntiAffinity is respected if matching 2 {Kubernetes e2e suite}
Looking into this...
trying to track this flake down. It looks like the pod is launched despite AntiAffinity in . In my local setup I saw the same, but noticed that the new admission controller was not activate. I have activate it and now get this:
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that InterPodAntiAffinity is respected if matching 2 {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that InterPodAntiAffinity is respected if matching 2 {Kubernetes e2e suite}
Should be fixed by

pod, err = c.Pods(ns).Get(podName) framework.ExpectNoError(err) <del> nodeName := pod.Spec.NodeName By("Trying to apply a random label on the found node.") k := "e2e.inter-pod-affinity.kubernetes.io/zone" v := "china-e2etest" framework.AddOrUpdateLabelOnNode(c, nodeName, k, v) framework.ExpectNodeHasLabel(c, nodeName, k, v) defer framework.RemoveLabelOffNode(c, nodeName, k) By("Trying to launch the pod, now with podAffinity with same Labels.") labelPodName := "with-podaffinity-" + string(uuid.NewUUID()) </del> <ins> By("Trying to launch another pod, now with podAntiAffinity with same Labels.") labelPodName := "with-podantiaffinity-" + string(uuid.NewUUID()) </ins> _, err = c.Pods(ns).Create(&api.Pod{ TypeMeta: unversioned.TypeMeta{ Kind: "Pod",


Question:
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that InterPodAntiAffinity is respected if matching 2 {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that InterPodAntiAffinity is respected if matching 2 {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that InterPodAntiAffinity is respected if matching 2 {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that InterPodAntiAffinity is respected if matching 2 {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that InterPodAntiAffinity is respected if matching 2 {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that InterPodAntiAffinity is respected if matching 2 {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that InterPodAntiAffinity is respected if matching 2 {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that InterPodAntiAffinity is respected if matching 2 {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that InterPodAntiAffinity is respected if matching 2 {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that InterPodAntiAffinity is respected if matching 2 {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that InterPodAntiAffinity is respected if matching 2 {Kubernetes e2e suite}
Looking into this...
trying to track this flake down. It looks like the pod is launched despite AntiAffinity in . In my local setup I saw the same, but noticed that the new admission controller was not activate. I have activate it and now get this:
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that InterPodAntiAffinity is respected if matching 2 {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that InterPodAntiAffinity is respected if matching 2 {Kubernetes e2e suite}
Should be fixed by

Image: framework.GetPauseImageName(f.Client), }, }, <ins> NodeSelector: map[string]string{k: v}, // only launch on our two nodes, contradicting the podAntiAffinity </ins> }, }) framework.ExpectNoError(err)


Question:
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that InterPodAntiAffinity is respected if matching 2 {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that InterPodAntiAffinity is respected if matching 2 {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that InterPodAntiAffinity is respected if matching 2 {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that InterPodAntiAffinity is respected if matching 2 {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that InterPodAntiAffinity is respected if matching 2 {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that InterPodAntiAffinity is respected if matching 2 {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that InterPodAntiAffinity is respected if matching 2 {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that InterPodAntiAffinity is respected if matching 2 {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that InterPodAntiAffinity is respected if matching 2 {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that InterPodAntiAffinity is respected if matching 2 {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that InterPodAntiAffinity is respected if matching 2 {Kubernetes e2e suite}
Looking into this...
trying to track this flake down. It looks like the pod is launched despite AntiAffinity in . In my local setup I saw the same, but noticed that the new admission controller was not activate. I have activate it and now get this:
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that InterPodAntiAffinity is respected if matching 2 {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that InterPodAntiAffinity is respected if matching 2 {Kubernetes e2e suite}
Should be fixed by

framework.Logf("Sleeping 10 seconds and crossing our fingers that scheduler will run in that time.") time.Sleep(10 * time.Second) <del> verifyResult(c, labelPodName, 2, 0, ns) </del> <ins> verifyResult(c, labelPodName, 3, 1, ns) </ins> }) // test the pod affinity successful matching scenario with multiple Label Operators.


Question:
https://k8s- Failed: [] Services should preserve source pod IP for traffic thru service cluster IP {Kubernetes e2e suite} Happened on a presubmit run in .
Sorry, it is a new e2e test I am still working on. It is not merged yet.
Don't worry about it, our flake detection isn't quite perfect yet :)
[FLAKE-PING] This flaky-test issue would love to have more attention...
[FLAKE-PING] This flaky-test issue would love to have more attention.
This new e2e test has been merged. Let's observe it few more days and close this issue if nothing bad happen.
[FLAKE-PING] This flaky-test issue would love to have more attention.
This does't look like a failure in new test, rather its a generic failure with serviceaccounts.
Thanks for the investigation. Yeah I think it is the serviceaccounts issue. I wonder when could we close this flaky issue. Seems like it is stable if serviceaccounts issue is fixed.
You can close it with a reference to the serviceaccounts issue - do you know the # for that one ?
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] Services should preserve source pod IP for traffic thru service cluster IP {Kubernetes e2e suite}
https://k8s- Failed: [] Services should preserve source pod IP for traffic thru service cluster IP {Kubernetes e2e suite}
https://k8s- Failed: [] Services should preserve source pod IP for traffic thru service cluster IP {Kubernetes e2e suite}
https://k8s- Failed: [] Services should preserve source pod IP for traffic thru service cluster IP {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
Have a fix for this, I'm now keeping running tests on my own cluster. Will send out PR once verify the fix works.

Expect(err).NotTo(HaveOccurred()) }() <del> // Waiting for service to expose endpoint </del> <ins> // Waiting for service to expose endpoint. </ins> validateEndpointsOrFail(c, ns, serviceName, PortsByPodName{serverPodName: {servicePort}}) By("Retrieve sourceip from a pod on the same node")


Question:
https://k8s- Failed: [] Services should preserve source pod IP for traffic thru service cluster IP {Kubernetes e2e suite} Happened on a presubmit run in .
Sorry, it is a new e2e test I am still working on. It is not merged yet.
Don't worry about it, our flake detection isn't quite perfect yet :)
[FLAKE-PING] This flaky-test issue would love to have more attention...
[FLAKE-PING] This flaky-test issue would love to have more attention.
This new e2e test has been merged. Let's observe it few more days and close this issue if nothing bad happen.
[FLAKE-PING] This flaky-test issue would love to have more attention.
This does't look like a failure in new test, rather its a generic failure with serviceaccounts.
Thanks for the investigation. Yeah I think it is the serviceaccounts issue. I wonder when could we close this flaky issue. Seems like it is stable if serviceaccounts issue is fixed.
You can close it with a reference to the serviceaccounts issue - do you know the # for that one ?
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] Services should preserve source pod IP for traffic thru service cluster IP {Kubernetes e2e suite}
https://k8s- Failed: [] Services should preserve source pod IP for traffic thru service cluster IP {Kubernetes e2e suite}
https://k8s- Failed: [] Services should preserve source pod IP for traffic thru service cluster IP {Kubernetes e2e suite}
https://k8s- Failed: [] Services should preserve source pod IP for traffic thru service cluster IP {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
Have a fix for this, I'm now keeping running tests on my own cluster. Will send out PR once verify the fix works.

timeout := 2 * time.Minute framework.Logf("Waiting up to %v for sourceIp test to be executed", timeout) cmd := fmt.Sprintf(`wget -T 30 -qO- %s:%d | grep client_address`, serviceIp, servicePort) <del> // need timeout mechanism because it may takes more times for iptables to be populated </del> <ins> // Need timeout mechanism because it may takes more times for iptables to be populated. </ins> for start := time.Now(); time.Since(start) < timeout; time.Sleep(2) { stdout, err = framework.RunHostCmd(execPod.Namespace, execPod.Name, cmd) if err != nil { framework.Logf("got err: %v, retry until timeout", err) continue } <ins> // Need to check output because wget -q might omit the error. if strings.TrimSpace(stdout) == "" { framework.Logf("got empty stdout, retry until timeout") continue } </ins> break } ExpectNoError(err) <del> // the stdout return from RunHostCmd seems to come with "n", so TrimSpace is needed // desired stdout in this format: client_address=x.x.x.x </del> <ins> // The stdout return from RunHostCmd seems to come with "n", so TrimSpace is needed. // Desired stdout in this format: client_address=x.x.x.x </ins> outputs := strings.Split(strings.TrimSpace(stdout), "=") <ins> sourceIp := "" </ins> if len(outputs) != 2 { <del> // fail the test if output format is unexpected </del> <ins> // Fail the test if output format is unexpected. </ins> framework.Failf("exec pod returned unexpected stdout format: [%v]n", stdout) <del> return execPodIp, "" </del> <ins> } else { sourceIp = outputs[1] </ins> } <del> return execPodIp, outputs[1] </del> <ins> return execPodIp, sourceIp </ins> }


Question:
On a cluster using GCI, I'm not able to create docker containers using the docker CLI. On further introspection, it looks like bridge is being deleted. I consider this a regression. K8s used to allow creating docker containers out of band and in-fact it is a feature that is taken into account as part of k8s node features. To further exacerbate this issue, exists on containervm. We need to match the behavior between GCI and container-vm and ideally stop deleting docker0 bridge. cc
During GCI start up, got deleted to avoid overlapping cidr with . ref:
there's no need to delete docker0 afaik. It was deleted on gci master to fix the "overlapping cidr" problem, i.e we were supplying to kubelet on GKE master, and this resulted in 2 bridges with the same cidr. At the time I'd suggested passing the correct cidr. Don't remember where this discussion ended. Note that containers on docker0 with/without cni are broken anyway, we used to start docker with --bridge=cbr0, so would result in an ip from cbr0. This is no longer the case. We use kubenet to put containers on cbr0, so will result in an ip from docker0, which is not connected to nanything else.
This shouldn't cause docker crashes? you should still be able to create containers, just no networking. And networking is broken through docker CLI anyway because of the cni issue mentioned, or are you observing something different?
will return error. Have to add
Since bridge has been deleted, docker run by default will fail unless one overrides the network ns to be host or none.
so what we're really losing is the ability to docker run then talk to others (external internet, other pods) without net=host on GCI. This is useful and we should fix it, as mentioned earlier we probably don't need to delete docker0. docker run + others talking to us is broken because of the cni change, and will be broken even with docker0 (IMO this is fine).
cc
Yeah. I'd prefer ignoring docker0 as long as it doesn't affect kubelet functionality. , Prashanth B wrote:
Vish pointed out that of docker dies, docker0 comes back. Given that we should be consistent. Either force docker not to create a bridge at all, EVER, or just ignore docker0. We may even want to consider assigning --bip= 169.254.123.0/24 or something, so we can reliably use 172.* for clusters eventually. , Vish Kannan wrote:

docker_opts+=" --log-level=warn" fi local use_net_plugin="true" <del> if [[ "${NETWORK_PROVIDER:-}" != "kubenet" && "${NETWORK_PROVIDER:-}" != "cni" ]]; then </del> <ins> if [[ "${NETWORK_PROVIDER:-}" == "kubenet" || "${NETWORK_PROVIDER:-}" == "cni" ]]; then # set docker0 cidr to private ip address range to avoid conflict with cbr0 cidr range docker_opts+=" --bip=169.254.123.1/24" else </ins> use_net_plugin="false" docker_opts+=" --bridge=cbr0" fi


Question:
On a cluster using GCI, I'm not able to create docker containers using the docker CLI. On further introspection, it looks like bridge is being deleted. I consider this a regression. K8s used to allow creating docker containers out of band and in-fact it is a feature that is taken into account as part of k8s node features. To further exacerbate this issue, exists on containervm. We need to match the behavior between GCI and container-vm and ideally stop deleting docker0 bridge. cc
During GCI start up, got deleted to avoid overlapping cidr with . ref:
there's no need to delete docker0 afaik. It was deleted on gci master to fix the "overlapping cidr" problem, i.e we were supplying to kubelet on GKE master, and this resulted in 2 bridges with the same cidr. At the time I'd suggested passing the correct cidr. Don't remember where this discussion ended. Note that containers on docker0 with/without cni are broken anyway, we used to start docker with --bridge=cbr0, so would result in an ip from cbr0. This is no longer the case. We use kubenet to put containers on cbr0, so will result in an ip from docker0, which is not connected to nanything else.
This shouldn't cause docker crashes? you should still be able to create containers, just no networking. And networking is broken through docker CLI anyway because of the cni issue mentioned, or are you observing something different?
will return error. Have to add
Since bridge has been deleted, docker run by default will fail unless one overrides the network ns to be host or none.
so what we're really losing is the ability to docker run then talk to others (external internet, other pods) without net=host on GCI. This is useful and we should fix it, as mentioned earlier we probably don't need to delete docker0. docker run + others talking to us is broken because of the cni change, and will be broken even with docker0 (IMO this is fine).
cc
Yeah. I'd prefer ignoring docker0 as long as it doesn't affect kubelet functionality. , Prashanth B wrote:
Vish pointed out that of docker dies, docker0 comes back. Given that we should be consistent. Either force docker not to create a bridge at all, EVER, or just ignore docker0. We may even want to consider assigning --bip= 169.254.123.0/24 or something, so we can reliably use 172.* for clusters eventually. , Vish Kannan wrote:

WantedBy=multi-user.target EOF <del> # Delete docker0 to avoid interference </del> <ins> # Flush iptables nat table </ins> iptables -t nat -F || true <del> ip link set docker0 down || true brctl delbr docker0 || true </del> systemctl start kubelet.service }


Question:
When I tried to use the following command to build kubernetes: I encountered the following errors: I'm using Ubuntu 14.04 and the code from master branch. will submit a patch then to fix this problem.
whoa, I just landed in the exact same place, I am trying to build k8s on a CentOS7 box. Tried with removing the unused utils and works like a charm !

"os" "k8s.io/kubernetes/pkg/healthz" <del> "k8s.io/kubernetes/pkg/util" </del> "k8s.io/kubernetes/pkg/util/flag" "k8s.io/kubernetes/pkg/util/logs" "k8s.io/kubernetes/pkg/version/verflag"


Question:
When I tried to use the following command to build kubernetes: I encountered the following errors: I'm using Ubuntu 14.04 and the code from master branch. will submit a patch then to fix this problem.
whoa, I just landed in the exact same place, I am trying to build k8s on a CentOS7 box. Tried with removing the unused utils and works like a charm !

"github.com/spf13/pflag" "k8s.io/kubernetes/contrib/mesos/pkg/executor/service" "k8s.io/kubernetes/contrib/mesos/pkg/hyperkube" <del> "k8s.io/kubernetes/pkg/util" </del> "k8s.io/kubernetes/pkg/util/flag" "k8s.io/kubernetes/pkg/util/logs" "k8s.io/kubernetes/pkg/version/verflag"


Question:
When I tried to use the following command to build kubernetes: I encountered the following errors: I'm using Ubuntu 14.04 and the code from master branch. will submit a patch then to fix this problem.
whoa, I just landed in the exact same place, I am trying to build k8s on a CentOS7 box. Tried with removing the unused utils and works like a charm !

"github.com/spf13/pflag" "k8s.io/kubernetes/contrib/mesos/pkg/hyperkube" "k8s.io/kubernetes/contrib/mesos/pkg/scheduler/service" <del> "k8s.io/kubernetes/pkg/util" </del> "k8s.io/kubernetes/pkg/util/flag" "k8s.io/kubernetes/pkg/util/logs" "k8s.io/kubernetes/pkg/version/verflag"


Question:
liveness probe was introduced in Heapster seems to work correctly so we should either fix it or revert liveness probe.
cc
port seems to be wrong

livenessProbe: httpGet: path: /healthz <del> port: 8080 </del> <ins> port: 8082 </ins> scheme: HTTP initialDelaySeconds: 180 timeoutSeconds: 5


Question: introduced a new requirement for node names to be DNS1123Labels. In case of hack/local-up-, an IP address is used as node name and thus endpoints aren't created for services, e.g. kube-dns doesn't work anymore. There are other projects that use IP addresses as node names for running simplified k8s dev environment: Probably there are other projects where it's convenient to have just an IP address as a node name. Verified against . On affected installs, service latency e2e test hangs, see e.g. Example ( some newlines):

if len(address.Hostname) > 0 { allErrs = append(allErrs, ValidateDNS1123Label(address.Hostname, fldPath.Child("hostname"))...) } <del> // During endpoint update, validate NodeName is DNS1123 compliant and transition rules allow the update </del> <ins> // During endpoint update, verify that NodeName is a DNS subdomain and transition rules allow the update </ins> if address.NodeName != nil { <del> allErrs = append(allErrs, ValidateDNS1123Label(*address.NodeName, fldPath.Child("nodeName"))...) </del> <ins> for _, msg := range ValidateNodeName(*address.NodeName, false) { allErrs = append(allErrs, field.Invalid(fldPath.Child("nodeName"), *address.NodeName, msg)) } </ins> } allErrs = append(allErrs, validateEpAddrNodeNameTransition(address, ipToNodeName, fldPath.Child("nodeName"))...) if len(allErrs) > 0 {


Question: introduced a new requirement for node names to be DNS1123Labels. In case of hack/local-up-, an IP address is used as node name and thus endpoints aren't created for services, e.g. kube-dns doesn't work anymore. There are other projects that use IP addresses as node names for running simplified k8s dev environment: Probably there are other projects where it's convenient to have just an IP address as a node name. Verified against . On affected installs, service latency e2e test hangs, see e.g. Example ( some newlines):

} } <del> func TestEndpointAddressNodeNameInvalidDNS1123(t *testing.T) { </del> <ins> func TestEndpointAddressNodeNameInvalidDNSSubdomain(t *testing.T) { </ins> // Check NodeName DNS validation <del> endpoint := newNodeNameEndpoint("illegal.nodename") </del> <ins> endpoint := newNodeNameEndpoint("illegal*.nodename") </ins> errList := ValidateEndpoints(endpoint) if len(errList) == 0 { t.Error("Endpoint should reject invalid NodeName") } } <ins> func TestEndpointAddressNodeNameCanBeAnIPAddress(t *testing.T) { endpoint := newNodeNameEndpoint("10.10.1.1") errList := ValidateEndpoints(endpoint) if len(errList) != 0 { t.Error("Endpoint should accept a NodeName that is an IP address") } } </ins>


Question:
During a manual upgrade test, described in , a kubelet failed to communicate via SSL to its master. This is currently blocking the 1.4 release. There is not a repro for this yet. The cluster was running 1.3, and was upgraded to 1.4 successfully, but upon downgrading to 1.3, the kubelets one one or more machines are not able to talk to the master. Current kubelet version is . (EDITED) The error when talking to the master is (lots like this): I think this means that the kubelet cannot verify the master's cert. I can't find the master's cert. Since there is not flag to kubelet to tell it where to find its config, I assume it is using the default location, which is . It seems wrong that clusters[0].cluster.certificate-authority-data is empty. That should have the cert for the master, right? I see this thing: but I don't understand what process generates , and if, presumably, from that template, why it isn't setting the env var CA_CERT.
in case this is related to kubelet bootstrap changes in 1.4.
Currently node is :
Correction. The node is running 1.4.0-beta.5
If the kubelet hasn't set the then it will not go through the bootstrap code path. Is the kubeconfig provided through the bootstrap flag or the kubeconfig flag?
Not set. For reference, args are:
Question: Could you please tell me why your nodes are running with GCI image at the first place? The test is starting with 1.3 release, which is using old debian-based containervm image, and I thought doesn't change the image at all (separate topic).
It does change to gci on gce with the default upgrade scripts, i brought this up on an email thread. I'm guessing something was dropped when we translated salt grains (containervm) to gci env vars.
What the default upgrade scripts you were referring to? I just double-checked with test cluster which was upgrade to 1.4 through , the node is stay with debian-based containervm, but Kubernetes version is bumped up to 1.4-beta.X
Ok, I am assigning this to to confirm. If upgrade logic from 1.3 to 1.4 changes the underneath os, that is wrong. I strongly believe this is just a simple operational error.
My understanding is that ignores base images altogether. , Dawn Chen wrote:
Hmm
I've tried and the node image did get updated to GCI. Before upgrade to v1.4: After upgrade to v1.4: After downgrade to v1.3: /cc
helped me debug this. He found that the metadata for the instance did not include the CA_CERT in it. That seems wrong.
Renamed from "x509 failure for kubelet after upgrade test"
Yeah, that's the error you'd get if the proper ca isn't loaded.
Will we get this problem if we don't upgrade to GCI (i.e. we stay with CVM) in -N? I guess we can try it and find out.
Manual upgrades have been successful if we don't switch image types. , David Oppenheimer < :
re: If you don't change the underneath image during the upgrade, the issue shouldn't occur.

function usage() { echo "!!! EXPERIMENTAL !!!" echo "" <del> echo "${0} [-M|-N|-P] -l | <version number or publication>" </del> <ins> echo "${0} [-M|-N|-P] -l -o | <version number or publication>" </ins> echo "  Upgrades master and nodes by default" echo "  -M:  Upgrade master only" echo "  -N:  Upgrade nodes only" echo "  -P:  Node upgrade prerequisites only (create a new instance template)" <ins> echo "  -o:  Use os distro sepcified in KUBE_NODE_OS_DISTRIBUTION for new nodes" </ins> echo "  -l:  Use local(dev) binaries" echo "" echo '  Version number or publication is either a proper version number'


Question:
During a manual upgrade test, described in , a kubelet failed to communicate via SSL to its master. This is currently blocking the 1.4 release. There is not a repro for this yet. The cluster was running 1.3, and was upgraded to 1.4 successfully, but upon downgrading to 1.3, the kubelets one one or more machines are not able to talk to the master. Current kubelet version is . (EDITED) The error when talking to the master is (lots like this): I think this means that the kubelet cannot verify the master's cert. I can't find the master's cert. Since there is not flag to kubelet to tell it where to find its config, I assume it is using the default location, which is . It seems wrong that clusters[0].cluster.certificate-authority-data is empty. That should have the cert for the master, right? I see this thing: but I don't understand what process generates , and if, presumably, from that template, why it isn't setting the env var CA_CERT.
in case this is related to kubelet bootstrap changes in 1.4.
Currently node is :
Correction. The node is running 1.4.0-beta.5
If the kubelet hasn't set the then it will not go through the bootstrap code path. Is the kubeconfig provided through the bootstrap flag or the kubeconfig flag?
Not set. For reference, args are:
Question: Could you please tell me why your nodes are running with GCI image at the first place? The test is starting with 1.3 release, which is using old debian-based containervm image, and I thought doesn't change the image at all (separate topic).
It does change to gci on gce with the default upgrade scripts, i brought this up on an email thread. I'm guessing something was dropped when we translated salt grains (containervm) to gci env vars.
What the default upgrade scripts you were referring to? I just double-checked with test cluster which was upgrade to 1.4 through , the node is stay with debian-based containervm, but Kubernetes version is bumped up to 1.4-beta.X
Ok, I am assigning this to to confirm. If upgrade logic from 1.3 to 1.4 changes the underneath os, that is wrong. I strongly believe this is just a simple operational error.
My understanding is that ignores base images altogether. , Dawn Chen wrote:
Hmm
I've tried and the node image did get updated to GCI. Before upgrade to v1.4: After upgrade to v1.4: After downgrade to v1.3: /cc
helped me debug this. He found that the metadata for the instance did not include the CA_CERT in it. That seems wrong.
Renamed from "x509 failure for kubelet after upgrade test"
Yeah, that's the error you'd get if the proper ca isn't loaded.
Will we get this problem if we don't upgrade to GCI (i.e. we stay with CVM) in -N? I guess we can try it and find out.
Manual upgrades have been successful if we don't switch image types. , David Oppenheimer < :
re: If you don't change the underneath image during the upgrade, the issue shouldn't occur.

'http://metadata/computeMetadata/v1/instance/attributes/kube-env'" 2>/dev/null } <ins> # Read os distro information from /os/release on node. # $1: The name of node # # Assumed vars: #   PROJECT #   ZONE function get-node-os() { gcloud compute ssh "$1"  --project "${PROJECT}"  --zone "${ZONE}"  --command  "cat /etc/os-release | grep "^ID=.*" | cut -c 4-" } </ins> # Assumed vars: #   KUBE_VERSION #   NODE_SCOPES


Question:
During a manual upgrade test, described in , a kubelet failed to communicate via SSL to its master. This is currently blocking the 1.4 release. There is not a repro for this yet. The cluster was running 1.3, and was upgraded to 1.4 successfully, but upon downgrading to 1.3, the kubelets one one or more machines are not able to talk to the master. Current kubelet version is . (EDITED) The error when talking to the master is (lots like this): I think this means that the kubelet cannot verify the master's cert. I can't find the master's cert. Since there is not flag to kubelet to tell it where to find its config, I assume it is using the default location, which is . It seems wrong that clusters[0].cluster.certificate-authority-data is empty. That should have the cert for the master, right? I see this thing: but I don't understand what process generates , and if, presumably, from that template, why it isn't setting the env var CA_CERT.
in case this is related to kubelet bootstrap changes in 1.4.
Currently node is :
Correction. The node is running 1.4.0-beta.5
If the kubelet hasn't set the then it will not go through the bootstrap code path. Is the kubeconfig provided through the bootstrap flag or the kubeconfig flag?
Not set. For reference, args are:
Question: Could you please tell me why your nodes are running with GCI image at the first place? The test is starting with 1.3 release, which is using old debian-based containervm image, and I thought doesn't change the image at all (separate topic).
It does change to gci on gce with the default upgrade scripts, i brought this up on an email thread. I'm guessing something was dropped when we translated salt grains (containervm) to gci env vars.
What the default upgrade scripts you were referring to? I just double-checked with test cluster which was upgrade to 1.4 through , the node is stay with debian-based containervm, but Kubernetes version is bumped up to 1.4-beta.X
Ok, I am assigning this to to confirm. If upgrade logic from 1.3 to 1.4 changes the underneath os, that is wrong. I strongly believe this is just a simple operational error.
My understanding is that ignores base images altogether. , Dawn Chen wrote:
Hmm
I've tried and the node image did get updated to GCI. Before upgrade to v1.4: After upgrade to v1.4: After downgrade to v1.3: /cc
helped me debug this. He found that the metadata for the instance did not include the CA_CERT in it. That seems wrong.
Renamed from "x509 failure for kubelet after upgrade test"
Yeah, that's the error you'd get if the proper ca isn't loaded.
Will we get this problem if we don't upgrade to GCI (i.e. we stay with CVM) in -N? I guess we can try it and find out.
Manual upgrades have been successful if we don't switch image types. , David Oppenheimer < :
re: If you don't change the underneath image during the upgrade, the issue shouldn't occur.

#                 compatible way? write-node-env <ins> if [[ "${env_os_distro}" == "false" ]]; then NODE_OS_DISTRIBUTION=$(get-node-os "${NODE_NAMES[0]}") source "${KUBE_ROOT}/cluster/gce/${NODE_OS_DISTRIBUTION}/node-helper.sh" # Reset the node image based on current os distro set-node-image fi </ins> # TODO(zmerlynn): Get configure-vm script from ${version}. (Must plumb this #                 through all create-node-instance-template implementations). local template_name=$(get-template-name-from-version ${SANITIZED_VERSION})


Question:
During a manual upgrade test, described in , a kubelet failed to communicate via SSL to its master. This is currently blocking the 1.4 release. There is not a repro for this yet. The cluster was running 1.3, and was upgraded to 1.4 successfully, but upon downgrading to 1.3, the kubelets one one or more machines are not able to talk to the master. Current kubelet version is . (EDITED) The error when talking to the master is (lots like this): I think this means that the kubelet cannot verify the master's cert. I can't find the master's cert. Since there is not flag to kubelet to tell it where to find its config, I assume it is using the default location, which is . It seems wrong that clusters[0].cluster.certificate-authority-data is empty. That should have the cert for the master, right? I see this thing: but I don't understand what process generates , and if, presumably, from that template, why it isn't setting the env var CA_CERT.
in case this is related to kubelet bootstrap changes in 1.4.
Currently node is :
Correction. The node is running 1.4.0-beta.5
If the kubelet hasn't set the then it will not go through the bootstrap code path. Is the kubeconfig provided through the bootstrap flag or the kubeconfig flag?
Not set. For reference, args are:
Question: Could you please tell me why your nodes are running with GCI image at the first place? The test is starting with 1.3 release, which is using old debian-based containervm image, and I thought doesn't change the image at all (separate topic).
It does change to gci on gce with the default upgrade scripts, i brought this up on an email thread. I'm guessing something was dropped when we translated salt grains (containervm) to gci env vars.
What the default upgrade scripts you were referring to? I just double-checked with test cluster which was upgrade to 1.4 through , the node is stay with debian-based containervm, but Kubernetes version is bumped up to 1.4-beta.X
Ok, I am assigning this to to confirm. If upgrade logic from 1.3 to 1.4 changes the underneath os, that is wrong. I strongly believe this is just a simple operational error.
My understanding is that ignores base images altogether. , Dawn Chen wrote:
Hmm
I've tried and the node image did get updated to GCI. Before upgrade to v1.4: After upgrade to v1.4: After downgrade to v1.3: /cc
helped me debug this. He found that the metadata for the instance did not include the CA_CERT in it. That seems wrong.
Renamed from "x509 failure for kubelet after upgrade test"
Yeah, that's the error you'd get if the proper ca isn't loaded.
Will we get this problem if we don't upgrade to GCI (i.e. we stay with CVM) in -N? I guess we can try it and find out.
Manual upgrades have been successful if we don't switch image types. , David Oppenheimer < :
re: If you don't change the underneath image during the upgrade, the issue shouldn't occur.

node_upgrade=true node_prereqs=false local_binaries=false <ins> env_os_distro=false </ins> while getopts ":MNPlh" opt; do case ${opt} in


Question:
During a manual upgrade test, described in , a kubelet failed to communicate via SSL to its master. This is currently blocking the 1.4 release. There is not a repro for this yet. The cluster was running 1.3, and was upgraded to 1.4 successfully, but upon downgrading to 1.3, the kubelets one one or more machines are not able to talk to the master. Current kubelet version is . (EDITED) The error when talking to the master is (lots like this): I think this means that the kubelet cannot verify the master's cert. I can't find the master's cert. Since there is not flag to kubelet to tell it where to find its config, I assume it is using the default location, which is . It seems wrong that clusters[0].cluster.certificate-authority-data is empty. That should have the cert for the master, right? I see this thing: but I don't understand what process generates , and if, presumably, from that template, why it isn't setting the env var CA_CERT.
in case this is related to kubelet bootstrap changes in 1.4.
Currently node is :
Correction. The node is running 1.4.0-beta.5
If the kubelet hasn't set the then it will not go through the bootstrap code path. Is the kubeconfig provided through the bootstrap flag or the kubeconfig flag?
Not set. For reference, args are:
Question: Could you please tell me why your nodes are running with GCI image at the first place? The test is starting with 1.3 release, which is using old debian-based containervm image, and I thought doesn't change the image at all (separate topic).
It does change to gci on gce with the default upgrade scripts, i brought this up on an email thread. I'm guessing something was dropped when we translated salt grains (containervm) to gci env vars.
What the default upgrade scripts you were referring to? I just double-checked with test cluster which was upgrade to 1.4 through , the node is stay with debian-based containervm, but Kubernetes version is bumped up to 1.4-beta.X
Ok, I am assigning this to to confirm. If upgrade logic from 1.3 to 1.4 changes the underneath os, that is wrong. I strongly believe this is just a simple operational error.
My understanding is that ignores base images altogether. , Dawn Chen wrote:
Hmm
I've tried and the node image did get updated to GCI. Before upgrade to v1.4: After upgrade to v1.4: After downgrade to v1.3: /cc
helped me debug this. He found that the metadata for the instance did not include the CA_CERT in it. That seems wrong.
Renamed from "x509 failure for kubelet after upgrade test"
Yeah, that's the error you'd get if the proper ca isn't loaded.
Will we get this problem if we don't upgrade to GCI (i.e. we stay with CVM) in -N? I guess we can try it and find out.
Manual upgrades have been successful if we don't switch image types. , David Oppenheimer < :
re: If you don't change the underneath image during the upgrade, the issue shouldn't occur.

l) local_binaries=true ;; <ins> o) env_os_distro=true ;; </ins> h) usage exit 0


Question:
During a manual upgrade test, described in , a kubelet failed to communicate via SSL to its master. This is currently blocking the 1.4 release. There is not a repro for this yet. The cluster was running 1.3, and was upgraded to 1.4 successfully, but upon downgrading to 1.3, the kubelets one one or more machines are not able to talk to the master. Current kubelet version is . (EDITED) The error when talking to the master is (lots like this): I think this means that the kubelet cannot verify the master's cert. I can't find the master's cert. Since there is not flag to kubelet to tell it where to find its config, I assume it is using the default location, which is . It seems wrong that clusters[0].cluster.certificate-authority-data is empty. That should have the cert for the master, right? I see this thing: but I don't understand what process generates , and if, presumably, from that template, why it isn't setting the env var CA_CERT.
in case this is related to kubelet bootstrap changes in 1.4.
Currently node is :
Correction. The node is running 1.4.0-beta.5
If the kubelet hasn't set the then it will not go through the bootstrap code path. Is the kubeconfig provided through the bootstrap flag or the kubeconfig flag?
Not set. For reference, args are:
Question: Could you please tell me why your nodes are running with GCI image at the first place? The test is starting with 1.3 release, which is using old debian-based containervm image, and I thought doesn't change the image at all (separate topic).
It does change to gci on gce with the default upgrade scripts, i brought this up on an email thread. I'm guessing something was dropped when we translated salt grains (containervm) to gci env vars.
What the default upgrade scripts you were referring to? I just double-checked with test cluster which was upgrade to 1.4 through , the node is stay with debian-based containervm, but Kubernetes version is bumped up to 1.4-beta.X
Ok, I am assigning this to to confirm. If upgrade logic from 1.3 to 1.4 changes the underneath os, that is wrong. I strongly believe this is just a simple operational error.
My understanding is that ignores base images altogether. , Dawn Chen wrote:
Hmm
I've tried and the node image did get updated to GCI. Before upgrade to v1.4: After upgrade to v1.4: After downgrade to v1.3: /cc
helped me debug this. He found that the metadata for the instance did not include the CA_CERT in it. That seems wrong.
Renamed from "x509 failure for kubelet after upgrade test"
Yeah, that's the error you'd get if the proper ca isn't loaded.
Will we get this problem if we don't upgrade to GCI (i.e. we stay with CVM) in -N? I guess we can try it and find out.
Manual upgrades have been successful if we don't switch image types. , David Oppenheimer < :
re: If you don't change the underneath image during the upgrade, the issue shouldn't occur.

MASTER_IMAGE_PROJECT=${KUBE_GCE_MASTER_PROJECT:-google-containers} fi <del> if [[ "${NODE_OS_DISTRIBUTION}" == "gci" ]]; then # If the node image is not set, we use the latest GCI image. # Otherwise, we respect whatever is set by the user. NODE_IMAGE=${KUBE_GCE_NODE_IMAGE:-${GCI_VERSION}} NODE_IMAGE_PROJECT=${KUBE_GCE_NODE_PROJECT:-google-containers} elif [[ "${NODE_OS_DISTRIBUTION}" == "debian" ]]; then NODE_IMAGE=${KUBE_GCE_NODE_IMAGE:-${CVM_VERSION}} NODE_IMAGE_PROJECT=${KUBE_GCE_NODE_PROJECT:-google-containers} fi </del> <ins> # Sets node image based on the specified os distro. Currently this function only # supports gci and debian. function set-node-image() { if [[ "${NODE_OS_DISTRIBUTION}" == "gci" ]]; then # If the node image is not set, we use the latest GCI image. # Otherwise, we respect whatever is set by the user. NODE_IMAGE=${KUBE_GCE_NODE_IMAGE:-${GCI_VERSION}} NODE_IMAGE_PROJECT=${KUBE_GCE_NODE_PROJECT:-google-containers} elif [[ "${NODE_OS_DISTRIBUTION}" == "debian" ]]; then NODE_IMAGE=${KUBE_GCE_NODE_IMAGE:-${CVM_VERSION}} NODE_IMAGE_PROJECT=${KUBE_GCE_NODE_PROJECT:-google-containers} fi } set-node-image </ins> # Verfiy cluster autoscaler configuration. if [[ "${ENABLE_CLUSTER_AUTOSCALER}" == "true" ]]; then


Question:
./hack/e2e- line 1192: local: `replica-pd=jenkins-us-central1-b-master-pd': not a valid identifier FYI - this looks related to your recent work? I don't see any obvious PR to blame, but I seem to remember you guys touching this part of the code recently? BTW, I deleted a bunch of leaked resources in the relevant test project on Friday (9/16), assuming that the test scripts would re-create whatever they needed. It's possible (but unlikely) that this has anything to do with that.
Here are some example logs to aid debugging: https://k8s-
I believe this to be the source of dysfunction: Which is turn caused by what you found, which causes the disk to not be cleaned up ahead of time: You can try this out in your local shell: git blames leads us to this recent commit: cc the author The root problem is unfortunatley very annoying: this is how bash tells you that variable names (in this case, $replica-pd) cannot contain dashes!
fix is up in
Awesome many thanks. I'll make sure it gets merged ASAP.

# Delete the master replica pd (possibly leaked by kube-up if master create failed). # TODO(jszczepkowski): remove also possibly leaked replicas' pds <del> local -r replica-pd="${REPLICA_NAME:-${MASTER_NAME}}-pd" if gcloud compute disks describe "${replica-pd}" --zone "${ZONE}" --project "${PROJECT}" &>/dev/null; then </del> <ins> local -r replica_pd="${REPLICA_NAME:-${MASTER_NAME}}-pd" if gcloud compute disks describe "${replica_pd}" --zone "${ZONE}" --project "${PROJECT}" &>/dev/null; then </ins> gcloud compute disks delete  --project "${PROJECT}"  --quiet  --zone "${ZONE}"  <del> "${replica-pd}" </del> <ins> "${replica_pd}" </ins> fi # Delete disk for cluster registry if enabled


Question:
After PR , I did a bit experiments on kube-dns graceful termination and got some unintended result as below: I guess it is because the other container was terminated before . DNS query from clients failed because is the actual connection point. After a bit discussion with and two solutions for this are: another program wrap up the dnsmasq and capture the SIGTERM signal. out a way to modify the sig mask of process without touching the original codes.
Something I have been meaning to try: Put in Dockerfile Build and push image We should deliver SIGCONT rather than SIGTERM. I did a quick test and it seems to work. Otherwise trapping SIGTERM in the shell and ignoring it is well documented... , Zihong Zheng wrote:
Thanks, I will take a look and test these.

<ins> # This Dockerfile will build an image that is configured # to run Fluentd with an Elasticsearch plug-in and the # provided configuration file. # TODO(satnam6502): Use a lighter base image, e.g. some form of busybox. # The image acts as an executable for the binary /usr/sbin/td-agent. # Note that fluentd is run with root permssion to allow access to # log files with root only access under /var/lib/docker/containers/* # Please see http://docs.fluentd.org/articles/install-by-deb for more # information about installing fluentd using deb package. FROM ubuntu:14.04 MAINTAINER Satnam Singh "satnam@google.com" # Ensure there are enough file descriptors for running Fluentd. RUN ulimit -n 65536 # Install prerequisites. RUN apt-get update &&  apt-get install -y curl &&  apt-get install -y -q libcurl4-openssl-dev make &&  apt-get clean # Install Fluentd. RUN /usr/bin/curl -L http://toolbelt.treasuredata.com/sh/install-ubuntu-trusty-td-agent2.sh | sh # Change the default user and group to root. # Needed to allow access to /var/log/docker/... files. RUN sed -i -e "s/USER=td-agent/USER=root/" -e "s/GROUP=td-agent/GROUP=root/" /etc/init.d/td-agent # Install the Elasticsearch Fluentd plug-in. RUN /usr/sbin/td-agent-gem install fluent-plugin-elasticsearch # Copy the Fluentd configuration file. COPY td-agent.conf /etc/td-agent/td-agent.conf # Copy a script that determines the name of the host machine # and then patch the Fluentd configuration files and then # run Fluentd in the foreground. ADD run.sh /run.sh # Always run the this setup script. ENTRYPOINT ["/run.sh"] </ins>


Question:
After PR , I did a bit experiments on kube-dns graceful termination and got some unintended result as below: I guess it is because the other container was terminated before . DNS query from clients failed because is the actual connection point. After a bit discussion with and two solutions for this are: another program wrap up the dnsmasq and capture the SIGTERM signal. out a way to modify the sig mask of process without touching the original codes.
Something I have been meaning to try: Put in Dockerfile Build and push image We should deliver SIGCONT rather than SIGTERM. I did a quick test and it seems to work. Otherwise trapping SIGTERM in the shell and ignoring it is well documented... , Zihong Zheng wrote:
Thanks, I will take a look and test these.

<ins> #!/bin/bash # Copyright 2014 Google Inc. All rights reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # #     http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # Build the fluentd-elasticsearch image and push # to google/fluentd-elasticsearch. sudo docker build -t kubernetes/fluentd-elasticsearch . sudo docker push kubernetes/fluentd-elasticsearch </ins>


Question:
After PR , I did a bit experiments on kube-dns graceful termination and got some unintended result as below: I guess it is because the other container was terminated before . DNS query from clients failed because is the actual connection point. After a bit discussion with and two solutions for this are: another program wrap up the dnsmasq and capture the SIGTERM signal. out a way to modify the sig mask of process without touching the original codes.
Something I have been meaning to try: Put in Dockerfile Build and push image We should deliver SIGCONT rather than SIGTERM. I did a quick test and it seems to work. Otherwise trapping SIGTERM in the shell and ignoring it is well documented... , Zihong Zheng wrote:
Thanks, I will take a look and test these.

<ins> #!/bin/bash # Copyright 2014 Google Inc. All rights reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # #     http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # WARNING! HORRIBLE HACK! We expect /outerhost to be mapped to # the enclosing /etc/host file so we can determine the name of # the host machine (super fragile). This is a temporary hack until # service IPs are done. OUTER_HOST=`tail -n 1 /outerhost | awk '{print $3}'` # Copy the Fluentd config file and patch it to refer to the # name of the host machine for ES_HOST. HACK! cp td-agent.conf /etc/td-agent sed -i -e "s/ES_HOST/${OUTER_HOST}/" /etc/td-agent/td-agent.conf /usr/sbin/td-agent </ins>


Question:
After PR , I did a bit experiments on kube-dns graceful termination and got some unintended result as below: I guess it is because the other container was terminated before . DNS query from clients failed because is the actual connection point. After a bit discussion with and two solutions for this are: another program wrap up the dnsmasq and capture the SIGTERM signal. out a way to modify the sig mask of process without touching the original codes.
Something I have been meaning to try: Put in Dockerfile Build and push image We should deliver SIGCONT rather than SIGTERM. I did a quick test and it seems to work. Otherwise trapping SIGTERM in the shell and ignoring it is well documented... , Zihong Zheng wrote:
Thanks, I will take a look and test these.

<ins> # This configuration file for Fluentd / td-agent is used # to watch changes to Docker log files that live in the # directory /var/lib/docker/containers/ which are then submitted to # Elasticsearch (running on the machine ES_HOST:9200) which # assumes the installation of the fluentd-elasticsearch plug-in. # See https://github.com/uken/fluent-plugin-elasticsearch for # more information about the plug-in. This file needs to be # patched to replace ES_HOST with the name of the actual # machine running Elasticsearch. # Maintainer: Satnam Singh (satnam@google.com) # # Exampe # ====== # A line in the Docker log file might like like this JSON: # # {"log":"2014/09/25 21:15:03 Got request with path wombatn", #  "stream":"stderr", #   "time":"2014-09-25T21:15:03.499185026Z"} # # The time_format specification below makes sure we properly # parse the time format produced by Docker. This will be # submitted to Elasticsearch and should appear like: # $ curl 'http://elasticsearch:9200/_search?pretty' # ... # { #      "_index" : "logstash-2014.09.25", #      "_type" : "fluentd", #      "_id" : "VBrbor2QTuGpsQyTCdfzqA", #      "_score" : 1.0, #      "_source":{"log":"2014/09/25 22:45:50 Got request with path wombatn", #                 "stream":"stderr","tag":"docker.container.all", #                 "@timestamp":"2014-09-25T22:45:50+00:00"} #    }, # ... <source> type tail format json time_key time path /var/lib/docker/containers/*/*-json.log time_format %Y-%m-%dT%H:%M:%S tag docker.container.all </source> <match docker.container.*> type elasticsearch log_level info include_tag_key true host ES_HOST port 9200 logstash_format true flush_interval 5s </match> </ins>


Question:
Kubernetes version (use ): 1.3.7 Environment: Cloud provider or hardware configuration: Bare-metal Ubuntu cluster OS (e.g. from /etc/os-release): Ubuntu 14.04.3 LTS Kernel (e.g. ): 3.19.0-47-generic Install tools: KUBERNETESPROVIDER=ubuntu ./kube- Others: Docker version 1.12.1 What happened: appends the same copy of each time we start and stop Kubernetes cluster (using Ubuntu's & ). For example, if we start Kubernetes cluster the first time and restart it again, the content of is doubled: This cause the following error when starting Kubernetes as seen in : What you expected to happen: content should not be concatenated every time Kubernetes is restarted. How to reproduce it (as minimally and precisely as possible): Start Kubernetes, stop it, and start it again, using and . Anything else do we need to know: One possible fix is to remove the in the following line in :
This is a bug and easy to fix. I'll open a PR to fix it
Bug has been fixed.

source /run/flannel/subnet.env source /etc/default/docker <del> echo DOCKER_OPTS="${DOCKER_OPTS} -H tcp://127.0.0.1:4243 -H unix:///var/run/docker.sock  </del> <ins> echo DOCKER_OPTS=" -H tcp://127.0.0.1:4243 -H unix:///var/run/docker.sock  </ins> --bip=${FLANNEL_SUBNET} --mtu=${FLANNEL_MTU}" > /etc/default/docker sudo service docker restart }


Question:
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite} Previous issues for this test:
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
We understand the problem here - I'm working on the fix.
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}

if len(tc.plugin) != 0 { c.AuthProvider = &clientcmdapi.AuthProviderConfig{Name: tc.plugin} } <del> tConfig, err := c.transportConfig() </del> <ins> tConfig, err := c.TransportConfig() </ins> if err != nil { // Unknown/bad plugins are expected to fail here. if !tc.expectErr {


Question:
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite} Previous issues for this test:
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
We understand the problem here - I'm working on the fix.
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}

// TLSConfigFor returns a tls.Config that will provide the transport level security defined // by the provided Config. Will return nil if no transport level security is requested. func TLSConfigFor(config *Config) (*tls.Config, error) { <del> cfg, err := config.transportConfig() </del> <ins> cfg, err := config.TransportConfig() </ins> if err != nil { return nil, err }


Question:
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite} Previous issues for this test:
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
We understand the problem here - I'm working on the fix.
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}

// or transport level security defined by the provided Config. Will return the // default http.DefaultTransport if no special case behavior is needed. func TransportFor(config *Config) (http.RoundTripper, error) { <del> cfg, err := config.transportConfig() </del> <ins> cfg, err := config.TransportConfig() </ins> if err != nil { return nil, err }


Question:
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite} Previous issues for this test:
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
We understand the problem here - I'm working on the fix.
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}

// the underlying connection (like WebSocket or HTTP2 clients). Pure HTTP clients should use // the higher level TransportFor or RESTClientFor methods. func HTTPWrappersForConfig(config *Config, rt http.RoundTripper) (http.RoundTripper, error) { <del> cfg, err := config.transportConfig() </del> <ins> cfg, err := config.TransportConfig() </ins> if err != nil { return nil, err } return transport.HTTPWrappersForConfig(cfg, rt) } <del> // transportConfig converts a client config to an appropriate transport config. func (c *Config) transportConfig() (*transport.Config, error) { </del> <ins> // TransportConfig converts a client config to an appropriate transport config. func (c *Config) TransportConfig() (*transport.Config, error) { </ins> wt := c.WrapTransport if c.AuthProvider != nil { provider, err := GetAuthProvider(c.Host, c.AuthProvider, c.AuthConfigPersister)


Question:
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite} Previous issues for this test:
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
We understand the problem here - I'm working on the fix.
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}

"fmt" "math" "math/rand" <ins> "net" "net/http" </ins> "os" "strconv" "sync"


Question:
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite} Previous issues for this test:
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
We understand the problem here - I'm working on the fix.
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}

"k8s.io/kubernetes/pkg/api" "k8s.io/kubernetes/pkg/client/clientset_generated/internalclientset" unversionedcore "k8s.io/kubernetes/pkg/client/clientset_generated/internalclientset/typed/core/unversioned" <ins> "k8s.io/kubernetes/pkg/client/restclient" "k8s.io/kubernetes/pkg/client/transport" </ins> client "k8s.io/kubernetes/pkg/client/unversioned" "k8s.io/kubernetes/pkg/labels" "k8s.io/kubernetes/pkg/util/intstr" <ins> utilnet "k8s.io/kubernetes/pkg/util/net" </ins> "k8s.io/kubernetes/test/e2e/framework" . "github.com/onsi/ginkgo"


Question:
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite} Previous issues for this test:
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
We understand the problem here - I'm working on the fix.
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}

namespaces = createNamespaces(f, nodeCount, itArg.podsPerNode) totalPods := itArg.podsPerNode * nodeCount <del> configs = generateRCConfigs(totalPods, itArg.image, itArg.command, c, namespaces) </del> <ins> configs = generateRCConfigs(totalPods, itArg.image, itArg.command, namespaces) </ins> var services []*api.Service // Read the environment variable to see if we want to create services createServices := os.Getenv("CREATE_SERVICES")


Question:
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite} Previous issues for this test:
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
We understand the problem here - I'm working on the fix.
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}

return namespaces } <ins> func createClients(numberOfClients int) ([]*client.Client, error) { clients := make([]*client.Client, numberOfClients) for i := 0; i < numberOfClients; i++ { config, err := framework.LoadConfig() Expect(err).NotTo(HaveOccurred()) config.QPS = 100 config.Burst = 200 if framework.TestContext.KubeAPIContentType != "" { config.ContentType = framework.TestContext.KubeAPIContentType } // For the purpose of this test, we want to force that clients // do not share underlying transport (which is a default behavior // in Kubernetes). Thus, we are explicitly creating transport for // each client here. transportConfig, err := config.TransportConfig() if err != nil { return nil, err } tlsConfig, err := transport.TLSConfigFor(transportConfig) if err != nil { return nil, err } config.Transport = utilnet.SetTransportDefaults(&http.Transport{ Proxy:               http.ProxyFromEnvironment, TLSHandshakeTimeout: 10 * time.Second, TLSClientConfig:     tlsConfig, MaxIdleConnsPerHost: 100, Dial: (&net.Dialer{ Timeout:   30 * time.Second, KeepAlive: 30 * time.Second, }).Dial, }) // Overwrite TLS-related fields from config to avoid collision with // Transport field. config.TLSClientConfig = restclient.TLSClientConfig{} c, err := client.New(config) if err != nil { return nil, err } clients[i] = c } return clients, nil } </ins> func computeRCCounts(total int) (int, int, int) { // Small RCs owns ~0.5 of total number of pods, medium and big RCs ~0.25 each. // For example for 3000 pods (100 nodes, 30 pods per node) there are:


Question:
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite} Previous issues for this test:
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
We understand the problem here - I'm working on the fix.
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}

return smallRCCount, mediumRCCount, bigRCCount } <del> func generateRCConfigs(totalPods int, image string, command []string, c *client.Client, nss []*api.Namespace) []*framework.RCConfig { </del> <ins> func generateRCConfigs(totalPods int, image string, command []string, nss []*api.Namespace) []*framework.RCConfig { </ins> configs := make([]*framework.RCConfig, 0) smallRCCount, mediumRCCount, bigRCCount := computeRCCounts(totalPods) <del> configs = append(configs, generateRCConfigsForGroup(c, nss, smallRCGroupName, smallRCSize, smallRCCount, image, command)...) configs = append(configs, generateRCConfigsForGroup(c, nss, mediumRCGroupName, mediumRCSize, mediumRCCount, image, command)...) configs = append(configs, generateRCConfigsForGroup(c, nss, bigRCGroupName, bigRCSize, bigRCCount, image, command)...) </del> <ins> configs = append(configs, generateRCConfigsForGroup(nss, smallRCGroupName, smallRCSize, smallRCCount, image, command)...) configs = append(configs, generateRCConfigsForGroup(nss, mediumRCGroupName, mediumRCSize, mediumRCCount, image, command)...) configs = append(configs, generateRCConfigsForGroup(nss, bigRCGroupName, bigRCSize, bigRCCount, image, command)...) // Create a number of clients to better simulate real usecase // where not everyone is using exactly the same client. rcsPerClient := 20 clients, err := createClients((len(configs) + rcsPerClient - 1) / rcsPerClient) framework.ExpectNoError(err) for i := 0; i < len(configs); i++ { configs[i].Client = clients[i%len(clients)] } </ins> return configs } <del> func generateRCConfigsForGroup(c *client.Client, nss []*api.Namespace, groupName string, size, count int, image string, command []string) []*framework.RCConfig { </del> <ins> func generateRCConfigsForGroup( nss []*api.Namespace, groupName string, size, count int, image string, command []string) []*framework.RCConfig { </ins> configs := make([]*framework.RCConfig, 0, count) for i := 1; i <= count; i++ { config := &framework.RCConfig{ <del> Client:     c, </del> <ins> Client:     nil, // this will be overwritten later </ins> Name:       groupName + "-" + strconv.Itoa(i), Namespace:  nss[i%len(nss)].Name, Timeout:    10 * time.Minute,


Question:
What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): vSphere, "message": "no endpoints available for service "kube-dns"", kube-dns, etc -- similar issues include: however it's not quite the same Is this a BUG REPORT or FEATURE REQUEST? (choose one): Bug Kubernetes version (use ): Environment: Cloud provider or hardware configuration: vSphere OS (e.g. from /etc/os-release): Using provided vmdk Kernel (e.g. ): and Install tools: GOVC + the script What happened: Our environment utilizes the RFC1918 10.0.0.0/8 block, so I edited to utilize an address space we are not using. This is my current configuration: After the deployment is finished, I am able to login to the dashboard via the URL provided via , however, when I check the DNS server I see: When I list services, I see: When I start a container and try to ping kube-dns, I am unable to: I can ping external IP addresses: Here's the configuration on each minion and the master: What you expected to happen: I expect containers can reach ping eachother. How to reproduce it (as minimally and precisely as possible): I've reproduced this problem using both 172.16.0.0/16 addresses as well as the 192.168.0.0/16 addresses. Anything else do we need to know: I followed the documentation here: and the only change I made was to the network configuration.
I just noticed that doesn't include any routes for the service IPs I defined (e.g. has no routing table entries on any of the minions, only ). I'm not sure how to fix it, but it looks like the DNS server can make requests to the upstream DNS, it just can't get responses back.
cc
Some additional info I probably should have included the first time:
I just noticed that while from inside a container I can ping google: I cannot ping our internal DNS which is at IP . So I think I'm getting closer to where the problem actually is. I also think the 192.168.120.120 was not actually unreachable, just not pingable. For example: Returns right away, while doing the same to IP hangs. I'm not sure why when using our DNS in the of my master/minions doesn't seem to give them the ability to resolve internal assets on our network.
Can you provide your output for iptables also
Hi imkin, Sure thing! Minion 1 - 10.248.30.38 Minion 2 - 10.248.30.40 Minion 3 - 10.248.30.37 Minion 4 - 10.248.30.39 Master - 10.248.30.36 Let me know what else I can provide, I haven't had much time to continue troubleshooting this week, but getting this to work would be a really great template for building our bare metal deployment. Thanks! ZC
Can you try with DNSSERVERIP="192.168.128.120"
Hi - I will edit that in the initial configuration and re-up my cluster this morning to see if that helps things work out of the box so to speak. Thanks! ZC
Hey I have edited my configuration as follows: I have re-run the script and everything came up and validated. I am still experiencing the same behavior, however. I now see this in the logs for the DNS service via the dashboard: I see that the service can't be reached by kubectl either (it doesn't even return kube-dns like it did previously): I also notice that on the minions, none of them seem to have routes for the higher IP ranges:
This looks to be the same as posting a PR shortly.
Awesome!! Thanks - I'm grabbing the from your pull request and will re-open this issue if that did not resolve it (once it's closed by the merge).
In the second case [when your kube-dns had the ip 192.168.128.120] where was your kube-dns?
based on the logs you posted you were definitely impacted by the issue in but I am not sure if that is the only issue on going on your setup. Please let us know how the fix goes for your environment.
There is no route for 192.168.3.0/24

MASTER_CPU=1 NODE_NAMES=($(eval echo ${INSTANCE_PREFIX}-minion-{1..${NUM_NODES}})) <del> NODE_IP_RANGES="10.244.0.0/16" </del> <ins> NODE_IP_RANGES="10.244.0.0/16" # Min Prefix supported is 16 </ins> MASTER_IP_RANGE="${MASTER_IP_RANGE:-10.246.0.0/24}" NODE_MEMORY_MB=2048 NODE_CPU=1


Question:
What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): vSphere, "message": "no endpoints available for service "kube-dns"", kube-dns, etc -- similar issues include: however it's not quite the same Is this a BUG REPORT or FEATURE REQUEST? (choose one): Bug Kubernetes version (use ): Environment: Cloud provider or hardware configuration: vSphere OS (e.g. from /etc/os-release): Using provided vmdk Kernel (e.g. ): and Install tools: GOVC + the script What happened: Our environment utilizes the RFC1918 10.0.0.0/8 block, so I edited to utilize an address space we are not using. This is my current configuration: After the deployment is finished, I am able to login to the dashboard via the URL provided via , however, when I check the DNS server I see: When I list services, I see: When I start a container and try to ping kube-dns, I am unable to: I can ping external IP addresses: Here's the configuration on each minion and the master: What you expected to happen: I expect containers can reach ping eachother. How to reproduce it (as minimally and precisely as possible): I've reproduced this problem using both 172.16.0.0/16 addresses as well as the 192.168.0.0/16 addresses. Anything else do we need to know: I followed the documentation here: and the only change I made was to the network configuration.
I just noticed that doesn't include any routes for the service IPs I defined (e.g. has no routing table entries on any of the minions, only ). I'm not sure how to fix it, but it looks like the DNS server can make requests to the upstream DNS, it just can't get responses back.
cc
Some additional info I probably should have included the first time:
I just noticed that while from inside a container I can ping google: I cannot ping our internal DNS which is at IP . So I think I'm getting closer to where the problem actually is. I also think the 192.168.120.120 was not actually unreachable, just not pingable. For example: Returns right away, while doing the same to IP hangs. I'm not sure why when using our DNS in the of my master/minions doesn't seem to give them the ability to resolve internal assets on our network.
Can you provide your output for iptables also
Hi imkin, Sure thing! Minion 1 - 10.248.30.38 Minion 2 - 10.248.30.40 Minion 3 - 10.248.30.37 Minion 4 - 10.248.30.39 Master - 10.248.30.36 Let me know what else I can provide, I haven't had much time to continue troubleshooting this week, but getting this to work would be a really great template for building our bare metal deployment. Thanks! ZC
Can you try with DNSSERVERIP="192.168.128.120"
Hi - I will edit that in the initial configuration and re-up my cluster this morning to see if that helps things work out of the box so to speak. Thanks! ZC
Hey I have edited my configuration as follows: I have re-run the script and everything came up and validated. I am still experiencing the same behavior, however. I now see this in the logs for the DNS service via the dashboard: I see that the service can't be reached by kubectl either (it doesn't even return kube-dns like it did previously): I also notice that on the minions, none of them seem to have routes for the higher IP ranges:
This looks to be the same as posting a PR shortly.
Awesome!! Thanks - I'm grabbing the from your pull request and will re-open this issue if that did not resolve it (once it's closed by the merge).
In the second case [when your kube-dns had the ip 192.168.128.120] where was your kube-dns?
based on the logs you posted you were definitely impacted by the issue in but I am not sure if that is the only issue on going on your setup. Please let us know how the fix goes for your environment.
There is no route for 192.168.3.0/24

# identify the subnet assigned to the node by the kubernetes controller manager. KUBE_NODE_BRIDGE_NETWORK=() for (( i=0; i<${#NODE_NAMES[@]}; i++)); do <del> printf " finding network of cbr0 bridge on node  ${NODE_NAMES[$i]}n" network=$(kube-ssh ${KUBE_NODE_IP_ADDRESSES[$i]} 'sudo ip route show | grep -E "dev cbr0" | cut -d     " " -f1') KUBE_NODE_BRIDGE_NETWORK+=("${network}") done </del> <ins> printf " finding network of cbr0 bridge on node  ${NODE_NAMES[$i]}n" network="" top2_octets_final=$(echo $NODE_IP_RANGES | awk -F "." '{ print $1 "." $2 }') # Assume that a 24 bit mask per node attempt=0 max_attempt=60 while true ; do attempt=$(($attempt+1)) network=$(kube-ssh ${KUBE_NODE_IP_ADDRESSES[$i]} 'sudo ip route show | grep -E "dev cbr0" | cut -d     " " -f1') top2_octets_read=$(echo $network | awk -F "." '{ print $1 "." $2 }') if [[ "$top2_octets_read" == "$top2_octets_final" ]]; then break fi if (( $attempt == $max_attempt )); then echo echo "(Failed) Waiting for cbr0 bridge to come up @ ${NODE_NAMES[$i]}" echo exit 1 fi printf "." sleep 5 done printf "n" KUBE_NODE_BRIDGE_NETWORK+=("${network}") done </ins> # Make the pods visible to each other and to the master. # The master needs have routes to the pods for the UI to work. local j for (( i=0; i<${#NODE_NAMES[@]}; i++)); do <del> printf "setting up routes for ${NODE_NAMES[$i]}" kube-ssh "${KUBE_MASTER_IP}" "sudo route add -net ${KUBE_NODE_BRIDGE_NETWORK[${i}]} gw ${KUBE_NODE_IP_ADDRESSES[${i}]}" for (( j=0; j<${#NODE_NAMES[@]}; j++)); do if [[ $i != $j ]]; then kube-ssh ${KUBE_NODE_IP_ADDRESSES[$i]} "sudo route add -net ${KUBE_NODE_BRIDGE_NETWORK[$j]} gw ${KUBE_NODE_IP_ADDRESSES[$j]}" fi done </del> <ins> printf "setting up routes for ${NODE_NAMES[$i]}n" printf " adding route to ${MASTER_NAME} for network ${KUBE_NODE_BRIDGE_NETWORK[${i}]} via ${KUBE_NODE_IP_ADDRESSES[${i}]}n" kube-ssh "${KUBE_MASTER_IP}" "sudo route add -net ${KUBE_NODE_BRIDGE_NETWORK[${i}]} gw ${KUBE_NODE_IP_ADDRESSES[${i}]}" for (( j=0; j<${#NODE_NAMES[@]}; j++)); do if [[ $i != $j ]]; then printf " adding route to ${NODE_NAMES[$j]} for network ${KUBE_NODE_BRIDGE_NETWORK[${i}]} via ${KUBE_NODE_IP_ADDRESSES[${i}]}n" kube-ssh ${KUBE_NODE_IP_ADDRESSES[$i]} "sudo route add -net ${KUBE_NODE_BRIDGE_NETWORK[$j]} gw ${KUBE_NODE_IP_ADDRESSES[$j]}" fi done printf "n" </ins> done }


Question:
What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): vSphere, "message": "no endpoints available for service "kube-dns"", kube-dns, etc -- similar issues include: however it's not quite the same Is this a BUG REPORT or FEATURE REQUEST? (choose one): Bug Kubernetes version (use ): Environment: Cloud provider or hardware configuration: vSphere OS (e.g. from /etc/os-release): Using provided vmdk Kernel (e.g. ): and Install tools: GOVC + the script What happened: Our environment utilizes the RFC1918 10.0.0.0/8 block, so I edited to utilize an address space we are not using. This is my current configuration: After the deployment is finished, I am able to login to the dashboard via the URL provided via , however, when I check the DNS server I see: When I list services, I see: When I start a container and try to ping kube-dns, I am unable to: I can ping external IP addresses: Here's the configuration on each minion and the master: What you expected to happen: I expect containers can reach ping eachother. How to reproduce it (as minimally and precisely as possible): I've reproduced this problem using both 172.16.0.0/16 addresses as well as the 192.168.0.0/16 addresses. Anything else do we need to know: I followed the documentation here: and the only change I made was to the network configuration.
I just noticed that doesn't include any routes for the service IPs I defined (e.g. has no routing table entries on any of the minions, only ). I'm not sure how to fix it, but it looks like the DNS server can make requests to the upstream DNS, it just can't get responses back.
cc
Some additional info I probably should have included the first time:
I just noticed that while from inside a container I can ping google: I cannot ping our internal DNS which is at IP . So I think I'm getting closer to where the problem actually is. I also think the 192.168.120.120 was not actually unreachable, just not pingable. For example: Returns right away, while doing the same to IP hangs. I'm not sure why when using our DNS in the of my master/minions doesn't seem to give them the ability to resolve internal assets on our network.
Can you provide your output for iptables also
Hi imkin, Sure thing! Minion 1 - 10.248.30.38 Minion 2 - 10.248.30.40 Minion 3 - 10.248.30.37 Minion 4 - 10.248.30.39 Master - 10.248.30.36 Let me know what else I can provide, I haven't had much time to continue troubleshooting this week, but getting this to work would be a really great template for building our bare metal deployment. Thanks! ZC
Can you try with DNSSERVERIP="192.168.128.120"
Hi - I will edit that in the initial configuration and re-up my cluster this morning to see if that helps things work out of the box so to speak. Thanks! ZC
Hey I have edited my configuration as follows: I have re-run the script and everything came up and validated. I am still experiencing the same behavior, however. I now see this in the logs for the DNS service via the dashboard: I see that the service can't be reached by kubectl either (it doesn't even return kube-dns like it did previously): I also notice that on the minions, none of them seem to have routes for the higher IP ranges:
This looks to be the same as posting a PR shortly.
Awesome!! Thanks - I'm grabbing the from your pull request and will re-open this issue if that did not resolve it (once it's closed by the merge).
In the second case [when your kube-dns had the ip 192.168.128.120] where was your kube-dns?
based on the logs you posted you were definitely impacted by the issue in but I am not sure if that is the only issue on going on your setup. Please let us know how the fix goes for your environment.
There is no route for 192.168.3.0/24

printf "Waiting for salt-master to be up on ${KUBE_MASTER} ...n" remote-pgrep ${KUBE_MASTER_IP} "salt-master" <ins> printf "Waiting for all packages to be installed on ${KUBE_MASTER} ...n" kube-check  ${KUBE_MASTER_IP} 'sudo salt "kubernetes-master" state.highstate -t 30 | grep -E "Failed:[[:space:]]+0"' </ins> local i for (( i=0; i<${#NODE_NAMES[@]}; i++)); do printf "Waiting for salt-minion to be up on ${NODE_NAMES[$i]} ....n" remote-pgrep ${KUBE_NODE_IP_ADDRESSES[$i]} "salt-minion" <ins> printf "Waiting for all salt packages to be installed on ${NODE_NAMES[$i]} .... n" kube-check  ${KUBE_MASTER_IP} 'sudo salt '"${NODE_NAMES[$i]}"' state.highstate -t 30 | grep -E "Failed:[[:space:]]+0"' </ins> printf " OKn" done <del> printf "Waiting for init highstate to be done on all nodes (this can take a few minutes) ...n" kube-check  ${KUBE_MASTER_IP} 'sudo salt '''*''' state.show_highstate -t 50' printf "Waiting for all packages to be installed on all nodes (this can take a few minutes) ...n" kube-check  ${KUBE_MASTER_IP} 'sudo salt '''*''' state.highstate -t 50 | grep -E "Failed:[[:space:]]+0"' </del> echo echo "Waiting for master and node initialization."


Question:
Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT Kubernetes version (use ): Environment: Cloud provider or hardware configuration: AWS OS (e.g. from /etc/os-release): Debian GNU/Linux 8 (jessie) Kernel (e.g. ): 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux What happened: Deployed a scheduled job from this manifest: After some time the jobs are no longer executed: What you expected to happen: Jobs are executed as defined in the manifest. How to reproduce it (as minimally and precisely as possible): Deploy a scheduled job and wait for some time.
Some logs from the controller manager:
Ping. More than 2 weeks and no action at all? Anything else i can do?
I'm also encountering this using both and as concurrency policies, fwiw.
I'm hitting this as well. Kube
Do the jobs start to reschedule if you delete old one? Try running . cc
From kube-controller-manager logs: The job is scheduled for 1:00. I think the "Multiple unmet start times" message is a clue of where things start going downhill. Also, notice how at 1:00 the controller wants to create job (for a whole day, it had failed on ).
Deleting (all) old jobs / pods fixes the problem temporarily. After some time the scheduled jobs face the same issue again.
There's a fix () that I've submitted against master to fix the Replace strategy, I'll mark it for inclusion in 1.4.
The other problem that I see here is this: It is possible that our is creating duplicates. I'll try to investigate that a bit more.
sorry for the delay in response time, I must have missed that while going through issues. Next time you're experiencing problems/have a question wrt Jobs/ScheduledJobs, just ping me directly in the issue.
No problem and I will. Thanks!
I'm still hitting this on
Fix in flight , I'll make sure to cherry-pick it back to 1.4
does this ship in to latest kubernetes release?
yes, this should be present in 1.5, 1.4 cherry-pick hasn't been merged yet.
so this issue is still present in v1.4.6?
yes - it is still present there. My fix was merged to 1.4 yesterday. You should wait for the next 1.4.x release.
thanks!
which according to should happen Dec 8th.
any news on the release?
See !msg/kubernetes-dev/o0At2-4KKQ8/_kPVM1XyDAAJ
so v1.4.7 will be released on that same day?
Look like should be available later today: !msg/kubernetes-dev/00lJpT9pxDc/YZfDzCxMDQAJ

"//pkg/runtime:go_default_library", "//pkg/types:go_default_library", "//pkg/util/errors:go_default_library", <del> "//pkg/util/hash:go_default_library", </del> "//pkg/util/metrics:go_default_library", "//pkg/util/runtime:go_default_library", "//pkg/util/wait:go_default_library",


Question:
Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT Kubernetes version (use ): Environment: Cloud provider or hardware configuration: AWS OS (e.g. from /etc/os-release): Debian GNU/Linux 8 (jessie) Kernel (e.g. ): 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux What happened: Deployed a scheduled job from this manifest: After some time the jobs are no longer executed: What you expected to happen: Jobs are executed as defined in the manifest. How to reproduce it (as minimally and precisely as possible): Deploy a scheduled job and wait for some time.
Some logs from the controller manager:
Ping. More than 2 weeks and no action at all? Anything else i can do?
I'm also encountering this using both and as concurrency policies, fwiw.
I'm hitting this as well. Kube
Do the jobs start to reschedule if you delete old one? Try running . cc
From kube-controller-manager logs: The job is scheduled for 1:00. I think the "Multiple unmet start times" message is a clue of where things start going downhill. Also, notice how at 1:00 the controller wants to create job (for a whole day, it had failed on ).
Deleting (all) old jobs / pods fixes the problem temporarily. After some time the scheduled jobs face the same issue again.
There's a fix () that I've submitted against master to fix the Replace strategy, I'll mark it for inclusion in 1.4.
The other problem that I see here is this: It is possible that our is creating duplicates. I'll try to investigate that a bit more.
sorry for the delay in response time, I must have missed that while going through issues. Next time you're experiencing problems/have a question wrt Jobs/ScheduledJobs, just ping me directly in the issue.
No problem and I will. Thanks!
I'm still hitting this on
Fix in flight , I'll make sure to cherry-pick it back to 1.4
does this ship in to latest kubernetes release?
yes, this should be present in 1.5, 1.4 cherry-pick hasn't been merged yet.
so this issue is still present in v1.4.6?
yes - it is still present there. My fix was merged to 1.4 yesterday. You should wait for the next 1.4.x release.
thanks!
which according to should happen Dec 8th.
any news on the release?
See !msg/kubernetes-dev/o0At2-4KKQ8/_kPVM1XyDAAJ
so v1.4.7 will be released on that same day?
Look like should be available later today: !msg/kubernetes-dev/00lJpT9pxDc/YZfDzCxMDQAJ

import ( "encoding/json" "fmt" <del> "hash/adler32" </del> "time" "github.com/golang/glog"


Question:
Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT Kubernetes version (use ): Environment: Cloud provider or hardware configuration: AWS OS (e.g. from /etc/os-release): Debian GNU/Linux 8 (jessie) Kernel (e.g. ): 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux What happened: Deployed a scheduled job from this manifest: After some time the jobs are no longer executed: What you expected to happen: Jobs are executed as defined in the manifest. How to reproduce it (as minimally and precisely as possible): Deploy a scheduled job and wait for some time.
Some logs from the controller manager:
Ping. More than 2 weeks and no action at all? Anything else i can do?
I'm also encountering this using both and as concurrency policies, fwiw.
I'm hitting this as well. Kube
Do the jobs start to reschedule if you delete old one? Try running . cc
From kube-controller-manager logs: The job is scheduled for 1:00. I think the "Multiple unmet start times" message is a clue of where things start going downhill. Also, notice how at 1:00 the controller wants to create job (for a whole day, it had failed on ).
Deleting (all) old jobs / pods fixes the problem temporarily. After some time the scheduled jobs face the same issue again.
There's a fix () that I've submitted against master to fix the Replace strategy, I'll mark it for inclusion in 1.4.
The other problem that I see here is this: It is possible that our is creating duplicates. I'll try to investigate that a bit more.
sorry for the delay in response time, I must have missed that while going through issues. Next time you're experiencing problems/have a question wrt Jobs/ScheduledJobs, just ping me directly in the issue.
No problem and I will. Thanks!
I'm still hitting this on
Fix in flight , I'll make sure to cherry-pick it back to 1.4
does this ship in to latest kubernetes release?
yes, this should be present in 1.5, 1.4 cherry-pick hasn't been merged yet.
so this issue is still present in v1.4.6?
yes - it is still present there. My fix was merged to 1.4 yesterday. You should wait for the next 1.4.x release.
thanks!
which according to should happen Dec 8th.
any news on the release?
See !msg/kubernetes-dev/o0At2-4KKQ8/_kPVM1XyDAAJ
so v1.4.7 will be released on that same day?
Look like should be available later today: !msg/kubernetes-dev/00lJpT9pxDc/YZfDzCxMDQAJ

"k8s.io/kubernetes/pkg/apis/batch" "k8s.io/kubernetes/pkg/runtime" "k8s.io/kubernetes/pkg/types" <del> hashutil "k8s.io/kubernetes/pkg/util/hash" </del> ) // Utilities for dealing with Jobs and CronJobs and time.


Question:
Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT Kubernetes version (use ): Environment: Cloud provider or hardware configuration: AWS OS (e.g. from /etc/os-release): Debian GNU/Linux 8 (jessie) Kernel (e.g. ): 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux What happened: Deployed a scheduled job from this manifest: After some time the jobs are no longer executed: What you expected to happen: Jobs are executed as defined in the manifest. How to reproduce it (as minimally and precisely as possible): Deploy a scheduled job and wait for some time.
Some logs from the controller manager:
Ping. More than 2 weeks and no action at all? Anything else i can do?
I'm also encountering this using both and as concurrency policies, fwiw.
I'm hitting this as well. Kube
Do the jobs start to reschedule if you delete old one? Try running . cc
From kube-controller-manager logs: The job is scheduled for 1:00. I think the "Multiple unmet start times" message is a clue of where things start going downhill. Also, notice how at 1:00 the controller wants to create job (for a whole day, it had failed on ).
Deleting (all) old jobs / pods fixes the problem temporarily. After some time the scheduled jobs face the same issue again.
There's a fix () that I've submitted against master to fix the Replace strategy, I'll mark it for inclusion in 1.4.
The other problem that I see here is this: It is possible that our is creating duplicates. I'll try to investigate that a bit more.
sorry for the delay in response time, I must have missed that while going through issues. Next time you're experiencing problems/have a question wrt Jobs/ScheduledJobs, just ping me directly in the issue.
No problem and I will. Thanks!
I'm still hitting this on
Fix in flight , I'll make sure to cherry-pick it back to 1.4
does this ship in to latest kubernetes release?
yes, this should be present in 1.5, 1.4 cherry-pick hasn't been merged yet.
so this issue is still present in v1.4.6?
yes - it is still present there. My fix was merged to 1.4 yesterday. You should wait for the next 1.4.x release.
thanks!
which according to should happen Dec 8th.
any news on the release?
See !msg/kubernetes-dev/o0At2-4KKQ8/_kPVM1XyDAAJ
so v1.4.7 will be released on that same day?
Look like should be available later today: !msg/kubernetes-dev/00lJpT9pxDc/YZfDzCxMDQAJ

return job, nil } <del> func getTimeHash(scheduledTime time.Time) uint32 { timeHasher := adler32.New() hashutil.DeepHashObject(timeHasher, scheduledTime) return timeHasher.Sum32() </del> <ins> // Return Unix Epoch Time func getTimeHash(scheduledTime time.Time) int64 { return scheduledTime.Unix() </ins> } // makeCreatedByRefJson makes a json string with an object reference for use in "created-by" annotation value


Question:
It fails with: cc
Looking at the generated I see that FEDERATIONIMAGETAG is "" which is resulting in this error. I see that before , FEDERATIONIMAGETAG was being set to but that docker tag file does not exist for hyperkube. I see the following exist in my : but does not exist there. Trying to figure out why.
cc
make sure you are running the scripts with env var exported. That is what controls whether gets created.
With , we are not generating the federation-apiserver binary and instead using the hyperkube binary.
sorry, i copied the wrong line. The droid you're looking for: File conditionally written during process File consumed when templating federation control plane yaml
Right. My is also empty. I am trying to run federation e2e tests. The command I am running is: which is what I have always used. I guess something changed in
that should definitely work. Please and then run the step, capture the output and email to me? I was literally doing exactly this process for another PR today, and I verified the image tag file was getting written correctly as well.
I figured out this is happening because after make clean or a fresh clone, kubectl will not be available when we are trying to get the semanticimagetag_version() in here.
Awesome! Thanks

kube::build::run_build_command make test-integration fi <ins> kube::build::copy_output </ins> if [[ "${FEDERATION:-}" == "true" ]];then ( source "${KUBE_ROOT}/build/util.sh"