Question:
It fails with: cc
Looking at the generated I see that FEDERATIONIMAGETAG is "" which is resulting in this error. I see that before , FEDERATIONIMAGETAG was being set to but that docker tag file does not exist for hyperkube. I see the following exist in my : but does not exist there. Trying to figure out why.
cc
make sure you are running the scripts with env var exported. That is what controls whether gets created.
With , we are not generating the federation-apiserver binary and instead using the hyperkube binary.
sorry, i copied the wrong line. The droid you're looking for: File conditionally written during process File consumed when templating federation control plane yaml
Right. My is also empty. I am trying to run federation e2e tests. The command I am running is: which is what I have always used. I guess something changed in
that should definitely work. Please and then run the step, capture the output and email to me? I was literally doing exactly this process for another PR today, and I verified the image tag file was getting written correctly as well.
I figured out this is happening because after make clean or a fresh clone, kubectl will not be available when we are trying to get the semanticimagetag_version() in here.
Awesome! Thanks

) fi <del> kube::build::copy_output </del> kube::release::package_tarballs kube::release::package_hyperkube


Question:
Implement unit (eventually, integration) tests for . This is an effort for Q4 2016 to be lead by and Refs
Has been implemented very much thanks to ;)

<ins> /* Copyright 2016 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package images import ( "fmt" "runtime" "testing" kubeadmapi "k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm" ) type getCoreImageTest struct { i string c *kubeadmapi.MasterConfiguration o string } const testversion = "1" func TestGetCoreImage(t *testing.T) { var tokenTest = []struct { t        getCoreImageTest expected string }{ {getCoreImageTest{o: "override"}, "override"}, {getCoreImageTest{ i: KubeEtcdImage, c: &kubeadmapi.MasterConfiguration{}}, fmt.Sprintf("%s/%s-%s:%s", gcrPrefix, "etcd", runtime.GOARCH, etcdVersion), }, {getCoreImageTest{ i: KubeAPIServerImage, c: &kubeadmapi.MasterConfiguration{KubernetesVersion: testversion}}, fmt.Sprintf("%s/%s-%s:%s", gcrPrefix, "kube-apiserver", runtime.GOARCH, testversion), }, {getCoreImageTest{ i: KubeControllerManagerImage, c: &kubeadmapi.MasterConfiguration{KubernetesVersion: testversion}}, fmt.Sprintf("%s/%s-%s:%s", gcrPrefix, "kube-controller-manager", runtime.GOARCH, testversion), }, {getCoreImageTest{ i: KubeSchedulerImage, c: &kubeadmapi.MasterConfiguration{KubernetesVersion: testversion}}, fmt.Sprintf("%s/%s-%s:%s", gcrPrefix, "kube-scheduler", runtime.GOARCH, testversion), }, {getCoreImageTest{ i: KubeProxyImage, c: &kubeadmapi.MasterConfiguration{KubernetesVersion: testversion}}, fmt.Sprintf("%s/%s-%s:%s", gcrPrefix, "kube-proxy", runtime.GOARCH, testversion), }, } for _, rt := range tokenTest { actual := GetCoreImage(rt.t.i, rt.t.c, rt.t.o) if actual != rt.expected { t.Errorf( "failed GetCoreImage:ntexpected: %snt  actual: %s", rt.expected, actual, ) } } } func TestGetAddonImage(t *testing.T) { var tokenTest = []struct { t        string expected string }{ {"matches nothing", ""}, { KubeDNSImage, fmt.Sprintf("%s/%s-%s:%s", gcrPrefix, "kubedns", runtime.GOARCH, kubeDNSVersion), }, { KubeDNSmasqImage, fmt.Sprintf("%s/%s-%s:%s", gcrPrefix, "kube-dnsmasq", runtime.GOARCH, dnsmasqVersion), }, { KubeExechealthzImage, fmt.Sprintf("%s/%s-%s:%s", gcrPrefix, "exechealthz", runtime.GOARCH, exechealthzVersion), }, { Pause, fmt.Sprintf("%s/%s-%s:%s", gcrPrefix, "pause", runtime.GOARCH, pauseVersion), }, } for _, rt := range tokenTest { actual := GetAddonImage(rt.t) if actual != rt.expected { t.Errorf( "failed GetCoreImage:ntexpected: %snt  actual: %s", rt.expected, actual, ) } } } </ins>


Question:
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
, it seems we forgot to add to ; I'll create a PR for that.
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!

package mount <del> type Mounter struct{} </del> <ins> type Mounter struct { mounterPath string } </ins> func (mounter *Mounter) Mount(source string, target string, fstype string, options []string) error { return nil


Question:
Similar to
is a dup of this. PR is just a quick try. Will spend more time on it to cover all cases.
status?
I sent PR which is waiting for reviewing from . Also, that PR is blocked by issue which is self-assigned by you but not fixed yet :)
That issue might be not hard to fix, but I didn't find how.So I need your help. IMO, we mignt need to change CI jobs configure in some files or images.
After discussing with reviewer ( this might be better suited to merge but not for 1.7.0, as it touches a part of the code that is quite old and stable. In other words, we probably need to give this change some time to soak in the e2e system as opposed to put right away into the release.

tags = ["automanaged"], deps = [ "//pkg/api:go_default_library", <ins> "//pkg/api/helper:go_default_library", </ins> "//pkg/api/service:go_default_library", "//pkg/api/testing:go_default_library", "//pkg/features:go_default_library",


Question:
Similar to
is a dup of this. PR is just a quick try. Will spend more time on it to cover all cases.
status?
I sent PR which is waiting for reviewing from . Also, that PR is blocked by issue which is self-assigned by you but not fixed yet :)
That issue might be not hard to fix, but I didn't find how.So I need your help. IMO, we mignt need to change CI jobs configure in some files or images.
After discussing with reviewer ( this might be better suited to merge but not for 1.7.0, as it touches a part of the code that is quite old and stable. In other words, we probably need to give this change some time to soak in the e2e system as opposed to put right away into the release.

} }() <del> if helper.IsServiceIPRequested(service) { // Allocate next available. ip, err := rs.serviceIPs.AllocateNext() if err != nil { // TODO: what error should be returned here?  It's not a // field-level validation failure (the field is valid), and it's // not really an internal error. return nil, errors.NewInternalError(fmt.Errorf("failed to allocate a serviceIP: %v", err)) } service.Spec.ClusterIP = ip.String() releaseServiceIP = true } else if helper.IsServiceIPSet(service) { // Try to respect the requested IP. if err := rs.serviceIPs.Allocate(net.ParseIP(service.Spec.ClusterIP)); err != nil { // TODO: when validation becomes versioned, this gets more complicated. el := field.ErrorList{field.Invalid(field.NewPath("spec", "clusterIP"), service.Spec.ClusterIP, err.Error())} return nil, errors.NewInvalid(api.Kind("Service"), service.Name, el) </del> <ins> var err error if service.Spec.Type != api.ServiceTypeExternalName { if releaseServiceIP, err = rs.initClusterIP(service); err != nil { return nil, err </ins> } <del> releaseServiceIP = true </del> } nodePortOp := portallocator.StartOperation(rs.serviceNodePorts) defer nodePortOp.Finish() <del> assignNodePorts := shouldAssignNodePorts(service) svcPortToNodePort := map[int]int{} for i := range service.Spec.Ports { servicePort := &service.Spec.Ports[i] allocatedNodePort := svcPortToNodePort[int(servicePort.Port)] if allocatedNodePort == 0 { // This will only scan forward in the service.Spec.Ports list because any matches // before the current port would have been found in svcPortToNodePort. This is really // looking for any user provided values. np := findRequestedNodePort(int(servicePort.Port), service.Spec.Ports) if np != 0 { err := nodePortOp.Allocate(np) if err != nil { // TODO: when validation becomes versioned, this gets more complicated. el := field.ErrorList{field.Invalid(field.NewPath("spec", "ports").Index(i).Child("nodePort"), np, err.Error())} return nil, errors.NewInvalid(api.Kind("Service"), service.Name, el) } servicePort.NodePort = int32(np) svcPortToNodePort[int(servicePort.Port)] = np } else if assignNodePorts { nodePort, err := nodePortOp.AllocateNext() if err != nil { // TODO: what error should be returned here?  It's not a // field-level validation failure (the field is valid), and it's // not really an internal error. return nil, errors.NewInternalError(fmt.Errorf("failed to allocate a nodePort: %v", err)) } servicePort.NodePort = int32(nodePort) svcPortToNodePort[int(servicePort.Port)] = nodePort } } else if int(servicePort.NodePort) != allocatedNodePort { if servicePort.NodePort == 0 { servicePort.NodePort = int32(allocatedNodePort) } else { err := nodePortOp.Allocate(int(servicePort.NodePort)) if err != nil { // TODO: when validation becomes versioned, this gets more complicated. el := field.ErrorList{field.Invalid(field.NewPath("spec", "ports").Index(i).Child("nodePort"), servicePort.NodePort, err.Error())} return nil, errors.NewInvalid(api.Kind("Service"), service.Name, el) } } </del> <ins> if service.Spec.Type == api.ServiceTypeNodePort || service.Spec.Type == api.ServiceTypeLoadBalancer { if err := rs.initNodePorts(service, nodePortOp); err != nil { return nil, err </ins> } }


Question:
Similar to
is a dup of this. PR is just a quick try. Will spend more time on it to cover all cases.
status?
I sent PR which is waiting for reviewing from . Also, that PR is blocked by issue which is self-assigned by you but not fixed yet :)
That issue might be not hard to fix, but I didn't find how.So I need your help. IMO, we mignt need to change CI jobs configure in some files or images.
After discussing with reviewer ( this might be better suited to merge but not for 1.7.0, as it touches a part of the code that is quite old and stable. In other words, we probably need to give this change some time to soak in the e2e system as opposed to put right away into the release.

// Update service from NodePort or LoadBalancer to ExternalName or ClusterIP, should release NodePort if exists. if (oldService.Spec.Type == api.ServiceTypeNodePort || oldService.Spec.Type == api.ServiceTypeLoadBalancer) && (service.Spec.Type == api.ServiceTypeExternalName || service.Spec.Type == api.ServiceTypeClusterIP) { <del> rs.releaseNodePort(oldService, nodePortOp) </del> <ins> rs.releaseNodePorts(oldService, nodePortOp) </ins> } // Update service from any type to NodePort or LoadBalancer, should update NodePort. if service.Spec.Type == api.ServiceTypeNodePort || service.Spec.Type == api.ServiceTypeLoadBalancer { <del> if err := rs.updateNodePort(oldService, service, nodePortOp); err != nil { </del> <ins> if err := rs.updateNodePorts(oldService, service, nodePortOp); err != nil { </ins> return nil, false, err } }


Question:
Similar to
is a dup of this. PR is just a quick try. Will spend more time on it to cover all cases.
status?
I sent PR which is waiting for reviewing from . Also, that PR is blocked by issue which is self-assigned by you but not fixed yet :)
That issue might be not hard to fix, but I didn't find how.So I need your help. IMO, we mignt need to change CI jobs configure in some files or images.
After discussing with reviewer ( this might be better suited to merge but not for 1.7.0, as it touches a part of the code that is quite old and stable. In other words, we probably need to give this change some time to soak in the e2e system as opposed to put right away into the release.

return servicePorts } <del> func shouldAssignNodePorts(service *api.Service) bool { switch service.Spec.Type { case api.ServiceTypeLoadBalancer: return true case api.ServiceTypeNodePort: return true case api.ServiceTypeClusterIP: return false case api.ServiceTypeExternalName: return false default: glog.Errorf("Unknown service type: %v", service.Spec.Type) return false } } </del> // Loop through the service ports list, find one with the same port number and // NodePort specified, return this NodePort otherwise return 0. func findRequestedNodePort(port int, servicePorts []api.ServicePort) int {


Question:
Similar to
is a dup of this. PR is just a quick try. Will spend more time on it to cover all cases.
status?
I sent PR which is waiting for reviewing from . Also, that PR is blocked by issue which is self-assigned by you but not fixed yet :)
That issue might be not hard to fix, but I didn't find how.So I need your help. IMO, we mignt need to change CI jobs configure in some files or images.
After discussing with reviewer ( this might be better suited to merge but not for 1.7.0, as it touches a part of the code that is quite old and stable. In other words, we probably need to give this change some time to soak in the e2e system as opposed to put right away into the release.

return false, nil } <del> func (rs *REST) updateNodePort(oldService, newService *api.Service, nodePortOp *portallocator.PortAllocationOperation) error { </del> <ins> func (rs *REST) initNodePorts(service *api.Service, nodePortOp *portallocator.PortAllocationOperation) error { svcPortToNodePort := map[int]int{} for i := range service.Spec.Ports { servicePort := &service.Spec.Ports[i] allocatedNodePort := svcPortToNodePort[int(servicePort.Port)] if allocatedNodePort == 0 { // This will only scan forward in the service.Spec.Ports list because any matches // before the current port would have been found in svcPortToNodePort. This is really // looking for any user provided values. np := findRequestedNodePort(int(servicePort.Port), service.Spec.Ports) if np != 0 { err := nodePortOp.Allocate(np) if err != nil { // TODO: when validation becomes versioned, this gets more complicated. el := field.ErrorList{field.Invalid(field.NewPath("spec", "ports").Index(i).Child("nodePort"), np, err.Error())} return errors.NewInvalid(api.Kind("Service"), service.Name, el) } servicePort.NodePort = int32(np) svcPortToNodePort[int(servicePort.Port)] = np } else { nodePort, err := nodePortOp.AllocateNext() if err != nil { // TODO: what error should be returned here?  It's not a // field-level validation failure (the field is valid), and it's // not really an internal error. return errors.NewInternalError(fmt.Errorf("failed to allocate a nodePort: %v", err)) } servicePort.NodePort = int32(nodePort) svcPortToNodePort[int(servicePort.Port)] = nodePort } } else if int(servicePort.NodePort) != allocatedNodePort { // TODO(xiangpengzhao): do we need to allocate a new NodePort in this case? // Note: the current implementation is better, because it saves a NodePort. if servicePort.NodePort == 0 { servicePort.NodePort = int32(allocatedNodePort) } else { err := nodePortOp.Allocate(int(servicePort.NodePort)) if err != nil { // TODO: when validation becomes versioned, this gets more complicated. el := field.ErrorList{field.Invalid(field.NewPath("spec", "ports").Index(i).Child("nodePort"), servicePort.NodePort, err.Error())} return errors.NewInvalid(api.Kind("Service"), service.Name, el) } } } } return nil } func (rs *REST) updateNodePorts(oldService, newService *api.Service, nodePortOp *portallocator.PortAllocationOperation) error { </ins> oldNodePorts := CollectServiceNodePorts(oldService) newNodePorts := []int{}


Question:
Similar to
is a dup of this. PR is just a quick try. Will spend more time on it to cover all cases.
status?
I sent PR which is waiting for reviewing from . Also, that PR is blocked by issue which is self-assigned by you but not fixed yet :)
That issue might be not hard to fix, but I didn't find how.So I need your help. IMO, we mignt need to change CI jobs configure in some files or images.
After discussing with reviewer ( this might be better suited to merge but not for 1.7.0, as it touches a part of the code that is quite old and stable. In other words, we probably need to give this change some time to soak in the e2e system as opposed to put right away into the release.

return nil } <del> func (rs *REST) releaseNodePort(service *api.Service, nodePortOp *portallocator.PortAllocationOperation) { </del> <ins> func (rs *REST) releaseNodePorts(service *api.Service, nodePortOp *portallocator.PortAllocationOperation) { </ins> nodePorts := CollectServiceNodePorts(service) for _, nodePort := range nodePorts {


Question:
Similar to
is a dup of this. PR is just a quick try. Will spend more time on it to cover all cases.
status?
I sent PR which is waiting for reviewing from . Also, that PR is blocked by issue which is self-assigned by you but not fixed yet :)
That issue might be not hard to fix, but I didn't find how.So I need your help. IMO, we mignt need to change CI jobs configure in some files or images.
After discussing with reviewer ( this might be better suited to merge but not for 1.7.0, as it touches a part of the code that is quite old and stable. In other words, we probably need to give this change some time to soak in the e2e system as opposed to put right away into the release.

"k8s.io/apiserver/pkg/registry/rest" utilfeature "k8s.io/apiserver/pkg/util/feature" "k8s.io/kubernetes/pkg/api" <ins> "k8s.io/kubernetes/pkg/api/helper" </ins> "k8s.io/kubernetes/pkg/api/service" "k8s.io/kubernetes/pkg/features" "k8s.io/kubernetes/pkg/registry/core/service/ipallocator"


Question:
Similar to
is a dup of this. PR is just a quick try. Will spend more time on it to cover all cases.
status?
I sent PR which is waiting for reviewing from . Also, that PR is blocked by issue which is self-assigned by you but not fixed yet :)
That issue might be not hard to fix, but I didn't find how.So I need your help. IMO, we mignt need to change CI jobs configure in some files or images.
After discussing with reviewer ( this might be better suited to merge but not for 1.7.0, as it touches a part of the code that is quite old and stable. In other words, we probably need to give this change some time to soak in the e2e system as opposed to put right away into the release.

if test.name == "Allocate specified ClusterIP" && test.svc.Spec.ClusterIP != "1.2.3.4" { t.Errorf("%q: expected ClusterIP %q, but got %q", test.name, "1.2.3.4", test.svc.Spec.ClusterIP) } <ins> if hasAllocatedIP { if helper.IsServiceIPSet(test.svc) { storage.serviceIPs.Release(net.ParseIP(test.svc.Spec.ClusterIP)) } } } } func TestInitNodePorts(t *testing.T) { storage, _ := NewTestREST(t, nil) nodePortOp := portallocator.StartOperation(storage.serviceNodePorts) defer nodePortOp.Finish() testCases := []struct { name                     string service                  *api.Service expectSpecifiedNodePorts []int }{ { name: "Service doesn't have specified NodePort", service: &api.Service{ ObjectMeta: metav1.ObjectMeta{Name: "foo"}, Spec: api.ServiceSpec{ Selector: map[string]string{"bar": "baz"}, Type:     api.ServiceTypeNodePort, Ports: []api.ServicePort{ { Name:       "port-tcp", Port:       53, TargetPort: intstr.FromInt(6502), Protocol:   api.ProtocolTCP, }, }, }, }, expectSpecifiedNodePorts: []int{}, }, { name: "Service has one specified NodePort", service: &api.Service{ ObjectMeta: metav1.ObjectMeta{Name: "foo"}, Spec: api.ServiceSpec{ Selector: map[string]string{"bar": "baz"}, Type:     api.ServiceTypeNodePort, Ports: []api.ServicePort{{ Name:       "port-tcp", Port:       53, TargetPort: intstr.FromInt(6502), Protocol:   api.ProtocolTCP, NodePort:   30053, }}, }, }, expectSpecifiedNodePorts: []int{30053}, }, { name: "Service has two same ports with different protocols and specifies same NodePorts", service: &api.Service{ ObjectMeta: metav1.ObjectMeta{Name: "foo"}, Spec: api.ServiceSpec{ Selector: map[string]string{"bar": "baz"}, Type:     api.ServiceTypeNodePort, Ports: []api.ServicePort{ { Name:       "port-tcp", Port:       53, TargetPort: intstr.FromInt(6502), Protocol:   api.ProtocolTCP, NodePort:   30054, }, { Name:       "port-udp", Port:       53, TargetPort: intstr.FromInt(6502), Protocol:   api.ProtocolUDP, NodePort:   30054, }, }, }, }, expectSpecifiedNodePorts: []int{30054, 30054}, }, { name: "Service has two same ports with different protocols and specifies different NodePorts", service: &api.Service{ ObjectMeta: metav1.ObjectMeta{Name: "foo"}, Spec: api.ServiceSpec{ Selector: map[string]string{"bar": "baz"}, Type:     api.ServiceTypeNodePort, Ports: []api.ServicePort{ { Name:       "port-tcp", Port:       53, TargetPort: intstr.FromInt(6502), Protocol:   api.ProtocolTCP, NodePort:   30055, }, { Name:       "port-udp", Port:       53, TargetPort: intstr.FromInt(6502), Protocol:   api.ProtocolUDP, NodePort:   30056, }, }, }, }, expectSpecifiedNodePorts: []int{30055, 30056}, }, { name: "Service has two different ports with different protocols and specifies different NodePorts", service: &api.Service{ ObjectMeta: metav1.ObjectMeta{Name: "foo"}, Spec: api.ServiceSpec{ Selector: map[string]string{"bar": "baz"}, Type:     api.ServiceTypeNodePort, Ports: []api.ServicePort{ { Name:       "port-tcp", Port:       53, TargetPort: intstr.FromInt(6502), Protocol:   api.ProtocolTCP, NodePort:   30057, }, { Name:       "port-udp", Port:       54, TargetPort: intstr.FromInt(6502), Protocol:   api.ProtocolUDP, NodePort:   30058, }, }, }, }, expectSpecifiedNodePorts: []int{30057, 30058}, }, { name: "Service has two same ports with different protocols but only specifies one NodePort", service: &api.Service{ ObjectMeta: metav1.ObjectMeta{Name: "foo"}, Spec: api.ServiceSpec{ Selector: map[string]string{"bar": "baz"}, Type:     api.ServiceTypeNodePort, Ports: []api.ServicePort{ { Name:       "port-tcp", Port:       53, TargetPort: intstr.FromInt(6502), Protocol:   api.ProtocolTCP, NodePort:   30059, }, { Name:       "port-udp", Port:       53, TargetPort: intstr.FromInt(6502), Protocol:   api.ProtocolUDP, }, }, }, }, expectSpecifiedNodePorts: []int{30059, 30059}, }, } for _, test := range testCases { err := storage.initNodePorts(test.service, nodePortOp) if err != nil { t.Errorf("%q: unexpected error: %v", test.name, err) continue } serviceNodePorts := CollectServiceNodePorts(test.service) if len(test.expectSpecifiedNodePorts) == 0 { for _, nodePort := range serviceNodePorts { if !storage.serviceNodePorts.Has(nodePort) { t.Errorf("%q: unexpected NodePort %d, out of range", test.name, nodePort) } } } else if !reflect.DeepEqual(serviceNodePorts, test.expectSpecifiedNodePorts) { t.Errorf("%q: expected NodePorts %v, but got %v", test.name, test.expectSpecifiedNodePorts, serviceNodePorts) } </ins> } } <del> func TestUpdateNodePort(t *testing.T) { </del> <ins> func TestUpdateNodePorts(t *testing.T) { </ins> storage, _ := NewTestREST(t, nil) nodePortOp := portallocator.StartOperation(storage.serviceNodePorts) defer nodePortOp.Finish()


Question:
Similar to
is a dup of this. PR is just a quick try. Will spend more time on it to cover all cases.
status?
I sent PR which is waiting for reviewing from . Also, that PR is blocked by issue which is self-assigned by you but not fixed yet :)
That issue might be not hard to fix, but I didn't find how.So I need your help. IMO, we mignt need to change CI jobs configure in some files or images.
After discussing with reviewer ( this might be better suited to merge but not for 1.7.0, as it touches a part of the code that is quite old and stable. In other words, we probably need to give this change some time to soak in the e2e system as opposed to put right away into the release.

} for _, test := range testCases { <del> err := storage.updateNodePort(test.oldService, test.newService, nodePortOp) </del> <ins> err := storage.updateNodePorts(test.oldService, test.newService, nodePortOp) </ins> if err != nil { t.Errorf("%q: unexpected error: %v", test.name, err) continue } <del> _ = nodePortOp.Commit() </del> serviceNodePorts := CollectServiceNodePorts(test.newService)


Question:
<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): Is this a BUG REPORT or FEATURE REQUEST? (choose one): FEATURE REQUEST <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): all :smile: Environment: Cloud provider or hardware configuration: GCE OS (e.g. from /etc/os-release): all Kernel (e.g. ): all Install tools: Others: What happened: uses the cli a lot and does a (in the commonly used function ) before doing things like . updates can introduce breaking changes (for example in the recent 132.0.0 release it removed a bunch of flags). This has caused cross-the-board test failures like . But this is not limited to just tests. Any kubernetes GCE users who rely on the and commands could have been broken. What you expected to happen: Stop updating gcloud automatically. We could either warn user to have the required dependencies installed, or pin to specific versions of dependencies like gcloud. How to reproduce it (as minimally and precisely as possible): download 1.4.4 release tarball run and Anything else do we need to know:
cc
/cc
cc I think it's reasonable to remove gcloud update calls from scripts, as I believe gcloud is updated out of band by k8s continuous testing. The scripts will still need to work with latest, so maybe they should check for a minimum version.
Good idea.
Yeah, we definitely want to continue using latest in CI so that we know if things break, but I think it's reasonable to remove the update itself from the kubernetes cluster util scripts.

KUBELET_CERT_BASE64=$(get-env-val "${master_env}" "KUBELET_CERT") KUBELET_KEY_BASE64=$(get-env-val "${master_env}" "KUBELET_KEY") } <ins> # Update or verify required gcloud components are installed # at minimum required version. # Assumed vars #   KUBE_PROMPT_FOR_UPDATE function update-or-verify-gcloud() { local sudo_prefix="" if [ ! -w $(dirname `which gcloud`) ]; then sudo_prefix="sudo" fi # update and install components as needed if [[ "${KUBE_PROMPT_FOR_UPDATE}" == "y" ]]; then ${sudo_prefix} gcloud ${gcloud_prompt:-} components install alpha ${sudo_prefix} gcloud ${gcloud_prompt:-} components install beta ${sudo_prefix} gcloud ${gcloud_prompt:-} components update else local version=$(${sudo_prefix} gcloud version --format=json) python -c' import json,sys from distutils import version minVersion = version.LooseVersion("1.3.0") required = [ "alpha", "beta", "core" ] data = json.loads(sys.argv[1]) rel = data.get("Google Cloud SDK") if rel != "HEAD" and version.LooseVersion(rel) < minVersion: print "gcloud version out of date ( < %s )" % minVersion exit(1) missing = [] for c in required: if not data.get(c): missing += [c] if missing: for c in missing: print ("missing required gcloud component "{0}"".format(c)) exit(1) ' """${version}""" fi } </ins>


Question:
<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): Is this a BUG REPORT or FEATURE REQUEST? (choose one): FEATURE REQUEST <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): all :smile: Environment: Cloud provider or hardware configuration: GCE OS (e.g. from /etc/os-release): all Kernel (e.g. ): all Install tools: Others: What happened: uses the cli a lot and does a (in the commonly used function ) before doing things like . updates can introduce breaking changes (for example in the recent 132.0.0 release it removed a bunch of flags). This has caused cross-the-board test failures like . But this is not limited to just tests. Any kubernetes GCE users who rely on the and commands could have been broken. What you expected to happen: Stop updating gcloud automatically. We could either warn user to have the required dependencies installed, or pin to specific versions of dependencies like gcloud. How to reproduce it (as minimally and precisely as possible): download 1.4.4 release tarball run and Anything else do we need to know:
cc
/cc
cc I think it's reasonable to remove gcloud update calls from scripts, as I believe gcloud is updated out of band by k8s continuous testing. The scripts will still need to work with latest, so maybe they should check for a minimum version.
Good idea.
Yeah, we definitely want to continue using latest in CI so that we know if things break, but I think it's reasonable to remove the update itself from the kubernetes cluster util scripts.

ALLOCATE_NODE_CIDRS=true <del> KUBE_PROMPT_FOR_UPDATE=y KUBE_SKIP_UPDATE=${KUBE_SKIP_UPDATE-"n"} </del> <ins> KUBE_PROMPT_FOR_UPDATE=${KUBE_PROMPT_FOR_UPDATE:-"n"} </ins> # How long (in seconds) to wait for cluster initialization. KUBE_CLUSTER_INITIALIZATION_TIMEOUT=${KUBE_CLUSTER_INITIALIZATION_TIMEOUT:-300}


Question:
<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): Is this a BUG REPORT or FEATURE REQUEST? (choose one): FEATURE REQUEST <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): all :smile: Environment: Cloud provider or hardware configuration: GCE OS (e.g. from /etc/os-release): all Kernel (e.g. ): all Install tools: Others: What happened: uses the cli a lot and does a (in the commonly used function ) before doing things like . updates can introduce breaking changes (for example in the recent 132.0.0 release it removed a bunch of flags). This has caused cross-the-board test failures like . But this is not limited to just tests. Any kubernetes GCE users who rely on the and commands could have been broken. What you expected to happen: Stop updating gcloud automatically. We could either warn user to have the required dependencies installed, or pin to specific versions of dependencies like gcloud. How to reproduce it (as minimally and precisely as possible): download 1.4.4 release tarball run and Anything else do we need to know:
cc
/cc
cc I think it's reasonable to remove gcloud update calls from scripts, as I believe gcloud is updated out of band by k8s continuous testing. The scripts will still need to work with latest, so maybe they should check for a minimum version.
Good idea.
Yeah, we definitely want to continue using latest in CI so that we know if things break, but I think it's reasonable to remove the update itself from the kubernetes cluster util scripts.

local cmd for cmd in gcloud gsutil; do if ! which "${cmd}" >/dev/null; then <del> local resp </del> <ins> local resp="n" </ins> if [[ "${KUBE_PROMPT_FOR_UPDATE}" == "y" ]]; then echo "Can't find ${cmd} in PATH.  Do you wish to install the Google Cloud SDK? [Y/n]" read resp <del> else resp="y" </del> fi if [[ "${resp}" != "n" && "${resp}" != "N" ]]; then curl https://sdk.cloud.google.com | bash


Question:
<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): Is this a BUG REPORT or FEATURE REQUEST? (choose one): FEATURE REQUEST <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): all :smile: Environment: Cloud provider or hardware configuration: GCE OS (e.g. from /etc/os-release): all Kernel (e.g. ): all Install tools: Others: What happened: uses the cli a lot and does a (in the commonly used function ) before doing things like . updates can introduce breaking changes (for example in the recent 132.0.0 release it removed a bunch of flags). This has caused cross-the-board test failures like . But this is not limited to just tests. Any kubernetes GCE users who rely on the and commands could have been broken. What you expected to happen: Stop updating gcloud automatically. We could either warn user to have the required dependencies installed, or pin to specific versions of dependencies like gcloud. How to reproduce it (as minimally and precisely as possible): download 1.4.4 release tarball run and Anything else do we need to know:
cc
/cc
cc I think it's reasonable to remove gcloud update calls from scripts, as I believe gcloud is updated out of band by k8s continuous testing. The scripts will still need to work with latest, so maybe they should check for a minimum version.
Good idea.
Yeah, we definitely want to continue using latest in CI so that we know if things break, but I think it's reasonable to remove the update itself from the kubernetes cluster util scripts.

fi fi done <del> if [[ "${KUBE_SKIP_UPDATE}" == "y" ]]; then return fi # update and install components as needed if [[ "${KUBE_PROMPT_FOR_UPDATE}" != "y" ]]; then gcloud_prompt="-q" fi local sudo_prefix="" if [ ! -w $(dirname `which gcloud`) ]; then sudo_prefix="sudo" fi ${sudo_prefix} gcloud ${gcloud_prompt:-} components install alpha || true ${sudo_prefix} gcloud ${gcloud_prompt:-} components install beta || true ${sudo_prefix} gcloud ${gcloud_prompt:-} components update || true </del> <ins> update-or-verify-gcloud </ins> } # Create a temp dir that'll be deleted at the end of this bash session.


Question:
<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): Is this a BUG REPORT or FEATURE REQUEST? (choose one): FEATURE REQUEST <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): all :smile: Environment: Cloud provider or hardware configuration: GCE OS (e.g. from /etc/os-release): all Kernel (e.g. ): all Install tools: Others: What happened: uses the cli a lot and does a (in the commonly used function ) before doing things like . updates can introduce breaking changes (for example in the recent 132.0.0 release it removed a bunch of flags). This has caused cross-the-board test failures like . But this is not limited to just tests. Any kubernetes GCE users who rely on the and commands could have been broken. What you expected to happen: Stop updating gcloud automatically. We could either warn user to have the required dependencies installed, or pin to specific versions of dependencies like gcloud. How to reproduce it (as minimally and precisely as possible): download 1.4.4 release tarball run and Anything else do we need to know:
cc
/cc
cc I think it's reasonable to remove gcloud update calls from scripts, as I believe gcloud is updated out of band by k8s continuous testing. The scripts will still need to work with latest, so maybe they should check for a minimum version.
Good idea.
Yeah, we definitely want to continue using latest in CI so that we know if things break, but I think it's reasonable to remove the update itself from the kubernetes cluster util scripts.

# Uses the config file specified in $KUBE_CONFIG_FILE, or defaults to config-default.sh <del> KUBE_PROMPT_FOR_UPDATE=y KUBE_SKIP_UPDATE=${KUBE_SKIP_UPDATE-"n"} </del> <ins> KUBE_PROMPT_FOR_UPDATE=${KUBE_PROMPT_FOR_UPDATE:-"n"} </ins> KUBE_ROOT=$(dirname "${BASH_SOURCE}")/../.. source "${KUBE_ROOT}/cluster/gke/${KUBE_CONFIG_FILE:-config-default.sh}" source "${KUBE_ROOT}/cluster/common.sh"


Question:
<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): Is this a BUG REPORT or FEATURE REQUEST? (choose one): FEATURE REQUEST <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): all :smile: Environment: Cloud provider or hardware configuration: GCE OS (e.g. from /etc/os-release): all Kernel (e.g. ): all Install tools: Others: What happened: uses the cli a lot and does a (in the commonly used function ) before doing things like . updates can introduce breaking changes (for example in the recent 132.0.0 release it removed a bunch of flags). This has caused cross-the-board test failures like . But this is not limited to just tests. Any kubernetes GCE users who rely on the and commands could have been broken. What you expected to happen: Stop updating gcloud automatically. We could either warn user to have the required dependencies installed, or pin to specific versions of dependencies like gcloud. How to reproduce it (as minimally and precisely as possible): download 1.4.4 release tarball run and Anything else do we need to know:
cc
/cc
cc I think it's reasonable to remove gcloud update calls from scripts, as I believe gcloud is updated out of band by k8s continuous testing. The scripts will still need to work with latest, so maybe they should check for a minimum version.
Good idea.
Yeah, we definitely want to continue using latest in CI so that we know if things break, but I think it's reasonable to remove the update itself from the kubernetes cluster util scripts.

if [[ "${KUBE_PROMPT_FOR_UPDATE}" == "y" ]]; then echo "Can't find gcloud in PATH.  Do you wish to install the Google Cloud SDK? [Y/n]" read resp <del> else resp="y" </del> fi if [[ "${resp}" != "n" && "${resp}" != "N" ]]; then curl https://sdk.cloud.google.com | bash


Question:
<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): Is this a BUG REPORT or FEATURE REQUEST? (choose one): FEATURE REQUEST <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): all :smile: Environment: Cloud provider or hardware configuration: GCE OS (e.g. from /etc/os-release): all Kernel (e.g. ): all Install tools: Others: What happened: uses the cli a lot and does a (in the commonly used function ) before doing things like . updates can introduce breaking changes (for example in the recent 132.0.0 release it removed a bunch of flags). This has caused cross-the-board test failures like . But this is not limited to just tests. Any kubernetes GCE users who rely on the and commands could have been broken. What you expected to happen: Stop updating gcloud automatically. We could either warn user to have the required dependencies installed, or pin to specific versions of dependencies like gcloud. How to reproduce it (as minimally and precisely as possible): download 1.4.4 release tarball run and Anything else do we need to know:
cc
/cc
cc I think it's reasonable to remove gcloud update calls from scripts, as I believe gcloud is updated out of band by k8s continuous testing. The scripts will still need to work with latest, so maybe they should check for a minimum version.
Good idea.
Yeah, we definitely want to continue using latest in CI so that we know if things break, but I think it's reasonable to remove the update itself from the kubernetes cluster util scripts.

exit 1 fi fi <del> if [[ "${KUBE_SKIP_UPDATE}" == "y" ]]; then return fi # update and install components as needed if [[ "${KUBE_PROMPT_FOR_UPDATE}" != "y" ]]; then gcloud_prompt="-q" fi local sudo_prefix="" if [ ! -w $(dirname `which gcloud`) ]; then sudo_prefix="sudo" fi ${sudo_prefix} gcloud ${gcloud_prompt:-} components install alpha || true ${sudo_prefix} gcloud ${gcloud_prompt:-} components install beta || true ${sudo_prefix} gcloud ${gcloud_prompt:-} components install kubectl|| true ${sudo_prefix} gcloud ${gcloud_prompt:-} components update || true </del> <ins> update-or-verify-gcloud </ins> } # Validate a kubernetes cluster


Question:
Is this a BUG REPORT or FEATURE REQUEST? (choose one): Bug Report Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2016-10-10T18:19:49Z", GoVersion:"go1.7.1", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"3", GitVersion:"v1.3.9", GitCommit:"", GitTreeState:"clean", BuildDate:"2016-10-20T20:37:18Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"linux/amd64"} Environment: Cloud provider or hardware configuration: GKE OS (e.g. from /etc/os-release): Local: El Capitan, Remote: GCI Kernel (e.g. ): Local: Darwin ... 15.6.0 Darwin Kernel Version 15.6.0: Thu Sep 1 15:01:16 PDT 2016; root:xnu-3248.60.11~2/RELEASEX8664 x86_64 Install tools: gcloud Others: n/a What happened: Provisioned 3 clusters each in different regions (us-central1, asia-east1, europe-west1). Each cluster is in 2 zones (w/ gcloud --zone... --additional-zones...) Provisioned Federation API Server and Controller (1.4.4) based on Verified clusters were ready. Provisioned federated service & deployed backends into each of the cluster. Verified the federated services were created, and all of the regional LB were created, and verified all the LB IPs were working w/ direct requests. DNS provider updated DNS w/ most of the service/zone CNAME records. But A records did not complete. Further investigation for the log points to multiple entries in rrdata w/ the same IP due to the multi-zone deployment: I1029 19:24:24. 1 ] Have recordset <(clouddns) "buttonmasher-" type=CNAME rrdatas=["buttonmasher-"] ttl=180. Need recordset <(clouddns) "buttonmasher-" type=A rrdatas=["130.211.203.196" "130.211.203.196"] ttl=180I1029 19:24:24. 1 ] Existing recordset <(clouddns) "buttonmasher-" type=CNAME rrdatas=["buttonmasher-"] ttl=180is not equivalent to needed recordset <(clouddns) "buttonmasher-" type=A rrdatas=["130.211.203.196" "130.211.203.196"] ttl=180, removing existing and adding needed. After creating the clusters w/ only one zone, the DNS provider worked properly. What you expected to happen: Cloud DNS Provider should update the DNS record properly w/ multi-zone clusters. How to reproduce it (as minimally and precisely as possible): 3 x multi-zone clusters Federation API Server and Controller clusters in to federation federated service and backends Anything else do we need to know: Using GKE and Cloud DNS
/cc
Also hit this. Any workarounds?
Sorry, no direct workarounds available yet. You could convert your multi-zone clusters into multiple single-zone clusters, and federate all of those.
FYI, although I'm sure you're aware - also affects AWS multi AZ clusters (1.6.2)
This seems bad. I'll try to take a look into it.
Fixed by

"//vendor/k8s.io/apimachinery/pkg/runtime/schema:go_default_library", "//vendor/k8s.io/apimachinery/pkg/types:go_default_library", "//vendor/k8s.io/apimachinery/pkg/util/runtime:go_default_library", <ins> "//vendor/k8s.io/apimachinery/pkg/util/sets:go_default_library", </ins> "//vendor/k8s.io/apimachinery/pkg/util/wait:go_default_library", "//vendor/k8s.io/apimachinery/pkg/watch:go_default_library", "//vendor/k8s.io/client-go/pkg/api/v1:go_default_library",


Question:
Is this a BUG REPORT or FEATURE REQUEST? (choose one): Bug Report Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2016-10-10T18:19:49Z", GoVersion:"go1.7.1", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"3", GitVersion:"v1.3.9", GitCommit:"", GitTreeState:"clean", BuildDate:"2016-10-20T20:37:18Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"linux/amd64"} Environment: Cloud provider or hardware configuration: GKE OS (e.g. from /etc/os-release): Local: El Capitan, Remote: GCI Kernel (e.g. ): Local: Darwin ... 15.6.0 Darwin Kernel Version 15.6.0: Thu Sep 1 15:01:16 PDT 2016; root:xnu-3248.60.11~2/RELEASEX8664 x86_64 Install tools: gcloud Others: n/a What happened: Provisioned 3 clusters each in different regions (us-central1, asia-east1, europe-west1). Each cluster is in 2 zones (w/ gcloud --zone... --additional-zones...) Provisioned Federation API Server and Controller (1.4.4) based on Verified clusters were ready. Provisioned federated service & deployed backends into each of the cluster. Verified the federated services were created, and all of the regional LB were created, and verified all the LB IPs were working w/ direct requests. DNS provider updated DNS w/ most of the service/zone CNAME records. But A records did not complete. Further investigation for the log points to multiple entries in rrdata w/ the same IP due to the multi-zone deployment: I1029 19:24:24. 1 ] Have recordset <(clouddns) "buttonmasher-" type=CNAME rrdatas=["buttonmasher-"] ttl=180. Need recordset <(clouddns) "buttonmasher-" type=A rrdatas=["130.211.203.196" "130.211.203.196"] ttl=180I1029 19:24:24. 1 ] Existing recordset <(clouddns) "buttonmasher-" type=CNAME rrdatas=["buttonmasher-"] ttl=180is not equivalent to needed recordset <(clouddns) "buttonmasher-" type=A rrdatas=["130.211.203.196" "130.211.203.196"] ttl=180, removing existing and adding needed. After creating the clusters w/ only one zone, the DNS provider worked properly. What you expected to happen: Cloud DNS Provider should update the DNS record properly w/ multi-zone clusters. How to reproduce it (as minimally and precisely as possible): 3 x multi-zone clusters Federation API Server and Controller clusters in to federation federated service and backends Anything else do we need to know: Using GKE and Cloud DNS
/cc
Also hit this. Any workarounds?
Sorry, no direct workarounds available yet. You could convert your multi-zone clusters into multiple single-zone clusters, and federate all of those.
FYI, although I'm sure you're aware - also affects AWS multi AZ clusters (1.6.2)
This seems bad. I'll try to take a look into it.
Fixed by

metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" pkgruntime "k8s.io/apimachinery/pkg/runtime" "k8s.io/apimachinery/pkg/util/runtime" <ins> "k8s.io/apimachinery/pkg/util/sets" </ins> "k8s.io/apimachinery/pkg/util/wait" "k8s.io/apimachinery/pkg/watch" "k8s.io/client-go/tools/cache"


Question:
Is this a BUG REPORT or FEATURE REQUEST? (choose one): Bug Report Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2016-10-10T18:19:49Z", GoVersion:"go1.7.1", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"3", GitVersion:"v1.3.9", GitCommit:"", GitTreeState:"clean", BuildDate:"2016-10-20T20:37:18Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"linux/amd64"} Environment: Cloud provider or hardware configuration: GKE OS (e.g. from /etc/os-release): Local: El Capitan, Remote: GCI Kernel (e.g. ): Local: Darwin ... 15.6.0 Darwin Kernel Version 15.6.0: Thu Sep 1 15:01:16 PDT 2016; root:xnu-3248.60.11~2/RELEASEX8664 x86_64 Install tools: gcloud Others: n/a What happened: Provisioned 3 clusters each in different regions (us-central1, asia-east1, europe-west1). Each cluster is in 2 zones (w/ gcloud --zone... --additional-zones...) Provisioned Federation API Server and Controller (1.4.4) based on Verified clusters were ready. Provisioned federated service & deployed backends into each of the cluster. Verified the federated services were created, and all of the regional LB were created, and verified all the LB IPs were working w/ direct requests. DNS provider updated DNS w/ most of the service/zone CNAME records. But A records did not complete. Further investigation for the log points to multiple entries in rrdata w/ the same IP due to the multi-zone deployment: I1029 19:24:24. 1 ] Have recordset <(clouddns) "buttonmasher-" type=CNAME rrdatas=["buttonmasher-"] ttl=180. Need recordset <(clouddns) "buttonmasher-" type=A rrdatas=["130.211.203.196" "130.211.203.196"] ttl=180I1029 19:24:24. 1 ] Existing recordset <(clouddns) "buttonmasher-" type=CNAME rrdatas=["buttonmasher-"] ttl=180is not equivalent to needed recordset <(clouddns) "buttonmasher-" type=A rrdatas=["130.211.203.196" "130.211.203.196"] ttl=180, removing existing and adding needed. After creating the clusters w/ only one zone, the DNS provider worked properly. What you expected to happen: Cloud DNS Provider should update the DNS record properly w/ multi-zone clusters. How to reproduce it (as minimally and precisely as possible): 3 x multi-zone clusters Federation API Server and Controller clusters in to federation federated service and backends Anything else do we need to know: Using GKE and Cloud DNS
/cc
Also hit this. Any workarounds?
Sorry, no direct workarounds available yet. You could convert your multi-zone clusters into multiple single-zone clusters, and federate all of those.
FYI, although I'm sure you're aware - also affects AWS multi AZ clusters (1.6.2)
This seems bad. I'll try to take a look into it.
Fixed by

non-nil error is also returned (possibly along with a partially complete list of resolved endpoints. */ func getResolvedEndpoints(endpoints []string) ([]string, error) { <del> resolvedEndpoints := make([]string, 0, len(endpoints)) </del> <ins> resolvedEndpoints := sets.String{} </ins> for _, endpoint := range endpoints { if net.ParseIP(endpoint) == nil { // It's not a valid IP address, so assume it's a DNS name, and try to resolve it, // replacing its DNS name with its IP addresses in expandedEndpoints ipAddrs, err := net.LookupHost(endpoint) if err != nil { <del> return resolvedEndpoints, err </del> <ins> return resolvedEndpoints.List(), err } for _, ip := range ipAddrs { resolvedEndpoints = resolvedEndpoints.Union(sets.NewString(ip)) </ins> } <del> resolvedEndpoints = append(resolvedEndpoints, ipAddrs...) </del> } else { <del> resolvedEndpoints = append(resolvedEndpoints, endpoint) </del> <ins> resolvedEndpoints = resolvedEndpoints.Union(sets.NewString(endpoint)) </ins> } } <del> return resolvedEndpoints, nil </del> <ins> return resolvedEndpoints.List(), nil </ins> } /* ensureDNSRrsets ensures (idempotently, and with minimum mutations) that all of the DNS resource record sets for dnsName are consistent with endpoints.


Question:
I just found a cluster like this cc/
do you know if anyone else is currently working on a fix for this? I've already started looking into a fix, would you be willing to review my PR soon?
I don't know of anyone who is working on it.
(I see your PR, and I'll review it.)

if observedReadyCondition.Status != v1.ConditionUnknown { currentReadyCondition.Status = v1.ConditionUnknown currentReadyCondition.Reason = "NodeStatusUnknown" <del> currentReadyCondition.Message = fmt.Sprintf("Kubelet stopped posting node status.") </del> <ins> currentReadyCondition.Message = "Kubelet stopped posting node status." </ins> // LastProbeTime is the last time we heard from kubelet. currentReadyCondition.LastHeartbeatTime = observedReadyCondition.LastHeartbeatTime currentReadyCondition.LastTransitionTime = nc.now() } } <del> // Like NodeReady condition, NodeOutOfDisk was last set longer ago than gracePeriod, so update // it to Unknown (regardless of its current value) in the master. // TODO(madhusudancs): Refactor this with readyCondition to remove duplicated code. _, oodCondition := v1.GetNodeCondition(&node.Status, v1.NodeOutOfDisk) if oodCondition == nil { glog.V(2).Infof("Out of disk condition of node %v is never updated by kubelet", node.Name) node.Status.Conditions = append(node.Status.Conditions, v1.NodeCondition{ Type:               v1.NodeOutOfDisk, Status:             v1.ConditionUnknown, Reason:             "NodeStatusNeverUpdated", Message:            fmt.Sprintf("Kubelet never posted node status."), LastHeartbeatTime:  node.CreationTimestamp, LastTransitionTime: nc.now(), }) } else { glog.V(4).Infof("node %v hasn't been updated for %+v. Last out of disk condition is: %+v", node.Name, nc.now().Time.Sub(savedNodeStatus.probeTimestamp.Time), oodCondition) if oodCondition.Status != v1.ConditionUnknown { oodCondition.Status = v1.ConditionUnknown oodCondition.Reason = "NodeStatusUnknown" oodCondition.Message = fmt.Sprintf("Kubelet stopped posting node status.") oodCondition.LastTransitionTime = nc.now() </del> <ins> // remaining node conditions should also be set to Unknown remainingNodeConditionTypes := []v1.NodeConditionType{v1.NodeOutOfDisk, v1.NodeMemoryPressure, v1.NodeDiskPressure} nowTimestamp := nc.now() for _, nodeConditionType := range remainingNodeConditionTypes { _, currentCondition := v1.GetNodeCondition(&node.Status, nodeConditionType) if currentCondition == nil { glog.V(2).Infof("Condition %v of node %v was never updated by kubelet", nodeConditionType, node.Name) node.Status.Conditions = append(node.Status.Conditions, v1.NodeCondition{ Type:               nodeConditionType, Status:             v1.ConditionUnknown, Reason:             "NodeStatusNeverUpdated", Message:            "Kubelet never posted node status.", LastHeartbeatTime:  node.CreationTimestamp, LastTransitionTime: nowTimestamp, }) } else { glog.V(4).Infof("node %v hasn't been updated for %+v. Last %v is: %+v", node.Name, nc.now().Time.Sub(savedNodeStatus.probeTimestamp.Time), nodeConditionType, currentCondition) if currentCondition.Status != v1.ConditionUnknown { currentCondition.Status = v1.ConditionUnknown currentCondition.Reason = "NodeStatusUnknown" currentCondition.Message = "Kubelet stopped posting node status." currentCondition.LastTransitionTime = nowTimestamp } </ins> } }


Question:
I just found a cluster like this cc/
do you know if anyone else is currently working on a fix for this? I've already started looking into a fix, would you be willing to review my PR soon?
I don't know of anyone who is working on it.
(I see your PR, and I'll review it.)

LastHeartbeatTime:  metav1.Date(2012, 1, 1, 0, 0, 0, 0, time.UTC), LastTransitionTime: fakeNow, }, <ins> { Type:               v1.NodeMemoryPressure, Status:             v1.ConditionUnknown, Reason:             "NodeStatusNeverUpdated", Message:            "Kubelet never posted node status.", LastHeartbeatTime:  metav1.Date(2012, 1, 1, 0, 0, 0, 0, time.UTC), LastTransitionTime: fakeNow, }, { Type:               v1.NodeDiskPressure, Status:             v1.ConditionUnknown, Reason:             "NodeStatusNeverUpdated", Message:            "Kubelet never posted node status.", LastHeartbeatTime:  metav1.Date(2012, 1, 1, 0, 0, 0, 0, time.UTC), LastTransitionTime: fakeNow, }, </ins> }, }, },


Question:
I just found a cluster like this cc/
do you know if anyone else is currently working on a fix for this? I've already started looking into a fix, would you be willing to review my PR soon?
I don't know of anyone who is working on it.
(I see your PR, and I'll review it.)

LastHeartbeatTime:  metav1.Date(2015, 1, 1, 12, 0, 0, 0, time.UTC), LastTransitionTime: metav1.Time{Time: metav1.Date(2015, 1, 1, 12, 0, 0, 0, time.UTC).Add(time.Hour)}, }, <ins> { Type:               v1.NodeMemoryPressure, Status:             v1.ConditionUnknown, Reason:             "NodeStatusNeverUpdated", Message:            "Kubelet never posted node status.", LastHeartbeatTime:  metav1.Date(2012, 1, 1, 0, 0, 0, 0, time.UTC), // should default to node creation time if condition was never updated LastTransitionTime: metav1.Time{Time: metav1.Date(2015, 1, 1, 12, 0, 0, 0, time.UTC).Add(time.Hour)}, }, { Type:               v1.NodeDiskPressure, Status:             v1.ConditionUnknown, Reason:             "NodeStatusNeverUpdated", Message:            "Kubelet never posted node status.", LastHeartbeatTime:  metav1.Date(2012, 1, 1, 0, 0, 0, 0, time.UTC), // should default to node creation time if condition was never updated LastTransitionTime: metav1.Time{Time: metav1.Date(2015, 1, 1, 12, 0, 0, 0, time.UTC).Add(time.Hour)}, }, </ins> }, Capacity: v1.ResourceList{ v1.ResourceName(v1.ResourceCPU):    resource.MustParse("10"),


Question:
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
should this issue be moved into the 1.5 milestone and is it a release blocker? The test seems to flake fairly infrequently but we need to get the issue triaged as to whether it's a 1.5 release blocker. Please update this issue by 1PM PST 30 November 2016. Thanks! cc:
Since the feature it is testing is introduced for 1.5, so it should have the 1.5 milestone? This test is flaking because it is not well written, hence I don't think it is a release blocker. And I already sent out a which got LGTM before but recently rebased.
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
Thanks for triaging
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}

"strings" "time" <ins> "k8s.io/kubernetes/pkg/api/resource" </ins> "k8s.io/kubernetes/pkg/api/v1" clientset "k8s.io/kubernetes/pkg/client/clientset_generated/release_1_5" "k8s.io/kubernetes/pkg/labels"


Question:
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
should this issue be moved into the 1.5 milestone and is it a release blocker? The test seems to flake fairly infrequently but we need to get the issue triaged as to whether it's a 1.5 release blocker. Please update this issue by 1PM PST 30 November 2016. Thanks! cc:
Since the feature it is testing is introduced for 1.5, so it should have the 1.5 milestone? This test is flaking because it is not well written, hence I don't think it is a release blocker. And I already sent out a which got LGTM before but recently rebased.
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
Thanks for triaging
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}

const ( DNSdefaultTimeout      = 5 * time.Minute <del> DNSscaleUpTimeout      = 5 * time.Minute DNSscaleDownTimeout    = 10 * time.Minute </del> DNSNamespace           = "kube-system" ClusterAddonLabelKey   = "k8s-app" KubeDNSLabelName       = "kube-dns"


Question:
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
should this issue be moved into the 1.5 milestone and is it a release blocker? The test seems to flake fairly infrequently but we need to get the issue triaged as to whether it's a 1.5 release blocker. Please update this issue by 1PM PST 30 November 2016. Thanks! cc:
Since the feature it is testing is introduced for 1.5, so it should have the 1.5 milestone? This test is flaking because it is not well written, hence I don't think it is a release blocker. And I already sent out a which got LGTM before but recently rebased.
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
Thanks for triaging
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}

var _ = framework.KubeDescribe("DNS horizontal autoscaling", func() { f := framework.NewDefaultFramework("dns-autoscaling") var c clientset.Interface <del> var nodeCount int </del> var previousParams map[string]string <del> DNSParams_1 := map[string]string{"linear": "{"nodesPerReplica": 1}"} DNSParams_2 := map[string]string{"linear": "{"nodesPerReplica": 2}"} DNSParams_3 := map[string]string{"linear": "{"nodesPerReplica": 3}"} </del> <ins> DNSParams_1 := DNSParamsLinear{map[string]string{"linear": "{"nodesPerReplica": 1}"}, 1.0, 0.0} DNSParams_2 := DNSParamsLinear{map[string]string{"linear": "{"nodesPerReplica": 2}"}, 2.0, 0.0} DNSParams_3 := DNSParamsLinear{map[string]string{"linear": "{"nodesPerReplica": 3, "coresPerReplica": 3}"}, 3.0, 3.0} </ins> BeforeEach(func() { framework.SkipUnlessProviderIs("gce") c = f.ClientSet <del> nodes := framework.GetReadySchedulableNodesOrDie(c) nodeCount = len(nodes.Items) Expect(nodeCount).NotTo(BeZero()) </del> <ins> Expect(len(framework.GetReadySchedulableNodesOrDie(c).Items)).NotTo(BeZero()) </ins> pcm, err := fetchDNSScalingConfigMap(c) <del> ExpectNoError(err) </del> <ins> Expect(err).NotTo(HaveOccurred()) </ins> previousParams = pcm.Data By("Replace the dns autoscaling parameters with testing parameters") <del> ExpectNoError(updateDNSScalingConfigMap(c, packDNSScalingConfigMap(DNSParams_1))) </del> <ins> Expect(updateDNSScalingConfigMap(c, packDNSScalingConfigMap(DNSParams_1.data))).NotTo(HaveOccurred()) </ins> By("Wait for kube-dns scaled to expected number") <del> ExpectNoError(waitForDNSReplicasSatisfied(c, nodeCount, DNSdefaultTimeout)) </del> <ins> getExpectReplicasLinear := getExpectReplicasFuncLinear(c, &DNSParams_1) Expect(waitForDNSReplicasSatisfied(c, getExpectReplicasLinear, DNSdefaultTimeout)).NotTo(HaveOccurred()) </ins> }) AfterEach(func() { By("Restoring intial dns autoscaling parameters") <del> ExpectNoError(updateDNSScalingConfigMap(c, packDNSScalingConfigMap(previousParams))) </del> <ins> Expect(updateDNSScalingConfigMap(c, packDNSScalingConfigMap(previousParams))).NotTo(HaveOccurred()) </ins> }) // This test is separated because it is slow and need to run serially


Question:
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
should this issue be moved into the 1.5 milestone and is it a release blocker? The test seems to flake fairly infrequently but we need to get the issue triaged as to whether it's a 1.5 release blocker. Please update this issue by 1PM PST 30 November 2016. Thanks! cc:
Since the feature it is testing is introduced for 1.5, so it should have the 1.5 milestone? This test is flaking because it is not well written, hence I don't think it is a release blocker. And I already sent out a which got LGTM before but recently rebased.
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
Thanks for triaging
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}

sum := 0 for _, mig := range strings.Split(framework.TestContext.CloudConfig.NodeInstanceGroup, ",") { size, err := GroupSize(mig) <del> ExpectNoError(err) </del> <ins> Expect(err).NotTo(HaveOccurred()) </ins> By(fmt.Sprintf("Initial size of %s: %d", mig, size)) originalSizes[mig] = size sum += size } <del> Expect(nodeCount).Should(Equal(sum)) </del> By("Manually increase cluster size") increasedSize := 0


Question:
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
should this issue be moved into the 1.5 milestone and is it a release blocker? The test seems to flake fairly infrequently but we need to get the issue triaged as to whether it's a 1.5 release blocker. Please update this issue by 1PM PST 30 November 2016. Thanks! cc:
Since the feature it is testing is introduced for 1.5, so it should have the 1.5 milestone? This test is flaking because it is not well written, hence I don't think it is a release blocker. And I already sent out a which got LGTM before but recently rebased.
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
Thanks for triaging
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}

increasedSize += increasedSizes[key] } setMigSizes(increasedSizes) <del> ExpectNoError(WaitForClusterSizeFunc(c, func(size int) bool { return size == increasedSize }, DNSscaleUpTimeout)) </del> <ins> Expect(WaitForClusterSizeFunc(c, func(size int) bool { return size == increasedSize }, scaleUpTimeout)).NotTo(HaveOccurred()) By("Wait for kube-dns scaled to expected number") getExpectReplicasLinear := getExpectReplicasFuncLinear(c, &DNSParams_1) Expect(waitForDNSReplicasSatisfied(c, getExpectReplicasLinear, DNSdefaultTimeout)).NotTo(HaveOccurred()) By("Replace the dns autoscaling parameters with another testing parameters") Expect(updateDNSScalingConfigMap(c, packDNSScalingConfigMap(DNSParams_3.data))).NotTo(HaveOccurred()) </ins> By("Wait for kube-dns scaled to expected number") <del> ExpectNoError(waitForDNSReplicasSatisfied(c, increasedSize, DNSdefaultTimeout)) </del> <ins> getExpectReplicasLinear = getExpectReplicasFuncLinear(c, &DNSParams_3) Expect(waitForDNSReplicasSatisfied(c, getExpectReplicasLinear, DNSdefaultTimeout)).NotTo(HaveOccurred()) </ins> By("Restoring cluster size") setMigSizes(originalSizes) <del> framework.ExpectNoError(framework.WaitForClusterSize(c, nodeCount, DNSscaleDownTimeout)) </del> <ins> Expect(framework.WaitForClusterSize(c, sum, scaleDownTimeout)).NotTo(HaveOccurred()) </ins> By("Wait for kube-dns scaled to expected number") <del> ExpectNoError(waitForDNSReplicasSatisfied(c, nodeCount, DNSdefaultTimeout)) </del> <ins> Expect(waitForDNSReplicasSatisfied(c, getExpectReplicasLinear, DNSdefaultTimeout)).NotTo(HaveOccurred()) </ins> }) It("kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios", func() { By("--- Scenario: should scale kube-dns based on changed parameters ---") <del> By("Replace the dns autoscaling parameters with the second testing parameters") ExpectNoError(updateDNSScalingConfigMap(c, packDNSScalingConfigMap(DNSParams_2))) </del> <ins> By("Replace the dns autoscaling parameters with another testing parameters") Expect(updateDNSScalingConfigMap(c, packDNSScalingConfigMap(DNSParams_3.data))).NotTo(HaveOccurred()) </ins> By("Wait for kube-dns scaled to expected number") <del> ExpectNoError(waitForDNSReplicasSatisfied(c, int(math.Ceil(float64(nodeCount)/2.0)), DNSdefaultTimeout)) </del> <ins> getExpectReplicasLinear := getExpectReplicasFuncLinear(c, &DNSParams_3) Expect(waitForDNSReplicasSatisfied(c, getExpectReplicasLinear, DNSdefaultTimeout)).NotTo(HaveOccurred()) </ins> By("--- Scenario: should re-create scaling parameters with default value when parameters got deleted ---") By("Delete the ConfigMap for autoscaler") err := deleteDNSScalingConfigMap(c) <del> ExpectNoError(err) </del> <ins> Expect(err).NotTo(HaveOccurred()) </ins> By("Wait for the ConfigMap got re-created") configMap, err := waitForDNSConfigMapCreated(c, DNSdefaultTimeout) <del> ExpectNoError(err) </del> <ins> Expect(err).NotTo(HaveOccurred()) </ins> By("Check the new created ConfigMap got the same data as we have") Expect(reflect.DeepEqual(previousParams, configMap.Data)).To(Equal(true)) <del> By("Replace the dns autoscaling parameters with the second testing parameters") ExpectNoError(updateDNSScalingConfigMap(c, packDNSScalingConfigMap(DNSParams_2))) </del> <ins> By("Replace the dns autoscaling parameters with another testing parameters") Expect(updateDNSScalingConfigMap(c, packDNSScalingConfigMap(DNSParams_2.data))).NotTo(HaveOccurred()) </ins> By("Wait for kube-dns scaled to expected number") <del> ExpectNoError(waitForDNSReplicasSatisfied(c, int(math.Ceil(float64(nodeCount)/2.0)), DNSdefaultTimeout)) </del> <ins> getExpectReplicasLinear = getExpectReplicasFuncLinear(c, &DNSParams_2) Expect(waitForDNSReplicasSatisfied(c, getExpectReplicasLinear, DNSdefaultTimeout)).NotTo(HaveOccurred()) </ins> By("--- Scenario: should recover after autoscaler pod got deleted ---") By("Delete the autoscaler pod for kube-dns") <del> ExpectNoError(deleteDNSAutoscalerPod(c)) </del> <ins> Expect(deleteDNSAutoscalerPod(c)).NotTo(HaveOccurred()) </ins> <del> By("Replace the dns autoscaling parameters with the third testing parameters") ExpectNoError(updateDNSScalingConfigMap(c, packDNSScalingConfigMap(DNSParams_3))) </del> <ins> By("Replace the dns autoscaling parameters with another testing parameters") Expect(updateDNSScalingConfigMap(c, packDNSScalingConfigMap(DNSParams_1.data))).NotTo(HaveOccurred()) </ins> By("Wait for kube-dns scaled to expected number") <del> ExpectNoError(waitForDNSReplicasSatisfied(c, int(math.Ceil(float64(nodeCount)/3.0)), DNSdefaultTimeout)) </del> <ins> getExpectReplicasLinear = getExpectReplicasFuncLinear(c, &DNSParams_1) Expect(waitForDNSReplicasSatisfied(c, getExpectReplicasLinear, DNSdefaultTimeout)).NotTo(HaveOccurred()) </ins> }) }) <ins> type DNSParamsLinear struct { data            map[string]string nodesPerReplica float64 coresPerReplica float64 } type getExpectReplicasFunc func(c clientset.Interface) int func getExpectReplicasFuncLinear(c clientset.Interface, params *DNSParamsLinear) getExpectReplicasFunc { return func(c clientset.Interface) int { var replicasFromNodes float64 var replicasFromCores float64 nodes := framework.GetReadySchedulableNodesOrDie(c).Items if params.nodesPerReplica > 0 { replicasFromNodes = math.Ceil(float64(len(nodes)) / params.nodesPerReplica) } if params.coresPerReplica > 0 { replicasFromCores = math.Ceil(float64(getScheduableCores(nodes)) / params.coresPerReplica) } return int(math.Max(1.0, math.Max(replicasFromNodes, replicasFromCores))) } } func getScheduableCores(nodes []v1.Node) int64 { var sc resource.Quantity for _, node := range nodes { if !node.Spec.Unschedulable { sc.Add(node.Status.Capacity[v1.ResourceCPU]) } } scInt64, scOk := sc.AsInt64() if !scOk { framework.Logf("unable to compute integer values of schedulable cores in the cluster") return 0 } return scInt64 } </ins> func fetchDNSScalingConfigMap(c clientset.Interface) (*v1.ConfigMap, error) { cm, err := c.Core().ConfigMaps(DNSNamespace).Get(DNSAutoscalerLabelName) if err != nil {


Question:
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
should this issue be moved into the 1.5 milestone and is it a release blocker? The test seems to flake fairly infrequently but we need to get the issue triaged as to whether it's a 1.5 release blocker. Please update this issue by 1PM PST 30 November 2016. Thanks! cc:
Since the feature it is testing is introduced for 1.5, so it should have the 1.5 milestone? This test is flaking because it is not well written, hence I don't think it is a release blocker. And I already sent out a which got LGTM before but recently rebased.
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
Thanks for triaging
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}
https://k8s- Failed: [] DNS horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {Kubernetes e2e suite}

return nil } <del> func waitForDNSReplicasSatisfied(c clientset.Interface, expected int, timeout time.Duration) (err error) { </del> <ins> func waitForDNSReplicasSatisfied(c clientset.Interface, getExpected getExpectReplicasFunc, timeout time.Duration) (err error) { </ins> var current int <del> framework.Logf("Waiting up to %v for kube-dns reach %v replicas", timeout, expected) </del> <ins> var expected int framework.Logf("Waiting up to %v for kube-dns to reach expected replicas", timeout) </ins> condition := func() (bool, error) { current, err = getDNSReplicas(c) if err != nil { return false, err } <ins> expected = getExpected(c) </ins> if current != expected { <ins> framework.Logf("replicas not as expected: got %v, expected %v", current, expected) </ins> return false, nil } return true, nil } <del> if err = wait.Poll(time.Second, timeout, condition); err != nil { </del> <ins> if err = wait.Poll(2*time.Second, timeout, condition); err != nil { </ins> return fmt.Errorf("err waiting for DNS replicas to satisfy %v, got %v: %v", expected, current, err) } <ins> framework.Logf("kube-dns reaches expected replicas: %v", expected) </ins> return nil }


Question:
Is this a request for help? No What keywords did you search in Kubernetes issues before filing this one? http proxy, httpproxy, httpsproxy, AWS, kubeadm, kubectl, cloud-provider Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT Kubernetes version (use ): v1.4.4 Environment: Cloud provider or hardware configuration: kubeadm on EC2 instances on AWS OS (e.g. from /etc/os-release): Ubuntu 16.04.1 LTS Kernel (e.g. ): Linux ip-10-0-133-99 4.4.0-45-generic -Ubuntu SMP Wed Oct 19 14:12:37 UTC 2016 x8664 x8664 x8664 GNU/Linux Install tools: kubeadm Others: httpproxy, httpsproxy and noproxy set in /etc/environment, Docker config as well on startup scripts that run What happened: won't complete and gets stuck on it's first deployment. When one looks through the logs it shows that the pods are downloading fine but we are stuck due to the kube-controller-manager continually crashing as it can't contact AWS. Elsewhere in kubernetes I see references to and if the AWS cloud provider could use that it would be able to connect as well. From the POD if I manually set httpproxy and httpsproxy and no_proxy I am able to use the AWS CLI without any issues. What you expected to happen: I would expect to successfully complete and use the environment proxies set up on the system so as not to depend on a default Internet Gateway being connected to the VPC just for AWS access. How to reproduce it (as minimally and precisely as possible): Set up a public VPC with Squid set up as a proxy Set up a private VPC peered to the public VPC Set up a master node and run Set up a regular node and run will never fully initialize. If you remove the --cloud-provider aws` then it will complete and nodes will register. Anything else do we need to know:
what do you have in proxy variables ? (env | grep -i proxy) it tries to connect to (tcp 176.32.118.53:443) directly. You're expecting to connect there via proxy ? then do you have something like in your no_proxy ?
Unfortunately on the pod itself for the the environment variables aren't passed through. As this is created using I don't have control over the pod definition. So basically none of the *_proxy variables are set inside the pod in the docker image, however if I and export the variables and try to wget that URL it works. Basically I'm hoping we can update to respect and pass the environment variables down to the system pods if they exist, eg, add: Without it I can't stand up a cluster with AWS integration on a completely private VPC and for such a small change it would make a rather big difference.
ok, so request is actually to pass proxy variables into containers. interesting. I'll check how it can be done. BTW, for noproxy value in your example: things like 10.0.0.0/8 would not work on 99% of software that respects noproxy variable. it must be always exact IP match.
you're right that what I'm actually asking for is to pass proxy variables through either from a commandline setting on kubeadm or pick it up from /etc/environment or shell environment that launches kubeadm. I didn't realize it till I dug in to try and think about how one might go about fixing this. At first I thought we would just pick up the variables from the system environment but then I realized kube-controller-manager is deployed in a container and so we'd need to update the yaml file that launches it to contain the environment variables. As I'd like to go to production with this soon hopefully this is a small change that can be .
This issue was moved to kubernetes/kubeadm

VolumeMounts:  volumeMounts, LivenessProbe: componentProbe(8080, "/healthz"), Resources:     componentResources("250m"), <ins> Env:           getProxyEnvVars(), </ins> }, volumes...), kubeControllerManager: componentPod(api.Container{ Name:          kubeControllerManager,


Question:
Is this a request for help? No What keywords did you search in Kubernetes issues before filing this one? http proxy, httpproxy, httpsproxy, AWS, kubeadm, kubectl, cloud-provider Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT Kubernetes version (use ): v1.4.4 Environment: Cloud provider or hardware configuration: kubeadm on EC2 instances on AWS OS (e.g. from /etc/os-release): Ubuntu 16.04.1 LTS Kernel (e.g. ): Linux ip-10-0-133-99 4.4.0-45-generic -Ubuntu SMP Wed Oct 19 14:12:37 UTC 2016 x8664 x8664 x8664 GNU/Linux Install tools: kubeadm Others: httpproxy, httpsproxy and noproxy set in /etc/environment, Docker config as well on startup scripts that run What happened: won't complete and gets stuck on it's first deployment. When one looks through the logs it shows that the pods are downloading fine but we are stuck due to the kube-controller-manager continually crashing as it can't contact AWS. Elsewhere in kubernetes I see references to and if the AWS cloud provider could use that it would be able to connect as well. From the POD if I manually set httpproxy and httpsproxy and no_proxy I am able to use the AWS CLI without any issues. What you expected to happen: I would expect to successfully complete and use the environment proxies set up on the system so as not to depend on a default Internet Gateway being connected to the VPC just for AWS access. How to reproduce it (as minimally and precisely as possible): Set up a public VPC with Squid set up as a proxy Set up a private VPC peered to the public VPC Set up a master node and run Set up a regular node and run will never fully initialize. If you remove the --cloud-provider aws` then it will complete and nodes will register. Anything else do we need to know:
what do you have in proxy variables ? (env | grep -i proxy) it tries to connect to (tcp 176.32.118.53:443) directly. You're expecting to connect there via proxy ? then do you have something like in your no_proxy ?
Unfortunately on the pod itself for the the environment variables aren't passed through. As this is created using I don't have control over the pod definition. So basically none of the *_proxy variables are set inside the pod in the docker image, however if I and export the variables and try to wget that URL it works. Basically I'm hoping we can update to respect and pass the environment variables down to the system pods if they exist, eg, add: Without it I can't stand up a cluster with AWS integration on a completely private VPC and for such a small change it would make a rather big difference.
ok, so request is actually to pass proxy variables into containers. interesting. I'll check how it can be done. BTW, for noproxy value in your example: things like 10.0.0.0/8 would not work on 99% of software that respects noproxy variable. it must be always exact IP match.
you're right that what I'm actually asking for is to pass proxy variables through either from a commandline setting on kubeadm or pick it up from /etc/environment or shell environment that launches kubeadm. I didn't realize it till I dug in to try and think about how one might go about fixing this. At first I thought we would just pick up the variables from the system environment but then I realized kube-controller-manager is deployed in a container and so we'd need to update the yaml file that launches it to contain the environment variables. As I'd like to go to production with this soon hopefully this is a small change that can be .
This issue was moved to kubernetes/kubeadm

VolumeMounts:  volumeMounts, LivenessProbe: componentProbe(10252, "/healthz"), Resources:     componentResources("200m"), <ins> Env:           getProxyEnvVars(), </ins> }, volumes...), kubeScheduler: componentPod(api.Container{ Name:          kubeScheduler,