Question:
For example, kubernetes-ro service appears with empty selector (this service was created by the hack/local cluster) - is that ok? What meaning such service has if it has no entity to fwd traffic to?
kubernetes and kubernetes-ro are currently "virtual" - they are not backed by running pods, but by the master itself. , abonas wrote:
if that's internal impl, why is it exposed to user? let's say a client pulls all entities and builds a UI around it. then a service with no selector arrives, and it's internal to kube - how the client should know not to present it/touch it? (similar to the "net" container in pods)
It's exposed exactly FOR users - so they can find the apiserver. It has a corresponding Endpoints object, it gets a portal IP and a kube-proxy rule, and it gets service env vars and a DNS name. The fact that it is virtual is an implementation detail. That it exists is all users should need to know. , abonas wrote:
And FWIW, the master components may one day actually be pods... , Tim Hockin wrote:
thanks for the explanation. so any general rule how to distinguish kube pods/services from user defined?
Do we need to distinguish them? We could add labels, I guess, if it is really important to distinguish them.
sometimes users would want to see only the things they defined and not implementation details of the system.
One could argue this is why the users should be using labels. I'm not against defining labels for "special" stuff, but it's new territory, so we should enter with caution. , abonas wrote:
I think the right answer might be to put them in a special "cluster" or "system" namespace. , Tim Hockin wrote:
See If we put them in a "kubernetes" ns, the DNS name would be :) I'm not against segmenting them into a different namespace. We just need to decide on what that namespace is called, plumb through that configuration, decide on compatibility for a short while or not, and decide if that namespace is "special" or just "different" - i.e. should the skydns, heapster, etc pods go into this namespace? , Brendan Burns wrote:
As said, the services are there for applications that need to contact the apiserver. Additionally, nil selectors can be used by users to import external services. Adding labels to the kubernetes services won't help. Users need to manage their objects using labels. /cc
Yeah, lots of consumers will expose "special" services. IN fact, in many clusters the "special" services will outweigh the unspecial ones. At this point, the namespace they are in is completely arbitrary, and I think because it is a name, it should be free for the end user to choose where they are surfaced.
Per the proposal here: , I think the namespace should continue to hold these services, and they require no special labels. I sense comment is made without thinking through what Kubernetes operators do to bootstrap a cluster. I think it's reasonable that namespace holds everything you need to run, and operators may or may not choose to segment their data in other namespaces. If they want to not see this information, its an exact reason for why they should use their own namespace. I don't think we need anything new here.
- I'm not clear on your meaning. Did you mean that we should make the "cluster admin" namespace a config param that everyone knows about and can be different cluster-to-cluster? Or did you mean that default is as arbitrary a name as any other? I'd be fine moving things out of default into "kube" or "kubernetes" or "cluster" or something, but I am less convinced that a flag is a good thing. , Clayton Coleman wrote:
- wherever we stick this stuff, it should be the namespace that is initially bootstrapped, which today is . I continue to believe that is as good a name as any other. I do not see a need to do more here.
I think as soon as you bring naming into the picture, making decisions based on names gets very tricky (what if I want my logging service to be everywhere?). If there is a magic namespace that exposes things to the rest of the cluster, we should ask whether there's only one magic namespace, or multiple? Are services supposed to cross namespaces? It's great that pods can find the master... but that's not the only thing they'll need to find in each namespace. And baking in special names (or customizable names) into your app code is fragile, because then your code only works inside Kubernetes. I'm just trying to pump the brakes on special services prior to us designing them... :) ----- Original Message -----
oh, to be clear, I don't want "special" services - I am merely asking if moving non-user-defined objects (where that is defined very loosely) out of "default" is worthwhile. I tend to agree with Derek that it's not worth energy. , Clayton Coleman wrote:
agree with you that lots of consumers will expose "special" services We should not use the (namespace + k8s-service + skydns) mechanism for publishing the location of those special services. We should decouple the choice of namespace to run the pods of the service in from the well-known dns address of special services. Otherwise, it will be hard to move the pods of the service to another namespace, which people will definitely want to do as they grow. Instead, we need a place to publish well-known services which is not tied to the namespace of the pods of that service.
you open a whole new topic. I've noodled with the idea of a Name resource type. We would generate a Name for each Service as "service.namespace", but users could also add their own Name resources. , Eric Tune wrote:
I'd like to understand better your comment regarding nil selectors used to export external services - can you elaborate? give an example? my understanding is that service redirects traffic to pods based on labels. with nil selector service does nothing, so why is it a service in the first place? I'm pro decoupling special services/internal implementation services from user defined entities. whether by separate namespace, or by saying that the default namespace is reserved, or by any other solution. because otherwise internal implementation and user defined content are mixed and that's no good for a proper management. just tagging you so you could see this issue and the related discussions.
Services abstract backends - they don't have to abstract pods. Some examples: You want to have an external database cluster in production, but in test you use your own databases. You would have a service for both and use the environment variables, but the production service has a nil selector with its endpoints set to the database cluster. You want to have an empty service because you're reserving a name (that isn't accessible yet) but will be in the future You want to point your service to a service in another namespace or on another cluster. You set the endpoints for your service to the publicIPs of the other service. ----- Original Message -----
, thanks. regarding the last point - I was under impression (based on some other discussion) that there would not be cross namespace interaction? that entities can can access only entities in the same namespace?
Cross-namespace is currently allowed and I think has to be allowed, though perhaps (in the fullness of time) that can be policy based rather than wide open. , abonas wrote:
please see this correspondence. my understanding is that there is isolation
Responded there. I think there's some less-than-crystal-clear words in play. , abonas wrote:
+1 to Eric's comment about ensuring there is a consistent manner for system services like kubernetes-ro, if that's not fixes for all clusters it's going to be a nightmare to write reusable containers for multiple different clusters. I don't care if we do that by using a constant namespace, or by giving system services two different names, but there needs to be at least one name that is constant. Regarding default vs. cluster namespace, I think the issue is one of clutter as a user is starting to understand k8s, you turn up a new cluster, launch a pod, or service, and it's hard to find your one object on the sea of system created pods and services. It's visual clutter that confuses people as they onboard. On Jan 23, 2015 3:01 PM, "Tim Hockin" wrote:
- can you drop me a line (my github name at google) - I want to ask you something, and you have no contact info :) , abonas wrote:
sorry for naive question but why would I care as an user that wants to run its existing workload on Kubernetes about kubernetes-ro ? If I write an application that uses kubernetes-ro then I'm locked into Kubernetes. I understand the need for kubernetes-ro being accessible by cluster level services such as monitoring, logging. I also can imagine that a power user might want to start its own cluster services that use kubernetes-ro. I don't see clearly why would majority of users care about kubernetes-ro being visible from their container. I see a reason for the cluster services being visible and accessible for Kubernetes developer, but from an user perspective such services only make the system more complex: why do I see these services (pods,rc) ? are they related to my application ? should I care about them (especially when I use hosted Kubernetes) ? why does my container get all these environment variables for cluster services ?
I will update file to explain why one would have service without a selector.
Right now, every service goes everywhere for simplicity. Name spacing and filtering probably make sense in the long run, but it's not a high priority for v1.0 Brendan On Mar 4, 2015 5:02 AM, "Rafał Sokołowski" wrote:

`service`, a client can simply connect to $MYAPP_SERVICE_HOST on port $MYAPP_SERVICE_PORT. <ins> ## Service without selector Services, in addition to providing clean abstraction to access pods, can also abstract any kind of backend: - you want to have an external database cluster in production, but in test you use your own databases. - you want to point your service to a service in another [`namespace`](namespaces.md) or on another cluster. - you are migrating your workload to Kubernetes and some of your backends run outside of Kubernetes. In any of these scenarios you can define a service without a selector: ```json "kind": "Service", "apiVersion": "v1beta1", "id": "myapp", "port": 8765 ``` then you can explicitly map the service to a specific endpoint(s): ```json "kind": "Endpoints", "apiVersion": "v1beta1", "id": "myapp", "endpoints": ["173.194.112.206:80"] ``` Access to the service without a selector works the same as if it had selector. The traffic will be routed to endpoints defined by the user (`173.194.112.206:80` in case of this example). </ins> ## How do they work? Each node in a Kubernetes cluster runs a `service proxy`.  This application


Question:
As long as we have it in the API, it should maybe be renamed to gcePersistentDisk ?
SGTM

EmptyDir *EmptyDir `json:"emptyDir"` // GCEPersistentDisk represents a GCE Disk resource that is attached to a // kubelet's host machine and then exposed to the pod. <del> GCEPersistentDisk *GCEPersistentDisk `json:"persistentDisk"` </del> <ins> GCEPersistentDisk *GCEPersistentDisk `json:"gcePersistentDisk"` </ins> // GitRepo represents a git repository at a particular revision. GitRepo *GitRepo `json:"gitRepo"` }


Question:
This whole set of tests (if it can't be run) should be omitted from the test execution, otherwise folks running on Mac can't verify everything else is clean in a sane fashion. Passive aggressive assign to Tim, I may get around to fixing this before he does.
Last I checked Travis + Go didn't work. We should check again, and enable OS X builds in Travis if it's possible.
You could argue that if the unsupported mount () is being called in a test, there is a problem anyway - it has to run as root. Will investigate. On Jan 25, 2015 9:09 AM, "Brendan Burns" wrote:
Without a Mac to test on, I feel like this would be better fixed by someone who uses Mac... Probably: just rename to I just discovered "implicit build constraints", sigh. Go is crossing from user-friendly to sort of absurd, IMO.

<del> // +build !windows /* Copyright 2014 Google Inc. All rights reserved. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package gce_pd import ( "os" "syscall" ) // Determine if a directory is a mountpoint, by comparing the device for the directory // with the device for it's parent.  If they are the same, it's not a mountpoint, if they're // different, it is. func isMountPoint(file string) (bool, error) { stat, err := os.Stat(file) if err != nil { return false, err } rootStat, err := os.Lstat(file + "/..") if err != nil { return false, err } // If the directory has the same device as parent, then it's not a mountpoint. return stat.Sys().(*syscall.Stat_t).Dev != rootStat.Sys().(*syscall.Stat_t).Dev, nil } </del>


Question:
This whole set of tests (if it can't be run) should be omitted from the test execution, otherwise folks running on Mac can't verify everything else is clean in a sane fashion. Passive aggressive assign to Tim, I may get around to fixing this before he does.
Last I checked Travis + Go didn't work. We should check again, and enable OS X builds in Travis if it's possible.
You could argue that if the unsupported mount () is being called in a test, there is a problem anyway - it has to run as root. Will investigate. On Jan 25, 2015 9:09 AM, "Brendan Burns" wrote:
Without a Mac to test on, I feel like this would be better fixed by someone who uses Mac... Probably: just rename to I just discovered "implicit build constraints", sigh. Go is crossing from user-friendly to sort of absurd, IMO.

<ins> // +build !windows /* Copyright 2014 Google Inc. All rights reserved. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package gce_pd import ( "os" "syscall" ) // Determine if a directory is a mountpoint, by comparing the device for the directory // with the device for it's parent.  If they are the same, it's not a mountpoint, if they're // different, it is. func isMountPoint(file string) (bool, error) { stat, err := os.Stat(file) if err != nil { return false, err } rootStat, err := os.Lstat(file + "/..") if err != nil { return false, err } // If the directory has the same device as parent, then it's not a mountpoint. return stat.Sys().(*syscall.Stat_t).Dev != rootStat.Sys().(*syscall.Stat_t).Dev, nil } </ins>


Question:
This whole set of tests (if it can't be run) should be omitted from the test execution, otherwise folks running on Mac can't verify everything else is clean in a sane fashion. Passive aggressive assign to Tim, I may get around to fixing this before he does.
Last I checked Travis + Go didn't work. We should check again, and enable OS X builds in Travis if it's possible.
You could argue that if the unsupported mount () is being called in a test, there is a problem anyway - it has to run as root. Will investigate. On Jan 25, 2015 9:09 AM, "Brendan Burns" wrote:
Without a Mac to test on, I feel like this would be better fixed by someone who uses Mac... Probably: just rename to I just discovered "implicit build constraints", sigh. Go is crossing from user-friendly to sort of absurd, IMO.

<del> // +build !linux </del> <ins> // +build windows </ins> /* Copyright 2014 Google Inc. All rights reserved.


Question: wrote: I just ran across this the other day. but I am seeing the error in the service. Here is a sample of the output. I am running v0.8.2. I can get the behavior to go away temporarily by rebooting the node or restarting the service but it always seems to come back eventually.

} if event.Type == watch.Error { util.HandleError(fmt.Errorf("error from watch during sync: %v", errors.FromObject(event.Object))) <ins> // Clear the resource version, this may cause us to skip some elements on the watch, // but we'll catch them on the synchronize() call, so it works out. *resourceVersion = "" </ins> continue } glog.V(4).Infof("Got watch: %#v", event) rc, ok := event.Object.(*api.ReplicationController) if !ok { <ins> if status, ok := event.Object.(*api.Status); ok { if status.Status == api.StatusFailure { glog.Errorf("failed to watch: %v", status) // Clear resource version here, as above, this won't hurt consistency, but we // should consider introspecting more carefully here. (or make the apiserver smarter) // "why not both?" *resourceVersion = "" continue } } </ins> util.HandleError(fmt.Errorf("unexpected object: %#v", event.Object)) continue }


Question:
We found yesterday that cAdvisor wasn't being started properly in 0.9.0 and 0.9.1 due to an issue with manifest file format changes (, fixed in ). If any of our e2e tests even checked that cAdvisor was running or exposing metrics on all nodes, it would have been caught before release. We should add at least some coverage for it to catch bugs like this.
If any of you are up for adding an e2e test for cadvisor, that'd be much appreciated!
I will work on adding one for cadvisor. , Alex Mohr wrote:
We already have a e2e test, which validate monitoring of Clusters using cAdvisor. We should figure out why the test passed without detecting such failure, and improve that test.
The existing test will succeed even if some of the nodes do not have cadvisor running. We can enhance the monitoring test to specifically check for cadvisor's health as well. , Dawn Chen wrote:
This test seems to be failing deterministically.
This was after fix to retry, but we're not exactly sure why.
Once I have emerged from the e2e test for cluster level logging with Elasticsearch quagmire (soon I hope -- perhaps by Wednesday) then I might also be willing to lend a hand with this since some of the experience might be relevant.
Actually latest error looks like So it's still not coming up after 5 minutes. The question is how long should it be taking to come up, given that the cadvisor containers get scheduled basically on node creation. The test finished in < 1s on my e2e cluster that has been up for awhile.
Or there's an actual bug here. can you look into it?
(I looked at the Jenkins log, and this test ran a good 10 minutes after startup, so it probably had 15 minutes to finish coming up. That seems wrong, given that cAdvisor is actually the first container we ever schedule, and it gets scheduled raw.)
Sorry, I hadn't seen -- not sure if that fixes this?
No, there are failures after as well. On Feb 20, 2015 5:39 PM, "David Oppenheimer" wrote:
I don't see any issue with the test itself. The test attempts to access a REST endpoint ('/stats') via the proxy on the api-server. Is it possible that the api-server doesn't get to a stable state for a while?
This test executes after several others that hit the apiserver? On Feb 22, 2015 4:15 PM, "Vish Kannan" wrote:
Noob question: How can we tell if it runs this test after the other tests?
You can't tell deterministically (i.e. it's not hardcoded). I was merely commenting that in several runs where this fails, it has run after many other tests. We've been randomizing the spec order for a while. You can look at the Jenkins console logs to get the random seed for a given run: And the console logs have the order just by manually scanning the horizontal bars, but it's kind of pain. (That seed was off of go/k8s-test/job/kubernetes-e2e-gce/2638/consoleFull)
The reason for this failure is . I am closing this issue since the test itself is fine.

<ins> /* Copyright 2015 Google Inc. All rights reserved. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package e2e import ( "fmt" "time" "github.com/GoogleCloudPlatform/kubernetes/pkg/client" . "github.com/onsi/ginkgo" ) var _ = Describe("Cadvisor", func() { var c *client.Client BeforeEach(func() { var err error c, err = loadClient() expectNoError(err) }) It("cadvisor should be healthy on every node.", func() { CheckCadvisorHealthOnAllNodes(c) }) }) func CheckCadvisorHealthOnAllNodes(c *client.Client) { By("getting list of nodes") nodeList, err := c.Nodes().List() expectNoError(err) for _, node := range nodeList.Items { // cadvisor is not accessible directly unless its port (4194 by default) is exposed. // Here, we access '/stats/' REST endpoint on the kubelet which polls cadvisor internally. statsResource := fmt.Sprintf("api/v1beta1/proxy/minions/%s/stats/", node.Name) By(fmt.Sprintf("Querying stats from node %s using url %s", node.Name, statsResource)) _, err = c.Get().AbsPath(statsResource).Timeout(1 * time.Second).Do().Raw() expectNoError(err) } } </ins>


Question: W0128 18:44:26. ] Pod from test failed validation, ignoring: namespace: required value '' E0128 18:44:26. ] Could not construct reference to: '&api.BoundPod{TypeMeta:api.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:api.ObjectMeta{Name:"foo", Namespace:"", SelfLink:"", UID:"foo", ResourceVersion:"", CreationTimestamp:util.Time{Time:time.Time{sec:0, nsec:0x0, loc:(time.Location)(nil)}}, Labels:map[string]string(nil), Annotations:map[string]string{"":""}}, Spec:api.PodSpec{Volumes:[]api.Volume(nil), Containers:[]api.Container(nil), RestartPolicy:api.RestartPolicy{Always:(api.RestartPolicyAlways)(0xe3d670), OnFailure:(api.RestartPolicyOnFailure)(nil), Never:(api.RestartPolicyNever)(nil)}, DNSPolicy:"ClusterFirst", NodeSelector:map[string]string(nil), Host:""}}' due to: 'selfLink was empty, can't make reference'. Will not report event: 'failedValidation' 'Error validating pod foo.<empty-namespacefrom test, ignoring: namespace: required value E0128 18:44:28. ] Unable to read config path "randomnonexistentpath": path does not exist, ignoring --- FAIL: TestReadFromFileWithoutID (0.20 seconds) Expected update, timeout instead

t.Fatalf("Expected %#v, Got %#v", expected, update) } <del> case <-time.After(2 * time.Millisecond): </del> <ins> case <-time.After(time.Second): </ins> t.Errorf("Expected update, timeout instead") } }


Question: W0128 18:44:26. ] Pod from test failed validation, ignoring: namespace: required value '' E0128 18:44:26. ] Could not construct reference to: '&api.BoundPod{TypeMeta:api.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:api.ObjectMeta{Name:"foo", Namespace:"", SelfLink:"", UID:"foo", ResourceVersion:"", CreationTimestamp:util.Time{Time:time.Time{sec:0, nsec:0x0, loc:(time.Location)(nil)}}, Labels:map[string]string(nil), Annotations:map[string]string{"":""}}, Spec:api.PodSpec{Volumes:[]api.Volume(nil), Containers:[]api.Container(nil), RestartPolicy:api.RestartPolicy{Always:(api.RestartPolicyAlways)(0xe3d670), OnFailure:(api.RestartPolicyOnFailure)(nil), Never:(api.RestartPolicyNever)(nil)}, DNSPolicy:"ClusterFirst", NodeSelector:map[string]string(nil), Host:""}}' due to: 'selfLink was empty, can't make reference'. Will not report event: 'failedValidation' 'Error validating pod foo.<empty-namespacefrom test, ignoring: namespace: required value E0128 18:44:28. ] Unable to read config path "randomnonexistentpath": path does not exist, ignoring --- FAIL: TestReadFromFileWithoutID (0.20 seconds) Expected update, timeout instead

t.Errorf("Unexpected UID: %s", update.Pods[0].ObjectMeta.UID) } <del> case <-time.After(2 * time.Millisecond): </del> <ins> case <-time.After(time.Second): </ins> t.Errorf("Expected update, timeout instead") } }


Question:
writes: As we start looking into etcd and event memory pressure [], we should look into switching over to etcd 2.0 which was released today.
cc/
+1 On Jan 28, 2015 5:29 PM, "Dawn Chen" wrote:
My long-running cluster tests resulted in a hard-crash on etcd: We should move to 2.0 ASAP in the hopes of (a) fixing the issue and if not (b) prioritizing the fix since we'll be complaining about head. Abhi, is this something you can pick up and drive to land in O(days) please?
Calling this P0 as it blocks further long-running cluster stability testing.
I will do this today. , Alex Mohr wrote:

#    shasum <tar> # 6. Update this file with new tar version and new hash <del> {% set etcd_version="v0.4.6" %} </del> <ins> {% set etcd_version="v2.0.0" %} </ins> {% set etcd_tar_url="https://storage.googleapis.com/kubernetes-release/etcd/etcd-%s-linux-amd64.tar.gz" | format(etcd_version)  %} <del> {% set etcd_tar_hash="sha1=5db514e30b9f340eda00671230d5136855ae14d7" %} </del> <ins> {% set etcd_tar_hash="sha1=b3cd41d1748bf882a58a98c9585fd5849b943811" %} </ins> etcd-tar: archive:


Question:
writes: As we start looking into etcd and event memory pressure [], we should look into switching over to etcd 2.0 which was released today.
cc/
+1 On Jan 28, 2015 5:29 PM, "Dawn Chen" wrote:
My long-running cluster tests resulted in a hard-crash on etcd: We should move to 2.0 ASAP in the hopes of (a) fixing the issue and if not (b) prioritizing the fix since we'll be complaining about head. Abhi, is this something you can pick up and drive to land in O(days) please?
Calling this P0 as it blocks further long-running cluster stability testing.
I will do this today. , Alex Mohr wrote:

DESC="The etcd key-value share configuration service" NAME=etcd DAEMON=/usr/local/bin/$NAME <del> DAEMON_ARGS="-peer-addr $HOSTNAME:7001 -name $HOSTNAME" </del> <ins> # DAEMON_ARGS="-peer-addr $HOSTNAME:7001 -name $HOSTNAME" host_ip=$(hostname -i) DAEMON_ARGS="-addr ${host_ip}:4001 -bind-addr ${host_ip}:4001 -data-dir /var/etcd -initial-advertise-peer-urls http://${HOSTNAME}:2380 -name ${HOSTNAME} -initial-cluster ${HOSTNAME}=http://${HOSTNAME}:2380" </ins> DAEMON_LOG_FILE=/var/log/$NAME.log PIDFILE=/var/run/$NAME.pid SCRIPTNAME=/etc/init.d/$NAME


Question:
writes: As we start looking into etcd and event memory pressure [], we should look into switching over to etcd 2.0 which was released today.
cc/
+1 On Jan 28, 2015 5:29 PM, "Dawn Chen" wrote:
My long-running cluster tests resulted in a hard-crash on etcd: We should move to 2.0 ASAP in the hopes of (a) fixing the issue and if not (b) prioritizing the fix since we'll be complaining about head. Abhi, is this something you can pick up and drive to land in O(days) please?
Calling this P0 as it blocks further long-running cluster stability testing.
I will do this today. , Alex Mohr wrote:

exit 1 fi <ins> version=$(etcd -version | cut -d " " -f 3) if [[ "${version}" < "2.0.0" ]]; then kube::log::usage "etcd version 2.0.0 or greater required." exit 1 fi </ins> # Start etcd ETCD_DIR=$(mktemp -d -t test-etcd.XXXXXX) <del> etcd -name test -data-dir ${ETCD_DIR} -addr ${host}:${port} >/dev/null 2>/dev/null & </del> <ins> kube::log::usage "etcd -data-dir ${ETCD_DIR} -addr ${host}:${port} >/dev/null 2>/dev/null" etcd -data-dir ${ETCD_DIR} -addr ${host}:${port} >/dev/null 2>/dev/null & </ins> ETCD_PID=$! <del> kube::util::wait_for_url "http://${host}:${port}/v2/keys/" "etcd: " </del> <ins> echo "Waiting for etcd to come up." while true; do if curl -L http://127.0.0.1:4001/v2/keys/test -XPUT -d value="test"; then break fi done kube::util::wait_for_url "http://${host}:${port}/v2/keys/test" "etcd: " </ins> } kube::etcd::cleanup() {


Question:
writes: As we start looking into etcd and event memory pressure [], we should look into switching over to etcd 2.0 which was released today.
cc/
+1 On Jan 28, 2015 5:29 PM, "Dawn Chen" wrote:
My long-running cluster tests resulted in a hard-crash on etcd: We should move to 2.0 ASAP in the hopes of (a) fixing the issue and if not (b) prioritizing the fix since we'll be complaining about head. Abhi, is this something you can pick up and drive to land in O(days) please?
Calling this P0 as it blocks further long-running cluster stability testing.
I will do this today. , Alex Mohr wrote:

KUBE_ROOT=$(dirname "${BASH_SOURCE}")/../.. <del> ETCD_VERSION=${ETCD_VERSION:-v0.4.6} </del> <ins> ETCD_VERSION=${ETCD_VERSION:-v2.0.0} </ins> cd "${KUBE_ROOT}/third_party" curl -sL https://github.com/coreos/etcd/releases/download/${ETCD_VERSION}/etcd-${ETCD_VERSION}-linux-amd64.tar.gz


Question:
reported a variety of failure modes in .
+1
I figured I'd add to this issue rather than create a new one. I attempted to create a vagrant cluster today (dev tree is up to date) with 1 Master and 1 Minion and the cluster creation is stuck in the 'Validating master' loop ( for over 10 minutes in the validating loop ). salt-master and salt-minion logs seem to be empty ( they don't appear to be running, in fact ).
I still have the cluster running in case anybody wants additional information.
/cc please let me know if there is any other information about my vagrant setup that I need to post here.
I have been working to get more stability in this environment over last few days. Can you try again from HEAD after this merges:
Can you also verify that you built a release prior to running cluster/Kube-up?
yes, I started from scratch and built a release before running cluster/kube- - cluster/kube- seems to fail fairly early in case there is no release build available.
It looks like got merged. I'll try again and get back to you.
The vagrant cluster seems to come up ( completes master and minion validation ). However, there is new credentials related error that is showing up at the end of running cluster/kube- :
I'll add one more problem I've encountered. Runing from head:
New to me. Will investigate more tomorrow. Sent from my iPhone
Also new to me :-) - will see if I can reproduce either. Sent from my iPhone
Your credendials and certificates are not working with new environment. Just delete your .kubernetesvagrant* files and recreate the cluster. It would be nice to automate this task, if you tear down the cluster.
I am able to reproduce the 502 Bad Gateway error and am working on determining what changed in order to provide a resolution over the last couple days. Please assign this issue to me.
The issue appears to me that the kube-apiserver is not getting provisioned by salt correctly. Parsing Salt logs to determine why.
Looks like the file is not getting laid down by salt for some reason ;-)
The Jinja template was changed as part of identifying the cluster. Fixing up now, and will send a PR to close this issue.
that was it, thanks!

# The IP of the master export MASTER_IP="10.245.1.2" <del> export INSTANCE_PREFIX=kubernetes </del> <ins> export INSTANCE_PREFIX="kubernetes" </ins> export MASTER_NAME="${INSTANCE_PREFIX}-master" # Map out the IPs, names and container subnets of each minion


Question:
reported a variety of failure modes in .
+1
I figured I'd add to this issue rather than create a new one. I attempted to create a vagrant cluster today (dev tree is up to date) with 1 Master and 1 Minion and the cluster creation is stuck in the 'Validating master' loop ( for over 10 minutes in the validating loop ). salt-master and salt-minion logs seem to be empty ( they don't appear to be running, in fact ).
I still have the cluster running in case anybody wants additional information.
/cc please let me know if there is any other information about my vagrant setup that I need to post here.
I have been working to get more stability in this environment over last few days. Can you try again from HEAD after this merges:
Can you also verify that you built a release prior to running cluster/Kube-up?
yes, I started from scratch and built a release before running cluster/kube- - cluster/kube- seems to fail fairly early in case there is no release build available.
It looks like got merged. I'll try again and get back to you.
The vagrant cluster seems to come up ( completes master and minion validation ). However, there is new credentials related error that is showing up at the end of running cluster/kube- :
I'll add one more problem I've encountered. Runing from head:
New to me. Will investigate more tomorrow. Sent from my iPhone
Also new to me :-) - will see if I can reproduce either. Sent from my iPhone
Your credendials and certificates are not working with new environment. Just delete your .kubernetesvagrant* files and recreate the cluster. It would be nice to automate this task, if you tear down the cluster.
I am able to reproduce the 502 Bad Gateway error and am working on determining what changed in order to provide a resolution over the last couple days. Please assign this issue to me.
The issue appears to me that the kube-apiserver is not getting provisioned by salt correctly. Parsing Salt logs to determine why.
Looks like the file is not getting laid down by salt for some reason ;-)
The Jinja template was changed as part of identifying the cluster. Fixing up now, and will send a PR to close this issue.
that was it, thanks!

dns_replicas: '$(echo "$DNS_REPLICAS" | sed -e "s/'/''/g")' dns_server: '$(echo "$DNS_SERVER_IP" | sed -e "s/'/''/g")' dns_domain: '$(echo "$DNS_DOMAIN" | sed -e "s/'/''/g")' <ins> instance_prefix: '$(echo "$INSTANCE_PREFIX" | sed -e "s/'/''/g")' </ins> EOF # Configure the salt-master


Question:
contains function get-password. This uses a template which gets the current context. But a user who has not yet used the kubectl contexts feature has no context set. This causes a nil pointer deref while evaluating the template.
I think kube-up creates the needed entries in the .kubeconfig file. But it calls get-password before it does that. Need to allow the kubectl config view command to fail.
okay, I have a fix.

#   KUBE_PASSWORD function get-password { # go template to extract the auth-path of the current-context user <del> local template='{{$ctx := index . "current-context"}}{{$user := index . "contexts" $ctx "user"}}{{index . "users" $user "auth-path"}}' </del> <ins> local template='{{with $ctx := index . "current-context"}}{{$user := index . "contexts" $ctx "user"}}{{index . "users" $user "auth-path"}}{{end}}' </ins> local file=$("${KUBE_ROOT}/cluster/kubectl.sh" config view -o template --template="${template}") <del> if [[ -r "$file" ]]; then </del> <ins> if [[ ! -z "$file" && -r "$file" ]]; then </ins> KUBE_USER=$(cat "$file" | python -c 'import json,sys;print json.load(sys.stdin)["User"]') KUBE_PASSWORD=$(cat "$file" | python -c 'import json,sys;print json.load(sys.stdin)["Password"]') return


Question:
I have been following the instructions here: to try to explore the kube UI. by executing: I get the following which shows it successfully started: When I try to access the page I get a 404:
could you try running ? I believe is a typo of
Also, try accessing instead.
thanks

Start the server: ```sh <del> cluster/kubectl.sh proxy -www=$PWD/www </del> <ins> cluster/kubectl.sh proxy --www=$PWD/www </ins> ``` The UI should now be running on [localhost](http://localhost:8001/static/index.html#/groups//selector)


Question:
I tried to create a pod without containers to see if it's supported or not. (does a pod without containers have a meaning in k8s?) I was getting "conflict 409 already exists" error although I used a new pod with a new name/id, so it's not clear whether it's supported or not, and whether this error is a mistake (409 instead of 400)
What is the reason to create pod without specifiing image / creating container? Please provide a use case when such functionality is necessry.
- to me it's not necessary. However if it's not supported, a validation should prevent it not to allow corrupted entities. And if it is supported, it will be good to understand why. For instance, service with nil selector is supported, though it might not make sense to some users. see this:
Yes, it should be allowed, and I'm pretty sure it used to work. Could you please post your object schema here or somewhere?
what do you mean by object schema? an example json of such pod or something broader? and btw what meaning a pod has without containers?
cc
I'm fine with banning this for now. The use cases where it would be useful (e.g., reserving resources, staging deployment, prefetching images, ...) aren't currently well supported. When we add the features that could take advantage of this (e.g., pod-level resources, additional of containers during update, container volumes, pod/volume init hooks), then we can make it legal again at that point.
in that case, a clear error message coming from REST would be in place when a pod with no containers is recieved by the server :)
I'll look into this. AFAIU we want to return meaningful error if user specifies pod with no containers. Is my understanding correct?
Correct. We should do it in validateContainers:
cc
should we document this somewhere (e.g. ). Validation is not versioned, so any change would apply to all existing APIs.
Yes, it should be documented, in the field descriptions, such as:
I'll send a PR for this.
cc

// PodSpec is a description of a pod type PodSpec struct { <del> Volumes       []Volume      `json:"volumes"` </del> <ins> Volumes []Volume `json:"volumes"` // Required: there must be at least one container in a pod. </ins> Containers    []Container   `json:"containers"` RestartPolicy RestartPolicy `json:"restartPolicy,omitempty"` // Required: Set DNS policy.


Question:
I tried to create a pod without containers to see if it's supported or not. (does a pod without containers have a meaning in k8s?) I was getting "conflict 409 already exists" error although I used a new pod with a new name/id, so it's not clear whether it's supported or not, and whether this error is a mistake (409 instead of 400)
What is the reason to create pod without specifiing image / creating container? Please provide a use case when such functionality is necessry.
- to me it's not necessary. However if it's not supported, a validation should prevent it not to allow corrupted entities. And if it is supported, it will be good to understand why. For instance, service with nil selector is supported, though it might not make sense to some users. see this:
Yes, it should be allowed, and I'm pretty sure it used to work. Could you please post your object schema here or somewhere?
what do you mean by object schema? an example json of such pod or something broader? and btw what meaning a pod has without containers?
cc
I'm fine with banning this for now. The use cases where it would be useful (e.g., reserving resources, staging deployment, prefetching images, ...) aren't currently well supported. When we add the features that could take advantage of this (e.g., pod-level resources, additional of containers during update, container volumes, pod/volume init hooks), then we can make it legal again at that point.
in that case, a clear error message coming from REST would be in place when a pod with no containers is recieved by the server :)
I'll look into this. AFAIU we want to return meaningful error if user specifies pod with no containers. Is my understanding correct?
Correct. We should do it in validateContainers:
cc
should we document this somewhere (e.g. ). Validation is not versioned, so any change would apply to all existing APIs.
Yes, it should be documented, in the field descriptions, such as:
I'll send a PR for this.
cc

// PodSpec is a description of a pod type PodSpec struct { <del> Volumes       []Volume      `json:"volumes" description:"list of volumes that can be mounted by containers belonging to the pod"` Containers    []Container   `json:"containers" description:"list of containers belonging to the pod; containers cannot currently be added or removed"` </del> <ins> Volumes []Volume `json:"volumes" description:"list of volumes that can be mounted by containers belonging to the pod"` // Required: there must be at least one container in a pod. Containers    []Container   `json:"containers" description:"list of containers belonging to the pod; containers cannot currently be added or removed; there must be at least one container in a Pod"` </ins> RestartPolicy RestartPolicy `json:"restartPolicy,omitempty" description:"restart policy for all containers within the pod; one of RestartPolicyAlways, RestartPolicyOnFailure, RestartPolicyNever"` // Optional: Set DNS policy.  Defaults to "ClusterFirst" DNSPolicy DNSPolicy `json:"dnsPolicy,omitempty" description:"DNS policy for containers within the pod; one of 'ClusterFirst' or 'Default'"`


Question:
I tried to create a pod without containers to see if it's supported or not. (does a pod without containers have a meaning in k8s?) I was getting "conflict 409 already exists" error although I used a new pod with a new name/id, so it's not clear whether it's supported or not, and whether this error is a mistake (409 instead of 400)
What is the reason to create pod without specifiing image / creating container? Please provide a use case when such functionality is necessry.
- to me it's not necessary. However if it's not supported, a validation should prevent it not to allow corrupted entities. And if it is supported, it will be good to understand why. For instance, service with nil selector is supported, though it might not make sense to some users. see this:
Yes, it should be allowed, and I'm pretty sure it used to work. Could you please post your object schema here or somewhere?
what do you mean by object schema? an example json of such pod or something broader? and btw what meaning a pod has without containers?
cc
I'm fine with banning this for now. The use cases where it would be useful (e.g., reserving resources, staging deployment, prefetching images, ...) aren't currently well supported. When we add the features that could take advantage of this (e.g., pod-level resources, additional of containers during update, container volumes, pod/volume init hooks), then we can make it legal again at that point.
in that case, a clear error message coming from REST would be in place when a pod with no containers is recieved by the server :)
I'll look into this. AFAIU we want to return meaningful error if user specifies pod with no containers. Is my understanding correct?
Correct. We should do it in validateContainers:
cc
should we document this somewhere (e.g. ). Validation is not versioned, so any change would apply to all existing APIs.
Yes, it should be documented, in the field descriptions, such as:
I'll send a PR for this.
cc

// PodSpec is a description of a pod type PodSpec struct { <del> Volumes       []Volume      `json:"volumes" description:"list of volumes that can be mounted by containers belonging to the pod"` Containers    []Container   `json:"containers" description:"list of containers belonging to the pod; cannot be updated; containers cannot currently be added or removed"` </del> <ins> Volumes []Volume `json:"volumes" description:"list of volumes that can be mounted by containers belonging to the pod"` // Required: there must be at least one container in a pod. Containers    []Container   `json:"containers" description:"list of containers belonging to the pod; cannot be updated; containers cannot currently be added or removed; there must be at least one container in a Pod"` </ins> RestartPolicy RestartPolicy `json:"restartPolicy,omitempty" description:"restart policy for all containers within the pod; one of RestartPolicyAlways, RestartPolicyOnFailure, RestartPolicyNever"` // Optional: Set DNS policy.  Defaults to "ClusterFirst" DNSPolicy DNSPolicy `json:"dnsPolicy,omitempty" description:"DNS policy for containers within the pod; one of 'ClusterFirst' or 'Default'"`


Question:
We use capitals for Pod fields (ex: Status.Host) and camelCase for Event fields (ex: involvedObject.resourceVersion). We should be consistent. Should it be capitals? camelCase? Or should we be case insensitive?
IMO, we should use camelCase for pod fields, to match the way it's serialized in JSON.
I agree with The goal here is to be consistent with the API, not with the internal Go representation.
I'd like to fix this before v1beta3 is cast in stone.
ok. Event field selectors are correct then. I will update the selectors for pods to make them camelCase.
I am wondering if I should change this only for v1beta3 or for 1 and 2 as well? This will be a breaking change for beta1 and 2 if we decide to do that.
v1beta3 only, though that's hard, we should reconsider. We're contemplating another "breakage day", and I doubt this is widely used.

case "name": return "name", value, nil case "DesiredState.Host": <del> return "Status.Host", value, nil </del> <ins> return "status.host", value, nil </ins> case "DesiredState.Status": podStatus := PodStatus(value) var internalValue newer.PodPhase newer.Scheme.Convert(&podStatus, &internalValue) <del> return "Status.Phase", string(internalValue), nil </del> <ins> return "status.phase", string(internalValue), nil </ins> default: return "", "", fmt.Errorf("field label not supported: %s", label) }


Question:
We use capitals for Pod fields (ex: Status.Host) and camelCase for Event fields (ex: involvedObject.resourceVersion). We should be consistent. Should it be capitals? camelCase? Or should we be case insensitive?
IMO, we should use camelCase for pod fields, to match the way it's serialized in JSON.
I agree with The goal here is to be consistent with the API, not with the internal Go representation.
I'd like to fix this before v1beta3 is cast in stone.
ok. Event field selectors are correct then. I will update the selectors for pods to make them camelCase.
I am wondering if I should change this only for v1beta3 or for 1 and 2 as well? This will be a breaking change for beta1 and 2 if we decide to do that.
v1beta3 only, though that's hard, we should reconsider. We're contemplating another "breakage day", and I doubt this is widely used.

switch label { case "name": fallthrough <del> case "Status.Phase": </del> <ins> case "status.phase": </ins> fallthrough <del> case "Status.Host": </del> <ins> case "status.host": </ins> return label, value, nil default: return "", "", fmt.Errorf("field label not supported: %s", label)


Question:
We use capitals for Pod fields (ex: Status.Host) and camelCase for Event fields (ex: involvedObject.resourceVersion). We should be consistent. Should it be capitals? camelCase? Or should we be case insensitive?
IMO, we should use camelCase for pod fields, to match the way it's serialized in JSON.
I agree with The goal here is to be consistent with the API, not with the internal Go representation.
I'd like to fix this before v1beta3 is cast in stone.
ok. Event field selectors are correct then. I will update the selectors for pods to make them camelCase.
I am wondering if I should change this only for v1beta3 or for 1 and 2 as well? This will be a breaking change for beta1 and 2 if we decide to do that.
v1beta3 only, though that's hard, we should reconsider. We're contemplating another "breakage day", and I doubt this is widely used.

label:       "label=qux", expectedIDs: util.NewStringSet("qux"), }, { <del> field:       "Status.Phase=Failed", </del> <ins> field:       "status.phase=Failed", </ins> expectedIDs: util.NewStringSet("baz"), }, { <del> field:       "Status.Host=barhost", </del> <ins> field:       "status.host=barhost", </ins> expectedIDs: util.NewStringSet("bar"), }, { <del> field:       "Status.Host=", </del> <ins> field:       "status.host=", </ins> expectedIDs: util.NewStringSet("foo", "baz", "qux", "zot"), }, { <del> field:       "Status.Host!=", </del> <ins> field:       "status.host!=", </ins> expectedIDs: util.NewStringSet("bar"), }, }


Question:
We use capitals for Pod fields (ex: Status.Host) and camelCase for Event fields (ex: involvedObject.resourceVersion). We should be consistent. Should it be capitals? camelCase? Or should we be case insensitive?
IMO, we should use camelCase for pod fields, to match the way it's serialized in JSON.
I agree with The goal here is to be consistent with the API, not with the internal Go representation.
I'd like to fix this before v1beta3 is cast in stone.
ok. Event field selectors are correct then. I will update the selectors for pods to make them camelCase.
I am wondering if I should change this only for v1beta3 or for 1 and 2 as well? This will be a breaking change for beta1 and 2 if we decide to do that.
v1beta3 only, though that's hard, we should reconsider. We're contemplating another "breakage day", and I doubt this is widely used.

func PodToSelectableFields(pod *api.Pod) labels.Set { return labels.Set{ "name":         pod.Name, <del> "Status.Phase": string(pod.Status.Phase), "Status.Host":  pod.Status.Host, </del> <ins> "status.phase": string(pod.Status.Phase), "status.host":  pod.Status.Host, </ins> } }


Question:
We use capitals for Pod fields (ex: Status.Host) and camelCase for Event fields (ex: involvedObject.resourceVersion). We should be consistent. Should it be capitals? camelCase? Or should we be case insensitive?
IMO, we should use camelCase for pod fields, to match the way it's serialized in JSON.
I agree with The goal here is to be consistent with the API, not with the internal Go representation.
I'd like to fix this before v1beta3 is cast in stone.
ok. Event field selectors are correct then. I will update the selectors for pods to make them camelCase.
I am wondering if I should change this only for v1beta3 or for 1 and 2 as well? This will be a breaking change for beta1 and 2 if we decide to do that.
v1beta3 only, though that's hard, we should reconsider. We're contemplating another "breakage day", and I doubt this is widely used.

case "v1beta1", "v1beta2": return "DesiredState.Host" default: <del> return "Status.Host" </del> <ins> return "status.host" </ins> } }


Question:
As Satnam mentioned in PR , our Fluentd pod tends to not collect the first chunk of logs from each container. I've noticed that it seems to miss different amounts of logs depending on how quickly the container generates them, so it may be a timing issue. cc
See

id: fluentd-to-gcp containers: - name: fluentd-gcp-container <del> image: kubernetes/fluentd-gcp:1.0 </del> <ins> image: kubernetes/fluentd-gcp:1.1 </ins> volumeMounts: - name: containers mountPath: /var/lib/docker/containers


Question:
As Satnam mentioned in PR , our Fluentd pod tends to not collect the first chunk of logs from each container. I've noticed that it seems to miss different amounts of logs depending on how quickly the container generates them, so it may be a timing issue. cc
See

.PHONY:	build push <del> TAG = 1.0 </del> <ins> TAG = 1.1 </ins> build: sudo docker build -t kubernetes/fluentd-gcp:$(TAG) .


Question:
As Satnam mentioned in PR , our Fluentd pod tends to not collect the first chunk of logs from each container. I've noticed that it seems to miss different amounts of logs depending on how quickly the container generates them, so it may be a timing issue. cc
See

path /var/lib/docker/containers/*/*-json.log pos_file /var/lib/docker/containers/containers.log.pos time_format %Y-%m-%dT%H:%M:%S <del> tag docker.container.* </del> <ins> tag docker.* read_from_head true </ins> </source> <del> <match docker.container.**> </del> <ins> <match docker.**> </ins> type google_cloud flush_interval 5s # Never wait longer than 5 minutes between retries.


Question:
Follow up from
Bumping this up to P1 given that 0.12.2 wasn't able to create a service with an external load balancer and we didn't notice before cutting the release.
No, to be clearer, 0.12.2 wasn't able to create a service with ELB on GKE. That was on our side entirely.
But it doesn't invalidate the need for this.
The example is using ELB.
are you actively working on this? If not, I'd be up for doing it today.
Maybe not actively but I know what to, however I won't be able to finish it today, so if you think you can do it today please move forward.
We run the e2e tests on GKE, so if we had a test for creating an ELB we would have known that the 0.12.2 release wouldn't work on GKE. I don't see how your point applies. The e2e tests are in place so that we don't have to test each feature manually to verify that it is working, and we only found out that this feature was broken on GKE after we pushed 0.12.2, hence the need for an e2e test.
I think and you are in fact in agreement
right now, k8petstore tests the external load balancer by curling down the amount of transactions ingested from the REST API. im playing with porting it as a e2e test now. The code in the shell script is a little raw, but you can see that it specifies a PUBLIC_IP and then curls down from it at the end ().
that sounds great. What I'd really like to see here though (and what is going to do early next week) is a very simple test that isolates creating a service with an ELB to verify that creating a load balancer works in isolation from a complex test of other parts of the system. This will give us a really clear signal when the test starts to fail instead of needing to debug lots of moving parts when we see a test failure.

import ( "fmt" <ins> "io/ioutil" "net/http" </ins> "sort" <ins> "strings" </ins> "time" "github.com/GoogleCloudPlatform/kubernetes/pkg/api"


Question:
Follow up from
Bumping this up to P1 given that 0.12.2 wasn't able to create a service with an external load balancer and we didn't notice before cutting the release.
No, to be clearer, 0.12.2 wasn't able to create a service with ELB on GKE. That was on our side entirely.
But it doesn't invalidate the need for this.
The example is using ELB.
are you actively working on this? If not, I'd be up for doing it today.
Maybe not actively but I know what to, however I won't be able to finish it today, so if you think you can do it today please move forward.
We run the e2e tests on GKE, so if we had a test for creating an ELB we would have known that the 0.12.2 release wouldn't work on GKE. I don't see how your point applies. The e2e tests are in place so that we don't have to test each feature manually to verify that it is working, and we only found out that this feature was broken on GKE after we pushed 0.12.2, hence the need for an e2e test.
I think and you are in fact in agreement
right now, k8petstore tests the external load balancer by curling down the amount of transactions ingested from the REST API. im playing with porting it as a e2e test now. The code in the shell script is a little raw, but you can see that it specifies a PUBLIC_IP and then curls down from it at the end ().
that sounds great. What I'd really like to see here though (and what is going to do early next week) is a very simple test that isolates creating a service with an ELB to verify that creating a load balancer works in isolation from a complex test of other parts of the system. This will give us a really clear signal when the test starts to fail instead of needing to debug lots of moving parts when we see a test failure.

}() }, 240.0) <ins> It("should be able to create a functioning external load balancer", func() { serviceName := "external-lb-test" ns := api.NamespaceDefault labels := map[string]string{ "key0": "value0", } service := &api.Service{ ObjectMeta: api.ObjectMeta{ Name: serviceName, }, Spec: api.ServiceSpec{ Port:                       80, Selector:                   labels, TargetPort:                 util.NewIntOrStringFromInt(80), CreateExternalLoadBalancer: true, }, } By("cleaning up previous service " + serviceName + " from namespace " + ns) c.Services(ns).Delete(serviceName) By("creating service " + serviceName + " with external load balancer in namespace " + ns) result, err := c.Services(ns).Create(service) Expect(err).NotTo(HaveOccurred()) defer func(ns, serviceName string) { // clean up when we're done By("deleting service " + serviceName + " in namespace " + ns) err := c.Services(ns).Delete(serviceName) Expect(err).NotTo(HaveOccurred()) }(ns, serviceName) if len(result.Spec.PublicIPs) != 1 { Failf("got unexpected number (%d) of public IPs for externally load balanced service: %v", result.Spec.PublicIPs, result) } ip := result.Spec.PublicIPs[0] port := result.Spec.Port pod := &api.Pod{ TypeMeta: api.TypeMeta{ Kind:       "Pod", APIVersion: "v1beta1", }, ObjectMeta: api.ObjectMeta{ Name:   "elb-test-" + string(util.NewUUID()), Labels: labels, }, Spec: api.PodSpec{ Containers: []api.Container{ { Name:  "webserver", Image: "kubernetes/test-webserver", }, }, }, } By("creating pod to be part of service " + serviceName) podClient := c.Pods(api.NamespaceDefault) defer func() { By("deleting pod " + pod.Name) defer GinkgoRecover() podClient.Delete(pod.Name) }() if _, err := podClient.Create(pod); err != nil { Failf("Failed to create pod %s: %v", pod.Name, err) } expectNoError(waitForPodRunning(c, pod.Name)) By("hitting the pod through the service's external load balancer") var resp *http.Response for t := time.Now(); time.Since(t) < 4*time.Minute; time.Sleep(5 * time.Second) { resp, err = http.Get(fmt.Sprintf("http://%s:%d", ip, port)) if err == nil { break } } Expect(err).NotTo(HaveOccurred()) defer resp.Body.Close() body, err := ioutil.ReadAll(resp.Body) Expect(err).NotTo(HaveOccurred()) if resp.StatusCode != 200 { Failf("received non-success return status %q trying to access pod through load balancer; got body: %s", resp.Status, string(body)) } if !strings.Contains(string(body), "test-webserver") { Failf("received response body without expected substring 'test-webserver': %s", string(body)) } }) </ins> It("should correctly serve identically named services in different namespaces on different external IP addresses", func() { serviceNames := []string{"services-namespace-test0"} // Could add more here, but then it takes longer. namespaces := []string{"namespace0", "namespace1"}   // As above.


Question:
We already "healthcheck" kubelet and docker periodically, but miss one for kube-proxy.
It would make sense to do (only bind to localhost) at the same time we add the healthchecking configuration.

- user: root - group: root - mode: 644 <ins> /etc/monit/conf.d/kube-proxy: file: - managed - source: salt://monit/kube-proxy - user: root - group: root - mode: 644 </ins> {% endif %} monit-service: service: - running <del> - name: monit </del> <ins> - name: monit </ins> - watch: <del> - pkg: monit </del> <ins> - pkg: monit </ins> - file: /etc/monit/conf.d/* {% endif %}


Question:
We already "healthcheck" kubelet and docker periodically, but miss one for kube-proxy.
It would make sense to do (only bind to localhost) at the same time we add the healthchecking configuration.

<ins> check process kube-proxy with pidfile /var/run/kube-proxy.pid group kube-proxy start program = "/etc/init.d/kube-proxy start" stop program = "/etc/init.d/kube-proxy stop" if does not exist then restart if failed port 10249 protocol HTTP request "/healthz" with timeout 10 seconds then restart </ins>


Question:
stack trace here: it's due to a shared variable:
Hi thanks for reporting the issue with a patch! Do you want to submit a PR so that you can take the credit for your fix? :)
done , Yu-Ju Hong wrote: 585.241.9488 (voice) 650.649.6071 (fax)
FYI, I think the same bug may still exist in your new PR.
There are multiple p
There are multiple PRs in flight, so I don't know which one you have in mind. One of them removes probing logic from controller altogether.
I was referring to , but I didn't read that PR closely so the bug may longer exist there. Again, just a FYI. :)

// Start syncing node list from cloudprovider. if syncNodeList && nc.isRunningCloudProvider() { go util.Forever(func() { <del> if err = nc.SyncCloudNodes(); err != nil { </del> <ins> if err := nc.SyncCloudNodes(); err != nil { </ins> glog.Errorf("Error syncing cloud: %v", err) } }, period)


Question:
stack trace here: it's due to a shared variable:
Hi thanks for reporting the issue with a patch! Do you want to submit a PR so that you can take the credit for your fix? :)
done , Yu-Ju Hong wrote: 585.241.9488 (voice) 650.649.6071 (fax)
FYI, I think the same bug may still exist in your new PR.
There are multiple p
There are multiple PRs in flight, so I don't know which one you have in mind. One of them removes probing logic from controller altogether.
I was referring to , but I didn't read that PR closely so the bug may longer exist there. Again, just a FYI. :)

// Start syncing or monitoring node status. if syncNodeStatus { go util.Forever(func() { <del> if err = nc.SyncProbedNodeStatus(); err != nil { </del> <ins> if err := nc.SyncProbedNodeStatus(); err != nil { </ins> glog.Errorf("Error syncing status: %v", err) } }, period) } else { go util.Forever(func() { <del> if err = nc.MonitorNodeStatus(); err != nil { </del> <ins> if err := nc.MonitorNodeStatus(); err != nil { </ins> glog.Errorf("Error monitoring node status: %v", err) } }, nodeMonitorPeriod)


Question:
Appears to deadlock or otherwise hang after a few seconds and then hit the 5-minute timeout. From start of log: followed by stack traces of zillions of goroutines that mostly report being blocked for 4+ minutes. Recent instances: https://travis- https://travis- cc
I tried to reproduce the issue by run kubelet unittests 200 times (100 for v1beta1, and 100 for v1beta3), but failed to reproduce the issue. I poked around, and found the only difference between my local machine and test machine is that I am running go 1.4.2 and test machine is running 1.3 (both failure cases posted by are with 1.3.0). I remembered from online I saw a deadlock issue reported against old golang version, but haven't found that issue yet. I will continue trying to reproduce the issue.
Saw this in travis runs too: https://travis-
Yes, there have been multiple failures today. Check shippable, also.
FWIW, these are not all kubelet tests failing, but all of them have many kubelet go routines in the trace - mostly stuck in managePodLoop().
I reran a couple of thousands time last night before I leave, and all passed. Yu-Ju just pointed out that the failure posted above by is go1.4. Shippable is in a really bad shape today, tons of different tests are failed, which should be handled separately.
Ok, with some hacky debugging message into kubelet package, even I couldn't reproduce the very failure reported here, I made some progress on understanding the issue here. Basically there are two issues: 1) The flaky tests reported here are TestServeExecInContainer. All reports posted here have the same stack trace below: 2) There are tons of goroutines, especially one created for each pod. Kubelet leak those goroutines. One can see when test initial run, there are 106 goroutines. Test itself create another 41 goroutines, but none of them are created for podWorkder.
FWIW, the tests in were creating go routines it didn't need at all. I just removed them in
Thanks for cleanup. Now the number down to 91:
cc/
Did something change related to this recently? I haven't heard any flakiness reports for a long time, and now they're starting up again. Any ideas? Sent from my iPhone
I didn't see any changes related to this feature. Actually I even couldn't reproduce it on my desktop. Please note that the comment I made above all those leftover goroutines are separate issue thought.
This has exposed a deadlock in spdystream. I'll work with to come up with a fix.
Upstream issue
Cool ncdc@ ! dchen1107 by the way, I've run across a few similar bugs recently where their reproducibility hinges mainly on how the test cluster was brought up. If the test cluster comes up right, then the tests pass 100% forever. If the cluster comes up bad, the tests fail, sometimes forever. Were you rerunning your tests against the same test cluster? Or rebuilding your test cluster between every test invocation?
This is unittests, not e2e. We saw test flakiness through travis and shippable, but never on my desktop.
This will hopefully fix the deadlock: I will do a bump commit once it's merged.
Thanks for your quick fix. I will clean up goroutines after that.
it's entirely possible there's some bugs in my exec & port forwarding code around leaking goroutines. Are you going to take an initial look to see what's sticking around?
Yes, your test leak some goroutine, but other tests too.

}, { "ImportPath": "github.com/docker/spdystream", <del> "Rev": "e731c8f9f19ffd7e51a469a2de1580c1dfbb4fae" </del> <ins> "Rev": "99515db39d3dad9607e0293f18152f3d59da76dc" </ins> }, { "ImportPath": "github.com/elazarl/go-bindata-assetfs",


Question:
Appears to deadlock or otherwise hang after a few seconds and then hit the 5-minute timeout. From start of log: followed by stack traces of zillions of goroutines that mostly report being blocked for 4+ minutes. Recent instances: https://travis- https://travis- cc
I tried to reproduce the issue by run kubelet unittests 200 times (100 for v1beta1, and 100 for v1beta3), but failed to reproduce the issue. I poked around, and found the only difference between my local machine and test machine is that I am running go 1.4.2 and test machine is running 1.3 (both failure cases posted by are with 1.3.0). I remembered from online I saw a deadlock issue reported against old golang version, but haven't found that issue yet. I will continue trying to reproduce the issue.
Saw this in travis runs too: https://travis-
Yes, there have been multiple failures today. Check shippable, also.
FWIW, these are not all kubelet tests failing, but all of them have many kubelet go routines in the trace - mostly stuck in managePodLoop().
I reran a couple of thousands time last night before I leave, and all passed. Yu-Ju just pointed out that the failure posted above by is go1.4. Shippable is in a really bad shape today, tons of different tests are failed, which should be handled separately.
Ok, with some hacky debugging message into kubelet package, even I couldn't reproduce the very failure reported here, I made some progress on understanding the issue here. Basically there are two issues: 1) The flaky tests reported here are TestServeExecInContainer. All reports posted here have the same stack trace below: 2) There are tons of goroutines, especially one created for each pod. Kubelet leak those goroutines. One can see when test initial run, there are 106 goroutines. Test itself create another 41 goroutines, but none of them are created for podWorkder.
FWIW, the tests in were creating go routines it didn't need at all. I just removed them in
Thanks for cleanup. Now the number down to 91:
cc/
Did something change related to this recently? I haven't heard any flakiness reports for a long time, and now they're starting up again. Any ideas? Sent from my iPhone
I didn't see any changes related to this feature. Actually I even couldn't reproduce it on my desktop. Please note that the comment I made above all those leftover goroutines are separate issue thought.
This has exposed a deadlock in spdystream. I'll work with to come up with a fix.
Upstream issue
Cool ncdc@ ! dchen1107 by the way, I've run across a few similar bugs recently where their reproducibility hinges mainly on how the test cluster was brought up. If the test cluster comes up right, then the tests pass 100% forever. If the cluster comes up bad, the tests fail, sometimes forever. Were you rerunning your tests against the same test cluster? Or rebuilding your test cluster between every test invocation?
This is unittests, not e2e. We saw test flakiness through travis and shippable, but never on my desktop.
This will hopefully fix the deadlock: I will do a bump commit once it's merged.
Thanks for your quick fix. I will clean up goroutines after that.
it's entirely possible there's some bugs in my exec & port forwarding code around leaking goroutines. Are you going to take an initial look to see what's sticking around?
Yes, your test leak some goroutine, but other tests too.

if timer != nil { timer.Stop() } <ins> // Start a goroutine to drain resetChan. This is needed because we've seen // some unit tests with large numbers of goroutines get into a situation // where resetChan fills up, at least 1 call to Write() is still trying to // send to resetChan, the connection gets closed, and this case statement // attempts to grab the write lock that Write() already has, causing a // deadlock. // // See https://github.com/docker/spdystream/issues/49 for more details. go func() { for _ = range resetChan { } }() </ins> i.writeLock.Lock() close(resetChan) i.resetChan = nil i.writeLock.Unlock() break Loop } }


Question:
goroutine 11 [running]: net/http.(ServeMux).Handle(0xc20800a8a0, 0x9e9490, 0xd, 0x7f7d12f190a0, 0xb0e7c0) +0x244 net/http.(ServeMux).HandleFunc(0xc20800a8a0, 0x9e9490, 0xd, 0xb0e7c0) +0x6d net/http.HandleFunc(0x9e9490, 0xd, 0xb0e7c0) +0x48 ·001() +0x58 created by (*SchedulerServer).Run +0x1c7
cc

client.BindClientConfigFlags(fs, &s.ClientConfig) fs.StringVar(&s.AlgorithmProvider, "algorithm_provider", s.AlgorithmProvider, "The scheduling algorithm provider to use") fs.StringVar(&s.PolicyConfigFile, "policy_config_file", s.PolicyConfigFile, "File with scheduler policy configuration") <del> fs.BoolVar(&s.EnableProfiling, "profiling", false, "Enable profiling via web interface host:port/debug/pprof/") </del> <ins> fs.BoolVar(&s.EnableProfiling, "profiling", true, "Enable profiling via web interface host:port/debug/pprof/") </ins> } // Run runs the specified SchedulerServer.  This should never exit.


Question:
goroutine 11 [running]: net/http.(ServeMux).Handle(0xc20800a8a0, 0x9e9490, 0xd, 0x7f7d12f190a0, 0xb0e7c0) +0x244 net/http.(ServeMux).HandleFunc(0xc20800a8a0, 0x9e9490, 0xd, 0xb0e7c0) +0x6d net/http.HandleFunc(0x9e9490, 0xd, 0xb0e7c0) +0x48 ·001() +0x58 created by (*SchedulerServer).Run +0x1c7
cc

} go func() { <ins> mux := http.NewServeMux() </ins> if s.EnableProfiling { <del> http.HandleFunc("/debug/pprof/", pprof.Index) http.HandleFunc("/debug/pprof/profile", pprof.Profile) http.HandleFunc("/debug/pprof/symbol", pprof.Symbol) </del> <ins> mux.HandleFunc("/debug/pprof/", pprof.Index) mux.HandleFunc("/debug/pprof/profile", pprof.Profile) mux.HandleFunc("/debug/pprof/symbol", pprof.Symbol) </ins> } <del> http.Handle("/metrics", prometheus.Handler()) http.ListenAndServe(net.JoinHostPort(s.Address.String(), strconv.Itoa(s.Port)), nil) </del> <ins> mux.Handle("/metrics", prometheus.Handler()) server := &http.Server{ Addr:    net.JoinHostPort(s.Address.String(), strconv.Itoa(s.Port)), Handler: mux, } glog.Fatal(server.ListenAndServe()) </ins> }() configFactory := factory.NewConfigFactory(kubeClient)


Question:
There is a funny subtle behvaiour in get namespaces, in which if i call more than one time, it seems that the removal from Terminating state is accelarated. It seems this might indicate a bug, or maybe its not a bug and the eventual semantics of is a little funny sometimes. in any case, here's what we are seeing : 1) when i delete a namespace the first time usning kubectl, we get a state change to . that is correct, of course. 2) but ... if make the call a second time, then the same status line is immediately removed from . Expected behaviour : A second call to "kubectl delete ..." shouldn't have any effect on the timing of removal of a namespace from 'Terminating' state*... So i guess there is some "eventual consistency" subtle bug that is resolved by repeated calls to
suggests this may not be e bug, maybe its just the etcd eventual deleter thingy.
Looks to definitely be a bug. The 2nd delete seems to have interrupted the delete process and artifacts (like events) from the namespace still exist in etcd.
investigating...
oops, see the error, will submit a fix.
I have a fix, but need a test case, will submit PR tomorrow AM

allErrs := errs.ValidationErrorList{} allErrs = append(allErrs, ValidateObjectMetaUpdate(&oldNamespace.ObjectMeta, &newNamespace.ObjectMeta).Prefix("metadata")...) newNamespace.Spec = oldNamespace.Spec <ins> if newNamespace.DeletionTimestamp.IsZero() { if newNamespace.Status.Phase != api.NamespaceActive { allErrs = append(allErrs, errs.NewFieldInvalid("Status.Phase", newNamespace.Status.Phase, "A namespace may only be in active status if it does not have a deletion timestamp.")) } } else { if newNamespace.Status.Phase != api.NamespaceTerminating { allErrs = append(allErrs, errs.NewFieldInvalid("Status.Phase", newNamespace.Status.Phase, "A namespace may only be in terminating status if it has a deletion timestamp.")) } } </ins> return allErrs }


Question:
There is a funny subtle behvaiour in get namespaces, in which if i call more than one time, it seems that the removal from Terminating state is accelarated. It seems this might indicate a bug, or maybe its not a bug and the eventual semantics of is a little funny sometimes. in any case, here's what we are seeing : 1) when i delete a namespace the first time usning kubectl, we get a state change to . that is correct, of course. 2) but ... if make the call a second time, then the same status line is immediately removed from . Expected behaviour : A second call to "kubectl delete ..." shouldn't have any effect on the timing of removal of a namespace from 'Terminating' state*... So i guess there is some "eventual consistency" subtle bug that is resolved by repeated calls to
suggests this may not be e bug, maybe its just the etcd eventual deleter thingy.
Looks to definitely be a bug. The 2nd delete seems to have interrupted the delete process and artifacts (like events) from the namespace still exist in etcd.
investigating...
oops, see the error, will submit a fix.
I have a fix, but need a test case, will submit PR tomorrow AM

} func TestValidateNamespaceStatusUpdate(t *testing.T) { <ins> now := util.Now() </ins> tests := []struct { oldNamespace api.Namespace namespace    api.Namespace valid        bool }{ <del> {api.Namespace{}, api.Namespace{}, true}, </del> <ins> {api.Namespace{}, api.Namespace{ Status: api.NamespaceStatus{ Phase: api.NamespaceActive, }, }, true}, </ins> {api.Namespace{ ObjectMeta: api.ObjectMeta{ Name: "foo"}}, api.Namespace{ ObjectMeta: api.ObjectMeta{ <del> Name: "foo"}, </del> <ins> Name:              "foo", DeletionTimestamp: &now}, </ins> Status: api.NamespaceStatus{ Phase: api.NamespaceTerminating, },


Question:
There is a funny subtle behvaiour in get namespaces, in which if i call more than one time, it seems that the removal from Terminating state is accelarated. It seems this might indicate a bug, or maybe its not a bug and the eventual semantics of is a little funny sometimes. in any case, here's what we are seeing : 1) when i delete a namespace the first time usning kubectl, we get a state change to . that is correct, of course. 2) but ... if make the call a second time, then the same status line is immediately removed from . Expected behaviour : A second call to "kubectl delete ..." shouldn't have any effect on the timing of removal of a namespace from 'Terminating' state*... So i guess there is some "eventual consistency" subtle bug that is resolved by repeated calls to
suggests this may not be e bug, maybe its just the etcd eventual deleter thingy.
Looks to definitely be a bug. The 2nd delete seems to have interrupted the delete process and artifacts (like events) from the namespace still exist in etcd.
investigating...
oops, see the error, will submit a fix.
I have a fix, but need a test case, will submit PR tomorrow AM

Name: "foo"}}, api.Namespace{ ObjectMeta: api.ObjectMeta{ <ins> Name: "foo"}, Status: api.NamespaceStatus{ Phase: api.NamespaceTerminating, }, }, false}, {api.Namespace{ ObjectMeta: api.ObjectMeta{ Name: "foo"}}, api.Namespace{ ObjectMeta: api.ObjectMeta{ </ins> Name: "bar"}, Status: api.NamespaceStatus{ Phase: api.NamespaceTerminating,


Question:
There is a funny subtle behvaiour in get namespaces, in which if i call more than one time, it seems that the removal from Terminating state is accelarated. It seems this might indicate a bug, or maybe its not a bug and the eventual semantics of is a little funny sometimes. in any case, here's what we are seeing : 1) when i delete a namespace the first time usning kubectl, we get a state change to . that is correct, of course. 2) but ... if make the call a second time, then the same status line is immediately removed from . Expected behaviour : A second call to "kubectl delete ..." shouldn't have any effect on the timing of removal of a namespace from 'Terminating' state*... So i guess there is some "eventual consistency" subtle bug that is resolved by repeated calls to
suggests this may not be e bug, maybe its just the etcd eventual deleter thingy.
Looks to definitely be a bug. The 2nd delete seems to have interrupted the delete process and artifacts (like events) from the namespace still exist in etcd.
investigating...
oops, see the error, will submit a fix.
I have a fix, but need a test case, will submit PR tomorrow AM

for i, test := range tests { test.namespace.ObjectMeta.ResourceVersion = "1" test.oldNamespace.ObjectMeta.ResourceVersion = "1" <del> errs := ValidateNamespaceStatusUpdate(&test.oldNamespace, &test.namespace) </del> <ins> errs := ValidateNamespaceStatusUpdate(&test.namespace, &test.oldNamespace) </ins> if test.valid && len(errs) > 0 { t.Errorf("%d: Unexpected error: %v", i, errs) t.Logf("%#v vs %#v", test.oldNamespace.ObjectMeta, test.namespace.ObjectMeta)


Question:
There is a funny subtle behvaiour in get namespaces, in which if i call more than one time, it seems that the removal from Terminating state is accelarated. It seems this might indicate a bug, or maybe its not a bug and the eventual semantics of is a little funny sometimes. in any case, here's what we are seeing : 1) when i delete a namespace the first time usning kubectl, we get a state change to . that is correct, of course. 2) but ... if make the call a second time, then the same status line is immediately removed from . Expected behaviour : A second call to "kubectl delete ..." shouldn't have any effect on the timing of removal of a namespace from 'Terminating' state*... So i guess there is some "eventual consistency" subtle bug that is resolved by repeated calls to
suggests this may not be e bug, maybe its just the etcd eventual deleter thingy.
Looks to definitely be a bug. The 2nd delete seems to have interrupted the delete process and artifacts (like events) from the namespace still exist in etcd.
investigating...
oops, see the error, will submit a fix.
I have a fix, but need a test case, will submit PR tomorrow AM

namespace := nsObj.(*api.Namespace) // upon first request to delete, we switch the phase to start namespace termination <del> if namespace.DeletionTimestamp == nil { </del> <ins> if namespace.DeletionTimestamp.IsZero() { </ins> now := util.Now() namespace.DeletionTimestamp = &now namespace.Status.Phase = api.NamespaceTerminating


Question:
There is a funny subtle behvaiour in get namespaces, in which if i call more than one time, it seems that the removal from Terminating state is accelarated. It seems this might indicate a bug, or maybe its not a bug and the eventual semantics of is a little funny sometimes. in any case, here's what we are seeing : 1) when i delete a namespace the first time usning kubectl, we get a state change to . that is correct, of course. 2) but ... if make the call a second time, then the same status line is immediately removed from . Expected behaviour : A second call to "kubectl delete ..." shouldn't have any effect on the timing of removal of a namespace from 'Terminating' state*... So i guess there is some "eventual consistency" subtle bug that is resolved by repeated calls to
suggests this may not be e bug, maybe its just the etcd eventual deleter thingy.
Looks to definitely be a bug. The 2nd delete seems to have interrupted the delete process and artifacts (like events) from the namespace still exist in etcd.
investigating...
oops, see the error, will submit a fix.
I have a fix, but need a test case, will submit PR tomorrow AM

// prior to final deletion, we must ensure that finalizers is empty if len(namespace.Spec.Finalizers) != 0 { <del> err = fmt.Errorf("Unable to delete namespace %v because finalizers is not empty %v", namespace.Name, namespace.Spec.Finalizers) </del> <ins> err = fmt.Errorf("Namespace %v termination is in progress, waiting for %v", namespace.Name, namespace.Spec.Finalizers) return nil, err </ins> } return r.Etcd.Delete(ctx, name, nil) }


Question:
There is a funny subtle behvaiour in get namespaces, in which if i call more than one time, it seems that the removal from Terminating state is accelarated. It seems this might indicate a bug, or maybe its not a bug and the eventual semantics of is a little funny sometimes. in any case, here's what we are seeing : 1) when i delete a namespace the first time usning kubectl, we get a state change to . that is correct, of course. 2) but ... if make the call a second time, then the same status line is immediately removed from . Expected behaviour : A second call to "kubectl delete ..." shouldn't have any effect on the timing of removal of a namespace from 'Terminating' state*... So i guess there is some "eventual consistency" subtle bug that is resolved by repeated calls to
suggests this may not be e bug, maybe its just the etcd eventual deleter thingy.
Looks to definitely be a bug. The 2nd delete seems to have interrupted the delete process and artifacts (like events) from the namespace still exist in etcd.
investigating...
oops, see the error, will submit a fix.
I have a fix, but need a test case, will submit PR tomorrow AM

if err != nil { t.Fatalf("unexpected error: %v", err) } <ins> } func TestDeleteNamespaceWithIncompleteFinalizers(t *testing.T) { now := util.Now() fakeEtcdClient, helper := newHelper(t) fakeEtcdClient.ChangeIndex = 1 fakeEtcdClient.Data["/registry/namespaces/foo"] = tools.EtcdResponseWithError{ R: &etcd.Response{ Node: &etcd.Node{ Value: runtime.EncodeOrDie(latest.Codec, &api.Namespace{ ObjectMeta: api.ObjectMeta{ Name:              "foo", DeletionTimestamp: &now, }, Spec: api.NamespaceSpec{ Finalizers: []api.FinalizerName{api.FinalizerKubernetes}, }, Status: api.NamespaceStatus{Phase: api.NamespaceActive}, }), ModifiedIndex: 1, CreatedIndex:  1, }, }, } storage, _, _ := NewStorage(helper) _, err := storage.Delete(api.NewDefaultContext(), "foo", nil) if err == nil { t.Fatalf("expected error: %v", err) } } </ins> <del> // TODO: when we add life-cycle, this will go to Terminating, and then we need to test Terminating to gone </del> <ins> func TestDeleteNamespaceWithCompleteFinalizers(t *testing.T) { now := util.Now() fakeEtcdClient, helper := newHelper(t) fakeEtcdClient.ChangeIndex = 1 fakeEtcdClient.Data["/registry/namespaces/foo"] = tools.EtcdResponseWithError{ R: &etcd.Response{ Node: &etcd.Node{ Value: runtime.EncodeOrDie(latest.Codec, &api.Namespace{ ObjectMeta: api.ObjectMeta{ Name:              "foo", DeletionTimestamp: &now, }, Spec: api.NamespaceSpec{ Finalizers: []api.FinalizerName{}, }, Status: api.NamespaceStatus{Phase: api.NamespaceActive}, }), ModifiedIndex: 1, CreatedIndex:  1, }, }, } storage, _, _ := NewStorage(helper) _, err := storage.Delete(api.NewDefaultContext(), "foo", nil) if err != nil { t.Fatalf("unexpected error: %v", err) } </ins> }


Question:
There is a funny subtle behvaiour in get namespaces, in which if i call more than one time, it seems that the removal from Terminating state is accelarated. It seems this might indicate a bug, or maybe its not a bug and the eventual semantics of is a little funny sometimes. in any case, here's what we are seeing : 1) when i delete a namespace the first time usning kubectl, we get a state change to . that is correct, of course. 2) but ... if make the call a second time, then the same status line is immediately removed from . Expected behaviour : A second call to "kubectl delete ..." shouldn't have any effect on the timing of removal of a namespace from 'Terminating' state*... So i guess there is some "eventual consistency" subtle bug that is resolved by repeated calls to
suggests this may not be e bug, maybe its just the etcd eventual deleter thingy.
Looks to definitely be a bug. The 2nd delete seems to have interrupted the delete process and artifacts (like events) from the namespace still exist in etcd.
investigating...
oops, see the error, will submit a fix.
I have a fix, but need a test case, will submit PR tomorrow AM

"testing" "github.com/GoogleCloudPlatform/kubernetes/pkg/api" <ins> "github.com/GoogleCloudPlatform/kubernetes/pkg/util" </ins> ) func TestNamespaceStrategy(t *testing.T) {


Question:
There is a funny subtle behvaiour in get namespaces, in which if i call more than one time, it seems that the removal from Terminating state is accelarated. It seems this might indicate a bug, or maybe its not a bug and the eventual semantics of is a little funny sometimes. in any case, here's what we are seeing : 1) when i delete a namespace the first time usning kubectl, we get a state change to . that is correct, of course. 2) but ... if make the call a second time, then the same status line is immediately removed from . Expected behaviour : A second call to "kubectl delete ..." shouldn't have any effect on the timing of removal of a namespace from 'Terminating' state*... So i guess there is some "eventual consistency" subtle bug that is resolved by repeated calls to
suggests this may not be e bug, maybe its just the etcd eventual deleter thingy.
Looks to definitely be a bug. The 2nd delete seems to have interrupted the delete process and artifacts (like events) from the namespace still exist in etcd.
investigating...
oops, see the error, will submit a fix.
I have a fix, but need a test case, will submit PR tomorrow AM

if StatusStrategy.AllowCreateOnUpdate() { t.Errorf("Namespaces should not allow create on update") } <ins> now := util.Now() </ins> oldNamespace := &api.Namespace{ ObjectMeta: api.ObjectMeta{Name: "foo", ResourceVersion: "10"}, Spec:       api.NamespaceSpec{Finalizers: []api.FinalizerName{"kubernetes"}}, Status:     api.NamespaceStatus{Phase: api.NamespaceActive}, } namespace := &api.Namespace{ <del> ObjectMeta: api.ObjectMeta{Name: "foo", ResourceVersion: "9"}, </del> <ins> ObjectMeta: api.ObjectMeta{Name: "foo", ResourceVersion: "9", DeletionTimestamp: &now}, </ins> Status:     api.NamespaceStatus{Phase: api.NamespaceTerminating}, } StatusStrategy.PrepareForUpdate(namespace, oldNamespace)


Question:
GCE has a length limit of 63 characters for its load balancer components (target pools and forwarding rules), and AWS has a length limit of 32 characters. Our current load balancer name construction of clustername-namespace-servicename can exceed both limits, especially AWS's. We should use a shorter name construction method less likely to hit these limits.
Thanks for filing this Alex. My suggestion would be to use a simple UID, as per utils.NewUID(). We'll need to store the mapping from {clustername, namespace, servicename} -UID somewhere stable like etcd. Using anything based directly on {cluster, namespace, name}, for example a stable hash of each, has the problem that the aforementioned tuple is not unique over time. This raises the possibility of e.g. the following scenario: creates mycluster/mynamespace/myservicename, which creates external load balancer named stablehash(mycluster,mynamespace,myservicename) deletes mycluster/mynamespace/myservicename, but at the time MyCloudProvider is down, so kubernetes can't delete ELB named stablehash(mycluster,mynamespace,myservicename). Kubernetes keeps trying in the background to delete the ELB. the mean time user creates a new service named the same as her old one mycluster/mynamespace/myservicename. Kubernetes tries to create an associated ELB named stablehash(mycluster,mynamespace,myservicename), but this fails, because the ELB exists already due to 2 above. ... etc. Badness ensues.
Assigning to Chao Xu, as this makes for a good starter bug.
Hmmmm.... can't seem to assign to caesarxuchao yet. Not sure why.
He has to have write privileges on the repo before issues/PRs can be assigned to him.
But we have many contributors with issues assigned to them that are not in the kubernetes-write group (that only has 48 members). I must be missing something. How do I grant him write privileges on the repo?
We do? That's not how I understand it to work: If you want to add him, go .
has been because the load balancer name changes if the service's UID changes (e.g. if it's deleted and recreated). Think we can change this? A hash of cluster name, namespace, and service name should get around it, but there may be better ideas as well.
Sure. I'm not confident that I can fix this quickly. If it's urgent, maybe you can assign some one more experienced to fix it? One question: I guess the ELB's name should stay unchanged as long as the "zone+namespace+service name" doesn't change, am I right?
I think I'm the guilty party here. I was unaware that when we create a new service with the same name as a previously existing service, that it needed to keep the same load balancer name and IP. This seems broken to me, but I may be missing the larger context. We can quite safely roll back the change while we decide on the best alternative solution. Q , Chao Xu wrote:
I think the real breakage is that we don't have any way to claim an LB-IP at a larger lifespan than a single service. Alex, can you go back to OP and get a sense of what's happening there to trigger this? , Quinton Hoole wrote:
Will do. Even if the name is kept consistent as it was before, with the way everything currently works the IP won't be consistent.
yeah, I'm not positive that the name has to be consistent. What we need is a way to claim the same public IP address, and we already have that (at least for GCE) Assigning to me, and I'll validate the length requirements, make sure public ip assignment works for AWS and close out this issue.
Note there are two separate things here: (1) a purely-name-based construction can exceed length limits given a single service and (2) some form of across-service consistency is apparently needed. (1) is trivially solvable by either always using a hash or alternatively using a two-stage process where you generate a desired name and only hashify if it's longer than the length limits by truncating it and appending a hash of said name.
AWS's load balancer hasn't been merged yet, so there's not much to check out there. I'm going to be reducing the churn of load balancers later today, which should help prevent IP address changes for services that are modified.
I think we need to be able to figure out which Service an LB is attached to from the Service, so that LBs can't get orphaned. regarding being able to claim an LB IP. I don't think that is covered in revamp... , Brendan Burns wrote:
I think that we need to roll back and think this issue through a bit better. See comment on that PR.

package gce_cloud import ( <ins> "crypto/md5" </ins> "fmt" "io" "io/ioutil"


Question:
GCE has a length limit of 63 characters for its load balancer components (target pools and forwarding rules), and AWS has a length limit of 32 characters. Our current load balancer name construction of clustername-namespace-servicename can exceed both limits, especially AWS's. We should use a shorter name construction method less likely to hit these limits.
Thanks for filing this Alex. My suggestion would be to use a simple UID, as per utils.NewUID(). We'll need to store the mapping from {clustername, namespace, servicename} -UID somewhere stable like etcd. Using anything based directly on {cluster, namespace, name}, for example a stable hash of each, has the problem that the aforementioned tuple is not unique over time. This raises the possibility of e.g. the following scenario: creates mycluster/mynamespace/myservicename, which creates external load balancer named stablehash(mycluster,mynamespace,myservicename) deletes mycluster/mynamespace/myservicename, but at the time MyCloudProvider is down, so kubernetes can't delete ELB named stablehash(mycluster,mynamespace,myservicename). Kubernetes keeps trying in the background to delete the ELB. the mean time user creates a new service named the same as her old one mycluster/mynamespace/myservicename. Kubernetes tries to create an associated ELB named stablehash(mycluster,mynamespace,myservicename), but this fails, because the ELB exists already due to 2 above. ... etc. Badness ensues.
Assigning to Chao Xu, as this makes for a good starter bug.
Hmmmm.... can't seem to assign to caesarxuchao yet. Not sure why.
He has to have write privileges on the repo before issues/PRs can be assigned to him.
But we have many contributors with issues assigned to them that are not in the kubernetes-write group (that only has 48 members). I must be missing something. How do I grant him write privileges on the repo?
We do? That's not how I understand it to work: If you want to add him, go .
has been because the load balancer name changes if the service's UID changes (e.g. if it's deleted and recreated). Think we can change this? A hash of cluster name, namespace, and service name should get around it, but there may be better ideas as well.
Sure. I'm not confident that I can fix this quickly. If it's urgent, maybe you can assign some one more experienced to fix it? One question: I guess the ELB's name should stay unchanged as long as the "zone+namespace+service name" doesn't change, am I right?
I think I'm the guilty party here. I was unaware that when we create a new service with the same name as a previously existing service, that it needed to keep the same load balancer name and IP. This seems broken to me, but I may be missing the larger context. We can quite safely roll back the change while we decide on the best alternative solution. Q , Chao Xu wrote:
I think the real breakage is that we don't have any way to claim an LB-IP at a larger lifespan than a single service. Alex, can you go back to OP and get a sense of what's happening there to trigger this? , Quinton Hoole wrote:
Will do. Even if the name is kept consistent as it was before, with the way everything currently works the IP won't be consistent.
yeah, I'm not positive that the name has to be consistent. What we need is a way to claim the same public IP address, and we already have that (at least for GCE) Assigning to me, and I'll validate the length requirements, make sure public ip assignment works for AWS and close out this issue.
Note there are two separate things here: (1) a purely-name-based construction can exceed length limits given a single service and (2) some form of across-service consistency is apparently needed. (1) is trivially solvable by either always using a hash or alternatively using a two-stage process where you generate a desired name and only hashify if it's longer than the length limits by truncating it and appending a hash of said name.
AWS's load balancer hasn't been merged yet, so there's not much to check out there. I'm going to be reducing the churn of load balancers later today, which should help prevent IP address changes for services that are modified.
I think we need to be able to figure out which Service an LB is attached to from the Service, so that LBs can't get orphaned. regarding being able to claim an LB IP. I don't think that is covered in revamp... , Brendan Burns wrote:
I think that we need to roll back and think this issue through a bit better. See comment on that PR.

"google.golang.org/cloud/compute/metadata" ) <ins> const LOAD_BALANCER_NAME_MAX_LENGTH = 63 </ins> // GCECloud is an implementation of Interface, TCPLoadBalancer and Instances for Google Compute Engine. type GCECloud struct { service          *compute.Service