A temporary workaround is to seed our random number generators. But really, apiserver should assign a guaranteed-unique identifier upon resource creation.
Internally we use RFC4122 UUIDs for identifying pods. Any objections to making this part of the pod setup? I guess it would really be a string (like "id") but with the strong suggestion that it be an encoded UUID. Or we could use docker-style 256 bit randoms, but that might get confusing. If we further lock down container names to RFC1035 labels, we can use <uuid.<nameas the docker container name, which seems much nicer than the current dashes and underscores :) What think?
Both points sound great to me. These encoded names are just plain ugly:) Id is also a required field (network containers break otherwise), so it seems weird to have them marked as 'omitempty'. This is probably more an issue for config files than anything else.
Was going to get familiar and try to fix this - sounds like the suggestion is to add "ID string" to Container, and fail PodRegistryStorage.Create() if Container.ID is empty? Or should PodRegistryStorage.Create() populate unset DesiredState.Manifest.Containers[].ID that are empty? Latter seems more flexible (server controls default UUID generation for clients)
I started in on some validation logic this morning. It's not clear whether unique ID is something users should have to spec or not. I lean towards not. That means the master (api server?) Has to generate a uuid upon acceptance. That uuid would have to flow down to Kubelet. Is that in the same vein as you were thinking? On Jun 25, 2014 6:35 PM, "Clayton Coleman" wrote:
As an API consumer I like not having to specify things that the server can do for me - having to generate a UUID on the command line to curl a new pod into existence feels wrong. I started but didn't pass down to kubelet yet.
One thing to note with global unique identifier is static containers. Today we setup cAdvisor as a static container and have no way to assign it a unique ID globally. , Clayton Coleman wrote:
Static as in "defined on each host via a config file"? Would it make sense for the Kubelet to auto assign a UUID for containers pulled from files based on the host MAC and the position in the file (or a SHA1 of the contents of the manifest plus the host MAC)?
Are you setting it up by config file? Can we generate a new uuid when we write the config file, or do we want it to be identical across machines (google prod style)? We cod do something like: if Kubelet finds a config without a uuid, it will assign it a uuid and log it. Internally we go one step further and define "master space" where each configuration originator can choose an ID and then manage ids within that space, meaning the master doesn't have to generate a UUID at all just a unique masterspace + pod name. Do we need to go that far? Uuids are notoriously human-hostile. On Jun 25, 2014 7:38 PM, "Victor Marmol" wrote:
Yes, it is a static file we leave on the machine. I don't think we have the ability to assign it a unique ID (outside of running some custom init script). Having the Kubelet assign it a UUID seems reasonable to me. , Clayton Coleman wrote:
To be clearer on masterspace. Each new pod gets fields such as: Masterspace: Name: "a name unique within this masterspace". I am sultaneously trying to argue that new should be DNS compatible, which lits them to 64 lowercase alphanums, and gives names semantic meaning. Making then bad as unique IDs. Need to think more about this... On Jun 25, 2014 7:46 PM, "Tim Hockin" wrote:
I want the container names from the different sources to have different namespaces, so there won't be any collisions. E.g., kubelet prepends/appends ".etcd" (or something) to etcd-sourced containers, ".cfg" to containers from the config file, ".http" to containers from the manifest url, etc. Then it is up to each source to stay unique. Api server can stay unique via uuid or counting up. Config files & manifest url stay unique by humans not screwing up; kubelet rejects them otherwise. This has the nice effect that the container name produced from a config file is predictable without having to do a lookup. This would be good for our own container vm image and anything like it.
This started structured and turned into a stream0-of-consiousness, sorry. I agree that unique names are desirable, but I'm not sure "etcd" and "httpd" are sufficient. The reason we have masterspace internally is because we also need to attach metadata about the master that created a pod (e.g. which cluster it thought it was in, what magic number it had (generation number) and so on). Here's what I think I have convinced myself of, so far. 1) All pods have a required name string, which is human-friendly and DNS friendly (probably rfc1035 subdomain rather than label). 2) All pods which are running have a uuid (suggest RFC4122 but could be something else). This of this as a cluster-wide PID. It has no semantic meaning and changes whenever a pod is started on a new host. 3) The kubelet API includes this uuid. If the uuid field is not specified, the kubelet will assign a new UUID. This does not have the property that Daniel wants - predictable container names. The problem is that you putting semantic meaning into the unique ID. There's a reason that databse best practices involve surrogate keys and that UNIX syscalls operate on PIDs rather than command names. Consider what happens if we get a phantom instance on a split network - both pods end up with the same name - not unique any more. That said, I could maybe be convinced. If we put the rule that the pod name had to be unique within a masterspace, we could sort of punt the problem a bit, for now. E.g. Pod { masterspace = "" id = "" containers [ { name = "apache" be created with container name If the apiserver did not care about phantoms, it could leave off the id8675309 noise. for Google people, this should look very familiar. I still have a vague foreboding about making the uniqueness be the master's problem, but it would get rid of the need for opaque UUIDs. Or would it? We need to handle static pods (e.g. cAdvisor). If they are allowed to collide, then some hyopthetical cluster-data aggregator can not use ID as primary key, and we will need to disambiguate queries by hostname. Blech. What if the rule is that masterspace is optional. If not specified, kubelet will make something up derived from the source. So cAdvsor would say: Pod { #no masterspace id = "cadvisor" containers [ { name = "cadvisor" be created with container name <host_fqdnThis still sort of sucks in that you can't aggregate by masterspace, e.g. "give me a list of all containers in the cadvisor masterspace". But maybe that's better served by labels anyway. Yes, I think so. Thoughts? I could go wither way (UUIDs or masterspace + unique name). Or does someone have other ideas? I feel like this got complicated pretty fast. Tim , Daniel Smith wrote:
Just to be clear, I only think that the property of predictable names is desirable for containers that come from the manifest url and maybe the container, because without that property, is unhelpful. But you've got me somewhat convinced that maybe I shouldn't care so much about that. I don't know if we want to wait to solve naming in general before we accept a better solution than what we do currently.
I could put my weight behind either approach. The problem with UUIDs (be they RFC4122 or SHA or whatever) is that they suck for humans. The problem with !UUIDs is that we have to count on the "master" to get uniqueness right. , Daniel Smith wrote:
"I feel like this got complicated pretty fast." Yup:) [I think I've read & parsed all of the comments, but I could be mistaken] I like the idea of DNS-styled hierarchical namespaces. I'm not super worried about multiple masters scheduling over the same cluster of minions, but it does make multi-tenant masters easier (i.e. a provider is running a master, and customers run their minions). It's also something that people are used to. I think it's also reasonable to require that users produce unique names for managing in the system. Machine-generated unique IDs are more useful for running containers (i.e. a global PID). Having this be globally unique is really handy for things like log aggregation, etc. Why not always have the kubelet generate that? The master can learn it when it lists the running containers. It's worth noting that the only restriction of docker names is that they are unique to the host. We can still encode useful human data (manifest+container names) in addition to the unique ID into the if we only parse out the unique ID! FWIW, I'm just polishing my ID cleanups to remove dead (or dieing) code. My goal was to normalize on docker IDs for many/most things inside the kubelet. I should have it out for review tomorrow.
Also, clarifying some terminology might be handy. Here's how I think of things (could easily be wrong!): is a collection of . are an instance of a . They can be running (or not). They are assigned to a host. Things that confuse me: have an Id. have a name. have an Id. Seems like the manifest should only have a name (but this gets to the discussion above about names vs ids). has a which really just wraps . Seems like overkill.
, Justin Huff wrote: I'm more worried about "masters" that are config files crafted by humans without coordination. Those should not accidentally collide :) Here "users" == apiserver? Yeah, this is doable (and is in fact closer to what we do internally) I think it is useful, but not critical that docker-reported names be human-friendly. I'm keenly interested in this, and I'd like to stabilize it all ASAP. Please do not merge this change until I have had time to feedback on it. Caveat: I am part-time the rest of this week.
, Justin Huff wrote: The names as defined in pkg/api are somewhat confusing. It all depends on perspective. We really have two APIs here (user -master; master -kubelet), and we're cramming the spec into one file. We should consider whether we really want to do that. From Kubelet's POV ContainerManifest == Pod. Yes, let's settle the discussion about UUIDs vs Names before we rename things. and then lets rename things. I think this was to leave room for growth. Kubelet should do the same and define a Pod type that has an api.ContainerManifest and maybe the UUID (if we do that). BTW: I have a change pending (not sent yet) to do validation of a ContainerManifest - so it will reject manifests that, for example, do not have an ID. The core of it is done, but it needs a test and has some open questions. Hope to have it out tomorrow.
Whew! This is long. This issue started with no context about what we're trying to do. Starting with the end: Yes, we should clean up the identifiers. Currently, every object/resource includes JSONBase, which has an ID (which probably should be Id). ContainerManifest also has an Id. Container has Name. Port has Name. I'll point out that Container and Port are not standalone resources right now. ContainerManifest is the way it is due to compatibility with the container-vm release. What are we trying to do? I saw several things mentioned in this issue: Human-friendly, memorable, semantically meaningful, short-ish references in operations Predictable references in operations and/or configuration files Unambiguous references in space, such as pods/containers created by multiple masters or by static configuration Unambiguous references in time, such as for logging Idempotent creation () Auto-generation of DNS names () Identification of sets for aggregation Both unique identifiers and human-friendly names have value. Human-friendly names can only be unique in space, not in time. We should use "Id" for the former and "Name" for the latter. Names could be used to ensure idempotent (at most once) creation, though a non-indexed resource value could be used for that, also. The replicationController would treat "Name" in the template as a prefix and would append relatively short random numbers for uniqueness. Static pods could be treated similarly. Label selectors should be used for set formation / aggregation. If users are providing Names, we could also use them for DNS (). I'd use "domain" rather than "masterspace". Argh, my laptop needs to be rebooted...
Continuing... I agree we want a mechanism that permits unique id allocation by Kubelets and that doesn't require centralized and/or persistent state. I like UUIDs. There are the issues of ensuring unique MAC addresses in VMs and/or namespaces, and determinism for testing, but I think we've found ways around these issues. I've considered not having human-friendly names for pods before and just using labels instead. Labels are predictable and human-friendly, but don't require uniqueness, and don't require concatenating lots of identifying info together in order to ensure uniqueness, which users WILL do for names (and they'll want to parameterize them). They aren't short, though. Also, I guess part of the problem is that Docker doesn't support labels. We should push on that. Idempotence could be ensured by a client-generated cookie, such as a fingerprint/hash or PR number. It wouldn't be required for static configs. DNS names for instances aren't super-useful if they aren't predictable and DNS-like human-friendly names aren't that friendly if they are long. Short nicknames don't have to be predictable (so they could have a short uniquifying suffix) and don't even need to be semantically relevant -- just memorable (hence Docker's silly auto-generated names, I guess). Services need predictable DNS names, OTOH, and ports need predictable names (for DNS SRV lookup or ENV vars or whatever), and pod-relative hostnames of containers should be predictable, so they can communicate with each other, though I don't know that they actually need to be FQDNs. Other types of services (e.g., master-elected services) and groups will need predictable DNS names, also. Is the main motivation for names for pods to be consistent across all resource types? If so, I could buy into DNS-like hierarchical names for them. I'd use something like "domain" or "namespace" instead of "masterspace".
Pod-relative hostnames and stable internal names are definitely valuable - and limiting "name" to rfc1035 subdomain has been extremely valuable in practice to us on OpenShift. Quasi-uniqueness I assume? How predictable? As a concrete example, with something like Zookeeper you need the container to have an identifier/name that is stable across restarts / reschedules in a shared config (so not a pod ID). I'd assumed you'd model this with a set of replication controllers (vs a shared controller) so you had 3 replication controllers with 1 item each, and you'd be able to either set an ENV per pod, or use the name in order to apply that. If name changes over pod instances that rules out the reuse there.
One last thought from me; it occurs to me that docker already generates a long container id. Perhaps we can consider using that directly as our spatial/temporal unique identifier, and use this hypothetical dns-style solution as the friendly human name. We'd need to investigate just how unique docker's id's are, and we may not want to depend on docker for that, but it would reduce the number of IDs needed. Also, as a footnote, if we end up with both pod.ID and Manifest.ID, IMO they should be the same identifier.
For clarity, let me suggest the convention that "name" means a friendly mostly human readable string, possibly in the style of dns names, and that "id" means an opaque, machine generated identifier, guaranteed unique at some level of resolution. Maybe everyone except for me is already using this convention. :) But I want to talk about ids and names in general without referring to a particular implementation. I think, in that case, the rep. controller itself has a name, which it can use to make names for the pods it creates (prepend/append indices or something). IDs for the pods could still be generated by the apisever or wherever we decide they need to be generated. I was trying to say that since there's a 1:1 relationship between pods and manifests, we shouldn't make different IDs for each.
Trying to collect thoughts and ideas into a proposal. It's tricky considering it from all angles (users, apiserver, replication controllers, kubelet). My background is node-centric, so I am counting on people to smack me if I say something dumb for higher-levels :) I have to run out right now, but I'm hoping we can distill the discussion into a design O(soon). NB: We spec names as RFC 1035 compatible, though we might extend that to allow purely numeric tokens (e.g. ) but I forget which RFC that is. Docker names allow underscores, which kubelet reserves for use in infrastructure containers. From kubelet's point of view: 1) All pods have a namespace string, which is human-friendly and DNS friendly (rfc1035 subdomain fragment). For example: "". This is used to indicate the provenance of the pod. If the namespace is not specified when creating a pod, kubelet will assign a namespace derived from the source. For example, a file "" on host " " might get masterspace " file-" (the big mess is a base32 encoding of the path). 2) All pods have a name string, which is human-friendly and DNS friendly (rfc1035 subdomain fragment). For example: "". These names are typically NOT user-provided, but are generated from infrastructure. For example, if the user asked for a job named "myjob" the apiserver must uniquify that into a pod name. 3) The namespace + name pair is assumed to be unique. This provides simple idempotency, for example when re-reading a config file. Open: Do we need UUIDs at all? For what purpose? The only argument I can come up with is to protect against accidentally non-unique names at cluster scope (e.g. in a giant database of namespace, name, uuid, stats) you could disambiguate colliding namespace+names pairs with uuid. Other justifications? From the apiserver's point of view: 1) The apiserver has a configured namespace. 2) All incoming pods have a name. I don't know what the formatting and uniqueness rules are here. 3) Upon acceptance, a pod is assigned a unique ID of some sort. 4) Upon assignment to a minion, the name and unique ID become the pod name. Open: Should the unique ID persist if the pod is moved to a new minion? Maybe we need two IDs - one that sticks across moves and one that does not? , Daniel Smith wrote:
Numeric host names:
Regarding the open question: is a pod on a new minion the same as the old pod? Is a move an action that is logically part of kubernetes, or is only "remove" and "create new" available? If it's the former, it seems like the ID should be the same, if it's the later it seems like it should be different. We have a use case for being able to move a pod from minion to minion (and any volumes that come with it) - however, since this requires the volume data on disk to be in a resting state, it's not an operation that seems to lend itself well to the replication controller (since the move is inherently stateful). So I would expect this to be managed as an operation above the replication controller, vs part of it. One namespace per apiserver can be limiting if the namespace is automatically bound to DNS and you're dealing with very large numbers of containers, but it doesn't sound unreasonable if you are using wildcard DNS.
BTW, it looks like docker IDs are more or less just 32 random bytes: They enforce them to be unique per-machine:
Unique ids disambiguate among multiple instances reusing the same name over time, such as when polling termination status. Beware of moving unique ids. Once we do that, they are no longer unique, such as in the cases of live migration or even just preloading. If we do move ids, we should have 2 ids, the movable one at the cluster level and a non-movable one at the node level, and the movable ids would correspond to zero or more of the non-movable ids.
OK, collecting thoughts again. If this is acceptable, I (or Clayton) can write up a short .md file covering identifiers. There are still a few FIXME comments in here. Please tell me if I am capturing any of this incorrectly. Global: We spec most names as RFC 1035/1123 compatible. Docker allows container names to include underscores, which we reserves for use by infrastructure. From kubelet's point of view: 1) All pods have a namespace string, which is human-friendly and DNS friendly (an rfc1035/1132 subdomain). For example: "". This is used to indicate the provenance of the pod. If the namespace is not specified when creating a pod, kubelet will assign a namespace derived from the source of the pod. For example, a file "" on host "" might get masterspace " file-" (the big mess is a base32 encoding of the path). (FIXME: do we need this if we have a UUID as spec'ed below?) 2) All pods have a name string, which is human-friendly and DNS friendly (an rfc1035/1132 subdomain fragment). For example: "". These names are typically NOT user-provided, but are generated by infrastructure. For example, if the user asked for a job named "myjob" the apiserver must uniquify that into a pod name. 3) The namespace + name pair is assumed to be unique. This provides simple idempotency, for example when re-reading a config file. 4) When starting an instance of a pod for the first time (i.e. not restarting an existing pod), kubelet will assign an rfc4122 compatible UUID to the pod. This provides an ID that is guaranteed unique across time and space. If the pod is stopped and an identical pod (same namespace + name) is started, a new UUID will be assigned. 5) Kubelet will use the aforementioned identifiers to produce unique container names, for example "". NB: The UUID is per-pod, not to be confused with Docker's own container IDs which are per-container. From the apiserver's point of view: 1) The apiserver has a configured namespace, for example "". 2) All incoming pods have a name assigned by the originator of the pod (be they human users or other infrastructure). Pod names must be unique in space, but not time. 3) Upon acceptance, a pod is assigned a unique ID (FIXME: just spec it as rfc4122 again? could maybe be simpler here?). This provides an easy way to disambiguate successive pods with the same name. This ID will persist for the lifetime of the pod, across restarts and moves. (FIXME: is this really needed or is name good enough?) 4) Upon assignment to a minion, the name and unique ID become the pod name. 5) The namespace + name together must be less no more than 255 characters long. If DNS service is to be configured automatically (not a feature yet, but has been discussed), the pod namespace + name will already be DNS compliant. , bgrant0607 wrote:
I think this seems like a good outline! Two comments: config files, maybe use a hash of the file vs a base64 encoding of the path as the component of the namespace. Just seems more natural. "These names are typically NOT user-provided, but are generated by infrastructure." I'd consider being more explicit about that behavior, even if it's as simple as a "unique prefix will be to the name". This sorta touches on the whole id/name discussion....
Another positive attribute of unique ids: they are typically more concise than names, and are therefore more efficient to use in cross references in logs. Based on experience in Omega, I think there's no question that we need unique ids. Also, if you play around with Docker for a short while, you'll find a pile of old objects that only have unique ids and can no longer be referenced by name. How do you query them, resurrect them, or delete them once their names have been recycled? Is there a compelling reason to use a different kind of unique identifier in the apiserver than in the kubelet? I don't think so. We could probably shorten it (e.g., remove the locator) when we combine it with the name to form the name for kubelet.
Definitely prefer hash of something in the config file (maybe hash of the JSON of the pod's manifest or something) for use in the namsepace component.
I'm happy to write this up as part of (along with some more tests and the apiserver implications) once folks reach closure.
One minor note regarding Docker ecosystem - "name" is currently being used in Docker for a lot of lightweight integrations (linking, sky dock dns, hostname in container). A side effect of any generated name for a Docker container is that those lightweight integrations may become more difficult for end admins. Is there a practical way to make the name appropriately unique on the minion without breaking those potential integrations (making the name a subdomain fragment by omitting '.' for instance)
I would also like to see us drive labels all the way down into Docker. That would enable us to abandon much of what we are encoding into the name today. Brendan On Fri, Jun 27, 2014, 8:39 AM, Clayton Coleman wrote:
+1 to what brendanburns@ wrote. Labels provide much cleaner solutions for many use cases.
, Justin Huff wrote: This is an implementation detail - human-friendliness of docker's names is a nice-to-have not critical, I think. Yeah, where is the line between "ID" and "name"? The apiserver COULD just send a UUID for the name, but I think we want the structure and content of a pod spec to stay largely the same throughout the entire system, at least for now. I don't think we've got any consensus on diverging it, anyway. For that reason, I think "name" makes sense. It is more human readable than not.
+1 to labels , brendanburns wrote:
What would break on those? We're not proposing to do anything with name that is not totally valid as per standalone Docker? , Clayton Coleman wrote:
It's not that the names aren't valid for Docker, its that something consuming those names (as a dns prefix a la skydock, or as a hostname) would break due to length or format on the Kube generated name. I don't think that is a blocker to the proposed naming patterns above, but it's a consideration when thinking about how other software that plays well with Docker might react. Concrete ideas might be to reduce the generated names' length and avoid using '.' as a separator.
I would argue (for the sake of arguing) that anyone who was making assumptions about container names deserves to be broken. But yeah, let's keep an eye on it. , Clayton Coleman wrote:
Agree, naming is hard
Writing up a summary md - does this belong in api/doc, , or another location?
I would go for or something , Clayton Coleman wrote:
Referenced pull includes a summary of this discussion, open questions: implication 3-1 (agree/disagree)? I defined "pod unique name" and "pod unique identifier" to represent thockin's kubelet items 3 and 4


If you are using a private docker registry, you first need to call something like and provide a user/pass/email. Any further calls to will pick up your identity from a file in $HOME/.dockercfg. Before a kubelet starts a container, it calls the equivalent of . Unfortunately, the auth information is not picked up from $HOME/.dockercfg, and there is no alternate way to configure it. The kubelet needs to support docker registry auth.
Here's where the auth should be getting passed to the docker client:
I'm not certain, but it looks like authenticated docker pulls would have worked before this patch:
See .
ah, search skills failed me again


Should be able to tell from logs if a minion isn't present because cloud provider didn't tell us about it, or if it's failing health checks.


https://travis- Seems surprising-- I will look into it later if no one else does first.
Got another flake. Looking into it.


if you delete a service and restart it the port is no longer accessible To reproduce: 1) start a pod 2) start a service All works fine port is assigned - traffic flows 3) delete service 4) start service the port is now not assigned Here are sample configs: pod: Service:
I thought this was a known issue, but I could not find another issue on the same topic. Note that all containers started after the first instance of the service was created will have environment variables referring to that instance and won't see environment variables referring to the new instance until they restart. This is a fundamental shortcoming of the current approach.
I understand what you just replied but that is not the issue here. The problem is that when you delete a service , no other service can resuse that port successfully. The port is removed from a netstat listing. When a new service tries to use that port it gets allocated but you can not communicate to the pod through it. It is hosed until I re-install the cluster. (I have not tried rebooting the cluster) BTW this is a bug not a design issue.
Ah. The previous issue I was thinking of was . It sounds like the issue hasn't been completely resolved.
I will take a look at this today.
Sorry for the problem, has the fix. --brendan
Thanks! On Aug 19, 2014, at 2:26 PM, brendandburns wrote:


We have an e2e test that runs the guestbook example. We should also have one that runs through the other example.


Running vagrant from scratch gives the error: For vagrant cluster, the last minion cannot be detected. The /tmp/minions file only shows two minions, but after a while, using the same command, i.e. will get three minions. There is a race between kube-up and validate-, we should wait a while before panic.
I can look to fix this.


As a sysadmin When using the Vagrant cluster provided And logged in as the vagrant user to the master or minions I would like to be able to call docker commands wihout requiring the 'sudo' prefix. For Fedora =19 and RHEL=7 this can be enabled by adding the vagrant user to the docker group in /etc/groups
This should be trivial, but its tricker to do because of the whole salt configuration process as the docker group is only created on each kubernetes-minion once docker is installed by the salt-minion. To do this properly, we would need a salt state change for this, and new flag in to let you specify additional users that should be in the docker group.


When creating replication controller (e.g. frontend php-redis from guestbook example), scheduler gets stuck and never recovers. This is a race, I have to run several times to see the error (A script to delete controller and pods, then create new controller, leave enough time in between for k8s to react). I've created PR for an attempted fix.
I saw this too while following the guest book and creating the php/frontend step. In my case, I had 2 minions while the replication controller configuration had configured 3 replicas, so 2 pods got scheduled -each on another minion, the third pod had empty Host (saw that when doing 'list pods'), and the 3 of them were stuck in "waiting" status. I deleted the controller and the pods, fixed the configuration in json file to be 2 replicas and recreated the controller and everything worked. Does your fix take care of this use case? (too many replicas, too few minions/one of the pods is not scheduled)
This is different from your use case. If you have more replicas than minions, then the scheduler will get stuck for sure (with the error msg probably being 'failed to find a fit for pod'). AFAICT, it's not support yet.
If you remove the HostPorts, it should work for you. thanks for report.
, thanks for your replies. I understand why it isn't working, but I'd expect one of the 2 options to happen instead of the current situation: the creation of replication controller if there are less minions than replicas, AND there is a hostPort property. In other words - validation of the content during creation. least schedule all replicas that can get a minion (in my case 2 out of 3) and leave only one at "waiting". Is there an issue open about the use case I describe?
should be true today. is less possible than it seems like-- minions can come and go, and become over or under loaded in the time between creation and when the pod shows up at the scheduler. So even if we check up front, we still have to check again later, and sometimes the first check will pass but not the second.


the file and line for these log statements is in the That's not helpful I1031 11:09:06. ] 'foo' has no storage object I1031 11:09:06. ] 'pods' is not a redirector I'd rather see Maybe use runtime.Caller in httplog to somehow fake the log to come from the caller's file/line.
Yeah-- actually this should be a feature of the glog package. Would be awesome to do:
Followup: glog is an opensource copy of a google-internal library; there's already something enabling this internally; we'll try to get a new version pushed externally in the next week or so. Then we can fix this.
Awesome. On Oct 31, 2014 2:53 PM, "Daniel Smith" wrote:
should meet your requirements.
+1, I would love to have this.


I suspect this is dues to the recent change to container VMs as the base image? ifconfig docker0 - exists but should not. iptables-save | grep docker - 4 docker rules that should not exist.
Is this still relevant?
was still a containervm turd the last time I checked. It's just not causing any harm.
Yeah, no harm, just ugly. , Zach Loafman wrote:
I think we should remove the docker0 from minion nodes, but do we still need this for master node? cc/
This is required by running etcd server as pod with flannel on gce nodes. Please sync your cluster with this fix.


It looks like we always try and hit the metadata server even if we aren't on GCE. We then print an error out. This should either be silent (assume after X attempts that we aren't on GCE?) or should only be activated with an option. Email thread: !topic/google-containers/w-ei5Xs6K0Y
cc:
FYI, the way the credentialprovider logic is set up, providers are asked once per keyring creation whether they are enabled. If they are enabled, then they are asked to provide their contribution to .dockercfg each time the lazy keyring is accessed. For efficiency, there is a caching provider, which composes with heavier providers and stores their .dockercfg contribution for a predetermined TTL. What was happening here was that "Enabled()" for our GCE-metadata implementations was logging about errors fetching "http://metadata", where the entire "Enabled()" check is "err == nil".
I believe that this was fixed by


0/8 is with a company's intranet, so it can be insecure. This is similiar to "For GCE, allow insecure registries anywhere in 10.0.0.0/8" Thanks!
Linking to and .
This was


it's impossible to GET replication controllers on v3. on v1 on the same server it works fine. example request: GET on http://localhost:8080/api/v1beta3/replicationControllers returns 404 http://localhost:8080/api/v1beta1/replicationControllers returns 200
v1beta3 moved to all lower case URL Would give that a try first Sent from my iPhone
looks like it's working. but it breaks backwards for clients who worked with v1. For instance, in my gem I can't now smoothly work with both versions because I need to treat the name of the resource differently. why don't at least redirect those who call replicationControllers to the lowercase too? Also, there's no way to know which resources are available and their names. A lot of REST api on their most top level at least publish the links to main resources, but if I call: http://localhost:8080/api/v1beta3/ it returns 404
Because we are actively trying to drop support for mixed case prior to 1.0, I did not continue supporting the old url paths. If you look at the go client, we have a "legacy" flag we use to control this and a few other behaviors.
well yes, but breaking backwards creates more work for the users... more "if this version ..." kind of stuff.
We got v1bet1/2 wrong and don't want to carry it forever. Stopping it early before v1 is our priority.
ok :( are the REST apis going through any kind of design review or code review?
v1beta1 is really an alpha quality API. It grew organically and did not go through any kind of design review, as evidenced by the many issues you've filed. We want to get rid of it ASAP. We intend v1beta3 to be the "release candidate" for the v1 API, which will be the stable API. As for knowing what exists, you can now browse or GET . Additionally, pull is in progress, and will return a list of valid paths to GET .
thanks. will check swagger for now


For example, kubernetes-ro service appears with empty selector (this service was created by the hack/local cluster) - is that ok? What meaning such service has if it has no entity to fwd traffic to?
kubernetes and kubernetes-ro are currently "virtual" - they are not backed by running pods, but by the master itself. , abonas wrote:
if that's internal impl, why is it exposed to user? let's say a client pulls all entities and builds a UI around it. then a service with no selector arrives, and it's internal to kube - how the client should know not to present it/touch it? (similar to the "net" container in pods)
It's exposed exactly FOR users - so they can find the apiserver. It has a corresponding Endpoints object, it gets a portal IP and a kube-proxy rule, and it gets service env vars and a DNS name. The fact that it is virtual is an implementation detail. That it exists is all users should need to know. , abonas wrote:
And FWIW, the master components may one day actually be pods... , Tim Hockin wrote:
thanks for the explanation. so any general rule how to distinguish kube pods/services from user defined?
Do we need to distinguish them? We could add labels, I guess, if it is really important to distinguish them.
sometimes users would want to see only the things they defined and not implementation details of the system.
One could argue this is why the users should be using labels. I'm not against defining labels for "special" stuff, but it's new territory, so we should enter with caution. , abonas wrote:
I think the right answer might be to put them in a special "cluster" or "system" namespace. , Tim Hockin wrote:
See If we put them in a "kubernetes" ns, the DNS name would be :) I'm not against segmenting them into a different namespace. We just need to decide on what that namespace is called, plumb through that configuration, decide on compatibility for a short while or not, and decide if that namespace is "special" or just "different" - i.e. should the skydns, heapster, etc pods go into this namespace? , Brendan Burns wrote:
As said, the services are there for applications that need to contact the apiserver. Additionally, nil selectors can be used by users to import external services. Adding labels to the kubernetes services won't help. Users need to manage their objects using labels. /cc
Yeah, lots of consumers will expose "special" services. IN fact, in many clusters the "special" services will outweigh the unspecial ones. At this point, the namespace they are in is completely arbitrary, and I think because it is a name, it should be free for the end user to choose where they are surfaced.
Per the proposal here: , I think the namespace should continue to hold these services, and they require no special labels. I sense comment is made without thinking through what Kubernetes operators do to bootstrap a cluster. I think it's reasonable that namespace holds everything you need to run, and operators may or may not choose to segment their data in other namespaces. If they want to not see this information, its an exact reason for why they should use their own namespace. I don't think we need anything new here.
- I'm not clear on your meaning. Did you mean that we should make the "cluster admin" namespace a config param that everyone knows about and can be different cluster-to-cluster? Or did you mean that default is as arbitrary a name as any other? I'd be fine moving things out of default into "kube" or "kubernetes" or "cluster" or something, but I am less convinced that a flag is a good thing. , Clayton Coleman wrote:
- wherever we stick this stuff, it should be the namespace that is initially bootstrapped, which today is . I continue to believe that is as good a name as any other. I do not see a need to do more here.
I think as soon as you bring naming into the picture, making decisions based on names gets very tricky (what if I want my logging service to be everywhere?). If there is a magic namespace that exposes things to the rest of the cluster, we should ask whether there's only one magic namespace, or multiple? Are services supposed to cross namespaces? It's great that pods can find the master... but that's not the only thing they'll need to find in each namespace. And baking in special names (or customizable names) into your app code is fragile, because then your code only works inside Kubernetes. I'm just trying to pump the brakes on special services prior to us designing them... :) ----- Original Message -----
oh, to be clear, I don't want "special" services - I am merely asking if moving non-user-defined objects (where that is defined very loosely) out of "default" is worthwhile. I tend to agree with Derek that it's not worth energy. , Clayton Coleman wrote:
agree with you that lots of consumers will expose "special" services We should not use the (namespace + k8s-service + skydns) mechanism for publishing the location of those special services. We should decouple the choice of namespace to run the pods of the service in from the well-known dns address of special services. Otherwise, it will be hard to move the pods of the service to another namespace, which people will definitely want to do as they grow. Instead, we need a place to publish well-known services which is not tied to the namespace of the pods of that service.
you open a whole new topic. I've noodled with the idea of a Name resource type. We would generate a Name for each Service as "service.namespace", but users could also add their own Name resources. , Eric Tune wrote:
I'd like to understand better your comment regarding nil selectors used to export external services - can you elaborate? give an example? my understanding is that service redirects traffic to pods based on labels. with nil selector service does nothing, so why is it a service in the first place? I'm pro decoupling special services/internal implementation services from user defined entities. whether by separate namespace, or by saying that the default namespace is reserved, or by any other solution. because otherwise internal implementation and user defined content are mixed and that's no good for a proper management. just tagging you so you could see this issue and the related discussions.
Services abstract backends - they don't have to abstract pods. Some examples: You want to have an external database cluster in production, but in test you use your own databases. You would have a service for both and use the environment variables, but the production service has a nil selector with its endpoints set to the database cluster. You want to have an empty service because you're reserving a name (that isn't accessible yet) but will be in the future You want to point your service to a service in another namespace or on another cluster. You set the endpoints for your service to the publicIPs of the other service. ----- Original Message -----
, thanks. regarding the last point - I was under impression (based on some other discussion) that there would not be cross namespace interaction? that entities can can access only entities in the same namespace?
Cross-namespace is currently allowed and I think has to be allowed, though perhaps (in the fullness of time) that can be policy based rather than wide open. , abonas wrote:
please see this correspondence. my understanding is that there is isolation
Responded there. I think there's some less-than-crystal-clear words in play. , abonas wrote:
+1 to Eric's comment about ensuring there is a consistent manner for system services like kubernetes-ro, if that's not fixes for all clusters it's going to be a nightmare to write reusable containers for multiple different clusters. I don't care if we do that by using a constant namespace, or by giving system services two different names, but there needs to be at least one name that is constant. Regarding default vs. cluster namespace, I think the issue is one of clutter as a user is starting to understand k8s, you turn up a new cluster, launch a pod, or service, and it's hard to find your one object on the sea of system created pods and services. It's visual clutter that confuses people as they onboard. On Jan 23, 2015 3:01 PM, "Tim Hockin" wrote:
- can you drop me a line (my github name at google) - I want to ask you something, and you have no contact info :) , abonas wrote:
sorry for naive question but why would I care as an user that wants to run its existing workload on Kubernetes about kubernetes-ro ? If I write an application that uses kubernetes-ro then I'm locked into Kubernetes. I understand the need for kubernetes-ro being accessible by cluster level services such as monitoring, logging. I also can imagine that a power user might want to start its own cluster services that use kubernetes-ro. I don't see clearly why would majority of users care about kubernetes-ro being visible from their container. I see a reason for the cluster services being visible and accessible for Kubernetes developer, but from an user perspective such services only make the system more complex: why do I see these services (pods,rc) ? are they related to my application ? should I care about them (especially when I use hosted Kubernetes) ? why does my container get all these environment variables for cluster services ?
I will update file to explain why one would have service without a selector.
Right now, every service goes everywhere for simplicity. Name spacing and filtering probably make sense in the long run, but it's not a high priority for v1.0 Brendan On Mar 4, 2015 5:02 AM, "Rafał Sokołowski" wrote:


As long as we have it in the API, it should maybe be renamed to gcePersistentDisk ?
SGTM


This whole set of tests (if it can't be run) should be omitted from the test execution, otherwise folks running on Mac can't verify everything else is clean in a sane fashion. Passive aggressive assign to Tim, I may get around to fixing this before he does.
Last I checked Travis + Go didn't work. We should check again, and enable OS X builds in Travis if it's possible.
You could argue that if the unsupported mount () is being called in a test, there is a problem anyway - it has to run as root. Will investigate. On Jan 25, 2015 9:09 AM, "Brendan Burns" wrote:
Without a Mac to test on, I feel like this would be better fixed by someone who uses Mac... Probably: just rename to I just discovered "implicit build constraints", sigh. Go is crossing from user-friendly to sort of absurd, IMO.


wrote: I just ran across this the other day. but I am seeing the error in the service. Here is a sample of the output. I am running v0.8.2. I can get the behavior to go away temporarily by rebooting the node or restarting the service but it always seems to come back eventually.


We found yesterday that cAdvisor wasn't being started properly in 0.9.0 and 0.9.1 due to an issue with manifest file format changes (, fixed in ). If any of our e2e tests even checked that cAdvisor was running or exposing metrics on all nodes, it would have been caught before release. We should add at least some coverage for it to catch bugs like this.
If any of you are up for adding an e2e test for cadvisor, that'd be much appreciated!
I will work on adding one for cadvisor. , Alex Mohr wrote:
We already have a e2e test, which validate monitoring of Clusters using cAdvisor. We should figure out why the test passed without detecting such failure, and improve that test.
The existing test will succeed even if some of the nodes do not have cadvisor running. We can enhance the monitoring test to specifically check for cadvisor's health as well. , Dawn Chen wrote:
This test seems to be failing deterministically.
This was after fix to retry, but we're not exactly sure why.
Once I have emerged from the e2e test for cluster level logging with Elasticsearch quagmire (soon I hope -- perhaps by Wednesday) then I might also be willing to lend a hand with this since some of the experience might be relevant.
Actually latest error looks like So it's still not coming up after 5 minutes. The question is how long should it be taking to come up, given that the cadvisor containers get scheduled basically on node creation. The test finished in < 1s on my e2e cluster that has been up for awhile.
Or there's an actual bug here. can you look into it?
(I looked at the Jenkins log, and this test ran a good 10 minutes after startup, so it probably had 15 minutes to finish coming up. That seems wrong, given that cAdvisor is actually the first container we ever schedule, and it gets scheduled raw.)
Sorry, I hadn't seen -- not sure if that fixes this?
No, there are failures after as well. On Feb 20, 2015 5:39 PM, "David Oppenheimer" wrote:
I don't see any issue with the test itself. The test attempts to access a REST endpoint ('/stats') via the proxy on the api-server. Is it possible that the api-server doesn't get to a stable state for a while?
This test executes after several others that hit the apiserver? On Feb 22, 2015 4:15 PM, "Vish Kannan" wrote:
Noob question: How can we tell if it runs this test after the other tests?
You can't tell deterministically (i.e. it's not hardcoded). I was merely commenting that in several runs where this fails, it has run after many other tests. We've been randomizing the spec order for a while. You can look at the Jenkins console logs to get the random seed for a given run: And the console logs have the order just by manually scanning the horizontal bars, but it's kind of pain. (That seed was off of go/k8s-test/job/kubernetes-e2e-gce/2638/consoleFull)
The reason for this failure is . I am closing this issue since the test itself is fine.


W0128 18:44:26. ] Pod from test failed validation, ignoring: namespace: required value '' E0128 18:44:26. ] Could not construct reference to: '&api.BoundPod{TypeMeta:api.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:api.ObjectMeta{Name:"foo", Namespace:"", SelfLink:"", UID:"foo", ResourceVersion:"", CreationTimestamp:util.Time{Time:time.Time{sec:0, nsec:0x0, loc:(time.Location)(nil)}}, Labels:map[string]string(nil), Annotations:map[string]string{"":""}}, Spec:api.PodSpec{Volumes:[]api.Volume(nil), Containers:[]api.Container(nil), RestartPolicy:api.RestartPolicy{Always:(api.RestartPolicyAlways)(0xe3d670), OnFailure:(api.RestartPolicyOnFailure)(nil), Never:(api.RestartPolicyNever)(nil)}, DNSPolicy:"ClusterFirst", NodeSelector:map[string]string(nil), Host:""}}' due to: 'selfLink was empty, can't make reference'. Will not report event: 'failedValidation' 'Error validating pod foo.<empty-namespacefrom test, ignoring: namespace: required value E0128 18:44:28. ] Unable to read config path "randomnonexistentpath": path does not exist, ignoring --- FAIL: TestReadFromFileWithoutID (0.20 seconds) Expected update, timeout instead


writes: As we start looking into etcd and event memory pressure [], we should look into switching over to etcd 2.0 which was released today.
cc/
+1 On Jan 28, 2015 5:29 PM, "Dawn Chen" wrote:
My long-running cluster tests resulted in a hard-crash on etcd: We should move to 2.0 ASAP in the hopes of (a) fixing the issue and if not (b) prioritizing the fix since we'll be complaining about head. Abhi, is this something you can pick up and drive to land in O(days) please?
Calling this P0 as it blocks further long-running cluster stability testing.
I will do this today. , Alex Mohr wrote:


reported a variety of failure modes in .
+1
I figured I'd add to this issue rather than create a new one. I attempted to create a vagrant cluster today (dev tree is up to date) with 1 Master and 1 Minion and the cluster creation is stuck in the 'Validating master' loop ( for over 10 minutes in the validating loop ). salt-master and salt-minion logs seem to be empty ( they don't appear to be running, in fact ).
I still have the cluster running in case anybody wants additional information.
/cc please let me know if there is any other information about my vagrant setup that I need to post here.
I have been working to get more stability in this environment over last few days. Can you try again from HEAD after this merges:
Can you also verify that you built a release prior to running cluster/Kube-up?
yes, I started from scratch and built a release before running cluster/kube- - cluster/kube- seems to fail fairly early in case there is no release build available.
It looks like got merged. I'll try again and get back to you.
The vagrant cluster seems to come up ( completes master and minion validation ). However, there is new credentials related error that is showing up at the end of running cluster/kube- :
I'll add one more problem I've encountered. Runing from head:
New to me. Will investigate more tomorrow. Sent from my iPhone
Also new to me :-) - will see if I can reproduce either. Sent from my iPhone
Your credendials and certificates are not working with new environment. Just delete your .kubernetesvagrant* files and recreate the cluster. It would be nice to automate this task, if you tear down the cluster.
I am able to reproduce the 502 Bad Gateway error and am working on determining what changed in order to provide a resolution over the last couple days. Please assign this issue to me.
The issue appears to me that the kube-apiserver is not getting provisioned by salt correctly. Parsing Salt logs to determine why.
Looks like the file is not getting laid down by salt for some reason ;-)
The Jinja template was changed as part of identifying the cluster. Fixing up now, and will send a PR to close this issue.
that was it, thanks!


contains function get-password. This uses a template which gets the current context. But a user who has not yet used the kubectl contexts feature has no context set. This causes a nil pointer deref while evaluating the template.
I think kube-up creates the needed entries in the .kubeconfig file. But it calls get-password before it does that. Need to allow the kubectl config view command to fail.
okay, I have a fix.


I have been following the instructions here: to try to explore the kube UI. by executing: I get the following which shows it successfully started: When I try to access the page I get a 404:
could you try running ? I believe is a typo of
Also, try accessing instead.
thanks


I tried to create a pod without containers to see if it's supported or not. (does a pod without containers have a meaning in k8s?) I was getting "conflict 409 already exists" error although I used a new pod with a new name/id, so it's not clear whether it's supported or not, and whether this error is a mistake (409 instead of 400)
What is the reason to create pod without specifiing image / creating container? Please provide a use case when such functionality is necessry.
- to me it's not necessary. However if it's not supported, a validation should prevent it not to allow corrupted entities. And if it is supported, it will be good to understand why. For instance, service with nil selector is supported, though it might not make sense to some users. see this:
Yes, it should be allowed, and I'm pretty sure it used to work. Could you please post your object schema here or somewhere?
what do you mean by object schema? an example json of such pod or something broader? and btw what meaning a pod has without containers?
cc
I'm fine with banning this for now. The use cases where it would be useful (e.g., reserving resources, staging deployment, prefetching images, ...) aren't currently well supported. When we add the features that could take advantage of this (e.g., pod-level resources, additional of containers during update, container volumes, pod/volume init hooks), then we can make it legal again at that point.
in that case, a clear error message coming from REST would be in place when a pod with no containers is recieved by the server :)
I'll look into this. AFAIU we want to return meaningful error if user specifies pod with no containers. Is my understanding correct?
Correct. We should do it in validateContainers:
cc
should we document this somewhere (e.g. ). Validation is not versioned, so any change would apply to all existing APIs.
Yes, it should be documented, in the field descriptions, such as:
I'll send a PR for this.
cc


We use capitals for Pod fields (ex: Status.Host) and camelCase for Event fields (ex: involvedObject.resourceVersion). We should be consistent. Should it be capitals? camelCase? Or should we be case insensitive?
IMO, we should use camelCase for pod fields, to match the way it's serialized in JSON.
I agree with The goal here is to be consistent with the API, not with the internal Go representation.
I'd like to fix this before v1beta3 is cast in stone.
ok. Event field selectors are correct then. I will update the selectors for pods to make them camelCase.
I am wondering if I should change this only for v1beta3 or for 1 and 2 as well? This will be a breaking change for beta1 and 2 if we decide to do that.
v1beta3 only, though that's hard, we should reconsider. We're contemplating another "breakage day", and I doubt this is widely used.


As Satnam mentioned in PR , our Fluentd pod tends to not collect the first chunk of logs from each container. I've noticed that it seems to miss different amounts of logs depending on how quickly the container generates them, so it may be a timing issue. cc
See


Follow up from
Bumping this up to P1 given that 0.12.2 wasn't able to create a service with an external load balancer and we didn't notice before cutting the release.
No, to be clearer, 0.12.2 wasn't able to create a service with ELB on GKE. That was on our side entirely.
But it doesn't invalidate the need for this.
The example is using ELB.
are you actively working on this? If not, I'd be up for doing it today.
Maybe not actively but I know what to, however I won't be able to finish it today, so if you think you can do it today please move forward.
We run the e2e tests on GKE, so if we had a test for creating an ELB we would have known that the 0.12.2 release wouldn't work on GKE. I don't see how your point applies. The e2e tests are in place so that we don't have to test each feature manually to verify that it is working, and we only found out that this feature was broken on GKE after we pushed 0.12.2, hence the need for an e2e test.
I think and you are in fact in agreement
right now, k8petstore tests the external load balancer by curling down the amount of transactions ingested from the REST API. im playing with porting it as a e2e test now. The code in the shell script is a little raw, but you can see that it specifies a PUBLIC_IP and then curls down from it at the end ().
that sounds great. What I'd really like to see here though (and what is going to do early next week) is a very simple test that isolates creating a service with an ELB to verify that creating a load balancer works in isolation from a complex test of other parts of the system. This will give us a really clear signal when the test starts to fail instead of needing to debug lots of moving parts when we see a test failure.


We already "healthcheck" kubelet and docker periodically, but miss one for kube-proxy.
It would make sense to do (only bind to localhost) at the same time we add the healthchecking configuration.


stack trace here: it's due to a shared variable:
Hi thanks for reporting the issue with a patch! Do you want to submit a PR so that you can take the credit for your fix? :)
done , Yu-Ju Hong wrote: 585.241.9488 (voice) 650.649.6071 (fax)
FYI, I think the same bug may still exist in your new PR.
There are multiple p
There are multiple PRs in flight, so I don't know which one you have in mind. One of them removes probing logic from controller altogether.
I was referring to , but I didn't read that PR closely so the bug may longer exist there. Again, just a FYI. :)


Appears to deadlock or otherwise hang after a few seconds and then hit the 5-minute timeout. From start of log: followed by stack traces of zillions of goroutines that mostly report being blocked for 4+ minutes. Recent instances: https://travis- https://travis- cc
I tried to reproduce the issue by run kubelet unittests 200 times (100 for v1beta1, and 100 for v1beta3), but failed to reproduce the issue. I poked around, and found the only difference between my local machine and test machine is that I am running go 1.4.2 and test machine is running 1.3 (both failure cases posted by are with 1.3.0). I remembered from online I saw a deadlock issue reported against old golang version, but haven't found that issue yet. I will continue trying to reproduce the issue.
Saw this in travis runs too: https://travis-
Yes, there have been multiple failures today. Check shippable, also.
FWIW, these are not all kubelet tests failing, but all of them have many kubelet go routines in the trace - mostly stuck in managePodLoop().
I reran a couple of thousands time last night before I leave, and all passed. Yu-Ju just pointed out that the failure posted above by is go1.4. Shippable is in a really bad shape today, tons of different tests are failed, which should be handled separately.
Ok, with some hacky debugging message into kubelet package, even I couldn't reproduce the very failure reported here, I made some progress on understanding the issue here. Basically there are two issues: 1) The flaky tests reported here are TestServeExecInContainer. All reports posted here have the same stack trace below: 2) There are tons of goroutines, especially one created for each pod. Kubelet leak those goroutines. One can see when test initial run, there are 106 goroutines. Test itself create another 41 goroutines, but none of them are created for podWorkder.
FWIW, the tests in were creating go routines it didn't need at all. I just removed them in
Thanks for cleanup. Now the number down to 91:
cc/
Did something change related to this recently? I haven't heard any flakiness reports for a long time, and now they're starting up again. Any ideas? Sent from my iPhone
I didn't see any changes related to this feature. Actually I even couldn't reproduce it on my desktop. Please note that the comment I made above all those leftover goroutines are separate issue thought.
This has exposed a deadlock in spdystream. I'll work with to come up with a fix.
Upstream issue
Cool ncdc@ ! dchen1107 by the way, I've run across a few similar bugs recently where their reproducibility hinges mainly on how the test cluster was brought up. If the test cluster comes up right, then the tests pass 100% forever. If the cluster comes up bad, the tests fail, sometimes forever. Were you rerunning your tests against the same test cluster? Or rebuilding your test cluster between every test invocation?
This is unittests, not e2e. We saw test flakiness through travis and shippable, but never on my desktop.
This will hopefully fix the deadlock: I will do a bump commit once it's merged.
Thanks for your quick fix. I will clean up goroutines after that.
it's entirely possible there's some bugs in my exec & port forwarding code around leaking goroutines. Are you going to take an initial look to see what's sticking around?
Yes, your test leak some goroutine, but other tests too.


goroutine 11 [running]: net/http.(ServeMux).Handle(0xc20800a8a0, 0x9e9490, 0xd, 0x7f7d12f190a0, 0xb0e7c0) +0x244 net/http.(ServeMux).HandleFunc(0xc20800a8a0, 0x9e9490, 0xd, 0xb0e7c0) +0x6d net/http.HandleFunc(0x9e9490, 0xd, 0xb0e7c0) +0x48 ·001() +0x58 created by (*SchedulerServer).Run +0x1c7
cc


There is a funny subtle behvaiour in get namespaces, in which if i call more than one time, it seems that the removal from Terminating state is accelarated. It seems this might indicate a bug, or maybe its not a bug and the eventual semantics of is a little funny sometimes. in any case, here's what we are seeing : 1) when i delete a namespace the first time usning kubectl, we get a state change to . that is correct, of course. 2) but ... if make the call a second time, then the same status line is immediately removed from . Expected behaviour : A second call to "kubectl delete ..." shouldn't have any effect on the timing of removal of a namespace from 'Terminating' state*... So i guess there is some "eventual consistency" subtle bug that is resolved by repeated calls to
suggests this may not be e bug, maybe its just the etcd eventual deleter thingy.
Looks to definitely be a bug. The 2nd delete seems to have interrupted the delete process and artifacts (like events) from the namespace still exist in etcd.
investigating...
oops, see the error, will submit a fix.
I have a fix, but need a test case, will submit PR tomorrow AM


GCE has a length limit of 63 characters for its load balancer components (target pools and forwarding rules), and AWS has a length limit of 32 characters. Our current load balancer name construction of clustername-namespace-servicename can exceed both limits, especially AWS's. We should use a shorter name construction method less likely to hit these limits.
Thanks for filing this Alex. My suggestion would be to use a simple UID, as per utils.NewUID(). We'll need to store the mapping from {clustername, namespace, servicename} -UID somewhere stable like etcd. Using anything based directly on {cluster, namespace, name}, for example a stable hash of each, has the problem that the aforementioned tuple is not unique over time. This raises the possibility of e.g. the following scenario: creates mycluster/mynamespace/myservicename, which creates external load balancer named stablehash(mycluster,mynamespace,myservicename) deletes mycluster/mynamespace/myservicename, but at the time MyCloudProvider is down, so kubernetes can't delete ELB named stablehash(mycluster,mynamespace,myservicename). Kubernetes keeps trying in the background to delete the ELB. the mean time user creates a new service named the same as her old one mycluster/mynamespace/myservicename. Kubernetes tries to create an associated ELB named stablehash(mycluster,mynamespace,myservicename), but this fails, because the ELB exists already due to 2 above. ... etc. Badness ensues.
Assigning to Chao Xu, as this makes for a good starter bug.
Hmmmm.... can't seem to assign to caesarxuchao yet. Not sure why.
He has to have write privileges on the repo before issues/PRs can be assigned to him.
But we have many contributors with issues assigned to them that are not in the kubernetes-write group (that only has 48 members). I must be missing something. How do I grant him write privileges on the repo?
We do? That's not how I understand it to work: If you want to add him, go .
has been because the load balancer name changes if the service's UID changes (e.g. if it's deleted and recreated). Think we can change this? A hash of cluster name, namespace, and service name should get around it, but there may be better ideas as well.
Sure. I'm not confident that I can fix this quickly. If it's urgent, maybe you can assign some one more experienced to fix it? One question: I guess the ELB's name should stay unchanged as long as the "zone+namespace+service name" doesn't change, am I right?
I think I'm the guilty party here. I was unaware that when we create a new service with the same name as a previously existing service, that it needed to keep the same load balancer name and IP. This seems broken to me, but I may be missing the larger context. We can quite safely roll back the change while we decide on the best alternative solution. Q , Chao Xu wrote:
I think the real breakage is that we don't have any way to claim an LB-IP at a larger lifespan than a single service. Alex, can you go back to OP and get a sense of what's happening there to trigger this? , Quinton Hoole wrote:
Will do. Even if the name is kept consistent as it was before, with the way everything currently works the IP won't be consistent.
yeah, I'm not positive that the name has to be consistent. What we need is a way to claim the same public IP address, and we already have that (at least for GCE) Assigning to me, and I'll validate the length requirements, make sure public ip assignment works for AWS and close out this issue.
Note there are two separate things here: (1) a purely-name-based construction can exceed length limits given a single service and (2) some form of across-service consistency is apparently needed. (1) is trivially solvable by either always using a hash or alternatively using a two-stage process where you generate a desired name and only hashify if it's longer than the length limits by truncating it and appending a hash of said name.
AWS's load balancer hasn't been merged yet, so there's not much to check out there. I'm going to be reducing the churn of load balancers later today, which should help prevent IP address changes for services that are modified.
I think we need to be able to figure out which Service an LB is attached to from the Service, so that LBs can't get orphaned. regarding being able to claim an LB IP. I don't think that is covered in revamp... , Brendan Burns wrote:
I think that we need to roll back and think this issue through a bit better. See comment on that PR.


Rackspace support has not ben updated in a few months. cluster/rackspace/cloud-config/minion-cloud- is using etcd_servers for kubelet and kube-proxy, but support for them has been removed since v0.9, I think. I'll remove it in a few days if no-one makes a counter-offer to get the config working with latest release.
There's been some work on this but it is blocked while we figure out some load-balancer design stuff. , Eric Tune wrote:
Okay, I will remove the link to rackspace until it is fixed (). Leaving this issue open to remind us to delete it in, say, a month, if no progress is made.
cc
See also
I'd prefer to try to get things updated and keep support. However I don't want to continually become stale. every few months either.
Hi all, I want to use kubernetes on my existing rackspace infrastructure but I don't understand if rackspace suppor was removed or not. In Doc, under "custom cloud solution" it's linked but I wasn't able to deploy it neither with install method nor with . With i got a error (I have 16GB or RAM!). thanks in advance, Fx
It looks like no support has happened on Rackspace for a year.
You should not expect it to work, and we should delete rackspace support.
ACK.
Unfortunately I think we'll need to drop support for Rackspace as a deployment target since I can't commit to keeping the deployment scripts up to date. Rackspace support as a cloudprovider should be maintained.
I would be willing to support the Rackspace deployment target, if there is interest to keep it alive.
There would be definitely an interest from our side - sad to see that Rackspace has no interest in maintaining it
No problem, I will take over the maintenance. It will take me a bit of time to get caught up, but I will get it done as soon as possible.
Do you think it would be hard to get the of LBaaS working ?
Sorry it took me a bit to respond. there is a quick update as to where I am. I was busy with some stuff, that prevented me from working on this. But that has been resolved, and should have updates by the end of next week. However what's causing the delays is the lack of a good API for Rackspace Cloud. I've used the gophercloud, it works but I'm not a fan. Additionally gophercloud doesn't have full API support (one of the things I'm not a fan). This would make it hard to implement LBaaS. I should have a better idea of your request sometime next week.
I have seen this api docu: - is it the same as you saw? I think it should be doable with it.


If another process is altering iptables rules at the time k8s attempts to add iptables rules it will fail with a locking error. k8s should use -w and retry after a delay. Below, an external process (openshift-sdn-node) modifies iptables at roughly the same time and k8s fails without retrying. Related to
Sorry, I misunderstood the -w flag, with it retry logic wouldn't be necessary as it'd wait. Perhaps simply retrying is a better solution as -w flag may not exist everywhere we'd like.
Looks like -w was to iptables 1.4.20(August 2013).
Yeah, we have to be pretty back-compatible with iptables. We always retry after a few seconds - is there really a bug here? , Scott Dodson wrote:
It definitely seems like there is some sort of a problem. We see this randomly on system startup. In our environment OpenShift is launching kubernetes around the same time other systemd units are modifying the firewall config. If one of these others units has the lock kubernetes will report the error Scott mentions and then will never create the required nat'ing for services. Here's how to reproduce this: At this point if you run you will see the exact same error kubernetes reports. It's probably not obvious why you have to set the breakpoint at 257 but if you set the breakpoint at 260 and then step through the code it appears that the compiler has optimized lines 263 and 257 to be the same. In any case running will verify if you have the lock or not. Once you've verified you have the lock just restart OpenShift (running kubernetes v0.14.1-582-gb12d75d): At this point you can press in gdb to continue the iptables invocation and release the lock. Afterwards I will see output from cadvisor every once in a while, however the will show the firewall is still blank: I initially restarted Kubernetes with a blank firewall at 11:21 and yet at 11:38 I still have nothing:
I probably should have made it more clear, I removed the lock by pressing 'c' in gdb to continue. That's when I started waiting. I'll update my comment to make that more clear.
So something for a holding the lock? I am out of your office for the rest of this week, but maybe you can figure out - is there a stale process holding a lock? What kind of lock is it? Flock or semaphore or ... ? On Apr 29, 2015 8:44 AM, "Brenton Leanhardt" wrote:
What we're saying is that on startup it's fairly common for multiple systemd units to make firewall additions. If one of them grabs the lock (rarely for more than a few seconds) it will cause kubernetes to fail to initialize iptables and it will never fix itself. The lock only need be grabbed the microsecond kubernetes tries to initialize iptables for this bug to present itself. We know exactly which program in our environment was concurrently making iptables updates. The problem is that kubernetes doesn't appear to be retrying if it fails to initialize iptables.
On RHEL the lock is the xtables unix socket lock in iptables to allow multiple programs to update iptables concurrently. Other platforms may use a flock since that is the default upstream. To solve this kubernetes needs one more more of the following: pass the flag to as scott and the logs mention (possibly only available on some platforms). If kubernetes desides to wait for the lock it will need a timeout. retry the firewall initialization quickly and maybe backoff if it still can't get the lock It's probably worth noting that if iptables on a platform doesn't have iptables lock support concurrency is likely broken already in other ways for that platform.
It does retry all up tables ops. Unless it somehow gets pathologically phase locked with your agent, or there is some deeper bug. We have e2e tests that prove you can delete tables and it will recreate them. Are you saying that it never retries (did you set -v=4?) or that every retry fails? On RHEL the lock is the xtables unix socket lock in iptables to allow multiple programs to update iptables concurrently. Other platforms may use a flock since that is the default upstream. To solve this kubernetes needs one more more of the following: pass the -w flag to iptables as scott and the logs mention (possibly only available on some platforms). If kubernetes desides to wait for the lock it will need a timeout. retry the firewall initialization quickly and maybe backoff if it still can't get the lock It's probably worth noting that if iptables on a platform doesn't have iptables lock support concurrency is likely broken already in other ways for that platform. — Reply to this email directly or view it on GitHub .
I can grab the lock for just the second that kubernetes needs it and then let it go. The kube-proxy nat'ing will never be rebuilt without intervention (restarting kubernetes). That seems like a bug to me and is what my original comment reproduces. In addition, once the environment is in this state it won't even pick up new service nat rules. Other nodes in my environment will pick them up but this one is dead. I highly suspect it has to do with the fact that iptables initialization is failing. We test all the time that once a system is up and running you can clean iptables and it will come back. This is a slightly different case.
This seems like a v1 blocker - we see this in every realistic environment we put the Kube-proxy in. Disagreement?
I still don't understand this issue. It is demonstrably true that the kube-proxy retries all of the iptables logic periodically. It's possible (given the mention of "won't even pick up new service nat rules") that this locking tickles a bug in kube-proxy, but I am having trouble reproducing it. Can you run your kube-proxy with -v=4, trigger the bug, and then take a look at the logs? If this is real, yeah it's a 1.0 blocker.
The problem is in the initialization process. If you hold the lock while the kube-proxy tries to initialize itself then it will be wedged forever. What you're showing is the kube-proxy can repair the iptables rules after the initialization code has run. The logs I pasted earlier show that after the initialization fails at it will never try again. I tend to run at loglevel 4 by default but I'll try to find time today to run through this again.
I have a fix coming - can you test the PR before commit?
I ran the exact same test as mentioned in my original comment. While the debug lines have changed the end result is the same: grab the lock (at this point even will fail) start the kube-proxy release the lock (at this point works) wait.. try creating another service wait.. The rules never come back. Here are the logs:
Sure, I'll test the PR. I'm betting it will apply fairly cleanly on to my OpenShift codebase. If not I'll launch Kubernetes directly.
Many of those log lines are not part of upstream. In fact, what upstream has is a glog.Fatalf() which will kill kube-proxy, and when it restarts it will try again. Did you guys break it by not actually exiting?
Looks like we did. We need to put it into a restart loop. ----- Original Message -----
is teh fix, though I assert we should not need it, and if you need it this fix may not help you anyway. I am still OK to commit this.
Although NewProxier sucks because it doesn't return a usable error. I'll take the todo to clean up that part. ----- Original Message -----
hey, it logs its error - nothing wrong with that :) , Clayton Coleman wrote:
:)
Opened and will move the rest back to openshift.
I restarted docker and it worked then


Following on from the Kubelet should accept YAML pod specifications. Currnelty it supports only JSON pod specifications.


As the title suggests, I can find no documentation on KUBEADMISSIONCONTROL (from the apiserver configuration), and I don't know golang well enough to determine the answer (yet) from the source code. Google search yields only 3 hits today, and all of them are copies of the configuration file itself with no commentary. As an admin, I'd be grateful for simple documentation on all the variables, their possible settings, and what those really mean. If that already exists, please identify where it exists. Though my formal issue is the documentation, the problem that leads me here is that my pods can't query the apiserver. Given the following, my connection is reset: Any assistance would be greatly appreciated, though my formal issue remains the lack of thorough documentation on the configuration. I'll continue learning golang in order to try to figure out what my problem may be.
I agree a document is needed, will look to put something together describing what each option does.
Thanks! :)
That being said, I recommend we scope this issue to just documentation being needed, and if you need assistance with your other topic, treat it as a separate issue.
Quite reasonable! If you think that documentation will be before the weekend, I'll wait until that's available and see if I can figure it out myself; otherwise, I'll make a separate issue for that.


We seem to have an occasional consistency issue between the client and server versions on our Jenkins GCE e2e project. Build 6295 failed this morning because the client version was newer than the server version: The server version matches the build was was immediately running in 6294, which was aborted early. I wonder if (somehow) the cleanup didn't work properly? (Which is a bit worrying if so.) Here's the version from 6294: A short while later, build 6297 failed, this time with the client version older than the server version: This one makes a little bit more sense: there was a new build published between downloading the client/test tarballs and setting up the cluster. I'm guessing that somewhere in the cluster setup scripts the latest CI release file was read again, rather than using the version read for the client/test tarballs. (I haven't yet worked out where exactly it's read, but it should be an easier fix.) cc
A side issue: we should probably figure out why the build git client is being marked dirty. Since we're checking out the head version from GitHub, it should ideally be a clean client.
The -dirty bit is now fixed by .
This is caused by different Jenkins jobs uploading to the same GCS bucket and clobbering each other. In we see: Over in
This also explains the issues seen in ; the job is also running in the same project.
Ah, good sleuthing. We avoided this issue in the PR builds by setting (used ), but maybe we should always be setting to include the Jenkins job name (and optionally executor/node number).
The original idea was to run every e2e job in it's own project, to avoid these sorts of name clashes. But we seem to have regressed and now have many jobs sharing the same projects. Is project-per-job not a better approach? It also helps with resource leaks and the likes. Just a thought.
I'd argue this is actually uncovering a (dev) deployment bug; I should be able to deploy different versions of Kubernetes to the same project. I know you and I disagree (and probably always will) on the strategy here. :smile:
fwiw I think I agree with that if project-per-job avoids this issue and helps identify resource leaks, it's a pragmatic near-term improvement in the state of the world that would result in more benefit than the cost of changing all the everythings to support multi-job-in-project. Re: devs deploying different versions of k8s to the same project, P3 feature request?
I was just about to suggest the same thing. I will file a separate issue, and fix this in the short term an easier way. is out to make it easier to isolate a run to a project.
Also it seems like a reasonable thing (higher than P2) to want to deploy multiple clusters to the same project safely in prod, not just dev. Disagree?
Multiple clusters in prod using well-defined build versions such as v1.1.4 = P0. Multiple clusters in prod using our cooked-up test-building system = non-goal?
SGTM; this is prod code, though, not test code; it's used every time we call , regardless of whether it's a CI build or released version.
FYI: kubernetes-e2e-gce-reboot/ has one reboot test failed due to kube-proxy image change. But that build doesn't include pr yet.
looks like it. That run used version , but the first build with was .


We need to be squelching the auth variables on upgrade anyways, but this complaint gets issued (and it proceeds anyways): cc


In case of cluster crash and status recovery, the persistentVolumeClaimBinder would not re-add available volumes to its internal index. Users would not be able to claim those volumes.
This is really bad. I think we should just nix Pending phase and always consider every Available volume for indexing - map lookups are cheap.


The "" E2E test will sometimes fail because it fails to delete the PD used in the test after the test is finished. The error: Detailed test log: The disk that fails to delete still shows up in : My attempts to manually try to delete the PD fail with: (the work around for ) doesn't change anything here:
is the disk still mounted ? It looks like the detach call did not succeed for some reason
I'll have to get another repro and verify.
okay cool. If you get a repro check to see if the disk is still mounted. If so then the cleanup code is not being called. If not then the clean up started, detach was called but failed for some reason; look in the logs for something like "googleapi: some error about detaching". It can also just be a race condition. That is, detach was called but is taking too long; if the disk does not detach then you cannot delete it.


If the container_probe test runs on a node without the nginx image, it sometimes takes 90s to get the pod in running. Most of this time goes in pulling the image, so I'm assuming it's registry related hiccup. Using might make it more reliable. Example failure: Note the following event in the logs:
a) we should not be pulling from docker in our tests - too hit-or-miss b) we should replace nginx (132 MB) with (4.5 MB) whenever possible. , Prashanth B wrote:


I'm conducting PoC about distributed build using Kubernetes. Jenkins master distribute jenkins slave pods through Kubernetes API Server. When build is done, jenkins slave pod is also deleted by kubernetes API Server. But, kubernetes didn't delete container. (Pod is deleted well) After searching, I found kubelet’s containerGC option. So, I adjusted below options to kubelet. --minimum-container-ttl-duration=0s --maximum-dead-containers=0 After adjusting both 2 options, container was deleted, too. But, kubernetes containerGC didn’t delete host volume path. (/var/lib/docker/vfs/dir/..) Due to aufs’s low performance, I used data volume of host instead of container inside. So, data volume was accumulated and host's inodes was fulled, too. Is there any solution to clean the containers wreckage? My environment was.. Docker: v1.6.2 Kubernetes: v0.18.2 (container based multi clustering)
are you using the host volumes you created yourself? If this is the case, kubelet does not clean the volumes since they are external to kubelet. If not, kubelet should clean the volumes before container GC kicks in.
No.. volumes were created by kubenetes API server. LGTM. Will it be adjusted next release?
You said "I used data volume of host instead of container inside" which says you used a hostPath mount - correct? That is explicitly NOT managed by kubernetes - you've gone to the host, you get to manage it. But you also said "Due to aufs’s low performance" - an emptyDir volume is NOT stored on aufs, but it IS managed with the pod. , gghonor wrote:
The issue as i understand it is dangling volumes, that is volumes that are created via a VOLUME directive in Dockerfile with no corresponding kubernetes volume in spec which therefore has no host path & is not cleaned up by the kubelet. These volumes reside in /var/lib/docker/vfs/. This waste of disk space has the potential to cause out of space errors on the node & therefore make the node unstable.
As the volume dir you mentioned is I assume you didn't specify any volume in your Kubernetes pod spec but did use the directive in your Dockerfile? As mentions I guess this isn't a Kubernetes managed volume as that would be in the Kubernetes volume dir, right? That dir is used by Docker for volumes specified in Dockerfiles or a docker container that is run with with no specified host path. The latter doesn't happen with Kubernetes managed containers AFAIK.
This makes sense. Thanks for the explanation :) I see the point of removing the dangling volumes. Even though kubelet doesn't manage those volumes, it doesn't hurt to remove them.
Do we have a full understanding of their lifetime, though? We don't control them, I don't see how we can know when to delete them. Docker's volume implementation is just too under-defined to actually be useful. , Yu-Ju Hong wrote:


The file: has instructions for AWS, GCE, and VMware. I'd like to see documentation for OpenStack as well. I'd be happy to do the work for this as I don't think it would take too much time. I'll add some instructions to this issue if its something that people think would be helpful.
I would greatly appreciate any work you can do with this. I've been trying, unsuccessfully, to get the OpenStack integration working.
Ok, I created instructions in this gist: If it looks good to you I'll create a pull request and merge changes in the document after the GCE section (as it appears we're going in Alphabetical order? )
Sending a PR sounds good.


If is passed to the kubelet, container metrics are not included in the Prometheus metrics endpoint.


The first line logged from the integration test is the last line from the service account token test is then the test panics with: I think the tests are just running too close the the timeout. On successful runs, the tests take ~115 seconds, which is ridiculously close to the threshold. For example: see
ok, do you want to send a PR to increase the timeout to 3 or 4 minutes?
Sure, I can do that tonight


Nodes Network when a minion node becomes unreachable [replication controller] recreates pods scheduled on the unreachable minion node AND allows scheduling of pods on a minion after it rejoins the cluster(Failed 17 times in the last 30 runs. Stability: 43 %)Failed with 0 retries
for triage
I'll look into this.
I talked to about this and he had a pretty good guess in about two seconds about what the problem is. ReplicationController sync method does this: but DaemonSet controller doesn't. (I guess when DaemonSet was copy-pasted from ReplicationController, this was accidentally removed.) Pod reflector and syncer are running in separate goroutines, and if pod reflector doesn't run then syncer will think there are no pods on the node.
To be clear: this is about the test Controller Manager should not create/delete replicas across restart which is the one that is actually failing.
I'm confused you're saying that this is about , but you're updating . I thought they are about two different tests...
Yes sorry, I put my comment in the wrong issue. Reassigning this one to you, and I'll copy my comment into the other one.
First failure: We check if all namespace are deleted. If other tests failed to delete their namespace this test will fail. I'm going to delete this logic as we already check if we have successfully deleted our namespace. See also for some more details.
Second failure: We disable network by running the following command: I checked and it seems that it results in ing master to hang. I suspect that Kubelet hangs on updating node status and depending when it breaks the connection (by timeout) it will fit in test timeouts or not.
What is the default timeout for connections using client libraries to apiserver? I'm checking the code, but can't find it.
I'm not sure if we set one, but we should. I guess I would expect to see that in the config, w/ default set by this function: , Filip Grzadkowski wrote:
So: let's add a timeout, and revert .
I think that adding a timeout and reverting are orthogonal. I we want to simulate network problems with node-master communication then using REJECT is better anyway. At the same time adding timeout makes sense for many other reasons.
OK then, let's add a timeout and test both DROP and REJECT-- no sort of network problem should hang a component.
I think it didn't hang. It just that the kubelet was retrying too slow and sometimes it fit test timeout, sometime it didn't.
Could we reduce the timeout to make it always fit the test timeout? , Filip Grzadkowski
I guess we can. I'll try tomorrow.
It seems we have a new type of failure: It seems that kubelet on a machine where network is blocked doesn't notice new pod. After reading logs I assume that kubelet doesn't notice that the watch is broken and doesn't try to relist everything. It notices new pod only after watch timeout.
Ugh, I hope we don't have to start sending watch heartbeats.
It seems that all recent failures are with the following error: which means the reason is here: - can you please take a look or delegate?
I did some more debugging of it. I was looking into this failure: There is something strange happening here: run performTemporaryNetworkFailure: succesfully block the load from kubelet: the succesfully wait for node to become not ready: some time nodecontroller successfully removes the pod: replication controller creates a new pod: here is something I don't undertand - because of some reason, the test doesn't observe that the pod actually disappeared. - any thoughts on it?
BTW - this is exactly the same situation in:
OK - I think this is related to default 30s deletion grace period.
Yes - it seems that this is exactly the case. I will send out PR fixing this out for review.
I'm reopening it for tracking. If there won't be any failures within next few days - I will move the test back from flaky and close this issue.
Thanks for fixing, .
It seems that the main issue is fixed. However, there was one failure in the last 30 runs: in which one pod simply didn't started (and it was "preparation" for the test, not the test itself).
During next 2 days there was exactly one more failure of this test: it is exactly the same as the previous one. However, those are duplicates of So I'm going to move this test out of flaky suite.


runs foo and then waits for it to be ready or terminated. If the pod is terminated, it prints the log, and if it's not terminated, it attaches the output. However, it does not handle the case that the pod terminates between returning the ready status and attaching, as can be the case in the : Options to fix this: Fix kubectl: If attach fails, fall back to printing the logs If attach fails, check the status again and if it's terminated, print the logs Fix the flaky test: Prevent the container from exiting (e.g. `sh -c 'echo running in container && sleep 30') Prevent the container from entering ready state (e.g. )
Thanks for debugging. I think we should probably do both: If attach fails and the container is now terminated, fall back to printing logs Make the e2e more robust by not exiting for a while to ensure time to attach. I'm happy to review PRs


I'm seeing a bug where the network plugin fails to configure networking, and yet the containers for this pod are still deployed (without networking configured). It looks like the following is occurring: 1) Kubelet attempts to create infra container in 2) Network plugin fails, returns error code. 3) Kubelet detects error and skips pod creation but does not tear down the infra container 4) On the next call, the kubelet detects that the infra container is running and deploys the other containers in the pod. The fix here is for the kubelet to clean up the infra container in when it receives the error from the network plugin and before it returns. Here are some logs illustrating the steps I described above:
Cleaning up infra container on network plugin error sounds reasonable/


Unit tests on our CI have apparently been failing for 2 weeks and no one noticed (oops!)... The PR that seems to have caused the failures is . The failing test is - . Full Test Log: I'm not sure why this is failing on our CI and not on shippable. Has anyone else seen something similar? That test only seems to fail on the second run through with . Our unit tests run in a container built specifically for testing k8s-mesos: The Dockerfile for the env: Is anyone else able to repro with the above env or their own?
/cc
that test was around a week ago, it's doesn't seem possible to me that it was failing since 2 weeks ago. Anyway, we have few duplications of this issue: this one And the PR where is working on: It's a little bit difficult to me to follow all the conversations in all the differente places about the same bug, could it be possible to unificate some? Since you provided the environment I will try and take a look to reproduce it locally. Thanks!
I think I've got a working patch in
Closing in favor of &


When trying to edit multiple resources, for example: will panic. Ref as the possible cause cc
if was the cause, any fix needs to happen in 1.1 as well, was already cherry picked there
Also shows that we need a test for this case.
is the cause. When we edit multiple resources (i.e., list-type objects), the list's Accessor.annotations (type ) is nil and will panic when UpdateApplyAnnotation tries to SetAnnotations ().


We have to spend times on figuring out what are the mount error code means from issues and . In , the error message is and is great for diagnosing mount problem. The console message better be returned rather than just error code.


Heapster pod: And logs: cc p0 for triage
Did the required node scope change?
Its either the API on the console or monitoring write scopes on the node. , Mike Danese wrote:
I think this affects only GCE
I will fix this issue since is on vacation.
Need to cherry pick so i'll leave open
is out for review.
has been closed and the cherrypick here () has been merged so I'm closing this issue. Thanks everyone for the debugging.


has TODOs for how to bring up cluster addons like monitoring and logging. We should fill these in. requested by user:


Currently watches only log when they complete, which can make analyzing kube-apiserver log files a bit tricky. (did component x never start watching after that list or did apiserver shut down?)